ML102460635

NSP000037-Revised Testimony of Northard/Petersen/Peterson-Root Cause Evaluation Report 01145695

ML102460635
Person / Time
Site:	Prairie Island
Issue date:	07/29/2008
From:	Xcel Energy
To:	Atomic Safety and Licensing Board Panel
SECY RAS
Shared Package
ML102460550	List: IR 05000282/2009009 IR 05000282/2009011 IR 05000282/2009015 Information Notice 2009-11, NSP000059-Revised Prefiled Testimony of Northard/Petersen/Peterson-NRC Information Notice 2009-11 L-PI-10-027, NSP000046-Revised Testimony of Northard/Petersen/Peterson-NSPM Response to NRC Annual Assessment Letter for Calendar Year 2009 ML102460551 ML102460552 ML102460553 ML102460554 ML102460555 ML102460556 ML102460557 ML102460558 ML102460559 ML102460560 ML102460562 ML102460563 ML102460564 ML102460565 ML102460566 ML102460568 ML102460569 ML102460615 ML102460616 ML102460617 ML102460618 ML102460619 ML102460620 ML102460621 ML102460624 ML102460625 ML102460626 ML102460627 ML102460628 ML102460629 ML102460630 ML102460631 ML102460633 ML102460634 ML102460635 ML102460636 ML102460637 ML102460638 ML102460639 ML102460715 ML102460716 ML102460718 ML102460719 ML102460720 ML102460722 ... further results
References
50-282-LR, 50-306-LR, ASLBP 08-871-01-LR-BD01, RAS 18555, RCE 01145695
Download: ML102460635 (90)

Text

NSP000037 QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 1 of 90 RCE REPORT Prairie Island Nuclear Generating Plant Component Cooling Piping Adjacent to HELB Location in Turbine Building Event Date: 7/29/2008 RCE: 01145695 RCE Team Members:

Dave Kettering/Scott Northard Management Sponsors Jeff Connors/Peter Wildenborg Team Leaders Christopher Lethgo System Engineering (Root Cause Investigator)

Nate Adams Design Engineering Ryan Cox Program Engineering Andy Notbohm Operations (Root Cause Investigator)

Kelsa Christopher Design Engineering (Monticello)

Dave Pennington Design Engineering (Monticello)

Deb Albarado Organizational Effectiveness Gene Woodhouse Performance Assessment (RCE Mentor)

Rob Sitek System Engineering James Sumpter CERTREC Consultant Kim Bromberek Administrative Assistant Betsy Rogers Training (Team Advisor)

Review:

Independent RCI Date Approvals:

RCE Team Leader Date Management Sponsor Date PARB Meeting Date NOTE: The above signatures may be documented via passport assignments.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 2 of 90 Table of Contents Page #

I. Executive Summary 3 II. Event Narrative and Timeline 12 III. Extent of Condition Assessment 18 IV. Operating Experience 27 V. Nuclear Safety Significance 33 VI. Reports to External Agencies & the NSPM Sites 35 VII. Data Analysis 36 A. Information & Fact Sources 36 B. Evaluation Methodology & Analysis Techniques 36 C. Causal Factors and Logic Ties Description 36 VIII. Root Cause and Contributing Causes 38 IX. Safety Culture 39 X. Corrective Actions (SMARTS) 40 XI. References 44 XII. Attachments 44 Note: The acronym CAP is used interchangeably in this RCE with the acronym AR. Both acronyms refer to an action request in the corrective action database.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 3 of 90 I. Executive Summary Problem Statement:

The station failed to ensure safety related functions of the component cooling water system were maintained for initiating events (HELB, tornado, seismic) in the Turbine Building.

Event Synopsis:

The genesis of this legacy issue began in December 1972 when the Atomic Energy Commission published a letter (the Giambusso letter) which required Prairie Island to address the consequences of pipe ruptures (high energy line breaks or HELB) outside containment and submit their analyses for review. Prairie Islands response is documented in Appendix I of the USAR. The station analyzed the effects of pipe whip, jet impingement, temperature and pressure from HELBs in the Auxiliary Building (AB) and the impact on the safeguards corridor in the Turbine Building (TB). The impact of HELBs in the TB on the Component Cooling Water System (CC) was not addressed.

There were updates to the HELB analyses for the AB in response to GL 87-11. Beginning in 1990, Prairie Island began identifying vulnerabilities with the CC with respect to single failure, QA classifications and QA boundary deficiencies that were discovered in response to GL 89-13. No actions were taken to address any of the vulnerabilities and operability of the CC system was only addressed from a high level perspective without full understanding of the issue. One of the vulnerabilities identified was that the surge tank could empty in six minutes if interface barriers failed.

In 1994, Prairie Island started work on an updated Turbine Building HELB analysis, though the focus was only on temperature and pressure impacts.

Completion of this analysis has been hindered by both technical and financial issues. In 2003 the site received an INPO AFI for Engineering programs not being managed effectively (including HELB) which resulted in commitments to INPO to have a Turbine Building HELB analysis completed by 2005. However, at the time of this investigation, the Turbine Building HELB analysis still has not been completed.

In 2005, the station decided to develop an Engineering resolution to the CC HELB issues in the TB for the cold chemistry laboratory piping independent of completion of the TB HELB analyses. However, progress on studies to develop the Engineering resolution have been delayed for various reasons and have not been completed at the time of this investigation.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 4 of 90 From 2000 through 2008, several opportunities existed for the CC/HELB interaction in the Turbine Building to be identified and placed in the Corrective Action Process. A few examples of these opportunities include (in addition to the INPO AFI in 2003):

• A 2000 Information Notice, IN 2000-20, regarding a HELB issue at DC Cook that mentioned HELBs in the Turbine Building.

• CAP 00737382, issued in 2004, identified seismic classification issues with the CC piping. An extent of condition review of the CC piping vulnerabilities was not conducted.

• Kewaunee OE from 2005 that notified the site of an event pertaining to a HELB interaction with the AFW Pump Suction line.

• July 2005 completion notes for CAP 00737382 state that the TB CC piping needs to be evaluated for HELB and tornado. A CAP is not written for this new condition. A decision is made to develop an engineering solution for the CC piping in the TB to the cold chemistry laboratory rather than wait for the results of the TB HELB analyses.

• August 2005 and July of 2006, Engineering Assistance Request (EAR) forms were presented to the Project Review Group (PRG)

Subcommittee for funding of studies to resolve CC/HELB issues but no questions were raised by this group regarding the significance of this issue. Draft studies from Sargent & Lundy state that CC does not isolate on seismic or tornado and that walkdowns found the CC piping near high energy feedwater lines. A CAP is not written for these newly identified conditions.

• Sargent & Lundy Q-List Report for the CC system received in December 2006 containing information regarding the need to upgrade the piping in the CC piping in the Turbine Building to prevent loss of CC system function. A CAP is not written because the study is not accepted by the station.

• Sargent & Lundy draft study received in June of 2007 (final received in January 2008) discussed CC vulnerabilities to a HELB, tornado, or seismic event.

Corrective actions completed between 2005 and 2007, as well as other project documentation (EARs and study proposals), indicate that concerns existed regarding the impact of a HELB, tornado, or seismic event on the CC piping. However, actions to address the concerns remained under the parent CAP for the seismic classification issues with the CC system (CAP 00737382), allowing the HELB and tornado vulnerabilities to remain at a low visibility level with no consideration of operability for any potential additional vulnerabilities. Studies investigating an Engineering resolution to the cold chemistry laboratory CC piping were not complete.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 5 of 90 In July of 2008, a walkdown of the CC/HELB interaction resulted in the initiation of CAP 01145695. Under this new CAP, the site addressed operability issues related to a HELB event in the TB and the impact on CC piping, which led to a fuller understanding of the significance of the issue.

Based on this discovery, the Unit 2 CC System was declared inoperable and action was taken immediately to restore operability to the CC System by isolating the CC system piping from the Turbine Building.

On August 5, 2009, Prairie Island received an NRC inspection report identifying a preliminary white finding pertaining to the CC/HELB interaction on Unit 2. This was followed by a final significance determination for the white finding on September 3, 2009.

Nuclear Safety Significance:

This evaluation found no significant evidence of Safety-Conscious Work Environment (SCWE) failures as part of this sequence of events.

However, there is evidence of a potential for resistance from site and fleet personnel to write a CAP when answers to an issue are not known, when an investigation is desired or when there are issues identified in studies or analyses received by the site. This, however, was not a significant contributor to this event. See Section V of this report for more information.

The events of this report have resulted in a White Finding from the NRC.

The finding is a violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control. The violation wording is as follows:

Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion III, Design Control, requires, in part, that measures be established to assure that the design basis for safety related functions of structures, systems, and components are correctly translated into specifications, drawings, procedures, and instructions. Further, Criterion III requires that the design control measures provide for verifying or checking the adequacy of designs.

Contrary to the above, as of July 29, 2008, the licensee failed to implement design control measures to ensure that the design basis for the component cooling water system was correctly translated into specifications, drawings, procedures, and instructions.

Specifically, the licensee failed to ensure that the safety-related function of the component cooling water system was maintained following a high energy line break, seismic, or tornado events in the Turbine Building.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 6 of 90 The root cause evaluation team evaluated Safety Culture Impacts for the root and contributing causes and for the extent of condition and extent of cause utilizing information in QF-0436 (Evaluation of Safety Culture Impacts) and NRC Inspection Manual 03-05. The evaluation is documented in Attachment 7 and summarized in the Nuclear Safety Significance Section of this report. Weaknesses in the following Safety Culture components were a root cause and contributing causes:

• H4c(Human Performance, Work Practices, Management and Supervisory Oversight)

• P1a(Problem Identification and Resolution, Corrective Action Program, Complete, Accurate and Timely Identification of Issues)

• P2a(Problem Identification and Resolution, Operating Experience, Systematic Evaluation of Relevant Internal and External Operating Experience)

Conclusions/Root Cause:

Conclusions:

This investigation has concluded that a number of failures occurred at Prairie Island that prevented the site from ensuring measures were in place and actions were taken to maintain the safety related functions of the CC system during initiating events in the Turbine Building. The main failure is that the site did not address issues like this one through the rigorous identification and timely resolution of known design basis deficiencies. Integral to this process is the knowledge of design and licensing bases requirements. The plant is required to be designed and built to meet all of these requirements. Additionally, all of these requirements need to be correctly translated into procedures, instructions, drawings, and specifications.

With respect to HELB, certain breaks of high energy lines were required to be assumed in the Turbine Building. However, the site did not have adequate documentation available to show that the breaks of these lines would not prevent other system, structures, and components (SSC) from meeting all their required safety related functions or even whether the proper breaks had been considered. This was a known deficiency but the significance of the deficiency did not appear to be well understood. This deficiency was pointed out to the site by INPO in 2003.

While activities were undertaken to identify and resolve CC TB HELB piping issues (TB HELB analyses to update the documentation and studies to develop an Engineering resolution for the cold chemistry laboratory CC piping), management of these activities was inadequate.

Engineering continued to pursue these activities with a normal priority Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 7 of 90 level, having only one engineer working on them part time and allowing them to be delayed by other emergent work. The activities were funded from the department line budget and therefore were not subject to the PRG review process. These continuing delays were not reviewed by the involved engineer or station management from the perspective of the increased risk of a legacy issue having potential further vulnerabilities that were remaining undiscovered. In addition, as a result, there were multiple cascading unintended consequences as discussed in Section VII.C.

There are two other contributing factors identified in this investigation.

One was related to the removal of barriers that may have identified the significance and complexity of this issue (e.g. more broadly focused Operability Determinations, management involvement, and independent evaluation of the vulnerabilities and the delays). All of these barriers were removed when CAPs were not written to document new identified conditions with the CC piping. A portion of this investigation centered on understanding why new CAPs were not written. Interviews indicated that personnel involved either did not recognize HELB as a separate issue from seismic or did not want to write a CAP until more information was available to show that there really was an issue. In addition, it was a practice not to write a CAP for issues identified in analyses or studies until those documents were accepted by the station.

When the significance of the issue was finally realized in July 2008, it was involvement of other engineers with a different mindset that led to a full understanding of the CC/HELB issue. There was an existing mindset (discussed in Section VII.C) regarding HELBs in the TB and the CC that hindered full understanding of the CC Licensing and Design Basis.

However, completion notes being added to AR 737382737382and statements from various draft studies examining options to resolve the cold chemistry laboratory HELB CC piping issue demonstrate that the station was beginning to grasp the significance of this issue. So, if CAPs had been written earlier in the process, it most likely would have resulted in the potential operability issue with the CC system being understood sooner and appropriate action would have been initiated. Also, the site had unclear guidance regarding the threshold for when to document a potential issue in a CAP.

Another contributing factor determined in this investigation is related to the missed opportunity that the site had to recognize the significance of the CC piping in the Turbine Building through evaluation of related OE. An extent of condition of the issues identified in OE was not conducted (not required by procedure) so that TB HELB impacts on CC piping were not identified.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 8 of 90 Root Cause:

There has been inadequate management of the Turbine Building HELB analyses and the cold chemistry laboratory component cooling water piping resolution studies.

The failure to effectively manage these activities so that they are still not finalized as of the date of this investigation has deprived the station of the opportunity to discover additional potential vulnerabilities in a legacy issue (HELB impacts in the Turbine Building) in a timely manner. This presents a challenge to the stations ability to complete a timely assessment of operability impacts and resolve identified issues.

There are two additional impacts that have resulted from the delay in completion of these activities:

1) A continuing lack of clarity and organization of the HELB documentation that is also not easily retrievable (as identified by INPO in 2003), and,

2) A continuing lack of understanding of the HELB Licensing Basis and difficulty in identifying and verifying design inputs and assumptions.

The consequences of the above impacts are:

1) Inadequate prioritization of HELB analyses and studies,

2) Related OPRs that are too narrowly focused,

3) CAPs not being written for related new conditions,

4) Related OE not being properly evaluated,

5) Incorrect assumptions regarding CC capabilities to handle certain events

6) Incorrect assumptions regarding Turbine Building HELB impacts on plant systems, and,

7) HELB analyses focused on temperature and pressure impacts to the exclusion of jet impingement and pipe whip While the RCE problem statement discussed initiating events in the Turbine Building for HELB, seismic, and tornado, the focus of this Root Cause is only on HELB. The reason for this is that HELB is the only one of the three where this issue could have been discovered as part of resolving known deficiencies. Documentation showing the ability of the site to mitigate a HELB in the Turbine Building is required documentation that the site does not have but was working on obtaining. If there had been adequate management of the HELB analyses and cold chemistry CC water piping resolution studies, these activities would have been completed and the required documentation and Engineering resolution in Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 9 of 90 place so that this issue would have resolved much sooner. For seismic and tornado, however, it is not likely that this issue would have been identified through routine efforts.

Contributing Cause:

Contributing Cause #1: Station management has not developed adequate standards for OE evaluations with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification.

Contributing Cause:

Contributing Cause #2: Engineering management has not developed expectations pertaining to CAP initiation for:

1) How long a potential issue can be investigated before it is documented in a CAP, and,

2) When a CAP should be written for valid issues identified in draft or otherwise unaccepted studies.

Corrective Action Synopsis:

The Root Cause is addressed by two CAPRs and two supporting CAs that work together to correct the root cause and ensure sustainability of the corrective actions. CAPR #1 develops and implements a HELB design basis document and program document. This effort will establish the HELB requirements at Prairie Island and complete actions necessary to ensure the site is in compliance with the requirements. The supporting CA

1. 2 and CA #3 determine the short term and long term personnel resource requirements for sustainability of the HELB program and develop a business case for these resources.

CAPR #2 revises 5AWI 6.0.0; Integrated Planning Process to ensure projects funded by department line budgets are subjected to the site project review process through the PRG and the tracking of all on-going PRG-approved O&M studies and analyses. A periodic status update of these activities would be provided to PRG. If any activity has been delayed or the scope changed, PRG will review the prioritization of the activity. Emphasis should be placed on those activities involving or potentially impacting risk-significant SSCs, particularly those activities still in the discovery stage. Depending on the nature of the study or analyses (for instance, when an OBN or OBD involved), a plan should be generated to recover the delay or justification provided for the scope change.

EFR #1 for CAPR #1 will assess the effectiveness of the HELB design basis and program documentation efforts by performing an external review of the program. This assessment will ensure that any discovered Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 10 of 90 deficiencies have been placed in the Corrective Action Process and that the program provides an adequate basis for future operability assessments involving HELB. EFR #2 for CAPR #2 will assess the revised PRG process and determine if it has been effectively implemented through personnel interviews and document reviews.

The associated extent of root cause is addressed by the following CAs:

CA #4 directs the development of program basis documents for other non-fleet programs to capture essential program elements in one location. CA

1. 5 identifies other studies and analyses that have the potential for discovering additional vulnerabilities to operability.

Contributing Cause #1 is addressed by TRRA #2 that evaluates training for all site personnel who currently perform OE evaluations. PCRA #1 will revise the fleet OE procedure (FP-PA-OE-01) to implement explicit requirements to consider an extent of condition evaluation be performed for OEs.

Contributing Cause #2 is addressed by CA #1 for the development of an expectation document covering the unique aspects of Engineering issues that could be potential CAQs. The document will address the management expectations for how long these issues can be investigated prior to the initiation of a CAP and when a CAP should be initiated for issues identified in studies and analyses. TRRA #1 directs training for all Engineering personnel on the revised Corrective Action Process expectations developed by CA #1. PCRA #2 will revise fleet Engineering guidance (FG-E-ARP-01) to incorporate the revised CAP expectations.

Reports to External Agencies:

The following reports were made to external agencies:

• August 7, 2008, CAP 01145695 was posted in the Internal Operating Experience Report (form QF-0407) that was submitted to the fleet.

• September 29, 2008, LER 2-08-1, Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break was submitted to the NRC.

• October 8, 2008, the CC/HELB issue was posted on the Nuclear Network as OE27559 related to CAP 01146027, which was closed to CAP 01145695.

• January 19, 2009, LER 2-08-1, Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break, Supplement 1 was submitted to the NRC.

• July 16, 2009, CAP 01145695 was again posted in the Internal Operating Experience Report that was submitted to the fleet. The Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 11 of 90 reposting contained additional information related to the CC/HELB issue as well as documentation that this was an NRC white finding.

Additionally, when this RCE is complete and approved by the PARB, the following actions will be taken:

• A follow-up posting will be made internally and the report will be shared with the fleet.

• OE27559 will be evaluated for update

• The NRC will be notified of the completion of the RCE and a copy will be provided.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 12 of 90 II. Event Narrative and Timeline Historical Information:

While the main scope of this Root Cause Evaluation focused on events occurring after 1990, there were a couple of historical events that should be mentioned as they are important with respect to an overall understanding of this issue. Based on the teams understanding of the historical events, the original design of the PINGP included the Component Cooling water being routed out of the Auxiliary Building into the Turbine Building. No information could be found regarding the details of why this was done or what the justification was for this design.

In December 1972, the Atomic Energy Commission published a letter (the Giambusso letter) which required Prairie Island to address the consequences of pipe ruptures (HELB) outside containment and submit their analyses for review. Prairie Islands response is documented in Appendix I of the USAR. The station analyzed the effects of pipe whip, jet impingement, temperature and pressure from HELBs in the Auxiliary Building (AB) and the impact on the safeguards corridor in the Turbine Building (TB) from HELBs. The impact of HELBs in the TB on the Component Cooling Water System (CC) was not addressed.

In 1977, based on the notes from Q-List Committee Meeting 77-12, the CC piping in the Turbine Building was downgraded to QA Type III. Again, no information could be found as to why this was done or what made it acceptable. While this downgrade may have played a role in the mindset of the Engineering staff during future investigations, it was not the main reason for the failure of the site to adequately protect the CC system.

In the late 1980s, Prairie Island updated the AB HELB analysis in response to GL 87-11. The analysis has been updated several times since then.

RCE Event Narrative:

In 1990, based on response to NRC Generic Letter (GL) 89-13, Service Water System Problems Affecting Safety-Related Equipment, Recommended Action IV, Prairie Island Nuclear Generating Plant contracted Pioneer Engineers & Consultants, Inc to prepare a single failure analysis of the Component Cooling (CC) System for Units 1 and 2 (documented in 1995 as ENG-ME-240). The completed analysis recognized several single failure and QAI-to-QAII/III boundary barrier deficiencies, and discrepancies in QA classifications. Recommendations were made for system improvements that would resolve the identified deficiencies (D6). One of the vulnerabilities identified was that the surge Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 13 of 90 tank could empty in six minutes if interface barriers failed. Follow-on-Item (FOI) A0108 focused on the SI pumps. An operability determination did not identify any immediate actions or address the operability of the CC system (D44). (Inappropriate Action #1)

During this same period of time the site was resolving other Cooling Water issues with greater perceived safety significance in preparation for the 1995 Service Water Safety Operational Inspection (SWSOPI) to demonstrate compliance with GL 89-13.

FOI A0863 was completed in May of 1995 to address concerns raised by the 1990 single failure analysis of the CC System for Units 1 and 2. The study indicated that the CC system had inadequate isolation in the Turbine Building. The FOI concluded that no impact on component or system operability existed, and no Justification for Continued Operation was required (D49). In addition, there was no recommended immediate or short term corrective actions since the FOI had no impact on operability or technical specifications. (Inappropriate Action #2)

A Turbine Building HELB analysis, Calc ATD-0312 was completed by Sargent & Lundy in December of 1994, of which Rev. 0 was received on December 8, 1994 and Rev. 1 received on April 19, 1995. In May 1999, Condition Report 19991622 (D52) was initiated following receipt of an S&L letter documenting errors and the affected calculations were cancelled. This calculation was also being used to support SE-419, Removal of Turbine Building Steam Exclusion Dampers which was completed on September 5, 1995 and rejected by the Operating Committee the following day. This project, 93L427, was cancelled a year later and SE-419 was cancelled in July 1997.

Work on the Turbine Building HELB analysis (temperature and pressure only) began in September of 2000 by contractor AES, but was terminated in 2001 as adequate funding was not available (D53). Additional funding was requested in April of 2002. The analysis was scheduled to be completed in November 2005.

In September 2000, IN 2000-20 was evaluated by the site concerning a HELB issue at DC Cook (D63). Though the IN mentioned HELBs in the Turbine Building, the site only evaluated safety-related equipment affected by HELBs and did not review HELB impact on the CC in the TB.

(Inappropriate Action #3)

In May of 2003 an INPO Evaluation was completed and resulted in AFI.2-2 which required a response by Prairie Island (D51). The AFI described that some Engineering programs and their supporting analyses and documentation, including environmental qualification, high-energy line Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 14 of 90 break, in-service inspection and the quality list, were not well organized and easily retrievable. Prairie Islands response, specifically with respect to high-energy lines, is below:

The High Energy Line Break (HELB) issue has been organized into two major tasks; Auxiliary building will be completed by December 2004, and Turbine building which will be completed by November 2005.

As of the completion of this investigation, the Turbine Building HELB analysis has yet to be finalized.

In April 2005, Prairie Island received OE 20291 (D30) reporting an event at Kewaunee NPS (then a member of the Nuclear Management Company).

The event pertained to an Auxiliary Feedwater Pump Suction / HELB interaction affecting operability of the Auxiliary Feedwater System. Rapid OE notification was conducted and a formal evaluation was conducted under OEER 810473 (D45). The OE evaluation focused on the AFW system and did not evaluate extent of condition (Inappropriate Action #6).

The RCE team also found that the Rapid OE and OE Report contained specific recommendations to perform extent of condition for other HELB issues.

In August of 2004, a year prior to the above OE, CAP 737382 (D3) was written indicating Component Cooling components were not Seismic Class 1 qualified. Several assignments were created from this CAP, however, no extent of condition evaluation was performed (Inappropriate Action #4).

Some of the assignments from this CAP include (but are not limited to) the following:

• 737382-1 Perform an OPR to determine potential non-conformance of CC system with respect to the non-seismic components

• 737382-2 Evaluate and if necessary initiate corrective actions

• 737382-3 OBD (00109) to track the plan for resolution of the operable but nonconforming condition.

• 737382-4 Perform seismic analysis of 1-CC-138 up to CC-71-1 and CC-71-2.

The operability assessment (OPR #509) only considered whether CC was on the Safe Shutdown Equipment List (SSEL) and was too narrowly focused (D8). (Inappropriate Action #5).

Completion notes to Assignment 4 of AR 737382737382dated 07/20/05 (D3) indicated that, the CC piping to Cold Chemistry laboratory and 123 Nitrogen Compressor was to be further evaluated and may have to be modified or isolated in view of the revised HELB analysis. A new CAP was not written even though the mention of HELB was a new condition. The HELB issue remained under the seismic AR. (Inappropriate Action #7).

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 15 of 90 From these completion notes, it is apparent that a three phase plan was put in place to complete the study and initiate modifications that would be necessary to resolve the existing seismic and HELB conditions with the CC System. An Engineering Assistance Request (EAR) 050120, Component Cooling Piping to Cold & Hot Chemistry Labs are not seismically qualified &

HELB is a concern (D26) was prepared and reviewed at the 08/30/2005 PRG Screening Meeting. The EAR noted the presence of safety related piping, pipe supports, and sample coolers in a non-safety area of the Turbine Building may require extensive analysis to justify their ability and that of the cold chemistry laboratory to withstand seismic and tornado loads. It was decided to develop an Engineering resolution to the problem rather than wait for the completion of the Turbine Building HELB analyses.

Initial funding was approved for the amount of $10,200 to perform a study.

This study was performed by S&L and recommended two options (D2):

1. Modifying CC Piping in order to isolate the Cold and Hot Chem. labs, chillers and 123 Nitrogen pump from safety related CC piping during an accident

2. Analytically qualify nonsafety related CC piping and nearby block-walls for all applicable loading conditions described in the USAR (this option was not recommended).

Option 1 for modifying CC piping to automatically isolate during an accident was determined not to be a feasible option and the study phase was started again. However, the study did indicate that the CC system will not isolate on seismic or tornado events. A synopsis of the 2005 study was prepared by a site structural engineer. That synopsis indicated that a walkdown found that CC lines to the Cold Chemistry Laboratory were very near high energy feedwater lines. However, the significance of this HELB interaction was not recognized. (Inappropriate Action #8).

The synopsis was used to contact three Architectural Engineering (AE)

Firms to provide Requests for Proposals. Three proposals were received from the AEs; one from Sargent & Lundy dated 02/28/06, one from AES dated 03/03/06, and one from Stevenson & Associates dated 03/02/06.

(D12, D25, and D62)

An Engineering Assistance Request was prepared that included the additional information gathered from the proposals received in February and March. This EAR was presented at the 07/24/06 PRG Screening Meeting (D59) and was reviewed and approved pending the outcome of the 2006 budget recovery actions. Sargent & Lundy provided an additional Proposal for Preparation of Refined Alternatives for Isolation of the Component Cooling (CC) System Piping to Cold Chemistry Lab February 15, 2007. This proposal was accepted, and on March 30, 2007 a contract Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 16 of 90 was issued to S&L for preparation of alternatives for isolation of the CC piping to the cold lab.

In December of 2006, the Q-List Report for the CC system was received by the site from Sargent & Lundy. The report was subsequently reviewed by five different site engineers, including the CC System Engineer and the HELB Engineer. The Q-List Report for the CC system discussed the vulnerabilities with the CC piping in the Turbine Building and the need for the piping and associated components to have a safety function of maintaining system pressure boundary. The document also discusses the drain down of the CC surge tank in approximately 6 minutes based on a break in a 3/4 schedule 80 pipe. No apparent action was taken as a result of these reviews by site Engineering since the report was not accepted due to errors in other portions of the report (Inappropriate Action # 9).

In June 2007, the Turbine Building HELB analyses contractor was changed from AES to Advent. The study is now seven years old and still not completed. The longer the delay in completion, the greater the chance that other potential vulnerabilities due to this legacy issue will not be discovered so that they can be assessed for operability impacts. The site failed to recognize the significance of these continuing delays. (Inappropriate Action #10).

On December 24, 2007, assignment 00737382-12 was completed and indicated that CC piping would not be able to withstand a HELB condition or tornado loads. CC System operability was not reassessed at this time as no new CAP was written. This was another opportunity to identify a new condition on the CC system similar to that with the completion notes to AR 737382737382in July 2005 (Inappropriate Action # 11) (D9).

A draft copy of the Sargent & Lundy document titled Chemistry Lab Component Cooling Study was received on site in June of 2007 for review (D28). No review was performed and there was no turnover provided about the study by the HELB engineer who had left the company. However, in December 2007, ECR 3183 was written to add a closed loop cooling system for the CC cold chemistry laboratory based on recommendations in the draft Sargent & Lundy study. (Inappropriate Action #12).

The final study was received in January 2008 (D28). The background section of the report discussed the purpose of the study. This section of the study presents the information that PINGP calculation ENG-ME-240 concludes that a 100 gal/min pipe break would drain the surge tank in approximately 6 minutes. It goes on to mention none of the piping to or from the cold chemistry laboratory has been analyzed for HELB and the CC piping is not analyzed for tornado impact. The study documented recommendations to address concerns with the CC/HELB interaction. A Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 17 of 90 subcommittee of the Study Review Board reviewed the study but did not read the background section and only focused on the recommendations.

The study was not accepted because the subcommittee was not sure of the purpose of the study. Not only was the significance of the information in the background section missed but the resolution of the cold chemistry laboratory CC piping issue continued to remain open, over 31/2 years after it was recognized that this issue needed to be resolved with an Engineering solution. (Inappropriate Action #13).

The governing procedure for engineering studies at the time this S&L study was received was SWI ENG-26, Development of Engineering Studies, Rev. 1. This procedure was not followed for this study. Per this procedure, externally prepared studies should receive an owners acceptance review and SHALL be reviewed by the project manager and approved by the project sponsor. Neither of these actions was completed for this study.

The procedure adherence aspect of this was addressed in ACE 01162511-01 (D2).

On July 29, 2008, AR 01145695 was initiated following a walkdown by design Engineering. This walkdown identified the same vulnerability between the CC lines and a feedwater line that had been identified in the 2005 timeframe. During the operability determination phase of this newly initiated CAP, it was determined that a HELB near the Cold Chemistry Lab had the potential to adversely affect the function of the entire CC system.

Therefore, at 13:45 on July 31, 2008, the station entered TS LCO 3.0.3 due to both U2 CC trains being inoperable. The CC lines to the Cold Chemistry Lab were isolated at 16:12 on July 31, 2008 which allowed the station to exit LCO 3.0.3. An Operability Recommendation (OPR) was completed by Engineering on August 1, 2008 which concluded that the Unit 2 CC System was operable but non-conforming.

On December 15, 2008, AR 01162511 (D2) was written to document missed opportunities to identify CC/HELB interactions from the Sargent &

Lundy study. An Apparent Cause Evaluation was conducted. The causal statement documented in the ACE was that Engineering did not properly follow the procedures for acceptance of the engineering study (S&L Study from January 2008), which was complicated by the wrong mindset when the staff requested the study.

On March 23, 2009, AR 01174370 was written to document another vulnerability with the CC system. CC piping to the 122 Spent Fuel Pool Heat Exchanger was not protected from a design basis tornado generated missile. This issue was discovered based on a question that came up during the significance determination process (SDP) for the CC/HELB issue.

ACE 01174370-02 was completed on 6/11/2009 for this issue. The causal statement documented in the ACE for this issue was that the lack of tornado Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 18 of 90 missile protection was most likely due to the addition of this heat exchanger by Pioneer to the original Westinghouse design without documenting acceptability of the change in configuration, protecting it to the same requirements as the 121 SFP HX, and clearly and permanently identifying the difference from the original heat exchanger.

Also discovered during the SDP for CC/HELB was a Turbine Building Flooding issue. On April 15, 2009, AR 01178236 was initiated due to the site not having a HELB flooding calculation for the Turbine Building. A CE was completed that performed a walkdown of the Turbine Building for flooding concerns related to high energy line breaks.

On August 5, 2009, Prairie Island received an NRC inspection report identifying a preliminary white finding pertaining to the CC/HELB interaction on Unit 2. This was followed by a final significance determination for the white finding on September 3, 2009.

During the conduct of this evaluation, it was determined that Prairie Island has not met the commitment made to INPO in 2003 to complete the Turbine Building HELB analysis by November 2005. AR 01192814 was written to document this issue.

III. Extent of Condition Assessment

• Extent of Condition Root Cause Extent of Condition:

The condition present in this event was that the CC system could not perform its safety-related function if a HELB occurred in the Turbine Building and severed the CC piping. The question to be evaluated in this extent of condition is: Are there other plant systems or equipment that cannot perform their safety-related functions during the same or similar events? While it is possible that other latent design issues remain from the time of original construction, a complete verification of the entire as-built configuration of the plant is beyond the scope of this evaluation. Additionally, it is difficult to identify problems that exist but have not been documented in the Corrective Action Process.

Various searches were made to capture sources of past data collection to capture potential issues that may exist in the plant that are not being supported, widely known, or possibly being misunderstood. In April 2003, a team of site personnel was established to respond to SOER 02-04 regarding the Reactor Pressure Vessel Head Degradation at Davis Besse.

The recommendations and team assignments were to interview plant personnel, perform an assessment of site personnel views on Nuclear Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 19 of 90 Safety, and identify and document abnormal and long term unexplained plant conditions. Results of the SOER evaluation identified 438 equipment, design, and process issues. The issues were ranked by the team based on safety significance and worst case outcome. None of the process issues were graded. Long term program concerns such as PRA, EQ, Fire Protection, and Maintenance Rule were identified by site personnel as lacking adequate funding and personnel to meet their respective program goals. Those issues that met the highest significance criteria were assessed in AR 0044901. (D42)

In May 2005, an all hands equipment reliability program was established.

Site personnel were requested to anonymously document any concerns they had with system operation, reliability, or system problems they thought were not getting adequate attention. Some of the issues identified were already documented in the Corrective Action Process but many were not.

Development of the 5 and 10 year plans were initially developed using these lists along with System teams. Most of the items identified were specific equipment or systems although comments were made to finish the HELB analysis and HELB related issues. A variety of concerns were identified regarding low design margins and issues.

In August 2009, Business Planning and Development generated a list of PRG approvals given for study phase money since 2005 per CA 01162511-

12. A number of the studies have never been returned to the PRG with their results. A number of design related issues are on the open list. This list is with the Engineering Design group for review and to address the products generated. Assignments for resolving these issues are included in AR 01162511. CA #5 also addresses any remaining issues. (D78)

As part of this root cause, revision 2.2 of the Probabilistic Risk Assessment (PRA) models was used to determine the plant systems that had the highest risk significance with different initiating events. The Top 11 systems/components were chosen from the list. Interviews were conducted with System Engineering owners to discuss if any issues were present with their systems regarding initiating events or any other potential design basis concerns such as seismic, tornado, and HELB. Systems reviewed were Cooling Water, Aux Feed water, Station Air, Screen House Vent, Component Cooling, Reactor Coolant, RHR, SI, SVCS, Reactor Protection and RWST. Interview results indicated that System Engineers were confident with the system design basis but in some cases may need additional support when justification is needed. System knowledge regarding design takes time to develop (several years) which may not always occur due to personnel or process changes taking place. Those interviewed believed a better mentoring process was needed to take place in lieu of the turnover checklist currently in use.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 20 of 90 An additional aspect of the extent of condition is related to identification of any additional vulnerabilities that may exist with the CC system or a HELB.

Corrective Action (CA) 01145695-02 was completed on 8/2/08 to address the extent of condition with respect to HELB issues in the Turbine Building.

This action was completed through the approval of EC 13000. This EC documented the extent of condition investigation to determine if any additional equipment located in the Turbine Building would be susceptible to pipe whip and/or jet impingement following a HELB. The result of the evaluation was that there are no additional concerns for the pipe whip or jet impingement for equipment located in the Turbine Building. This will be validated by the completion of CAPR #1.

Other vulnerabilities are related to tornado missiles. An issue was identified on March 23, 2009 in AR 01174370 that documented the fact that CC piping to the 122 Spent Fuel Pool (SFP) Heat Exchanger was not protected from a design basis tornado generated missile. As documented in ACE 01174370-02, on 5/15/09, a structural engineer, an Engineering supervisor, and a contractor inspected the Auxiliary Building, Turbine Building, and Screen House walls and doors from inside and outside and roofs and roof structures from outside for vulnerability to applicable natural phenomena of tornadoes, external flooding, and design snow load. No discrepancies noted and all open issues were resolved. Additionally, as part of this CAP, CE 01174370-07 was completed to perform an extent of condition with respect to a HELB in the Auxiliary Building. The CE determined that no HELB concerns were noted in the Auxiliary Building.

As a result of the CAP written for the SFP heat exchanger, Operations identified an additional concern in CAP 01174493 with CC piping going to the ADT evaporator and hydrogen recombiner. These lines are located in the fuel handling area where they could be susceptible to missiles from a tornado. As a result of this issue, an extent of condition was performed per CA 01174493-04. For this extent of condition, a walkdown of the fuel handling area was conducted. This walkdown determined that there was no additional CC piping or other equipment which would require missile protection located in the area.

On April 15, 2009, AR 01178236 was initiated due to the site not having a HELB flooding calculation for the Turbine Building. A CE was completed that performed a walkdown of the Turbine Building for flooding concerns related to high energy line breaks. Several large pipe interactions were noted on each unit. Some of these interactions have been screened out based on Appendix I of the USAR.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 21 of 90 Conclusions and Actions Needed Based on the extent of condition actions that have already been completed, no additional actions will be generated. The concern expressed by Engineering personnel regarding inadequate turnover is being addressed by CA 01165133 from RCE 01165133.

Contributing Cause #2 Extent of Condition:

An Extent of Condition was performed for Contributing Cause #2 because uncertainty regarding CAP initiation is considered a significant contributor to this event.

A missed opportunity to identify the significance of the CC/HELB issue found during the course of this investigation was the presentation of an EAR (Engineering Assistance Request) and RPA (Request for Phased Approval) to the PRG sub-committee to request funding for a study to determine solutions to the vulnerability of the CC piping in the Turbine Building. When the CC/HELB issue was presented to the PRG sub-committee, there was not a separate CAP, potentially contributing to a lack of awareness of the issue.

A sampling of RPAs was reviewed to ensure there was a CAP associated with the issue being addressed.

RPA Title AR Status Modify Louvered Fire Doors 1022720, 1026878 In progress for NFPA Code Compliance Steam Exclusion Damper Multiple ARs and an LER Study in 2010 Replacement Study related to SE Dampers in 1998.

Proposal to Prepare Scope Appears to be a maintenance Study for the Replacement of issue with multiple CAPs Cooling Water (CL) Valves written.

Unit 1 and Unit 2 Pressurizer 1156123 Study in 2010 PORV Backup Air Bus Load Sequencer Multiple CAPs written to In Study phase Processor Upgrade document MRFF.

D5/D6 Crankcase Breather Multiple CAPs written to EC 11013 currently at Modification document unplanned LCOs. a Canceled status Statements of LCOs being entered and this project would resolve the issue.

Replacement of References that multiple Study in 2010 Westinghouse DB-50 CAPs have been written on Breakers the breakers.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 22 of 90 Nuclear measurements Multiple CAPs written to Study in 2010 Corporation (NMC) Radiation document the issue. Concern Monitoring Upgrade Study in the RPA is failing equipment.

The RPA regarding the Louvered Door is similar to the CC/HELB issue as it relates to not ensuring design requirements were met. From the RPA it appears the site did not completely understand the requirements for the fire areas with the installation of the louvered doors. There does appear to have been appropriate use of the Corrective Action Process once the deficiency was recognized.

In most of the issues reviewed, the RPAs were written to address specific equipment issues. Accordingly, there were references to CAPs that had been written on the piece of equipment or statements that specifically state or imply operability/functionality was addressed.

Conclusions and Actions Needed Based upon this review of same and similar processes and issues, it could not be determined that attempting to solve problems without formally documenting them in the Corrective Action Process is an ongoing issue. No additional actions are required to address this extent of condition for Contributing Cause #2.

• Extent of Cause The root cause was inadequate management of the Turbine Building HELB analyses and the cold chemistry laboratory component cooling water piping resolution studies. The Extent of Cause looked at other areas where problems exist that are related to the development of programs important to ensuring the plant meets design basis requirements.

RCE 1182488, 12 Circ. Water Pump lock out and RX Trip. The root cause was found to be that the Cable Condition Monitoring Program development and implementation was not given sufficient priority.

AR 1132987, EIC Programs not recognized as Fleet Programs and AR 1148972, Motor Program not adequate for industry standards and site needs - both address programs that may not be receiving the proper priority.

AR 576240576240 INPO AFI EN.2-2 Engineering Programs. In 2004, INPO identified that some Engineering programs and their supporting analyses and documentation, including environmental qualification, high-energy line break, in-service inspection and the quality list, are not Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 23 of 90 well organized or easily retrievable. Weaknesses in this area make it difficult for station personnel to readily establish licensing basis requirements Management oversight of these programs is not sufficient to ensure program effectiveness.

These identified issues demonstrate that the extent of cause extends to other programs. CA #4 directs a review of non-fleet programs that currently do not have a program basis document (H-series procedure). This action should identify programs, other than HELB, that currently do not have a program basis document that may be required so that operability or other evaluations potentially supported by those programs may be conducted.

Contributing Cause #1 is that Station Management has not developed adequate standards for OE evaluations with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification. The extent of cause looked at a sample of recent OE evaluations to see if the extent of condition was adequate, as well as weaknesses with the OE process as a whole.

A draft of Common Cause Evaluation 01183142, Trend in Ineffective Resolution of OE Items, identified the Common Cause to be a Lack of Clearly Defined and Consistent Priorities, Lack of Functioning Engineering Work Management System, and Incorrect Operational Focus. Some of the items evaluated also show signs of incomplete understanding of the issues. The incomplete understanding of issues is revealed where issues were identified but resolution has not been completed. There is also evidence from previously-evaluated operating experience noted in other Root Cause Evaluations, specifically RCE 01132717 (Site Response to Issues with SI-9-5) that extent of condition and cause are not adequately addressed in OE evaluations.

The lack of adequate standards for OE evaluation is considered a site issue.

TRRA # 2 directs training of all personnel conducting OE evaluations to ensure that extent of condition reviews need to be considered when performing OE evaluations.

CC # 2 was that Engineering Management has not effectively communicated expectations of how long a potential issue should be investigated before it is documented in a CAP and when a CAP should be written for valid issues identified in draft or otherwise unaccepted studies.

The extent of cause for this Contributing Cause looked at challenges with using the Corrective Action Process across the entire site.

Comments from the 2009 Problem Identification and Resolution (PI&R) inspection exit meeting showed a concern with the implementation of the Corrective Action Process. The NRC identified that procedures associated Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 24 of 90 with Unit 2 Heater Drain Tank Pump, specifically the swapping of the pumps, did not meet the standard or requirements associated with Procedure Use and Adherence. This is an example of not properly identifying a deficiency that has been in place for a long period of time.

RCE 1141755, Identified NRC Crosscutting Issues, identified issues with the implementation of the Corrective Action Program. The RCE focused on resolution of issues and had actions to revise the Operational Decision Making procedure to provide the right level of response to degraded plant conditions and to train on the process. This is applicable to this RCE as it shows additional issues with the implementation of the Corrective Action Process.

Assignment 01 for CAP 01075890, PI125 PCD Scoping Enrichment Error, evaluates the issue identified for a department clock reset. The evaluation document for this CAP identified a timeliness problem in that there was indication of a problem on January 18, 2007 but a CAP for the issue wasnt written until February 6, 2007. This is applicable to this RCE as it shows the initiator not having a clear expectation as to the appropriate timeliness of CAP initiation.

Assignment 01 for CAP 01039647, Resetting of 1LM-750 using cabinet RESET switch, evaluates this issue for a clock reset. The evaluation document states the following: A CAP was not written by the SE as he felt that all necessary actions had been put in place to correct the lock up of Train A ICCM and at the time ICCM was not required to be operable. The idea that a compensatory action was being performed to keep A Train ICCM operable by resetting the ICCM rack weekly was overlooked by the SE and by the Crew 2 STA. Had a CAP been written to document the need to reset ICCM based on the replacement CMOS board giving a Ram Error code one week after replacement, the idea of a compensatory action being performed would have been caught during screening. This would have resulted in the issuance of an OPR at that time which in turn would have prevented the unplanned LCO entry on 7/12/06.

This is applicable to this RCE as the mindset demonstrated by the engineer and Operations is very similar to the mindset demonstrated by the engineers involved with the CC/HELB issue. The initiation of a CAP was overlooked as actions were in place to address this issue.

CAP 01121831, CAP not written for water leaking from 12 Desurger regulator. This CAP is also applicable as it relates to a CAP not being written when an equipment issue was identified. The individual who observed the issue did not write a CAP or inform the control room.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 25 of 90 CAP 01164967, CAP Not Issued for Breaker As Found Data Out of Tolerance. This CAP documents that a breaker PM work order was completed with notes that the as found testing was out-of-tolerance for one of the tests but no CAP was ever issued to identify this condition. This is applicable to the RCE as it is a situation where a CAP should have been written but wasnt.

Two recent CAPs demonstrate that expectations for when to initiate a CAP and the information needed to write a CAP are not clearly understood by the site. AR 01193003 documents unclear expectations for CAP documentation following management observations. AR 01193499 documents that some CAPs have an insufficient description of the issue. An adequate description of the issue aids in understanding the significance of the issue and initiating appropriate actions to resolve the issue. These CAPs indicate that there is not alignment on expectations for CAP initiation.

The above information demonstrates that while this contributing cause evaluation is centered mainly on the Engineering department, it can also be applied to the site as a whole. RCE 1211532 addresses the CAP initiation issues for the whole site. TRRA #1 directs Corrective Action Process training for all Engineering personnel for CAP expectations developed as part of CA #1.

Conclusions and Actions Needed -

The extent of cause of the root cause extends to other studies and analyses not being completed in a timely manner so that potential vulnerabilities are not identified. CA #4 directs a review of non-fleet programs that currently do not have a program basis document (H-series procedure). This action should identify programs, other than HELB, that currently do not have a program basis document that may be required when conducting operability or other evaluations that affect those programs. Additionally, CAPR #1 institutes a policy for ensuring projects funded by department line budgets are subjected to the site project review process and projects in the PRG process that are delayed or experience a scope change are re-reviewed.

CA #5 identifies other studies and analyses that have the potential for discovering additional vulnerabilities to operability. These actions satisfy the need to identify and fix potential deficiencies in other Engineering programs.

The extent of cause for CC #1 indicates that the lack of adequate standards for OE evaluation should be considered a site issue. TRRA # 2 directs training of all personnel conducting OE evaluations to ensure that extent of condition reviews need to be considered when performing OE evaluations.

This training satisfies the need to align site personnel for properly performing extent of condition assessments for OE evaluations.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 26 of 90 The extent of cause for CC #2 found that revised CAP initiation criteria should be applied to the Engineering organization and the site. TRRA #1 directs Corrective Action Process training for all Engineering personnel for CAP expectations developed as part of CA #1. This satisfies the Engineering personnel training needs for the extent of cause for CC #2. .

RCE 1211532 addresses the CAP initiation issues for the whole site.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 27 of 90 IV. Operating Experience:

• Internal OE -

LER 95-06 Determination that some component cooling system alignments are not within the Intent of Technical Specifications. Reviews of the component cooling system revealed that a single failure could make both component cooling trains of one unit inoperable when both trains of that unit are cross-tied. Corrective actions to correct the discrepancy were taken, but no extent of condition review was performed.

LER 1-00 Flooding from Postulated Failure of Air/Vacuum Valve has Potential to Disable Both Trains of Essential Service (Cooling Water). Opportunities to identify and evaluate an initial design deficiency were missed in 1990 and 1995. Contributing to the failure was the situation arising from the reviews and corrective actions performed for Generic Letter 89-13 and the Service Water Operational Performance Inspection. These activities involved many extensive and significant issues which overshadowed this initial design deficiency and may have contributed to the failure of Engineering staff to otherwise identify and evaluate it. Corrective actions to correct the discrepancy were taken, but no extent of condition review was performed.

AR 01111291, 01112915, 01113170, 01117260 - These CAPs document issues with operability determinations for equipment issues. Each CAP resulted in reopening the original CAP and updating the status notes to reflect basis for operability determination. All of these CAPs referenced deficiencies in implementing FP-OP-OL-01. These CAPs reflect the continuing trend in less-than-adequate operability reviews.

AR 01120989 (12/12/2007, SITE FAILED TO RECOGNIZE A POTENTIAL OPERABILITY ISSUE) - This CAP documented that no CAP was written for a leak on a safety-related component. The CAP also documented that the site may not have recognized the significance of the leak. This CAP was closed to AR 01120914 which conducted a root cause on the issues surrounding the component. CAPRs from this root cause addressed only the equipment aspects, not the questions about the sites ability to recognize issues.

AR 01183142 (5/26/2009, TREND IN INEFFECTIVE RESOLUTION OF OE ITEMS) - This CAP documents an identified trend in the resolution of OE items. A common cause evaluation is in progress. This CAP is an example of not fully evaluating and correcting deficiencies related to Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 28 of 90 industry OE and will address issues raised in this RCE pertaining to the sites effective use of OE.

RCE 01132717 (Site Response to Issues with SI-9-5) - Over a period of 2 years from 2006 to 2008, the site failed to address operability issues related to internal leakage of SI-9-5 (the first-off check valve from the high pressure reactor vessel to the low pressure safety injection system).

During 1R24, SI-9-5 failed the internal leakage surveillance (SP 1070).

This failure was not recognized for its impact on operability of the valve and no causal evaluation was conducted. Failure to recognize a potentially inoperable condition led to failures in the Corrective Action, Work Management and Operability/Functionality processes. The root cause was determined that the organization has not developed a process for review of relevant engineering data as inputs to decision-making and prioritization processes. The corrective actions included developing a procedure for use by Engineering personnel to include preparation and review of engineering data input to site decision-making and prioritization processes (specifically Work Management Screening, Outage Scope Creation, Outage Scope Change, AR Screening, Engineering Change, Plant Health Committee, Project Review Group, and Procedure Change).

This RCE also concluded that there were adequate opportunities through OE for the site to understand the implications and importance of preconditioning, but a lack of rigorous adherence to FP-PA-OE-01 led to inadequate identification of applicability to the site.

ACE 01131913 (3/20/08, Monticello, HELB Program documentation deficiencies) - Gaps exist in the MNGP HELB program with respect to industry standards. The apparent cause of this event is that personnel have an overall lack of knowledge of the HELB design and licensing basis.

A contributing cause was ineffective management oversight of the HELB program to ensure industry standard is maintained. Corrective actions included having the HELB program owner conduct a review of the MNGP HELB license basis to improve understanding, and identify opportunities for enhancements or deltas between program documentation and license basis, including updating the HELB Design Basis Documents as required.

Additionally, the HELB program owner will perform informal industry benchmarking.

RCE 01100615-01, (7/11/07, CAPRs Closure Conflicts with Procedural Requirements) - Identified previous attempts to resolve the issue have not been fully effective. Human performance failure modes included wrong assumptions, inadequate verifications, inadequate tracking, and time and schedule pressure. From the report: Corrective actions to directly address the HU failures were considered, but not implemented. Previous efforts to address the HU aspects were not successful. The root cause cited wrong assumptions, inadequate verification and inadequate task management.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 29 of 90 The CAPR action cited to result in assurance that a plan exists to facilitate CAPR closure (a pre-planning meeting with requirement that owed to report the results) does not appear to have been proceduralized RCE 01141755-01, (6/27/08, Identified NRC Crosscutting Issues) -

Concluded that the operational philosophy currently in place relies on skill sets and knowledge that no longer exist within the station and the organization is not placing appropriate focus on plant issues, strategies, and/or the appropriate priority when they are identified. The root cause identified is the roles and responsibilities previously held by Engineering to address plant issues have not been effectively transferred to Operations to promote a strong operational focus with contributing causes cited; high workload without proper prioritization and lack of critical skills throughout the organization. The created CAPRs, still in progress, are to revise the ODMI procedure and train Operations and Engineering. Corrective actions are in process to address the contributing causes. This effort is ongoing, due June 2010.

ACE 01174370-02, (6/22/09 No Tornado Protection of CC Piping for 122 SFP-HX) - One of the contributing causes was evaluation of internal and external operating experience was either not extensive or not used.

CCE 01183142, (8/20/09 Trend in Ineffective Resolution of OE Items -

Identified the Common Cause to be a Lack of Clearly Defined and Consistent Priorities), Lack of Functioning Engineering Work Management System, and Incorrect Operational Focus. Some of the items evaluated also show signs of incomplete understanding of the issues. The incomplete understanding of issues is revealed where issues were identified but resolution has not been completed.

RCE 01157726, (10/30/09 PI Rad Shipment Arrives at Consignee above DOT Rad Limits) - This RCE identified the fact that industry experience had not been effectively incorporated into the RMSP as a Contributing Cause.

RCE 01182488-03, (10/17/09 12 Circulating Water Pump Lock Out with Turbine/Reactor Trip) - 12 Circulating Water Pump had an electrical ground fault that resulted in a lockout of the pump. Lockout of the 12 CW Pump caused the circulating water flow rate through the condensers to be reduced to one half the normal flow. This led to a loss of vacuum and a Unit 1 reactor trip. The failure of the pump was caused by age related degradation of the 12 CWP power cable. The root cause was found to be that the Cable Condition Monitoring Program development and implementation was not given sufficient priority.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 30 of 90 AR 866805866805 (7/14/05 Radiation Monitoring System [Top 10 Issue]) - A project to replace the containment air monitors due to repeated LCOs and outdated radiation monitor modules (NMC-71) was initiated. Numerous delays were encountered for this project and the monitors have yet to be replaced (2/10). On 12/31/09, the module replacement project was restarted after being inappropriately closed in 9/06.

AR 1170596, (2/24/09 Identified Vendor Performance Issues with EDO

- S&L) - The vendor missed several due dates and failed to provide adequate design outputs for work associated with the hydrogen storage project leading to delays. The vendor also failed to provide follow-on materials for the Security Barrier Upgrade project. This is the same vendor that delayed the HELB project.

AR 576325576325 (2/14/05 Prepare a Project Package for Lead Shielding) -

This CAP requests Engineering to initiate a study to review lead shielding that is installed in the plant that does not meet the current plant design standards. To date (2/10) little progress has been made on this project.

• External OE -

A search for operating experience was conducted by searching the INPO Plant Events Database for CC systems and HELB events. A search was also conducted using the OE search homepage for the phrases Component Cooling and High Energy Line Break. Below is a summary of related OE events along with descriptions of Prairie Islands response to each one:

Plant Event 316-980715-1, DC Cook, POTENTIAL FOR HIGH ENERGY LINE BREAK TO DEGRADE COMPONENT COOLING WATER SYSTEM. On July 15, 1998, with both units in an extended shutdown, the station determined that the potential existed for a postulated critical crack in the Unit 2 main steam line to degrade the ability of adjacent component cooling water (CCW) pumps to perform their design function. The pumps are in a semi-enclosed area of the Auxiliary Building with a Unit 2 main steam line chase accessible from any of three doors. There is no calculation available to show that these doors can withstand the energy release from a postulated crack in the main steam line. DC Cook evaluated the event as NOTEWORTHY because the stations ongoing analysis had the potential to indicate the CCW pumps would not be able to perform their design basis functions.

There is no evidence that this issue was reviewed by the staff at Prairie Island. At the time of the INPO change date (09/20/1998) it was not part of the Prairie Island OE process to investigate Plant Events from other Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 31 of 90 plants. Therefore, the relevancy to our plant, and the close similarities between a HELB degrading Cooks CCW pumps and a HELB degrading Prairie Islands CCW piping, was never officially investigated.

INFORMATION NOTICE 2000-20, POTENTIAL LOSS OF REDUNDANT SAFETY RELATED EQUIPMENT BECAUSE OF THE LACK OF HIGH-ENERGY LINE BREAK BARRIERS. This information notice was issued by the NRC on December 11, 2000 in response to the above issue and similar issues at DC Cook 1 and 2. This notice used issues at the DC Cook site to lead discussion on four conditions that must coexist in order to produce a risk-significant configuration like that at Cook. They are 1) lack of HELB barrier between the redundant trains of a system that is needed to mitigate accidents, 2) the lack of environmental qualification for the redundant components of trains located in the same area, 3) the presence of high-energy piping in adjacent areas, and 4) the lack of a HELB barrier between adjacent piping and the redundant safety system trains. It states that these conditions cannot be present for essential systems and components.

XOE 20006028, CONDITION REPORT: POTENTIAL LOSS OF REDUNDANT SAFETY RELATED EQUIPMENT BECAUSE OF THE LACK OF HIGH-ENERGY LINE BREAK BARRIERS. This was Prairie Islands assessment of the issue reported in IN 2000-20. It had only discussed the pressurizer PORVs as the only SSC with a HELB issue as concerning the Information Notice since the DC panels located in the Auxiliary Building, providing power to the PORVs, could be affected by a Main Steamline Break. It talked about a design change currently in place at the time that would move those panels to a mild environment location.

This report did not properly address the entire issue of HELBs. It only looked at locations where the environmental conditions following a HELB would be detrimental to the SSC, such as described in the IN concerning DC Cook, and did not take into account pipe whip or jet impingement.

OE23897 (updating OE20291), Kewaunee, THE CONDENSATE MAKEUP LINE TO AFW PUMP IS VULNERABLE TO A FW LINE BREAK. The Condensate Makeup (CMU) line from the Condensate Storage Tank (CST) to the suction of the Auxiliary Feedwater (AFW) pumps is routed in close proximity to the Main Feedwater (MFW) piping. If a HELB were to occur in the MFW piping, pipe whip or jet impingement could impact the CMU line. The large amount of force from the MFW line break could significantly damage or break the CMU line. The following event scenario was listed in the OE as being possible; The MFW line break results in failure of the CMU line. The reactor trips and all AFW pumps auto start due to low-low Steam Generator level. The failed CMU line may introduce air into the AFW pumps due to its location on the suction side of the AFW pump. Air in the CMU line and/or pumps can Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 32 of 90 cause heavy cavitations within the AFW pumps. Air ingestion or heavy cavitations due to suction loss could result in damage to all three AFW pumps, rendering them inoperable. Because of this scenario all three AFW pumps were declared inoperable. Immediate and long term changes were made to bring these pumps back to an operable status. It also included the corrective actions that were to take place. Of these actions, one was to perform Extent of Condition walkdowns to look at other potential HELB issues and to provide assurance of design basis compliance for high energy line breaks.

OE 20291 was reviewed by Prairie Island staff via Operating Experience Evaluation Request (OEER) 00810473. It was also recorded on T-Track number OE037389. The OE was submitted on 02/21/2005 and the report was completed on 4/15/2005. In the report it was stated that if the CST piping were damaged, the resulting draining of the CST out the damaged line while still supplying the AFWPs, there would be an increased head loss at the AFWPs. It went on to say that Prairie Island has suction pressure switches installed to protect the pumps upon loss of suction so that air would not be able to be ingested into the pumps. It did not uncover any potential inoperability issues with the plant equipment. The review was only focused on the effects of the line/system damage to the CMU system, such as described in the OE, and it did not take into account similar hazards a HELB may have on other systems, such as the CC system, even though the OE stated that SSCs should be re-reviewed for system inter-relationships and associated implications and the OE stated a corrective action of performing walkdowns to find other HELB issues.

This is seen as an instance of a missed opportunity to identify the issue of a HELB affecting a safety related system. Instead of the HELB issue being reviewed as part of the general OE process, the focus was aimed only at the result on the affected system. It was never applied to the broader cause of the incident which was the postulated HELB event.

Per the Operating Experience Program procedure FP-PA-OE-01, revision 2 (Attachment 4) which was the revision used during the timeframe this OEER was written, the Operating Experience Evaluation Guideline did not prompt the OEER writer to investigate other systems or programs which may be effected by the same event contributors (i.e.: HELBs). The current revision of FP-PA-OE-01 (revision 12) does have a section that tells the writer to describe how the event relates to applicable plant equipment, procedures/ processes/programs, human performance (barriers currently affected).

Prior to the official release of OE 23897, NMC issued an Internal Operating Experience Rapid Notification Report on February 21, 2005.

This report was very similar to the OE filed through INPO. However, in the section titled Technical Considerations for Other NMC Plants, it makes the suggestion to Review the adequacy of equipment protection from Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 33 of 90 HELB events. This was another opportunity for Prairie Island to identify the issue of CC piping located next to HELB piping. There was no documentation found of a response to this internal report.

OE 20291 was later updated by OE 23897. The INPO change date for updating OE 20291 to OE 23897 was listed as 03/30/2005 which was prior to the actual OE 23897 release date of 12/20/2006. Therefore, there was no evaluation done on OE 23897 because the date did not fall into Prairie Islands search criteria for new Operating Experiences. However, OE 23897 is very similar to OE 20291 in that it utilizes the same Abstract, Reason For Message, Description, Causes, Corrective Actions, and Safety Significance. With this understanding it can be reasoned that OEER for OE 20291 would suffice for OE 23897.

The review of operating experience shows that numerous similar events occurred at other sites and that Prairie Islands lack of fully evaluating operating experience resulted in a missed opportunity to identify and correct this issue sooner.

• Conclusions and Actions Needed -

A review of the previous similar events and assessments demonstrate that the site has a history of issues with respect to management of engineering studies and analyses, the OE process, and the Corrective Action Process.

Corrective actions to address weakness in all of these areas have been identified in the Corrective Action section of this report. The review of operating experience shows that numerous similar events occurred at other sites and that Prairie Islands lack of fully evaluating operating experience resulted in a missed opportunity to identify and correct this issue sooner.

V. Nuclear Safety Significance This evaluation found no significant evidence of Safety-Conscious Work Environment (SCWE) failures as part of this sequence of events. However, discussions with engineers revealed the potential for some resistance from site and fleet personnel with respect to the identification of potential issues, specifically when all of the answers to the issue are not known, when more investigation is desired or when there are issues identified in studies or analyses received by the site. Investigation of this issue determined that the resistance comes from a lack of clear understanding between engineers and Engineering management with respect to the expectations for CAP initiation. In addition, this appears to be a culture or mindset that has developed at the site over time. While this has not necessarily had a direct chilling effect, the continued tolerance by management of this culture or mindset could be considered as having an indirect chilling effect. The Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 34 of 90 consequence of this is that potential plant concerns can linger for long periods of time without ever being known or understood by Operations and the site. While this was not found to be a significant contributor to the events identified in this report, the behaviors and interactions of management must continue to encourage the identification and free flow of information related to raising nuclear safety issues in the Corrective Action Process in a timely manner. It is expected that the completion of actions for Contributing Cause

1. 2 will address this issue.

The events of this report have resulted in a White Finding from the NRC.

The finding is a violation of 10 CFR Part 50, Appendix B, Criterion III, Design Control. PINGP failed to implement design control measures to ensure that the design basis for the component cooling water system was correctly translated into specifications, drawings, procedures, and instructions. Specifically, PINGP failed to ensure that the safety-related function of the component cooling water system was maintained following a high energy line break, seismic, or tornado events in the turbine building.

• The site failed to ensure that the Safety-Related function of the Component Cooling Water System was maintained following the initiating event of a High Energy Line Break (HELB) in the Turbine Building.

• During a HELB event, an unisolated section of the CC piping leading to the cold lab could break, potentially resulting in draining of the CC system within 6 minutes.

• The NRC conducted a phase 3 assessment of the impact of this event and found a delta core damage frequency of 3.2E-6 (White).

• The NRC finding is related to the cross-cutting aspect Human Performance, Decision Making.

The root cause evaluation team evaluated Safety Culture Impacts for the root and contributing causes and for the extent of condition and extent of cause utilizing information in QF-0436 (Evaluation of Safety Culture Impacts) and NRC Inspection Manual 03-05. The evaluation is documented in Attachment 7. Weaknesses in the following Safety Culture components were a root cause and contributing causes:

H4a(Human Performance, Work Practices)

Basis: There was inadequate supervisory and management oversight of the TB HELB analyses and the Engineering resolution studies for CC cold chemistry laboratory. Delays in completion of these activities deprived the station of an opportunity to discover potential vulnerabilities in a legacy issue (HELB impacts in the TB) in a timely manner. The longer the delays, the greater the chance these Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 35 of 90 vulnerabilities will not be discovered so that they can be assessed for operability impacts. In addition, the delays were not reviewed by the involved lead engineer or station management from the perspective of the increased risk of a legacy issue having potential further vulnerabilities that were remaining undiscovered. The activities were funded from the department line budget and therefore were not subject to a control process such as through the PRG.

Actions: This aspect is addressed by the Root Cause.

P1a(Problem Identification and Resolution, Corrective Action Program)

Basis: New CAPs were not generated for the CC/HELB interaction at various times when aspects of this issue were identified, precluding the Corrective Action Process from potentially taking appropriate corrective action. Station Management had not developed standards pertaining to CAP initiation,

1) for how long a potential issue can be investigated before it is documented in a CAP, and 2) when a CAP should be written for valid issues identified in draft or otherwise unacceptable studies Actions: This aspect is addressed by Contributing Cause #2.

P2a(Problem Identification and Resolution, Operating Experience)

Basis: There were missed opportunities to evaluate internal events and external events at other plants that were related to this event.

Actions: This aspect is addressed by Contributing Cause #1 VI. Reports to External Agencies & the NSPM Sites The following reports were made to external agencies:

• August 7, 2008, CAP 01145695 was posted in the Internal Operating Experience Report (form QF-0407) that was submitted to the fleet.

• September 29, 2008, LER 2-08-1, Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break was submitted to the NRC.

• October 8, 2008, this issue was posted on the Nuclear Network as OE27559 related to CAP 01146027, which was closed to CAP 01145695.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 36 of 90

• January 19, 2009, LER 2-08-1, Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break, Supplement 1 was submitted to the NRC.

• July 16, 2009, CAP 01145695 was again posted in the Internal Operating Experience Report that was submitted to the fleet. The reposting contained additional information related to the issue as well as documentation that this was an NRC white finding.

Additionally, when this RCE is complete and approved by the PARB, the following actions will be taken:

• A follow-up posting will be made internally and the report will be shared with the fleet.

• OE27559 will be evaluated for update

• The NRC will be notified of the completion of the RCE and a copy will be provided.

VII. Data Analysis A. Information & Fact Sources This root cause evaluation utilized the following as information and fact sources:

• Interviews - Interview list in Attachment 3.

• Procedures - A detailed list of referenced procedures is included in Attachment 5.

• Passport CAP Database - A detailed list of referenced CAPs is included in Attachment 5.

• INPO OE Database - A detailed list of referenced OE is included in Attachment 5.

• Other references as described in Attachment 5.

B. Evaluation Methodology & Analysis Techniques This root cause evaluation utilized the Event and Causal Factor Chart to summarize and link events and Why Staircases to determine Causal Factors, Contributing Causes and Root Causes. Failure Mode Analysis (Attachment 8), Safety Culture Analysis (Attachment 7), Barrier Analysis (Attachment 9), and Change Analysis (Attachment 10) were utilized to determine causal factors.

C. Casual Factors and Logic Ties Description The pertinent events to this investigation are described in the event narrative and illustrated in the Event and Causal Factor Chart. An Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 37 of 90 examination of this chart demonstrates that there are several recurring themes that eventually cascade into and impact decisions made over a period of almost twenty years. The original TB HELB analysis was incomplete in that it did not evaluate TB HELB impacts on CC piping. Per the 2003 INPO AFI, the HELB documentation was not clear, well organized, or easily retrievable. This resulted in a continuing lack of understanding of the HELB Licensing Basis and difficulty in identifying and verifying design inputs and assumptions.

This lack of understanding led in turn to a number of incorrect assumptions and a mindset regarding the TB HELB impact on CC piping, as demonstrated by IA #1, IA#2, IA#5, IA#8 and, IA#11. These included:

• The original plant construction must be correct since it had been approved by the NRC

• CC was not impacted by design basis events

• The USAR indicates that CC could withstand pipe breaks for events other than LOCA

• Loss of the CC due to pipe break can be addressed by an AOP

• The CC seismic design made it adequate for other design basis events

• A CC leak could be isolated within an acceptable time

• CC not on SSEL list so HELB impacts need not be considered

• HELB analysis only includes temperature and pressure effects to the exclusion of pipe whip and jet impingement These incorrect and unverified assumptions cascaded into and governed decisions made over many years including:

• Inadequate prioritization of HELB analyses and studies

• Related OPRs that were too narrowly focused

• CAPs not being written for related new conditions

• Related OE not being properly evaluated When it was recognized that the TB HELB analysis needed to be updated in 1994 and later, in 2005, that an Engineering resolution was needed for the cold chemistry laboratory CC piping, the station was on a path to correct this legacy issue. However, neither the analysis nor the studies developing an Engineering resolution were completed in a timely manner (IA #10). When delays due to funding and technical issues occurred, the mindset discussed previously resulted in them not being reviewed by the involved engineer or station management from the perspective of the increased risk of a legacy issue having potential further vulnerabilities that were remaining undiscovered. As of the date of this investigation these activities are still not complete, which led to this event and the NRC violation.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 38 of 90 Contributing to this event were failures to perform adequate OE evaluations for related issues as demonstrated by IA #3 and IA #6 and to write a CAP when a TB CC HELB condition was identified or properly evaluate a related CAP as demonstrated by IA #4, IA #7 and IA #9.

In conclusion, while there was a lack of understanding of the HELB Licensing Basis and TB HELB impacts on CC piping, the station did undertake the correct actions that would have successfully resolved this issue. Failure to complete those activities in a timely manner eventually led to this event. These delays deprived the station of an opportunity to discover potential vulnerabilities in a legacy issue (HELB impacts in the TB) in a timely manner. The longer the delays, the greater the chances these potential vulnerabilities would not be discovered so that they can be assessed for operability impacts.

VIII. Root and Contributing Causes Root Cause:

Lack of adequate documentation for HELB was, and is, a known deficiency.

An update to the Turbine Building HELB analysis was first attempted in the 1990s. Multiple attempts have been made to complete it without success.

While money was typically made available for work on the analysis, the level of effort from site Engineering was very limited. The HELB analyses were being worked on by only one civil/structural engineer who was also responsible for Tornado and Seismic issues and other emergent activities.

This is evident in the need to replace the original contractor after over seven years into the analysis effort due to errors in the analysis. This most pointedly demonstrates a lack of adequate management of this activity.

This inadequate management of HELB has allowed the site to operate without full assurance of being able to maintain the safety related functions of the CC system during HELB events in the Turbine Building.

Equally important is the failure to timely complete the Engineering resolution of the TB HELB CC cold chemistry laboratory piping issue. The decision was made to resolve this issue independent of the TB HELB analysis.

Considerable funding has been expended with various contractors with no firm resolution to this issue after 41/2 years of studies. Again, this most pointedly indicates a lack of adequate management of this activity.

Failure to adequately manage these activities so that they would have been completed in a timely manner has resulted in this event. These delays deprived the station of an opportunity to discover potential vulnerabilities in a legacy issue (HELB impacts in the TB) in a timely manner. The longer the delays, the greater the chances these potential vulnerabilities will not be discovered so that they can be assessed for operability impacts.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 39 of 90 Contributing to the event has been the failure to address the operability of the CC/HELB issue by means of initiating new CAPs when various aspects of this issue were discovered. Important to the nuclear safety of the plant is the identification of potential concerns that could challenge the ability of the plant to function as designed. Engineering management needs to develop and communicate what the expectations are for CAP initiation, specifically when potential degraded or non-conforming conditions are identified in studies or analyses. The failure of Engineering management to maintain proper oversight of the Engineering staff by allowing this potential concern to go undocumented in the Corrective Action Process was a contributing failure to this event.

Root Cause:

There has been inadequate management of the Turbine Building HELB analyses and the cold chemistry laboratory cooling water piping resolution studies.

Contributing Causes:

Contributing Cause #1: Station management has not developed adequate standards for OE evaluations with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification.

Contributing Cause #2: Engineering management has not developed expectations for CAP initiation for:

1) How long a potential issue can be investigated before it is documented in a CAP, and,

2) When a CAP should be written for valid issues identified in draft or otherwise unaccepted studies IX. Safety Culture - conclusions The root cause evaluation team evaluated Safety Culture Impacts for the root cause, contributing causes, extent of condition, and extent of cause and identified weaknesses in the Safety Culture components of H4a(Human Performance, Work Practices), P1a(Problem Identification and Resolution, Corrective Action Program), and P2a(Problem Identification and Resolution, Operating Experience). The weakness in H4a led to delays in completing the necessary HELB studies and analyses. Weaknesses in P1a and P2a led personnel to continually miss the opportunity to identify CC/HELB vulnerabilities through the prompt initiation of CAPs or conducting extent of condition evaluations for OE. As indicated in Section V, these weaknesses are addressed by the corrective actions identified by this root cause evaluation.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 40 of 90 X. Corrective Actions (SMARTS)

Corrective Actions to Restore (broke-fix)

• CC-20-4 (CC to Chem Labs from Unit 1 Supply Header) and CC-20-6 (CC from Chem Labs to Unit One Return Header) were verified closed to isolate the Cold Chem Lab from the rest of the CC System. This action is documented in the Ops Status Notes of AR 01145695.

Interim Corrective Actions (mitigation)

• The first quarter 2009 Human Performance for Engineers training used the CC/HELB issue as a case study regarding missed opportunities to identify a design flaw. (D83)

• Fleet (internal) Operating Experience on this event was distributed.

• A root cause team was formed to perform the evaluation regarding this issue.

Corrective Actions to Prevent Recurrence (CAPRs)

• CAPR 01145695-XX (CAPR #1) o Due Date: 11/30/2012 o Owner: Design Engineering Manager o

Description:

Develop a HELB Design Basis Document (H-series procedure) that provides an overall understanding of all HELB requirements for PINGP. Also, develop a HELB Program Document that demonstrates how the site meets all of the identified HELB requirements.

• CAPR 01145695-XX (CAPR #2) o Due Date: 6/10/2010 o Owner: Engineering Director o

Description:

Revise 5AWI 6.0.0 to implement a requirement that when activities funded within an Engineering department line budget become projects, they are required to be entered into the site project review process through the Project Review Group (PRG). In addition, revise the AWI to identify responsibility for tracking the status of ongoing, PRG-approved, O&M studies and analyses. Require each of these ongoing activities to include a plan that details follow-on actions once a study or analysis is completed. Require periodic updates of the status of these activities to the PRG. If any activity is delayed or the scope changes, the prioritization of all activities should be re-reviewed. Factors to take into account should include those activities involving risk-significant SSCs, particularly studies or analyses still in the discovery stage or those involving OBDs or OBNs. Depending on the nature of the study or analysis, a plan Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 41 of 90 to recover any delay should be developed and presented to PRG.

Other Corrective Actions

• CA 01145695-XX (CA #1) o Due Date: 4/15/2010 o Owner: Engineering Support Manager o

Description:

Develop Engineering expectations, for incorporation into FG-E-ARP-01 per PCRA #2, that cover at least the following aspects of CAP initiation:

When there is a potential CAQ identified, how long should the issue be investigated prior to initiation of a CAP?

When a vendor document (such as a study or report, draft or final) is received that documents a potential CAQ, what level of validation should be performed prior to initiation of a CAP?

• CA 01145695-XX (CA #2) o Due Date: 6/1/2010 o Owner: Design Engineering Manager o

Description:

As a support action for CAPR #1, determine short term personnel resource requirements for the HELB recovery program and develop a business case to support those requirements.

• CA 01145695-XX (CA #3) o Due Date: 6/1/2010 o Owner: Design Engineering Manager o

Description:

As a support action for CAPR #1, determine long term personnel resource requirements for sustainability of the HELB program and develop a business case to support those requirements.

• CA 01145695-XX (CA #4) o Due Date: 6/18/2010 o Owner: Programs Engineering Manager o

Description:

Review non-fleet programs (other than HELB) and develop, as appropriate, program basis documents for these programs to capture the essential program elements in a single location. (This action is similar to and should replace CA 01182488-14.)

• CA 01145695-XX (CA #5) o Due Date: 7/31/2010 o Owner: Design Engineering Manager Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 42 of 90 o

Description:

Identify PRG-approved studies and analyses that are not complete (scope has changed or there has been a delay). If they involve or impact risk-significant SSCs, particularly those activities still in the discovery stage, prioritization should be reconsidered and options explored to complete the discovery phase so that any vulnerabilities can be identified and assessed for operability. Depending on the nature of the study or analyses (for instance, is an OBN or OBD involved); a plan should be generated to recover the delay or justification provided for the scope change. In addition, any studies or analyses being funded by department line budgets that have not been through the PRG process should also be examined using the same criteria.

• TRRA 01145695-XX (TRRA #1) o Due Date: 7/16/2010 o Owner: Engineering Director o

Description:

Engineering TAC to evaluate training need for Engineering personnel that reinforces the need for all potential concerns to be addressed in the revised FG-E-ARP-01.

• TRRA 01145695-XX (TRRA #2) o Due Date: 5/26/2010 o Owner: Engineering Director o

Description:

TOC to evaluate training need for all site personnel who perform OE evaluations. The training will emphasize what the requirements are for evaluating OE. Emphasis will be placed on the use of a broad mindset that considers extent of condition as well as the need for review of the OE by other individuals or groups. This training will be required for the continued performance of OEEs.

• PCRA 01145695-XX (PCRA #1) o Due Date: 8/1/2010 o Owner: Fleet Performance Assessment Manager o

Description:

Revise FP-PA-OE-01, specifically Attachments 2 and 4, to include an explicit requirement to consider Extent of Condition when conducting OE evaluations.

• PCRA 01145695-XX (PCRA #2) o Due Date: 6/15/2010 o Owner: Fleet Design Engineering Director o

Description:

Revise FG-E-ARP-01, to incorporate the Engineering expectations for CAP initiation as developed in CA 01145695-XX (CA #1).

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 43 of 90

• GAR (GAR #1) o Due Date: 5/1/2010 o Owner: Project Engineering Manager o

Description:

Develop a policy that assigns a project manager to selected O&M projects based on criteria such as dollar amount, multi-year, etc. to relieve the engineer of the project management responsibilities.

Effectiveness Reviews:

• EFR 01145695-XX (EFR #1) o Due Date: 3/01/2013 o Owner: Design Engineering Manager o

Description:

Complete an effectiveness review of the HELB program by performing an external assessment of the program.

Effectiveness will be determined by no significant deficiencies (conditions adverse to quality exceeding a C level in AR screening) found in the established HELB program. This assessment will review the studies generated for the HELB program to ensure any CAQs have been identified in the corrective action process. It will also assess whether the HELB documentation provides a reasonable basis for future operability determinations involving HELB.

• EFR 01145695-XX (EFR #2) o Due Date: 9/30/2010 o Owner: Design Engineering Manager o

Description:

Complete an effectiveness review of the revised Business Planning and Development process for activities funded by the Engineering department line budget or activities that change status (delay or scope change) while in the PRG process. Effectiveness will be determined by (1) satisfactory results from interviews of Engineering personnel concerning their perception of the proper implementation of the process and (2) reviews of the required documentation of safety-significant decision-making to ensure that the proper evaluation was completed for a selected sample of activities funded within the department that were determined to be projects and were subjected to the site project review process or that changed status while in the PRG process.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 44 of 90 XI. References See Attachment 5.

XII. Attachments Attachment 1: Event and Causal Factor Chart Attachment 2: Root Cause Evaluation Charter Attachment 3: Interview List Attachment 4: Corrective Action Matrix Attachment 5: Supporting Evidence/Reference Documents Attachment 6: Why Staircases Attachment 7: Safety Culture Analysis Attachment 8: Failure Mode Analysis Attachment 9: Barrier Analysis Attachment 10: Change Analysis Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 45 of 90 Attachment 1: Event and Causal Factor Chart May 1990 December 5, 1990 D6 D44 Response to GL 89-13. Pioneer Study 892106 Operability evaluation of identifies vulnerabilities with CC system with respect to Pioneer 829106 study A single failure and QA boundaries and classifications. noted several Surge tank could empty in 6 minutes if interface discrepancies with CC barriers fail. HELB interaction with CC piping not system. No immediate mentioned. actions identified. SI Eventually documented as ENG-ME-240. pumps are considered operable.

IA #1 The evaluation only focused on SI pumps, remainder of CC system operability not addressed.

(FOI A0108)

IA #2 May 1995 WS 1 D49 December, 1994 Pioneer Study 892106 CC D53 Single Failure Analysis S&L Aux Building October 1995 October 1999 Open Issues. Inadequate May 5, 1995 update and TB D6 D52 and untimely evaluation of D47 A HELB analyses CC single failure Site notified that B operability for isolation LER-95-06-00 completed. Never analysis entered into previous HELB analysis capability in the CC LER identified some reviewed or records as ENG-ME- from S&L had software system. CC alignments that are approved due to 240. Identifies errors. No impact to FO1 A0863 not single failure proof.

large number of inadequate isolation in PINGP.

Inadequate evaluation of errors. Analyses Turbine Building.

loss of CC due to pipe focused only on rupture.

temperatures and pressures.

Vendor calculation ATD-0312 WS 2 Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 46 of 90 Attachment 1: Event and Causal Factor Chart (Contd)

_a_ --

051,000 INPO Ev~ (,&,R 576;140) ""'1 1 2 _

~- .....

...... ~f"Ig s ,, ~ ,2tOI ~Olior>~

f,.~

~

~

mHa.e

~-.

SeI*oT_ 2t, 21M IN 2t'00-20

~

'"ev_ od III' by EO. Hae,lST anlQ-lliI or" cd wei

<<l/IriZod (HEll!

M' ,

,_dtIg 2OXI-2Q ev-..:l ~odionlO' Cen<:eIO<I..-.te<

00' _~toGt.

",cMO'P'd to O!. !I'll st.

11>0 .ce--.I£& I'BB onoI\'SiI). DifII<>.A to EQIo_1lI fQtoev_ T8

• -~ ,ea.

0 87.11. AE'S 8'/.11 h_n;.

AES - __oIOCCo<*

.OC COOIL * .,oIlt>I1~.;r,g

& ...-11...... _

...giI _ _ mort 01"" ,HELB ono/y.iS ,

A,_

t~ot "'.

1_01",. 10tT1'0f""'*,O,

!"",-IlIUt

& IICU"dIoY

_po ......

............. -..,_ody 11' _ _* &

presoo.re ~

10""09" efta rrty, EWR lWROOH007.S4~ **

~ tmIfo:iert AA .66513

~.t_

I'El B. (I by i'I3..!lt

.....-HEL.Ban (IN N

.....-. Ha8In Iho _

Ihe_~)

Wlc&'>gj.

~ s.eO CC

",,,<JPUrll>>

"ll* &3 S"jSI .....

.ysl .... roc

"",~n._

u

~

ApIU. _

L§:J ~,

~,

OHR81 ~73 OEERSlOol73 Of2ln91(~

0E2O?!I1 (~

ntlJ1,~20, 23897 , ~20 ,

A ~2, M4 Au!r*1, (N!.

CAP ",aen OJ>

00 2 004 737382)

(.tJI137J12) on CC

""ttffi "" ._-

OPRIWY~od C4' 10R

~

~

_ ~od

.....'!l7l . _

Of CoMen,oIeW WWP

-~)-

(IUR ttfUI..-..;

.... HBB (IE' 10'

~ . lIl.Inlo

.tltiClr>g roe to (Kew.......,).

(JEER say WWP ar.

SyoI .... - .-,..;sil...... .ole nw * * ..,

."*. The<eiSroo

~ b sue me<tiDn oj _ rU B

...-.01_ i'ti.B

~~.

..,~~

EOC-..ote

~--

U ~,

~,~ ~

exte<tot r'IOff","" Crii

~~

-~

cb

~

m

~

-~

~

=

~

_. ~ -,

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 47 of 90 Attachment 1: Event and Causal Factor Chart (Contd) 2 5. ZOIS

.July 25.

.July 28. 2 MS 2MS saL ",onh oiIe 10' to site 10 SmII ~ 11'01:<".01 PfOlXlMl st~

Ie< studyi">;j 00

......... :z.S n lS C~lelion rdes

~Ie!ion cde~ 1", 737382 state AA 7CR382 IOf.oR stole "=-e, 1o oIemotives to

~,

est_h e s t _ h CC 0:: ~ yyslem stern ltJoI TB CC~i"og lhat CC~ ;.-,g r>eedsr-.eedslobe10 be

,t..BHELB_~m

,&.III-ftB_~m ~yno~td I:o.-J<Wy ir<; ~ he(

v _ e d l1<r ee~_ed er !-£LB HELB (pipe ...-.:1 je1 (pipe- wtip ,..-,j jet corrp.te

~ . TIl TB HElEI H El1l

...-.:1 cokt chern 1M> CC on<! cokf 0::

~tnenl ) & Il()f~

~menl Of~ ,..-,j ...-.:1 "WI "Wi h!ove Mve

._than_

AES~is AES .......,."is opt""",_ stat""ltJoIO--

options . stol""lhatO-10to be moo:tfied Of iooloted.

lIl<l<ified '" i.ololed. EAR SOl 50120 20 UsIr-.-q_

list~ _

c~edWrd c<Jffpe!ed WI rod _ oIed. Decide generated. Decide 10 to Ii< CC system system 10 to acc~ed acc~ed '<t.>e Of"""

" ' 10 ..-"""

cold...-.:1td cold,..-,j td ccl>em l>em 1M> 1M> ,_ltJIon ..... Re-.tion ResoUion Projec1 100..0:1 loo.n:l 1M! thai de>>;Jl J'fojec1 des>;J1 Msis_is 1<r .,=-Jts

=Jjs of TB TIlt£LB......,.,..,.

HOLB <>noIyses evmco.Jdc""",,!oos evmco.Jdc......,loo~

01 u; Of U; ..... e<tory rue to n>e<tory Ood: o lad: lSOO1tion n 01l is_ion ... lhe the Ttrtone~

Tcrtone~

. ~

~~

~~

AA 1<r AR 1<1" I-ftB HELB

...-.:1 tOffl<ldQ on<! 1 ()f~

is",-,",,_Itten is~_-'

Oct _ _ 31.

Oct_ 31. 280S 2MS Aug.Bt _.

Aug.s1 _ . 2811S 28DS W JuIy~.

J uIy ~ . 28DS 2811S rn.

Octo_ 20, 28e5 Odo_20. 2et1!i saLCC~

saL "'

CC boon:Iory y

-- "" 00 0<< _ _ 28.

Oct_ 28. 280S 2MS

~ m.Iy,oc~ .

stlly.oeeived HaB HB..B Projec1 J'fojec1 Cost PRO Sa--.ir>;i Saeeoiro;l SIb-

'"<<>roved Committee Appro'ied

~letionrod e s

~letionrde , 10<

73738:2-04: TB 0::

737382-04:

IOf AR CC AA

= Cost

~

HELB Projec1 J'fojec1 Co,,",

Focus"" on I-ElEI.

Foc="" f£11l.

~- study lor to" CC 0:: pipir-.;I pipO-.;I ",, ;.-,g oeeds

"" .... oeeds lobe 10 be ~,~

~

stMe~ OC doe, stoles does rd rod 0

0- - e e_ _e ed d Ifacr I-£LB HELB is_eon is_ e on S _Of

~ _()f (EAA 5(120)

(EAR SOl20) tornado e_ sroe e --*s sn:.e cd on SOlJG list .

rdonSOUG Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 48 of 90 Attachment 1: Event and Causal Factor Chart (Contd)

--=

--.=

~~

sbAySl . _ ex

_ _ foo,rd CC fetoruory,

,.~

• 7 _- ..... 1,

"*" _11._

_10 n " to cold _ 0Ib .....,,24,_ Hf' Api!,

-= --.....

hos 10 0:<*1 lID .........

........,2.11

~,

rdn.-y ' ........

f-..ory' ........... ~

,ZM' tou<tling!Wl>

Iau<:hr!g loQI\_lIJ

_IIY IlIU, oee PlIO Sr;reri'lg IoIrlg

_ _ ex Pipngo"

~U , _

~,

"" ,-= ~~

~,

~.

TBHaBanoIysiS Tete.S.....,.

025, 025. Dti2. 06lIII _ _ 01 saL Q..LisI sa

_101>00001

_1r>e*RFI'>

-~

-~

S&l.

-~

cold

-~

I-Ei., _ $&A SIL, AES,"'"

10 lob ""'to~

cdd lob

~ cc SUo to ....,.. OC fix ... Ie..II, CGt:I 0.. UI> por-.tng

~ '" :!OOS f!o.dgeI

---~tAR501120 RIIaIip of o.tJOI repOo1 to CC tt_~

,, -~

"~

~

IOU",,_

.-~

",.ft

.~ .. ~.......

."",.o.e;IOAdY<<1I

. . .oESIO_

_ ,O.,ES

-- rr.....

,~

ot~'"

HEUlwe.

Has elO10

_ Irtetoction

--"~

001

~=-

_m

"~

_sUI d

""",*"e ,.

tt_

0 I ! . -.. ..... ~.,

~-.

-~

~ed_

~.

~.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 49 of 90 Attachment 1: Event and Causal Factor Chart (Contd) 2 007 _ .J ..... ary

.July 2001 u y 0,2000 0 . 2000 Oe c eJrt>et- 24.

OeceJrt>et- 2 4. 2001 2007 c eJrt>et- 24.

Oe ceJrt>et- 2 4. 2001 2 007 saL study 0'"

O~

study rec_e d =

~

Note 10 AR 737382737382 ECR31 ECR 3183 OW

_ill .... 10 83_1Ite.--.lo

.J .....

.Ja 20 0 1 u..-y, 2008 ou"'Y, .Janu..-y,

.J 2000 _

.. nuooy, 2000_

_h recommendd:ions wthrecommend..tions Determine CC pPno;J~ oodd clo~ed-Ioop MId c lo ~ ed-Ioop ~"

~o May. 2008 May, 2001 f er or t he CC ~ to the n noI_lo'NIhsti>nd o t _ Io ...... h st.,..-.;:f c~ s.,."tern fer

~ f er cokl cold SaL rele .. ~ e~ fin&!

S aL r_""ses

~ioncfCC

~ctCC

~"

~o propo ~ es .,..-.;:f S&L propo~es ....-.d cold _ .. study cold !&b study or lorr>Oldo HELB Of' lornoodo. chern chern I"b 1<tIl per SaL a"n dr"fI HELB study.study OOUIho<;:<e d 10

~ ""-'Iho<il

e is indic..te~ CC indic"",e~ C C in TB not develop study . Approved 2.6.08 study. 2!6J08 develop cokf _

cokf !&b de~>7ted fer desio;;roed for I-ELB HELB by PRO .

byPRO HELB fi fix.

x.

~e j ~tnic 01' tornado seismic ternado:>

IA #'11

1. '1 1 Ne<N A R not NewAR

_enfOf"

'MItten fot" HELB CC IA #'12 IA 1>'13

1. 13 is~ues lholt

~sues E CR C R 3183 012,D41 O t 2 ,D41

~-"

~sue d using

"sued lJ<'ing R esolution e solution of cokf resufted in resulled poIrti<oly pa.-liooly chern_CC chern !&bCC

~R..

~R revie<hed reV'ie<.o.ed SaL S aL piping is~ue pipin!l  ; ~~u e study ____ is re""""",

rem<W>s open.

H Apo-il,

~,

0" 2 00 1 Apo-il , 200. May, 2000 Au _ _ , 2 August, 2 000 2 001 008 .July 1 0 , 2000 2000

.July H , 2 008 0'

001 3 1 , 2000

.July 31 O~

2000 1 346 3 46

" AR 737C22

,,=>;)ned rfrom Action "=9>ed 737332 . Determine om I-- SaL propos'"

~O

~,

propos&! for lemp temp f-0'"

0'0 Turbine EIuiIdirv BuiIdirJg HELB CAP C A P _itlenlo

_ittenlo document CC P PiJoin!;!

ipin\j ;"is Ente red TS LCO 3.0.3 due 10 both U2 CC 3.0 .3 f.....-.:liro;J d eh.ys f.....-.:tino;J_,,'r-'

de~ign requiremerts design requirements f er HELB in TB.

mod 10 re~otve re~oIve cokl HELB issue 1<tIl cold I"b

~ "'-Ie

~entto

-... HE LB

,.q&Cert 10 HELB lr _ ~ hope<"

Ir"in" hope<"bIe... bIe .

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 50 of 90 Attachment 1: Event and Causal Factor Chart (Contd)

December 15, 2008 July 31, July 31, 2008 September 18, 2008 D2 2008 1612 D66 August 6, 2008 D2 Missed Opportunities August 1, 2008 D70 D20 CAP written to Address ACE 01145695 to Identify HELB/CC D5 ECR 3653 J I Isolated U2 Unit 2 CC/HELB No CAP initiated in System Interactions OPR Complete written to provide CC Supply to issues. Closed to AR July 2006 to determine from S&L Study.

for CC System cooling to hot & cold Cold Chem 01145695. operability for impact of AR 01162511 Lab. Exited chem lab. (duplicate to HELB on CC. ACE completed LCO 3.0.3. ECR 3183 from Dec. 1/15/09.

2007)

January, 2009 August 7, 2009 D51 Untimely January 15, S&L resolution of cold 2009 July 30, Prairie Island has not Proposal to lab HELB issue D5 2009 met commitment to J resolve cold with plant mod RCE RFP to S&L S&L study INPO made in 2003 lab HELB proposals. Began 01145695 for CC piping approved by related to completion of issue since 7/05 - still not study. PRG. HELB analysis by temp mod fixed in 1/2010.

worked November 2005. AR stopped. 01192814 submitted.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 51 of 90 Attachment 1: Event and Causal Factor Chart (Contd)

IA Description 1 Inadequate OPERABILITY determination 2 Inadequate OPERABILITY determination 3 Inadequate OE extent of condition evaluation.

4 Extent of Condition was not applied to CC CAP 5 Inadequate OPERABILITY determination 6 Inadequate OE extent of condition evaluation.

7 A new AR was not written to document the new HELB issue 8 Failure to recognize potential HELB effects 9 No action was taken in response to the Q-List report for CC 10 Safety significance of incomplete TB HELB analysis not considered 11 A new AR was not written to document the new HELB issue 12 ECR issued without proper justification 13 Untimely resolution of cold chem lab CC piping issue Event Presumptive Contributing 1 IA # Barrier Condition Cause Logic Tie Terminal Change Event Presumptive A Before Now Failed Barrier Causal Root WS #

Event Factor Cause Chronological Tie Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 52 of 90 Attachment 2: Root Cause Evaluation Charter Root Cause Evaluation Charter CAR AR # 01145695 RCE# 01145695 Manager Sponsor: Scott Northard Problem Statement:

The station failed to ensure safety related functions of the component cooling water system were maintained for initiating events (HELB, tornado, seismic) in the Turbine Building Investigation Scope:

• This RCE will o evaluate the breakdowns of the processes and execution of those processes that allowed design deficiencies to remain uncorrected following the initial discovery

• This RCE will also evaluate the extent of conditions and cause(s) that are determined.

The output of the investigation will be corrective actions to prevent similar issues in the future.

Investigation Methodology:

To determine the root cause the team will employ event and causal factor charting, task analysis, barrier analysis and personnel interviews. NRC Inspection Criteria from 95001 will be used as a guide. This will include review of the extent of condition, extent of cause, Safety Culture attributes, review of the oversight and monitoring by the organization, and the past evaluation of the condition and the actions taken.

Team Members:

Team Leader Jeff Connors Design Engineering Team Member Chris Lethgo System Engineering Team Member Nate Adams Design Engineering (new engineer)

Team Member Ryan Cox Program Engineering Team Member Andy Notbohm Operations Team Member Kara Hernandez(Christopher) Monticello Systems Team Member Dave Pennington Monticello Systems Team Member Deb Albarado Organizational Eff.

RCE mentor Gene Woodhouse Performance Assessment Consultant Bob Hite Radiation Protection Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 53 of 90 Attachment 2: Root Cause Evaluation Charter (Contd)

Rewrite Team Members:

Team Leader Pete Wildenborg RP/Chem Team Member Jim Sumpter CERTRC Team Member Rob Sitek Engineering Admin Support Kim Bromberek BPA Team Advisor Betsy Rogers Training Team Advisor Andy Notbohm Operations Milestones: Rewrite Milestones:

Date Assigned: 8/03/09 1/11/10 Status Update: 08/17/09 1/18/10 Draft Report: 08/24/09 2/3/10 Final Report: 08/31/09 2/12/10 PARB graded and approved Communication Plan:

A copy of the approved RCE will be provided to licensing for submittal to the NRC within two weeks of approval.

Approved: Date:

Scott Northard Approved by: Screen Team / PARB on 1/8/2010 (circle one) (Date)

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 54 of 90 Attachment 3: Interview List Interview List Design Engineer A Design Engineer B Design Engineer C Design Engineer D Design Supervisor A Design Manager A Design Manager B Design Manager C System Engineer A System Engineer B Project Manager A Finance Engineer A Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 55 of 90 Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION RC: There has been inadequate CAPR 01145695-XX (CAPR #1) Develop a HELB Design Basis Document (H-management of the Turbine Building Due Date: 11/30/2012 series procedure) that provides an overall HELB analyses and the cold chemistry Owner: Design Engineering understanding of all HELB requirements for laboratory component cooling water Manager PINGP. Also, develop a HELB Program piping resolution studies. Document that demonstrates how the site meets all of the identified HELB requirements.

CAPR 01145695-XX (CAPR #2) Revise 5AWI 6.0.0 to implement a requirement Due Date: 6/10/2010 that when activities funded within an Engineering Owner: Engineering Director department line budget become projects, they are required to be entered into the site project review process through the Project Review Group (PRG).

In addition, revise the AWI to identify responsibility for tracking the status of ongoing, PRG-approved, O&M studies and analyses. Require each of these ongoing activities to include a plan that details follow-on actions once a study or analysis is completed. Require periodic updates of the status of these activities to the PRG. If any activity is delayed or the scope changes, the prioritization of all activities should be re-reviewed. Factors to take into account should include those activities involving risk-significant SSCs, particularly studies or analyses still in the discovery stage or those involving OBDs or OBNs. Depending on the nature of the study or analysis, a plan to recover any delay should be developed and presented to PRG.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 56 of 90 Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CA 01145695-XX (CA #2) As a support action for CAPR #1, determine short Due Date: 6/1/2010 term personnel resource requirements for the Owner: Design Engineering HELB recovery program and develop a business Manager case to support those requirements.

CA 01145695-XX (CA #3) As a support action for CAPR #1, determine long Due Date: 6/1/2010 term personnel resource requirements for Owner: Design Engineering sustainability of the HELB Program and develop a Manager business case to support these requirements.

EFR 01145695-XX (EFR #1) Complete an effectiveness review of the HELB Due Date: 3/01/2013 program by performing an external assessment of Owner: Design Engineering the program. Effectiveness will be determined by Manager no significant deficiencies (conditions adverse to quality exceeding a C level in AR screening) found in the established HELB program. This assessment will review the studies generated for the HELB program to ensure any CAQs have been identified in the corrective action process. It will also assess whether the HELB documentation provides a reasonable basis for future operability determinations involving HELB.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 57 of 90 Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION EFR 01145695-XX (EFR #2) Complete an effectiveness review of the revised Due Date: 9/30/2010 Business Planning and Development process for Owner: Design Engineering activities funded by the Engineering department Manager line budget or activities that change status (delay or scope change) while in the PRG process.

Effectiveness will be determined by (1) satisfactory results from interviews of Engineering personnel concerning their perception of the proper implementation of the process and (2) reviews of the required documentation of safety-significant decision-making to ensure that the proper evaluation was completed for a selected sample of activities funded within the department that were determined to be projects and were subjected to the site project review process or that changed status while in the PRG process.

Extent of Root Cause: CA 01145695-XX (CA #4) Review non-fleet programs (other than HELB) and Due Date: 6/18/2010 develop, as appropriate, program basis documents Owner: Programs Engineering for these programs to capture the essential Manager program elements in a single location. (This action is similar to and should replace CA 01182488-14.)

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 58 of 90 Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CA 01145695-XX (CA #5) Identify PRG-approved studies and analyses that Due Date: 7/31/2010 are not complete (scope has changed or there has Owner: Design Engineering been a delay). If they involve or impact risk-Manager significant SSCs, particularly those activities still in the discovery stage, prioritization should be reconsidered and options explored to complete the discovery phase so that any vulnerabilities can be identified and assessed for operability. Depending on the nature of the study or analyses (for instance, is an OBN or OBD involved), a plan should be generated to recover the delay or justification provided for the scope change. In addition, any studies or analyses being funded by department line budgets that have not been through the PRG process should also be examined using the same criteria.

CC #1: Station management has not TRRA 01145695-XX (TRRA #2) TOC to evaluate training need for all site personnel developed adequate standards for OE Due Date: 5/26/2010 who perform OE evaluations. The training will evaluations with respect to Extent of Owner: Engineering Director emphasize what the requirements are for Condition resulting in a lack of rigor evaluating OE. Emphasis will be placed on the applied to new issue identification. use of a broad mindset that considers extent of condition as well as the need for review of the OE by other individuals or groups. This training will be required for the continued performance of OEEs.

PCRA 01145695-XX (PCRA #1) Revise FP-PA-OE-01, specifically Attachments 2 Due Date: 8/1/2010 and 4, to include an explicit requirement to Owner: Fleet Performance consider Extent of Condition when conducting OE Assessment Manager Evaluations.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 59 of 90 Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CC #2: Engineering management has not CA 01145695-XX (CA #1) Develop Engineering expectations, for developed expectations pertaining to CAP Due Date: 4/15/2010 incorporation into FG-E-ARP-01 per PCRA #2, that initiation for: Owner: Engineering Support cover at least the following aspects of CAP Manager initiation:

1) How long a potential issue can be

• When there is a potential CAQ identified, investigated before it is documented in a how long should the issue be investigated CAP, and, prior to initiation of a CAP?

• When a vendor document (such as a study

2) When a CAP should be written for valid or report, draft or final) is received that issues identified in draft or otherwise documents a potential CAQ, what level of unaccepted studies. validation should be performed prior to initiation of a CAP?

TRRA 01145695-XX (TRRA #1) Engineering TAC to evaluate training need for Due Date: 7/16/2010 Engineering personnel that reinforces the need for Owner: Engineering Director all potential concerns to be addressed in the revised FG-E-ARP-01.

PCRA 01145695-XX (PCRA #2) Revise FG-E-ARP-01, to incorporate the Due Date: 6/15/2010 Engineering expectations for CAP initiation as Owner: Fleet Design Engineering developed in CA 01145695-XX (CA #1).

Director OTHER GAR (GAR #1) Develop a policy that assigns a project manager to Due Date: 5/11/2010 selected O&M projects based on criteria such as Owner: Project Engineering dollar amount, multi-year, etc. to relieve the Manager engineer of the project management responsibilities.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 60 of 90 Attachment 5 Supporting Evidence/Reference Documents

1. Identifier Description D1 FG-PA-RCE-01 (Rev 15) Root Cause Evaluation Manual D2 AR 01162511 Missed Opportunity to ID Issue from S&L Report D3 AR 00737382 Non-Seismic Equipment in CC System Pressure Boundary D4 SWI ENG-26 (Rev 2) (Legacy) Development of Engineering Studies D5 AR 01145695 CC Piping Adjacent to HELB (Original ACE Conducted)

D6 ENG-ME-240 (Rev 0) CC Single Failure Analysis D7 FP-E-SDY-01 (Rev 0) Development of Conceptual Design Concept Studies D8 OPR #509 (Rev 0) Non-Seismic Equipment in CC Pressure Boundary D9 AS 00737382-12 CC Piping not able to withstand HELB D10 ECR 3183 Add closed loop cooling system D11 FP-E-MOD-04 (Rev 3) Design Inputs D12 SLRP-2006-029 S&L Proposal D13 FP-PA-ARP-01 (Rev 10) CAP Process D14 FP-PA-ARP-01 (Rev 18) CAP Process D15 FP-PA-ARP-01 (Rev 22) CAP Process D16 FP-PA-ARP-01 (Rev 3) CAP Process D17 AR 000826114 Perform Seismic Analysis D18 PRG Minutes, 8/30/05 PRG Minutes D19 AR 01143812 Turbine Building Funding Delays D20 Operations Log Entries from 7/31/2008 D21 RCE 01013473 (Rev 0) D6 High Crankcase Pressure resulting in Unit 2 Shutdown D22 RCE 000185 (Rev 0) EHC Project Over Budget and Behind Schedule D23 RCE 888596 (Rev 1) Organizational Response to Operational Issues D24 RCE 01115585 (Rev 0) D5 Inoperability - Organizational Issues D25 AES Proposal, 3/3/06 Provide engineering services for CC piping Project D26 EAR 05 Funding CC project D27 EAR 06 Funding CC Project D28 SL-11973-014 (01/08) Chem Lab Component Cooling Study, Draft and Final D29 Plant Event 316-980715- Potential for HELB to degrade CC system 1

D30 OE 20291/23897 Condensate MU line to AFWP is vulnerable to FW line break D31 Plant Event 36335 Several plant locations discovered to be unprotected HELB areas D32 Plant Event 315-981116- HELB could result in condition outside design basis of 1 AF D33 RCE 01141755 Identified NRC Cross-cutting issues D34 RCE 01132717 Site Response to Issues with SI-9-5 D35 RCE 01132098 Site Response to 11 TD AFWP Turbine Bearing Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 61 of 90 Attachment 5 Supporting Evidence/Reference Documents

1. Identifier Description Failure D36 RCE 01144249 TSC Ventilation System Not Maintained Functional D37 RCE 01157726 Radioactive Material Shipment Exceeded DOT Limits D38 RCE 01166830 Inadequate CAP Resolution of Significant Issues D39 ACE 01154831 Failure in application of conservative assumptions resulted in a delay in implementation of corrective actions D40 FP-E-PHC-01 (Rev 0) Plant Health Committee D41 Interviews Interview Sheets D42 AR 449041449041 SOER 02-04 3a Review D43 1987 CC System 1987 CC System Upgrade Study D44 FOI A0108 Determine SI Pump Seal and Lube Oil Cooling Requirements D45 OEER 00810473 OEER For OE 20291 D46 LER 1-00-03 Flooding from postulated failure of air/vacuum valve has potential to disable both trains Essential Service (Cooling) Water System D47 LER 95-06-00 Determination that some CC alignments are not within the intent of TS D48 FOI-A0862 CL single failure analysis open issues D49 FOI-A0863 CC single failure analysis open issues D50 AR 01072605 Revised U2 FW analysis has new HELB cracks requiring evaluation D51 AR 576240576240 INPO Commitments Engineering Programs AFI D52 CR 19991622 HELB Analysis from S&L contains errors D53 AR 34876/34885 Turbine Building HELB analysis, documented finding problems D54 RPAs Reviewed for EOC D55 LRP Reviewed for EOC D56 AR 60820 Steam Exclusion Dampersexcessive leakage D57 AR 32717 Steam Exclusion Dampers D58 NRC IR Related to ineffective corrective actions D59 PRG Minutes 7/24/2006 PRG Screening Committee Minutes D60 FP-OP-OL-01, Rev 0 Operability Determination Process D61 FP-PA-OE-01, Rev 2 Operating Experience Program D62 06PBOS-1020 S&A Proposal for CC Piping Project D63 IN 2000-20 NRC Information Notice, dated Dec 11, 2000 D64 AR 871749871749 HELB Project Cost Overruns D65 AR 01002268 HLEB Project Cost Overruns D66 AR 01146027 Unit 2 CC/HELB Issues D67 CC Report, Rev 0 Q-List CC report, dated 12/11/06 D68 RFPs and proposals for CC piping D69 S&L Cost Estimate for Study to Upgrade Hot and Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 62 of 90 Attachment 5 Supporting Evidence/Reference Documents

1. Identifier Description Cold Labs 1/30/08 D70 ECR 3653 Alternate Cooling for the Cold Lab D71 ECR 13000 Turbine Building HELB Walkdown D72 PI SDP for CC/HELB D73 5AWI 3.7.0 rev.3 Operating Experience Assessment D74 TS 3.7.7 Component Cooling Water (CC) System D75 Kewaunee RCE, AFW Pumps Susceptible to Damage from Air Entrainment D76 5AWI 3.15.5 rev 0 Operability Determinations D77 5AWI 3.15.5 rev 14 Operability Determinations D78 Studies that have not progressed through the PRG D79 GL 87-02 Responses Selected PI responses to GL 87-02.

D80 USAR App I USAR HELB Evaluation D81 CR 19992212 Revised HELB temperatures for the 715, 735, and 755 Auxiliary Building D82 Outstanding OBDs and OBNs D83 P7550L-0822 Human Performance for Engineers P7550L-1001 Design & Licensing Basis Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 63 of 90 Attachment 6: Why Staircases WS 1 IA #1 The evaluation only focused on SI pumps, remainder of CC system operability not addressed.

(FOI A0108)

D44 Verification and Validation/ Verification and Validation/

Worker Practices Supervisory Oversight Supervisory Oversight No Operations CAs for CC system Relied on CC being a not addressed until involvement.* F2 QA1 system. J3, J4 1996 in preparation for upcoming NRC inspection.** (D41)

F3 Organization was Lack of Engineering-led understanding of rather than HELB licensing basis Operations-led.

and difficulty in identifying and verifying inputs and assumptions.

The organization is not operationally focused.

TB HELB analysis HELB documentation was incomplete so lacked clarity, didnt realize HELB organization, and not significance. ***

easily retrievable. *** (D41) F3

• Addressed by current procedures.

• • Lack of timeliness addressed by WS2.

• • • Addressed by RC1.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 64 of 90 Attachment 6: Why Staircases (Contd)

WS 2 IA #2 Inadequate and untimely evaluation of operability and isolation capability in the CC system.

Did not understand the full extent of the issue (D41) K6 Verification and Validation/

Worker Practices Supervisory Oversight Verification and Validation/ Verification and Validation/ Procedure Supervisory Oversight Worker Practices Project study No Operations Misinterpreted the Resolution of CC results addressed involvement* (D41) Engineers had mindset USAR to believe a issues was untimely. through WO & FOIS F2 that original plant leak could be isolated (D41) not CAP*

construction must be (D41) within acceptable time correct since it had NRC to mitigate (D49) J3, approval, CC not impacted J4 by design basis events, Organization was and USAR wording Resolution given Engineering-led regarding CC capability to lower priority due to rather than withstand piping breaks Believed loss of more important Operations-led. taken out of context to CC system due to issues. (D41) apply to events other than pipe rupture LOCA (D41, D49) J3 addressed by AOP (D41) J3, J4 The organization is not operationally Lack of focused. understanding of Believed CC seismic HELB licensing basis design made it and difficulty in adequate for other identifying and design basis events verifying inputs and so CC issue is less assumptions.

important. (D41)

TB HELB analysis HELB documentation was incomplete so lacked clarity, didnt realize HELB organization, and not significance. **

• Addressed by current procedures. easily retrievable. ** (D41) F3

• • Addressed by RC1.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 65 of 90 Attachment 6: Why Staircases (Contd)

WS 3 IA #3 Site evaluated SR equipment affected by HELBs (IN mentions HELBs in the Turbine Building).

Missed CC pumps &

system. EOC inadequate.

Verification and Validation/

Worker Practices/

Supervisory Oversight OEER only evaluated redundant safety related equipment affected by HELB temperature and pressure. Pipe whip and jet impingement effects not addressed.

Procedure Engineers had Procedure 5AWI mindset that HELB 3.7.0 did not only included specify temperature and requirements for pressure effects. J3 extent of condition review.* (D61)

RR1 Lack of understanding of HELB documentation HELB licensing basis lacked clarity, and difficulty in organization, and not identifying and easily retrievable. **

verifying inputs and assumptions. **

• Addressed by PCRA #1.

• • Addressed by RC1.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 66 of 90 Attachment 6: Why Staircases (Contd)

WS 4 IA #4 Extent of Condition not considered Worker Practices/ Procedure/ Supervisory Oversight No ACE / RCE was generated (D3, D41)

Procedure didnt require an ACE /

RCE at that time.*

(D16) RR1

• Addressed by current procedures.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 67 of 90 Attachment 6: Why Staircases (Contd)

WS 5 IA #5 OPR too narrow. Only considered whether CC on SSEL rather than consequences of seismic failure. D8 Verification and Validation/ Verification and Validation/

Worker Practices Supervisory Oversight Did not understand No Operations the full extent of the involvement* (D41) issue. K6 F2 Organization was Engineering-led Did not consider all rather than aspects of HELB, Operations-led.

only looked at seismic issues. D8 J3, J4 The organization is Lack of not operationally understanding of focused.

HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.

TB HELB analysis HELB documentation was incomplete so lacked clarity, didnt realize HELB organization, and not significance. ** easily retrievable. **

(D41) F3

• Addressed by current procedures.

• • Addressed by RC1.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 68 of 90 Attachment 6: Why Staircases (Contd)

WS 6 IA #6 The site did not adequately evaluate OE, especially EOC.

OEER only considered AFW aspects (D45)

Procedure Worker Practices OE Screen team OE Evaluator did Closeout review only sent not look at HELB did not recognize evaluation to AFW aspect omission (D45)

System Engineer (D45)

(D45)

Focused on Supervisory System Oversight/Procedure Procedure Implications (D45)

Screen team Review focused on Procedure did not Procedure requires assigns OE to one procedural specify consultation with group per OE compliance rather requirements for other groups as procedure.* (D41) Did not Consider than content (D45) review (D61) RR1 required. * (D61)

Extent of Condition (D45) RR1 Supervisory Oversight/Procedure/Job Planning and Preparation Procedure FP-PA-OE-01 does Assignment scope not specify an did not specify Extent of Condition Extent of Condition Review (D61) RR1 Review (D45)

CC #1 Management did not develop adequate standards of OE

• Addressed by PCRA #1. evaluations with respect to extent of condition Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 69 of 90 Attachment 6: Why Staircases (Contd)

WS 7 WS 7 IA #11 IA #7 New AR not No new written for HELB AR for HELB CC issues that and tornado would have issues written. resulted in OPR.

Supervisory Verification and Validation/

Oversight/Procedure Worker Practices New CAP not written for HELB/Tornado (D3, D41)

Still Supervisory Oversight Verification and Validation/

investigating Lack of Worker Practices the issue with Supervisor HELB assumed HELB/CC Oversight to be addressed (D41) (D41) under AR 737382737382 Did not recognize new Procedure condition.

Lack of understanding of Inadequate Perception that HELB licensing basis funding for HELB CAP isnt and difficulty in Engineers had mindset that analyses.

written until identifying and original plant construction must problem is verifying inputs and be correct since it had NRC known (D41) assumptions. approval, CC not impacted by design basis events, and USAR wording regarding CC capability to withstand piping Some No clear guidance or HELB breaks taken out of context to TB HELB analysis supervisors too expectation for how documentation apply to events other than was incomplete so involved with long issues can be lacked clarity, LOCA (D41, D49) J3 didnt realize HELB issue resolution All disciplines investigated before a organization, and significance. (D41) reporting to one CAP is written (D16, not easily (D41) F3 supervisor for D41) retrievable.

analysis activities (D41)

Low resources require supervisors to become involved (D41)

CC #2 RC1 Engineering Management has Inadequate not developed expectations management of pertaining to CAP initiation for HELB analyses this aspect Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 70 of 90 Attachment 6: Why Staircases (Contd)

WS 8 IA #8 Failure to recognize significance of HELB interaction of feedwater and CC lines. D25 Did not understand the full extent of the issue. K6 Verification and Validation/

Verification and Validation/ Worker Practices Worker Practices Engineers had mindset that original plant construction Review of S&L study must be correct since it had focuses only on NRC approval, CC not HELB temperature impacted by design basis and pressure effects.

events, and USAR wording regarding CC capability to withstand piping breaks taken out of context to apply to events other than LOCA (D41, D49) J3 Engineers had mindset that HELB only included temperature and pressure.

Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.

HELB TB HELB analysis documentation was incomplete so lacked clarity, didnt realize HELB organization, and significance.

not easily (D41) F3 retrievable.

RC1 Inadequate management of HELB analyses Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 71 of 90 Attachment 6: Why Staircases (Contd)

WS 9 IA #9 Site review was completed but no action taken in response to Q-List report for CC Worker Practices/ Supervisory Supervisory Supervisory Oversight Oversight Oversight Q-List Project Project did not agree CAPs werent written philosophy was to with some of the until there was an validate and then conclusions in the report actual concern (D41) initiate CAPs (D41) and therefore considered the entire report questionable (D41)

Some items Procedure included in the report were viewed as too conservative (D41)

No clear guidance for writing ARs for valid issues identified in draft or otherwise unaccepted Engineering studies.

CC #2 Engineering Management has not developed expectations pertaining to CAP initiation for this aspect Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 72 of 90 Attachment 6: Why Staircases (Contd)

WS 10 IA #10 Turbine Building HELB analyses still not complete, safety significance of delay not considered. D41 Verification and Validation/

Worker Practices Supervisory Supervisory Supervisory Supervisory Supervisory Oversight Oversight Oversight Oversight Oversight Funding issues:

funding was limited, Staffing issues: Mindset issues:

funding process is insufficient Engineers had mindset cumbersome and staffing, only one that original plant not well understood. person assigned construction must be (D41) Workload issues: High to HELB. (D41) correct since it had NRC workload, transition to approval, CC not NMC and then Xcel, impacted by design basis Supervisory issues: working Passport rollout, other events, and USAR supervisors, frequent issues with higher priority, Technical issues: wording regarding CC management/supervisor changes, emergent regulatory Problems with capability to withstand inadequate turnovers, supervisor issues, forced outage, content and quality piping breaks taken out positions vacant for extended inadequate turnover of vendor studies. of context to apply to periods, limited interaction between engineers. (D41) (D41) events other than LOCA between supervisor and reporting (D41, D49) J3 engineers. (D41) Funding was accomplished using a department line TB HELB analysis budget that was was incomplete so outside the site didnt realize HELB significance.

review process.

(D41) F3 (D41)

RC1 Inadequate management of HELB analyses Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 73 of 90 Attachment 6: Why Staircases (Contd)

WS 11 IA #12 ECR 3183 issued using partially reviewed draft S&L study as basis.

D41 Supervisory Supervisory Worker Practices/ Supervisory Oversight Oversight Oversight No turnover or ECR initiator was discussion of CA requesting the working in Outage report with ECR had an Control Center manager by SME.

impending due during a forced F1 date. A9 outage. S3 SME had left company.

Human performance error.

Turnover is addressed by CA 13 from RCE 01165133 Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 74 of 90 Attachment 6: Why Staircases (Contd)

WS 12 IA #13 Resolution of cold chem lab CC piping issue remains open.

Worker Practices/ Supervisory Supervisory Worker Practices/ Supervisory Oversight/ Procedure Oversight Oversight S&L final Study does No turnover or not get owners Studies on cold discussion of acceptance review chem lab CC piping report with per SWI-ENG-26.

issue still incomplete. manager. F1 D12, D41 WS 10 Reviewers had erroneous Not all information understanding of in study was reasons for the study reviewed. J3 and significance of CC/HELB.

Owners Acceptance of study not complete Lack of (D2) J4 understanding of HELB licensing basis and difficulty in identifying and verifying inputs No clear owner of and assumptions. the study. (D41)

SWI ENG-26 HELB TB HELB analysis not followed documentation was incomplete so (D2) lacked clarity, didnt realize organization, and HELB significance.

not easily (D41) F3 retrievable.

RC1 Procedure Inadequate Compliance issue -

management of Addressed by ACE HELB analyses. 01162511-01 (D2)

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 75 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H1a The site makes safety-significant or risk-significant decisions Applicable . The root cause team did not find using a systematic process, especially when faced with uncertain that this aspect was present.

or unexpected plant conditions, to ensure safety is maintained.

This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions.

H1b The licensee uses conservative assumptions in decision making Applicable There were a number of and adopts a requirement to demonstrate that the proposed assumptions made concerning TB action is safe in order to proceed rather than a requirement to HELB impacts on CC piping that demonstrate that it is unsafe in order to disapprove the action. were not verified.

The licensee conducts effectiveness reviews of safety-significant decisions to verify the validity of the underlying assumptions, identify possible unintended consequences, and determine how to improve future decisions.

H1c The licensee communicates decisions and the basis for decisions Not Applicable The root cause team did not find that to personnel who have a need to know the information in order to this aspect was present.

perform work safely and in a timely manner.

H2a The licensee ensures that personnel, equipment, procedures, and Not Applicable The root cause team did not find that other resources are available and adequate to ensure nuclear this aspect was present.

safety. Specifically, those necessary for: Maintaining long term plant safety by maintenance of design margins, minimization of long-standing equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs are low enough to support safety.

H2b The licensee ensures that personnel, equipment, procedures, and Applicable Resources for effectively other resources are available and adequate to ensure nuclear implementing required HELB safety. Specifically, those necessary for: Sufficient qualified activities were insufficient. One personnel are trained and available to maintain work hours within person was responsible for the working hours guidelines. HELB program only part time.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 76 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H2c The licensee ensures that personnel, equipment, procedures, and Applicable There was a lack of design basis other resources are available and adequate to ensure nuclear documentation with regard to HELB.

safety. Specifically, those necessary for: Complete, accurate and Design documentation was not up-up-to-date design documentation, procedures, and work to-date. OE procedures did not packages, and correct labeling of components. address extent of condition.

Corrective Action Process procedures did not provide expectations on CAP initiation for certain situations.

H2d The licensee ensures that personnel, equipment, procedures, and Not Applicable The root cause team did not find that other resources are available and adequate to ensure nuclear this aspect was present.

safety. Specifically, those necessary for: Adequate and available facilities and equipment, including physical improvements, simulator fidelity and emergency facilities and equipment.

H3a The licensee appropriately plans work activities by incorporating: Not Applicable The root cause team did not find that risk insights this aspect was present.

job site conditions, including environmental conditions, which may impact human performance; plant structures, systems, and components; human-system interface; or radiological safety The need for planned contingencies, compensatory actions, and abort criteria.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 77 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H3b The licensee appropriately coordinates work activities by Not Applicable The root cause team did not find that incorporating actions to address: this aspect was present.

The impact of changes to the work scope or activity on the plant and human performance.

The impact of the work on different job activities, the need for work groups to maintain interfaces with offsite organizations, and communicate, coordinate, and cooperate with each other during activities in which interdepartmental coordination is necessary to assure plant and human performance.

The need to keep personnel apprised of work status, the operational impact of work activities, and plant conditions that may affect work activities Long-term equipment reliability by limiting temporary modifications, operator work-arounds, safety systems unavailability, and reliance on manual actions.

Maintenance scheduling is more preventive than reactive.

H4a The licensee communicates human error prevention techniques, Applicable Pre-job brief was not performed prior such as holding pre-job briefings, self and peer checking, and to evaluation of OE by system proper documentation of activities. These techniques are used engineer.

commensurate with the risk of the assigned task, such that work activities are performed safely. Personnel are fit for duty. In addition, personnel do not proceed in the face of uncertainty or unexpected circumstances.

H4b The licensee defines and effectively communicates expectations Not Applicable Owners review of S&L study did not regarding procedural compliance. Personnel follow procedures. follow SWI-ENG-26.

H4c The licensee ensures supervisory and management oversight of Applicable Supervisory and management work activities, including contractors, such that nuclear safety is oversight of the CC/HELB analyses supported. and studies was not effective in that these activities are still not complete.

Delays in completion of these activities were not reviewed for their impact on a legacy issue and the risk of potential vulnerabilities remaining undiscovered.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 78 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS P1a The licensee implements a corrective action program with a low Applicable CAPs were not generated for the threshold for identifying issues. The licensee identifies such CC/HELB interaction on discovery, issues completely, accurately, and in a timely manner precluding the CAP from potentially commensurate with their safety significance. taking appropriate corrective action.

P1b The licensee periodically trends and assesses information from Not Applicable The root cause team did not find that the CAP and other assessments in the aggregate to identify this aspect was present.

programmatic and Issue common cause problems. The licensee communicates the results of the trending to applicable personnel.

P1c The licensee thoroughly evaluates problems such that the Applicable CAPs that were written did not result resolutions address causes and extent of conditions, as in an extent of condition review.

necessary. This includes properly classifying, prioritizing, and evaluating for operability and reportability conditions adverse to quality. This also includes, for significant problems, conducting effectiveness reviews of corrective actions to ensure that the problems are resolved.

P1d The licensee takes appropriate corrective actions to address Applicable safety issues and adverse trends in a timely manner, Failure to review the impact of commensurate with their safety significance and complexity. delays in completion of HELB activities resulted in corrective actions not being completed in a timely manner.

P1e If an alternative process (i.e., a process for raising concerns that Not Applicable The root cause team did not find that is an alternate to the licensees corrective action program or line this aspect was present.

management) for raising safety concerns exists, then it results in appropriate and timely resolutions of identified problems.

P2a The licensee systematically collects, evaluates, and Applicable Evaluation of events at other plants communicates to affected internal stakeholders in a timely and internal events that were related manner relevant internal and external OE. to this event did not consider extent of condition.

P2b The licensee implements and institutionalizes OE through Applicable The root cause team did not find that changes to station processes, procedures, equipment, and this aspect was present training programs.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 79 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS P3a The licensee conducts self-assessments at an appropriate Not Applicable The root cause team did not find that frequency; such assessments are of sufficient depth, are this aspect was present.

comprehensive, are appropriately objective, and are self-critical.

The licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies.

P3b The licensee tracks and trends safety indicators which provide an Not Applicable The root cause team did not find that accurate representation of performance. this aspect was present.

P3c The licensee coordinates and communicates results from Not Applicable The root cause team did not find that assessments to affected personnel, and takes corrective actions this aspect was present.

to address issues commensurate with their significance.

S1a Behaviors and interactions encourage free flow of information Not Applicable The root cause team did not find that related to raising nuclear safety issues, differing professional this aspect was present.

opinions, and identifying issues in the CAP and through self assessments. Such behaviors include supervisors responding to employee safety concerns in an open, honest, and non-defensive manner and providing complete, accurate, and forthright information to oversight, audit, and regulatory organizations. Past behaviors, actions, or interactions that may reasonably discourage the raising of such issues are actively mitigated. As a result, personnel freely and openly communicate in a clear manner conditions or behaviors, such as fitness for duty issues, that may impact safety and personnel raise nuclear safety issues without fear of retaliation.

S1b IF alternative processes (i.e., a process for raising concerns or Not Applicable The root cause team did not find that resolving differing professional opinions that are alternates to the this aspect was present.

licensees corrective action program or line management) for raising safety concerns or resolving differing professional opinions exists, THEN they are communicated, accessible, have an option to raise issues in confidence, and are independent, in the sense that the program does not report to line management (i.e., those who would in the normal course of activities be responsible for addressing the issue raised).

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 80 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS S2a All personnel are effectively trained that harassment and Not Applicable The root cause team did not find that retaliation for raising safety concerns is a violation of law and this aspect was present.

policy and will not be tolerated.

S2b Claims of discrimination are investigated consistent with the Not Applicable The root cause team did not find that content of the regulations regarding employee protection and any this aspect was present.

necessary corrective actions are taken in a timely manner, including actions to mitigate any potential chilling effect on others due to the personnel action under investigation.

S2c The potential chilling effects of disciplinary actions and other Not Applicable The root cause team did not find that potentially adverse personnel actions (e.g., reductions, this aspect was present.

outsourcing, and reorganizations) are considered and compensatory actions are taken when appropriate.

OTH1a Accountability is maintained for important safety decisions in that Not Applicable The root cause team did not find that the system of rewards and sanctions is aligned with nuclear this aspect was present.

safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority.

OTH1b Management reinforces safety standards and displays behaviors Not Applicable The root cause team did not find that that reflect safety as an overriding priority. this aspect was present.

OTH1c The workforce demonstrates a proper safety focus and reinforces Not Applicable The root cause team did not find that safety principles among their peers. this aspect was present.

OTH2a The licensee provides adequate training and knowledge transfer Not Applicable The root cause team did not find that to all personnel on site to ensure technical competency. this aspect was present.

OTH2b Personnel continuously strive to improve their knowledge, skills, Not Applicable The root cause team did not find that and safety performance through activities such as benchmarking, this aspect was present.

being receptive to feedback, and setting performance goals. The licensee effectively communicates information learned from internal and external sources about industry and plant issues.

OTH3 Management uses a systematic process for planning, Not Applicable The root cause team did not find that coordinating, and evaluating the safety impacts of decisions this aspect was present.

related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 81 of 90 Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS OTH4a These policies require and reinforce that individuals have the right Not Applicable The root cause team did not find that and responsibility to raise nuclear safety issues through available this aspect was present.

means, including avenues outside their organizational chain of command and to external agencies, and obtain feedback on the resolution of such issues.

OTH4b Personnel are effectively trained on these policies. Not Applicable The root cause team did not find that this aspect was present.

OTH4c Organizational decisions and actions at all levels of the Not Applicable The root cause team did not find that organization are consistent with the policies. Production, cost this aspect was present.

and schedule goals are developed, communicated, and implemented in a manner that reinforces the importance of nuclear safety.

OTH4d Senior managers and corporate personnel periodically Not Applicable The root cause team did not find that communicate and reinforce nuclear safety such that personnel this aspect was present.

understand that safety is of the highest priority.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 82 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS HUMAN PERFORMANCE FAILURE MODES Inattention (A1) Not paying attention to the task requirements. Not paying Not Applicable The root cause team did not Type - SB attention to information in the immediate environment. find that this failure mode was present.

Bored (A2) Inadequate level of mental activity due to performance of Not Applicable The root cause team did not Type - SB repetitive actions or lack of activity. find that this failure mode was present.

Habit / Reflex (A3) Ingrained or automated pattern of actions attributed to the Applicable When dealing with the CC Type - SB repetitive nature of a well-practiced task or a natural response. Seismic issues, it appears there was a reflexive action to continue adding actions to the existing CAP instead of writing a new AR.

Tired & Fatigued (A4) Degradation of physical or mental abilities due to illness, a lack Not Applicable The root cause team did not Type - SB/RB/KB of rest, or influences associated with body rhythms. find that this failure mode was present.

Distracted & Interrupted (A5) Conditions of task or the work environment require the individual Applicable The individual assigned to the Type - SB to stop and restart a task, diverting the individuals attention from HELB program had many the task at hand. other responsibilities that would have resulted in stopping and starting a task many times.

Multi Tasking (A6) Performing two or more tasks simultaneously and neglecting to Applicable Multiple Interviews indicated Type - SB perform a required element of one or more of the tasks. that high work loads and multiple issues (specifically AFWP Bearing issues) were occurring at the same time.

Lapse of Memory (A7) Momentary loss of memory regarding information previously Not Applicable The root cause team did not Type - SB learned and known. find that this failure mode was present.

Inadequate Tracking (Place Method used to maintain control of information, necessary Not Applicable The root cause team did not Keeping) (A8) requirements, or status was not properly used. find that this failure mode was Type - SB/RB present.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 83 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Time & Schedule Pressure (A9) Urgency or excessive pace required to perform the task. No Not Applicable The root cause team did not Type - SB/RB/KB spare time allotted or perception by the individual that a tight find that this failure mode was schedule exists. present.

Fear of Failure (A10) Apprehension regarding potential adverse consequences if the Not Applicable The root cause team did not Type - SB/RB/KB individual fails to perform at a high level, resulting in undesirable find that this failure mode was behaviors. present.

Imprecise Communication Miscommunication resulting from error of omission or Not Applicable The root cause team did not (A11) commission by the sender or receiver. This includes find that this failure mode was Type - SB/RB breakdowns of the three-part communication process. present.

Cognitive Overload (J1) Mental demands on the individual to maintain a high level of Not Applicable The root cause team did not Type - RB/SB concentration while requiring recall of excessive amounts of find that this failure mode was information. present.

Spatial Disorientation (J2) Loss or misjudgment of place or time; wrong component, wrong Not Applicable The root cause team did not Type - SB/RB train and wrong unit errors due to similarities in the environment. find that this failure mode was present.

Mindset / Preconceived Idea The tendency of an individual to make a judgment based upon a Applicable There was a mindset that the (J3) preconceived mental model or preconditioned bias that is not answer to an issue had to be Type - RB based upon the current information, conditions or indications. known prior to initiating a CAP.

Wrong Assumptions (J4) Judgments are made without verification of the facts and are Applicable There was a preconceived Type - RB usually based upon the individuals perception of recent idea that the CC System was experiences or events. not required for a HELB, without verifying the validity of the assumption.

Inadequate Verification (J5) Insufficient verification of the facts, and is usually based upon Applicable There was a lack of design Type - RB inaccurate information or the lack of information. basis documentation with regard to HELB which prevented verification of the assumption that the CC System was not required for a HELB.

Inadequate Motivation (J6) Low morale or low interest in performing well. Not Applicable The root cause team did not Type - SB/RB/KB find that this failure mode was present.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 84 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Shortcuts Taken (J7) Actions to allow the job to go easier or faster, contrary to Not Applicable The root cause team did not Type - RB prescribed requirements. find that this failure mode was present.

Work Around (J8) Compensatory or non-standard actions to meet a requirement Not Applicable The root cause team did not Type - RB are taken by the worker due to uncorrected material condition, find that this failure mode was programmatic deficiencies, or long-standing problems. present.

Over Confident (K1) Underestimating the difficulty or complexity of the task. Self- Applicable Individuals did not write a Type - KB/RB/SB satisfaction or confidence with a situation in which actual second AR, in part (based on hazards or dangers exist, but the worker is not aware of them. interviews), because there was a belief that there was adequate understanding of the issue.

Unfamiliar or Infrequent Task Tasks that have not been performed before or are performed Not Applicable The root cause team did not (K2) infrequently. find that this failure mode was Type - KB present.

Misdiagnosis (K3) Decisions made with accurate information that is used or Applicable There was an incorrect Type - KB interpreted incorrectly when reaching a decision. diagnosis of the impact of HELB on the CC System.

Tunnel Vision (K4) Decisions are made without considering all the available options Not Applicable The root cause team did not Type - KB or information needed to adequately assess the situation. find that this failure mode was present.

Inadequate Knowledge of Insufficient knowledge of fundamentals needed for task, such as Not Applicable The root cause team did not Fundamentals (K5) heat transfer, fluid flow, structural analysis, etc. find that this failure mode was Type - KB present.

Inadequate Knowledge of Insufficient knowledge of codes, standards, design basis, Applicable The lack of well developed Standards (K6) licensing basis, regulations, etc. needed to perform the task. HELB design basis Type - KB documentation meant that personnel did not adequately understand the design requirements of the CC system and how HELB would impact operation.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 85 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Flawed Analytical Process or Decisions based on a flawed analysis, such as using qualitative Applicable A narrow focus when Model (K7) versus quantitative data, insufficient determination of performing OE evaluations Type - KB/RB problem/solution scope, improper computer modeling, or played a role in this event.

inadequate sample scope.

ORGANIZATIONAL AND MANAGEMENT FAILURE MODES Inadequate Span of Control (S1) Horizontal organizational design - the number of personnel Not Applicable The root cause team did not which a supervisor is responsible for is too large or too few for find that this failure mode was the groups oversight & responsibilities. This often creates present.

problems with task assignment and accountability.

Inadequate Levels in the Vertical organizational design - the number of levels or layers, Not Applicable The root cause team did not Organization (S2) from senior manager to employee is too many or too few for the find that this failure mode was given activity. Creates problems with communication of present.

expectations.

Insufficient Staffing (S3) Comprehensive organizational design - the total number of Applicable Interviews indicated that high employees for which the company or group is designed are not work loads played a role in filled. Often causes staff work overload and poor accountability. this event.

Inadequate Communication A breakdown in communication (written or verbal) within one Applicable Inadequate supervisory within an Organization (F1) organization or work group. Often leads to important issues not control in that standards and being addressed and critical process breakdown. expectations were not clearly reinforced.

Inadequate Communication A breakdown in communication (written or verbal) among two or Not Applicable The root cause team did not among Organizations (F2) more organizations or work groups. Often leads to a breakdown find that this failure mode was in processes that require several groups to participate. present.

Inadequate Prioritization (F3) Deficiencies in determining which work takes precedence over other Applicable The HELB program was not work. Often leads to unexpected equipment failures or failure to meet adequately prioritized.

regulatory requirements.

Inadequate Planning (F4) Deficiencies in determining what work must be done, by whom, Not Applicable The root cause team did not when, and how long it will take. Often leads to staff work find that this failure mode was overload, budget over-runs and low morale. present.

Inadequate Emerging Issues Deficiencies in determining how to deal effectively with Not Applicable The root cause team did not Management (F5) unexpected issues. Often leads to continual crisis find that this failure mode was management and low morale. present.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 86 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Inadequate Program Inadequate oversight of critical work processes to ensure they Applicable The CAP and OE programs Management (F6) function smoothly and effectively. Often results in program were not effective in resolving degradation over time or increased problems within those the issue.

processes.

Inadequate Trust (C1) A lack of confidence in the workgroup or members of the Not Applicable The root cause team did not workgroup, or a disbelief in information shared. Often results in find that this failure mode was fractured work completion and stress levels. present.

Inadequate Teamwork (C2) Constant friction among the workforce, or an unwillingness to Not Applicable The root cause team did not work with one another. This problem could exist within find that this failure mode was organizations or between organizations. Results in confusion present.

within the ranks and a lack of information flow among the groups.

Inadequate Knowledge (C3) An inadequate understanding of the work to be performed and how the Not Applicable The root cause team did not work ties into the overall goals. Often causes individual errors to find that this failure mode was occur. present.

Lack of Commitment (C4) A lack of dedication to the work. Often results in inconsistent or Not Applicable The root cause team did not unreliable performance by an individual or group. find that this failure mode was present.

Inadequate Self Assessment A failure to continually encourage feedback, listen to customer Not Applicable The root cause team did not (C5) input, or look at better ways to perform. Often creates a false find that this failure mode was sense of security and leads to complacency. present.

PROCESS FAILURE MODES Actions Not Specified (RR1) The action(s) that an individual or group must perform to Applicable FP-PA-ARP-01, Revs 9-21 accomplish a task are not contained in the document or did not specify to create a instruction. new CAP for a new issue identified during evaluations of existing issues. FP-PA-OE-01 does not contain explicit guidance to include Extent of Condition in OE evaluations.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 87 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Actions Not Clear (RR2) The action(s) that an individual or group must perform to Applicable The OE and CAP procedures accomplish a task are not clearly described in the document or lacked guidance that could instruction. have helped in problem identification Actions not within Control of the The action(s) that an individual or group must perform to Not Applicable The root cause team did not Individual (RR3) accomplish a task cannot be performed as specified (physical find that this failure mode was constraints, do not have authority to dictate results, etc.). present.

Actions Conflict with Another The action(s) that an individual or group must perform to Not Applicable The root cause team did not Process (RR4) accomplish a task conflict or contradict the actions specified by find that this failure mode was another document or instruction. present.

Actions Not Tied to Another The action(s) contained within one document or instruction does Not Applicable The root cause team did not Process When Necessary not reference supporting documents or instructions when find that this failure mode was (RR5) necessary. present.

Methods Not Clearly Defined Action(s) are required by the document or instruction, but the Not Applicable The root cause team did not (RR6) method to accomplish the actions is not clearly specified by the find that this failure mode was document or instruction. present.

Unnecessary Actions Required The document or instruction require the performance of certain Not Applicable The root cause team did not (RR7) actions that is not really necessary to successfully perform the find that this failure mode was action. present.

Wrong Information (RR8) The information provided in the document or instruction is Not Applicable The root cause team did not incorrect. find that this failure mode was present.

Critical Actions Not Verified Critical actions required to successfully perform a task are not Not Applicable The root cause team did not (AR1) verified within the process. find that this failure mode was present.

Excessive Verifications (AR2) The document or instruction requires excessive verification of Not Applicable The root cause team did not completed steps or tasks. Actions are verified, regardless of find that this failure mode was criticality to the task or the task has multiple reviews and present.

verifications instead of a single, specific review.

No Process Monitoring (AR3) There is no established means of monitoring the success or Not Applicable The root cause team did not failure of the process. find that this failure mode was present.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 88 of 90 Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Only Monitoring Problems The only method of monitoring process performance is to Not Applicable The root cause team did not (AR4) observe problems when they occur. find that this failure mode was present.

No Acceptance Criteria (AR5) No acceptable performance parameters have been established Not Applicable The root cause team did not for the process, procedure or task. find that this failure mode was present.

No One Specified to Perform No one is specified (either by title, group, or other means) as Not Applicable The root cause team did not Task (I1) responsible for completion of the actions required by a find that this failure mode was document or instruction. present.

More Than One Person More than one person or group is specified (either by title, Not Applicable The root cause team did not Specified to Perform Task (I2) group, or other means) as responsible for completion of the find that this failure mode was actions required by a document or instruction. present.

Person Specified Not Able to The person or group specified (either by title, group, or other Not Applicable The root cause team did not Perform Task (I3) means) as responsible for the completion of the required actions find that this failure mode was in a document or instruction is unable to perform the action. present.

Typically because they do not have the skill or knowledge.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 89 of 90 Attachment 9 BARRIER ANALYSIS HAZARD BARRIER ASSESSMENT TARGET Inadequate Procedure (FP-OE-OL-01)

• No clear guidance on performing extent of condition review. Failed to ensure that the evaluation of

• Requirements for supervisory review focused on administration rather than safety related function of OE content of the evaluation. the CCW system was

• No requirements for a pre-job brief. maintained.

Engineers work practice

• Performed a very narrow evaluation.

• Did not evaluate all the recommendations in the OE report.

Supervisory Oversight

• Did not review OE evaluation to ensure high quality product.

Job Planning and

• No pre-job brief Preparation Inadequate Verification and Validation

• Took information from USAR out of context. Failed to ensure that the investigation

• Assumed original plant construction must be correct - did not question or safety related function of of study validate this assumption the CCW system was findings

• Did not involve other departments maintained.

regarding CC

• Thought check valves in the CCW system or operator action would prevent the system from draining.

Procedure

• No clear CAP process when report first received Supervisory Oversight

• Supervisors did not insist study findings be discussed outside of the engineering department Extent of Procedure (FP-PA-ARP-01)

• Did not require extent of condition for condition evaluations Failed to ensure that the Condition Worker Practices

• It is good engineering practice to perform extent of condition assessments safety related function of Assessment Supervisory Oversight

• Supervisor did not ensure extent of condition was performed. the CCW system was maintained.

Operability of Worker Practices

• Did not feel another CAP was required Failed to ensure that the CCW was not

• Excessive investigation time safety related function of considered for

• Overconfidence the CCW system was HELB/Tornado Verification and Validation

• Did not verify if CCW system would be needed for a HELB event maintained.

in Turb. Bldg.

• Single point of expertise limited chances to challenge/ verify experts opinion.

• Took information from USAR out of context.

• Did not involve other departments Supervisory Oversight

• Supervisor did not require an additional CAP to be written.

• Untimely investigation when TB HELB issues were noted.

Procedure

• Procedure did not provide clear guidance for how long a potential problem should be investigated before writing a CAP Procedure

• The Development of Engineering Studies procedure defines scope of review.

This includes verifying clear definition of the problem with specific supporting information.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.

QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 90 of 90 Attachment 10 Change Analysis Potential HELB interactions were known in 2005 but no immediate actions were taken. In 2008 the problem was rediscovered and appropriate actions were taken.

2005 Instance 2008 Instance Change / Difference Impact / Assessment Seen as a long term problem Recognized immediate problem The issue was dealt with on an When operability was scrutinized

- Wrote EAR - Wrote CAP operability level for the 2008 per design requirements, the issue

- Attempted to get study - Initiated new OPR instance. The 2005 instance was finally known fully.

funded - Isolated CC to the Cold Chem. Lab was more interested in closure

- Study done over the of the issue following year Experienced Engineer New Engineer (<2 years) The experienced engineer used The more detailed investigation

- Knew past justifications - Had nothing written down. Had to past experience to guide during the 2008 instance led to a

- Had preconceived ideas search for justifying documentation decisions in place of rigorous fuller understanding of the design

- Would rather resolve issue - Not as much historical knowledge. investigative work. The new requirements. Understanding of the without a CAP Less preconceived ideas. engineer had no previous design requirements led to the

- Relied on Tribal Knowledge - Trained in higher level experience to rely on. This appropriate operability assessment.

expectations for writing CAPs required him to fully investigate

- Had no Tribal Knowledge - Had the issue. The new engineer to research all information had to rely on procedures and retrievable documentation.

Discipline - Civil Discipline - Mechanical The engineers involved in each There is some evidence that the

- No program to explain the - No program to explain the instance were of a different 2005 instance was more focused on requirements. requirements. engineering discipline and had the Seismic aspect.

- Focused on Seismic - Was aware of need to investigate different responsibilities. Having responsibility for all three

- Responsible for Seismic, HELB (HELB, Seismic, and Tornado)

HELB, and tornado issues - Responsible for HELB but not areas could result in a more limited seismic or tornado. focus on the requirements for each area and less of an ability to recognize when new issues arise.

Form retained in accordance with record retention schedule identified in FP-G-RM-01.