ML102460635
Text
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 1 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
RCE REPORT Prairie Island Nuclear Generating Plant Component Cooling Piping Adjacent to HELB Location in Turbine Building Event Date: 7/29/2008 RCE: 01145695 RCE Team Members:
Dave Kettering/Scott Northard Management Sponsors Jeff Connors/Peter Wildenborg Team Leaders Christopher Lethgo System Engineering (Root Cause Investigator)
Nate Adams Design Engineering Ryan Cox Program Engineering Andy Notbohm Operations (Root Cause Investigator)
Kelsa Christopher Design Engineering (Monticello)
Dave Pennington Design Engineering (Monticello)
Deb Albarado Organizational Effectiveness Gene Woodhouse Performance Assessment (RCE Mentor)
Rob Sitek System Engineering James Sumpter CERTREC Consultant Kim Bromberek Administrative Assistant Betsy Rogers Training (Team Advisor)
Review: Independent RCI Date Approvals:
RCE Team Leader Date Management Sponsor Date PARB Meeting Date NOTE: The above signatures may be documented via passport assignments.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 2 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Table of Contents Page #
I. Executive Summary 3
II. Event Narrative and Timeline 12
III. Extent of Condition Assessment 18
IV. Operating Experience 27
V. Nuclear Safety Significance 33
VI. Reports to External Agencies & the NSPM Sites 35
VII. Data Analysis 36
A. Information & Fact Sources 36 B. Evaluation Methodology & Analysis Techniques 36 C. Causal Factors and Logic Ties Description 36
VIII. Root Cause and Contributing Causes 38
IX. Safety Culture 39
X. Corrective Actions (SMARTS) 40
XI. References 44
XII. Attachments 44
Note: The acronym "CAP" is used interchangeably in this RCE with the acronym "AR". Both acronyms refer to an action request in the corrective action database.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 3 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
I. Executive Summary Problem Statement:
The station failed to ensure safety related functions of the component cooling water system were maintai ned for initiating events (HELB, tornado, seismic) in the Turbine Building.
Event Synopsis:
The genesis of this legacy issue began in December 1972 when the Atomic Energy Commission published a letter (the "Giambusso letter")
which required Prairie Island to address the consequences of pipe ruptures (high energy line breaks or HELB) outside containment and
submit their analyses for review. Prai rie Island's response is documented in Appendix I of the USAR. The stat ion analyzed the effects of pipe whip, jet impingement, temperature and pressu re from HELBs in the Auxiliary Building (AB) and the impact on the safeguards corridor in the Turbine
Building (TB). The impact of HELBs in the TB on the Component Cooling Water System (CC) was not addressed.
There were updates to the HELB analyses for the AB in response to GL 87-11. Beginning in 1990, Prairie Island began identifying vulnerabilities
with the CC with respect to single failure, QA classifications and QA boundary deficiencies that were discover ed in response to GL 89-13. No actions were taken to address any of the vulnerabilities and operability of the CC system was only addressed from a high level perspective without full understanding of the issue. One of the vulnerabilities identified was that the surge tank could empty in six minutes if interface barriers failed.
In 1994, Prairie Island started work on an updated Turbine Building HELB analysis, though the focus was only on te mperature and pressure impacts.
Completion of this analysis has been hindered by both technical and financial issues. In 2003 the site received an INPO AFI for Engineering
programs not being managed effectively (including HELB) which resulted in commitments to INPO to have a Turbine Building HELB analysis completed by 2005. However, at the time of this investigation, the Turbine Building HELB analysis still has not been completed.
In 2005, the station decided to devel op an Engineering resolution to the CC HELB issues in the TB for t he cold chemistry laboratory piping independent of completion of the TB HELB analyses. However, progress on studies to develop the Engineering resolution have been delayed for
various reasons and have not been completed at the time of this investigation.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 4 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
From 2000 through 2008, several opportunities existed for the CC/HELB interaction in the Turbine Building to be identified and placed in the Corrective Action Process. A few ex amples of these opportunities include (in addition to the INPO AFI in 2003):
- A 2000 Information Notice, IN 2000-20, regarding a HELB issue at DC Cook that mentioned HELBs in the Turbine Building.
- CAP 00737382, issued in 2004, i dentified seismic classification issues with the CC piping. An ext ent of condition review of the CC piping vulnerabilities was not conducted.
- Kewaunee OE from 2005 that notifi ed the site of an event pertaining to a HELB interaction with the AFW Pump Suction line.
- July 2005 completion notes for CAP 00737382 state that the TB CC piping needs to be evaluated for HE LB and tornado. A CAP is not written for this new condition.
A decision is made to develop an engineering solution for the CC piping in the TB to the cold chemistry laboratory rather than wait for the results of the TB HELB analyses.
- August 2005 and July of 2006, Engineering Assistance Request (EAR) forms were presented to t he Project Review Group (PRG)
Subcommittee for funding of studies to resolve CC/HELB issues but no questions were raised by this group regarding the significance of
this issue. Draft studies from Sargent & Lundy state that CC does not isolate on seismic or tor nado and that walkdo wns found the CC piping near high energy feedwater li nes. A CAP is not written for these newly identified conditions.
- Sargent & Lundy Q-List Report for the CC system received in December 2006 containing information regarding the need to
upgrade the piping in the CC piping in the Turbine Building to prevent loss of CC system function. A CAP is not written because the study is not accepted by the station.
- Sargent & Lundy draft study received in June of 2007 (final received in January 2008) discussed CC vulnerabilities to a HELB, tornado, or
seismic event.
Corrective actions completed betw een 2005 and 2007, as well as other project documentation (EARs and study pr oposals), indicate that concerns existed regarding the impac t of a HELB, tornado, or seismic event on the CC piping. However, actions to addr ess the concerns remained under the parent CAP for the seismic classifi cation issues with the CC system (CAP 00737382), allowing the HELB and tornado vulnerabilities to remain at a low visibility level with no considerat ion of operability for any potential additional vulnerabilities. Studies investigating an Engineering resolution to the cold chemistry laborator y CC piping were not complete.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 5 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
In July of 2008, a walkdown of the CC/HELB interaction resulted in the initiation of CAP 01145695. Under this new CAP, the site addressed operability issues related to a HELB event in the TB and the impact on CC piping, which led to a fuller understanding of the significance of the issue.
Based on this discovery, the Unit 2 CC System was declared inoperable and action was taken immediately to restore operability to the CC System by isolating the CC system piping from the Turbine Building.
On August 5, 2009, Prairie Island received an NRC inspection report
identifying a preliminary white finding pertaining to the CC/HELB interaction on Unit 2. This wa s followed by a final significance determination for the white fi nding on September 3, 2009.
Nuclear Safety Significance:
This evaluation found no significant evidence of Safety-Conscious Work Environment (SCWE) failures as part of this sequence of events. However, there is evidence of a potentia l for resistance from site and fleet personnel to write a CAP when answers to an issue are not known, when an investigation is desired or when there are issues identified in studies or analyses received by the site. This, however, was not a significant contributor to this event. See Section V of this report for more information.
The events of this report have result ed in a White Findi ng from the NRC.
The finding is a violation of 10 CFR Pa rt 50, Appendix B, Criterion III, "Design Control." The violation wording is as follows:
Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion III, "Design Control," requ ires, in part, that measures be established to assure that the design basis for safety related functions of structures, systems, and components are correctly translated into specifications, drawings, procedures, and
instructions. Further, Criterion III requires that the design control measures provide for verifyi ng or checking the adequacy of designs.
Contrary to the above, as of July 29, 2008, the licensee failed to implement design control measures to ensure that the design basis for the component cooling water s ystem was correctly translated into specifications, drawings, procedures, and instructions.
Specifically, the licensee failed to ensure that the safety-related function of the component coolin g water system was maintained following a high energy line break, se ismic, or tornado events in the Turbine Building.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 6 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
The root cause evaluation team eval uated Safety Culture Impacts for the root and contributing causes and for the extent of condition and extent of cause utilizing information in QF-0436 (Evaluation of Safety Culture Impacts) and NRC Inspection Manual 03-05. The evaluation is
documented in Attachment 7 and su mmarized in the Nuclear Safety Significance Section of this report. Weaknesses in the following Safety
Culture components were a root cause and contributing causes:
- H4c(Human Performance, Wo rk Practices, Management and Supervisory Oversight)
- P1a(Problem Identification and Resolution, Corrective Action Program, Complete, Accurate and Ti mely Identification of Issues)
- P2a(Problem Identific ation and Resolution, O perating Experience, Systematic Evaluation of Relevant Internal and External Operating Experience)
Conclusions/Root Cause:
Conclusions:
This investigation has concluded t hat a number of fa ilures occurred at Prairie Island that prevented the site from ensuring measures were in place and actions were taken to maintain the safety related functions of the CC system during initiating events in the Turbine Building. The main failure is that the site did not address issues like this one through the rigorous identification and timely resolution of known design basis deficiencies. Integral to this pr ocess is the knowledge of design and licensing bases requirements. The plant is required to be designed and
built to meet all of these require ments. Additiona lly, all of these requirements need to be correctly transla ted into procedures, instructions, drawings, and specifications.
With respect to HELB, certain breaks of high energy lines were required to
be assumed in the Turbine Building.
However, the site did not have adequate documentation available to show that the breaks of these lines would not prevent other system, structures, and components (SSC) from meeting all their required safety related functions or even whether the proper breaks had been considered. This was a known deficiency but the significance of the deficiency did not appear to be well understood. This deficiency was pointed out to the site by INPO in 2003.
While activities were undertaken to identify and resolve CC TB HELB piping issues (TB HELB analyses to update the documentation and studies to develop an Engineering reso lution for the cold chemistry
laboratory CC piping), managem ent of these activities was inadequate.
Engineering continued to pursue these activities with a normal priority QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 7 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
level, having only one engineer worki ng on them part time and allowing them to be delayed by other emergent work. The activities were funded from the department line budget and therefore were not subject to the PRG review process. These conti nuing delays were not reviewed by the involved engineer or station managem ent from the perspective of the increased risk of a legacy issue having potential further vulnerabilities that were remaining undiscovered. In additi on, as a result, th ere were multiple cascading unintended consequences as discussed in Section VII.C.
There are two other contri buting factors identified in this investigation.
One was related to the removal of ba rriers that may hav e identified the significance and complexity of this issue (e.g. more broadly focused
Operability Determinations, management involvement, and independent evaluation of the vulnerabilities and t he delays). All of these barriers were removed when CAPs were not wri tten to document new identified conditions with the CC piping. A portion of this investigation centered on understanding why new CAPs we re not written. Interviews indicated that personnel involved either did not re cognize HELB as a separate issue from seismic or did not want to wr ite a CAP until more information was available to show that there really was an issue. In addition, it was a practice not to write a CAP for issues identified in analyses or studies until those documents were acc epted by the station.
When the significance of the issue was finally realized in July 2008, it was involvement of other engineers with a different mindset that led to a full understanding of the CC/HELB issue.
There was an existing mindset (discussed in Section VII.C) regarding HELBs in the TB and the CC that hindered full understanding of the CC Licensing and Design Basis. However, completion notes being added to AR 737382737382and statements from various draft studies examining opt ions to resolve the cold chemistry laboratory HELB CC piping issue demonstrate that the station was beginning to grasp the significance of this issue. So, if CAPs had been written earlier in the process, it most likely would have resulted in the potential operability issue with the CC system being understood sooner and appropriate action would have been in itiated. Also, the site had unclear guidance regarding the threshol d for when to document a potential issue in a CAP.
Another contributing factor determined in this investigation is related to the missed opportunity that the site had to recognize the significance of the
CC piping in the Turbine Building thr ough evaluation of related OE. An extent of condition of the issues i dentified in OE was not conducted (not required by procedure) so that TB HELB impacts on CC piping were not identified.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 8 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Root Cause:
There has been inadequate management of the Turbine Building HELB analyses and the cold chemistry laboratory component
cooling water piping r esolution studies.
The failure to effectively manage these ac tivities so that they are still not finalized as of the date of this investigation has deprived the station of the opportunity to discover additional potential vulnerabilities in a legacy issue (HELB impacts in the Turbine Building) in a timely manner. This presents
a challenge to the stati on's ability to complete a timely assessment of operability impacts and resolve identified issues.
There are two additional impacts that have resulted from the delay in
completion of these activities:
- 1) A continuing lack of clar ity and organization of the HELB documentation that is also not eas ily retrievable (as identified by
INPO in 2003), and,
- 2) A continuing lack of understanding of the HELB Licensing Basis and difficulty in identifying and verifying design inputs and
assumptions.
The consequences of the above impacts are:
- 1) Inadequate prioritization of HELB analyses and studies, 2) Related OPRs that are too narrowly focused, 3) CAPs not being written for related new conditions, 4) Related OE not bei ng properly evaluated, 5) Incorrect assumptions regardi ng CC capabilities to handle certain events 6) Incorrect assumptions regarding Tu rbine Building HELB impacts on plant systems, and, 7) HELB analyses focused on tem perature and pressure impacts to the exclusion of jet impingement and pipe whip While the RCE problem statement di scussed initiating events in the Turbine Building for HELB, seismic, and tornado, the focus of this Root Cause is only on HELB. The reason for this is that HELB is the only one
of the three where this issue c ould have been discovered as part of resolving known deficiencies. Docum entation showing the ability of the site to mitigate a HELB in the Turb ine Building is required documentation that the site does not have but was working on obtaining. If there had been adequate management of the HELB analyses and cold chemistry CC water piping resolution studies, these activities would have been
completed and the required documentati on and Engineering resolution in QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 9 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
place so that this issue would have resolved much sooner. For seismic and tornado, however, it is not likely that this issue would have been
identified through r outine efforts.
Contributing Cause:
Contributing Cause #1:
Station management has not developed adequate standards for OE evaluat ions with respect to Extent of Condition resulting in a lack of rigor applied to new issue
identification.
Contributing Cause:
Contributing Cause #2: Engi neering management has not developed expectations pertaining to CAP initiation for:
- 1) How long a potential issue can be investigated before it is documented in a CAP, and, 2) When a CAP should be written for valid issues identified in draft or otherwise unaccepted studies.
Corrective Action Synopsis:
The Root Cause is addressed by two CAPRs and two supporting CAs that work together to correct the root cause and ensure sustainability of the corrective actions. CAPR #1 develops and implements a HELB design basis document and program document. This effort will establish the
HELB requirements at Prairie Island and complete actions necessary to ensure the site is in compliance with the requirements. The supporting CA
- 2 and CA #3 determine the short term and long term personnel resource requirements for sustainability of the HELB program and develop a business case for these resources.
CAPR #2 revises 5AWI 6.0.0; "Int egrated Planning Proc ess" to ensure projects funded by department line budgets are subj ected to the site project review process through the PRG and the tracking of all on-going PRG-approved O&M studies and analyses.
A periodic status update of these activities would be provided to PRG. If any activity has been delayed or the scope changed, PRG will review the prioritization of the activity. Emphasis should be placed on those activities involving or potentially impacting risk-significant SSCs, particularly those activities still in the discovery stage. Depending on the nature of the study or analyses (for instance, when an OBN or OBD in volved), a plan should be generated to recover the delay or justification provided for the scope change.
EFR #1 for CAPR #1 will assess the e ffectiveness of the HELB design
basis and program documentation effort s by performing an external review
of the program. This assessment will ensure that any discovered QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 10 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
deficiencies have been placed in the Corrective Action Process and that the program provides an adequate basis for future operability assessments involving HELB. EFR #2 for CAPR #2 will assess the
revised PRG process and determine if it has been effectively implemented through personnel interviews and document reviews.
The associated extent of root cause is addressed by the following CAs:
CA #4 directs the development of program basis documents for other non-
fleet programs to capture essential pr ogram elements in one location. CA
- 5 identifies other stud ies and analyses that have the potential for discovering additional vulnerabilities to operability.
Contributing Cause #1 is addressed by TRRA #2 that evaluates training for all site personnel who currently per form OE evaluations. PCRA #1 will revise the fleet OE procedure (FP-PA-OE-01) to implement explicit requirements to consider an extent of condition evaluation be performed for OEs.
Contributing Cause #2 is addressed by CA #1 for the development of an expectation document covering the unique aspects of Engineering issues that could be potential CAQs. The document will address the
management expectations for how long these issues can be investigated prior to the initiation of a CAP and when a CAP should be initiated for issues identified in studies and analyses. TRRA #1 directs training for all
Engineering personnel on the revised Corrective Action Process expectations developed by CA #1. PCRA #2 will revise fleet Engineering guidance (FG-E-ARP-01) to incorpor ate the revised CAP expectations.
Reports to External Agencies:
The following reports were made to external agencies:
- August 7, 2008, CAP 01145695 was pos ted in the "Internal Operating Experience Report" (form QF-0407) t hat was submitted to the fleet.
- September 29, 2008, LE R 2-08-1, "Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break" was submitted to the NRC.
- October 8, 2008, the CC/HELB issue was posted on the Nuclear Network as OE27559 related to CAP 01146027, which was closed to CAP 01145695.
- January 19, 2009, LER 2-08-1, "Unana lyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break, Supplement 1" was submitted to the NRC.
- July 16, 2009, CAP 01145695 was again posted in the "Internal Operating Experience Report" that was submitted to the fleet. The QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 11 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
reposting contained additional info rmation related to the CC/HELB issue as well as documentation that this was an NRC white finding.
Additionally, when this RCE is comple te and approved by the PARB, the following actions will be taken:
- A follow-up posting will be made internally and the report will be shared with the fleet.
- OE27559 will be evaluated for update
- The NRC will be notified of the completion of the RCE and a copy will be provided.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 12 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
II. Event Narrative and Timeline Historical Information:
While the main scope of this Root Cause Evaluation focused on events occurring after 1990, there were a couple of historical events that should be mentioned as they are impor tant with respect to an overall understanding of this issue. Based on the team's understanding of the historical events, the original design of the PINGP included the Component Cooling water being routed out of the Auxiliary Building into the Turbine Building. No information could be found regarding the details of why this was done or what the
justification was for this design.
In December 1972, the Atomic Energy Commission published a letter (the "Giambusso letter") which required Prairie Island to address the consequences of pipe ruptures (HEL B) outside containment and submit their analyses for review. Prairie Island's response is documented in Appendix I of the USAR. The station analyzed the effects of pipe whip, jet impingement, temperature and pressure from HELBs in the Auxiliary Building (AB) and the impact on the safeguards corridor in the Turbine Building (TB) from HELBs. The impact of HELBs in the TB on the Component Cooling Water System (CC) was not addressed.
In 1977, based on the notes from Q-List Committee Meeting 77-12, the CC
piping in the Turbine Building was downgraded to QA Type III. Again, no information could be found as to why this was done or what made it acceptable. While this downgrade may have played a role in the mindset of the Engineering staff during future investigations, it was not the main reason for the failure of the site to adequately protect the CC system.
In the late 1980s, Prairie Island updated the AB HELB analysis in response to GL 87-11. The analysis has been updated several times since then.
RCE Event Narrative:
In 1990, based on response to NRC Generic Letter (GL) 89-13, "Service Water System Problems Affecti ng Safety-Related Equipment,"
Recommended Action IV, Prairie Island Nuclear Generating Plant contracted Pioneer Engineers
& Consultants, Inc to prepare a single failure analysis of the Component Cooling (CC) System for Units 1 and 2 (documented in 1995 as ENG-ME-240). The completed analysis recognized several single failure and QAI-to-QAII/III boundary barrier deficiencies, and discrepancies in QA classifications. Recommendations were made for system improvements that would resolve the identified deficiencies (D6). One of the vulnerabilities ident ified was that the surge QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 13 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
tank could empty in six minutes if interface barriers failed.
Follow-on-Item (FOI) A0108 focused on the SI pumps.
An operability determination did not identify any immediate actions or addr ess the operability of the CC system (D44). (Inappropriate Action #1)
During this same period of time the si te was resolving other Cooling Water issues with greater perceived safety significance in preparation for the 1995 Service Water Safety Operational Inspection (SWSOPI) to demonstrate compliance with GL 89-13.
FOI A0863 was completed in May of 1995 to address concerns raised by
the 1990 single failure analysis of the CC System for Units 1 and 2. The study indicated that the CC system had inadequate isolation in the Turbine Building. The FOI concluded that no impact on component or system operability existed, and no Justification for Continued Operation was
required (D49). In additi on, there was no recommended immediate or short term corrective actions since t he FOI had no impact on operability or technical specifications. (Inappropriate Action #2)
A Turbine Building HELB analysis, Calc ATD-0312 was completed by Sargent & Lundy in December of 1994, of which Rev. 0 was received on
December 8, 1994 and Rev. 1 received on April 19, 1995. In May 1999, Condition Report 19991622 (D52) was init iated following receipt of an S&L letter documenting errors and the affected calculations were cancelled. This calculation was also being used to support SE-419, Removal of Turbine
Building Steam Exclusion Dampers wh ich was completed on September 5, 1995 and rejected by the Operating Committee the following day. This project, 93L427, was cancelled a year later and SE-419 was cancelled in July 1997.
Work on the Turbine Building HELB analysis (temperature and pressure only) began in September of 2000 by c ontractor AES, but was terminated in 2001 as adequate funding was not availa ble (D53). Additional funding was
requested in April of 2002. The analysis was scheduled to be completed in November 2005.
In September 2000, IN 2000-20 was evaluated by the site concerning a HELB issue at DC Cook (D63). Though the IN mentioned HELBs in the
Turbine Building, the site only eval uated safety-related equipment affected by HELBs and did not review HELB impact on the CC in the TB. (Inappropriate Action #3)
In May of 2003 an INPO Evaluation was completed and resulted in AFI.2-2 which required a response by Prairie Island (D51). The AFI described that some Engineering programs and their supporting analyses and documentation, includi ng environmental qualific ation, high-energy line QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 14 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
break, in-service inspection and the quality list, were not well organized and easily retrievable. Prairie Island's re sponse, specifically with respect to high-energy lines, is below:
"The High Energy Line Break (HELB) issue has been organized into two major tasks; Auxiliary bui lding will be completed by December 2004, and Turbine building which will be completed by November 2005."
As of the completion of this invest igation, the Turbine Building HELB analysis has yet to be finalized.
In April 2005, Prairie Island receiv ed OE 20291 (D30) reporting an event at Kewaunee NPS (then a member of t he Nuclear Management Company).
The event pertained to an Auxiliary Feedwater Pump Suction / HELB interaction affecting operability of t he Auxiliary Feedwater System. Rapid OE notification was conducted and a formal evaluation was conducted under OEER 810473 (D45). The OE evaluation focused on the AFW
system and did not evaluate extent of condition (Inappropriate Action #6). The RCE team also found that the Rapid OE and OE Report contained
specific recommendations to perform extent of condition for other HELB issues.
In August of 2004, a year prior to the above OE, CAP 737382 (D3) was
written indicating Component Coolin g components were not Seismic Class 1 qualified. Several assignments were created from this CAP, however, no
extent of condition evaluation was performed (Inappropriate Action #4). Some of the assignments from this CAP include (but are not limited to) the following:
- 737382-1 Perform an OPR to determine potential non-conformance of CC system with respect to the non-seismic components
- 737382-2 Evaluate and if necessary initiate corrective actions
- 737382-3 OBD (00109) to track the pl an for resolution of the operable but nonconforming condition.
- 737382-4 Perform seismic analysis of 1-CC-138 up to CC-71-1 and CC-71-2. The operability assessment (OPR #509) only considered whether CC was on the Safe Shutdown Equipment List (SSEL) and was too narrowly focused (D8). (Inappropriate Action #5). Completion notes to Assignment 4 of AR 737382737382dated 07/20/05 (D3) indicated that, the CC piping to Cold Chemistry laboratory and 123 Nitrogen Compressor was to be further evaluat ed and may have to be modified or isolated in view of the revised HELB analysis. A new CAP was not written even though the mention of HELB was a new condition. The HELB issue remained under the seismic AR. (Inappropriate Action #7).
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 15 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
From these completion notes, it is apparent that a three phase plan was put in place to complete the study and in itiate modifications that would be necessary to resolve the existing seis mic and HELB conditions with the CC System. An Engineering A ssistance Request (EAR) 050120, "Component Cooling Piping to Cold & Hot Chemistry Labs are not seismically qualified &
HELB is a concern" (D26) was pr epared and reviewed at the 08/30/2005 PRG Screening Meeting. The EAR noted "the presence of safety related piping, pipe supports, and sample cool ers in a non-safety area of the Turbine Building may require extensive analysis to justify their ability and
that of the cold chemistry laborat ory to withstand seismic and tornado loads." It was decided to develop an Engineering resolution to the problem rather than wait for the co mpletion of the Turbine Bu ilding HELB analyses.
Initial funding was approved for the am ount of $10,200 to perform a study.
This study was performed by S&L and recommended two options (D2):
- 1. Modifying CC Piping in order to is olate the Cold and Hot Chem. labs, chillers and 123 Nitrogen pump from sa fety related CC piping during an accident 2. Analytically qualify nonsafety re lated CC piping and nearby block-walls for all applicable loading conditions described in the USAR (this option
was not recommended).
Option 1 for modifying CC piping to automatically isolate during an accident was determined not to be a feasible option and the study phase was started again. However, the study did indicate that t he CC system will not isolate on seismic or tornado events. A sy nopsis of the 2005 study was prepared by a site structural engineer. That synopsis indicated that a walkdown found that CC lines to the Cold Chem istry Laboratory were very near high energy feedwater lines. However, the significance of this HELB interaction was not recognized. (Inappropriate Action #8).
The synopsis was used to contact thr ee Architectural E ngineering (AE)
Firms to provide Requests for Proposals. Three proposals were received
from the AE's; one from Sargent & Lundy dat ed 02/28/06, one from AES dated 03/03/06, and one from Stevenson
& Associates dated 03/02/06. (D12, D25, and D62)
An Engineering Assistance Reques t was prepared that included the additional information gathered from the proposals received in February and March. This EAR was presented at the 07/24/06 PRG Screening Meeting (D59) and was "reviewed and approved
- pending the outcome of the 2006 budget recovery actions." Sargent & Lundy provided an additional
"Proposal for Preparation of Refined Alternatives for Isolation of the Component Cooling (CC) System Piping to Cold Chemistry Lab" February 15, 2007. This proposal was accept ed, and on March 30, 2007 a contract QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 16 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
was issued to S&L for preparation of "alternatives for isolation of the CC piping to the cold lab".
In December of 2006, the Q-List Report for the CC system was received by the site from Sargent & Lundy. The report was subsequently reviewed by five different site engineers, includ ing the CC System Engineer and the HELB Engineer. The Q-List Report for the CC system discussed the vulnerabilities with the CC piping in the Turbine Building and the need for the piping and associated components to have a safety function of maintaining system pressure boundary. The document also discusses the drain down of the CC surge tank in approximately 6 minutes based on a break in a 3/4" schedule 80 pipe. No apparent action was taken as a result of
these reviews by site Engineering si nce the report was not accepted due to errors in other portions of the report (Inappropriate Action # 9).
In June 2007, the Turbine Building HE LB analyses contractor was changed from AES to Advent. The study is now seven years old and still not
completed. The longer the delay in co mpletion, the greater the chance that other potential vulnerabilit ies due to this legacy issue will not be discovered so that they can be assessed for oper ability impacts. The site failed to recognize the significance of these continuing delays. (Inappropriate Action #10).
On December 24, 2007, assignm ent 00737382-12 was completed and indicated that CC piping would not be able to withstand a HELB condition or
tornado loads. CC System operability was not reassessed at this time as no new CAP was written. This was another opportunity to identify a new
condition on the CC system similar to that with the completion notes to AR
737382 in July 2005 (Inappropriate Action # 11) (D9).
A draft copy of the Sargent &
Lundy document titled "Chemistry Lab Component Cooling Study" was received on site in June of 2007 for review (D28). No review was performed and there was no turnover provided about the study by the HELB engineer who had left the company. However, in December 2007, ECR 3183 was written to add a closed loop cooling system
for the CC cold chemistry laboratory based on recommendations in the draft Sargent & Lundy study. (Inappropriate Action #12).
The final study was received in January 2008 (D28). The background section of the report discussed the purpose of the study. This section of the study presents the information that "PINGP calculation ENG-ME-240 concludes that a 100 gal/min pipe break would drain the surge tank in approximately 6 minutes." It goes on to mention "none of the piping to or from the cold chemistry laborat ory has been analyzed for HELB" and "the CC piping is not analyzed for tornado impact." The study documented
recommendations to address concerns with the CC/HELB interaction. A QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 17 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
subcommittee of the Study Review B oard reviewed the study but did not read the background section and only focused on the recommendations.
The study was not accepted because the subcommittee was not sure of the
purpose of the study. Not only was the significance of the information in the
background section missed but the resolution of the cold chemistry
laboratory CC piping issue continued to re main open, over 31/2 years after it was recognized that this issue needed to be resolved with an Engineering solution. (Inappropriate Action #13).
The governing procedure for engineering st udies at the time this S&L study was received was SWI ENG-26, "Dev elopment of Engineering Studies", Rev. 1. This procedure was not follow ed for this study. Per this procedure, externally prepared studies should re ceive an owners' acceptance review and SHALL be reviewed by the project manager and approved by the
project sponsor. Neither of these actions was comple ted for this study.
The procedure adherence aspect of th is was addressed in ACE 01162511-01 (D2).
On July 29, 2008, AR 01145695 was in itiated following a walkdown by design Engineering. This walkdown identified the same vulnerability
between the CC lines and a feedwater li ne that had been identified in the 2005 timeframe. During the operability determination phase of this newly initiated CAP, it was determined that a HELB near the Cold Chemistry Lab
had the potential to adversely affect the function of the entire CC system.
Therefore, at 13:45 on July 31, 2008, the station entered TS LCO 3.0.3 due to both U2 CC trains being inoperable.
The CC lines to the Cold Chemistry Lab were isolated at 16:12 on July 31, 2008 which allowed the station to exit LCO 3.0.3. An Operability Recommendation (OPR) was completed by Engineering on August 1, 2008 which conc luded that the Unit 2 CC System
was operable but non-conforming.
On December 15, 2008, AR 01162511 (D2) was written to document
missed opportunities to identify CC/HELB interactions from the Sargent &
Lundy study. An Apparent Cause Evaluat ion was conducted. The causal statement documented in the ACE was that Engi neering did not properly follow the procedures for acceptance of the engineering study (S&L Study from January 2008), which was complic ated by the wrong mindset when the staff requested the study.
On March 23, 2009, AR 01174370 was written to document another vulnerability with the CC syst em. CC piping to the 122 Spent Fuel Pool Heat Exchanger was not protected from a design basis tornado generated missile. This issue was discover ed based on a question that came up during the significance determination process (SDP) for the CC/HELB issue.
ACE 01174370-02 was comple ted on 6/11/2009 for this issue. The causal statement documented in t he ACE for this issue was that the lack of tornado QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 18 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
missile protection was most likely due to the addition of this heat exchanger by Pioneer to the original We stinghouse design without documenting acceptability of the change in configuration, prot ecting it to the same requirements as the 121 SFP HX, and clearly and permanently identifying the difference from the or iginal heat exchanger.
Also discovered during the SDP for CC/HELB was a Turbine Building Flooding issue. On April 15, 2009, AR 01178236 was initiated due to the
site not having a HELB flooding calculat ion for the Turbine Building. A CE was completed that performed a walk down of the Turbine Building for flooding concerns related to high energy line breaks.
On August 5, 2009, Prairie Island received an NRC inspection report
identifying a preliminary white finding pertaining to the CC/HELB interaction on Unit 2. This was followed by a final significance determination for the white finding on September 3, 2009.
During the conduct of this evaluation, it was determined that Prairie Island has not met the commitment made to INPO in 2003 to complete the Turbine Building HELB analysis by Novem ber 2005. AR 01192814 was written to document this issue.
III. Extent of Condition Assessment
- Extent of Condition Root Cause Extent of Condition:
The condition present in this even t was that the CC system could not perform its safety-related function if a HELB occurred in the Turbine Building and severed the CC piping. The question to be evaluated in this extent of condition is: Are there ot her plant systems or equipment that cannot perform their safety-related functions during the sa me or similar events? While it is possible that other latent design issues remain from the time of original construction, a complete verification of the entire as-built configuration of the plant is beyond the scope of this evaluation.
Additionally, it is difficult to identify problems that ex ist but have not been documented in the Corrective Action Process.
Various searches were made to captur e sources of past data collection to capture potential issues t hat may exist in the pl ant that are not being supported, widely known, or possibly be ing misunderstood. In April 2003, a team of site personnel was est ablished to respond to SOER 02-04 regarding the Reactor Pressure Vessel Head Degradation at Davis Besse. The recommendations and team assignments were to interview plant
personnel, perform an assessment of site personnel views on Nuclear QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 19 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Safety, and identify and document abnormal and long term unexplained plant conditions. Results of the SOER evaluation identified 438 equipment, design, and process issues. The issues were ranked by the team based on
safety significance and worst case out come. None of the process issues were graded. Long term program c oncerns such as PRA, EQ, Fire Protection, and Maintenance Rule were identified by site personnel as lacking adequate funding and personnel to meet their respective program
goals. Those issues that met the highest significance criteria were assessed in AR 0044901. (D42)
In May 2005, an all hands equipment reliability program was established.
Site personnel were requested to anonymously document any concerns
they had with system operation, reliability, or system problems they thought were not getting adequate attention. So me of the issues identified were already documented in the Corrective Ac tion Process but many were not.
Development of the 5 and 10 year plans were initially developed using these lists along with System team
- s. Most of the items i dentified were specific equipment or systems although comments were made to finish the HELB analysis and HELB related issues. A va riety of concerns were identified regarding low design margins and issues.
In August 2009, Business Planning and Development generated a list of PRG approvals given for study phase money since 2005 per CA 01162511-
- 12. A number of the studies have never been returned to the PRG with their results. A number of des ign related issues are on the open list. This list is
with the Engineering Design group for review and to address the products generated. Assignments for resolving these issues are included in AR 01162511. CA #5 also addresses any remaining issues. (D78)
As part of this root cause, revision 2.2 of the Probabilistic Risk Assessment (PRA) models was used to determine the plant systems that had the highest risk significance with different initiating events. The Top 11 systems/components were chosen from the list. Interviews were conducted with System Engineering owners to discuss if any issues were present with their systems regarding initiating events or any other potential design basis concerns such as seismic, tornad o, and HELB. Systems reviewed were Cooling Water, Aux Feed water, Station Air, Screen House Vent, Component Cooling, Reactor Coolant, RHR, SI, SVCS, Reactor Protection and RWST. Interview results indica ted that System Engineers were confident with the system design basis but in some cases may need additional support when justificati on is needed. System knowledge regarding design takes time to devel op (several years) which may not always occur due to personnel or process changes taking place. Those
interviewed believed a better mentoring process was needed to take place
in lieu of the turnover che cklist currently in use.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 20 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
An additional aspect of the extent of condition is related to identification of any additional vulnerabilities that may exist with the CC system or a HELB. Corrective Action (CA) 01145695-02 was co mpleted on 8/2/08 to address the extent of condition with respect to HELB issues in the Turbine Building.
This action was completed through the approval of EC 13000. This EC documented the extent of condition investigation to determine if any additional equipment located in the Turbine Building would be susceptible to pipe whip and/or jet im pingement following a HELB.
The result of the evaluation was that there are no additional concerns for the pipe whip or jet
impingement for equipment located in t he Turbine Building. This will be validated by the completion of CAPR #1.
Other vulnerabilities are related to to rnado missiles. An issue was identified on March 23, 2009 in AR 01174370 that documented the fact that CC piping to the 122 Spent Fuel Pool (SFP) Heat Exchanger was not protected from a design basis tornado generated missile.
As documented in ACE 01174370-02, on 5/15/09, a structural engin eer, an Engineering supervisor, and a contractor inspected the Auxiliary Building, Turbine Building, and Screen House walls and doors from insi de and outside and roofs and roof structures from outside for vulnerability to applicable natural phenomena of tornadoes, external flooding, and design snow load. No discrepancies noted and all open issues were resolved.
Additionally, as part of this CAP, CE 01174370-07 was completed to per form an extent of condition with respect to a HELB in the Auxiliary Building. The CE determined that no HELB concerns were noted in the Auxiliary Building.
As a result of the CAP written for the SFP heat exc hanger, Operations identified an additional concern in CAP 01174493 with CC piping going to the ADT evaporator and hydrogen recombi ner. These lines are located in the fuel handling area where they coul d be susceptible to missiles from a tornado. As a result of this issue, an extent of condition was performed per CA 01174493-04. For this extent of condition, a walkdown of the fuel handling area was conducted. This wa lkdown determined that there was no additional CC piping or other equipment which would require missile protection located in the area.
On April 15, 2009, AR 01178236 was initiated due to the site not having a HELB flooding calculation for the Turbine Building. A CE was completed that performed a walkdown of the Turbine Building for flooding concerns related to high energy line breaks. Several large pipe interactions were
noted on each unit. Some of these interactions have been screened out
based on Appendix I of the USAR.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 21 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Conclusions and Actions Needed Based on the extent of condition actions that have already been completed, no additional actions will be generated.
The concern expressed by Engineering personnel regarding inadequate turnover is being addressed by CA 01165133 from RCE 01165133.
Contributing Cause #2 Ex tent of Condition:
An Extent of Condition was performed for Contributing Cause #2 because uncertainty regarding CAP initiation is c onsidered a significant contributor to this event.
A missed opportunity to identify the significance of the CC/HELB issue found
during the course of this investig ation was the presentation of an EAR (Engineering Assistance Request) and RPA (Request for Phased Approval)
to the PRG sub-committee to req uest funding for a study to determine
solutions to the vulnerability of the CC piping in the Turbine Building. When the CC/HELB issue was presented to the PRG sub-committee, there was not
a separate CAP, potentially contributing to a lack of awareness of the issue.
A sampling of RPAs was reviewed to ensure there was a CAP associated with the issue being addressed.
RPA Title AR Status Modify Louvered Fire Doors for NFPA Code Compliance 1022720, 1026878 In progress Steam Exclusion Damper
Replacement Study Multiple AR's and an LER
1998. Study in 2010 Proposal to Prepare Scope Study for the Replacement of Cooling Water (CL) Valves Appears to be a maintenance issue with multiple CAPs
written. Unit 1 and Unit 2 Pressurizer PORV Backup Air 1156123 Study in 2010 Bus Load Sequencer
Processor Upgrade Multiple CAPs written to
document MRFF.
In Study phase D5/D6 Crankcase Breather
Modification Multiple CAPs written to
document unplanned LCOs.
Statements of LCOs being
entered and this project would
resolve the issue.
EC 11013 currently at a Canceled status Replacement of
Westinghouse DB-50
Breakers References that multiple
CAPs have been written on
the breakers.
Study in 2010 QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 22 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Nuclear measurements Corporation (NMC) Radiation Monitoring Upgrade Study Multiple CAPs written to
document the issue. Concern in the RPA is failing equipment.
Study in 2010
The RPA regarding the Louvered Door is similar to the CC/HELB issue as it relates to not ensuring design requirements were met. From the RPA it appears the site did not completely under stand the requirements for the fire areas with the installation of the louvered doors. There does appear to have been appropriate use of the Corrective Action Process once the deficiency
was recognized.
In most of the issues reviewed, the RPAs were written to address specific equipment issues. Accordingly, there were references to CAPs that had been written on the piece of equipment or statements that specifically state
or imply operability/functionality was addressed.
Conclusions and Actions Needed Based upon this review of same and similar processes and issues, it could not be determined that attempting to solve problems without formally documenting them in the Corrective Action Process is an ongoing issue. No additional actions are required to address this extent of condition for Contributing Cause #2.
- Extent of Cause The root cause was inadequate managemen t of the Turbine Building HELB analyses and the cold chemistry laboratory component cooling water piping resolution studies.
The Extent of Cause look ed at other areas where problems exist that are related to t he development of programs important to ensuring the plant meets design basis requirements.
RCE 1182488, 12 Circ. Water Pump lock out and RX Trip. The root
cause was found to be that the Cable Condition Monitoring Program development and implementation was not given sufficient priority.
AR 1132987, EIC Programs not recognized as Fleet Programs and AR 1148972, Motor Program not adequate for industry standards and site needs - both address programs that may not be receiving the proper priority.
AR 576240576240 INPO AFI EN.2-2 Engi neering Programs.
In 2004, INPO identified that "some Engineer ing programs and their supporting analyses and documentation, including environmental qualification, high-energy line break, in-service ins pection and the quality list, are not QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 23 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
well organized or easily retrievable.
Weaknesses in this area make it difficult for station personnel to readily establish licensing basis requirements- Management oversight of these programs is not sufficient to ensure program effectiveness."
These identified issues demonstrate that the extent of cause extends to other programs. CA #4 directs a revi ew of non-fleet progr ams that currently do not have a program basis document (H-series procedure). This action
should identify programs, other than HE LB, that currently do not have a program basis document that may be requi red so that operability or other evaluations potentially supported by those programs may be conducted.
Contributing Cause #1 is that St ation Management has not developed adequate standards for OE evaluations with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification. The extent of cause looked at a sample of recent OE evaluations to see if the extent of condition was adequate, as well as weaknesses with the OE process as a
whole.
A draft of Common Cause Evaluat ion 01183142, Trend in Ineffective Resolution of OE Items, identified the Common Cause to be a "Lack of Clearly Defined and Consistent Priorities
", "Lack of Functioning Engineering Work Management System", and "Incorrect Operational Focus". Some of the items evaluated also show signs of incomplete understanding of the issues. The incomplete understanding of issues is revealed where issues were identified but resolution has not been completed. There is also evidence from previously-evaluated operating experience noted in other Root Cause Evaluations, specifically RCE 01132717 (Site Response to
Issues with SI-9-5) that extent of condition and cause are not adequately addressed in OE evaluations.
The lack of adequate standards for OE evaluation is considered a site issue.
TRRA # 2 directs training of all personnel conducting OE evaluations to ensure that extent of condition reviews need to be considered when performing OE evaluations.
CC # 2 was that Engineering Ma nagement has not effectively communicated expectations of how long a potential issue should be investigated before it is document ed in a CAP and when a CAP should be written for valid issues identified in draft or otherwise unaccepted studies.
The extent of cause for this Contri buting Cause looked at challenges with using the Corrective Action Process across the entire site.
Comments from the 2009 Problem Identification and Resolution (PI&R) inspection exit meeting showed a c oncern with the impl ementation of the Corrective Action Process. The NRC identified that procedures associated QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 24 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
with Unit 2 Heater Drain Tank Pump , specifically the swapping of the pumps, did not meet the standard or requirements associated with Procedure Use and Adherence. This is an example of not properly identifying a deficiency that has been in place for a long period of time.
RCE 1141755, "Identified NRC Crosscutting Issues", identified issues with
the implementation of the Corrective Action Program. The RCE focused on resolution of issues and had actions to revise the Operational Decision Making procedure to provide the right level of response to degraded plant conditions and to train on the process. This is applicable to this RCE as it shows additional issues with the impl ementation of the Corrective Action Process.
Assignment 01 for CAP 01075890, "PI125 PCD Scoping Enrichment Error", evaluates the issue identified for a department clock reset. The evaluation
document for this CAP identified a ti meliness problem in that there was indication of a problem on January 18, 2007 but a CAP for the issue wasn't
written until February 6, 2007. This is applicable to this RCE as it shows the initiator not having a clear expectation as to the appropriate timeliness of CAP initiation.
Assignment 01 for CAP 01039647, "Res etting of 1LM-750 using cabinet 'RESET' switch", evaluates this issue for a clock reset. The evaluation document states the following: "A CAP was not wr itten by the SE as he felt that all necessary actions had been put in place to correct the lock up of Train A ICCM and at the time ICCM wa s not required to be operable. The idea that a compensatory action was bei ng performed to keep A Train ICCM operable by resetting the ICCM rack weekly was overlooked by the SE and by the Crew 2 STA. Had a CAP been written to document the need to reset ICCM based on the replacement CMOS board giving a Ram Error code one week after replacement, the idea of a compensatory action being performed would have been caught during screening. Th is would have resulted in the issuance of an OPR at that time which in turn would have prevented the unplanned LCO entry on 7/12/06."
This is applicable to this RCE as t he mindset demonstrated by the engineer and Operations is very similar to the mindset demonstrated by the engineers involved with the CC/HELB i ssue. The initiation of a CAP was overlooked as actions were in place to address this issue.
CAP 01121831, "CAP not wr itten for water leaki ng from 12 Desurger regulator". This CAP is also applicabl e as it relates to a CAP not being written when an equipment issue was i dentified. The individual who observed the issue did not write a CAP or inform the control room.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 25 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
CAP 01164967, "CAP Not Issued for Br eaker "As Found" Data Out of Tolerance". This CAP documents t hat a breaker PM work order was completed with notes that the as f ound testing was out-of-tolerance for one of the tests but no CAP was ever issued to identify this condition. This is applicable to the RCE as it is a situation where a CAP should have been written but wasn't.
Two recent CAPs demonstrate that expec tations for when to initiate a CAP and the information needed to write a CAP are not clearly understood by the site. AR 01193003 documents unclear expectations for CAP documentation following management observations. AR 01193499 documents that some
CAPs have an insufficient description of the issue. An adequate description of the issue aids in understanding the significance of the issue and initiating appropriate actions to resolve the issue.
These CAPs indicate that there is not alignment on expectations for CAP initiation.
The above information demonstrates that while this contributing cause evaluation is centered mainly on the Engineering department, it can also be applied to the site as a whole.
RCE 1211532 addresses the CAP initiation issues for the whole site. TRRA #1 directs Corrective Action Process training for all Engineering personnel for CAP expectations developed as part of CA #1.
Conclusions and Actions Needed
- The extent of cause of the root c ause extends to other studies and analyses not being completed in a timely manner so that potential vu lnerabilities are not identified. CA #4 directs a review of non-fleet programs that currently do
not have a program basis document (H-series procedure). This action should identify programs, other than HE LB, that currently do not have a program basis document that may be required when conduc ting operability or other evaluations that affect th ose programs. A dditionally, CAPR #1 institutes a policy for ensuring pr ojects funded by department line budgets are subjected to the site project review process and projects in the PRG process that are delayed or experi ence a scope change are re-reviewed.
CA #5 identifies other studies and analyses that have the potential for discovering additional vulnerabilities to oper ability. These actions satisfy the need to identify and fix potential deficiencies in other Engineering programs.
The extent of cause for CC #1 indica tes that the lack of adequate standards for OE evaluation should be considered a site issue. TRRA # 2 directs training of all personnel conducting OE ev aluations to ensure that extent of condition reviews need to be considered when performing OE evaluations.
This training satisfies the need to align site personnel for properly
performing extent of condition assessments for OE evaluations.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 26 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
The extent of cause for CC #2 found that revised C AP initiation criteria should be applied to the Engineering organization and the site. TRRA #1 directs Corrective Action Process training for all Engineering personnel for CAP expectations developed as part of CA #1. This satisfies the Engineering personnel training needs for the extent of cause for CC #2. . RCE 1211532 addresses the CAP initiati on issues for the whole site.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 27 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
IV. Operating Experience:
- Internal OE -
LER 95-06-00
- Determination that some component cooling system alignments are not within the Intent of Technical Specifications. Reviews of the component cooling system reveal ed that a single failure could make both component cooling trains of one unit inoperable when both trains of that unit are cross-tied. Corrective ac tions to correct the discrepancy were taken, but no extent of condition review was performed.
LER 1-00 Flooding from Postulated Failure of Air/Vacuum Valve has Potential to Disable Both Trai ns of Essential Service (Cooling Water). Opportunities to identify and evaluate an initial design deficiency were missed in 1990 and 1995. Contri buting to the failure "was the situation arising from the reviews and corrective actions performed for Generic Letter 89-13 and the Servic e Water Operational Performance Inspection." These activities involved many extensive and significant issues which overshadowed this initial design deficiency and may have contributed to the failure of Engi neering staff to otherwise identify and evaluate it. Corrective actions to correct the discrepancy were taken, but no extent of condition review was performed.
AR 01111291, 01112915, 01113170, 01117260 - These CAPs document
issues with operability determinations for equipment issues. Each CAP resulted in reopening the original C AP and updating the status notes to reflect basis for operability determinati on. All of these CAPs referenced deficiencies in implementing FP-OP-OL-01. These CAPs reflect the continuing trend in less-than-adequate operability reviews.
AR 01120989 (12/12/2007, SITE FAILED TO RECOGNIZE A POTENTIAL OPERABILITY ISSUE)
- This CAP documented that no CAP was written for a leak on a safe ty-related component. The CAP also documented that the site may not have recognized the significance of the leak. This CAP was closed to AR 01120914 which conducted a root
cause on the issues surrounding the co mponent. CAPRs from this root cause addressed only the equipment as pects, not the questions about the site's ability to recognize issues.
AR 01183142 (5/26/2009, TREND IN INEFFECTIVE RESOLUTION OF OE ITEMS) - This CAP documents an identif ied trend in the resolution of OE items. A common caus e evaluation is in progress. This CAP is an example of not fully evaluating and correcting deficiencies related to QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 28 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
industry OE and will address issues ra ised in this RCE pertaining to the site's effective use of OE.
RCE 01132717 (Site Response to Issues with SI-9-5)
- Over a period of 2 years from 2006 to 2008, the site failed to address operability issues related to internal leakage of SI-9-5 (t he first-off check va lve from the high pressure reactor vessel to the low pressure safety injection system).
During 1R24, SI-9-5 failed the internal leakage surveillance (SP 1070).
This failure was not recognized for its impact on operability of the valve and no causal evaluation was conducted. Failure to recognize a
potentially inoperable condition led to failures in the Corrective Action, Work Management and Operability/Functionality processes. The root
cause was determined that the organi zation has not developed a process for review of relevant engineering data as input s to decision-making and prioritization processes. The corrective actions included developing a procedure for use by Engineering personnel to include preparation and review of engineering data input to si te decision-making and prioritization processes (specifically Work Management Screening, Outage Scope Creation, Outage Scope Change, AR Screening, Engineering Change, Plant Health Committee, Project Re view Group, and Procedure Change).
This RCE also concluded that ther e were adequate opportunities through OE "for the site to understand t he implications and importance of preconditioning, but a lack of rigor ous adherence to FP-PA-OE-01 led to inadequate identification of applicability to the site."
ACE 01131913 (3/20/08, Monticello, HELB Program documentation deficiencies) - Gaps exist in the MNGP HELB program with respect to industry standards. The apparent cause of this event is that personnel
have an overall lack of knowledge of the HELB design and licensing basis.
A contributing cause was ineffective management oversight of the HELB program to ensure industry standard is maintained. Corrective actions included having the HELB program ow ner conduct a review of the MNGP
HELB license basis to improve understanding, and identify opportunities for enhancements or deltas between pr ogram documentation and license basis, including updating the HELB De sign Basis Documents as required. Additionally, the HELB program ow ner will perform informal industry benchmarking.
RCE 01100615-01, (7/11/07, CAPRs Closure Conflicts with Procedural Requirements) - Identified previous attempts to resolve the issue have not been fully effective. Human perfo rmance failure modes included wrong assumptions, inadequate verificati ons, inadequate tracking, and time and schedule pressure. From the report: "Co rrective actions to directly address the HU failures were considered, but not implemented. Previous efforts to address the HU aspects were not successf ul." The root cause cited wrong assumptions, inadequate verification and inadequate task management.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 29 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
The CAPR action cited to result in assurance that a plan exists to facilitate CAPR closure (a pre-planning meeti ng with requirement that owed to report the results) does not appear to have been proceduralized
RCE 01141755-01, (6/27/08, Identified NRC Crosscutting Issues)
- Concluded that the operational philosophy currently in place relies on skill sets and knowledge that no longer exist within the station and the organization is not placing appropriat e focus on plant issues, strategies, and/or the appropriate prio rity when they are identified. The root cause identified is the roles and responsibilities previously held by Engineering to address plant issues have not been effectively transferred to Operations to
promote a strong oper ational focus with contributing causes cited; high workload without proper prioritization and lack of critical skills throughout the organization. The created CAPRs, still in progress, are to revise the ODMI procedure and train Operations and Engineering. Corrective actions are in process to address the contributing causes. This effort is ongoing, due June 2010.
ACE 01174370-02, (6/22/09 No Torna do Protection of CC Piping for 122 SFP-HX) - One of the contributing causes was "evaluation of internal and external operating experience was ei ther not extensive or not used."
CCE 01183142, (8/20/09 Trend in Ine ffective Resolution of OE Items - Identified the Common Cause to be a "Lack of Clearly Defined and Consistent Priorities), "Lack of Functioning Engineering Work Management System", and "Incorrect Operational Focus". Some of the items evaluated also show signs of incomplete understanding of the issues. The incomplete understanding of issues is revealed where issues were identified but resolution has not been completed.
RCE 01157726, (10/30/09 PI Rad Sh ipment Arrives at Consignee above DOT Rad Limits)
- This RCE identified the fact that industry experience had not been effectively incorporated into the RMSP as a Contributing Cause.
RCE 01182488-03, (10/17/09 12 Circulating Water Pump Lock Out with Turbine/Reactor Trip) - 12 Circulating Water Pump had an electrical ground fault that resulted in a lockout of the pump. Lockout of the 12 CW Pump caused the circulating water flow rate through the condensers to be reduced to one half the normal flow. Th is led to a loss of vacuum and a Unit 1 reactor trip. The failure of the pump was caused by age related degradation of the 12 CWP power cable. The root cause was found to be
that the Cable Condition Moni toring Program development and implementation was not given sufficient priority.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 30 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
AR 866805866805 (7/14/05 Radiation Monitoring System [Top 10 Issue])
- A project to replace the containment air monitors due to repeated LCOs and outdated radiation monitor modules (NMC-71) was initiated. Numerous delays were encountered for this proj ect and the monitors have yet to be replaced (2/10). On 12/31/09, th e module replacement project was restarted after being inappropr iately closed in 9/06.
AR 1170596, (2/24/09 Iden tified Vendor Performance Issues with EDO
- S&L) - The vendor missed several due dates and failed to provide adequate design outputs for work asso ciated with the hydrogen storage project leading to delays. The vendor also failed to provide follow-on materials for the Security Barrier Upgrade project. This is the same vendor that delayed the HELB project.
AR 576325576325 (2/14/05 Prep are a Project Package for Lead Shielding)
- This CAP requests Engineering to initiate a study to review lead shielding
that is installed in the plant that does not meet the current plant design standards. To date (2/10) little pr ogress has been made on this project.
- External OE -
A search for operating experience was conducted by searching the INPO Plant Events Database for CC systems and HELB events. A search was also conducted using the OE s earch homepage for the phrases Component Cooling and High Energy Li ne Break. Below is a summary of related OE events along with descriptions of Prairie Island's response to each one:
Plant Event 316-980715-1, DC Cook, POTENTIAL FOR HIGH ENERGY LINE BREAK TO DEGRADE COMPONENT COOLING WATER SYSTEM. On July 15, 1998, with both units in an extended shutdown, the station determined that the potential ex isted for a postulated critical crack in the Unit 2 main steam line to degr ade the ability of adjacent component cooling water (CCW) pumps to perform their design function. The pumps are in a semi-enclosed area of the Auxiliary Building with a Unit 2 main steam line chase accessible from any of three doors. There is no calculation available to show that these doors can withstand the energy release from a postulated crack in the main steam line. DC Cook evaluated the event as NOTEWORTHY because the station's ongoing analysis had the potential to indicate the CCW pumps would not be able to perform their design basis functions.
There is no evidence that this issue was reviewed by the staff at Prairie Island. At the time of the INPO change date (09/20/1998) it was not part of the Prairie Island OE process to investigate Plant Events from other QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 31 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
plants. Therefore, the relevancy to our plant, and the close similarities between a HELB degrading Cook's CCW pumps and a HELB degrading Prairie Island's CCW piping, was never officially investigated.
INFORMATION NOTICE 2000-20, POTENTIAL LOSS OF REDUNDANT SAFETY RELATED EQUIPMENT BECAUSE OF THE LACK OF HIGH-ENERGY LINE BREAK BARRIERS. This information notice was issued by the NRC on December 11, 2000 in response to the above issue and similar issues at DC Cook 1 and 2. This notice used issues at the DC Cook site to lead discussion on four c onditions that must coexist in order to produce a risk-significant configurati on like that at Cook. They are 1) lack of HELB barrier between the r edundant trains of a system that is needed to mitigate accidents, 2) the lack of environmental qualification for the redundant components of trains located in the same area, 3) the
presence of high-energy piping in adj acent areas, and 4) the lack of a
HELB barrier between adjacent pi ping and the redundant safety system trains. It states that these conditions cannot be present for essential systems and components.
XOE 20006028, CONDITION REPORT: POTENTIAL LOSS OF REDUNDANT SAFETY RELATED EQUIPMENT BECAUSE OF THE LACK OF HIGH-ENERGY LINE BREAK BARRIERS.
This was Prairie Island's assessment of the issue r eported in IN 2000-20. It had only discussed the pressurizer PORVs as the only SSC with a HELB issue as concerning the Information Notice since the DC panels located in the Auxiliary Building, provid ing power to the PORVs, could be affected by a Main Steamline Break. It talked about a design change currently in place at the time that would move those panels to a mild environment location.
This report did not properly address t he entire issue of HELBs. It only looked at locations where the envir onmental conditions following a HELB would be detrimental to the SSC, such as described in the IN concerning DC Cook, and did not take into acc ount pipe whip or jet impingement.
OE23897 (updating OE20291), Kewaunee, THE CONDENSATE MAKEUP LINE TO AFW PUMP IS VULNERABLE TO A FW LINE BREAK. The Condensate Makeup (CMU) line from the Condensate Storage Tank (CST) to the suction of the Auxiliary Feedwater (AFW) pumps is routed in close proximity to the Main Feedwater (MFW) piping. If a HELB were to occur in the MFW pi ping, pipe whip or jet impingement could impact the CMU line. The lar ge amount of force from the MFW line break could significantly damage or break the CMU line.
The following event scenario was listed in the OE as being possible; The MFW line break results in failure of the CMU line. The reactor trips and all AFW pumps auto start due to lo w-low Steam Generator le vel. The failed CMU line may introduce air into the AFW pumps due to its location on the
suction side of the AF W pump. Air in the CM U line and/or pumps can QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 32 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
cause heavy cavitations within the AFW pumps. Air ingestion or heavy cavitations due to suction loss could result in damage to all three AFW
pumps, rendering them inoperable. Be cause of this scenario all three AFW pumps were declared inoperable.
Immediate and long term changes were made to bring these pumps back to an operable status. It also included the corrective actions that were to take place. Of these actions, one was to perform Extent of Condition walkdowns to look at other potential HELB issues and to provide assurance of design basis compliance for high energy line breaks.
OE 20291 was reviewed by Prairie Island staff via Operating Experience
Evaluation Request (OEER) 00810473. It was also recorded on T-Track number OE037389. The OE was subm itted on 02/21/2005 and the report was completed on 4/15/2005. In the r eport it was stated that if the CST piping were damaged, the resulting dr aining of the CST out the damaged line while still supplying the AFWPs, there would be an increased head loss at the AFWPs. It went on to say that Prairie Island has suction pressure switches installed to protect the pumps upon loss of suction so
that air would not be able to be inges ted into the pumps. It did not uncover any potential inoperability issues with the plant equipment. The
review was only focused on the effect s of the line/system damage to the CMU system, such as described in the OE , and it did not take into account
similar hazards a HELB may have on other systems, such as the CC system, even though the OE stated that SSCs should be "re-reviewed for system inter-relationships and associat ed implications" and the OE stated a corrective action of performing walkdowns to find other HELB issues.
This is seen as an instance of a missed opportunity to identify the issue of a HELB affecting a safety related system. Instead of the HELB issue being reviewed as part of the general OE process, the focus was aimed only at the result on the affected syst em. It was never applied to the broader "cause" of the in cident which was the po stulated HELB event.
Per the Operating Experience Program procedure FP-PA-OE-01, revision 2 (Attachment 4) which was the revisi on used during the timeframe this OEER was written, the O perating Experience Eval uation Guideline did not prompt the OEER writer to invest igate other systems or programs which may be effected by the same event cont ributors (i.e.: HELBs). The current
revision of FP-PA-OE-01 (revision 12) does have a section that tells the
writer to "describe ho w the event relates to applicable plant equipment, procedures/ processes/programs, hum an performance (barriers currently affected)."
Prior to the official release of OE 23897, NMC issued an Internal Operating Experience Rapid Notifica tion Report on February 21, 2005.
This report was very similar to the OE filed through INPO. However, in the section titled "Technical Considerations for Other NMC Plants," it makes the suggestion to "Review the adequa cy of equipment protection from QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 33 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
HELB events." This was another opportunity for Prairie Island to identify the issue of CC piping located next to HELB piping. There was no documentation found of a response to this internal report.
OE 20291 was later updated by OE 23897. The INPO change date for updating OE 20291 to OE 23897 was listed as 03/30/2005 which was prior
to the actual OE 23897 release date of 12/20/2006. Ther efore, there was no evaluation done on OE 23897 because t he date did not fall into Prairie Island's search criteria for "new" Operating Experiences. However, OE 23897 is very similar to OE 20291 in t hat it utilizes the same Abstract, Reason For Message, Description, Causes, Corrective Actions, and
Safety Significance. With this understanding it can be reasoned that OEER for OE 20291 would suffice for OE 23897.
The review of operating experience s hows that numerous similar events occurred at other sites and that Prairie Island's lack of fully evaluating
operating experience resulted in a missed opportunity to identify and correct this issue sooner.
- Conclusions and Actions Needed -
A review of the previous similar events and assessments demonstrate that the site has a history of issues wit h respect to management of engineering studies and analyses, the OE process, and the Corrective Action Process.
Corrective actions to address weakness in all of these areas have been
identified in the Corrective Action sect ion of this report. The review of operating experience shows that numerous similar ev ents occurred at other sites and that Prairie Island's lack of fully evaluating operating experience resulted in a missed opportunity to identify and correct this issue sooner.
V. Nuclear Safety Significance This evaluation found no significant evidence of Safety-Conscious Work
Environment (SCWE) failures as part of this sequence of events. However, discussions with engineers revealed the po tential for some resistance from site and fleet personnel with respect to t he identification of potential issues, specifically when all of the answers to the issue are not known, when more investigation is desired or when there are issues identified in studies or
analyses received by the site. Investigat ion of this issue determined that the resistance comes from a lack of clear understanding between engineers and Engineering management with respect to the expectations for CAP initiation. In addition, this appears to be a culture or mindset that has developed at the site over time. While this has not necessarily had a direct chilling effect, the continued toleranc e by management of this culture or mindset could be considered as having an indirect chilling effect. The QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 34 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
consequence of this is that potentia l plant concerns can linger for long periods of time without ever being known or understood by Operations and the site. While this was not found to be a significant contributor to the events identified in this report, the behaviors and interactions of management must continue to encourage the identificati on and free flow of information related to raising nuclear safety issues in the Corrective Action Process in a timely
manner. It is expected t hat the completion of acti ons for Contributing Cause #2 will address this issue.
The events of this report have result ed in a White Findi ng from the NRC.
The finding is a violation of 10 CFR Pa rt 50, Appendix B, Criterion III, "Design Control." PINGP failed to im plement design control measures to ensure that the design basis for the component cooling water system was correctly translated into specific ations, drawings, procedures, and instructions. Specifically, PINGP failed to ensure that the safety-related function of the component cooling water system was maintained following a high energy line break, seismic, or tornado events in the turbine building.
- The site failed to ensure that the Safety-Related function of the Component Cooling Water System was maintained following the initiating event of a High Energy Line Break (HELB) in the Turbine Building.
- During a HELB event, an unisolated section of the CC piping leading to the cold lab could break, potentiall y resulting in draining of the CC system within 6 minutes.
- The NRC conducted a phase 3 assessment of the impact of this event and found a delta core damage frequency of 3.2E-6 (White).
- The NRC finding is related to t he cross-cutting aspect Human Performance, Decision Making.
The root cause evaluation team eval uated Safety Culture Impacts for the root and contributing causes and for the extent of condition and extent of cause utilizing information in QF-0436 (Evaluation of Safety Culture Impacts) and NRC Inspection Manual 03-05. The evaluation is documented
in Attachment 7. Weaknesses in t he following Safety Culture components were a root cause and contributing causes:
H4a(Human Performanc e, Work Practices)
Basis: There was inadequate su pervisory and management oversight of the TB HELB analyses and the Engineering resolution studies for CC cold chemistry laboratory. Delays
in completion of these activiti es deprived the station of an opportunity to discover potential vulnerabilities in a legacy
issue (HELB impacts in the TB) in a timely manner. The longer the delays, the gr eater the chance these QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 35 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
vulnerabilities will not be discovered so that they can be assessed for operability impacts. In addition, the delays were not reviewed by the involved lead engineer or station management from the perspective of the increased risk of a legacy issue having potential fu rther vulnerabilities that were remaining undiscovered. The activities were funded
from the department line budget and therefore were not subject to a control process such as through the PRG.
Actions: This aspect is addressed by the Root Cause.
P1a(Problem Identification and Resolution, Corrective Action Program) Basis: New CAPs were not generated for the CC/HELB interaction at various times when aspects of this issue were identified, precluding the Corrective Acti on Process from potentially taking appropriate corrective action. Station Management had not developed standards pertaining to CAP initiation,
- 1) for how long a potential iss ue can be investigated before it is documented in a CAP, and 2) when a CAP should be written for valid issues ident ified in draft or otherwise
unacceptable studies Actions: This aspect is addressed by Contributing Cause #2.
P2a(Problem Identification and Resolution, Operating Experience)
Basis: There were missed opportunities to evaluate internal events and external events at other plants that were
related to this event.
Actions: This aspect is address ed by Contributing Cause #1 VI. Reports to External Agencies & the NSPM Sites The following reports were made to external agencies:
- August 7, 2008, CAP 01145695 was pos ted in the "Internal Operating Experience Report" (form QF-0407) t hat was submitted to the fleet.
- September 29, 2008, LE R 2-08-1, "Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break" was submitted to the NRC.
- October 8, 2008, this issue was posted on the Nuclear Network as OE27559 related to CAP 01146027, which was closed to CAP 01145695.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 36 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
- January 19, 2009, LER 2-08-1, "Unanalyzed Condition Due to Both Trains of Component Cooling Being Susceptible to a Postulated High Energy Line Break, Supplement 1" was submitted to the NRC.
- July 16, 2009, CAP 01145695 was again pos ted in the "Internal Operating Experience Report" that was submi tted to the fleet. The reposting contained additional information related to the issue as well as documentation that this was an NRC white finding.
Additionally, when this RCE is comple te and approved by the PARB, the following actions will be taken:
- A follow-up posting will be made internally and the report will be shared with the fleet.
- OE27559 will be evaluated for update
- The NRC will be notified of the completion of the RCE and a copy will be provided.
VII. Data Analysis A. Information & Fact Sources This root cause evaluation utilized the following as information and fact sources:
- Interviews - Interview list in Attachment 3.
- Procedures - A detailed list of referenced procedures is included in .
- Other references as described in Attachment 5.
B. Evaluation Methodology & Analysis Techniques This root cause evaluation utilized the Event and Causal Factor Chart to summarize and link events and Why Staircases to determine Causal Factors, Contributing Causes and R oot Causes. Failure Mode Analysis (Attachment 8), Safety Culture Analysis (Attachment 7), Barrier Analysis (Attachment 9), and Change Analysis (Attachment 10) were utilized to determine causal factors.
C. Casual Factors and Logic Ties Description
The pertinent events to this investigation are described in the event
narrative and illustrated in the Event and Causal Factor Chart. An QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 37 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
examination of this char t demonstrates that ther e are several recurring themes that eventually cascade into and impact decisions made over a period of almost twenty years. Th e original TB HELB analysis was incomplete in that it did not evaluat e TB HELB impacts on CC piping. Per the 2003 INPO AFI, the HELB document ation was not clear, well organized, or easily retrievable. This resulted in a continuing lack of understanding of the HELB Licensing Basis and difficulty in identifying and verifying design
inputs and assumptions.
This lack of understanding led in turn to a number of incorrect assumptions and a mindset regarding the TB HE LB impact on CC piping, as demonstrated by IA #1, IA#2, IA#5, IA#8 and, IA#11. These included:
- The original plant construction must be correct since it had been approved by the NRC
- CC was not impacted by design basis events
- The CC seismic design made it adequate for other design basis events
- A CC leak could be isolat ed within an acceptable time
- HELB analysis only includes temper ature and pressure effects to the exclusion of pipe whip and jet impingement These incorrect and unverified assumptions cascaded into and governed
decisions made over many years including:
- Inadequate prioritization of HELB analyses and studies
- Related OPRs that were too narrowly focused
- CAPs not being written fo r related new conditions
- Related OE not being properly evaluated When it was recognized that the TB HELB analysis needed to be updated in 1994 and later, in 2005, that an Engineering resolution was needed for the cold chemistry laboratory CC piping, the station was on a path to correct this legacy issue. However, neither t he analysis nor the studies developing an Engineering resolution were completed in a timely manner (IA #10). When delays due to funding and technical iss ues occurred, the mindset discussed previously resulted in them not being reviewed by the involved engineer or station management from the perspective of the increased risk of a legacy issue having potential further vulner abilities that were remaining undiscovered. As of the date of this investigation these activities are still not complete, which led to this event and the NRC violation.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 38 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Contributing to this event were failu res to perform adequate OE evaluations for related issues as demonstrated by IA #3 and IA #6 and to write a CAP when a TB CC HELB condition was identif ied or properly evaluate a related CAP as demonstrated by IA #4, IA #7 and IA #9.
In conclusion, while there was a lack of understanding of the HELB Licensing Basis and TB HELB impac ts on CC piping, the station did undertake the correct actions that would have successfully resolved this issue. Failure to complete those acti vities in a timely manner eventually led to this event. These delays deprived the station of an opportunity to discover potential vulnerabilities in a legacy issue (HELB impacts in the TB) in a timely manner. The longer the dela ys, the greater the chances these potential vulnerabilities would not be discovered so that they can be assessed for operability impacts.
VIII. Root and Contributing Causes Root Cause:
Lack of adequate documentation for HELB wa s, and is, a known deficiency.
An update to the Turbine Building HELB analysis was first attempted in the 1990s. Multiple attempts have been made to complete it without success. While money was typically made available for work on the analysis, the level
of effort from site Engineering was very limited. The HELB analyses were being worked on by only one civil/structural engineer who was also responsible for Tornado and Seismic iss ues and other emergent activities.
This is evident in the need to replace t he original contractor after over seven years into the analysis effort due to errors in the analysis. This most
pointedly demonstrates a lack of adequate management of this activity.
This inadequate management of HELB has allowed the si te to operate without full assurance of being able to ma intain the safety related functions of the CC system during HELB ev ents in the Turbine Building.
Equally important is the failure to time ly complete the Engineering resolution of the TB HELB CC cold chemistry laboratory piping issue. The decision was made to resolve this issue independent of the TB HELB analysis.
Considerable funding has been expended with various contractors with no firm resolution to this iss ue after 41/2 years of studi es. Again, this most pointedly indicates a lack of adequat e management of this activity.
Failure to adequately manage these activi ties so that they would have been completed in a timely manner has re sulted in this event. These delays deprived the station of an opportunity to discover potential vulnerabilities in a legacy issue (HELB impacts in the TB) in a timely manner. The longer the delays, the greater the chances these potential vulnerabilities will not be discovered so that they can be assessed for operability impacts.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 39 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Contributing to the event has been the failure to address the operability of the CC/HELB issue by means of initiating new CAPs when various aspects of this issue were discovered. Important to the nuclear safety of the plant is the identification of potential concerns that could challenge the ability of the plant to function as designed.
Engineering management needs to develop and communicate what the expectations are for CAP initiation, specifically when potential degraded or non-confo rming conditions are identified in studies or analyses. The failure of Engineering management to maintain proper oversight of the E ngineering staff by allowi ng this potential concern to go undocumented in the Corrective Ac tion Process was a contributing failure to this event.
Root Cause:
There has been inadequate management of the Turbine Building HELB analyses and the cold chemistry laboratory cooling water piping
resolution studies.
Contributing Causes:
Contributing Cause #1: Statio n management has not developed adequate standards for OE evaluat ions with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification.
Contributing Cause #2: Engineer ing management has not developed expectations for CAP initiation for:
- 1) How long a potential issue can be investigated before it is documented in a CAP, and, 2) When a CAP should be written fo r valid issues identified in draft or otherwise unaccepted studies IX. Safety Culture - conclusions The root cause evaluation team eval uated Safety Culture Impacts for the root cause, contributing causes, extent of condition, and extent of cause and identified weaknesses in the Safety Culture components of H4a(Human Performance, Work Practices), P1a(P roblem Identification and Resolution, Corrective Action Program), and P2a(Problem Identification and Resolution, Operating Experience). T he weakness in H4a led to delays in completing the necessary HELB studies and analyses. Weaknesses in P1a
and P2a led personnel to continually miss the opportunity to identify CC/HELB vulnerabilities through the prompt initiation of CAPs or conducting extent of condition evaluations for OE. As indicated in Section V, these weaknesses are addressed by the correctiv e actions identified by this root cause evaluation.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 40 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
X. Corrective Actions (SMARTS)
Corrective Actions to Restore (broke-fix)
- CC-20-4 (CC to Chem Labs from Unit 1 Supply Header) and CC-20-6 (CC from Chem Labs to Unit One Return Header) were verified closed to isolate the Cold Chem Lab from the rest of the CC System. This action is documented in the Op s Status Notes of AR 01145695.
Interim Corrective Actions (mitigation)
- The first quarter 2009 Human Performance for Engineers training used the CC/HELB issue as a case study regarding missed opportunities to identify a design flaw. (D83)
- Fleet (internal) Operating Experie nce on this event was distributed.
- A root cause team was formed to perform the evaluation regarding this issue.
Corrective Actions to Prevent Recurrence (CAPRs)
o
Description:
Develop a HELB Design Basis Document (H-series procedure) that provides an overall understanding of all
HELB requirements for PINGP. Also, develop a HELB Program Document that demonstrates how the site meets all of the identified HELB requirements.
o Owner: Engineering Director
o
Description:
Revise 5AWI 6.
0.0 to implement a requirement that when activities funded within an Engineering department line budget become projects, they are required to be entered into the site project review process through the Project Review
Group (PRG). In addition, re vise the AWI to identify responsibility for tracking the status of ongoing, PRG-approved, O&M studies and analyses. Require each of these ongoing activities to include a plan that details follow-on actions once a
study or analysis is completed. Require periodic updates of the status of these activities to the PRG. If any activity is delayed or the scope changes, the prioritization of all activities should be
re-reviewed. Factors to take into account should include those activities involving risk-signific ant SSCs, particularly studies or analyses still in the discovery stage or those involving OBDs or
OBNs. Depending on the nature of the study or analysis, a plan QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 41 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
to recover any delay should be developed and presented to PRG. Other Corrective Actions
- CA 01145695-XX (CA #1) o Due Date: 4/15/2010
o Owner: Engineering Support Manager
o
Description:
Develop Engineering expectations, for incorporation into FG-E-ARP-01 per PCRA #2, that cover at least the following aspects of CAP initiation: When there is a potential CAQ identified, how long should the issue be investigated prior to initiation of a
CAP? When a vendor document (such as a study or report, draft or final) is received that documents a potential CAQ, what level of validation should be performed prior to initiation of a CAP?
- CA 01145695-XX (CA #2) o Due Date: 6/1/2010
o Owner: Design Engineering Manager
o
Description:
As a support acti on for CAPR #1, determine short term personnel resource requirements for the HELB recovery program and develop a busine ss case to support those requirements.
- CA 01145695-XX (CA #3) o Due Date: 6/1/2010 o Owner: Design Engineering Manager
o
Description:
As a support action for CAPR #1, determine long term personnel resource requirements for sustainability of the HELB program and develop a business case to support those
requirements.
- CA 01145695-XX (CA #4) o Due Date: 6/18/2010
o Owner: Programs Engineering Manager
o
Description:
Review non-fleet programs (other than HELB) and develop, as appropriate, program basis documents for these programs to capture the essentia l program elements in a single location. (This action is similar to and should replace CA
01182488-14.)
- CA 01145695-XX (CA #5) o Due Date: 7/31/2010
o Owner: Design Engineering Manager QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 42 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
o
Description:
Identify PRG-approved studies and analyses that are not complete (scope has changed or there has been a delay). If they involve or impact risk-significant SSCs, particularly those activities still in the discovery stage, prioritization should be reconsidered and options explored to complete the discovery phase so that any vulnerabilities can be identified and assessed for oper ability. Depending on the nature of the study or analyses (f or instance, is an OBN or OBD involved); a plan should be generated to recover the delay or
justification provided for the scope change. In addition, any
studies or analyses being f unded by department line budgets that have not been through the PRG process should also be examined using the same criteria.
- TRRA 01145695-XX (TRRA #1) o Due Date: 7/16/2010
o Owner: Engineering Director
o
Description:
Engineering TAC to evaluate training need for Engineering personnel that reinfo rces the need for all potential concerns to be addressed in the revised FG-E-ARP-01.
- TRRA 01145695-XX (TRRA #2) o Due Date: 5/26/2010
o Owner: Engineering Director o
Description:
TOC to evaluate tr aining need for all site personnel who perform OE evaluations. T he training will emphasize what the requirements are for evaluating OE. Emphasis will be placed on the use of a broad mindset that considers extent of condition as well as the need for review of the OE by other individuals or groups. This tr aining will be required for the continued performance of OEEs.
- PCRA 01145695-XX (PCRA #1) o Due Date: 8/1/2010
o Owner: Fleet Performance Assessment Manager
o
Description:
Revise FP-PA-OE-01, specifically Attachments 2 and 4, to include an explicit require ment to consider Extent of Condition when conducting OE evaluations.
- PCRA 01145695-XX (PCRA #2) o Due Date: 6/15/2010 o Owner: Fleet Design Engineering Director
o
Description:
Revise FG-E-A RP-01, to incorporate the Engineering expectations for CAP initiation as developed in CA
01145695-XX (CA #1).
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 43 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
- GAR (GAR #1) o Due Date: 5/1/2010 o Owner: Project E ngineering Manager o
Description:
Develop a policy th at assigns a project manager to selected O&M projects based on criteria such as dollar amount, multi-year, etc. to relieve the engineer of the project management responsibilities.
Effectiveness Reviews:
- EFR 01145695-XX (EFR #1) o Due Date: 3/01/2013 o Owner: Design Engineering Manager
o
Description:
Complete an effectiveness review of the HELB program by performing an external assessment of the program. Effectiveness will be determined by no significant deficiencies (conditions adverse to quality exceeding a C level in AR
screening) found in the establis hed HELB program. This assessment will review the studies generated for the HELB program to ensure any CAQs have been identified in the
corrective action process. It will also assess whether the HELB documentation provides a reasonable basis for future operability determinations involving HELB.
- EFR 01145695-XX (EFR #2) o Due Date: 9/30/2010
o Owner: Design Engineering Manager
o
Description:
Complete an effect iveness review of the revised Business Planning and Developm ent process for activities funded by the Engineering department line budget or activities that change status (delay or scope change) while in the PRG
process. Effectiveness will be de termined by (1) satisfactory results from interviews of Engineering personnel concerning their perception of the proper im plementation of the process and (2) reviews of the required docum entation of safety-significant decision-making to ensure that the proper ev aluation was completed for a selected sample of activities funded within the
department that were determined to be projects and were subjected to the site project review process or that changed status while in the PRG process.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 44 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
XI. References See Attachment 5.
XII. Attachments
- Event and Causal Factor Chart : Root Cause Evaluation Charter : Interview List : Corrective Action Matrix : Supporting Evidence/Reference Documents
- Why Staircases
- Safety Culture Analysis
- Failure Mode Analysis : Barrier Analysis 0: Change Analysis
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 45 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event a nd Causal Factor Chart May 1990 D6Response to GL 89-13. Pioneer Study 892106 identifies vulnerabilities with CC system with respect to single failure and QA boundaries and classifications. Surge tank could empty in 6 minutes if interface barriers fail. HELB interaction with CC piping not mentioned.Eventually documented as ENG-ME-240.December 5, 1990D44Operability evaluation of Pioneer 829106 study noted several discrepancies with CC system. No immediate actions identified. SI pumps are considered operable.A AOctober 1995 D6CC single failure analysis entered into records as ENG-ME-240. Identifies inadequate isolation in Turbine Building.October 1999D52Site notified that previous HELB analysis from S&L had software errors. No impact to PINGP.IA #2May 1995 D49Pioneer Study 892106 CC Single Failure Analysis Open Issues. Inadequate and untimely evaluation of operability for isolation capability in the CC system. FO1 A0863Inadequate evaluation of loss of CC due to pipe rupture.WS 2May 5, 1995D47LER-95-06-00LER identified some CC alignments that are not single failure proof.December, 1994D53S&L Aux Building update and TB HELB analyses completed. Never reviewed or approved due to large number of errors. Analyses focused only on temperatures and pressures.Vendor calculation ATD-0312 BIA #1The evaluation only focused on SI pumps, remainder of CC system operability not addressed.(FOI A0108) WS 1 QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 46 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
- , f * ...,., -00' ea. A,_ ............. 00 (N!. 737382) mH a.e ",cMO'P'd to O!. 87.11. AESh_n;. 10tT1'0f""'*, presoo.re
& 10""09" eft a rrty, EWR 007.S4 OJ> ""tt ffi "" C C SyoI .... -U exte<tot m _. SeI*oT_ 2t, 21M I N 2t'00-20 ev_od b y 11>0 st. ,_dtIg I'BB _ .OC C OOIL ". -
b y I'E l B. (I N .....-HEL.Ban
", ,,<> 3 S"jSI ..... roc --L§:J -'-C4'R '!l7l 10 ..... _._ .-,..;sil
...... ** r'IOff","" Crii = -, --051,000 I NPO (,&,R 576;140) ""'1 1 2 _ ..... ......
EO. Hae,lST lliI or" cd wei <<l/IriZod (HEll! onoI\'SiI). DifII<>.A to
- _ &...-11 ...... _01"" ...giI __ mort tmIfo:iert AA .665 13 ** -, Of 10' HBB .tltiClr>g CoMen,oIeW roe t o WWP (Kew.......,). (JEER say WWP ar. ." *. The<eiSroo me<tiDn oj _ rU B -EOC-..ot e cb --> "" Cen<:eIO<I..-.te<
fQtoev_ T8 HELB ono/y.iS , , , * -00 , .. 1_01",. _po ...... ..... Au!r*1 , M4 00 (.tJI137J12)
-.
8'/.11 AE'S...... !"",-IlIUt O , 11' __ * & IICU"dIoY
- _. -_. _a_ '" IN 2OXI-2Q ev-..:l III' !I'll .ce--.I£&
_oIOCCo<*
- ...--by i'I3..!lt (I N .....-.H a8In Iho _ Wlc&'>gj. CC i>J"ll* & .ysl .... roc --,-
._-..... -** -.. -" -'" u ApIU._ OEERSlOol73 0E2O?!I1
,..., (IE' .... ttfUI..-..; (IUR __ .... .ole nw ** .., ...-.01_ i'ti.B --cb -*' M' EQIo_1lI , .. -*
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 47 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
......... nl S corrp.te. TIl HElEI AES .......,."is c<Jffpe!ed WI rod
'"' 10 ..-""" 28DS HaB J'fojec1 Cost --Aug.Bt _. 28 DS rn. P RO Saeeoiro;l Committee
'"<<>roved study to" CC pipir-.;I 0-.July 28. 2 MS 00 1", AA 737382 stol e lhat TB r-.eedslobe e v_ed l er HELB (pipe wtip ...-.:1 je1 &
...-.:1 "Wi Mv e 10 be lIl<l<ified
'" i.ololed. EAR 50120 _ oIed. Decide 1 0 Ii< CC system 1 0 cold...-.:1td cl>em 1M> ._than_ 1<r .=Jjs of TIlt£L B......,.,..,. AA 1<r HELB is",-,",,_Itten Octo_ 20 , 28e5 00 10< AA 73 7 382-04: TB CC ",,;.-,g oeeds 10 be e_ed I cr HEL B (EAA 5(120) Oct_ 28. 2MS HELB J'fojec1 Cost .July 2 5. ZOI S "' saL 11'01:<".01 1 0 oiI e 10' "=-e , 1o e st_h CC
...-.:1 cokt chern 1M> CC opt""",_ stol""lhatO-
_ Re-.tion J'fojec1 loo.n:l 1M! des>;J1 Msis 01 U; n>e<tory rue to Ood: 01 is_ion ... l he Oct_ 31. 2MS "' stlly.oeeived F ocus"" on I-ElEI.
OC does rod tornado e--*s sn:.e cd on SOlJG list. .........
- z.S
<t.>e 1 0 Of"""
2811S HB..B Projec1 Cost Aug.s1 _. 2811S "" PR O Sa--.ir>;i SIb-Committee Appro'ied study lor 0:: pipO-.;I 0-.July 28. 2MS 00 rdes I Of.oR 7CR382 state ltJoI TB r>eeds 10 be 1<r !-£LB (pipe-wtip ,..-,j jet & ,..-,j "WI h!ove t o be moo:tfied Of i ooloted. EAR SOl 20 generated. Decide t o Ii< CC system t o cold,..-,j td c l>em 1M> ,_ltJIon ..... 1<r ,=-Jts of TB HO LB <>noIyses AR 1<1" I-ftB on<! t Offl<ldQ Odo_20.2et1!i 00 I Of AR 73 7 38:2-04: TB 0:: "" .... oeeds l obe e_ed f a I-£LB (EAR SOl20) 0<< __ 28. 280S = HEL B Projec1 Co,,", .July 25. ZOIS "' saL ",onh PfOlXlMl t o site Ie< studyi">;j oIemotive s to est_h 0::
I:o.-J<Wy he( on<! cokf chern 1M> 0:: options. stat""ltJoIO--
UsIr-.-q_
ResoUion Projec1 100..0:1 thai de>>;Jl _is evmco.Jdc""",,!oos Of u; ..... e<tory rue to lad: o l lSOO1tion n t he Oct __ 3 1. 280S W saL CC boon:Iory F oc="" on f£11l. stoles OC doe , rd is_eon S_Of tornado e_ sroe rdonSOUG list.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 48 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
......,,,---.= sbAySl ... _ __ foo,rd C C n" t o cold _ 0Ib tou<tling!Wl>
_IIY -.. -.. --_101>00001
_1r>e*RFI'> ** r ..... elO Has Irtetoction 0I!.-....... tt_ f-..ory' ........ -025, Dti2. oee S&l. AES ,"'" $&A """""'* 10 cc cold lob __ 01 saL Q..LisI repOo1 to CC ** fetoruory, *7_ ..... 1, ,-= IOU",,_ .........
,2.11 TBHaBanoIysiS
--..... ."",.o.e;IOAdY<<1I .ft .. ", __ ,O.,ES --* ** ,-.........
"" sUI 001 """,*"e , .-..., _m
........ ,,---= ---___ ex hos 10 0:<*1 "*" lID Iau<:hr!g loQI\_lIJ .... -_10 ....... ** r .... 10 ._. H EUlwe. __ ot __ ... ,,-rdn.-y' ...........
-025.IlIU , 06lIII SIL, I-Ei., _ SUo """""'* to ....,.. OC cdd lob fix ... Ie..II .....,,24,_ PlIO Sr;reri'lg IoIrlg __ ex Pipngo" CGt:I 0.. UI> por-.tng '" :!OOS f!o.dgeI tAR501120
_11._ "" RIIaIip of sa o.tJOI --" ** _. Hf'-Api!, ., -= ._--........ ,ZM' Tete.S.....,. .... ... .oESIO_ ....... ---...... * **
.... _d ---. --
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 49 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
" .July 2 007 _ .J ..... u y 0,2 000 0'" saL stud y rec_e d wthrecommend..tions f or t he CC to the cold !&b. stud y C C in TB not for HELB 01' tornado Apo-il , 2 00 1 0" Action ,,=>;)ned from AR 737C22. Determine design requirements f er HE LB in TB. O e c eJrt>et-2 4. 2007 Note 10 AR 737382737382Determine CC not_Io ...... h st.,..-.;:f HELB or lorr>Oldo IA #'1 1 NewA R not 'MItten fot" HELB CC lholt resufted in May , 2 00 0 Au __ , 2 00 1 I--SaL propos&! for temp mod 10 cold I"b HELB f-O e c eJrt>et-2 4. 2 00 7 OW ECR 3183 _ill .... 10 oodd s.,."tern f er cold c hern I"b per SaL a"n study. Approved 2.6.08 by PRO. IA #'12 E C R 3183 ;"sued using poIrt i<oly reV'ie<.o.ed S aL study __ ;". .July 1 0 , 200 0 0'0 Tur bine BuiIdirJg HELB f.....-.:liro;J d eh.ys .J a ou"'Y, 2 0 0 1 S aL rele .. fin&!
HELB study IA 1>'13 O t 2 ,D41 R e solution of cokf chern !&bCC piping re""""", open. .July H , 2 00 1 0' CAP _ittenlo document CC PiJoin!;!
- " -... .J .. nuooy , 200 0_ May. 200 1 S&L .,..-.;
- f OOUIho<;:<e d 10 develop cokf !&b HELB fix. .July 3 1 , 200 0 1 346 Ent e red TS LCO 3.0.3 due 10 both U2 CC Ir"in" hope<"bIe. , H .July 2001 _ .J ..... ary 0.2000 saL study rec_e d _h recommendd:ions f er t he CC to the cold _. study CC in TB not desio;;roed fer I-ELB seismic 01' ternado:>
Apo-il , 200. Action "=9>ed r om AR 737332737332 Determine requiremerts f er HELB in TB. OeceJrt>et-
IA #'11 Ne<N A R not _enfOf" HELB CC lholt resulled in May , 2000 August, 2 00 8 I--SaL propos'" for lemp mod 10 cokl 1<tIl HELB issue f-O e ceJrt>et-24. 2001 OW ECR3 1 83_1Ite.--.lo MId s.,."tern fer cokl chern 1<tIl per SaL dr"fI study. Approved 2!6J08 byPRO. IA #'12 E CR 3183 lJ<'ing pa.-liool y revie<hed SaL study __ is. .July 1 0 , 2000 0'" Turbine EIuiIdirv HELB f.....-.:tino;J_
,,'r-' .J ..... u..-y, 200 8 SaL r_""ses fin&!
HELB study. IA #13 012,D41 R es olution of cokf chern_CC pipin!l rem<W>s open. .July H , 2 00 8 0' C A P _itlenlo document CC P ipin\j is ,.q&Cert 10 HELB -... .Ja nu..-y , 2000_ May , 2008 S&L
....-.d is ""-'Iho<il:
e d 10 develop cokf _ HELB fi x. .July 31 , 2000 1 3 46 Ent e red TS LCO 3.0.3 due 10 both U2 CC hope<" .. bIe. ,
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 50 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
July 31, 2008D66CAP written to Address Unit 2 CC/HELB issues. Closed to AR 01145695.July 31, 2008 1612D20Isolated U2 CC Supply to Cold ChemLab. Exited LCO 3.0.3.August 1, 2008 D5OPR Complete for CC SystemAugust 6, 2008D70ECR 3653written to provide cooling to hot & cold chemlab. (duplicate to ECR 3183 from Dec. 2007)September 18, 2008 D2ACE 01145695No CAP initiated in July 2006 to determine operability for impact of HELB on CC.December 15, 2008 D2Missed Opportunities to Identify HELB/CC System Interactions from S&L Study.AR 01162511ACE completed 1/15/09.January 15, 2009 D5RFP to S&L for CC piping study.July 30, 2009S&L study approved by PRG.January, 2009S&L Proposal to resolve cold lab HELB issue since temp mod worked stopped.August 7, 2009 D51Prairie Island has not met commitment to INPO made in 2003 related to completion of HELB analysis by November 2005. AR 01192814 submitted.Untimely resolution of cold lab HELB issue with plant mod proposals. Began 7/05 -still not fixed in 1/2010.
I J JRCE 01145695 QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 51 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 1: Event and Ca usal Factor Chart (Cont'd)
AEventFailed BarrierBarrierBeforeNowChangePresumptiveEvent WS #Terminal Event IA #No action was taken in response to the Q-List report for CC 9Failure to recognize potential HELB effects 8A new AR was not written to document the new HELB issue 11Safety significance of incomplete TB HELB analysis not considered 10ECR issued without proper justification 12Untimely resolution of cold chemlab CC piping issue 13Inadequate OPERABILITY determination 2Inadequate OE extent of condition evaluation.
3Inadequate OE extent of condition evaluation.
6A new AR was not written to document the new HELB issue 7Inadequate OPERABILITY determination 5Extent of Condition was not applied to CC CAP 4Inadequate OPERABILITY determination 1Description IA 1Logic TieChronological TieContributing CauseRootCauseCausalFactorPresumptiveCondition QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 52 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 2: Root Cause Evaluation Charter Root Cause Evaluation Charter CAR AR # 01145695 RCE# 01145695 Manager Sponsor: Scott Northard Problem Statement:
The station failed to ensure safety relate d functions of the component cooling water system were maintained for initiating events (HELB, tornado, seismic) in the Turbine Building Investigation Scope:
- This RCE will o evaluate the breakdowns of the proce sses and execution of those processes that allowed design deficiencies to remain uncorrected following the initial discovery
- This RCE will also evaluate the extent of conditions and cause(s) that are determined.
The output of the investigation will be corrective actions to prevent similar issues in the future.
Investigation Methodology:
To determine the root cause the team will employ event and causal factor charting, task analysis, barrier analysis and personnel interviews. NRC Inspection Criteria from 95001 will be used as a guide. This will include review of the extent of condition, extent of cause, Safety Culture attributes, review of the oversight and monitoring by the organization, and the past evaluation of the condition and the actions taken.
Team Members:
Team Leader Jeff Connors Design Engineering Team Member Chris Lethgo System Engineering Team Member Nate Adams Design Engineering (new engineer)
Team Member Ryan Cox Program Engineering Team Member Andy Notbohm Operations Team Member Kara Hernandez(Christopher) Monticello Systems Team Member Dave Pennington Monticello Systems Team Member Deb Albarado Organizational Eff.
RCE mentor Gene Woodhouse Performance Assessment Consultant Bob Hite Radiation Protection
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 53 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 2: Root Cause Evaluation Charter (Cont'd)
Rewrite Team Members:
Team Leader Pete Wildenborg RP/Chem Team Member Jim Sumpter CERTRC Team Member Rob Sitek Engineering Admin Support Kim Bromberek BPA Team Advisor Betsy Rogers Training Team Advisor Andy Notbohm Operations
Milestones: Rewrite Milestones:
Date Assigned: 8/03/09 1/11/10 Status Update: 08/17/09 1/18/10 Draft Report: 08/24/09 2/3/10 Final Report: 08/31/09 2/12/
10 PARB graded and approved Communication Plan:
A copy of the approved RCE will be provided to licensing for submittal to the NRC within two weeks of approval.
Approved:
Date: Scott Northard Approved by: Screen Team / PARB on 1/8/2010 (circle one) (Date)
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 54 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 3: Interview List Interview List
Design Engineer A
Design Engineer B
Design Engineer C
Design Engineer D
Design Supervisor A
Design Manager A
Design Manager B
Design Manager C System Engineer A System Engineer B
Project Manager A Finance Engineer A
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 55 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CAPR 01145695-XX (CAPR #1)
Due Date: 11/30/2012 Owner: Design Engineering
Manager Develop a HELB Design Basis Document (H-series procedure) that provides an overall
understanding of all HELB requirements for PINGP. Also, develop a HELB Program Document that demonstrates how the site meets all of the identified HELB requirements. RC: There has been inadequate management of the Turbine Building HELB analyses and the cold chemistry
laboratory component cooling water piping resolution studies.
Due Date: 6/10/2010 Owner: Engineering Director
Revise 5AWI 6.0.0 to implement a requirement that when activities funded within an Engineering department line budget become projects, they are required to be entered into the site project review process through the Project Review Group (PRG). In addition, revise the AWI to identify responsibility for tracking the status of ongoing, PRG-approved, O&M studies and analyses. Require each of these ongoing activities to include a plan that details follow-on actions once a study or analysis is completed. Require periodic updates of the status of these activities to the PRG. If any activity is delayed or the scope changes, the prioritization of all activities should be re-reviewed. Factors to take into account should include those activities involving risk-significant SSCs, particularly studies or analyses still in the discovery stage or those involving OBDs or OBNs. Depending on the nature of the study or analysis, a plan to recover any delay should be developed and presented to PRG.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 56 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CA 01145695-XX (CA #2)
Due Date: 6/1/2010 Owner: Design Engineering
Manager As a support action for CAPR #1, determine short term personnel resource requirements for the
HELB recovery program and develop a business case to support those requirements.
CA 01145695-XX (CA #3)
Due Date: 6/1/2010 Owner: Design Engineering
Manager As a support action for CAPR #1, determine long term personnel resource requirements for sustainability of the HELB Program and develop a business case to support these requirements. EFR 01145695-XX (EFR #1)
Due Date: 3/01/2013 Owner: Design Engineering Manager Complete an effectiveness review of the HELB program by performing an external assessment of the program. Effectiveness will be determined by no significant deficiencies (conditions adverse to quality exceeding a C level in AR screening) found
in the established HELB program. This assessment will review the studies generated for the HELB program to ensure any CAQs have been identified in the corrective action process. It will also assess whether the HELB documentation provides a reasonable basis for future operability
determinations involving HELB.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 57 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION EFR 01145695-XX (EFR #2)
Due Date: 9/30/2010 Owner: Design Engineering
Manager Complete an effectiveness review of the revised Business Planning and Development process for activities funded by the Engineering department line budget or activities that change status (delay or scope change) while in the PRG process. Effectiveness will be determined by (1) satisfactory results from interviews of Engineering personnel concerning their perception of the proper implementation of the process and (2) reviews of the required documentation of safety-significant decision-making to ensure that the proper evaluation was completed for a selected sample of activities funded within the department that were determined to be projects and were subjected to the site project review process or that changed status while in the PRG process.
Extent of Root Cause: CA 01145695-XX (CA #4)
Due Date: 6/18/2010 Owner: Programs Engineering
Manager Review non-fleet programs (other than HELB) and develop, as appropriate, program basis documents for these programs to capture the essential program elements in a single location. (This action is similar to and should replace CA 01182488-14.)
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 58 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CA 01145695-XX (CA #5)
Due Date: 7/31/2010 Owner: Design Engineering
Manager Identify PRG-approved studies and analyses that are not complete (scope has changed or there has been a delay). If they involve or impact risk-significant SSCs, particularly those activities still in the discovery stage, prioritization should be reconsidered and options explored to complete the discovery phase so that any vulnerabilities can be identified and assessed for operability. Depending on the nature of the study or analyses (for instance, is an OBN or OBD involved), a plan should be generated to recover the delay or justification provided for the scope change. In addition, any studies or analyses being funded by department line budgets that have not been through the PRG process should also be examined using the same criteria. TRRA 01145695-XX (TRRA #2)
Due Date: 5/26/2010 Owner: Engineering Director TOC to evaluate training need for all site personnel who perform OE evaluations. The training will emphasize what the requirements are for evaluating OE. Emphasis will be placed on the use of a broad mindset that considers extent of condition as well as the need for review of the OE by other individuals or groups. This training will be required for the continued performance of OEEs.
CC #1: Station management has not developed adequate standards for OE evaluations with respect to Extent of Condition resulting in a lack of rigor applied to new issue identification.
PCRA 01145695-XX (PCRA #1)
Due Date: 8/1/2010 Owner: Fleet Performance Assessment Manager
Revise FP-PA-OE-01, specifically Attachments 2 and 4, to include an explicit requirement to
consider Extent of Condition when conducting OE
Evaluations.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 59 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 4 CORRECTIVE ACTION MATRIX CAUSAL FACTOR AR # ACTION CA 01145695-XX (CA #1)
Due Date: 4/15/2010 Owner: Engineering Support
Manager Develop Engineering expectations, for incorporation into FG-E-ARP-01 per PCRA #2, that cover at least the following aspects of CAP
initiation:
- When there is a potential CAQ identified, how long should the issue be investigated prior to initiation of a CAP?
- When a vendor document (such as a study or report, draft or final) is received that documents a potential CAQ, what level of validation should be performed prior to initiation of a CAP?
TRRA 01145695-XX (TRRA #1)
Due Date: 7/16/2010 Owner: Engineering Director Engineering TAC to evaluate training need for Engineering personnel that reinforces the need for all potential concerns to be addressed in the revised FG-E-ARP-01.
CC #2: Engineering management has not developed expectations pertaining to CAP initiation for:
- 1) How long a potential issue can be investigated before it is documented in a
CAP, and,
- 2) When a CAP should be written for valid issues identified in draft or otherwise unaccepted studies.
PCRA 01145695-XX (PCRA #2)
Due Date: 6/15/2010
Owner: Fleet Design Engineering
Director Revise FG-E-ARP-01, to incorporate the Engineering expectations for CAP initiation as developed in CA 01145695-XX (CA #1).
OTHER GAR (GAR #1)
Due Date: 5/11/2010 Owner: Project Engineering Manager Develop a policy that assigns a project manager to selected O&M projects based on criteria such as dollar amount, multi-year, etc. to relieve the engineer of the project management
responsibilities.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 60 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 5 Supporting Evidence/Reference Documents
- Identifier Description D1 FG-PA-RCE-01 (Rev 15)Root Cause Evaluation Manual D2 AR 01162511 Missed Opportunity to ID Issue from S&L Report D3 AR 00737382 Non-Seismic Equi pment in CC System Pressure Boundary D4 SWI ENG-26 (Rev 2) (Legacy) Development of Engineering Studies D5 AR 01145695 CC Piping Adjac ent to HELB (Original ACE Conducted) D6 ENG-ME-240 (Rev 0) CC Single Failure Analysis D7 FP-E-SDY-01 (Rev 0) Development of Conceptual Design Concept Studies D8 OPR #509 (Rev 0) Non-Seismic E quipment in CC Pressure Boundary D9 AS 00737382-12 CC Piping not able to withstand HELB D10 ECR 3183 Add closed loop cooling system D11 FP-E-MOD-04 (Rev 3) Design Inputs D12 SLRP-2006-029 S&L Proposal D13 FP-PA-ARP-01 (Rev 10) CAP Process D14 FP-PA-ARP-01 (Rev 18) CAP Process D15 FP-PA-ARP-01 (Rev 22) CAP Process D16 FP-PA-ARP-01 (Rev 3) CAP Process D17 AR 000826114 Perform Seismic Analysis D18 PRG Minutes, 8/30/05 PRG Minutes D19 AR 01143812 Turbine Building Funding Delays D20 Operations Log En tries from 7/31/2008 D21 RCE 01013473 (Rev 0) D6 High Crankca se Pressure resulting in Unit 2 Shutdown D22 RCE 000185 (Rev 0) EHC Project Over Budget and Behind Schedule D23 RCE 888596 (Rev 1) Organizational Response to Operational Issues D24 RCE 01115585 (Rev 0) D5 Inoperab ility - Organizational Issues D25 AES Proposal, 3/3/06 Provide engineering services for CC piping Project D26 EAR '05 Funding CC project D27 EAR '06 Funding CC Project D28 SL-11973-014 (01/08) Chem Lab Component Cooling Study, Draft and Final D29 Plant Event 316-980715-1 Potential for HELB to degrade CC system D30 OE 20291/23897 Condensate MU line to AFWP is vulnerable to FW line break D31 Plant Event 36335 Several plant locations discovered to be unprotected HELB areas D32 Plant Event 315-981116-1 HELB could result in conditi on outside design basis of AF D33 RCE 01141755 Identified NRC Cross-cutting issues D34 RCE 01132717 Site Response to Issues with SI-9-5 D35 RCE 01132098 Site Response to 11 TD AFWP Turbine Bearing QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 61 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 5 Supporting Evidence/Reference Documents
- Identifier Description Failure D36 RCE 01144249 TSC Ventilation System Not Maintained Functional D37 RCE 01157726 Radioactive Material Shipment Exceeded DOT Limits D38 RCE 01166830 Inadequate CAP Resolu tion of Significant Issues D39 ACE 01154831 Failure in application of conservative assumptions resulted in a delay in implementation of corrective actions D40 FP-E-PHC-01 (Rev 0) Plant Health Committee D41 Interviews Interview Sheets D42 AR 449041449041SOER 02-04 3a Review D43 1987 CC System 1987 CC System Upgrade Study D44 FOI A0108 Determine SI Pump Seal and Lube Oil Cooling Requirements D45 OEER 00810473 OEER For OE 20291 D46 LER 1-00-03 Flooding from postula ted failure of air/vacuum valve has potential to disable both trains Essential Service (Cooling) Water System D47 LER 95-06-00 Determination that some CC alignments are not within the intent of TS D48 FOI-A0862 CL single failure analysis open issues D49 FOI-A0863 CC single failure analysis open issues D50 AR 01072605 Revised U2 FW analysis has new HELB cracks requiring evaluation D51 AR 576240576240INPO Commitments Engineering Programs AFI D52 CR 19991622 HELB Analysis from S&L contains errors D53 AR 34876/34885 Turbine Building HELB analysis, documented finding problems D54 RPA's Reviewed for EOC D55 LRP Reviewed for EOC D56 AR 60820 Steam Exclusion Dampers-excessive leakage D57 AR 32717 Steam Exclusion Dampers D58 NRC IR Related to ineffective corrective actions D59 PRG Minutes 7/24/2006 PRG Screening Committee Minutes D60 FP-OP-OL-01, Rev 0 Operability Determination Process D61 FP-PA-OE-01, Rev 2 Operating Experience Program D62 06PBOS-1020 S&A Proposal for CC Piping Project D63 IN 2000-20 NRC Information Notice, dated Dec 11, 2000 D64 AR 871749871749HELB Project Cost Overruns D65 AR 01002268 HLEB Project Cost Overruns D66 AR 01146027 Unit 2 CC/HELB Issues D67 CC Report, Rev 0 Q-List CC report, dated 12/11/06 D68 RFPs and proposals for CC piping D69 S&L Cost Estimate fo r Study to Upgrade Hot and QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 62 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 5 Supporting Evidence/Reference Documents
- Identifier Description Cold Labs 1/30/08 D70 ECR 3653 Alternate Cooling for the Cold Lab D71 ECR 13000 Turbine Building HELB Walkdown D72 PI SDP for CC/HELB D73 5AWI 3.7.0 rev.3 Operating Experience Assessment D74 TS 3.7.7 Component Coo ling Water (CC) System D75 Kewaunee RCE, "AFW Pumps Susceptible to Damage from Air Entrainment" D76 5AWI 3.15.5 rev 0 Operability Determinations D77 5AWI 3.15.5 rev 14 Operability Determinations D78 Studies that have not progressed through the PRG D79 GL 87-02 Responses Selected PI responses to GL 87-02. D80 USAR App I USAR HELB Evaluation D81 CR 19992212 Revised HELB temper atures for the 715', 735', and 755' Auxiliary Building D82 Outstanding OBDs and OBNs D83 P7550L-0822 P7550L-1001 Human Performance for Engineers Design & Licensing Basis
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 63 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases WS 1IA #1The evaluation only focused on SI pumps, remainder of CC system operability not addressed.(FOI A0108)D44No Operations involvement.* F2Relied on CC being a QA1 system. J3, J4*Addressed by current procedures.**Lack of timeliness addressed by WS2.
- Addressed b y RC1.CAsfor CC system not addressed until 1996 in preparation for upcoming NRC inspection.** (D41)
F3Organization was Engineering-led rather than Operations-led.The organization is not operationally focused.Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.HELB documentation lacked clarity, organization, and not easily retrievable. ***TB HELB analysis was incomplete so didn't realize HELB significance. ***(D41) F3Verification and Validation/Supervisory OversightVerification and Validation/Worker PracticesSupervisory Oversight
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 64 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 2IA #2Inadequate and untimely evaluation of operability and isolation capability in the CC system.Did not understand the full extent of the issue (D41) K6No Operations involvement* (D41)
F2Project study results addressed through WO & FOIS not CAP*(D41)Engineers had mindset that original plant construction must be correct since it had NRC approval, CC not impacted by design basis events, and USAR wording regarding CC capability to withstand piping breaks taken out of context to apply to events other than LOCA (D41, D49) J3Misinterpreted the USAR to believe a leak could be isolated within acceptable time to mitigate (D49) J3, J4TB HELB analysis was incomplete so didn't realize HELB significance. **(D41) F3Organization was Engineering-led rather than Operations-led.Believed loss of CC system due to pipe rupture addressed by AOP(D41) J3, J4*Addressed by current procedures.** Addressed by RC1.Resolution of CC issues was untimely. (D41)Resolution given lower priority due to more important issues. (D41)Believed CC seismic design made it adequate for other design basis events so CC issue is less important. (D41)Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.HELB documentation lacked clarity, organization, and not easily retrievable. **The organization is not operationally focused.Supervisory OversightVerification and Validation/Supervisory OversightVerification and Validation/Worker PracticesVerification and Validation/Worker PracticesProcedure QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 65 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 3IA #3Site evaluated SR equipment affected by HELBs(IN mentions HELBsin the Turbine Building). Missed CC pumps & system. EOC inadequate.OEER only evaluated redundant safety related equipment affected by HELB temperature and pressure. Pipe whip and jet impingement effects not addressed.Engineers had mindset that HELB only included temperature and pressure effects. J3Procedure 5AWI 3.7.0 did not specify requirements for extent of condition review.* (D61) RR1HELB documentation lacked clarity, organization, and not easily retrievable. **Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions. ***Addressed by PCRA #1.** Addressed by RC1.Verification and Validation/Worker Practices/Supervisory OversightProcedure QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 66 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 4IA #4Extent of Condition not consideredNo ACE / RCE was generated (D3, D41)Procedure didn't require an ACE / RCE at that time.* (D16) RR1Worker Practices/ Procedure/ Supervisory Oversight* Addressed by current procedures.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 67 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 5IA #5OPR too narrow. Only considered whether CC on SSEL rather than consequences of seismic failure. D8Did not consider all aspects of HELB, only looked at seismic issues. D8 J3, J4TB HELB analysis was incomplete so didn't realize HELB significance. **(D41) F3No Operations involvement* (D41)
F2Did not understand the full extent of the issue. K6Organization was Engineering-led rather than Operations-led.HELB documentation lacked clarity, organization, and not easily retrievable. **Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.The organization is not operationally focused.*Addressed by current procedures.** Addressed by RC1.Verification and Validation/Supervisory OversightVerification and Validation/Worker Practices
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 68 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 6IA #6The site did not adequately evaluate OE, especially EOC.Closeout review did not recognize omission (D45)OEER only considered AFW aspects (D45)OE Screen team only sent evaluation to AFW System Engineer (D45)OE Evaluator did not look at HELB aspect(D45)Focused on System Implications (D45)Procedure did not specify requirements for review (D61) RR1FP-PA-OE-01 does not specify an Extent of Condition Review (D61) RR1Review focused on procedural compliance rather than content (D45)Did not Consider Extent of Condition (D45) RR1Screen team assigns OE to one group per OE procedure.* (D41)Procedure requires consultation with other groups as required. * (D61)Assignment scope did not specify Extent of Condition Review (D45)CC #1Management did not develop adequate standards of OE evaluations with respect to extent of conditionWorker PracticesProcedureSupervisory Oversight/ProcedureProcedureSupervisory Oversight/Procedure/Job Planning and PreparationProcedure* Addressed by PCRA #1.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 69 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 7IA #11New AR not written for HELB CC issues that would have resulted in OPR.New CAP not written for HELB/Tornado (D3, D41)Still investigating the issue with HELB/CC (D41)Lack of Supervisor Oversight (D41)Some supervisors too involved with issue resolution (D41)All disciplines reporting to one supervisor for analysis activities (D41)Perception that CAP isn't written until problem is known (D41)No clear guidance or expectation for how long issues can be investigated before a CAP is written (D16, D41)RC1Inadequate management of HELB analysesLow resources require supervisors to become involved (D41)IA #7No newAR for HELB and tornado issues written.
WS 7HELB assumed to be addressed under AR 737382737382 Did not recognize new condition.Engineers had mindset that original plant construction must be correct since it had NRC approval, CC not impacted by design basis events, and USAR wording regarding CC capability to withstand piping breaks taken out of context to apply to events other than LOCA (D41, D49) J3Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.HELB documentation lacked clarity, organization, and not easily retrievable.TB HELB analysis was incomplete so didn't realize HELB significance.(D41) F3Inadequate funding for HELB analyses.CC #2Engineering Management has not developed expectations pertaining to CAP initiation for this aspectSupervisory Oversight/ProcedureSupervisory OversightVerification and Validation/Worker PracticesVerification and Validation/Worker PracticesProcedure QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 70 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 8IA #8Failure to recognize significance of HELB interaction of feedwaterand CC lines. D25Did not understand the full extent of the issue. K6Review of S&L study focuses only on HELB temperature and pressure effects.Engineers had mindsetthat original plant construction must be correct since it had NRC approval, CC not impacted by design basis events, and USAR wording regarding CC capability to withstand piping breaks taken out of context to apply to events other than LOCA (D41, D49) J3Engineers had mindset that HELB only included temperature and pressure.Lack of understanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.HELB documentation lacked clarity, organization, and not easily retrievable.TB HELB analysis was incomplete so didn't realize HELB significance.(D41) F3 RC1Inadequate management of HELB analysesVerification and Validation/Worker PracticesVerification and Validation/Worker Practices
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 71 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 9IA #9Site review was completed but no action taken in response to Q-List report for CCCAPsweren't written until there was an actual concern (D41)Project did not agree with some of the conclusions in the report and therefore considered the entire report questionable (D41)Some items included in the report were viewed as too conservative (D41)Q-List Project philosophy was to validate and then initiate CAPs(D41)Worker Practices/ Supervisory OversightSupervisory OversightSupervisory OversightNo clear guidance for writing ARsfor valid issues identified in draft or otherwise unaccepted Engineering studies.CC #2Engineering Management has not developed expectations pertaining to CAP initiation for this aspectProcedure QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 72 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 10IA #10Turbine Building HELB analyses still not complete, safety significance of delay not considered. D41Workload issues:High workload, transition to NMC and then Xcel, Passport rollout, other issues with higher priority, emergent regulatory issues, forced outage, inadequate turnover between engineers. (D41)Mindset issues:Engineers had mindset that original plant construction must be correct since it had NRC approval, CC not impacted by design basis events, and USAR wording regarding CC capability to withstand piping breaks taken out of context to apply to events other than LOCA (D41, D49) J3Supervisory issues:working supervisors, frequent management/supervisor changes, inadequate turnovers, supervisor positions vacant for extended periods, limited interaction between supervisor and reporting engineers. (D41)Funding issues:funding was limited, funding process is cumbersome and not well understood. (D41)Staffing issues:insufficient staffing, only one person assigned to HELB. (D41)Technical issues:Problems with content and quality of vendor studies. (D41)RC1Inadequate management of HELB analysesSupervisory OversightSupervisory OversightSupervisory OversightSupervisory OversightSupervisory OversightVerification and Validation/Worker PracticesFunding was accomplished using a department line budget that was outside the site review process.(D41)TB HELB analysis was incomplete so didn't realize HELB significance.(D41) F3 QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 73 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 11IA #12ECR 3183 issued using partially reviewed draft S&L study as basis.
D41No turnover or discussion of report with manager by SME.
F1CA requesting the ECR had an impending due date. A9ECR initiator was working in Outage Control Center during a forced outage. S3SME had left company.Human performance error.Turnover is addressed by CA 13 from RCE 01165133 Supervisory OversightSupervisory OversightWorker Practices/ Supervisory Oversight QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 74 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 6: Why Staircases (Cont'd)
WS 12IA #13Resolution of cold chemlab CC piping issue remains open.Not all information in study was reviewed. J3Owner's Acceptance of study not complete (D2) J4SWI ENG-26 not followed (D2)Procedure Compliance issue -Addressed by ACE 01162511-01 (D2)No turnover or discussion of report with manager. F1Reviewers had erroneous understanding of reasons for the study and significance of CC/HELB.Studies on cold chemlab CC piping issue still incomplete. No clear owner of the study. (D41)S&L final Study does not get owner's acceptance review per SWI-ENG-26.D12, D41WS 10Lack ofunderstanding of HELB licensing basis and difficulty in identifying and verifying inputs and assumptions.HELB documentation lacked clarity, organization, and not easily retrievable.TB HELB analysis was incomplete so didn't realize HELB significance.(D41) F3 RC1Inadequate management of HELB analyses.Supervisory OversightWorker Practices/ Supervisory OversightWorker Practices/ Supervisory Oversight/ Procedure QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 75 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H1a The site makes safety-significant or risk-significant decisions using a systematic process, especially when faced with uncertain or unexpected plant conditions, to ensure safety is maintained. This includes formally defining the authority and roles for decisions affecting nuclear safety, communicating these roles to applicable personnel, and implementing these roles and authorities as designed and obtaining interdisciplinary input and reviews on safety-significant or risk-significant decisions. Applicable . The root cause team did not find that this aspect was present. H1b The licensee uses conservative assumptions in decision making and adopts a requirement to demonstrate that the proposed action is safe in order to proceed rather than a requirement to demonstrate that it is unsafe in order to disapprove the action. The licensee conducts effectiveness reviews of safety-significant decisions to verify the validity of the underlying assumptions, identify possible unintended consequences, and determine how to improve future decisions. Applicable There were a number of assumptions made concerning TB HELB impacts on CC piping that were not verified. H1c The licensee communicates decisions and the basis for decisions to personnel who have a need to know the information in order to perform work safely and in a timely manner. Not Applicable The root cause team did not find that this aspect was present. H2a The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to ensure nuclear safety. Specifically, those necessary for: Maintaining long term plant safety by maintenance of design margins, minimization of long-standing equipment issues, minimizing preventative maintenance deferrals, and ensuring maintenance and engineering backlogs are low enough to support safety. Not Applicable The root cause team did not find that this aspect was present. H2b The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to ensure nuclear safety. Specifically, those necessary for: Sufficient qualified personnel are trained and available to maintain work hours within working hour's guidelines. Applicable Resources for effectively implementing required HELB activities were insufficient. One person was responsible for the HELB program only part time.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 76 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H2c The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to ensure nuclear safety. Specifically, those necessary for: Complete, accurate and up-to-date design documentation, procedures, and work packages, and correct labeling of components. Applicable There was a lack of design basis documentation with regard to HELB.
Design documentation was not up-to-date. OE procedures did not address extent of condition.
Corrective Action Process procedures did not provide expectations on CAP initiation for certain situations. H2d The licensee ensures that personnel, equipment, procedures, and other resources are available and adequate to ensure nuclear safety. Specifically, those necessary for: Adequate and available facilities and equipment, including physical improvements, simulator fidelity and emergency facilities and equipment. Not Applicable The root cause team did not find that this aspect was present. H3a The licensee appropriately plans work activities by incorporating: risk insights job site conditions, including environmental conditions, which may impact human performance; plant structures, systems, and components; human-system interface; or radiological safety The need for planned contingencies, compensatory actions, and abort criteria. Not Applicable The root cause team did not find that this aspect was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 77 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS H3b The licensee appropriately coordinates work activities by incorporating actions to address: The impact of changes to the work scope or activity on the plant and human performance. The impact of the work on different job activities, the need for work groups to maintain interfaces with offsite organizations, and communicate, coordinate, and cooperate with each other during activities in which interdepartmental coordination is necessary to assure plant and human performance. The need to keep personnel apprised of work status, the operational impact of work activities, and plant conditions that may affect work activities Long-term equipment reliability by limiting temporary modifications, operator work-arounds, safety systems unavailability, and reliance on manual actions.
Maintenance scheduling is more preventive than reactive. Not Applicable The root cause team did not find that this aspect was present. H4a The licensee communicates human error prevention techniques, such as holding pre-job briefings, self and peer checking, and proper documentation of activities. These techniques are used commensurate with the risk of the assigned task, such that work activities are performed safely. Personnel are fit for duty. In addition, personnel do not proceed in the face of uncertainty or unexpected circumstances. Applicable Pre-job brief was not performed prior to evaluation of OE by system engineer. H4b The licensee defines and effectively communicates expectations regarding procedural compliance. Personnel follow procedures. Not Applicable Owner's review of S&L study did not follow SWI-ENG-26. H4c The licensee ensures supervisory and management oversight of work activities, including contractors, such that nuclear safety is supported. Applicable Supervisory and management oversight of the CC/HELB analyses and studies was not effective in that these activities are still not complete. Delays in completion of these activities were not reviewed for their impact on a legacy issue and the risk of potential vulnerabilities remaining undiscovered.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 78 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS P1a The licensee implements a corrective action program with a low threshold for identifying issues. The licensee identifies such issues completely, accurately, and in a timely manner commensurate with their safety significance. Applicable CAPs were not generated for the CC/HELB interaction on discovery, precluding the CAP from potentially taking appropriate corrective action. P1b The licensee periodically trends and assesses information from the CAP and other assessments in the aggregate to identify programmatic and Issue common cause problems. The licensee communicates the results of the trending to applicable personnel. Not Applicable The root cause team did not find that this aspect was present. P1c The licensee thoroughly evaluates problems such that the resolutions address causes and extent of conditions, as necessary. This includes properly classifying, prioritizing, and evaluating for operability and reportability conditions adverse to quality. This also includes, for significant problems, conducting effectiveness reviews of corrective actions to ensure that the problems are resolved. Applicable CAPs that were written did not result in an extent of condition review. P1d The licensee takes appropriate corrective actions to address safety issues and adverse trends in a timely manner, commensurate with their safety significance and complexity.
Applicable Failure to review the impact of delays in completion of HELB activities resulted in corrective actions not being completed in a timely manner. P1e If an alternative process (i.e., a process for raising concerns that is an alternate to the licensee's corrective action program or line management) for raising safety concerns exists, then it results in appropriate and timely resolutions of identified problems. Not Applicable The root cause team did not find that this aspect was present. P2a The licensee systematically collects, evaluates, and communicates to affected internal stakeholders in a timely manner relevant internal and external OE. Applicable Evaluation of events at other plants and internal events that were related to this event did not consider extent of condition. P2b The licensee implements and institutionalizes OE through changes to station processes, procedures, equipment, and training programs. Applicable The root cause team did not find that this aspect was present QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 79 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS P3a The licensee conducts self-assessments at an appropriate frequency; such assessments are of sufficient depth, are comprehensive, are appropriately objective, and are self-critical. The licensee periodically assesses the effectiveness of oversight groups and programs such as CAP, and policies. Not Applicable The root cause team did not find that this aspect was present. P3b The licensee tracks and trends safety indicators which provide an accurate representation of performance. Not Applicable The root cause team did not find that this aspect was present. P3c The licensee coordinates and communicates results from assessments to affected personnel, and takes corrective actions to address issues commensurate with their significance. Not Applicable The root cause team did not find that this aspect was present. S1a Behaviors and interactions encourage free flow of information related to raising nuclear safety issues, differing professional opinions, and identifying issues in the CAP and through self assessments. Such behaviors include supervisors responding to employee safety concerns in an open, honest, and non-defensive manner and providing complete, accurate, and forthright information to oversight, audit, and regulatory organizations. Past behaviors, actions, or interactions that may reasonably discourage the raising of such issues are actively mitigated. As a result, personnel freely and openly communicate in a clear manner conditions or behaviors, such as fitness for duty issues, that may impact safety and personnel raise nuclear safety issues without fear of retaliation. Not Applicable The root cause team did not find that this aspect was present. S1b IF alternative processes (i.e., a process for raising concerns or resolving differing professional opinions that are alternates to the licensee's corrective action program or line management) for raising safety concerns or resolving differing professional opinions exists, THEN they are communicated, accessible, have an option to raise issues in confidence, and are independent, in the sense that the program does not report to line management (i.e., those who would in the normal course of activities be responsible for addressing the issue raised). Not Applicable The root cause team did not find that this aspect was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 80 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS S2a All personnel are effectively trained that harassment and retaliation for raising safety concerns is a violation of law and policy and will not be tolerated. Not Applicable The root cause team did not find that this aspect was present. S2b Claims of discrimination are investigated consistent with the content of the regulations regarding employee protection and any necessary corrective actions are taken in a timely manner, including actions to mitigate any potential chilling effect on others due to the personnel action under investigation. Not Applicable The root cause team did not find that this aspect was present. S2c The potential chilling effects of disciplinary actions and other potentially adverse personnel actions (e.g., reductions, outsourcing, and reorganizations) are considered and compensatory actions are taken when appropriate. Not Applicable The root cause team did not find that this aspect was present. OTH1a Accountability is maintained for important safety decisions in that the system of rewards and sanctions is aligned with nuclear safety policies and reinforces behaviors and outcomes which reflect safety as an overriding priority. Not Applicable The root cause team did not find that this aspect was present. OTH1b Management reinforces safety standards and displays behaviors that reflect safety as an overriding priority. Not Applicable The root cause team did not find that this aspect was present. OTH1c The workforce demonstrates a proper safety focus and reinforces safety principles among their peers. Not Applicable The root cause team did not find that this aspect was present. OTH2a The licensee provides adequate training and knowledge transfer to all personnel on site to ensure technical competency. Not Applicable The root cause team did not find that this aspect was present. OTH2b Personnel continuously strive to improve their knowledge, skills, and safety performance through activities such as benchmarking, being receptive to feedback, and setting performance goals. The licensee effectively communicates information learned from internal and external sources about industry and plant issues. Not Applicable The root cause team did not find that this aspect was present. OTH3 Management uses a systematic process for planning, coordinating, and evaluating the safety impacts of decisions related to major changes in organizational structures and functions, leadership, policies, programs, procedures, and resources. Management effectively communicates such changes to affected personnel. Not Applicable The root cause team did not find that this aspect was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 81 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 7 Safety Culture Analysis NRC ID DEFINITION APPLICABILITY BASIS OTH4a These policies require and reinforce that individuals have the right and responsibility to raise nuclear safety issues through available means, including avenues outside their organizational chain of command and to external agencies, and obtain feedback on the resolution of such issues. Not Applicable The root cause team did not find that this aspect was present. OTH4b Personnel are effectively trained on these policies. Not Applicable The root cause team did not find that this aspect was present. OTH4c Organizational decisions and actions at all levels of the organization are consistent with the policies. Production, cost and schedule goals are developed, communicated, and implemented in a manner that reinforces the importance of nuclear safety. Not Applicable The root cause team did not find that this aspect was present. OTH4d Senior managers and corporate personnel periodically communicate and reinforce nuclear safety such that personnel understand that safety is of the highest priority. Not Applicable The root cause team did not find that this aspect was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 82 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS HUMAN PERFORMANCE FAILURE MODES Inattention (A1) Type - SB Not paying attention to the task requirements. Not paying attention to information in the immediate environment. Not Applicable The root cause team did not find that this failure mode was present. Bored (A2) Type - SB Inadequate level of mental activity due to performance of repetitive actions or lack of activity. Not Applicable The root cause team did not find that this failure mode was present. Habit / Reflex (A3) Type - SB Ingrained or automated pattern of actions attributed to the repetitive nature of a well-practiced task or a natural response. Applicable When dealing with the CC Seismic issues, it appears there was a reflexive action to continue adding actions to the existing CAP instead of writing a new AR. Tired & Fatigued (A4) Type - SB/RB/KB Degradation of physical or mental abilities due to illness, a lack of rest, or influences associated with body rhythms. Not Applicable The root cause team did not find that this failure mode was present. Distracted & Interrupted (A5) Type - SB Conditions of task or the work environment require the individual to stop and restart a task, diverting the individual's attention from the task at hand. Applicable The individual assigned to the HELB program had many
other responsibilities that would have resulted in stopping and starting a task many times.
Multi Tasking (A6) Type - SB Performing two or more tasks simultaneously and neglecting to perform a required element of one or more of the tasks.
Applicable Multiple Interviews indicated that high work loads and multiple issues (specifically AFWP Bearing issues) were occurring at the same time. Lapse of Memory (A7) Type - SB Momentary loss of memory regarding information previously learned and known. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Tracking (Place Keeping) (A8) Type - SB/RB Method used to maintain control of information, necessary requirements, or status was not properly used. Not Applicable The root cause team did not find that this failure mode was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 83 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Time & Schedule Pressure (A9) Type - SB/RB/KB Urgency or excessive pace required to perform the task. No spare time allotted or perception by the individual that a tight schedule exists. Not Applicable The root cause team did not find that this failure mode was present. Fear of Failure (A10) Type - SB/RB/KB Apprehension regarding potential adverse consequences if the individual fails to perform at a high level, resulting in undesirable behaviors. Not Applicable The root cause team did not find that this failure mode was present. Imprecise Communication (A11) Type - SB/RB Miscommunication resulting from error of omission or commission by the sender or receiver. This includes breakdowns of the three-part communication process. Not Applicable The root cause team did not find that this failure mode was present. Cognitive Overload (J1) Type - RB/SB Mental demands on the individual to maintain a high level of concentration while requiring recall of excessive amounts of information. Not Applicable The root cause team did not find that this failure mode was present. Spatial Disorientation (J2) Type - SB/RB Loss or misjudgment of place or time; wrong component, wrong train and wrong unit errors due to similarities in the environment. Not Applicable The root cause team did not find that this failure mode was present. Mindset / Preconceived Idea (J3) Type - RB The tendency of an individual to make a judgment based upon a preconceived mental model or preconditioned bias that is not based upon the current information, conditions or indications. Applicable There was a mindset that the answer to an issue had to be
known prior to initiating a
CAP. Wrong Assumptions (J4) Type - RB Judgments are made without verification of the facts and are usually based upon the individual's perception of recent experiences or events. Applicable There was a preconceived idea that the CC System was not required for a HELB, without verifying the validity of the assumption. Inadequate Verification (J5) Type - RB Insufficient verification of the facts, and is usually based upon inaccurate information or the lack of information. Applicable There was a lack of design basis documentation with regard to HELB which prevented verification of the assumption that the CC System was not required for a HELB. Inadequate Motivation (J6) Type - SB/RB/KB Low morale or low interest in performing well. Not Applicable The root cause team did not find that this failure mode was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 84 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Shortcuts Taken (J7) Type - RB Actions to allow the job to go "easier" or faster, contrary to prescribed requirements. Not Applicable The root cause team did not find that this failure mode was present. Work Around (J8) Type - RB Compensatory or non-standard actions to meet a requirement are taken by the worker due to uncorrected material condition, programmatic deficiencies, or long-standing problems. Not Applicable The root cause team did not find that this failure mode was present. Over Confident (K1) Type - KB/RB/SB Underestimating the difficulty or complexity of the task. Self-satisfaction or confidence with a situation in which actual hazards or dangers exist, but the worker is not aware of them. Applicable Individuals did not write a second AR, in part (based on interviews), because there was a belief that there was adequate understanding of the issue. Unfamiliar or Infrequent Task (K2) Type - KB Tasks that have not been performed before or are performed infrequently. Not Applicable The root cause team did not find that this failure mode was present. Misdiagnosis (K3) Type - KB Decisions made with accurate information that is used or interpreted incorrectly when reaching a decision. Applicable There was an incorrect diagnosis of the impact of HELB on the CC System. Tunnel Vision (K4) Type - KB Decisions are made without considering all the available options or information needed to adequately assess the situation. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Knowledge of Fundamentals (K5) Type - KB Insufficient knowledge of fundamentals needed for task, such as heat transfer, fluid flow, structural analysis, etc. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Knowledge of
Standards (K6) Type - KB Insufficient knowledge of codes, standards, design basis, licensing basis, regulations, etc. needed to perform the task. Applicable The lack of well developed HELB design basis documentation meant that personnel did not adequately understand the design requirements of the CC system and how HELB would impact operation.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 85 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Flawed Analytical Process or Model (K7) Type - KB/RB Decisions based on a flawed analysis, such as using qualitative versus quantitative data, insufficient determination of problem/solution scope, improper computer modeling, or inadequate sample scope. Applicable A narrow focus when performing OE evaluations played a role in this event. ORGANIZATIONAL AND MANAGEMENT FAILURE MODES Inadequate Span of Control (S1) Horizontal organizational design - the number of personnel which a supervisor is responsible for is too large or too few for the groups oversight & responsibilities. This often creates problems with task assignment and accountability. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Levels in the Organization (S2) Vertical organizational design - the number of levels or layers, from senior manager to employee is too many or too few for the given activity. Creates problems with communication of expectations. Not Applicable The root cause team did not find that this failure mode was present. Insufficient Staffing (S3) Comprehensive organizational design - the total number of employees for which the company or group is designed are not filled. Often causes staff work overload and poor accountability. Applicable Interviews indicated that high work loads played a role in this event. Inadequate Communication within an Organization (F1) A breakdown in communication (written or verbal) within one organization or work group. Often leads to important issues not being addressed and critical process breakdown. Applicable Inadequate supervisory control in that standards and expectations were not clearly reinforced. Inadequate Communication among Organizations (F2) A breakdown in communication (written or verbal) among two or more organizations or work groups. Often leads to a breakdown in processes that require several groups to participate. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Prioritization (F3) Deficiencies in determining which work takes precedence over other work. Often leads to unexpected equipment failures or failure to meet regulatory requirements. Applicable The HELB program was not adequately prioritized. Inadequate Planning (F4) Deficiencies in determining what work must be done, by whom, when, and how long it will take. Often leads to staff work overload, budget over-runs and low morale. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Emerging Issues Management (F5) Deficiencies in determining how to deal effectively with unexpected issues. Often leads to continual "crisis management" and low morale. Not Applicable The root cause team did not find that this failure mode was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 86 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Inadequate Program Management (F6) Inadequate oversight of critical work processes to ensure they function smoothly and effectively. Often results in program degradation over time or increased problems within those processes. Applicable The CAP and OE programs were not effective in resolving the issue. Inadequate Trust (C1) A lack of confidence in the workgroup or members of the workgroup, or a disbelief in information shared. Often results in fractured work completion and stress levels. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Teamwork (C2) Constant friction among the wo rkforce, or an unwillingness to work with one another. This problem could exist within organizations or between organizations. Results in confusion within the ranks and a lack of information flow among the groups. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Knowledge (C3) An inadequate understanding of the work to be performed and how the work ties into the overall goals. Often causes individual errors to occur. Not Applicable The root cause team did not find that this failure mode was present. Lack of Commitment (C4) A lack of dedication to the work. Often results in inconsistent or unreliable performance by an individual or group. Not Applicable The root cause team did not find that this failure mode was present. Inadequate Self Assessment (C5) A failure to continually encourage feedback, listen to customer input, or look at better ways to perform. Often creates a false sense of security and leads to complacency. Not Applicable The root cause team did not find that this failure mode was present. PROCESS FAILURE MODES Actions Not Specified (RR1) The action(s) that an individual or group must perform to accomplish a task are not contained in the document or instruction. Applicable FP-PA-ARP-01, Revs 9-21 did not specify to create a new CAP for a new issue identified during evaluations of existing issues. FP-PA-OE-01 does not contain explicit guidance to include Extent of Condition in OE evaluations.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 87 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Actions Not Clear (RR2) The action(s) that an individual or group must perform to accomplish a task are not clearly described in the document or instruction. Applicable The OE and CAP procedures lacked guidance that could have helped in problem identification Actions not within Control of the Individual (RR3) The action(s) that an individual or group must perform to accomplish a task cannot be performed as specified (physical constraints, do not have authority to dictate results, etc.). Not Applicable The root cause team did not find that this failure mode was present. Actions Conflict with Another Process (RR4) The action(s) that an individual or group must perform to accomplish a task conflict or contradict the actions specified by another document or instruction. Not Applicable The root cause team did not find that this failure mode was present. Actions Not Tied to Another Process When Necessary (RR5) The action(s) contained within one document or instruction does not reference supporting documents or instructions when necessary. Not Applicable The root cause team did not find that this failure mode was present. Methods Not Clearly Defined (RR6) Action(s) are required by the document or instruction, but the method to accomplish the actions is not clearly specified by the document or instruction. Not Applicable The root cause team did not find that this failure mode was present. Unnecessary Actions Required (RR7) The document or instruction require the performance of certain actions that is not really necessary to successfully perform the action. Not Applicable The root cause team did not find that this failure mode was present. Wrong Information (RR8) The information provided in the document or instruction is incorrect.
Not Applicable The root cause team did not find that this failure mode was present. Critical Actions Not Verified (AR1) Critical actions required to successfully perform a task are not verified within the process. Not Applicable The root cause team did not find that this failure mode was present. Excessive Verifications (AR2) The document or instruction requires excessive verification of completed steps or tasks. Actions are verified, regardless of criticality to the task or the task has multiple reviews and verifications instead of a single, specific review. Not Applicable The root cause team did not find that this failure mode was present. No Process Monitoring (AR3) There is no established means of monitoring the success or failure of the process. Not Applicable The root cause team did not find that this failure mode was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 88 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 8 FAILURE MODE ANALYSIS FAILURE MODE DEFINITION APPLICABILITY BASIS Only Monitoring Problems (AR4) The only method of monitoring process performance is to observe problems when they occur. Not Applicable The root cause team did not find that this failure mode was present. No Acceptance Criteria (AR5) No acceptable performance parameters have been established for the process, procedure or task. Not Applicable The root cause team did not find that this failure mode was present. No One Specified to Perform
Task (I1) No one is specified (either by title, group, or other means) as responsible for completion of the actions required by a document or instruction. Not Applicable The root cause team did not find that this failure mode was present. More Than One Person Specified to Perform Task (I2) More than one person or group is specified (either by title, group, or other means) as responsible for completion of the actions required by a document or instruction. Not Applicable The root cause team did not find that this failure mode was present. Person Specified Not Able to Perform Task (I3) The person or group specified (either by title, group, or other means) as responsible for the completion of the required actions in a document or instruction is unable to perform the action. Typically because they do not have the skill or knowledge. Not Applicable The root cause team did not find that this failure mode was present.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 89 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 9 BARRIER ANALYSIS HAZARD BARRIER ASSESSMENT TARGET Procedure (FP-OE-OL-01)
- No clear guidance on performing extent of condition review.
- Requirements for supervisory review focused on administration rather than content of the evaluation.
- No requirements for a pre-job brief. Engineer's work practice
- Performed a very narrow evaluation.
- Did not evaluate all the recommendations in the OE report. Supervisory Oversight
- Did not review OE evaluation to ensure high quality product. Inadequate evaluation of
OE Job Planning and Preparation
- No pre-job brief Failed to ensure that the safety related function of the CCW system was maintained. Verification and Validation
- Took information from USAR out of context.
- Assumed original plant construction must be correct - did not question or validate this assumption
- Did not involve other departments
- Thought check valves in the CCW system or operator action would prevent the system from draining. Procedure
- No clear CAP process when report first received Inadequate
investigation of study findings regarding CC Supervisory Oversight
- Supervisors did not insist study findings be discussed outside of the engineering department Failed to ensure that the safety related function of the CCW system was maintained.
Procedure (FP-PA-ARP-01)
- Did not require extent of condition for condition evaluations Worker Practices
- It is good engineering practice to perform extent of condition assessments Extent of
Condition Assessment Supervisory Oversight
- Supervisor did not ensure extent of condition was performed. Failed to ensure that the safety related function of the CCW system was maintained. Worker Practices
- Did not feel another CAP was required
- Excessive investigation time
- Overconfidence Verification and Validation
- Single point of expertise limited chances to challenge/ verify expert's opinion.
- Took information from USAR out of context.
- Did not involve other departments Supervisory Oversight
- Supervisor did not require an additional CAP to be written.
- Procedure did not provide clear guidance for how long a potential problem should be investigated before writing a CAP Operability of CCW was not considered for HELB/Tornado in Turb. Bldg.
Procedure
- The Development of Engineering Studies procedure defines scope of review. This includes verifying clear definition of the problem with specific supporting information. Failed to ensure that the safety related function of the CCW system was maintained.
QF-0433, Rev 3, (FG-PA-RCE-01) RCE Report Template Page 90 of 90 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
Attachment 10 Change Analysis Potential HELB interactions were known in 2005 but no immediate actions were taken. In 2008 the problem was rediscovered and appropriate actions were taken.
2005 Instance 2008 Instance Change / Difference Impact / Assessment Seen as a long term problem
- Wrote EAR
- Attempted to get study funded - Study done over the following year Recognized immediate problem
- Wrote CAP
- Initiated new OPR
- Isolated CC to the Cold Chem. Lab The issue was dealt with on an operability level for the 2008 instance. The 2005 instance was more interested in closure of the issue When operability was scrutinized per design requirements, the issue was finally known fully. Experienced Engineer
- Knew past justifications
- Had preconceived ideas
- Would rather resolve issue without a CAP
- Relied on "Tribal Knowledge" New Engineer (<2 years)
- Had nothing written down. Had to search for justifying documentation
- Not as much historical knowledge. Less preconceived ideas.
- Trained in higher level expectations for writing CAPs
- Had no "Tribal Knowledge" - Had to research all information The experienced engineer used past experience to guide decisions in place of rigorous investigative work. The new engineer had no previous experience to rely on. This required him to fully investigate the issue. The new engineer had to rely on procedures and retrievable documentation. The more detailed investigation during the 2008 instance led to a fuller understanding of the design requirements. Understanding of the design requirements led to the appropriate operability assessment. Discipline - Civil
- No program to explain the requirements.
- Focused on Seismic
- Responsible for Seismic, HELB, and tornado issues Discipline - Mechanical
- No program to explain the requirements.
- Was aware of need to investigate HELB - Responsible for HELB but not seismic or tornado. The engineers involved in each instance were of a different engineering discipline and had different responsibilities. There is some evidence that the 2005 instance was more focused on the Seismic aspect. Having responsibility for all three (HELB, Seismic, and Tornado) areas could result in a more limited focus on the requirements for each area and less of an ability to recognize when new issues arise.