05000528/LER-2004-009
| Docket Numbersequential Revday Year Year Number No. Month Day Year Pvngs Unit 2 05000529Month | |
| Event date: | 07-30-2004 |
|---|---|
| Report date: | 5-1-1200 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat |
| Initial Reporting | |
| ENS 40913 | 10 CFR 50.72(b)(3)(v)(B), Loss of Safety Function - Remove Residual Heat, 10 CFR 50.72(b)(3)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident |
| 5282004009R01 - NRC Website | |
NOTE: THIS SUPPLEMENT IS A SUBSTANTIAL REWRITE OF THE ORIGINAL LER AND THEREFORE NO REVISION BARS HAVE BEEN INCLUDED.
1. REPORTING REQUIREMENT(S):
This event is being reported pursuant to 10 CFR 50.73(a)(2)(v)(B) and (D) due to an air void in Emergency Core Cooling System suction piping from the containment sump that may have prevented the fulfillment of the system safety function to remove residual heat and mitigate the consequences of a Loss of Coolant Accident (LOCA). This condition existed in Palo Verde Nuclear Generating Station (PVNGS) Units 1, 2, and 3. Initial event notification was made to the NRC headquarters operation officer on July 31, 2004 (reference ENS # 40913).
2. DESCRIPTION OF EVENT RELATED STRUCTURE(S), SYSTEM(S) AND
COMPONENT(S):
Emergency Core Cooling System (ECCS) [EllS: BQj The emergency core cooling system (ECCS) is comprised of three functional systems:
emergency core cooling (safety injection (SI)), residual heat removal (shutdown cooling, (SDC)) and containment heat removal (containment spray (CS)). During an event requiring ECCS actuation, a flow path is established to supply borated water from the refueling water tank (RWT) to the reactor coolant system (RCS) via the SI pumps and their respective supply headers.
The RWT supplies two ECCS trains by separate, redundant supply headers. Each header also supplies one train of the Containment Spray System. A motor operated isolation valve is provided in each header to allow the operator to isolate the RWT from the ECCS after the pump suction has been automatically transferred to the containment sump following depletion of the RWT during a loss of coolant accident (LOCA). The signal that places the sumps in service is a recirculation actuation signal (RAS) and is generated by a low level in the RWT. The RAS, in addition to opening the sump valves, also trips the Low Pressure Safety Injection pumps to ensure adequate net positive suction head (NPSH) to the High Pressure Safety Injection (HPSI) and CS pumps.
3. INITIAL PLANT CONDITIONS:
On July 30, 2004, at approximately 0700 Mountain Standard Time (MST), PVNGS Units 1, 2 and 3 were operating at approximately 100 percent power, when engineering personnel [utility, non-licensed] informed control room personnel [licensed] that a concern had developed regarding a voided section of ECCS suction piping and that the condition was being evaluated for its impact on the fulfillment of the safety function to remove residual heat and mitigate the consequences of a LOCA.
EVENT DESCRIPTION:
The design basis for the PVNGS is specified, in part, in the plant Updated Final Safety Analysis Report (UFSAR). Section 6.3 of the UFSAR, "Emergency Core Cooling System," states, in part, that the safety injection piping will be maintained filled with water, and that during recirculation mode, the available net positive suction head for the containment spray and high pressure safety injection pumps is 25.8 feet and 28.8 feet, respectively (values that assume the pump suction piping is filled with water.) Contrary to the above, from initial plant licensing until July 2004, the design control measures established by the licensee were not adequate to assure that the design basis for the PVNGS emergency core cooling system (ECCS) was appropriately translated into specifications, procedures, and instructions. The licensee had no specifications, procedures or instructions in place to assure that this aspect of the design basis for the ECCS system was maintained. Specifically, except for limited periods of time following ECCS leak testing prior to 1992, PVNGS failed to maintain portions of the containment sump safety injection recirculation piping filled with water in accordance with the UFSAR, a nonconformance that affected the ability of the containment spray and high pressure safety injection pumps to deliver the required head and flow as described in the UFSAR. This condition existed at Units 1, 2 and 3 of the PVNGS facility from initial plant operation (1985, 1986 and 1987, respectively) until August 2004, at which time corrective actions were taken to fill the affected piping.
5. ASSESSMENT OF SAFETY CONSEQUENCES:
PVNGS conducted tests to determine whether the ECCS pumps would perform their safety function given the unfilled suction piping. Based upon this testing, PVNGS was unable to determine that the high pressure safety injection (HPSI) pumps would deliver the required head and flow during a small break LOCA (SBLOCA).
Additionally, the Nuclear Regulatory Commission (NRC) concluded that the HPSI pumps might be unable to perform their function for some portion of the medium break LOCAs (MBLOCAs) For the time frame that the three units operated with the voided piping, none of the units experienced an event that called for the Emergency Core Cooling System(s) (ECCS) to perform their safety function(s) that required the initiation of a RAS. This condition did not result in any challenges to the fission product bafflers or result in any offsite releases. Using its Significance Determination Process, the NRC assessed a 'Yellow' finding for Palo Verde's failure to maintain portions of the ECCS filled with water in accordance with design control requirements. The NRC issues 'Yellow' findings for performance deficiencies that the NRC evaluates as having a substantial importance to safety or "a significant reduction in safety margin.
6. CAUSE OF THE EVENT:
PVNGS conducted an investigation for the condition in Units 1, 2, and 3 where voids in Emergency Core Cooling System containment sump piping may have prevented the fulfillment of the system safety function to remove residual heat and mitigate the consequences of a Loss of Coolant Accident and has performed an extensive investigation into the causes. The investigation evaluated the Initial failure" to fill the pipe during original construction and startup, as well, as the "missed opportunities" since that time to have identified and corrected the condition. In addition, comprehensive extent of condition and extent of cause evaluations have been completed. Summarized below are the direct cause, nine root causes, and nine contributing causes for the violation.
Direct Cause Direct Cause — Procedures Did Not Contain Necessary Requirements. The design intent that the suction line be filled with water was not translated into start-up, surveillance, and operating procedures.
Root Causes for the Initial Failure Root Cause No. 1 (RC-1) — Lack of Specific Provisions in the Design and Licensing Basis The design and licensing basis documents did not contain explicit statements requiring the emergency core cooling system (ECCS) suction lines to be filled. The reason for not explicitly stating these requirements was not positively ascertained. The following root and contributing causes provide amplifying causal information.
Root Cause No. 2 (RC-2) — Ineffective Questioning Attitude and Technical Rigor of Individuals Some PVNGS personnel had a narrow focus and an incorrect mindset (i.e., incorrect belief in a self-venting theory) in reviewing information provided in various design documents that indicated the need to keep the ECCS suction lines filled. There was a general ineffective use of a QV&V process. (QV&V ["Qualify, Validate, and Verify"] is a three-step tool used to obtain accurate information during critical decision-making.) Root Cause No. 3 (RC-3) — Inadequate Communication of Design Information The need to keep the ECCS suction lines filled was identified but not appropriately communicated. Follow-through for ensuring start-up procedures contained provisions for filling and venting the system was inadequate.
Root Causes for the Missed Opportunities* * Focus on missed opportunities for identifying and correcting the unanalyzed condition, after the initial design configuration error.
Root Cause No. 4 (RC-4) — Lack of Specific Provisions in the Design Basis Personnel missed opportunities to identify the unanalyzed condition involving the unfilled suction lines because the design basis documents did not contain an explicit statement that required the lines to be filled.
Root Cause No. 5 (RC-5) — Ineffective Questioning Attitude and Technical Rigor of Individuals Some PVNGS personnel had a narrow focus and an incorrect mindset (i.e., incorrect belief in a self-venting theory) in reviewing various documents and information related to the ECCS suction lines. There was general ineffective use of a QV&V process. (QV&V ["Qualify, Validate, and Verify"] is a three-step tool used to obtain accurate information during critical decision-making.) Root Cause No. 6 (RC-6) — Inadequate Communication of Design Information The need to keep the ECCS suction lines filled was identified but not appropriately communicated.
Root Cause No. 7 (RC-7) — Inadequate Problem Identification and Resolution Issues related to the acceptability of the unfilled ECCS suction lines were not documented on a Condition Report/Disposition Request (CRDR) due to unclear procedural guidance.
Root Cause No. 8 (RC-8) — Less than Adequate Technical Reviews As a result of inadequate technical reviews, PVNGS personnel overlooked information regarding the need,to fill the ECCS suction lines or did not review identified issues that could have led to identification of the unanalyzed condition involving the suction lines.
Root Cause No 9 (RC-9) — Limited Operating Experience Program The PVNGS Operating Experience Program did not require reviews of some types of operating experience reports related to the ECCS suction lines.
Contributing Causes for the Initial Failure Contributing Cause No. 1 (CC-1) — Inappropriate reliance on standard Combustion Engineering design The design of the ECCS suction lines at PVNGS was different than the design at other CE plants, but the PVNGS design did not account for the significance of those differences.
Contributing Cause No. 2 (CC-2) — Limited experience and training PVNGS personnel with responsibility for start-up did not have adequate system design or licensing basis training or experience to be able to detect the need for filling of the suction lines.
Contributing Cause No. 3 (CC-3) — Allocation of Resources During start-up, the Safety Injection engineers were under a high workload and had multiple tasks to perform, which deterred them from raising questions on issues not directly related to resolving the specific issues assigned to them.
Contributing Causes for the Missed Opportunities* * Focus on missed opportunities for identifying and correcting the unanalyzed condition, after the initial design configuration error.
Contributing Cause No. 4 (CC-4) — Weak Operating Experience Program The Operating Experience Program had little guidance applicable to the review of the Industry Operating Experience Reports related to the ECCS suction lines and gave low priority to the reviews, resulting in a narrow focus to the reviews and a lack of review by the Nuclear Assurance Department.
Contributing Cause No. 5 (CC-5) — Limited experience and training PVNGS personnel with responsibility for the Safety Injection System had limited training and experience to be able to detect the need for filling of the suction lines.
Contributing Cause No. 6 (CC-6) — Limited Resources System engineers had been under a high workload and had multiple tasks to perform, which deterred them from raising questions on issues not directly related to resolving the specific issues assigned to them. Reviews of 10E reports were generally narrowly focused and limited to addressing the specific issue raised in the report.
Contributing Cause No. 7 (CC-7) — Limited Verification and Validation By design, the "100% validation" of the Design Bases Manuals (DBM) was comprehensive and focused on validation of the information in the DBMs but was not 100%.
Contributing Cause No. 8 (CC-8) — Limited Procedural Guidance The DBM Writer's Guide (Procedure 83DP-4CCO2) lacked detailed guidance on how to review source documents during preparation of the DBMs (e.g., there was no requirement to review the entire source document).
Contributing Cause No. 9 (CC-9) — Limited Nuclear Assurance Department (NAD) Oversight NAD has not had a systematic approach for assessing safety significant or high risk technical specification or design configuration issues.
A collective evaluation was performed of the root and contributing causes, the organizational and programmatic (O&P) issues identified by the Event and Causal Factor Analysis and the Barrier Analysis, and PVNGS operating experience. This collective evaluation identified the following organizational weaknesses (OW). These weaknesses are viewed as deficiencies provoking conditions or degrading defenses associated with the unanalyzed condition involving the unfilled suction piping and the missed opportunities to identify the condition. 5 f,
- There has been evidence of insufficient or ineffective questioning attitude and technical rigor in Engineering and Operations at various times, particularly when analyzing design and licensing bases and configuration management issues.
- Problem identification and resolution has not always been fully effective.
- Safety Injection System Engineering resources were limited, which presented challenges in effectively attending to routine and emergent issues.
- There were limited system specific training, training materials, and formal system turnover requirements in Engineering.
- The Operating Experience Program did not receive effective support from the site organization.
- The Independent Safety Review (ISR) process did not conduct performance-based assessments to improve plant safety.
Inadequate control of the design and licensing bases was also evaluated for possible classification as an organizational weakness. However, the condition involving the unfilled suction piping appears to be isolated, and reviews of the UFSAR, Design Basis Manual, Licensee Event Reports (LERs), and Engineering Evaluation Requests (EERs), did not identify any generic concern with respect to the design and licensing basis requirements.
Since the condition was isolated, inadequate control of the design and licensing bases was not classified as an organizational weakness.
7. EXTENT OF CONDITION
The direct extent of condition was addressed by reviewing other sections of safety related piping to ensure they were filled as needed and tested by providing flow through the pipe as part of a routine test or operational evolution. The review identified no additional sections of piping that were maintained in an unfilled condition except where there was a clear design requirement that they be unfilled (e.g., Containment Spray discharge spray headers). - 1 In order to determine if other design requirements may have been missed in station procedures, a sample review of the UFSAR of selected plant systems was completed. Any design requirements that were not clearly maintained in station procedures have been entered into the PVNGS corrective action program.
PVNGS Licensee Event Reports (LER) were reviewed to identify those associated with design control. This subset of LERs was reviewed to determine if there was a generic weakness in transmitting design requirements into the station's design/operating basis. No weakness was identified. However, there was indication that individual knowledge of system, structures and component design requirements was weak. Corrective actions were initiated.
Based upon the results of the review of the extent of condition, it is concluded that the missed design requirement regarding the operating configuration of the ECCS suction line was isolated, and that there is reasonable assurance that similar safety significant configuration conditions do not exist in other fluid, I&C, or electrical systems.
8. EXTENT OF CAUSE
Issues involving problem identification and resolution and human performance (which encompass questioning attitude and technical rigor) have previously been identified as NRC cross-cutting issues at PVNGS. PVNGS has initiated separate CRDRs to assess these areas (including extent of cause evaluations) and to take corrective action.
The other root causes and organizational weaknesses pertain (either directly or indirectly) to the control of the design basis (e.g., limited training contributed to the failure of personnel to recognize that the design intent for filled suction lines was not satisfied). To determine whether these root causes and organizational weaknesses could have broader implications related to the:control of the design basis, a review was conducted of PVNGS licensee event reports (LERs) since the beginning of operation to determine whether significant conditions have been identified due to a failure to translate the design basis into requirements. Based upon this review, a concern was identified with respect to the knowledge of engineers. This is similar to one of the root causes and organizational weaknesses identified by the Event & Casual Factor analysis.
In addition, three other reviews were conducted. First, a review was performed of the eight Independent Design Review (IDR) reports for Containment Systems, Auxiliary Feedwater System, Alternating Current System, Auxiliary Systems, Fire Protection System, Environmental Qualification, Control Systems and Balance of Plant (BOP) I&C Systems, and Direct Current Power Systems to determine if the design requirements identified in these reports have been incorporated into design documents. Second, a review was performed of five systems (plus portions of two other systems) to verify that the Design Basis Manuals incorporate the design requirements in the UFSAR and that the plant has not been inappropriately altered by other maintenance, test, or modifications. This review included walkdowns of the Safety Injection (SI) and Auxiliary Feedwater (AF) Systems.
Third, a 95/95 probabilistic sample of all Engineering Evaluation Requests (EER) for the SI, AF and Diesel Generator (DG) systems was reviewed to determine if plant changes were made outside formal design change processes.
The results of the extent of cause reviews for the design basis issues did not identify any safety significant conditions; i.e., the reviews did not identify any missed design or licensing bases that would impact any system's or component's ability to perform their safety functions. These results indicate that while there were some organizational weaknesses at PVNGS, there is reasonable assurance that they did not have any safety significant effects except with respect to the unfilled ECCS suction line.
Finally PVNGS performed an assessment of the technical adequacy of a sample of high- tiered Industry. Operating Experience (10E) evaluations since circa 1980 (i.e., Significant Operating Experience Reports [SOERs], Significant Event Reports [SERs], Significant Event Notifications (SENsl, NRC Information Notices [°Ns], NRC Generic Letters [GL], NRC Bulletins, and Operations and Maintenance Reminders [O&MRs]). This assessment determined that while a number of 10E reviews were weak in the area of either technical rigor or insufficient scope, the team identified no instance where a weak 10E review would have left a latent design weakness in the plant that would adversely impact the , performance of a safety function:
The investigation concluded that the safety significant deficiency involving the ECCS suction line is also partially attributable to the limited pre-operational testing of the as-built system, the unique design of the suction lines at PVNGS, and the relative lack of functional use of the system (e.g., the suction piping is not used to flow water during routine operations). Unlike PVNGS, most of the other CE plants had suction lines inherently self- venting, which apparently led some personnel to believe that at PVNGS the suction lines were also self-venting. Additionally, PVNGS used standard CE products (e.g., CESSAR, generic start-up and operating procedures, etc.) and material from other CE plants (which appear to be predicated upon a self-venting suction line design), to develop the design basis and procedures at PVNGS which further contributed to the error at PVNGS. Some personnel did not appreciate the significance of the difference in design and did not exhibit the proper questioning attitude or technical rigor and follow-through.
9. CORRECTIVE ACTIONS:
The ECCS Sump suction lines and sumps in Units 1, 2 and 3 were filled by August 4, 2004.
Since then, modifications have been installed in all three units to maintain the ECCS suction piping filled and the ECCS sump dry. The modifications added additional vent, drain and fill connections on the SI piping to facilitate filling and maintaining the lines in a filled condition.
At Procedure 40ST-9S104, "Containment Spray Valve Verification," has been updated to vent the sump suction lines every 31 days. This surveillance test verifies that the piping is maintained filled.
Procedures were revised to require the ECCS suction lines to be filled with borated water prior to returning the system to a mode where the ECCS is required to be OPERABLE.
This places the system in the required condition at the completion of testing and maintenance.
The SI Design Basis Manual (DBM), the UFSAR, and the Technical Requirements Manual (TRM) have been revised to reflect the design requirement for this piping to be maintained full. This anchors the design requirement that was not'translated into the ECCS operating and/or testing procedures.
As an initial step to bring the site's attention to the root cause of ineffective questioning attitude and technical rigor, Senior Management has communicated to all hands that it is essential that all employees have a strong and effective questioning attitude and demonstrate technical rigor and to challenge assumptions and/or any situations which do not appear to be safe, per design, per procedure, per expectation, or in general do not seem appropriate. The communication re-emphasized the use of the QV&V (Qualify, Validate, & Verify) prevent event tool with a short primer on what it is and how to use it.
A Senior Management Sponsor has been designated who is responsible for a site-wide improvement in the culture relating to questioning attitude and technical rigor. The Senior Management Sponsor has developed a plan to improve and anchor the organizational culture with respect to effective questioning attitude and technical rigor.
Checklists for key technical products (High-tiered 10E evaluations [SOER, SEN, SER, IN, O&MR, Topical reports, etc.], Self-assessments, Audits, Significant CRDR evaluations) have been developed for use by both the evaluator and supervisory review to include guidance for performing more expansive OE reviews so that personnel do not focus only on the particular conditions identified in the OE report.
A RAS event case study has been developed and has been presented in a training setting to non-admin PVNGS personnel in Operations, Engineering, Nuclear Fuels Management, Regulatory Affairs and Nuclear Assurance. A few make-up sessions remain to be completed.
"Questioning attitude and technical rigor" tools and processes have been provided to the Palo Verde engineering organization. These tools have been designed to drive situations into "rule-based" processes instead of "knowledge-based" processes. A program to monitor the use of these tools has been implemented to reinforce a questioning attitude and appropriate technical rigor.
In parallel with the evaluation of the voided sump suction piping, evaluations of the substantive cross-cutting issues in problem identification and resolution and human performance were separately performed: The results of these evaluations along with other internal and external assessments were then used as inputs to the performance of an integrated organizational effectiveness evaluation. -The results of the integrated evaluation are being implemented in the Palo Verde'Performance Improvement Plan (PIP) which has been separately reviewed by and discussed with the NRC staff.
10. PREVIOUS SIMILAR EVENTS:
There have been no previous similar events in the past three years that had a similar failure mechanism or that should have been prevented by previously implemented corrective actions.