ML061300182

Revision to Reply to Notice of Violation; EA-04-221

ML061300182
Person / Time
Site:	Palo Verde
Issue date:	04/27/2006
From:	James M. Levine Arizona Public Service Co
To:	Document Control Desk, NRC/OE
References
102-05476-JML/CKS/RJR, EA-04-221, IR-04-014
Download: ML061300182 (15)

Text

10 CFR 2.201 NOV EA-04-221 ASlA~~

A subsidiary ofPinnacle West Capital Corporation Palo Verde Nuclear Generating Station James M. Levine Executive Vice President Generation Tel (623) 393-5300 Fax (623) 393-6077 Mail Station 7602 PO Box 52034 Phoenix, Arizona 85072-2034 102-05476-JMUCKSIRJR April 27, 2006 U.S. Nuclear Regulatory Commission ATTN: Document Control Desk Washington, DC 20555-0001

Reference:

1. Letter from B. S. Mallett (NRC) to G. R. Overbeck (APS) dated April 8, 2005,

Subject:

Final Significance Determination for a Yellow Finding and Notice of Violation - NRC Special Inspection Report 2004-014 -

Palo Verde Nuclear Generating Station

2. APS letter 102-05290-GRO/RAS, "Reply to a Notice of Violation; EA-04-221," dated June 7, 2005

3. Letter from T. W. Pruett (NRC) to G. R. Overbeck (APS) dated June 16, 2005,

Subject:

Palo Verde Nuclear Generating Station - Reply to Notice of Violation (NRC Inspection report 05000528/2004014, 05000529/2004014, 05000530/2004014)

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Units 1, 2 and 3 Docket Nos. STN 50-528, 50-529 and 50-530 Revision to the Reply to Notice of Violation; EA-04-221 In Reference 1, the NRC documented and issued Notice of Violation (NOV) EA-04-221.

In Reference 2, APS provided the response to the NOV. APS' original response identified that the root cause investigation for this violation was ongoing.

Subsequently, in Reference 3, the NRC requested that APS provide an additional written response to the NOV when the root cause evaluation was completed describing any additional reasons for the violation and/or corrective steps that will be taken to avoid further violations.

The root cause investigation has been completed. As requested by Reference 3 and pursuant to the requirements of 10 CFR 2.201 and the original Notice of Violation, EA-04-221, APS is submitting a revision to its original reply. Enclosure 1 to this letter contains a restatement of the violation. The revised response to NOV EA-04-221 is provided in Enclosure 2 and reflects the conclusions of the completed investigation.

A member of the STARS (Strategic Teaming and Resource Sharing) Alliance Callaway

• Comanche Peak

• Diablo Canyon

• Palo Verde 0 South Texas Project

• Wolf Creek SIE L1

U.S. Nuclear Regulatory Commission Document Control Desk Revision to the Reply to Notice of Violation; EA-04-221 Page 2 No commitments are being made to the NRC by this letter. This letter supersedes the original response (Reference 2) in its entirety. Should you have questions regarding this submittal, please contact Mr. Scott A. Bauer at (623) 393-5978.

Sincerely, JML/CKS/RJR/ca

Enclosures:

1. Restatement of Violation, EA-04-221

2. Revision to the Reply to NOV EA-04-221 cc:

T. W. Pruett B. S. Mallett M. B. Fields G. G. Warnick Chief Project Branch D, Division of Reactor Projects, USNRC, Region IV Administrator, Region IV, USNRC Project Manager, Nuclear Reactor Regulation, USNRC Senior Resident Inspector, PVNGS, USNRC Restatement of Violation, EA-04-221 During an NRC inspection completed December 8, 2004, a violation of NRC requirements was identified. In accordance with the "General Statement of Policy and Procedure for NRC Enforcement Actions," NUREG-1 600, the violation is listed below:

10 CFR Part 50, Appendix B, Criterion l1l, Design Control states, in part, that measures shall be established to assure that the design basis is correctly translated into specifications, procedures, and instructions. The design basis for the Palo Verde Nuclear Generating Station (PVNGS) is specified, in part, in the plant Updated Final Safety Analysis Report (UFSAR). Section 6.3 of the UFSAR, "Emergency Core Cooling System," states, in part, that the safety injection piping will be maintained filled with water, and that during recirculation mode, the available net positive suction head for the containment spray and high pressure safety injection pumps is 25.8 feet and 28.8 feet, respectively (values that assume the pump suction piping is filled with water.)

Contrary to the above, from initial plant licensing until July 2004, the design control measures established by the licensee were not adequate to assure that the design basis for the PVNGS emergency core cooling system (ECCS) was appropriately translated into specifications, procedures, and instructions. The licensee had no specifications, procedures or instructions in place to assure that the design basis for the ECCS system was maintained. Specifically, except for limited periods of time following ECCS leak testing prior to 1992, the licensee failed to maintain portions of the containment sump safety injection recirculation piping filled with water in accordance with the UFSAR, a nonconformance that affected the available net positive suction head for the containment spray and high pressure safety injection pumps as described in the UFSAR. This condition existed at Units 1, 2 and 3 of the PVNGS facility from initial plant operation (1985, 1986 and 1987, respectively) until August 2004, at which time corrective actions were taken to fill the affected piping.

This violation is associated with a Yellow SDP finding.

Page 1 Revised Reply to Notice of Violation EA-04-221

1.

Reason for the Violation APS admits to the violation and has performed an extensive investigation into the causes. This investigation evaluated the "initial failure" to fill the pipe during original construction and startup, as well, as the "missed opportunities" since that time to have identified and corrected the condition. In addition, comprehensive extent of condition and extent of cause evaluations have been completed. Summarized below are the direct cause, nine root causes, and nine contributing causes for the violation.

Direct Cause Direct Cause (DC-1) - Procedures Did Not Contain Necessary Requirements. The design intent that the suction line be filled with water was not translated into start-up, surveillance, and operating procedures.

Root Causes for the Initial Failure Root Cause No. I (RC-1) - Lack of Specific Provisions in the Design and Licensing Basis The design and licensing basis documents did not contain explicit statements requiring the emergency core cooling system (ECCS) suction lines to be filled. The reason for not explicitly stating these requirements was not positively ascertained. The following root and contributing causes provide amplifying causal information.

Root Cause No. 2 (RC-2) - Ineffective Questioning Attitude and Technical Rigor of Individuals Some PVNGS personnel had a narrow focus and an incorrect mindset (i.e., incorrect belief in a self-venting theory) in reviewing information provided in various design Page 1 Revised Reply to Notice of Violation EA-04-221 documents that indicated the need to keep the ECCS suction lines filled. There was a general ineffective use of a QV&V process. (QV&V ("Qualify, Validate, and Verify"] is a three-step tool used to obtain accurate information during critical decision-making.)

Root Cause No. 3 (RC-3) - Inadequate Communication of Design Information The need to keep the ECCS suction lines filled was identified but not appropriately communicated. Follow-through for ensuring start-up procedures contained provisions for filling and venting the system was inadequate.

Root Causes for the Missed Opportunities*

Focus on missed opportunities for identifying and correcting the unanalyzed condition, after the initial design configuration error.

Root Cause No. 4 (RC-4) - Lack of Specific Provisions in the Design Basis Personnel missed opportunities to identify the unanalyzed condition involving the unfilled suction lines because the design basis documents did not contain an explicit statement that required the lines to be filled.

Root Cause No. 5 (RC-5) - Ineffective Questioning Attitude and Technical Rigor of Individuals Some PVNGS personnel had a narrow focus and an incorrect mindset (i.e., incorrect belief in a self-venting theory) in reviewing various documents and information related to the ECCS suction lines. There was general ineffective use of a QV&V process.

(QV&V ["Qualify, Validate, and Verify"] is a three-step tool used to obtain accurate information during critical decision-making.)

Page 2 Revised Reply to Notice of Violation EA-04-221 Root Cause No. 6 (RC-6) - Inadequate Communication of Design Information The need to keep the ECCS suction lines filled was identified but not appropriately communicated.

Root Cause No. 7 (RC-7) - Inadequate Problem Identification and Resolution Issues related to the acceptability of the unfilled ECCS suction lines were not documented on a Condition Report/Disposition Request (CRDR) due to unclear procedural guidance.

Root Cause No. 8 (RC-8) - Less than Adequate Technical Reviews As a result of inadequate technical reviews, PVNGS personnel overlooked information regarding the need to fill the ECCS suction lines or did not review identified issues that could have led to identification of the unanalyzed condition involving the suction lines.

Root Cause No 9 (RC-9) - Limited Operating Experience Program The PVNGS Operating Experience Program did not require reviews of some types of operating experience reports related to the ECCS suction lines.

Contributing Causes for the Initial Failure Contributing Cause No. I (CC-1) - Inappropriate reliance on standard Combustion Engineering design The design of the ECCS suction lines at PVNGS was different than the design at other CE plants, but the PVNGS design did not account for the significance of those differences.

Page 3

Revised Reply to Notice of Violation EA-04-221 Contributing Cause No. 2 (CC-2) - Limited experience and training PVNGS personnel with responsibility for start-up did not have adequate system design or licensing basis training or experience to be able to detect the need for filling of the suction lines.

Contributing Cause No. 3 (CC-3) - Allocation of Resources During start-up, the Safety Injection engineers were under a high workload and had multiple tasks to perform, which deterred them from raising questions on issues not directly related to resolving the specific issues assigned to them.

Contributing Causes for the Missed Opportunities*

• Focus on missed opportunities for identifying and correcting the unanalyzed condition, after the initial design configuration error.

Contributing Cause No. 4 (CC-4) - Weak Operating Experience Program The Operating Experience Program had little guidance applicable to the review of the Industry Operating Experience Reports related to the ECCS suction lines and gave low priority to the reviews, resulting in a narrow focus to the reviews and a lack of review by the Nuclear Assurance Department.

Contributing Cause No. 5 (CC-5) - Limited experience and training PVNGS personnel with responsibility for the Safety Injection System had limited training and experience to be able to detect the need for filling of the suction lines.

Page 4 Revised Reply to Notice of Violation EA-04-221 Contributing Cause No. 6 (CC-6) - Limited Resources System engineers had been under a high workload and had multiple tasks to perform, which deterred them from raising questions on issues not directly related to resolving the specific issues assigned to them. Reviews of IOE reports were generally narrowly focused and limited to addressing the specific issue raised in the report.

Contributing Cause No. 7 (CC-7) - Limited Verification and Validation By design, the "100% validation" of the Design Bases Manuals (DBM) was comprehensive and focused on validation of the information in the DBMs but was not 100%.

Contributing Cause No. 8 (CC-8)- Limited Procedural Guidance The DBM Writer's Guide (Procedure 83DP-4CC02) lacked detailed guidance on how to review source documents during preparation of the DBMs (e.g., there was no requirement to review the entire source document).

Contributing Cause No. 9 (CC-9) - Limited Nuclear Assurance Department (NAD)

Oversight NAD has not had a systematic approach for assessing safety significant or high risk technical specification or design configuration issues.

A collective evaluation was performed of the root and contributing causes, the organizational and programmatic (O&P) issues identified by the Event and Causal Factor Analysis and the Barrier Analysis, and PVNGS operating experience. This collective evaluation identified the following organizational weaknesses (OWM. These weaknesses are viewed as deficiencies provoking conditions or degrading defenses Page 5 Revised Reply to Notice of Violation EA-04-221 associated with the unanalyzed condition involving the unfilled suction piping and the missed opportunities to identify the condition.

• There has been evidence of insufficient or ineffective questioning attitude and technical rigor in Engineering and Operations at various times, particularly when analyzing design and licensing bases and configuration management issues.

• Problem identification and resolution has not always been fully effective.

• Safety Injection System Engineering resources were limited, which presented challenges in effectively attending to routine and emergent issues.

• There were limited system specific training, training materials, and formal system turnover requirements in Engineering.

• The Operating Experience Program did not receive effective support from the site organization.

• The Independent Safety Review (ISR) process did not conduct performance-based assessments to improve plant safety.

Inadequate control of the design and licensing bases was also evaluated for possible classification as an organizational weakness. However, the condition involving the unfilled suction piping appears to be isolated, and reviews of the UFSAR, Design Basis Manual, Licensee Event Reports (LERs), and Engineering Evaluation Requests (EERs),

did not identify any generic concern with respect to the design and licensing basis requirements. Since the condition was isolated, inadequate control of the design and licensing bases was not classified as an organizational weakness.

Page 6 Revised Reply to Notice of Violation EA-04-221

2.

Corrective Steps That Have Been Taken and the Results Achieved Extent of Condition The direct extent of condition was addressed by reviewing other sections of safety related piping to ensure they were filled as needed and tested by providing flow through the pipe as part of a routine test or operational evolution. The review identified no additional sections of piping that were maintained in an unfilled condition except where there was a clear design requirement that they be unfilled (e.g., Containment Spray discharge spray headers).

In order to determine if other design requirements may have been missed in station procedures, a sample review of the UFSAR of selected plant systems was completed.

Any design requirements that were not clearly maintained in station procedures have been entered into the PVNGS corrective action program.

PVNGS Licensee Event Reports (LER) were reviewed to identify those associated with design control. This subset of LERs was reviewed to determine if there was a generic weakness in transmitting design requirements into the station's design/operating basis.

No weakness was identified. However, there was indication that individual knowledge of system, structures and component design requirements was weak. Corrective actions were initiated.

Based upon the results of the review of the extent of condition, it is concluded that the missed design requirement regarding the operating configuration of the ECCS suction line was isolated, and that there is reasonable assurance that similar safety significant configuration conditions do not exist in other fluid, I&C, or electrical systems.

Page 7 Revised Reply to Notice of Violation EA-04-221 Extent of Cause Issues involving problem identification and resolution and human performance (which encompass questioning attitude and technical rigor) have previously been identified as NRC cross-cutting issues at PVNGS. PVNGS has initiated separate CRDRs to assess these areas (including extent of cause evaluations) and to take corrective action.

The other root causes and organizational weaknesses pertain (either directly or indirectly) to the control of the design basis (e.g., limited training contributed to the failure of personnel to recognize that the design intent for filled suction lines was not satisfied). To determine whether these root causes and organizational weaknesses could have broader implications related to the control of the design basis, a review was conducted of PVNGS licensee event reports (LERs) since the beginning of operation to determine whether significant conditions have been identified due to a failure to translate the design basis into requirements. Based upon this review, a concern was identified with respect to the knowledge of engineers. This is similar to one of the root causes and organizational weaknesses identified by the Event & Casual Factor analysis.

In addition, three other reviews were conducted. First, a review was performed of the eight Independent Design Review (IDR) reports for Containment Systems, Auxiliary Feedwater System, Alternating Current System, Auxiliary Systems, Fire Protection System, Environmental Qualification, Control Systems and Balance of Plant (BOP) l&C Systems, and Direct Current Power Systems to determine if the design requirements identified in these reports have been incorporated into design documents. Second, a review was performed of five systems (plus portions of two other systems) to verify that the Design Basis Manuals incorporate the design requirements in the UFSAR and that the plant has not been inappropriately altered by other maintenance, test, or modifications. This review included walkdowns of the Safety Injection (SI) and Auxiliary Feedwater (AF) Systems. Third, a 95/95 probabilistic sample of all Engineering Evaluation Requests (EER) for the SI, AF and Diesel Generator (DG) systems was Page 8 Revised Reply to Notice of Violation EA-04-221 reviewed to determine if plant changes were made outside formal design change processes.

The results of the extent of cause reviews for the design basis issues did not identify any safety significant conditions; i.e., the reviews did not identify any missed design or licensing bases that would impact any system's or component's ability to perform their safety functions. These results indicate that while there were some organizational weaknesses at PVNGS, there is reasonable assurance that they did not have any safety significant effects except with respect to the unfilled ECCS suction line.

Finally PVNGS performed an assessment of the technical adequacy of a sample of high-tiered Industry Operating Experience (IQE) evaluations since circa 1980 (i.e.,

Significant Operating Experience Reports [SOERs], Significant Event Reports [SERs],

Significant Event Notifications [SENs], NRC Information Notices [INs], NRC Generic Letters [GLU, NRC Bulletins, and Operations and Maintenance Reminders [O&MRs]).

This assessment determined that while a number of IOE reviews were weak in the area of either technical rigor or insufficient scope, the team identified no instance where a weak IOE review would have left a latent design weakness in the plant that would adversely impact the performance of a safety function.

The investigation concluded that the safety significant deficiency involving the ECCS suction line is also partially attributable to the limited pre-operational testing of the as-built system, the unique design of the suction lines at PVNGS, and the relative lack of functional use of the system (e.g., the suction piping is not used to flow water during routine operations). Unlike PVNGS, most of the other CE plants had suction lines inherently self-venting, which apparently led some personnel to believe that at PVNGS the suction lines were also self-venting. Additionally, PVNGS used standard CE products (e.g., CESSAR, generic start-up and operating procedures, etc.) and material from other CE plants (which appear to be predicated upon a self-venting suction line design), to develop the design basis and procedures at PVNGS which further contributed to the error at PVNGS. Some personnel did not appreciate the significance Page 9 Revised Reply to Notice of Violation EA-04-221 of the difference in design and did not exhibit the proper questioning attitude or technical rigor and follow-through.

Corrective Actions The ECCS Sump suction lines and sumps in Units 1, 2 and 3 were filled by August 4, 2004. Since then, modifications have been installed in all three units to maintain the ECCS suction piping filled and the ECCS sump dry. The modifications added additional vent, drain and fill connections on the SI piping to facilitate filling and maintaining the lines in a filled condition.

Procedure 40ST-9SI04, "Containment Spray Valve Verification," has been updated to vent the sump suction lines every 31 days. This surveillance test verifies that the piping is maintained filled.

Procedures were revised to require the ECCS suction lines to be filled with borated water prior to returning the system to a mode where the ECCS is required to be OPERABLE. This places the system in the required condition at the completion of testing and maintenance.

The SI Design Basis Manual (DBM), the UFSAR, and the Technical Requirements Manual (TRM) have been revised to reflect the design requirement for this piping to be maintained full. This anchors the design requirement that was not translated into the ECCS operating and/or testing procedures.

As an initial step to bring the site's attention to the root cause of ineffective questioning attitude and technical rigor, Senior Management has communicated to all hands that it is essential that all employees have a strong and effective questioning attitude and demonstrate technical rigor and to challenge assumptions and/or any situations which do not appear to be safe, or per design, or per procedure, or per expectation, or in general do not seem appropriate. The communication re-emphasized the use of the Page 10 Revised Reply to Notice of Violation EA-04-221 QV&V (Qualify, Validate, & Verify) prevent event tool with a short primer on what it is and how to use it.

A Senior Management Sponsor has been designated who is responsible for a site-wide improvement in the culture relating to questioning attitude and technical rigor. The Senior Management Sponsor has developed a plan to improve and anchor the organizational culture with respect to effective questioning attitude and technical rigor.

Checklists for key technical products (High-tiered IOE evaluations [SOER, SEN, SER, IN, O&MR, Topical reports, etc.], Self-assessments, Audits, Significant CRDR evaluations) have been developed for use by both the evaluator and supervisory review to include guidance for performing more expansive OE reviews so that personnel do not focus only on the particular conditions identified in the OE report.

A RAS event case study has been developed and has been presented in a training setting to non-admin PVNGS personnel in Operations, Engineering, Nuclear Fuels Management, Regulatory Affairs and Nuclear Assurance. A few make-up sessions remain to be completed.

"Questioning attitude and technical rigor" tools and processes have been provided to the Palo Verde engineering organization. These tools have been designed to drive situations into "rule-based" processes instead of "knowledge-based" processes. A program to monitor the use of these tools has been implemented to trend and reinforce a questioning attitude and appropriate technical rigor.

3.

Corrective Steps That Will Be Taken to Avoid Further Violations In parallel with the evaluation of the voided sump suction piping, evaluations of the substantive cross-cutting issues in problem identification and resolution and human performance were separately performed. The results of these evaluations along with other internal and external assessments were then used as inputs to the performance of Page 1 1 Revised Reply to Notice of Violation EA-04-221 an integrated organizational effectiveness evaluation. The results of the integrated evaluation are being implemented in the Palo Verde Integrated Improvement Plan which has been separately reviewed by and discussed with the NRC staff.

4.

Date when Full Compliance Will Be Achieved Full compliance was achieved on August 12, 2004, when procedure 400P-9SI02, "Recovery from Shutdown Cooling to Normal Operating Lineup," was revised to include requirements for filling the containment recirculation sump with borated water and for adding demineralized water to the sump for makeup of evaporative losses.

Paae 12