ML063250343

APS Response to NRC Inspection Report 05000528-06-011, 05000529-06-011, & 05000530-06-011

ML063250343
Person / Time
Site:	Palo Verde
Issue date:	11/14/2006
From:	James M. Levine Arizona Public Service Co
To:	Document Control Desk, NRC Region 4
References
102-05593-JML/SAB/JAP/DFH, EA-06-221, IR-06-011
Download: ML063250343 (29)

Text

A--

• A subsidiaryof Pinnacle West CapitalCorporation James M. Levine Mail Station 7602 Palo Verde Nuclear Executive Vice President Tel (623) 393-5300 PO Box 52034 Generating Station Generation Fax (623) 393-6077 Phoenix, Arizona 85072-2034 102-05593-J ML/SAB/JAP/DFH November 14, 2006 U.S. Nuclear Regulatory Commission ATTN: Document Control Desk Washington, DC 20555

Subject:

Palo Verde Nuclear Generating Station (PVNGS)

Units 1, 2 and 3 Docket Nos. STN 50-528, 50-529, and 50-530 APS Response to NRC Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011; EA 06-221 In NRC Special Inspection Report, dated September 28, 2006, the NRC indicated that APS failed to recognize that improper chemistry controls for the spray pond systems in all three units caused degraded performance over a period of years. The report discusses a finding that may have a greater than very low safety significance. APS has reviewed the NRC Inspection Report and has no substantive disagreement with the facts as documented in the report.

In accordance with the Inspection Manual Chapter 0609, the NRC is currently evaluating the safety significance of the finding. At a November 20, 2006 Regulatory Conference in Arlington, Texas, APS will provide the NRC its perspective on the facts and analytical assumptions relevant to determining the safety significance of the finding.

Regardless of the ultimate significance of the finding, APS takes this matter seriously.

We have implemented immediate actions to assure the system remains within the design basis, have implemented or planned additional actions to prevent recurrence, and have considered this issue in light of our ongoing improvement plans.

The purpose of this letter is to provide the results of APS' evaluation of the spray pond issue in advance of the Regulatory Conference to assure the NRC that APS recognizes the programmatic and organizational implications of this matter. APS is providing this information to facilitate a focused discussion at the conference on the safety significance of the spray pond degraded performance.

U.S. Nuclear Regulatory Commission ATTN: Document Control Desk APS Response to NRC Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011; EA 06-221 Page 2 to this letter contains a summary of APS' evaluation and how APS considered the results in the context of other ongoing corrective actions. Enclosure 1 also includes an APS response to the five apparent violations related to the potentially greater than green finding.

Finally, the non-cited violations described in the Inspection Report are not addressed in this submittal because APS agrees with the violations which have been entered into, and will be resolved in the corrective action program.

APS noted some minor corrections for the subject Inspection Report and is submitting for completeness.

The actions described in Enclosure 1 represent corrective actions and are not regulatory commitments. There are no regulatory commitments in this letter.

If you have any questions, please contact James A. Proctor at (623) 393-5730.

Sincerely, JML/SAB/JAP/DFH/gt

Enclosures:

1. Summary of APS Investigation and Corrective Actions

2. Corrections to NRC Inspection Report cc: B. S. Mallett NRC Region IV Regional Administrator M. B. Fields NRC NRR Project Manager G. G. Warnick NRC Senior Resident Inspector for PVNGS

ENCLOSURE 1 Summary of APS Investigation Into Loss of Thermal Performance of the Essential Cooling Water (EW)

And Emergency Diesel Generator (EDG) Intercooler Heat Exchangers, And Corrective Actions I. Introduction On March 30, 2006 Arizona Public Service Company (APS) met with the NRC to discuss progress of the comprehensive integrated improvement plan, which addresses performance issues at the Palo Verde Nuclear Generating Station (PVNGS). APS outlined the elements of its Performance Improvement Plan (PIP), and how, as part of that plan, APS management is monitoring performance and adjusting its improvement initiatives, as appropriate. APS has considered and incorporated in its PIP the organizational issues resulting from its review of the Recirculation Actuation Signal (RAS) Sump Event, the substantive cross-cutting issues in Problem Identification and Resolution (PI&R) and Human Performance (HU), and other indicators of performance.

A separate, stand-alone 95002 action plan was developed to address the issues directly related to the YELLOW finding for the RAS.

APS recognizes that its corrective actions for the 95002 action plan as well as the PIP have not yet resulted in the desired improved performance. As a result, APS recently set forth to revise the 95002 action plan to include additional corrective actions to address the remaining issues; particularly, improving the adequacy of technical rigor, reinforcement of a questioning attitude among personnel, and improving metrics to better measure performance. APS has similarly set forth to revise the PIP to include further corrective actions to address the two substantive cross-cutting issues (i.e., PI&R and HU).

Recognizing that the actions that have been taken have not been fully effective, a more rigorous approach has been adopted for developing the revision to the 95002 action plan and PIP. The approach includes development of problem statements and comparing these to the actions previously taken. From an understanding of the shortcomings of those previous actions, additional actions to be taken will be identified.

In order to monitor the effectiveness of the actions, a more rigorous approach will be applied to establishing the appropriate metrics. First the intent of the metric will be defined. From this the criteria to be used for populating the metric will be established and the data sources will be identified. The metrics will then be developed and populated with the appropriate data.

This enhanced approach has been applied to the 95002 action plan and is being applied to the PIP action plans for the substantive cross-cutting issues. These three revised action plans will be submitted to the NRC by December 15, 2006. This revised approach is also being applied to the three other focus areas of the PIP, namely leadership, accountability and standards.

1

In Section II of this enclosure, APS sets forth corrective actions to address the SP root and contributing causes. However, APS did not limit its considerations of appropriate corrective actions to the SP; rather, APS considered the broader implications of the root cause investigation. Specifically, many of the root and contributing causes identified in the spray pond event are substantially the same as those identified in the RAS sump event and other self-assessments APS has conducted. For example, root cause three and contributing cause two (personnel failed to effectively identify problems, and missed numerous opportunities to resolve the foulant problem); contributing cause six (personnel did not always exhibit a sufficiently questioning attitude); and contributing cause seven (personnel did not apply appropriate technical rigor in their evaluations) reflect the very performance deficiencies that the aforementioned action plans are designed to correct. For this reason, the investigation relies, in part, on the established corrective actions in the revised 95002 and PIP action plans.

The purpose of this enclosure is to summarize APS' investigation, to describe the corrective actions, and to summarize further corrective actions, particularly in response to the five apparent violations related to the NRC's potentially greater than green finding.

This enclosure does not address APS' position on the significance of the NRC finding, as that will be the subject of a Regulatory Conference between APS and the NRC on November 20, 2006.

I.A. Summary of Key Events Leading to Identification of Possible Degraded Performance The spray pond (SP) system is the ultimate safety-related heat sink and is an open water, standby cooling system. The key safety functions for the SP system are heat removal and maintenance of vital auxiliaries. The primary function of the SP system is to remove heat from all essential components required for normal and emergency shutdown of the plant. The SP system also provides cooling to the Emergency Diesel Generators (EDG), which is an essential function, necessary for a reliable source of on-site AC electrical power.

The design bases for the SP system assume that the system will function for 26 days without makeup water, losing approximately 85% of its original volume of water to evaporation and drift.

During the mid-1 990s, APS initiated monitoring of performance of the Essential Cooling Water (EW) heat exchangers in part due to NRC and industry initiatives, such as Generic Letter (GL) 89-13, "Service Water System Problems Affecting Safety-Related Equipment." EW thermal performance is monitored and reported as a percent of the design value, which includes the capability to remove assumed design heat loads, with bounding coolant conditions and with an industry standard for service fouling. As manufactured, the performance capability of the EW heat exchangers for new conditions is greater than the design value. A margin of 0% reflects performance at the 2

design value, including the service fouled condition. Positive margin reflects heat exchanger performance above the design value. Negative margin does not necessarily reflect an inability to perform its safety function. This is because actual heat removal conditions (e.g., coolant temperatures, environmental, actual heat loads) are typically less severe than the assumed bounding design conditions.

In 1994, zinc-phosphate treatment was introduced into the SP system to mitigate internal piping corrosion caused by small localized failures of the internal coating, specifically in underground spray pond piping. Leaks detected in EW heat exchanger tubes and inspection of spray pond piping internals promoted greater attention to corrosion control.

In the spring of 1995, there was a release of corrosion nodules that caused limited blockage of EDG heat exchanger tubes. APS performed additional inspections to assess the extent of condition. After the initial release of corrosion nodules, the amount quickly subsided and has remained within the inspection acceptance criteria established as a result of this event. This condition was documented in Licensee Event Report 05000528/1995-005-00.

In the late 1990s, APS first identified fouling of EDG intercoolers through trending of diesel intake temperatures. There were losses of EDG intercooler performance and visual inspection of all EDG heat exchangers revealed a white film, generally associated with the zinc-phosphate chemical treatment. The Palo Verde maintenance team addressed this issue by mechanical cleaning initially, and followed this with a chemical cleaning process in 2001. During the time period from 1995 to 2001, APS identified no visual fouling in the EW heat exchangers.

In 2001, Palo Verde changed chemical vendors to GE-Betz from Calgon. Prior to 2001, the dispersant used for the essential spray ponds was Calgon PCL-401. In 2001, GE-Betz provided Palo Verde with a dispersant (HPS-1) that GE-Betz personnel asserted to be equivalent to the Calgon product.

In the Spring 2002 Unit 2 refueling outage, APS identified the first negative thermal performance margin of any EW heat exchanger. Data collected during the refueling outage on the EW B heat exchanger was analyzed and determined to be negative 9.5%.

In September 2002, a low margin (positive 3.6%) was measured on Unit 1 EW B heat exchanger.

In April 2003 Unit 3 EW A thermal performance was measured at a negative 22%

margin.

In September 2003, the Unit 2 EW A heat exchanger exhibited a 1% margin, before cleaning.

3

In November 2003, APS initiated corrective action based on a loss of thermal margin in all 3 units' EW heat exchangers. APS concluded that the EW heat exchangers were fouling due to a reaction between the new dispersant (HPS-1) and the non-oxidizing biocide used to control algae growth. APS concluded, at that time, that these two chemicals were reacting to form a polymer, thus fouling the heat exchangers. During the course of APS' current investigation, the investigation team determined that this conclusion was incorrect.

In March 2004, the Chemistry department began using a new dispersant (DN2317),

which was acquired from the vendor, GE-Betz. The new dispersant was used to address the increase in fouling rate that had been concluded to be associated with the interaction of the HPS-1 dispersant and biocide.

Immediately following the change in dispersant the thermal performance of the EW heat exchangers improved, further supporting the incorrect conclusion regarding the previous dispersant. Because the new dispersant was not directly measurable, a molybdenum tracer had been added to the dispersant to allow detection and to provide an indirect correlation to the amount of dispersant available in the ponds. However, there was a difference in the consumption rate of the dispersant, and the molybdenum. Over time, the molybdenum tracer provided a false indication that sufficient levels of dispersant were present in the ponds. Consequently, no dispersant was added for an extended period of time.

As a result, in October of 2005, when Unit 1 was in a refueling outage and Unit 2 and 3 were in forced shutdowns, a loss in thermal performance margin was identified indicating that the fouling rate had increased. Unit 1 and 3 EW heat exchangers were cleaned during their refueling outages (Fall 2005 for Unit 1 and Spring 2006 for Unit 3).

When it was projected that the rate of fouling would not support Unit 2 EW function during the summer of 2006, both Unit 2 EW heat exchangers were cleaned during online equipment maintenance outages.

I.B. APS Identification of Possible Degraded Performance and Initiation of an Investigation Team On May 17, 2006, APS identified that the EDG 2B intake air temperature was higher than the maximum limits specified in the data collection logs. APS declared the EDG inoperable on May 19, 2006 and inspected and cleaned the intercooler. At that time, APS took immediate corrective actions to clean the intercoolers on all six EDGs and inspect the EDG Jacket Water (JW) and Lube Oil (LO) heat exchangers in Unit 1. Unit 1 EW heat exchangers were also re-cleaned prior to the summer.

As a result of these issues, APS initiated an investigation team that consisted of individuals from the Chemistry Department (Palo Verde and APS Corporate Chemistry),

Engineering Department, Performance Improvement Team members with plant operations experience, and industry consultants. The scope of the investigation included a review of all heat exchangers that are cooled by spray pond water and 4

addressed the lack of dispersant control for the SP from approximately March of 2004 until June of 2005. The investigation team evaluated both past and current plant documents including sampling results, reviewed industry information, obtained independent analysis of foulant samples and conducted limited inspections of the system to support their investigation and to draw their conclusions.

The next section summarizes the results of the investigation team, the causal analysis results and APS' corrective actions. Subsequent to this effort, APS has reviewed, along with EPRI and other experts, the safety significance of these issues. As indicated in the cover letter, APS will provide its perspective on the safety significance of this matter during the November 20, 2006 Regulatory Conference.

II. Results, Causal Analysis and Corrective Actions II.A. Direct Cause and Extent of Condition II.A.1 Direct Cause The direct cause of the loss of thermal performance from the EDG intercooler and EW heat exchangers was the formation of an insulating precipitant on the SP side of the heat exchanger surfaces.

This precipitant, calcium-zinc-phosphate, formed when the chemical constituents were present in sufficient concentrations and the control factors (i.e., pH and dispersant) were not adequately controlled.

II.A.2 Extent of Condition APS took a broad perspective in considering the extent of condition and assessed the plant heat transfer systems, beginning with components of the essential spray system.

Three types of foulant were identified during the investigation: zinc-hydroxide film, scale (calcium-phosphate or calcium carbonate) and precipitant (calcium-zinc-phosphate).

Other Spray Pond Heat Exchangers In addition to the EDG intercoolers and the EW heat exchangers, three other heat exchangers on the EDGs are cooled by the SP system.

EDG Jacket Water and Lube Oil Coolers The EDG JW and LO heat exchangers were less susceptible to fouling than the EDG intercooler or EW heat exchanger. It was not possible to quantify the amount of thermal performance loss in the EDG JW and LO coolers by calculational methods; but, it was determined there was ample margin to respond to a DBA and cope with minor degrees of fouling based on the following:

5

• Less precipitant existed in the EDG JW and LO heat exchangers than the EDG intercooler and EW heat exchangers.

• Original tube plugging design margin remains intact.

• Performance monitoring of JW and LO parameters provides assurance that the thermostatic control valves, which regulate temperature, are functioning within their setpoint band.

• Assuming full load and peak Design Basis Accident (DBA) spray pond temperatures, should any thermostatic valve reach its full open position, there is additional margin beyond the limit of the thermostatic valve to the manufacturer's operating limit.

" Loads will be less than the design values when peak SP temperature occurs.

EDG Fuel Oil Coolers The fuel oil heat exchanger was supplied as original equipment on the EDG. It is a carry-over from earlier diesel models, with an original purpose to support extended idle, no-load, operation. The fuel oil piping system recirculates excess fuel oil to the suction of the fuel oil transfer pump, which heats the fuel oil by pump heat. The fuel oil cooler was intended to prevent fuel oil from overheating during extended no-load operation. The fuel consumption of a KSV-20 at no-load is sufficient to preclude the need for cooling; therefore, EDG safety functions would not have been impacted by a loss of thermal performance in the EDG Fuel Oil cooler.

Potential for SP Foulant to Manifest as Scale Another immediate concern was that with the presence of the constituents in sufficient amounts to form a precipitant there was a possibility that the chemical constituents may also form scale. Scale has a crystalline structure, tenaciously adheres to surfaces, and has an increased likelihood of forming with rising temperature, because scale forming constituents become more insoluble.

The corrective actions to establish correct chemistry control in the spray ponds address the precipitant and also resolve the propensity to form scale. Additionally, an analysis was performed by the Electric Power Research Institute (EPRI) on APS' behalf to assess the possibility and extent of scale formation in light of the concentration of chemical constituents during the period in question. The results of this analysis predict a minor amount of scale formation under DBA conditions. The results of the aggregate impacts of the precipitant and scaling propensity will be presented at the November 20, 2006 Regulatory Conference.

Applicability to Other Open Cooling Water Systems The investigation team also evaluated the susceptibility of other open cooling water systems to fouling problems.

6

The SP fouling problem was viewed as being directly applicable to the Circulating Water (CW) System, which is also an open cooling water system. Unlike the SP system, the CW system is a continuously operating system and does not fulfill a safety function, but is relied upon for power operations. It was concluded there was no immediate impact to CW but to ensure lessons learned from this event are considered by the Water Reclamation Facility (WRF) personnel who maintain CW chemistry, the results of the evaluation will be reviewed with WRF personnel for lessons learned and technical issues.

Applicability to Heat Exchangers Addressed by NRC Generic Letter 89-13 The EW and EDG heat exchangers are within the scope of equipment addressed by GL 89-13. In view of the issues identified with the spray pond it was concluded an assessment of GL 89-13 implementation should be performed. See corrective actions to be taken in Section II.A.3 below.

II.A.3. Corrective Actions - Direct Cause and Extent of Condition Corrective Actions Taken and Results Achieved The SP system chemistry was corrected as follows:

• Dispersant additions were resumed in all spray ponds.

" Additional acid was added to lower pH in all ponds.

" Feed and bleed was used to reduce calcium and phosphate concentrations in each spray pond.

EW heat exchangers were cleaned in all three units starting with the Unit 3 heat exchangers in the spring 2006 refueling outage.

EDG intercoolers were cleaned in all three units and a Standing Order was issued to increase the EDG test frequency to assess intercooler performance. This standing order continued through September 29, 2006. It was removed after APS gained confidence that the immediate corrective actions were effective.

Procedure 40DP-9OP08, "Diesel Generator Test Records," was revised to require a work order to be generated to clean any EDG intercooler if temperature exceeds 120F, to ensure the system does not exceed its Design Basis Manual (DBM) limit of 130F.

The SPs have been cleaned, facilitating improved chemistry control and accessibility for inspections.

Additional cleaning and inspection activities of the EDG and EW heat exchangers have been scheduled to ensure the effectiveness of the completed corrective actions and to ensure there are no unintended consequences of the chemistry changes.

7

Corrective Actions to Be Taken Modifications will be installed to add higher capacity acid and dispersant pumps. (Due date: December 31, 2006).

Systems Engineering will conduct a self-assessment of the SP systems to include NRC GL 89-13 and industry operating experience on service water systems. (Due date:

February 28, 2007)

II.B Five Root Causes Identified The investigation identified the following five Root Causes:

" Root Cause #1: Inadequate Chemistry Control Program

" Root Cause #2: Managed to Inadequate Chemistry Metrics

" Root Cause #3: Inadequate Resolution of Performance Problems

• Root Cause #4: Management Reliance on an Expert

" Root Cause #5: Ineffective Change Management This section summarizes each root cause and the associated corrective actions.

II.B.1 Root Cause #1 and Corrective Actions - Inadequate Chemistry Control Program The Palo Verde chemistry control program for the SP system was ineffective. The program used control ranges that were inadequate for some parameters and lacked other necessary elements.

Corrective Actions and Results Achieved Procedure 74DP-9CY04, "System Chemistry Specifications," was revised to establish control limits, sampling frequencies and action levels that ensure the spray ponds are able to perform their specified function in a DBA.

The strategic plan for SP Chemistry Control was revised to include chemistry control requirements for fouling, corrosion control, and biological control.

The dispersant was changed to one that is directly measurable.

The Chemistry Department was trained in the direct, root and contributing causes of the investigation.

Extent of Cause Corrective Actions An assessment of the chemistry controls for closed cooling water systems is in progress. (Due date: November 30, 2006) 8

A Chemistry Design Basis Manual will be developed. (Due date: December 24, 2007)

II.B.2 Root Cause #2 and Corrective Actions - Managed to Inadequate Chemistry Metrics A single metric, corrosion rate, existed for the SP system, which had the unintended affect of causing personnel to make decisions based on this metric and misunderstand the health of the SP chemistry. There was no metric for fouling.

Corrective Actions Taken and Results Achieved The Chemistry Department revised the performance indicators for the SP Chemistry Control Program to ensure measurable targets are established for corrosion, fouling, and biological control.

Corrective Actions to Be Taken Training will be provided to the Chemistry Technical Advisors, their backups, and Chemistry Supervision on the indications of fouling mechanisms associated with the chemicals used in the SP system. The training will also include changes to the Chemistry Management Expectations on Qualification, Validation and Verification (QV&V) and technical rigor. (Due date: November 21, 2006)

The System Health Report for the SP system will be updated in the engineering program database to reflect the metrics established by Chemistry. (Due date: January 15, 2007)

Extent of Cause Corrective Actions The site's metric guideline was revised to identify that a thorough review of a proposed metric is necessary to identify and measure necessary objectives so that unintended consequences are minimized (e.g., single metric such as reliability without a complementary metric of availability).

An effectiveness review of site and department metrics will be conducted to determine if appropriate metrics have been established and are being used by leaders to guide actions. (Due date: March 30, 2007)

Other Chemistry Department metrics will be reviewed to ensure complementary metrics exist where appropriate. (Due date: April 30, 2007) 9

ll.B.3 Root Cause #3 and Corrective Actions - Inadequate Resolution of Performance Problems Palo Verde personnel (leaders and frontline) failed to effectively resolve previously identified problems with a zinc-related foulant despite numerous opportunities to do so.

One specific example of inadequate resolution of performance problems was cited as an apparent violation of 10 CFR 50 Appendix B, Criterion XVI, "Corrective Actions", for the failure to correct the degraded performance of EW 2B, when identified in March 2002.

Corrective Actions Taken and Results Achieved Procedure 74DP-OCY01, "Specifications for Bulk Chemicals," was revised to require a chemical compatibility test when changing chemicals.

Tools have been developed to improve engineering work product quality, including the Engineering Department Guides (EDG) 01 and 02 to minimize human performance errors.

The Condition Reporting procedure was revised to include checklists for processing CRDRs to improve resolution of conditions that are adverse to quality.

Management Expectations for questioning attitude and technical rigor have been incorporated into Chemistry Department Policy (CDP-01).

Corrective Actions to Be Taken Procedure 70TI-9EW01, "Thermal Performance Testing of Essential Cooling Water Heat Exchangers", will be revised to improve heat exchanger performance management. (Due date: January 31, 2007)

The Systems Engineering Department performance monitoring database will be revised to include a parameter for the differential temperature between the EDG intake air manifold and the SP cooling water to detect degradation of the intercoolers. (Due date:

February 28, 2007)

Extent of Cause Corrective Action This cause is illustrative of the issues being addressed by the revised PIP action plan for PI&R. The investigation compared the issues for the SP event to the action being taken in the revised PI&R action plan and concluded that plan will appropriately address the extent of cause. As noted in Section I. Introduction, the revised PIP action plan for the PI&R cross-cutting issue will be submitted to the NRC by December 15, 2006.

10

ll.B.4 Root Cause #4 and Corrective Actions - Management Reliance on an Expert There were instances where the chemistry advisor's working assumptions were flawed and the lack of technical support from other technical specialists resulted in missed opportunities to preclude the precipitant from occurring.

Corrective Actions Taken and Results Achieved Chemistry Standards and Expectations have been revised to require independent review of calculations that relate to changes to chemistry design controls or the Chemistry Design Basis Manual.

Corrective Actions to Be Taken Procedure 74DP-9CY04, "System Chemistry Specifications" will be revised to require a technically independent review when making changes to chemistry control regimes.

(Due date: November 17, 2006)

The Chemistry Department will designate backups to selected advisors in the chemistry department, will develop training plans and identify critical attributes for these backups.

(Due date: November 30, 2006).

A training program specific to chemistry will be developed and implemented, using the Engineering Training Program Description (TPD) as a reference. Critical job skills and functions of Chemistry Technical Support staff and Instrument Maintenance staff will be identified and a structure for additional development of current staff and for development of replacement personnel will be included. (Due date: April 30, 2007)

Extent of Cause Corrective Actions Each department will evaluate susceptibility to reliance on experts and take appropriate corrective actions. (Due date: December 15, 2006) ll.B.5 Root Cause #5 and Corrective Actions - Ineffective Change Management APS did not adequately assess the significance or impacts of changes made to the Chemistry Control Program or activities that impact the program.

Corrective Actions Taken and Results Achieved Procedure 93DP-OLC17, "10 CFR 50.59 and 72.48 Guidance Manual," was revised to require a 50.59 screening when changing chemicals and corresponding changes were made to Chemistry procedure 74DP-OCY01, "Specifications for Bulk Chemicals."

The commitments for Generic Letter 89-13 were verified to be accurately reflected in the Regulatory Commitment Tracking System.

11

Ineffective Change Management has been captured in the Performance Improvement Plan and led to the development of the "RAPID" (Recognize, Assess, Plan, Implement, and Drive) program to help manage the change process at Palo Verde.

Corrective Actions to Be Taken Systems Engineering will conduct a self-assessment of the SP systems to include NRC GL 89-13 and industry operating experience on service water systems. (Due date:

February 28, 2007)

Recognizing similarities to the issues identified in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation will be performed to ensure outstanding issues are appropriately identified, prioritized and are being properly addressed. (Due date: March 1, 2007)

II.C Contributing Causes The investigation identified the following nine contributing causes:

" Contributing Cause #1: Palo Verde did not solicit outside assistance

• Contributing Cause #2: Inadequate Prioritization of the SP System

• Contributing Cause #3: Narrow Focus

• Contributing Cause #4: Emphasis on Cost Control

• Contributing Cause #5: Erosion of Design Margin

" Contributing Cause #6: Questioning Attitude of Workers

" Contributing Cause #7: Lack of Rigor in Evaluations

" Contributing Cause #8: Living with Problems

• Contributing Cause #9: Removing UFSAR Requirements This section summarizes each contributing cause and the associated corrective actions.

I1.C.1 Palo Verde Did Not Solicit Outside Assistance Palo Verde personnel did not effectively compare their chemical control program for the SP system with other plants because personnel considered the ultimate heat sink for Palo Verde to be unique.

Corrective Actions Taken and Results Achieved Chemistry Department Policy (CDP-01) was revised to incorporate a requirement for Corporate Chemistry or an outside individual to conduct a review of the SP system Chemical Control Program on a semi-annual basis.

12

Palo Verde's Nuclear Assurance Department revised the master assessment plan to require procedures selected for reviews during audits to include a full basis review of the entire procedure against the applicable licensing and design basis.

II.C.2 Inadequate Prioritization of the SP System Palo Verde management did not effectively prioritize the SP system problems to ensure they were resolved in a timely manner.

Corrective Actions to Be Taken Modifications will be installed to add higher capacity acid and dispersant pumps. (Due date: December 31, 2006).

The Tower Blowdown (TB) makeup line will be restored to Unit 1 spray ponds. (Due date: March 30, 2007)

The temporary modification for zinc and biocide chemical addition will be replaced by a modification in all three units. (Due date: September 30, 2008)

The feasibility of installing a SP filter system will be performed. (Due date: June 30, 2007)

II.C.3 Narrow Focus Palo Verde personnel occasionally displayed a narrow focus to problem solving, which resulted in missed opportunities to address the cause, rather than the symptoms. In some cases, the corrective actions that were put in place to solve one symptom resulted in the manifestation of new problems.

Corrective Actions Taken and Results Achieved Procedure 74DP-9CY04, "System Chemistry Specifications," was revised to establish control limits, sampling frequencies and action levels that ensure the spray ponds are able to perform their specified function in a DBA.

Implemented a program to walk down safety-significant systems weekly on a 12 week schedule. This walk down is performed by Operations and Engineering personnel.

Corrective Actions to Be Taken This cause is illustrative of the issues being addressed by the revised 95002 and PIP action plans. The investigation compared the issues for the SP event to the action being taken in those action plans and concluded they will appropriately address this contributing cause.

13

An engineering project, referred to as the Component Design Basis Review (CDBR) began in October of 2006. The purpose of the review is to verify the design bases have been correctly implemented for high risk-significant components and operator actions.

The review is estimated to take 24-48 months to complete for approximately 250 components and operator actions.

II.C.4 Emphasis on Cost Control Palo Verde personnel, management and frontline, have occasionally made inappropriate cost basis decisions that result in inaction or delay for equipment with perceived lower safety significance.

Corrective Actions Taken and Results Achieved The closed cooling water chemical addition systems were evaluated and it was verified they are adequate to control chemistry.

Corrective Actions to Be Taken Recognizing similarities to the issues identified in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation will be performed to ensure outstanding issues are appropriately identified, prioritized and are being properly addressed. (Due date: March 1, 2007)

II.C.5 Erosion of Design Margin Personnel used various corrective action programs and processes to accept progressively greater amounts of degradation, eventually changing the organization's perception of normal.

Corrective Actions to Be Taken Modifications will be installed to add higher capacity acid and dispersant pumps. (Due date: December 31, 2006).

The TB makeup line will be restored to Unit 1 spray ponds. (Due date: March 30, 2007)

A Chemistry Design Basis Manual will be developed. (Due date: December 24, 2007)

Recognizing similarities to the issues identified in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation will be performed to ensure outstanding issues are appropriately identified, prioritized and are being properly addressed. (Due date: March 1, 2007) 14

I1.C.6 Questioning Attitude of Workers Palo Verde personnel did not always question what they saw, which contributed to the misunderstanding that ultimately impacted the SP heat exchanger performance.

Corrective Actions to Be Taken Develop and implement site-wide training on a case study, based on the SP investigation, emphasizing lessons learned from this event. (Due date: October 1, 2007)

This cause is illustrative of the issues being addressed by the revised 95002 action plan. The investigation compared the issues for the SP event to the actions, being taken in this action plan and concluded it will appropriately address this contributing cause.

I1.C.7 Lack of Rigor in Evaluations Palo Verde personnel did not always apply an appropriate amount of technical rigor in evaluations related to the thermal performance of the SP heat exchangers.

Corrective Actions to Be Taken This cause is illustrative of the issues being addressed by the revised 95002 action plan. The investigation compared the issues for the SP event to the actions being taken in this action plan and concluded it will appropriately address this contributing cause.

II.C.8 Living with Problems Palo Verde personnel have, in some cases, accepted and worked around problems.

Corrective Actions to Be Taken Recognizing similarities to the issues identified in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation will be performed to ensure outstanding issues are appropriately identified, prioritized and are being properly addressed. (Due date: March 1, 2007)

This cause is illustrative of the issues being addressed by the revised 95002 and PIP action plans. The investigation compared the issues for the SP event to the action being taken in those action plans and concluded they will appropriately address this contributing cause.

II.C.9 Removing UFSAR Requirements Personnel removed information from the UFSAR that may have helped preclude this event.

15

Corrective Actions to Be Taken Revise the UFSAR to include spray pond chemical addition equipment necessary to maintain the design basis. (Due date: March 30, 2007)

Revise the UFSAR to incorporate the TB make up line modification to Unit 1. (Due date: March 30, 2007)

III. Review of Latent Design Issues APS recognizes both the RAS event as well as the SP event revealed significant, latent design basis issues. As a result, APS will address the possibility of other latent design issues that remain unidentified. The cornerstone of the effort to identify these issues is the component design basis review (CDBR) which APS committed to perform in a letter to the NRC dated June 2, 2006. The purpose of the review will be to verify the design bases have been correctly implemented for highly risk-significant components and operator actions as defined in the June 2, 2006 letter. The review will verify the capability of these components to perform their intended safety functions and will use the inspection methodologies detailed in NRC Inspection Procedure (IP) 71111.21, dated December 2, 2005.

APS will implement this component design basis review in two phases. The first phase will include the highly risk-significant components and operator actions that are in the Mitigating Systems Performance Indicator systems. It is expected that at least 20 of these components and operator actions will be complete by the end of the first quarter 2007. The second phase will include the remaining highly risk-significant components and operator actions.

In addition to the CDBR, APS will initiate a Plant Health Committee (PHC) with the distinct purpose of providing a high level management overview of current and latent issues with the potential to impact plant operations. The PHC will focus senior management representatives from Operations, Engineering, and Maintenance on issues that need management attention and resolution, including: weekly system health reviews, corrective actions status, single point vulnerabilities and resolutions, prioritization of plant modifications, and station program health and implementation reviews.

The Plant Health Committee represents a function that is much more comprehensive in management oversight and intrusiveness into plant performance than previous System Team Steering Committee efforts. The details of the PHC will be discussed further at the November 20, 2006 regulatory conference.

16

IV. Response to Apparent Violations This section sets forth APS' position on the five apparent violations and summarizes corrective actions taken or planned that are directly related to the apparent violations.

IV.A. Apparent Violation of Technical Specification 3.7.7 Restatement of Apparent Violation An apparent violation of Technical Specification 3.7.7 was identified because Train B of the Essential Cooling Water System in Unit 2 was not capable of performing its safety function for approximately 6.8 months ending on September 27, 2003.

Admission APS' significance determination has concluded there was no violation of TS 3.7.7. The details leading to this conclusion will be presented by APS during the November 20, 2006, Regulatory Conference. Specifically, our significance determination demonstrates that the system would have been able to perform its safety function due to the large margin in the EW heat exchanger design in spite of the degraded conditions.

Corrective Actions The corrective actions delineated above in Section II of this Enclosure address the degraded conditions of the spray pond and have returned it to conformance with its design basis.

IV.B. Apparent Violation of Criterion Xl, Test Control Restatement of Apparent Violation An apparent violation of 10 CFR Part 50, Appendix B, Criterion X1, "Test Control," was identified because the two procedures that were performed to measure essential cooling water heat exchanger performance were implemented in a way that was inadequate to ensure the timely determination that the requirements and acceptance limits contained in applicable design documents were met.

Admission APS admits this apparent violation.

Cause The potential violation has been entered into the Palo Verde Corrective Action Program and the apparent cause evaluation is currently in progress.

17

Corrective Actions Taken and Results Achieved Since September 2002, an evaluation of the EW heat exchanger thermal performance test data has occurred during the outage in which the data was collected.

Corrective Actions to Be Taken Procedure 70TI-9EW01, "Thermal Performance Testing of the Essential Cooling Water Heat Exchangers," will be revised to require heat exchanger performance data analysis to be completed prior to entry into Mode 4. (Due date: November 15, 2006)

VI.C. Apparent Violation of 10 CFR 50.59 Restatement of Apparent Violation An apparent violation of 10 CFR 50.59 was identified for making nine revisions to Procedure 74DP-9CY04, "System Chemistry Specification," a procedure described in the Updated Final Safety Analysis Report, between 1998 and 2004 without performing evaluations of the potential impact of the changes on the safety-related components in the spray pond system; the changes revised spray pond chemistry parameter limits which were subsequently determined to have contributed to heat exchanger fouling.

Admission APS admits this apparent violation.

Cause APS determined that the causes of the apparent violation of 10 CFR 50.59 were as follows:

• Changes to chemistry limits were viewed as administrative and related to maintenance.

" There was a failure to rigorously evaluate the changes to the spray pond chemistry control program Corrective Actions Taken and Results Achieved Chemistry procedure 74DP-9CY04 was restored to the correct condition and 10 CFR 50.59 was applied to these changes to implement the current chemistry regime of the SP System.

The 10 CFR 50.59 guidance manual (93DP-OLC17) was revised to provide specific requirements for chemistry related changes.

18

Corrective Actions to Be Taken In order to determine if other chemistry procedures were changed without the proper 10 CFR 50.59 evaluations, a random sample of 58 chemistry procedure changes are being reviewed. (Due date: November 17, 2006)

Chemistry personnel will be trained on the changes to the 10 CFR 50.59 guidance manual. (Due date: November 17, 2006)

In addition to a' review of chemistry related procedure changes, a review of the performance monitoring results of the 10 CFR 50.59 performance review team from the past year will be performed to determine if there are any adverse trends or areas needing additional review. (Due date: December 30, 2006)

IV.D. Apparent Violation of Criterion XVI, Corrective Actions Restatement of Apparent Violation An apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions," was identified. On March 19, 2002, performance testing for Essential Cooling Water Heat Exchanger 2B indicated that the system would not be capable of performing its design function, but this significant condition adverse to quality was not promptly identified, the cause determined, or corrective actions taken to restore the required heat exchanger performance. The failure to correct this degraded performance contributed to the continued degradation and eventual loss of function for an estimated period of 6.8 months.

Admission APS admits this apparent violation with the following clarification. As stated in Section IV.A, APS concluded that the EW B heat exchanger in Unit 2 would have been able to perform its function during the period in question. APS admits that actions were not taken to promptly identify, accurately determine the cause and to establish effective corrective actions to prevent thermal performance degradation.

Cause The evaluation of the EW 2B condition by engineering personnel lacked technical rigor.

Correction Actions The corrective actions for this violation are provided in Section ll.B.3 of this enclosure.

19

IV.E. Apparent Violation of Criterion III, Design Control Restatement of Apparent Violation An apparent violation of 10 CFR Part 50, Appendix B, Criterion Ill, "Design Control,"

was identified for failure to correctly evaluate the scaling potential of the safety-related heat exchangers cooled by the emergency spray pond during a design basis accident.

An error in the EPRI SEQUIL calculation caused the licensee to incorrectly conclude that scaling would not occur under the conditions established in the chemistry control program.

Admission APS admits this apparent violation.

Cause Incorrect use of a computer model, erroneously predicted that calcium phosphate scale would not form under the input conditions. Chemistry personnel selected input parameters to simulate conditions for evaluation. The individual performing the modeling was unfamiliar with the software and while inputting data into the model and selecting the parameters for the calculation, failed to set the super-saturation parameter correctly. This process relied on a single individual to perform the computer modeling predictions with no independent reviews or peer checks.

Corrective Actions Taken and Results Achieved Utilizing the correct input parameters and settings Chemistry reran the EPRI SEQUIL calculation. The revised data indicated that there was a high potential for calcium phosphate scaling.

Chemistry Standards and Expectations have been revised to require independent review of calculations that relate to changes to chemistry design controls. This requirement will also apply to the Chemistry Design Basis Manual, once developed.

Management expectations for questioning attitude and technical rigor have been incorporated into Chemistry Department Policy (CDP-01).

Corrective Actions to Be Taken Procedure 74DP-9CY04, "System Chemistry Specifications" will be revised to require a technically independent review when making changes to chemistry control regimes.

(Due date: November 17, 2006) 20

Chemistry Department will designate backups to the selected advisors in the chemistry department, will develop training plans and identify critical attributes for these backups.

(Due date: November 30, 2006).

A training program specific to chemistry will be developed and implemented, using the Engineering Training Program Description (TPD) as a reference. Critical job skills and functions of Chemistry Technical Support staff and Instrument Maintenance staff will be identified and a structure for additional development of current staff and for development of replacement personnel will be included. (Due date: April 30, 2007)

Each department will evaluate susceptibility to reliance on experts and take appropriate corrective actions. (Due date: December 15, 2006)

Chemistry will develop a plan that will provide for the proper use and independent review of software generated evaluations, calculations and predictions. (Due date:

January 22, 2007)

V. Conclusion APS takes the matter of the spray pond chemistry controls seriously, and has considered this issue in light of its ongoing improvement plans. Immediate actions have been taken to assure the spray pond systems remain within the design basis.

Additional actions have been taken and others planned to provide greater confidence that similar events will not occur.

21

Enclosure 2 Corrections to NRC Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011 Page 2, first subparagraph, second sentence:

Change from: Train A Change to: Train B Page 14, Enforcement. (1), third sentence:

Change from: Train A Change to: Train B Page 29, third paraaraph, third column:

Change from: 2/04 1 B -17%

Change to: 2/04 1B -18%

Page A3-1, third row, first column:

Change from: 4/95, 1 R4 perf test 56%

and 33.8%

margin at outage beginning and end Change to: 5/95, 1 R5 33.8% margin Page A3-1, third row, second column:

Change from: 4/95, 1R4 perf test 39.8%

margin Change to: 4/95, 1R5 39.8%

margin 1

Page A3-1, fourth row, first column:

Change from: 10/96, 1R6; end of outage perf test 34.3%

margin Change to: 10/96, 1 R6; end of outage perf test 55.7%

margin Page A3-1, fourth row, second column:

Change from: 9/96, 1R6; early outage perf test 55.7%

margin Change to: 9/96, 1 R6; early outage perf test 34.3%

margin Page A3-2, fourth row, fith column:

Change from: 4/03, 3R10; early outage perf test

-22.0%

margin before cleaning, 26.8% after.

All tubes inspected with boroscope.

2

Change to: 4/03, 3R10; early outage perf test

-22.0%

margin before ECT Inspection of all tubes, 26.8% after.

17 tubes inspected with boroscope.

Page A3-3. first row, second column:

Change from: 2/04, midcycle outage perf test -17.1%

margin.

Change to: 2/04, midcycle outage perf test -18%

margin.

Page A3-3, first row, third column:

Change from: 3/04, midcycle outage perf test 28.6%

before cleaning Change to: 3/04, midcycle outage perf test 28.6%

no cleaning 3

Page A3-3. first row, fourth column:

Change from: 3/04, midcycle outage, cleaning performed but no test.

Change to: 3/04, midcycle outage, no cleaning no test.

Page A3-3, first row, sixth column:

Change from: 3/04, midcycle outage; cleaning performed but no test.

Change to: 2/04, midcycle outage; cleaning performed but no test.

Page A3-3. second row, fith column:

Change from: 10/04, 3R11 EWHX 3A cleaned but not tested.

Change to: 11/04, 3R11 EW HX 3A tested at 26.8%

after cleaning.

4

Page A3-3, second row, sixth column:

Change from: 10/04, 3R1 1 34.6%

margin before cleaning, 26.8% after.

Change to: 10/04, 3R1 1 34.6%

margin before cleaning.

Page A3-3, sixth row, first column:

Change from: 6/06, midcycle outage; perf test 28.6%

before cleaning, 36.3% after.

Change to: 6/06, midcycle outage; perf test 28.6%

after cleaning.

Page A3-3, sixth row, second column:

Change from: 3/06, midcycle outage; perf test 32.9%

margin before cleaning.

Change to: 3/06, midcycle outage; perf test 32.9%

margin no cleaning.

5

Page A4-2, sixth paragraph, second column:

Change from: Essential Cooling Water 2B performance measured at -17.1 percent.

Change to: Essential Cooling Water 1B performance measured at -18 percent.

Paae A4-2, tenth paragraph, first column:

Change from: 4/04 Change to: 5/04 6