ML063250343

APS Response to NRC Inspection Report 05000528-06-011, 05000529-06-011, & 05000530-06-011

ML063250343
Person / Time
Site:	Palo Verde
Issue date:	11/14/2006
From:	James M. Levine Arizona Public Service Co
To:	Document Control Desk, NRC Region 4
References
102-05593-JML/SAB/JAP/DFH, EA-06-221, IR-06-011
Download: ML063250343 (29)

See also: IR 05000528/2006011

Text

A-- A subsidiary

of Pinnacle West Capital Corporation

James M. Levine Mail Station 7602 Palo Verde Nuclear Executive

Vice President

Tel (623) 393-5300 PO Box 52034 Generating

Station Generation

Fax (623) 393-6077 Phoenix, Arizona 85072-2034

102-05593-J

ML/SAB/JAP/DFH

November 14, 2006 U.S. Nuclear Regulatory

Commission

ATTN: Document Control Desk Washington, DC 20555 Dear Sir: Subject: Palo Verde Nuclear Generating

Station (PVNGS)Units 1, 2 and 3 Docket Nos. STN 50-528, 50-529, and 50-530 APS Response to NRC Inspection

Report 05000528/2006011;

05000529/2006011;

05000530/2006011;

EA 06-221 In NRC Special Inspection

Report, dated September

28, 2006, the NRC indicated

that APS failed to recognize

that improper chemistry

controls for the spray pond systems in all three units caused degraded performance

over a period of years. The report discusses

a finding that may have a greater than very low safety significance.

APS has reviewed the NRC Inspection

Report and has no substantive

disagreement

with the facts as documented

in the report.In accordance

with the Inspection

Manual Chapter 0609, the NRC is currently evaluating

the safety significance

of the finding. At a November 20, 2006 Regulatory

Conference

in Arlington, Texas, APS will provide the NRC its perspective

on the facts and analytical

assumptions

relevant to determining

the safety significance

of the finding.Regardless

of the ultimate significance

of the finding, APS takes this matter seriously.

We have implemented

immediate

actions to assure the system remains within the design basis, have implemented

or planned additional

actions to prevent recurrence, and have considered

this issue in light of our ongoing improvement

plans.The purpose of this letter is to provide the results of APS' evaluation

of the spray pond issue in advance of the Regulatory

Conference

to assure the NRC that APS recognizes

the programmatic

and organizational

implications

of this matter. APS is providing

this information

to facilitate

a focused discussion

at the conference

on the safety significance

of the spray pond degraded performance.

U.S. Nuclear Regulatory

Commission

ATTN: Document Control Desk APS Response to NRC Inspection

Report 05000528/2006011;

05000529/2006011;

05000530/2006011;

EA 06-221 Page 2 Enclosure

1 to this letter contains a summary of APS' evaluation

and how APS considered

the results in the context of other ongoing corrective

actions. Enclosure

1 also includes an APS response to the five apparent violations

related to the potentially

greater than green finding.Finally, the non-cited

violations

described

in the Inspection

Report are not addressed

in this submittal

because APS agrees with the violations

which have been entered into, and will be resolved in the corrective

action program.APS noted some minor corrections

for the subject Inspection

Report and is submitting

Enclosure

2 for completeness.

The actions described

in Enclosure

1 represent

corrective

actions and are not regulatory

commitments.

There are no regulatory

commitments

in this letter.If you have any questions, please contact James A. Proctor at (623) 393-5730.Sincerely, JML/SAB/JAP/DFH/gt

Enclosures:

1. Summary of APS Investigation

and Corrective

Actions 2. Corrections

to NRC Inspection

Report cc: B. S. Mallett NRC Region IV Regional Administrator

M. B. Fields NRC NRR Project Manager G. G. Warnick NRC Senior Resident Inspector

for PVNGS

ENCLOSURE

1 Summary of APS Investigation

Into Loss of Thermal Performance

of the Essential

Cooling Water (EW)And Emergency

Diesel Generator (EDG) Intercooler

Heat Exchangers, And Corrective

Actions I. Introduction

On March 30, 2006 Arizona Public Service Company (APS) met with the NRC to discuss progress of the comprehensive

integrated

improvement

plan, which addresses performance

issues at the Palo Verde Nuclear Generating

Station (PVNGS). APS outlined the elements of its Performance

Improvement

Plan (PIP), and how, as part of that plan, APS management

is monitoring

performance

and adjusting

its improvement

initiatives, as appropriate.

APS has considered

and incorporated

in its PIP the organizational

issues resulting

from its review of the Recirculation

Actuation

Signal (RAS) Sump Event, the substantive

cross-cutting

issues in Problem Identification

and Resolution (PI&R) and Human Performance (HU), and other indicators

of performance.

A separate, stand-alone

95002 action plan was developed

to address the issues directly related to the YELLOW finding for the RAS.APS recognizes

that its corrective

actions for the 95002 action plan as well as the PIP have not yet resulted in the desired improved performance.

As a result, APS recently set forth to revise the 95002 action plan to include additional

corrective

actions to address the remaining

issues; particularly, improving

the adequacy of technical

rigor, reinforcement

of a questioning

attitude among personnel, and improving

metrics to better measure performance.

APS has similarly

set forth to revise the PIP to include further corrective

actions to address the two substantive

cross-cutting

issues (i.e., PI&R and HU).Recognizing

that the actions that have been taken have not been fully effective, a more rigorous approach has been adopted for developing

the revision to the 95002 action plan and PIP. The approach includes development

of problem statements

and comparing

these to the actions previously

taken. From an understanding

of the shortcomings

of those previous actions, additional

actions to be taken will be identified.

In order to monitor the effectiveness

of the actions, a more rigorous approach will be applied to establishing

the appropriate

metrics. First the intent of the metric will be defined. From this the criteria to be used for populating

the metric will be established

and the data sources will be identified.

The metrics will then be developed

and populated

with the appropriate

data.This enhanced approach has been applied to the 95002 action plan and is being applied to the PIP action plans for the substantive

cross-cutting

issues. These three revised action plans will be submitted

to the NRC by December 15, 2006. This revised approach is also being applied to the three other focus areas of the PIP, namely leadership, accountability

and standards.

1

In Section II of this enclosure, APS sets forth corrective

actions to address the SP root and contributing

causes. However, APS did not limit its considerations

of appropriate

corrective

actions to the SP; rather, APS considered

the broader implications

of the root cause investigation.

Specifically, many of the root and contributing

causes identified

in the spray pond event are substantially

the same as those identified

in the RAS sump event and other self-assessments

APS has conducted.

For example, root cause three and contributing

cause two (personnel

failed to effectively

identify problems, and missed numerous opportunities

to resolve the foulant problem);

contributing

cause six (personnel

did not always exhibit a sufficiently

questioning

attitude);

and contributing

cause seven (personnel

did not apply appropriate

technical

rigor in their evaluations)

reflect the very performance

deficiencies

that the aforementioned

action plans are designed to correct. For this reason, the investigation

relies, in part, on the established

corrective

actions in the revised 95002 and PIP action plans.The purpose of this enclosure

is to summarize

APS' investigation, to describe the corrective

actions, and to summarize

further corrective

actions, particularly

in response to the five apparent violations

related to the NRC's potentially

greater than green finding.This enclosure

does not address APS' position on the significance

of the NRC finding, as that will be the subject of a Regulatory

Conference

between APS and the NRC on November 20, 2006.I.A. Summary of Key Events Leading to Identification

of Possible Degraded Performance

The spray pond (SP) system is the ultimate safety-related

heat sink and is an open water, standby cooling system. The key safety functions

for the SP system are heat removal and maintenance

of vital auxiliaries.

The primary function of the SP system is to remove heat from all essential

components

required for normal and emergency shutdown of the plant. The SP system also provides cooling to the Emergency

Diesel Generators (EDG), which is an essential

function, necessary

for a reliable source of on-site AC electrical

power.The design bases for the SP system assume that the system will function for 26 days without makeup water, losing approximately

85% of its original volume of water to evaporation

and drift.During the mid-1 990s, APS initiated

monitoring

of performance

of the Essential

Cooling Water (EW) heat exchangers

in part due to NRC and industry initiatives, such as Generic Letter (GL) 89-13, "Service Water System Problems Affecting

Safety-Related

Equipment." EW thermal performance

is monitored

and reported as a percent of the design value, which includes the capability

to remove assumed design heat loads, with bounding coolant conditions

and with an industry standard for service fouling. As manufactured, the performance

capability

of the EW heat exchangers

for new conditions

is greater than the design value. A margin of 0% reflects performance

at the 2

design value, including

the service fouled condition.

Positive margin reflects heat exchanger

performance

above the design value. Negative margin does not necessarily

reflect an inability

to perform its safety function.

This is because actual heat removal conditions (e.g., coolant temperatures, environmental, actual heat loads) are typically less severe than the assumed bounding design conditions.

In 1994, zinc-phosphate

treatment

was introduced

into the SP system to mitigate internal piping corrosion

caused by small localized

failures of the internal coating, specifically

in underground

spray pond piping. Leaks detected in EW heat exchanger tubes and inspection

of spray pond piping internals

promoted greater attention

to corrosion

control.In the spring of 1995, there was a release of corrosion

nodules that caused limited blockage of EDG heat exchanger

tubes. APS performed

additional

inspections

to assess the extent of condition.

After the initial release of corrosion

nodules, the amount quickly subsided and has remained within the inspection

acceptance

criteria established

as a result of this event. This condition

was documented

in Licensee Event Report 05000528/1995-005-00.

In the late 1990s, APS first identified

fouling of EDG intercoolers

through trending of diesel intake temperatures.

There were losses of EDG intercooler

performance

and visual inspection

of all EDG heat exchangers

revealed a white film, generally

associated

with the zinc-phosphate

chemical treatment.

The Palo Verde maintenance

team addressed

this issue by mechanical

cleaning initially, and followed this with a chemical cleaning process in 2001. During the time period from 1995 to 2001, APS identified

no visual fouling in the EW heat exchangers.

In 2001, Palo Verde changed chemical vendors to GE-Betz from Calgon. Prior to 2001, the dispersant

used for the essential

spray ponds was Calgon PCL-401. In 2001, GE-Betz provided Palo Verde with a dispersant (HPS-1) that GE-Betz personnel

asserted to be equivalent

to the Calgon product.In the Spring 2002 Unit 2 refueling

outage, APS identified

the first negative thermal performance

margin of any EW heat exchanger.

Data collected

during the refueling outage on the EW B heat exchanger

was analyzed and determined

to be negative 9.5%.In September

2002, a low margin (positive

3.6%) was measured on Unit 1 EW B heat exchanger.

In April 2003 Unit 3 EW A thermal performance

was measured at a negative 22%margin.In September

2003, the Unit 2 EW A heat exchanger

exhibited

a 1% margin, before cleaning.3

In November 2003, APS initiated

corrective

action based on a loss of thermal margin in all 3 units' EW heat exchangers.

APS concluded

that the EW heat exchangers

were fouling due to a reaction between the new dispersant (HPS-1) and the non-oxidizing

biocide used to control algae growth. APS concluded, at that time, that these two chemicals

were reacting to form a polymer, thus fouling the heat exchangers.

During the course of APS' current investigation, the investigation

team determined

that this conclusion

was incorrect.

In March 2004, the Chemistry

department

began using a new dispersant (DN2317), which was acquired from the vendor, GE-Betz. The new dispersant

was used to address the increase in fouling rate that had been concluded

to be associated

with the interaction

of the HPS-1 dispersant

and biocide.Immediately

following

the change in dispersant

the thermal performance

of the EW heat exchangers

improved, further supporting

the incorrect

conclusion

regarding

the previous dispersant.

Because the new dispersant

was not directly measurable, a molybdenum

tracer had been added to the dispersant

to allow detection

and to provide an indirect correlation

to the amount of dispersant

available

in the ponds. However, there was a difference

in the consumption

rate of the dispersant, and the molybdenum.

Over time, the molybdenum

tracer provided a false indication

that sufficient

levels of dispersant

were present in the ponds. Consequently, no dispersant

was added for an extended period of time.As a result, in October of 2005, when Unit 1 was in a refueling

outage and Unit 2 and 3 were in forced shutdowns, a loss in thermal performance

margin was identified

indicating

that the fouling rate had increased.

Unit 1 and 3 EW heat exchangers

were cleaned during their refueling

outages (Fall 2005 for Unit 1 and Spring 2006 for Unit 3).When it was projected

that the rate of fouling would not support Unit 2 EW function during the summer of 2006, both Unit 2 EW heat exchangers

were cleaned during online equipment

maintenance

outages.I.B. APS Identification

of Possible Degraded Performance

and Initiation

of an Investigation

Team On May 17, 2006, APS identified

that the EDG 2B intake air temperature

was higher than the maximum limits specified

in the data collection

logs. APS declared the EDG inoperable

on May 19, 2006 and inspected

and cleaned the intercooler.

At that time, APS took immediate

corrective

actions to clean the intercoolers

on all six EDGs and inspect the EDG Jacket Water (JW) and Lube Oil (LO) heat exchangers

in Unit 1. Unit 1 EW heat exchangers

were also re-cleaned

prior to the summer.As a result of these issues, APS initiated

an investigation

team that consisted

of individuals

from the Chemistry

Department (Palo Verde and APS Corporate

Chemistry), Engineering

Department, Performance

Improvement

Team members with plant operations

experience, and industry consultants.

The scope of the investigation

included a review of all heat exchangers

that are cooled by spray pond water and 4

addressed

the lack of dispersant

control for the SP from approximately

March of 2004 until June of 2005. The investigation

team evaluated

both past and current plant documents

including

sampling results, reviewed industry information, obtained independent

analysis of foulant samples and conducted

limited inspections

of the system to support their investigation

and to draw their conclusions.

The next section summarizes

the results of the investigation

team, the causal analysis results and APS' corrective

actions. Subsequent

to this effort, APS has reviewed, along with EPRI and other experts, the safety significance

of these issues. As indicated

in the cover letter, APS will provide its perspective

on the safety significance

of this matter during the November 20, 2006 Regulatory

Conference.

II. Results, Causal Analysis and Corrective

Actions II.A. Direct Cause and Extent of Condition II.A.1 Direct Cause The direct cause of the loss of thermal performance

from the EDG intercooler

and EW heat exchangers

was the formation

of an insulating

precipitant

on the SP side of the heat exchanger

surfaces.This precipitant, calcium-zinc-phosphate, formed when the chemical constituents

were present in sufficient

concentrations

and the control factors (i.e., pH and dispersant)

were not adequately

controlled.

II.A.2 Extent of Condition APS took a broad perspective

in considering

the extent of condition

and assessed the plant heat transfer systems, beginning

with components

of the essential

spray system.Three types of foulant were identified

during the investigation:

zinc-hydroxide

film, scale (calcium-phosphate

or calcium carbonate)

and precipitant (calcium-zinc-phosphate).

Other Spray Pond Heat Exchangers

In addition to the EDG intercoolers

and the EW heat exchangers, three other heat exchangers

on the EDGs are cooled by the SP system.EDG Jacket Water and Lube Oil Coolers The EDG JW and LO heat exchangers

were less susceptible

to fouling than the EDG intercooler

or EW heat exchanger.

It was not possible to quantify the amount of thermal performance

loss in the EDG JW and LO coolers by calculational

methods; but, it was determined

there was ample margin to respond to a DBA and cope with minor degrees of fouling based on the following:

5

• Less precipitant

existed in the EDG JW and LO heat exchangers

than the EDG intercooler

and EW heat exchangers.

• Original tube plugging design margin remains intact.* Performance

monitoring

of JW and LO parameters

provides assurance

that the thermostatic

control valves, which regulate temperature, are functioning

within their setpoint band.* Assuming full load and peak Design Basis Accident (DBA) spray pond temperatures, should any thermostatic

valve reach its full open position, there is additional

margin beyond the limit of the thermostatic

valve to the manufacturer's

operating

limit." Loads will be less than the design values when peak SP temperature

occurs.EDG Fuel Oil Coolers The fuel oil heat exchanger

was supplied as original equipment

on the EDG. It is a carry-over

from earlier diesel models, with an original purpose to support extended idle, no-load, operation.

The fuel oil piping system recirculates

excess fuel oil to the suction of the fuel oil transfer pump, which heats the fuel oil by pump heat. The fuel oil cooler was intended to prevent fuel oil from overheating

during extended no-load operation.

The fuel consumption

of a KSV-20 at no-load is sufficient

to preclude the need for cooling; therefore, EDG safety functions

would not have been impacted by a loss of thermal performance

in the EDG Fuel Oil cooler.Potential

for SP Foulant to Manifest as Scale Another immediate

concern was that with the presence of the constituents

in sufficient

amounts to form a precipitant

there was a possibility

that the chemical constituents

may also form scale. Scale has a crystalline

structure, tenaciously

adheres to surfaces, and has an increased

likelihood

of forming with rising temperature, because scale forming constituents

become more insoluble.

The corrective

actions to establish

correct chemistry

control in the spray ponds address the precipitant

and also resolve the propensity

to form scale. Additionally, an analysis was performed

by the Electric Power Research Institute (EPRI) on APS'behalf to assess the possibility

and extent of scale formation

in light of the concentration

of chemical constituents

during the period in question.

The results of this analysis predict a minor amount of scale formation

under DBA conditions.

The results of the aggregate

impacts of the precipitant

and scaling propensity

will be presented

at the November 20, 2006 Regulatory

Conference.

Applicability

to Other Open Cooling Water Systems The investigation

team also evaluated

the susceptibility

of other open cooling water systems to fouling problems.6

The SP fouling problem was viewed as being directly applicable

to the Circulating

Water (CW) System, which is also an open cooling water system. Unlike the SP system, the CW system is a continuously

operating

system and does not fulfill a safety function, but is relied upon for power operations.

It was concluded

there was no immediate

impact to CW but to ensure lessons learned from this event are considered

by the Water Reclamation

Facility (WRF) personnel

who maintain CW chemistry, the results of the evaluation

will be reviewed with WRF personnel

for lessons learned and technical

issues.Applicability

to Heat Exchangers

Addressed

by NRC Generic Letter 89-13 The EW and EDG heat exchangers

are within the scope of equipment

addressed

by GL 89-13. In view of the issues identified

with the spray pond it was concluded

an assessment

of GL 89-13 implementation

should be performed.

See corrective

actions to be taken in Section II.A.3 below.II.A.3. Corrective

Actions -Direct Cause and Extent of Condition Corrective

Actions Taken and Results Achieved The SP system chemistry

was corrected

as follows:* Dispersant

additions

were resumed in all spray ponds." Additional

acid was added to lower pH in all ponds." Feed and bleed was used to reduce calcium and phosphate

concentrations

in each spray pond.EW heat exchangers

were cleaned in all three units starting with the Unit 3 heat exchangers

in the spring 2006 refueling

outage.EDG intercoolers

were cleaned in all three units and a Standing Order was issued to increase the EDG test frequency

to assess intercooler

performance.

This standing order continued

through September

29, 2006. It was removed after APS gained confidence

that the immediate

corrective

actions were effective.

Procedure

40DP-9OP08, "Diesel Generator

Test Records," was revised to require a work order to be generated

to clean any EDG intercooler

if temperature

exceeds 120F, to ensure the system does not exceed its Design Basis Manual (DBM) limit of 130F.The SPs have been cleaned, facilitating

improved chemistry

control and accessibility

for inspections.

Additional

cleaning and inspection

activities

of the EDG and EW heat exchangers

have been scheduled

to ensure the effectiveness

of the completed

corrective

actions and to ensure there are no unintended

consequences

of the chemistry

changes.7

Corrective

Actions to Be Taken Modifications

will be installed

to add higher capacity acid and dispersant

pumps. (Due date: December 31, 2006).Systems Engineering

will conduct a self-assessment

of the SP systems to include NRC GL 89-13 and industry operating

experience

on service water systems. (Due date: February 28, 2007)II.B Five Root Causes Identified

The investigation

identified

the following

five Root Causes: " Root Cause #1: Inadequate

Chemistry

Control Program" Root Cause #2: Managed to Inadequate

Chemistry

Metrics" Root Cause #3: Inadequate

Resolution

of Performance

Problems* Root Cause #4: Management

Reliance on an Expert" Root Cause #5: Ineffective

Change Management

This section summarizes

each root cause and the associated

corrective

actions.II.B.1 Root Cause #1 and Corrective

Actions -Inadequate

Chemistry

Control Program The Palo Verde chemistry

control program for the SP system was ineffective.

The program used control ranges that were inadequate

for some parameters

and lacked other necessary

elements.Corrective

Actions and Results Achieved Procedure

74DP-9CY04, "System Chemistry

Specifications," was revised to establish control limits, sampling frequencies

and action levels that ensure the spray ponds are able to perform their specified

function in a DBA.The strategic

plan for SP Chemistry

Control was revised to include chemistry

control requirements

for fouling, corrosion

control, and biological

control.The dispersant

was changed to one that is directly measurable.

The Chemistry

Department

was trained in the direct, root and contributing

causes of the investigation.

Extent of Cause Corrective

Actions An assessment

of the chemistry

controls for closed cooling water systems is in progress. (Due date: November 30, 2006)8

A Chemistry

Design Basis Manual will be developed. (Due date: December 24, 2007)II.B.2 Root Cause #2 and Corrective

Actions -Managed to Inadequate

Chemistry Metrics A single metric, corrosion

rate, existed for the SP system, which had the unintended

affect of causing personnel

to make decisions

based on this metric and misunderstand

the health of the SP chemistry.

There was no metric for fouling.Corrective

Actions Taken and Results Achieved The Chemistry

Department

revised the performance

indicators

for the SP Chemistry Control Program to ensure measurable

targets are established

for corrosion, fouling, and biological

control.Corrective

Actions to Be Taken Training will be provided to the Chemistry

Technical

Advisors, their backups, and Chemistry

Supervision

on the indications

of fouling mechanisms

associated

with the chemicals

used in the SP system. The training will also include changes to the Chemistry

Management

Expectations

on Qualification, Validation

and Verification (QV&V) and technical

rigor. (Due date: November 21, 2006)The System Health Report for the SP system will be updated in the engineering

program database to reflect the metrics established

by Chemistry. (Due date: January 15, 2007)Extent of Cause Corrective

Actions The site's metric guideline

was revised to identify that a thorough review of a proposed metric is necessary

to identify and measure necessary

objectives

so that unintended

consequences

are minimized (e.g., single metric such as reliability

without a complementary

metric of availability).

An effectiveness

review of site and department

metrics will be conducted

to determine

if appropriate

metrics have been established

and are being used by leaders to guide actions. (Due date: March 30, 2007)Other Chemistry

Department

metrics will be reviewed to ensure complementary

metrics exist where appropriate. (Due date: April 30, 2007)9

ll.B.3 Root Cause #3 and Corrective

Actions -Inadequate

Resolution

of Performance

Problems Palo Verde personnel (leaders and frontline)

failed to effectively

resolve previously

identified

problems with a zinc-related

foulant despite numerous opportunities

to do so.One specific example of inadequate

resolution

of performance

problems was cited as an apparent violation

of 10 CFR 50 Appendix B, Criterion

XVI, "Corrective

Actions", for the failure to correct the degraded performance

of EW 2B, when identified

in March 2002.Corrective

Actions Taken and Results Achieved Procedure

74DP-OCY01, "Specifications

for Bulk Chemicals," was revised to require a chemical compatibility

test when changing chemicals.

Tools have been developed

to improve engineering

work product quality, including

the Engineering

Department

Guides (EDG) 01 and 02 to minimize human performance

errors.The Condition

Reporting

procedure

was revised to include checklists

for processing

CRDRs to improve resolution

of conditions

that are adverse to quality.Management

Expectations

for questioning

attitude and technical

rigor have been incorporated

into Chemistry

Department

Policy (CDP-01).Corrective

Actions to Be Taken Procedure

70TI-9EW01, "Thermal Performance

Testing of Essential

Cooling Water Heat Exchangers", will be revised to improve heat exchanger

performance

management. (Due date: January 31, 2007)The Systems Engineering

Department

performance

monitoring

database will be revised to include a parameter

for the differential

temperature

between the EDG intake air manifold and the SP cooling water to detect degradation

of the intercoolers. (Due date: February 28, 2007)Extent of Cause Corrective

Action This cause is illustrative

of the issues being addressed

by the revised PIP action plan for PI&R. The investigation

compared the issues for the SP event to the action being taken in the revised PI&R action plan and concluded

that plan will appropriately

address the extent of cause. As noted in Section I. Introduction, the revised PIP action plan for the PI&R cross-cutting

issue will be submitted

to the NRC by December 15, 2006.10

ll.B.4 Root Cause #4 and Corrective

Actions -Management

Reliance on an Expert There were instances

where the chemistry

advisor's

working assumptions

were flawed and the lack of technical

support from other technical

specialists

resulted in missed opportunities

to preclude the precipitant

from occurring.

Corrective

Actions Taken and Results Achieved Chemistry

Standards

and Expectations

have been revised to require independent

review of calculations

that relate to changes to chemistry

design controls or the Chemistry

Design Basis Manual.Corrective

Actions to Be Taken Procedure

74DP-9CY04, "System Chemistry

Specifications" will be revised to require a technically

independent

review when making changes to chemistry

control regimes.(Due date: November 17, 2006)The Chemistry

Department

will designate

backups to selected advisors in the chemistry department, will develop training plans and identify critical attributes

for these backups.(Due date: November 30, 2006).A training program specific to chemistry

will be developed

and implemented, using the Engineering

Training Program Description (TPD) as a reference.

Critical job skills and functions

of Chemistry

Technical

Support staff and Instrument

Maintenance

staff will be identified

and a structure

for additional

development

of current staff and for development

of replacement

personnel

will be included. (Due date: April 30, 2007)Extent of Cause Corrective

Actions Each department

will evaluate susceptibility

to reliance on experts and take appropriate

corrective

actions. (Due date: December 15, 2006)ll.B.5 Root Cause #5 and Corrective

Actions -Ineffective

Change Management

APS did not adequately

assess the significance

or impacts of changes made to the Chemistry

Control Program or activities

that impact the program.Corrective

Actions Taken and Results Achieved Procedure

93DP-OLC17, "10 CFR 50.59 and 72.48 Guidance Manual," was revised to require a 50.59 screening

when changing chemicals

and corresponding

changes were made to Chemistry

procedure

74DP-OCY01, "Specifications

for Bulk Chemicals." The commitments

for Generic Letter 89-13 were verified to be accurately

reflected

in the Regulatory

Commitment

Tracking System.11

Ineffective

Change Management

has been captured in the Performance

Improvement

Plan and led to the development

of the "RAPID" (Recognize, Assess, Plan, Implement, and Drive) program to help manage the change process at Palo Verde.Corrective

Actions to Be Taken Systems Engineering

will conduct a self-assessment

of the SP systems to include NRC GL 89-13 and industry operating

experience

on service water systems. (Due date: February 28, 2007)Recognizing

similarities

to the issues identified

in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation

will be performed

to ensure outstanding

issues are appropriately

identified, prioritized

and are being properly addressed. (Due date: March 1, 2007)II.C Contributing

Causes The investigation

identified

the following

nine contributing

causes: " Contributing

Cause #1: Palo Verde did not solicit outside assistance

• Contributing

Cause #2: Inadequate

Prioritization

of the SP System* Contributing

Cause #3: Narrow Focus* Contributing

Cause #4: Emphasis on Cost Control* Contributing

Cause #5: Erosion of Design Margin" Contributing

Cause #6: Questioning

Attitude of Workers" Contributing

Cause #7: Lack of Rigor in Evaluations" Contributing

Cause #8: Living with Problems* Contributing

Cause #9: Removing UFSAR Requirements

This section summarizes

each contributing

cause and the associated

corrective

actions.I1.C.1 Palo Verde Did Not Solicit Outside Assistance

Palo Verde personnel

did not effectively

compare their chemical control program for the SP system with other plants because personnel

considered

the ultimate heat sink for Palo Verde to be unique.Corrective

Actions Taken and Results Achieved Chemistry

Department

Policy (CDP-01) was revised to incorporate

a requirement

for Corporate

Chemistry

or an outside individual

to conduct a review of the SP system Chemical Control Program on a semi-annual

basis.12

Palo Verde's Nuclear Assurance

Department

revised the master assessment

plan to require procedures

selected for reviews during audits to include a full basis review of the entire procedure

against the applicable

licensing

and design basis.II.C.2 Inadequate

Prioritization

of the SP System Palo Verde management

did not effectively

prioritize

the SP system problems to ensure they were resolved in a timely manner.Corrective

Actions to Be Taken Modifications

will be installed

to add higher capacity acid and dispersant

pumps. (Due date: December 31, 2006).The Tower Blowdown (TB) makeup line will be restored to Unit 1 spray ponds. (Due date: March 30, 2007)The temporary

modification

for zinc and biocide chemical addition will be replaced by a modification

in all three units. (Due date: September

30, 2008)The feasibility

of installing

a SP filter system will be performed. (Due date: June 30, 2007)II.C.3 Narrow Focus Palo Verde personnel

occasionally

displayed

a narrow focus to problem solving, which resulted in missed opportunities

to address the cause, rather than the symptoms.

In some cases, the corrective

actions that were put in place to solve one symptom resulted in the manifestation

of new problems.Corrective

Actions Taken and Results Achieved Procedure

74DP-9CY04, "System Chemistry

Specifications," was revised to establish control limits, sampling frequencies

and action levels that ensure the spray ponds are able to perform their specified

function in a DBA.Implemented

a program to walk down safety-significant

systems weekly on a 12 week schedule.

This walk down is performed

by Operations

and Engineering

personnel.

Corrective

Actions to Be Taken This cause is illustrative

of the issues being addressed

by the revised 95002 and PIP action plans. The investigation

compared the issues for the SP event to the action being taken in those action plans and concluded

they will appropriately

address this contributing

cause.13

An engineering

project, referred to as the Component

Design Basis Review (CDBR)began in October of 2006. The purpose of the review is to verify the design bases have been correctly

implemented

for high risk-significant

components

and operator actions.The review is estimated

to take 24-48 months to complete for approximately

250 components

and operator actions.II.C.4 Emphasis on Cost Control Palo Verde personnel, management

and frontline, have occasionally

made inappropriate

cost basis decisions

that result in inaction or delay for equipment

with perceived

lower safety significance.

Corrective

Actions Taken and Results Achieved The closed cooling water chemical addition systems were evaluated

and it was verified they are adequate to control chemistry.

Corrective

Actions to Be Taken Recognizing

similarities

to the issues identified

in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation

will be performed

to ensure outstanding

issues are appropriately

identified, prioritized

and are being properly addressed. (Due date: March 1, 2007)II.C.5 Erosion of Design Margin Personnel

used various corrective

action programs and processes

to accept progressively

greater amounts of degradation, eventually

changing the organization's

perception

of normal.Corrective

Actions to Be Taken Modifications

will be installed

to add higher capacity acid and dispersant

pumps. (Due date: December 31, 2006).The TB makeup line will be restored to Unit 1 spray ponds. (Due date: March 30, 2007)A Chemistry

Design Basis Manual will be developed. (Due date: December 24, 2007)Recognizing

similarities

to the issues identified

in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation

will be performed

to ensure outstanding

issues are appropriately

identified, prioritized

and are being properly addressed. (Due date: March 1, 2007)14

I1.C.6 Questioning

Attitude of Workers Palo Verde personnel

did not always question what they saw, which contributed

to the misunderstanding

that ultimately

impacted the SP heat exchanger

performance.

Corrective

Actions to Be Taken Develop and implement

site-wide

training on a case study, based on the SP investigation, emphasizing

lessons learned from this event. (Due date: October 1, 2007)This cause is illustrative

of the issues being addressed

by the revised 95002 action plan. The investigation

compared the issues for the SP event to the actions, being taken in this action plan and concluded

it will appropriately

address this contributing

cause.I1.C.7 Lack of Rigor in Evaluations

Palo Verde personnel

did not always apply an appropriate

amount of technical

rigor in evaluations

related to the thermal performance

of the SP heat exchangers.

Corrective

Actions to Be Taken This cause is illustrative

of the issues being addressed

by the revised 95002 action plan. The investigation

compared the issues for the SP event to the actions being taken in this action plan and concluded

it will appropriately

address this contributing

cause.II.C.8 Living with Problems Palo Verde personnel

have, in some cases, accepted and worked around problems.Corrective

Actions to Be Taken Recognizing

similarities

to the issues identified

in the Davis Besse SOER 02-4, a review of the SOER and Palo Verde's evaluation

will be performed

to ensure outstanding

issues are appropriately

identified, prioritized

and are being properly addressed. (Due date: March 1, 2007)This cause is illustrative

of the issues being addressed

by the revised 95002 and PIP action plans. The investigation

compared the issues for the SP event to the action being taken in those action plans and concluded

they will appropriately

address this contributing

cause.II.C.9 Removing UFSAR Requirements

Personnel

removed information

from the UFSAR that may have helped preclude this event.15

Corrective

Actions to Be Taken Revise the UFSAR to include spray pond chemical addition equipment

necessary

to maintain the design basis. (Due date: March 30, 2007)Revise the UFSAR to incorporate

the TB make up line modification

to Unit 1. (Due date: March 30, 2007)III. Review of Latent Design Issues APS recognizes

both the RAS event as well as the SP event revealed significant, latent design basis issues. As a result, APS will address the possibility

of other latent design issues that remain unidentified.

The cornerstone

of the effort to identify these issues is the component

design basis review (CDBR) which APS committed

to perform in a letter to the NRC dated June 2, 2006. The purpose of the review will be to verify the design bases have been correctly

implemented

for highly risk-significant

components

and operator actions as defined in the June 2, 2006 letter. The review will verify the capability

of these components

to perform their intended safety functions

and will use the inspection

methodologies

detailed in NRC Inspection

Procedure (IP) 71111.21, dated December 2, 2005.APS will implement

this component

design basis review in two phases. The first phase will include the highly risk-significant

components

and operator actions that are in the Mitigating

Systems Performance

Indicator

systems. It is expected that at least 20 of these components

and operator actions will be complete by the end of the first quarter 2007. The second phase will include the remaining

highly risk-significant

components

and operator actions.In addition to the CDBR, APS will initiate a Plant Health Committee (PHC) with the distinct purpose of providing

a high level management

overview of current and latent issues with the potential

to impact plant operations.

The PHC will focus senior management

representatives

from Operations, Engineering, and Maintenance

on issues that need management

attention

and resolution, including:

weekly system health reviews, corrective

actions status, single point vulnerabilities

and resolutions, prioritization

of plant modifications, and station program health and implementation

reviews.The Plant Health Committee

represents

a function that is much more comprehensive

in management

oversight

and intrusiveness

into plant performance

than previous System Team Steering Committee

efforts. The details of the PHC will be discussed

further at the November 20, 2006 regulatory

conference.

16

IV. Response to Apparent Violations

This section sets forth APS' position on the five apparent violations

and summarizes

corrective

actions taken or planned that are directly related to the apparent violations.

IV.A. Apparent Violation

of Technical

Specification

3.7.7 Restatement

of Apparent Violation An apparent violation

of Technical

Specification

3.7.7 was identified

because Train B of the Essential

Cooling Water System in Unit 2 was not capable of performing

its safety function for approximately

6.8 months ending on September

27, 2003.Admission APS' significance

determination

has concluded

there was no violation

of TS 3.7.7. The details leading to this conclusion

will be presented

by APS during the November 20, 2006, Regulatory

Conference.

Specifically, our significance

determination

demonstrates

that the system would have been able to perform its safety function due to the large margin in the EW heat exchanger

design in spite of the degraded conditions.

Corrective

Actions The corrective

actions delineated

above in Section II of this Enclosure

address the degraded conditions

of the spray pond and have returned it to conformance

with its design basis.IV.B. Apparent Violation

of Criterion

Xl, Test Control Restatement

of Apparent Violation An apparent violation

of 10 CFR Part 50, Appendix B, Criterion

X1, "Test Control," was identified

because the two procedures

that were performed

to measure essential cooling water heat exchanger

performance

were implemented

in a way that was inadequate

to ensure the timely determination

that the requirements

and acceptance

limits contained

in applicable

design documents

were met.Admission APS admits this apparent violation.

Cause The potential

violation

has been entered into the Palo Verde Corrective

Action Program and the apparent cause evaluation

is currently

in progress.17

Corrective

Actions Taken and Results Achieved Since September

2002, an evaluation

of the EW heat exchanger

thermal performance

test data has occurred during the outage in which the data was collected.

Corrective

Actions to Be Taken Procedure

70TI-9EW01, "Thermal Performance

Testing of the Essential

Cooling Water Heat Exchangers," will be revised to require heat exchanger

performance

data analysis to be completed

prior to entry into Mode 4. (Due date: November 15, 2006)VI.C. Apparent Violation

of 10 CFR 50.59 Restatement

of Apparent Violation An apparent violation

of 10 CFR 50.59 was identified

for making nine revisions

to Procedure

74DP-9CY04, "System Chemistry

Specification," a procedure

described

in the Updated Final Safety Analysis Report, between 1998 and 2004 without performing

evaluations

of the potential

impact of the changes on the safety-related

components

in the spray pond system; the changes revised spray pond chemistry

parameter

limits which were subsequently

determined

to have contributed

to heat exchanger

fouling.Admission APS admits this apparent violation.

Cause APS determined

that the causes of the apparent violation

of 10 CFR 50.59 were as follows:* Changes to chemistry

limits were viewed as administrative

and related to maintenance." There was a failure to rigorously

evaluate the changes to the spray pond chemistry

control program Corrective

Actions Taken and Results Achieved Chemistry

procedure

74DP-9CY04

was restored to the correct condition

and 10 CFR 50.59 was applied to these changes to implement

the current chemistry

regime of the SP System.The 10 CFR 50.59 guidance manual (93DP-OLC17)

was revised to provide specific requirements

for chemistry

related changes.18

Corrective

Actions to Be Taken In order to determine

if other chemistry

procedures

were changed without the proper 10 CFR 50.59 evaluations, a random sample of 58 chemistry

procedure

changes are being reviewed. (Due date: November 17, 2006)Chemistry

personnel

will be trained on the changes to the 10 CFR 50.59 guidance manual. (Due date: November 17, 2006)In addition to a' review of chemistry

related procedure

changes, a review of the performance

monitoring

results of the 10 CFR 50.59 performance

review team from the past year will be performed

to determine

if there are any adverse trends or areas needing additional

review. (Due date: December 30, 2006)IV.D. Apparent Violation

of Criterion

XVI, Corrective

Actions Restatement

of Apparent Violation An apparent violation

of 10 CFR Part 50, Appendix B, Criterion

XVI, "Corrective

Actions," was identified.

On March 19, 2002, performance

testing for Essential

Cooling Water Heat Exchanger

2B indicated

that the system would not be capable of performing

its design function, but this significant

condition

adverse to quality was not promptly identified, the cause determined, or corrective

actions taken to restore the required heat exchanger

performance.

The failure to correct this degraded performance

contributed

to the continued

degradation

and eventual loss of function for an estimated

period of 6.8 months.Admission APS admits this apparent violation

with the following

clarification.

As stated in Section IV.A, APS concluded

that the EW B heat exchanger

in Unit 2 would have been able to perform its function during the period in question.

APS admits that actions were not taken to promptly identify, accurately

determine

the cause and to establish

effective corrective

actions to prevent thermal performance

degradation.

Cause The evaluation

of the EW 2B condition

by engineering

personnel

lacked technical

rigor.Correction

Actions The corrective

actions for this violation

are provided in Section ll.B.3 of this enclosure.

19

IV.E. Apparent Violation

of Criterion

III, Design Control Restatement

of Apparent Violation An apparent violation

of 10 CFR Part 50, Appendix B, Criterion

Ill, "Design Control," was identified

for failure to correctly

evaluate the scaling potential

of the safety-related

heat exchangers

cooled by the emergency

spray pond during a design basis accident.An error in the EPRI SEQUIL calculation

caused the licensee to incorrectly

conclude that scaling would not occur under the conditions

established

in the chemistry

control program.Admission APS admits this apparent violation.

Cause Incorrect

use of a computer model, erroneously

predicted

that calcium phosphate

scale would not form under the input conditions.

Chemistry

personnel

selected input parameters

to simulate conditions

for evaluation.

The individual

performing

the modeling was unfamiliar

with the software and while inputting

data into the model and selecting

the parameters

for the calculation, failed to set the super-saturation

parameter correctly.

This process relied on a single individual

to perform the computer modeling predictions

with no independent

reviews or peer checks.Corrective

Actions Taken and Results Achieved Utilizing

the correct input parameters

and settings Chemistry

reran the EPRI SEQUIL calculation.

The revised data indicated

that there was a high potential

for calcium phosphate

scaling.Chemistry

Standards

and Expectations

have been revised to require independent

review of calculations

that relate to changes to chemistry

design controls.

This requirement

will also apply to the Chemistry

Design Basis Manual, once developed.

Management

expectations

for questioning

attitude and technical

rigor have been incorporated

into Chemistry

Department

Policy (CDP-01).Corrective

Actions to Be Taken Procedure

74DP-9CY04, "System Chemistry

Specifications" will be revised to require a technically

independent

review when making changes to chemistry

control regimes.(Due date: November 17, 2006)20

Chemistry

Department

will designate

backups to the selected advisors in the chemistry department, will develop training plans and identify critical attributes

for these backups.(Due date: November 30, 2006).A training program specific to chemistry

will be developed

and implemented, using the Engineering

Training Program Description (TPD) as a reference.

Critical job skills and functions

of Chemistry

Technical

Support staff and Instrument

Maintenance

staff will be identified

and a structure

for additional

development

of current staff and for development

of replacement

personnel

will be included. (Due date: April 30, 2007)Each department

will evaluate susceptibility

to reliance on experts and take appropriate

corrective

actions. (Due date: December 15, 2006)Chemistry

will develop a plan that will provide for the proper use and independent

review of software generated

evaluations, calculations

and predictions. (Due date: January 22, 2007)V. Conclusion

APS takes the matter of the spray pond chemistry

controls seriously, and has considered

this issue in light of its ongoing improvement

plans. Immediate

actions have been taken to assure the spray pond systems remain within the design basis.Additional

actions have been taken and others planned to provide greater confidence

that similar events will not occur.21

Enclosure

2 Corrections

to NRC Inspection

Report 05000528/2006011;

05000529/2006011;

05000530/2006011

Page 2, first subparagraph, second sentence: Change from: Train A Change to: Train B Page 14, Enforcement.

(1), third sentence: Change from: Train A Change to: Train B Page 29, third paraaraph, third column: Change from: 2/04 1 B -17%Change to: 2/04 1 B -18%Page A3-1, third row, first column: Change from: 4/95, 1 R4 perf test 56%and 33.8%margin at outage beginning and end Change to: 5/95, 1 R5 33.8% margin Page A3-1, third row, second column: Change from: 4/95, 1R4 perf test 39.8%margin Change to: 4/95, 1 R5 39.8%margin 1

Page A3-1, fourth row, first column: Change from: 10/96, 1R6;end of outage perf test 34.3%margin Change to: 10/96, 1 R6;end of outage perf test 55.7%margin Page A3-1, fourth row, second column: Change from: 9/96, 1 R6;early outage perf test 55.7%margin Change to: 9/96, 1 R6;early outage perf test 34.3%margin Page A3-2, fourth row, fith column: Change from: 4/03, 3R10;early outage perf test-22.0%margin before cleaning, 26.8% after.All tubes inspected with boroscope.

2

Change to: 4/03, 3R10;early outage perf test-22.0%margin before ECT Inspection

of all tubes, 26.8% after.17 tubes inspected with boroscope.

Page A3-3. first row, second column: Change from: 2/04, midcycle outage perf test -17.1%margin.Change to: 2/04, midcycle outage perf test -18%margin.Page A3-3, first row, third column: Change from: 3/04, midcycle outage perf test 28.6%before cleaning Change to: 3/04, midcycle outage perf test 28.6%no cleaning 3

Page A3-3. first row, fourth column: Change from: 3/04, midcycle outage, cleaning performed but no test.Change to: 3/04, midcycle outage, no cleaning no test.Page A3-3, first row, sixth column: Change from: 3/04, midcycle outage;cleaning performed but no test.Change to: 2/04, midcycle outage;cleaning performed but no test.Page A3-3. second row, fith column: Change from: 10/04, 3R11 EWHX 3A cleaned but not tested.Change to: 11/04, 3R11 EW HX 3A tested at 26.8%after cleaning.4

Page A3-3, second row, sixth column: Change from: 10/04, 3R1 1 34.6%margin before cleaning, 26.8% after.Change to: 10/04, 3R1 1 34.6%margin before cleaning.Page A3-3, sixth row, first column: Change from: 6/06, midcycle outage; perf test 28.6%before cleaning, 36.3% after.Change to: 6/06, midcycle outage; perf test 28.6%after cleaning.Page A3-3, sixth row, second column: Change from: 3/06, midcycle outage; perf test 32.9%margin before cleaning.Change to: 3/06, midcycle outage; perf test 32.9%margin no cleaning.5

Page A4-2, sixth paragraph, second column: Change from: Essential

Cooling Water 2B performance

measured at -17.1 percent.Change to: Essential

Cooling Water 1 B performance

measured at -18 percent.Paae A4-2, tenth paragraph, first column: Change from: 4/04 Change to: 5/04 6