IR 05000528/2006011

IR 05000528-06-011, 05000529-06-011 & 05000530-06-011; on 06/6-9/26/2006; Arizona Public Service Company; Palo Verde Nuclear Plant; Special Inspection, 50.59, Operability Evaluations, Design Control, Corrective Actions, Biennial Heat Sink,

ML062710607
Person / Time
Site:	Palo Verde
Issue date:	09/28/2006
From:	Chamberlain D Division of Reactor Safety IV
To:	James M. Levine Arizona Public Service Co
References
EA-06-221 IR-06-011
Download: ML062710607 (62)

Text

ber 28, 2006

SUBJECT:

Palo Verde Nuclear Generating Station, Units 1, 2, and 3 - NRC SPECIAL INSPECTION REPORT 05000528/2006011; 05000529/2006011; 05000530/2006011

Dear Mr. Levine:

On September 26, 2006, the US Nuclear Regulatory Commission (NRC) completed a special inspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. This inspection examined your staffs response to the high intake air temperature condition identified in Emergency Diesel Generator 2B on May 17, 2006, and the fouling of safety-related heat exchangers serviced by the emergency spray pond system. The NRC's initial evaluation met both the risk criteria and several of the deterministic criteria for initiating a special inspection.

The basis for initiating this special inspection is further discussed in the Charter, which is included as Attachment 2 to this report.

The enclosed special inspection report documents the inspection findings which were discussed on August 18, 2006, with Mr. C. Eubanks, Vice President, Nuclear Operations, Mr. D. Mauldin, Vice President, Engineering, and other members of your staff. A supplemental exit was conducted on September 26, 2006, with Mr. and other members of your staff to discuss the preliminary significance determination and the characterization of the performance deficiency. The determination that the inspection would be conducted was made by the NRC on June 6, 2006, and the inspection started on June 12, 2006.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

The report documents two findings that were determined to be violations of very low safety significance. Also, a licensee-identified violation which was determined to be of very low safety significance is listed in this report. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If

Arizona Public Service Company -2-you contest any noncited violation in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U. S. Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington, DC 20555-0001; and the NRC Resident Inspector at Palo Verde Nuclear Generating Station.

In addition, the attached report discusses a finding that appears to have greater than very low safety significance. As described in Section 5.1 of this report, the failure to recognize that improperly implemented chemistry controls for the emergency spray pond systems in all three units caused degraded performance in all emergency diesel generators and emergency cooling water systems over a period of years. This finding was assessed based on the best available information, including influential assumptions, using the applicable Significance Determination Process (SDP) and was preliminarily determined to be a Greater Than Green Finding. The final resolution of this finding will convey the increment in the importance to safety by assigning the corresponding color i.e., [(white) a finding with some increased importance to safety, which may require additional NRC inspection; (yellow) a finding with substantial importance to safety that will result in additional NRC inspection and potentially other NRC action; (red) a finding of high importance to safety that will result in increased NRC inspection and other NRC action]. This finding has a preliminary greater than very low safety significance because it has been estimated that in Unit 2, Train B of essential cooling water was incapable of performing its safety function for 6.8 months in 2003, based on extrapolation of performance data for this heat exchanger. Accident sequences that would cause high pond temperatures, primarily loss-of-coolant accidents, could challenge this system to the point where the essential chiller would fail, causing a loss of room cooling to important mitigating systems. In addition, there is uncertainty associated with the amount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of an accident scenario.

Due to the complexity of this issue, there was insufficient information for a number of important aspects to complete a final Phase 3 significance determination evaluation in a timely manner.

During the inspection process, it was apparent that your staff is working to develop the information and calculations needed to complete an accurate assessment. Therefore, the preliminary significance was based on a Phase 2 evaluation in accordance with Inspection Manual Chapter 0609. In order to complete a final significance determination, additional information is needed. The specific information needed is described in the assessment portion of Section 5.1.

This finding does not represent an immediate safety concern because of the immediate and long term corrective actions that have been implemented to reduce chemical addition, remove excess impurities from the ponds, clean all the affected heat exchangers, and adjust control limits appropriately. You are also in the process of removing sediment and sludge from the ponds.

There are five apparent violations associated with this finding being considered for escalated enforcement action in accordance with the NRC Enforcement Policy. The current Enforcement Policy is included on the NRCs web site at http://www.nrc.gov/reading-rm/adams.html.

In accordance with Inspection Manual Chapter 0609, we intend to complete our evaluation using the best available information and issue our final determination of safety significance within 90 days of this letter.

Arizona Public Service Company -3-The significance determination process encourages an open dialog between the staff and the licensee, however the dialogue should not impact the timeliness of the staffs final determination. Before we make a final decision on this matter, we are providing you an opportunity (1) to present to the NRC your perspectives on the facts and assumptions used by the NRC to arrive at the finding and its significance at a Regulatory Conference or (2) submit your position on the finding to the NRC in writing. If you request a Regulatory Conference, it should be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least 1 week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open for public observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of the receipt of this letter.

Please contact Linda J. Smith at (817) 860-8137 within 10 business days of the date of this receipt of this letter to notify the NRC of your intentions. If we have not heard from you within 10 days, we will continue with our significance determination and enforcement decision and you will be advised by separate correspondence of the results of our deliberations on this matter.

Since the NRC has not made a final determination in this matter, no Notice of Violation is being issued for this inspection finding at this time. In addition, please be advised that the number and characterization of apparent violations described in the enclosed inspection report may change as a result of further NRC review.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Dwight D. Chamberlain, Director Division of Reactor Safety Dockets: 50-528; 50-529; 50-530 Licenses: NPF-41; NPF-51; NPF-74

Enclosure:

Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011 w/Attachment: Supplemental Information

REGION IV==

Dockets: 50-528; 50-529; 50-530 Licenses: NPF-41; NPF-51; NPF-74 Report No.: 05000528/2006011; 05000529/2006011; 05000530/2006011 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station, Units 1, 2, and 3 Location: 5951 S. Wintersburg Road Tonopah, Arizona Dates: June 6 through September 26, 2006 Inspectors: N. O'Keefe, Team Leader P Goldberg, Reactor Inspector, Engineering Branch 1 Y. Diaz-Castillo, Chemical Engineer, Office of Nuclear Reactor Regulation (NRR)

K. Parczewiski, Senior Chemical Engineer, NRR M. Runyan, Senior Reactor Analyst Approved Dwight D. Chamberlain, Director By: Division of Reactor Safety

SUMMARY

OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE . . . . . . . . . 34 1R07 Biennial Heat Sink Performance (71111.07B) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4OA6 Meetings, Including Exit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4OA7 Licensee-Identified Violation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 ATTACHMENT 1: SUPPLEMENTAL INFORMATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 KEY POINTS OF CONTACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

LIST OF DOCUMENTS REVIEWED . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ATTACHMENT 2: SPECIAL INSPECTION CHARTER . . . . . . . . . . . . . . . . . . . . . . . . . . .

ATTACHMENT 3: SUMMARY OF HEAT EXCHANGER PERFORMANCE TEST RESULTSA3-1 ATTACHMENT 4: SEQUENCE OF EVENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ii

SUMMARY OF FINDINGS IR 05000528/2006011; 05000529/2006011; 05000530/2006011; 6/6-9/26/2006; Palo Verde Nuclear Generating Station, Units 1, 2, and 3: Special Inspection, 50.59, Operability Evaluations, Design Control, Corrective Actions, Biennial Heat Sink, Test Control.

The report covered a 3-month period of inspection by two region-based inspectors and two chemistry subject matter experts from headquarters. In addition to performing a special inspection, this report includes the results of the announced biennial heat sink inspection, which was performed concurrently. One finding was identified with five apparent violations, which were determined to have potential safety significance greater than very low safety significance.

Additionally, two NRC-identified noncited violations and one licensee-identified noncited violation with very low safety significance were identified. The significance of most findings is indicated by its color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609,

"Significance Determination Process." Findings for which the significance determination process does not apply may be green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, dated July 2000.

Summary of Event and Inspection Results The NRC conducted a special inspection to better understand the circumstances surrounding the heat exchanger fouling at Palo Verde Nuclear Generating Station. The fouling came to the NRCs attention as a result of unusual temperatures noted during a surveillance test of Emergency Diesel Generator 2B conducted on May 17, 2006. After 2 days, it was determined that this potentially impacted the ability of this safety-related system to perform its safety function during a design basis accident. It also became apparent that the fouling that was experienced had existed for a long period of time. The cause of the fouling was not fully understood, and affected other equipment in all trains of all three units. In accordance with NRC Management Directive 8.3, it was determined that this event had sufficient risk significance and met several deterministic criteria to warrant a special inspection.

This inspection identified a pattern of behavior where heat exchanger performance data and internal inspection results were not understood, entered into the corrective action program, or adequately assessed for operability impact. When the degradation increased and the appearance of the foulant changed, this did not cause an increased response, other than to clean heat exchangers more frequently. The chemistry control credited to prevent fouling was determined to have no mechanism to actively monitor fouling. Further, the chemistry control limits were changed over time to actually promote fouling and increase the potential for scaling during design basis accidents. This pattern of behavior existed for the last 10 years.

Cornerstone: Mitigating Systems

• TBD. A finding with five apparent violations was identified associated with fouling of safety-related heat exchangers cooled by the emergency spray pond system. Between 1995 and May, 2006, the licensee failed to recognize that improperly implemented chemistry controls for the emergency spray pond system caused a significant condition adverse to quality which degraded the performance of all emergency diesel generators and emergency cooling water systems. The degraded performance was primarily due to heat exchanger fouling caused by adding excessive amounts of chemicals. The conditions that existed also had the potential to cause scaling after an accident starts. In one instance, it is estimated that this resulted in degrading the performance of Emergency Cooling Water Heat Exchanger 2B to the point where it would not have been capable of performing its intended safety function for approximately 6.8 months in 2003.

An apparent violation of Technical Specification 3.7.7 was identified because Train A of the Essential Cooling Water System in Unit 2 was not capable of performing its safety function for approximately 6.8 months ending on September 27, 2003.

An apparent violation of 10 CFR Part 50, Appendix B, Criterion XI, "Test Control," was identified because the two procedures that were performed to measure essential cooling water heat exchanger performance were implemented in a way that was inadequate to ensure the timely determination that the requirements and acceptance limits contained in applicable design documents were met.

An apparent violation of 10 CFR 50.59 was identified for making nine revisions to Procedure 74DP-9CY04, "System Chemistry Specification," a procedure described in the Updated Final Safety Analysis Report, between 1998 and 2004 without performing evaluations of the potential impact of the changes on the safety-related components in the spray pond system; the changes revised spray pond chemistry parameter limits which were subsequently determined to have contributed to heat exchanger fouling.

An apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI,

"Corrective Actions," was identified. On March 19, 2002, performance testing for Essential Cooling Water Heat Exchanger 2B indicated that the system would not be capable of performing its design function, but this significant condition adverse to quality was not promptly identified, the cause determined, or corrective actions taken to restore the required heat exchanger performance. The failure to correct this degraded performance contributed to the continued degradation and eventual loss of function for an estimated period of 6.8 months.

An apparent violation of 10 CFR Part 50, Appendix B, Criterion III,

"Design Control," was identified for failure to correctly evaluate the scaling potential of the safety-related heat exchangers cooled by the emergency spray pond during a design basis accident. An error in the SEQUIL calculation caused the licensee to incorrectly conclude that scaling would not occur under the conditions established in the chemistry control program.

The performance deficiency associated with these apparent violations was more than minor because it impacted the equipment performance attribute of the Mitigating Systems Cornerstone objective to maintain the availability and reliability of systems needed to mitigate accidents. Specifically, Essential Cooling Water (EW) Train B in Unit 2 was estimated to have been incapable of performing its function under existing conditions for approximately 6.8 months. A Phase 2 significance determination process concluded that this finding has potential safety significance greater than very low safety significance because some accident sequences, notably loss of coolant accidents, were expected to elevate the ultimate heat sink temperature to the point where the degraded essential cooling water heat exchanger would be challenged. This was expected to cause failure of the essential chiller, and the resulting loss of room cooling to safety-related equipment increased the plant risk. In addition, there is uncertainty associated with the amount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of an accident scenario. Additional information was needed to perform a final Phase 3 assessment, due to the complexity of the issue. (Section 5.1)

Cornerstone: Mitigating Systems

Green.

Two examples of a noncited violation of 10 CFR Part 50, Appendix B,

Criterion III, "Design Control," were identified involving the failure to adequately translate the design basis of the spray ponds into procedures. Design Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis Analysis," Revision 7, which demonstrated that the spray pond system could adequately limit spray pond temperature during a design basis accident did not account for any reduced heat capacity caused by sediment buildup.

However, this fact was not translated into procedures, so approximately 400 cubic yards of sediment had built up in each of the six spray ponds when the team questioned the impact to the heat removal function. Also, the same calculation demonstrated that sufficient water was available to provide adequate cooling during a design basis accident, but did not account for any leakage from the ponds. The team determined that the licensee did not translate this into a procedure to ensure that the condition of the spray pond was maintained such that leakage did not occur. Procedure 81DP-0ZZ01, "Civil System, Structure, and Component Monitoring Program," Revision 11, was used to monitor the condition of the pond structures. The team identified that it examined only the exposed concrete surfaces, which constituted about 7 percent of the surface area and almost none of the water-containing volume. Cracks had been identified and repaired in this area, but the inspections were not expanded to the underwater surfaces. This issue was documented in Condition Report/Disposition Requests 2906671 and 2910912.

Failure to adequately translate the design basis of the spray ponds into procedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, the finding could become a more significant safety concern. This finding affected the Mitigating Systems Cornerstone. This performance deficiency screened as having very low safety significance in a Phase 1 significance determination process because the licensee was able to demonstrate that the sediment would not have resulted in a loss of safety function, and that significant leakage did not exist. The licensee was able to revise the calculation to take credit for heat absorption by the concrete walls, and scheduled inspections by divers of underwater portions of the ponds to follow sediment removal. (Section 5.2)

Cornerstone: Mitigating Systems

Green.

A noncited violation of 10 CFR Part 50, Appendix B, Criterion V,

"Instructions, Procedures and Drawings," with multiple examples was identified for failure to adequately assess the impact to operability of degraded heat exchangers in the emergency diesel generators and essential cooling water system. Specifically, the licensee failed to follow Procedure 40DP-9OP26,

"Operability Determination and Functional Assessment," Revision 16, in assessing indications of degraded heat exchanger performance, an activity affecting quality. Key support organizations were not always involving operations personnel with questions that had a potential to affect the operability of safety-related equipment, or were informing operators only after the support organization had fully evaluated the condition, delaying actions that were required to be prompt. Also, operations personnel did not always insist on a rigorous evaluation. This issue was documented in Condition Report/Disposition Requests 2918892, 2901815, and 2898237.

Failure to adequately implement the operability assessment process was a performance deficiency. This finding was more than minor because it impacted the equipment performance attribute of the Mitigating Systems Cornerstone objective to maintain the availability and reliability of systems needed to mitigate accidents. This finding screened as having very low safety significance in a Phase 1 significance determination process, because the examples used for this violation were confirmed not to involve any loss of safety function. This finding had cross-cutting aspects in the area of human performance because the licensee did not follow their systematic process for operability decision making when information was not brought to the right decision makers. (Section 5.3)

B. Licensee-Identified Findings A violation of very low safety significance, which was identified by the licensee, has been reviewed by the inspectors. Corrective actions taken or planned by the licensee have been entered into the licensee's corrective action program. This violation and corrective actions are listed in Section 4OA7 of this report.

REPORT DETAILS

1.0 SPECIAL

INSPECTION SCOPE

The NRC conducted this special inspection to better understand the circumstances surrounding the heat exchanger fouling at Palo Verde Nuclear Generating Station. The fouling came to the NRCs attention as a result of unusual temperatures noted during a surveillance test of Emergency Diesel Generator (EDG) 2B on May 17, 2006. Over the following several days, it was determined that this potentially impacted the ability of this safety-related system to perform its safety function during a design basis accident. It also became apparent that the fouling that was experienced had existed for a long period of time. The cause of the fouling was not fully understood, and affected other equipment in all trains of all three units. In accordance with NRC Management Directive 8.3, it was determined that this event had sufficient risk significance and met several deterministic criteria to warrant a special inspection.

The team used NRC Inspection Procedure 93812, "Special Inspection Procedure," to conduct the inspection. The special inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern.

The team interviewed key station personnel regarding the event, the root cause analysis, and corrective actions. A list of specific documents reviewed is provided in

1. The charter for the special inspection effort is provided as Attachment 2.

2.0 SYSTEM AND EVENT DESCRIPTION 2.1 Event Summary On May 17, 2006, Palo Verde Nuclear Generating Station noted elevated temperatures in the intake air for EDG 2B while the engine was running for a surveillance test. The engine intake air is compressed and heated by the turbocharger, and is then cooled by two parallel intercoolers before entering the engine. The elevated temperatures were noted at the outlet of the intercoolers, which was indicative of inadequate cooling in the intercoolers.

The elevated temperatures were discussed among operations and engineering personnel, and it was concluded that this condition did not affect operability. Emergency Diesel Generator 2B was returned to service, and the following day the EDG Train A and essential cooling water (EW) heat exchanger were removed from service for approximately 23 hours2.662037e-4 days <br />0.00639 hours <br />3.80291e-5 weeks <br />8.7515e-6 months <br />. On May 19, following the return to service of Train A, the operability of EDG 2B was revisited. An immediate operability determination was completed at 9:55 a.m. which concluded EDG 2B was operable. After additional review, at 5:54 p.m., it was concluded that EDG 2B was not operable, and Technical Specification 3.8.1 was entered. Because the condition was potentially a common mode failure concern for the EDGs, all EDGs were run to assess the performance of the intercoolers in accordance with Technical Specification 3.8.1.2.

The licensee inspected the EDG 2B intercoolers and found they were fouled on the cooling water (spray pond system) side. The emergency spray pond system (SP)provides the cooling water to both the EDG coolers (two intercoolers, a jacket water cooler, and a lube oil cooler), as well as cooling the large EW heat exchanger. The intercoolers' water side had a white slimy substance which was apparently reducing the ability to transfer heat.

That same week, the licensee had taken each of the Unit 2 EW heat exchangers out of service for tube cleaning. This was in response to earlier heat transfer degradation which was judged to be degrading fast enough that the Unit 2 EW heat exchangers would not remain operable through the full operating cycle and needed to be cleaned before summer. The degradation was not fully understood, but was suspected to be related to a similar white slimy substance.

As NRC inspectors got involved and asked key questions related to the operability of these components, it became apparent that the scope of the problem was more than just EDG 2B.

2.2 System Descriptions The spray pond system for each unit is comprised of two trains. Each train has a concrete lined spray pond of 6.2 million gallons of water, which serves as the ultimate heat sink. The ponds are 345 feet by 172 feet, and are 15.5 feet deep. The spray pump for each train takes a suction on one corner of the pond, at a level 6 feet below the bottom of the main portion of the pond. The pump delivers 16,300 gpm through buried piping to the EW heat exchanger and the EDG coolers. The water then returns to the pond through over 300 spray nozzles spaced around the pond above the surface.

The spray nozzles atomize the water, allowing for evaporative cooling. The spray pond system is normally in a standby condition with no flow. The spray pond piping was made of carbon steel with a protective coating on the inside surfaces.

Each EDG has five coolers which have heat removed by the spray pond system: the jacket water cooler; lube oil cooler: fuel oil cooler, and a pair of turbocharger intercoolers. While the EDG is normally in standby, the jacket water system and lube oil system continue to circulate and heat their process fluids. The fuel oil cooler has a very small flow, and is not required for the EDG to function. The intercoolers function at the highest temperature, approximately 290F air inlet temperature.

The EW heat exchangers transfer heat from the systems it cools to the spray pond system. The loads supplied are the essential chillers, shutdown cooling heat exchanger, and spent fuel pool cooling system. The essential chillers provide cooling to the control room ventilation, safety-related switchgear rooms, and emergency core cooling pump rooms. The shutdown cooling heat load during an accident constitutes over 94 percent of the total EW heat load.

The spray ponds were provided with a filtering system. Pond water was pumped through a pair of sand filters to remove larger impurities at 600 gpm each (1200 gpm total). The sand filters were backwashed to remove the impurities about once per day under normal conditions.

3.0 CHEMISTRY DESCRIPTION 3.1 Fouling Mechanisms The degraded heat transfer capabilities in the EDG coolers and the EW heat exchangers occurred in all trains in all units. The degree of degradation varied somewhat among the units. The EDG intercoolers and the EW heat exchangers were observed to have fouled the most, while the jacket water and lube oil coolers fouling was minor and had no impact to operation.

The team determined that the licensee did not take photographs or have samples of the substances found in these heat exchangers analyzed. When the May 2006 event occurred, samples were analyzed, but the exact substances could not be determined because it was an amorphous mix of a number of substances. The major constituents were determined to be zinc (38 percent), phosphorous (29 percent), and calcium (32 percent).

Two different substances have been found in the heat exchangers, though this was not recognized. In earlier years, a thin coating that turned white as it dried was observed in heat exchanger inspections. This is now believed to be zinc hydroxide. In more recent years, a thicker coating that has been described as white, slimy, and lotion-like was observed. It was believed for years that the coating was zinc hydroxide which the chemistry control program was intentionally trying to deposit on the system piping, so it was largely accepted. The team noted that the licensee did not reassess what the material was when the appearance began to change.

The licensee's root cause assessment concluded that improper chemical control resulted initially in excessive zinc hydroxide precipitating out of solution, which is a heat transfer foulant. Later, zinc phosphate precipitated as a thicker mix. This precipitant then provided a localized seed site for other chemicals to also precipitate out of solution.

The combined effect was an amorphous lotion-like substance that coated heat exchanger tubes and degraded heat transfer by interrupting cooling water flow at the tube wall. Specifically, the fluffy film caused a low flow zone near the tube wall and forced the turbulent flow away from the tube wall; since efficient convective heat transfer required strong turbulent flow at the tube wall, heat transfer was significantly impacted even though the film was fluffy and not tightly adherent.

It should be noted that the above fouling mechanisms were different than scaling.

Scaling involved hard mineral deposits, such as calcium carbonate and calcium phosphate which occur at high temperatures. Scaling concerns are addressed in Section 5.1.5.

3.2 The Intended Chemistry Control Program The licensee's program was intended to meet two goals: minimize corrosion in the spray pond piping and components, and minimize fouling. The team determined that the Palo Verde Nuclear Generating Station program was similar to that used in cooling water systems in the electric power industry, both nuclear and non-nuclear. Though complex, this type of program is typically effective. When initially implemented, the spray pond chemistry control program used the chemical supply vendors recommended limits; however, over time these limits were changed numerous times, eventually departing from these recommendations.

The Palo Verde Nuclear Generating Station program treated the two parts of the spray pond system separately. The spray ponds were treated only to prevent gross biological fouling. The spray pond system piping and components were treated with a complicated combination of chemicals to meet the goals stated above. Chemicals were added at the pump suction during pump runs for piping treatment, and pond chemicals were added at the same location but were not effectively dispersed through the pond.

The carbon steel piping was protected from corrosion by adding zinc and phosphoric acid. This combination was intended to form a very thin layer (a few molecules thick) of zinc, with phosphate in solution to protect any active corrosion sites. The root cause assessment team concluded that this film should not be thick enough to be visible on the piping. The zinc coating must be replaced periodically, so extra zinc is added to stay in solution to be available as replacement if needed. However, zinc has a low solubility and will precipitate out of solution unless the conditions are carefully managed.

The makeup water for the system has a relatively high alkalinity and some potentially scale-forming impurities (such as calcium). Scale is more likely to form at higher temperatures (such as those expected in heat exchangers), and under higher pH values. The alkalinity from makeup water and evaporating of water caused pH to naturally increase. Therefore, the licensee added sulphuric acid to lower pH and improve the solubility conditions.

Dispersant was added to keep the impurities and the chemicals that were intentionally added in solution and avoid precipitation and fouling. The dispersant was very important to control solubility and avoid fouling and scaling. However, the team determined that the dispersant in use at the time of the event could not be measured directly, and the method used to estimate dispersant concentration was no longer reflecting the actual conditions.

Finally, the licensee added biocide and hypochlorite to control biological growth. The piping was treated during pump runs; pond treatments were started during pump runs, but continued for hours after the pump was secured. All chemicals were added to the ponds near the pumps.

The team identified that the chemistry program objectives to minimize fouling was not being monitored. The effectiveness of fouling prevention was not being tracked by chemistry personnel.

3.3 How The Chemistry Control Was Actually Implemented A summary of the chemistry control program and changes made over time is provided in Section 6.1.

The team determined that the chemistry control program was being implemented in a way that caused fouling of safety-related heat exchangers. Chemical additives (zinc and phosphate) were being added in excess of what was needed and above the solubility limits for the existing conditions. The pH was also being maintained too high, reducing the solubility limits. Also, dispersant was not being accurately being monitored, and as a result, was not being added in sufficient quantities to effectively control the solubility of the existing chemicals. These conditions combined to cause the fouling.

The team reached the following conclusions about the actual implementation of the spray pond chemistry control program:

The Chemistry personnel implementing the program did not fully understand how the spray pond chemistry control program was supposed to work. As a result, they made a series of inappropriate changes to chemistry limits which negatively impacted solubility without assessing or understanding the potential impacts.

The specified pH band was too high to maintain zinc hydroxide in solution, so it precipitated out of solution and formed a foulant.

The zinc control band was too high, resulting in injecting too much and precipitating significant quantities out of solution. The precipitate collected in the EW heat exchangers and intercoolers, and may have also precipitated into the sediment at the bottom of the ponds. The ideal control concentration was determined to be about 0.5 ppm by the root cause team, while the actual control band was 1.0 - 3.0 ppm at the time of the event.

Excess phosphate was being added, contributing to the precipitation problems. Phosphoric acid was added in a 4:1 ratio with the zinc; zinc was constantly removed from solution, while phosphate built up in the pond and piping. The ideal phosphate level was determined by the root cause team to be about 4.0 ppm, while at times, samples recorded over 20 ppm phosphates in Unit 2. The high phosphate levels exceeded the maximum solubility for the bulk water conditions, and precipitated out of solution in the EW heat exchangers and intercoolers, and probably also precipitated into the sediment at the bottom of the ponds.

Calcium and other impurities from makeup water were allowed to concentrate excessively over time, and under the relatively high pH conditions they contributed to precipitation problems. The root cause team concluded that the zinc phosphate precipitant provided a seed site for other compounds containing zinc, calcium, and phosphate to attach.

Dispersant was intended to improve the solubility conditions, but there was too little dispersant present to be effective. The dispersant in use could not be measured directly, so a "tag" chemical (molybdate) which could be measured was added in a known proportion to the dispersant. The licensee did not recognize that this tag chemical was not subject to the same loss and use mechanisms as the dispersant. As a result, the tag chemical analyses registered incorrectly high results. The team determined that Chemistry personnel reduced or stopped adding dispersant for long periods of time because the molybdate concentrations indicated dispersant levels were high. Contributing to this, the tag chemical was only checked once per month. Also, when the licensee allowed the high levels of impurities to build up, they did not recognize that this should have required raising the amount of dispersant to maintain the same effectiveness.

The team identified that the chemistry control practices for this system were not typical of other plant systems with the same importance. Chemicals were added based on thumbrules and past additions, rather than basing them on recent sample results. Also, many key samples were taken only once per month, but the associated chemical additions were made several times per week. Some important samples had no controls associated with undesirable levels. For example calcium, an impurity, which is a scale-forming concern, had no action level to trigger action to reduce the concentration. The spray pond system was a concentrator for chemicals, so as calcium levels accumulated over the years, Chemistry raised the limits to stay ahead of the actual levels, rather than reducing the existing concentrations. When sample results indicated that molybdate levels were high and it was presumed that this meant dispersant levels were high, dispersant additions were stopped. The molybdate samples were taken monthly, but the analyses were sometimes delayed by up to 2 to 3 weeks (Condition Report/

Disposition Request (CRDR) 2800653). This created an untimely feedback mechanism for adjusting the chemical addition.

The team identified that chemistry personal had focused on treating the pipes and did not effectively manage the chemistry condition in bulk pond water. This approach was viewed as economical, and is a common industrial practice. However, this did not account for the unique nature of the spray ponds at Palo Verde Nuclear Generating Station. Unlike typical systems, the spray pond system was not designed or operated in a way that regular water turnover occurred, such that chemicals and impurities would be removed. It was not recognized that the chemistry control program in use typically involved a system that ran continuously and had a much higher water turnover. The turnover should result in removal of impurities and cause the tag chemicals to more closely reflect dispersant concentrations. The continual buildup of chemicals and impurities at Palo Verde Nuclear Generating Station caused a departure from the conditions assumed when the vendor recommendation for dispersant levels was made.

Most sampling was done during pump runs by taking samples from the pipes, with very little sampling of the bulk water in the ponds. As a result, the ponds were allowed to accumulate chemicals and impurities. This was further complicated by allowing sediment to accumulate at the bottom of the ponds, where precipitants could be concealed but still affect water chemistry.

Biological fouling in the ponds was not being effectively controlled. The team concluded that the biocide treatments and monitoring of the ponds were ineffective. During the onsite portion of the inspection, contract divers began vacuuming sediment from the Unit 1 ponds. Condition Report/Disposition Request 2905162 was written on June 21, 2006, to document that the diving operations were dislodging "sheets" of algae from the spray pond walls. The licensee was unaware that algae existed in the ponds prior to this time. The team reviewed the pond treatment practices and concluded that the biocide addition point was in one corner of the ponds, insufficient mixing was provided to distribute the chemicals. This was, in part, because much of the biocide chemicals were added after securing the spray pond pumps. The licensees evaluation concluded that the existing algae would not impact operability, and the team concluded that this was a reasonable conclusion.

Finally, the water chemistry control program being used was challenged by the licensee's practice of leaving the systems idle, without constant flow, for most of the time. Typical cooling water systems run pumps continuously, which have the benefits for flushing pipes and heat exchangers out and mixing the chemicals. The Palo Verde Nuclear Generating Station spray pond systems were normally run only for chemistry control (adding chemicals 3 times per week), infrequent testing, and during plant outages. This resulted in the system having no flow most of the time.

The team noted that the chemistry control program documents stated that dispersant additions were intended to be based on makeup water addition. However, that had never been the actual practice. The team identified that the licensee did not track makeup water additions to the spray ponds, and had no way of measuring the quantity of water flowing to any one pond. In an idle pond, evaporative losses ranged from 2,200 to 10,000 gallons per day. When the system was run, evaporation was much higher, and "drift" losses occurred when water droplets were carried out of the pond berm by wind. Despite these large and highly variable losses which required makeup and caused loss of dispersant, the team identified that the licensee had stopped adding dispersant for long periods, in part because they were not adding dispersant based on makeup water addition.

The team identified that the chemical supply vendor site representative visited the site several times per month, but this source of expertise was not utilized. The vendor was not asked to assess the performance of the program, and was not asked to observe heat exchanger foulant or analyze the material. The site representative was involved in trying to identify the lotion-like substance found in the spray pond sand filters, though he was not provided with a sample as a basis for the discussion.

The team concluded that the licensee was no longer maintaining the spray pond chemistry control program within the bounds that were originally intended. They had made changes without understanding the effect the changes would have on the conditions and without understanding the complex interactions that existed. As a result, the fouling was self-induced by the chemistry control program.

4.0 HEAT EXCHANGER PERFORMANCE MONITORING 4.1 Essential Cooling Water (EW) Heat Exchanger Performance Testing The licensee committed to conducting heat exchanger performance testing on the EW heat exchangers. This was typically done for one train at the beginning of each refueling outage during the primary plant cooldown, when decay heat load allowed enough heat load to be able to get accurate results. Later, the licensee began to also conduct EW heat exchanger performance testing during the primary plant heatup at the end of refueling outages. These tests had a much lower heat loading, so the accuracy of the results was potentially lower, but the results would still be expected to provide an indication of the relative effectiveness of any tube cleaning during that outage.

The team performed a detailed review of the licensees performance testing methods and results, including internal inspections and cleaning practices. This included reviewing documented results since 2001. This is further discussed in Section 1R07.

The results of the performance testing conducted on the EW heat exchangers is summarized in Attachment 3. It can be seen that the data for any one heat exchanger is very limited, since it might be tested only once every three years. It can also be seen that significant reductions in heat exchanger performance were indicated by test results, yet very few instances were found where this degradation was documented in the corrective action program. This is further discussed in Section 5.4.

4.2 Emergency Diesel Generator (EDG) Performance Testing The licensee did not conduct performance testing of any of the EDG heat exchangers.

The primary reasons are briefly summarized:

Lube oil coolers and jacket water coolers: The shell side fluids (lube oil and jacket water) have automatically controlled flows that adjust to the conditions to maintain a constant temperature. The thermostatic control valves cannot be manually overridden or their positions measured. Also, the systems do not have installed flow measurement devices, and the pipe configuration prevents an accurate flow measurement using an external device.

Intercoolers: These heat exchangers have a relatively small heat load, and use air as a working fluid. Both these facts make it difficult to accurately measure heat exchanger performance.

Instead of performance testing, Palo Verde Nuclear Generating Stations commitment in their Generic Letter 89-13 response was to regularly clean and inspect these heat exchangers to ensure there was not excessive fouling. Since these inspections identified that the intercoolers in each unit experienced fouling, the licensee was trending the relative performance of the intercoolers during EDG surveillance testing.

This was done by assessing the difference between air outlet temperature and water inlet temperature over time. Cleanings were scheduled when this difference became high, but the condition was documented as a degraded condition in the corrective action program only once, although the degraded condition occurred on numerous occasions.

This is further discussed in Section 5.3.

5.0 FINDINGS 5.1 Performance Deficiencies that Directly Related to Fouling to the Point of Inoperability a.

Description.

In 1994, Palo Verde Nuclear Generating Station implemented a new chemistry control program for the emergency spray pond system to control corrosion and prevent fouling of the safety-related components. This system provides the ultimate heat sink function.

Throughout the period of 1994 through May 2006, the licensee made a series of changes to this program which created a chemical environment that was progressively more conducive to fouling of the heat exchangers which were relied upon to transfer heat from the reactor, containment, EDGs, and safety-related equipment rooms to the ultimate heat sink. The foulant was determined to be a buildup of excess chemicals which were added as part of the chemistry control program, but which were subsequently determined to have exceeded the limitations of solubility. Despite years of test results which showed degraded heat exchanger performance, numerous heat exchanger inspections which documented chemical buildup, and an increasing need to clean the heat exchangers more frequently, the licensee failed to recognize the safety significance of the problem or determine and correct the cause. Degraded performance was observed in all trains in all three units. Because of significant inherent design margins, only one example was identified where fouling may have been sufficient to cause a loss of safety function. Specifically, the Unit 2 Train B EW heat exchanger was potentially inoperable between about April and October 2003.

Also, during the inspection it was recognized that calculations which demonstrated that heat exchangers cooled by the spray pond system would not be subject to scaling during a design basis accident contained an error. When re-performed, the calculations indicated that scaling was possible. This would have the effect of further reducing the heat transfer capability of these heat exchangers during certain accident scenarios.

This had the potential to affect components cooled by the spray pond in all units.

The team reviewed the degraded performance and the remaining capabilities of these components over the period 1994 through 2006. The licensee performed computer modeling and analysis that adequately demonstrated that the EDG systems were capable of performing under the degraded conditions. This was due to considerable margins inherent in the design, and because it could be shown that the fouling in EDG 2B that triggered this inspection was clearly the worst case that had been experienced. For the EW heat exchangers, the team identified the following four periods where the EW heat exchangers had less than the design basis heat removal capability, a degraded condition which constituted a significant condition adverse to quality. Of these, it appears that only 2B was incapable of performing its function under the existing, less challenging conditions in 2003.

EW HX 1B was possibly as low as -20.8 percent margin entering 1R11 in 2004.

EW HX 2B was possibly as low as -49.78 percent margin entering 2R11 in 2003.

EW HX 3B was possibly as low as -14.4% percent margin entering a midcycle outage in 2004. .

EW HX 3A was measured at -22 percent margin entering 3R10 in 2003.

These estimates do not include the possible effect due to scaling discussed in subsection

(5) below. The impact due to fouling will be reassessed in all units during the final significance determination.

Note that throughout this report, heat exchanger capacity is discussed in terms of "percent margin." This term is meant to refer to margin above or below the design capacity of a heat exchanger. In this context, "0 percent margin" means the heat exchanger would have exactly the capacity required to transfer the design heat load under limiting design conditions. Positive or negative margin expresses how much more or less than that capability actually existed. This was used to be consistent with licensee documentation.

b. Statement of the Performance Deficiency.

Between 1995 and May, 2006, the licensee failed to recognize that improperly implemented chemistry controls for the emergency spray pond system caused a significant condition adverse to quality which degraded the performance of all emergency diesel generators and emergency cooling water systems. The degraded performance was because of heat exchanger fouling from chemical additives, and potentially because of scaling after an accident starts. This resulted in degrading the performance of EW Heat Exchanger 2B to the point where it would not have been capable of performing its intended safety function for approximately 6.8 months in 2003.

In addition, there is uncertainty associated with the amount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of an accident scenario.

c.

Enforcement.

(1) Technical Specification 3.7.7 requires that two trains of essential cooling water be operable in Mode 1, with an allowed outage time for one train of 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />. Contrary to this, Train A of the EW system in Unit 2 was not capable of performing its safety function for approximately 6.8 months ending on September 27, 2003, when the plant shut down for Refueling Outage 11. This will be treated as an apparent violation pending a final significance determination: AV 05000529/2006011-01, EW Train 2B Inoperable Longer than Allowed Outage Time. This issue was entered into the licensee's corrective action program under CRDR 2905161.

The period during which the train was incapable of performing its intended safety function was determined through straight-line extrapolation of the two prior heat exchanger performance tests for this train. The minimum acceptable performance was determined by licensee calculation dated June 22, 2006, which revised Design Basis Calculation 13-MC-SP-307 to remove all known margins. The minimum acceptable performance was 204 Btu/hr-ft2-F, or about -35 percent margin compared to the full design basis value. This calculation included the most limiting values for spray pond temperature and air temperature actually recorded during the exposure period, and accounted for the existing number of tubes with plugs installed. The team reviewed this calculation and determined that appropriate values and assumptions were used for the purpose of determining heat exchanger performance capability.

The licensee showed documentation that indicated that eddy current testing of this heat exchanger was performed during 2R10 in March, 2002. The licensee stated that the probes were known to push out the majority of the foulant and improve heat exchanger performance. The team acknowledged this fact, but could not quantify any potential improvement because no data existed to quantify this for Unit 2 EW heat exchangers.

The Unit 2 EW heat exchangers were the only ones with sleeves at each end; the sleeves necessitated using a smaller probe than was used in the other units, and thus had a bigger gap between the probe and the majority of the tube wall length. This fact precluded using test data which was available for the other units, particularly because the heat transfer characteristics of the foulant were not known.

(2) Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XI, "Test Control," requires that a test program shall be established to assure that all testing required to demonstrate that structures, systems, and components will perform satisfactorily in service is identified and performed in accordance with written test procedures which incorporate the requirements and acceptance limits contained in applicable design documents. Test results are required to be documented and evaluated to assure that test requirements have been satisfied. Test Procedure 70TI-9EW01, "Thermal Performance Testing of Essential Cooling Water Heat Exchangers," contained instructions for conducting performance testing.

Procedure 73DP-9ZZ10, "Guidelines for Heat Exchanger Thermal Performance Analysis," contained instructions for calculating the results of EW heat exchanger performance testing and verifying them against the design basis requirements in Calculation 13-MC-SP-0307, "SP/EW System Performance Design Bases Analysis,"

Revision 007. Contrary to this, the results of performance testing for EW Heat Exchanger 2B conducted on March 19, 2002, did not meet the design basis requirements, but this was not correctly evaluated to determine whether the system would be capable of performing its design function until August 22, 2002. As a result, Unit 2 was allowed to restart and run an entire operating cycle without correcting the degraded performance. This will be treated as an apparent violation pending final significance determination: AV 05000528; 05000529;05000530/2006011-02, Inadequate Test Control to Promptly Identify Unacceptable Performance Test Results.

This issue has been entered into the licensee's corrective action program under CRDR 2928230.

The team determined that the performance test was scheduled and conducted under a maintenance order. However, the calculation of the results and comparison to acceptance criteria was not scheduled. Further, the test results were not administratively tracked to ensure that the unit was not restarted before acceptable performance was documented. As a result, the March 19, 2002, test results were determined after Unit 2 started up from Refueling Outage 2R10. The test results were unexpectedly low, but this was determined to involve a calculation error. When the error was corrected with results, which indicated low performance, the test method was evaluated. On August 22, 2002, the test package was completed and verified to demonstrate -9.5 percent capability. This untimely determination of unacceptable performance did not trigger a review of operability, nor was continued degradation considered. As a result, this component continued to degrade for 18 months after demonstrating unacceptable performance. Had this determination been made prior to the end of the outage, corrective actions could have been taken to clean the tubes.

Technical Specifications require that both trains be operable prior to making plant mode changes, so the unit should not have be restarted with this condition, had it been identified.

(3) Part 50.59 of Title 10 of the Code of Federal Regulations allows licensees to make changes to procedures as described in the Updated Final Safety Analysis Report without obtaining a license amendment only if the change satisfies the criteria in 50.59.c. It goes on to specify that changes made to procedures must include written evaluations which provide the bases for the determination that the change does not require a license amendment.

The Palo Verde Nuclear Generating Station Updated Final Safety Analysis Report, Section 9.2.1.2.F, specified that the spray pond system would be protected from organic fouling and inorganic buildup by proper water treatment. Procedure 74DP-9CY04, "System Chemistry Specification," implemented the licensee's program, including chemical limits and addition frequencies, in order to control corrosion and fouling in the emergency spray pond system.

Contrary to the above, between 1998 and 2004, the licensee made multiple changes to the chemistry control program specified in Procedure 74DP-9CY04, "System Chemistry Specification," without adequately assessing the impact to the safety-related systems affected by the changes. Revisions 3, 6, 8, 10, 12, 24, 28, 32, and 36 to Procedure 74DP-9CY04 made changes between March 1998 and February 2006, which affected spray pond chemistry parameter limits which were subsequently determined to contribute to heat exchanger fouling. Specifically:

• Revisions 3, 6, 8, 12, 24, 28, and 32 were made without any documented 50.59 review. The team determined that these revisions contained changes to chemistry limits in the spray pond which had the potential to negatively impact fouling, and therefore were required to be evaluated per 50.59. These included changing pH, phosphate, and zinc limits, and instructions on the frequency of dispersant additions.

• The screening performed for Revision 10 was determined to be inadequate because the revision increased the maximum allowable pH in Unit 1, but the screening dated 6/30/99 did not evaluate this aspect of the revision. This limit had been lowered in Revision 6 to account for the decision to use well water as the makeup source for the Unit 2 spray pond. This water source had a significantly higher alkalinity, and the lower pH was determined to be necessary to avoid scaling at the time the modification was implemented. However, this aspect of the change was not evaluated in this screening.

• The screening performed for Revision 36 was determined to be inadequate because the revision increased the maximum allowable zinc concentration by a factor of two, but Screening Number S-06-0049, Revision 1, did not address this aspect of the revision. Excessive zinc additions were subsequently determined to be a primary cause of fouling.

This will be treated as an apparent violation pending final significance determination:

AV 05000528; 05000529;05000530/2006011-03, 50.59 Reviews Not Performed or Inadequate for Multiple Changes to Spray Pond Chemistry Control Procedure.

This issue was documented in CRDR 2902498. The team concluded that there were two prior opportunities to have identified that chemistry limit changes were being made without 50.59 reviews which did not identify this set of examples (CRDRs 2683642 and 2864575); these are further discussed in Section 4OA7.

(4) Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XVI, "Corrective Actions," requires that significant conditions adverse to quality shall be promptly identified, and that the cause shall be determined and corrective action shall be taken to preclude repetition. Contrary to this, on March 19, 2002, performance testing for EW Heat Exchanger 2B indicated that the system would not be capable of performing its design function, but this significant condition adverse to quality was not promptly identified, the cause determined, or corrective actions taken to restore the required heat exchanger performance. Specifically, the unacceptable performance was not promptly identified, because the test results were not correctly calculated until August 22, 2002, which was after operating mode changes and returning the unit to power following the outage. When the test results were finalized, the fact that the design basis capability was not met was not recognized or entered into the corrective action program. These failures to correct this degraded performance contributed to the continued degradation and eventual loss of function for a period of approximately 6.8 months. This will be treated as an apparent violation pending final significance determination: AV 05000528; 05000529;05000530/2006011-04, Inadequate Identification and Corrective Action for Degraded EW Heat Exchanger Performance.

This issue was entered into the corrective action program under CRDR 2897810.

(5) Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion III, requires that design control measures be established to verify the adequacy of design of structures, systems and components. A SEQUIL calculation was intended to demonstrate that the heat exchangers cooled by the emergency spray pond system would be capable of transferring the required heat load to the spray pond because scaling would not occur during a design basis accident. Contrary to this, the calculation used an improper setting which caused the calculated result to incorrectly show that scaling would not occur. This issue was documented in CRDR 2913430. This violation has the potential to increase the exposure time when EW Heat Exchanger 2B was incapable of performing its safety function, since the 6.8 month exposure time did not account for any impact because of scaling. There is uncertainty associated with the amount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of an accident scenario. This will be treated as an apparent violation pending final significance determination: AV 05000528; 05000529;05000530/2006011-05, Inadequate Design Control to Ensure No EW Heat Exchanger Scaling.

This finding does not represent an immediate safety concern because immediate and long term corrective actions that have been implemented to reduce chemical addition, remove excess impurities from the ponds, clean all the affected heat exchangers, and adjust control limits appropriately. The licensee was also in the process of removing sediment and sludge from the ponds.

d. Assessment.

The performance deficiency associated with these violations was more than minor because it impacted the equipment performance attribute of the Mitigating Systems cornerstone objective to maintain the availability and reliability of systems needed to mitigate accidents. Specifically, EW Train B in Unit 2 was estimated to have been incapable of performing its function under existing conditions for approximately 6.8 months. This was determined through straight-line extrapolation of heat exchanger performance test data. The minimum required performance for the EW heat exchanger was determined by the licensee by modifying the design basis calculation to credit the use of all known margin and evaluated the condition for the worst case pond temperature known to exist during the period of concern.

This performance deficiency affected the Mitigating Systems functions of short-term primary heat removal (high and low pressure safety injection) and long-term heat removal (emergency core cooling system recirculation). Because this performance deficiency involved the loss of function for a single train for longer than its Technical Specification allowed outage time, Phase 1 of the significance determination process required a Phase 2 evaluation.

To challenge the safety function of the EW heat exchanger, it was determined that there must be both a high heat load and a high pond temperature. The sequences of concern were determined to be small, medium and large break loss-of-coolant accidents, loss-of-offsite power, plus, stuck open primary relief valve sequences. Loss of dc Bus B was also evaluated as a consequential failure.

The performance deficiency was assumed to cause a loss of function of the EW heat exchanger. As a consequence, this has the following assumed impacts:

Loss of cooling to shutdown heat exchanger - reduced containment heat removal function credit from multi-train to single train.

Loss of cooling to the essential chiller, which fails and causes loss of room cooling and eventual failure of the following equipment:

Motor driven auxiliary feedwater pump - reduced credit from multi-train to single train Containment spray pump - reduced credit from multi-train to single train High pressure safety injection pump - reduced credit from multi-train to single train Low pressure safety injection pump - reduced credit from multi-train to Single Train EW pump - reduced credit from multi-train to single train Control room complex - no Phase 2 impact because of redundant train and non-safety cooling supply Direct current equipment room - required evaluation of the loss of dc Bus B special initiator Recovery credit was considered for the EW heat exchanger, because the condition was a degradation in capability, not a complete loss of function. The requirements in Manual Chapter 0609 for recovery credit were not explicitly met, but it was judged that normal operator training and procedures would be sufficient to maintain containment/core heat removal.

Recovery credit was considered for mitigating the effects of loss of room cooling.

The licensee had procedure steps to open doors for the affected rooms, and meet the requirements for recovery credit. It was uncertain whether this action would be effective, so this is an area where additional information is needed to make a final significance determination.

Manual Chapter 0609 directs that findings of this type be evaluated for large early release frequency (LERF) contribution. For large, dry type containment buildings (e.g., Palo Verde Nuclear Generating Station) Appendix H and associated basis document indicates that Type A findings at pressurized-water reactors that affect containment heat removal do not contribute to LERF, because they involve late containment failure probability only. Therefore, the risk of this performance deficiency is best represented by CDF.

The results of the Phase 2 SDP determined that this condition represented a risk that was potentially greater than very low risk significance (potentially greater than Green). The Phase 2 results and assumptions were reviewed by Senior Risk Analysts from Region IV and Headquarters.

A preliminary Phase 3 SDP analysis was performed. However, it was determined that several important aspects and assumptions contained considerable uncertainty or could not yet be verified. While the licensee was working on a number of evaluations and calculations to resolve the uncertainty, these were not complete. The preliminary Phase 3 analysis and modified Phase 2 using the same assumptions as the Phase 3 analysis indicated that the risk could be lower than the Phase 2 analysis, but the results were highly dependent on the uncertainty of some of the information.

In accordance with Manual Chapter 0609, NRC senior management determined that the preliminary significance would be based on the Phase 2 SDP results, pending a final significance determination which would account for additional information to be provided by the licensee. This was intended to improve the timeliness of regulatory decision-making in situations where the complexity of the significance determination would otherwise cause an excessive delay.

Information on the following aspects was still needed from the licensee in order to complete a final significance determination:

The effect on heat exchanger heat transfer capability that scaling may have played in the first 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of a design basis loss-of-coolant accident for the worst chemistry conditions that existed in each of the six spray ponds. Also, whether this impact would cause other periods of degraded performance because of fouling to challenge the capability of the system.

Whether the peak spray pond temperature would be significantly lower for loss-of-coolant accident break sizes less than analyzed in the large break design basis accident. If so, what would the resulting limiting overall heat transfer coefficient values be for medium and small break sizes given the lower peak pond temperatures.

Best available calculations (technical basis and results) for room temperature profiles during loss-of-coolant accidents with loss of room cooling to safety-related equipment because of a failed essential chiller. Also, best available information of the failure probability and basis associated with the safety related pumps affected by the loss of room cooling.

An assessment of human error probabilities for important operator actions under the specific conditions expected to be present if operator action is to be credited in reducing the risk estimation. Specifically, if the licensee plans to credit controlling EW flow through the shutdown cooling heat exchanger as a way of delaying or eliminating a loss of the essential chiller.

The technical basis for any "inadvertent cleaning" effect the licensee intends to credit. This should include an analysis of how the any test data to be used relates to the geometry and conditions in the Unit 2 EW heat exchangers (i.e.

eddy current probe size, the heat exchanger tube sleeves, normal tube diameter, thermal conductivity of remaining precipitant, etc.).

An analysis of the contribution to risk for external events associated with this issue.

5.2 Design Control Issues Three examples of a violation of 10 CFR Part 50, Appendix B, Criterion III, "Design Control," were identified. The example related to scaling calculations is discussed in Section 5.1.5.

(1) Spray Pond Allowed to Build Up Sediment

Introduction.

An example of a noncited violation of Criterion III was identified for failure to adequately translate the design basis of the spray ponds into procedures. The design basis calculation that demonstrated that the spray pond system could adequately limit spray pond temperature during a design basis accident did not account for reduced heat capacity caused by sediment. However, this was not translated into procedures to ensure that sediment did not build up in the spray ponds.

Description.

During early discussions of the design and condition of the spray ponds, it became apparent that the spray ponds had sediment along most of the bottom of each pond. It was also determined that the licensee had never drained or cleaned the ponds, nor did they monitor or quantify the sediment. The team questioned the impact the sediment had on the ponds ability to perform their intended functions.

Engineering and chemistry personnel initially stated that the ponds needed 18 inches of at the bottom of the pond to assure adequate spray pond system pump net positive suction head. This was therefore an unusable water volume. By this reasoning, this volume of water could be replaced by sediment with no impact, since sediment did not affect the net-positive suction head (NPSH). Informal reports from contract divers had indicated that sediment was about 4 inches thick on average, and did not exist near the pump suctions.

The licensee determined that a preventive maintenance basis document from the early 1990s had specified that a preventive maintenance item should be created to verify that sediment did not accumulate and impact system capability. However, this was never implemented.

Additionally, engineering personnel reviewed Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis Analysis," Revision 7, and determined that this calculation relied on both the usable and unusable water volume contained in the ponds to absorb the heat rejected during a design basis accident. Using the entire water volume yielded the result that under limiting conditions, the maximum pond temperature was maintained just below the maximum allowable temperature of 110F.

Further, the heat capacity of the sediment was about 40 percent compared to that of the water that it displaced. Therefore, the team concluded that the spray ponds were all outside the analyzed condition of the design basis.

Engineering revised the calculation to account for the quantity of sediment believed to exist in each pond. In doing so, it was necessary to credit heat transfer to the concrete in the spray pond walls, which was not previously credited. The team reviewed the calculation and determined that it used reasonable assumptions and demonstrated adequate margin.

As part of the corrective actions, the licensee began a campaign to remove and quantify the sediment from each spray pond. By the time of this report, three of the six spray ponds had been vacuumed, removing approximately 400 cubic yards of sediment from each pond. This was within the volume assumed in the calculation.

Analysis.

Failure to adequately translate the design basis of the spray ponds into procedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, the finding could become a more significant safety concern. This finding affected the Mitigating Systems cornerstone. This performance deficiency screened as having very low safety significance (Green) in a Phase 1 SDP because the licensee was able to demonstrate that the sediment would not have resulted in a loss-of-safety function.

Enforcement.

Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion III, requires that design control measures be established to ensure that design basis information is translated into instructions, procedures, and drawings. The Calculation 13-MC-SP-0307 was intended to demonstrate that the heat exchangers cooled by the emergency spray pond system would be capable of transfering the required heat load to the spray pond because scaling would not occur during a design basis accident. This calculation relied upon the entire volume of water available at the Technical Specification minimum water level to demonstrate that the pond could perform its intended safety function. Contrary to this, the licensee did not ensure that the water volume was maintained as assumed in the calculation through instructions, procedures, or drawings. This issue was documented in CRDR 2906671. This example of a violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000528; 05000529; 05000530/2006011-05a, Inadequate Design Control to Ensure Spray Pond Sediment Would Not Impact Operability.

(2) Spray Pond Integrity

Introduction.

An example of a noncited violation of Criterion III was identified for failure to adequately translate the design basis of the spray ponds into procedures. The design basis calculation, which demonstrated that sufficient water was available to provide adequate cooling during a design basis accident did not account for any leakage from the ponds. The team determined that the licensee did not translate this into a procedure to ensure that the condition of the spray pond was maintained such that leakage did not occur.

Discussion. The team reviewed all CRDRs associated with the spray pond system initiated since 2000. In CRDRs 114987, and 2886930, and CORs 00-1-001, 00-2-001, and 00-3-001, the licensee documented cracks that were observed in the accessible portions of the spray pond walls. In some instances, minor weepage through the wall was observed. The conditions were repaired.

The team reviewed the licensees treatment of the system within their maintenance rule program, and determined that it was subject to condition monitoring. Condition monitoring was performed using Procedure 81DP-0ZZ01, "Civil System, Structure, and Component Monitoring Program, Revision 11. At Palo Verde Nuclear Generating Station, structural monitoring was performed in one unit every 10 years, rather than in each unit. The program required that if problems were identified in one units structure, the similar structures in the other units would be inspected.

The team was concerned that licensees Maintenance Rule Program only required inspections of the accessible portions of the pond structures. The licensee considered that the pond bottom and inner walls were inaccessible because of the presence of water. The team concluded that the portions of the spray pond liners considered by the licensee to be accessible represented a small fraction of the structure (less than seven percent). Almost the entire portion that was subject to inspections was above the water line, were structural integrity was not needed, whereas the uninspected portion was critical to the safety function.

The team identified that the licensee had never drained any the spray ponds or conducted formal or complete inspections using divers. Further, the pond bottom constituted the majority of the pond liner surface, and this could not be inspected because the licensee had allowed approximately 4 inches of sediment to buildup on the pond bottom. Also, the licensee was not able to measure spray pond makeup, nor were they able to estimate spray pond water loss because of the large daily evaporative losses, which were highly variable. The licensee calculated that there was sufficient water maintained above the Technical Specification minimum water level to be able to tolerate a 5 gpm leak, which was small compared to the daily losses.

The licensee acknowledged the need to inspect the pond liners. This was scheduled as part of CRDR 2901589 to be performed by divers trained to perform this inspection upon completion of vacuuming in each pond. Inspections of the three completed ponds indicated no integrity problems.

The licensee performed an assessment which concluded that no significant leakage existed. This was based largely on the judgement that the geology and hydrology of the site would cause significant leakage to be observable on the surface, and no such indication had ever been observed. This was supported by recent efforts to identify the source of detectable tritium in ground water, which included drilling some holes in the vicinity of the spray ponds. Many of the holes collected no water, and none of the water that was collected had chemical results which would indicate that it included spray pond water. Also, the construction methods ensured that there were no joints that could provide leakage paths through the walls or bottom. The observed cracking was judged to be too tight to allow significant leakage.

The team reviewed the photographs from the inspections, the design and construction of the liner, and the results of ground water studies, and concluded that it was reasonable to conclude that no significant leakage existed.

Analysis.

Failure to adequately translate the design basis of the spray ponds into procedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, this finding would be a more significant safety concern. This finding affected the Mitigating Systems cornerstone. This performance deficiency screened as having very low safety significance (Green) in a Phase 1 SDP because it did not result in a loss-of-safety function.

Enforcement.

Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion III, requires that design control measures be established to ensure that design basis information is translated into instructions, procedures, and drawings.

Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis Analysis," Revision 7, which demonstrated that the system contained sufficient water to provide adequate cooling during a design basis accident did not provide any margin for leakage from the ponds. Contrary to this, the team determined that the licensee did not translate this information into an instruction, procedure or drawing to ensure that the condition of the spray pond was maintained such that leakage did not occur. This issue was documented in CRDR 2910912. This example of a violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy:

NCV 05000528; 05000529; 05000530/2006011-06b, Inadequate Design Control to Ensure Spray Pond Integrity.

5.3 Inadequate Implementation Of Operability Assessment Process

Introduction.

A Green noncited violation with multiple examples was identified for failure to follow Procedure 40DP-9OP26, "Operability Determination and Functional Assessment," Revision 16. Key support organizations were not always involving operations personnel with questions that had a potential to affect operability of safety-related equipment, or were informing operators only after the support organization had fully evaluated the condition, delaying actions that were required to be prompt by days or weeks. Also, operations personnel did not always insist on a rigorous evaluation.

This violation was determined to be of very low safety significance because it was determined to not involve a loss-of-safety function in a Phase 1 SDP evaluation. This finding has Human Performance Cross-cutting aspects associated with decision making.

Description.

Charter Item 7 required a review of operability evaluations associated with heat exchanger fouling and an assessment of the compensatory actions and initial corrective actions to address the issues.

During this review, the team noted a number of examples where the assessment of operability did not meet the NRCs expectations as delineated in Regulatory Issue Summary 2005-20 or follow the operability assessment process in Procedure 40DP-9OP26, "Operability Determination and Functional Assessment."

More specifically, the team concluded that key support organizations (engineering, chemistry, and maintenance) were not always involving operations personnel with questions that had a potential to affect operability. In a few cases, operations was informed only after the support organization had fully evaluated the condition, delaying actions that were required to be prompt by days or weeks. In a larger number of cases, support organizations decided not to involve operations. Also, operations personnel became involved but did not always insist on a rigorous evaluation. The following are the more significant examples that illustrate these conclusions:

• During a surveillance test of EDG 2B on May 17, 2006, high out of specification intercooler air temperatures were noted. The licensee failed to adequately assess this degraded condition or formally enter the operability determination process until May 19. Emergency diesel generator operability was informally discussed. The EDG system engineer assured the operations shift manager that the emergency diesel was operable based on a call with a consultant to the Cooper-Bessemer Owners Group (inappropriately represented as a representative of the vendor). The team determined that the scope of the discussions had been limited to the impact to the engine and had failed to evaluate other aspects, such as the effect of higher exhaust temperatures on turbocharger blade creep, increased fuel consumption, and load control. While these were eventually addressed after NRC inspector involvement, this initial informal and undocumented operability assessment allowed the licensee to remove the opposite train from service when EDG 2B was degraded and the impact was not properly understood. When operability was properly addressed on May 19, it was decided that EDG 2B should not have been considered to have a reasonable expectation of operability.

• On May 19, 2006, Work Order 2896333 was initiated to inspect and clean the EDG 2B intercoolers. During review of this work order following EDG 2A restoration, the on-shift operators determined the high temperatures warranted an operability determination. Condition Report/Disposition Report 2896661 initially concluded that the condition was isolated to EDG 2B. The NRC inspectors identified that this ignored fouling in both EW heat exchangers on the common spray pond system side that had necessitated cleaning that same week to be able to assure operability through the coming hot summer months.

However, the team noted that operations did correctly decide to run all EDGs in accordance with Technical Specification Surveillance Requirement 3.8.1.2 to assess their operability. Additionally, the inspectors noted that the initial evaluation assessed performance using the existing pond and air temperatures, rather than using the more limiting design basis conditions; when this was included, the degradation was shown to potentially exceed the design basis temperature limit during accident conditions in two additional EDGs. Following prompting from NRC inspectors, on May 24, 2006, the licensee appropriately expanded the scope of their evaluation. Additionally, the team noted that the licensee initiated CRDR 2898120 that states, in part, "It appears that trending and monitoring and/or extent of condition considerations of the essential water heat exchanger fouling condition should have provided for identification of the intercooler fouling condition and prevented the emergent condition discovered on May 17, 2006."

• The Prompt Operability Determination that was issued on June 3, 2006, concluded that heat exchangers were adversely affected by a low temperature fouling mechanism that was already known. The team concluded that this initial conclusion was based on an incorrect understanding of the situation that dated back to 2003, and was the basis of the decision to stop adding dispersant to the spray ponds. This was eventually shown to be an inappropriate compensatory action. However, when a technical evaluation of facts progressed, the cause assessment correctly triggered a reassessment of operability and more appropriate compensatory measures.

• On June 10, 2006, the licensee identified the failure to implement routine preventative maintenance tasks to remove biological fouling agents, corrosion products, and sedimentation from the essential water cooling spray ponds. The concern was documented in CRDR 2901737 and was determined to not meet the procedural requirements for contacting the control room operators as an operability concern based on the incorrect conclusion that the failure to implement this task only affected the preventative maintenance process, even though it was known that the ponds had several inches of sediment buildup.

This issue should have been reviewed by the control room operators in accordance with procedural requirements because the failure to remove debris from the ponds could adversely affect the safety function of the ultimate heat sink based on water displacement and heat transfer considerations. The team concluded that the reviewers of this CRDR prevented a timely review of operability. This technical issue is further discussed in Attachment 4.

• On June 10, 2006, the NRC inspectors discussed with the licensee concerns that sedimentation buildup of the essential water spray ponds could potentially affect operability of the ultimate heat sink. On June 21, 2006, licensee engineering personnel provided the inspectors a calculation revision demonstrating that any sedimentation accumulation had placed the ultimate heat sink outside its analyzed condition for assuring the peak pond temperature limits could be met.

The inspectors noted that engineering should have recognized that all three units were outside their analyzed condition, but chose to evaluate the condition before discussing the operability concern with operations (CRDR 2906487 was written with June 28 listed as the discovery date, and CRDR 2906671 was written for the untimely identification of the issue). The team concluded that site personnel lacked a questioning attitude and technical rigor while evaluating this degraded condition since the accumulation of sedimentation had been known for years at the site without documenting or evaluating the possible impact to operability until questioned by the NRC.

• The team questioned whether the spray ponds had leakage which could challenge their ability to maintain enough water to perform their intended safety function during a design basis accident. This question was raised when it was identified that there was a history of cracking in the ponds concrete liners, and that the ponds were receiving inadequate inspection and monitoring (see Section 4.2.3). The team concluded that an operability assessment was required by the licensees procedure for this condition. The licensee wrote CRDR 2910912 and documented many of the points which would have been required in an operability assessment, although a discovery checklist was inexplicably used to make the case that an operability determination was not required. This document reiterated the conclusion of a 2000 assessment in addressing the structural integrity function of the concrete, rather than the specific concern raised by the NRC team, which was water loss. The team was told by the engineering supervisor that performed the evaluation that a hole of considerable size would be required to allow enough leakage to cause a loss of operability, so his evaluation had not addressed this aspect. The team noted that this evaluation was based on engineering judgement, was not complete in addressing the functions affected, and was based on a perception of lack of evidence of leakage, rather than a solid body of facts. The team found that operations had exhibited a questioning attitude, and when it was noted that the evaluation had not addressed the leakage aspect, and operations had performed their own calculation to show that only 5.3 gpm leakage could be tolerated without losing operability, and this small margin was made possible by crediting the amount of water normally maintained above the Technical Specification minimum water level. The team noted that this practice was reasonable for an operability assessment, but did not address the design problem of having no design margin reserved for realistic leakage values.

Assessment. Failure to adequately implement the operability assessment process was a performance deficiency. This finding was more than minor because it impacted the equipment performance attribute of the Mitigating Systems cornerstone objective to maintain the availability and reliability of systems needed to mitigate accidents. This finding screened as having very low safety significance in a Phase 1 SDP, since it was confirmed not to involve any loss-of-safety function. This finding had cross-cutting aspects in the area of human performance because the licensee did not follow their systematic process for operability decision making when information was not brought to the right decision makers.

Enforcement.

Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion V, "Instructions, Procedures and Drawings," requires that activities affecting quality shall be prescribed by instructions, procedures, or drawings, and shall be accomplished in accordance with those instructions, procedures, and drawings. The assessment of operability of safety-related equipmnet needed to mitigate accidents was an activity affecting quality, and was implemented by Procedure 40DP-9OP26, "Operability Determination and Functional Assessment," Revision 16. Failure to follow Procedure 40DP-9OP26, "Operability Determination and Functional Assessment,"

Revision 16, to properly and promptly evaluate operability constituted a violation of Criterion V. This issue was documented in CRDRs 2918892, 2901815, and 2898237.

This violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000528; 05000529;05000530/2006011-07, Multiple Examples of Failure to Properly Implement Operability Assessment Process.

5.4 Examples of Inadequate Response Under the Corrective Action Program

Introduction.

Section 5.1.4 briefly discusses an apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions." This section provides more details and specific examples which demonstrate a pattern of behavior which did not recognize and address degraded heat exchanger performance between 1995 and May, 2006. Fouling and degradation of these heat exchangers and cracking in the spray pond structure was apparent from inspections, performance tests, and surveillance tests, but the licensee failed to enter most instances into the corrective action program. Of the instances that were documented, the licensee failed to recognize that some instances constituted a significant condition adverse to quality, failed to determine the cause, and failed to implement corrective actions to prevent recurrence.

Description.

The Palo Verde Nuclear Generating Station corrective actions program is documented in Procedure 90DP-0IP10, "Condition Reporting," with Revision 28 being the current revision at the time of the inspection. Section 5.1.4 documented examples where the licensee failed to properly implement the corrective action program that directly contributed to a loss of safety function in one train that had potentially greater than very low safety significance. The team identified the following additional examples of a violation where the safety significance was determined to be very low.

Specific examples where the licensees corrective action program was ineffective include:

• Emergency diesel generator intercooler degradation was evident from available documentation starting in 1995, but was not recognized until January, 1999 when it was documented in CRDR 36287. After that, the team identified no other problem reporting in the corrective action program associated with this ongoing EDG intercooler fouling until May, 2006. Instead, engineering attempted to manage the problem outside the corrective action program by repeatedly scheduling cleanings when degradation reached a threshold which was not formally established. This was determined to be an example of inadequate problem identification for a condition adverse to quality.

• Problem reporting for EW heat exchanger fouling was incomplete and appears to have been based on a relative departure from "expected fouling" instead of from an absolute reference. In the early years after implementing the zinc-phosphate chemistry control program, the team noted that EW heat exchanger performance tests measured up to 75 percent excess capacity, and on paper, these components had more than 100 percent excess capacity. By 2004, engineering was considering 30 percent excess capacity to be "expected."

This amounted to tolerating significant performance loss without reporting it into the corrective action program. The following examples associated with the EW heat exchangers degraded performance illustrate this performance deficiency:

Heat Performance Date Exchanger Margin Comments 4/96 2B 3.2% Untimely identification (CRDR 80645 initiated 4 months after test).

9/02 1B 3.6% No CRDR written.

4/03 3A -22.0% Condition Report/Disposition Request 2598216 failed to identifying that this was a significant condition adverse to quality, address the cause, or implement corrective actions to preclude repetition.

10/03 2A 1.0% Condition Report/Disposition Request 2653867 addressed fouling and the informally attempted to determine the cause, but failed to conclude it was a significant condition adverse to quality when it was recognized that the problem affected multiple systems in all trains and all units.

2/04 1B -17% No CRDR written, no cleaning or corrective action, so condition was allowed to further degrade until refueling outage in 5/04. This constituted a significant condition adverse to quality that was not entered into the corrective action program, corrected, or a cause determined.

3/04 2A 28.6% No CRDR written. This constituted a condition adverse to quality, if considered individually.

5/04 1A 9.9% No CRDR written. This constituted a condition adverse to quality, if considered individually.

4/05 2B 25.8% No CRDR written. This constituted a condition adverse to quality, if considered individually.

5/05 2A 29% No CRDR written. This constituted a condition adverse to quality, if considered individually.

10/05 2B 11.1% No CRDR written. This constituted a condition adverse to quality, if considered individually.

10/05 3B 19.3% No CRDR written. This constituted a condition adverse to quality, if considered individually.

10/05 1B 1.4% Condition Report/Disposition Request 2835865 written. In 11/05, CRDR 2860763 was written and took 3 months to address EW heat exchanger fouling in all units. Not classified as significant condition, and did not consider EDG impact. Operability Determination performed for performance test results in 10/05 was not completed until 1/06.

Note that 30 percent margin was used as the threshold by the team in assessing when sufficient degradation had occurred to constitute a condition adverse to quality, to be consistent with the licensees stated expectations. It could be argued that a higher threshold could be used, since no degradation should be present if the chemistry control program was performing as intended. However, it was considered sufficient to document the above examples to illustrate the extensive pattern of behavior that is being addressed in this violation.

• In June, 2006, engineering personnel reviewed historical performance testing results and noted that EW Heat Exchanger 2B may have been significantly degraded in 2002-2003. This was not entered into the corrective action program for a week, and not until the team asked. Condition Report/Disposition Request 2905161 was written to document this issue.

• The corrective action process was ineffective in determining the significance of the fouling. The team determined that there was enough evidence to have recognized that the fouling problems affected both trains and all units as early as 1999, which should have raised the issue to a "significant" classification, but this did not occur because the licensee did not adequately consider the extent of condition or identify the cause.

• In December 2005, when it was recognized that the fouling was occurring faster in EW heat exchangers to the point where their capacity would be assured only during cooler weather, some of these heat exchangers were taken out of service while the associated plant was on line to perform midcycle cleanings. However, the fact that the EDG coolers should also be expected to be experiencing the same degradation was not considered. The worsening of the degradation rate did not trigger a reassessment of the cause or significance.

• In May 2005, chemistry wrote CRDR 2800653 to report that molybdenum analyses yielded erratic results. They concluded that the analysis method was not sufficiently accurate (actual accuracy was estimated at 30-50 percent) to maintain it in the desired control band (225 - 270 PPB). Chemistry began to question whether the molybdenum tag was accomplishing what they wanted it to and requested an evaluation of alternatives. The licensee also identified that samples were untimely because of lack of qualified analysts (would sometimes wait 2 to 3 weeks to analyze samples). The CRDR also documented that they had stopped adding dispersant in November 2005, but still recorded increases in molybdenum or other unexpected results in over half of the samples. The team concluded that this represented a missed opportunity to recognize that they were not adding dispersant, and that dispersant concentration was no longer coupled with the tag chemical. The evaluation of alternatives considered other tag chemicals, but did not consider other dispersants which could be measured. The team noted that this CRDR was closed with the only change being to qualify more analysts to perform the molybdate analysis.

Assessment. The significance of this finding will be assessed as part of the final significance determination. See Section 5.1 for the assessment of this finding.

Enforcement.

See Section 5.1.4 for the enforcement discussion associated with this finding.

6.0 ADDITIONAL INFORMATION 6.1 Summary of Chemistry Control Changes There were three distinct periods of interest in this inspection. From 1994 to 2001, the licensee used Nalco (later bought by Calgon) chemicals. In 2001, the Arizona Public Service corporate system changed to Betz chemicals. Palo Verde Nuclear Generating Station changed to what was believed to be equivalent chemicals, but maintained essentially the same chemistry control program for the spray ponds. In 2004, the dispersant was changed because it was thought that the dispersant was reacting with the biocide and forming an acrylic polymer and fouling heat exchangers.

The licensee implemented zinc-phosphate chemistry control in the SP systems in 1994.

The proprietary specialty chemicals used were:

Dispersant: CALGON PCL-401 (AA/AMPS)

Corrosion inhibitor: CALGON MSW-109 (zinc chloride in phosphoric acid (3:1)

Biocide: BULAB 6002 Chemical supplier: NALCO/CALGON Initially, zinc limits were 1.0 - 1.5 ppm, the pH band was 7.6-8.2, and dispersant was added three times per week.

Between 1995 and 1999, some degraded performance was noted in EDG intercooler outlet temperatures. This was not documented until 1999. Also, in 1995, the spray pond system began to have corrosion problems where localized corrosion nodules formed and broke off, blocking heat exchanger tubes.

On 4/24/96, the pH band was lowered from 7.6 - 8.2 to 7.4 - 8.0.

In 1997, the licensee attempted to chemically passivate spray pond piping by significantly lowering pH to remove corrosion products, then reestablish a protective zinc layer. This was unsuccessful.

On 3/3/98, pH specification was raised from 7.4 - 8.0 to 7.6 - 8.6. On 8/13/98, the dispersant addition time was increased from 2.5 to 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br />.

On 9/18/98, the licensee split the requirements so that Unit 1 had different limits, since they were changing Unit 1 so makeup water came from wells. Since this water had higher alkalinity and thus a higher scaling potential, Unit 1 was given a lower pH band to improve solubility and avoid scaling. Unit 1 had a pH specification of 7.8 - 8.2, while Units 2 and 3 had 8.0 - 8.4. However, on 7/6/99, this was reversed, and all units had a pH specification of 8.0 - 8.4. No explanation was given why the increased potential for scaling was acceptable.

On 7/11/00, the upper zinc specification was changed from 1.0 to 1.5 ppm.

In 2001, the corporate chemistry group for Arizona Public Service changed chemical supply companies. On 5/30/01, the licensee began adding the following new chemicals:

Dispersant: BETZ PY5200 (AA/HPS-1)

Corrosion inhibitor: BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002 Chemical supplier: BETZ In October 2003, the licensee performed the first EW heat exchanger tube cleaning in

2R11 in response to fouling. Since then, 100 percent of the EW heat exchanger tubes

are cleaned every refueling outage.

On 1/29/04, the licensee concluded from informal lab testing that EW heat exchanger fouling was being caused by an undesirable interaction between the dispersant and the biocide. This was based on discussions with the chemical suppliers site representative, and included informal lab tests which showed that a white substance could be formed by mixing the chemicals, although the conditions of this test were not documented and the substance formed was not analyzed. The team noted that the conclusions drawn from this informal test dominated the licensees response through June, 2006.

In March 2004, the dispersant was changed to address EW heat exchanger fouling.

The chemicals were now:

Dispersant: BETZ DN2317 (AA/APES)

Corrosion inhibitor: BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002 Chemical supplier: BETZ The target dispersant concentration was 12.5 - 15 ppm. The previous dispersant was not removed from the ponds through feed and bleed, but was expected to continue to react and be removed by spray pond sand filters. There was no evaluation of adding a new dispersant with the previous dispersant still present, or an analysis of the acceptability fo the new dispersant for the system (LIST CRDR). Also, the zinc specification was doubled from 0.5 - 1.5 ppm to 1.0 - 3.0 ppm.

In November 2004, a white "lotion-like" substance identified in Spray Pond 2C sand filter. The cause was believed to be having the old dispersant and the new dispersant, polymerizing and coagulating into larger particles (essentially doing what they were supposed to do, but at a higher rate). It was not believed to be a problem for heat exchangers because of the mobility of the substance and most recent EW heat exchanger performance test showed improving trend (the team noted that this was actually after a midcycle cleaning with only 6 months of operation, so it was not useful to consider this data). The licensee implemented more frequent backwashing of spray pond sand filters, believing that this would remove the problem chemical. Chemistry explained that this took 10 months to happen after changing dispersants by saying it was because of cooler temperatures in the pond. However, the team noted that this did not take into consideration that the dispersant change took place in January when the ponds were also cold.

In May 2005, chemistry wrote CRDR 2800653 to report that molybdenum analyses yielded erratic results. They concluded that the analysis method was not sufficiently accurate (actual accuracy was estimated at 30-50 percent) to maintain it in the desired control band (225 - 270 PPB). Chemistry began to question whether molybdenum tag was accomplishing what they wanted it to and requested an evaluation of alternatives.

The licensee also identified that samples were untimely because of lack of qualified analysts (would sometimes wait 2 to 3 weeks to analyze samples). The CRDR also documented that they had stopped adding dispersant in November 2005, but still recorded increases in molybdenum or other unexpected results in over half of the samples. The team concluded that this represented a missed opportunity to recognize that they were not adding dispersant, and that dispersant concentration was no longer coupled with the tag chemical. The evaluation of alternatives considered other tag chemicals, but did not consider other dispersants which could be measured. The team noted that this CRDR was closed with the only real change being to qualify more analysts to perform the molybdate analysis.

On 10/17/05, the pH specification was changed from 8.0 - 8.4 to 7.8 - 8.4 in order to improve chlorine effectiveness and zinc solubility.

The team noted that between October 2005 and May 2006, both spray ponds in Units 2 and 3 had molybdate results that were above the target level of 200-300 ppb almost continuously, even though dispersant was largely not being added. The validity of these results was not questioned. The team considered that this was imprudent, since evaporation required adding 2,200 to 10,000 gallons of makeup water per day if the pond was idle, more if the sprays were run. Of note, Unit 1 spray ponds were maintained within the target band during most of this period. The team reviewed the dispersant usage by assessing tank levels in each unit. Between October 2004 and August 2005 little or no dispersant was added to any of the ponds. From August 2005 until May 2006, dispersant was added fairly regularly in Unit 1, less regularly in Unit 3, and infrequently in Unit 2. This was known to the chemistry line organization, but not to management or the root cause team until the inspection team requested the information. The team also compared the recent dispersant usage to earlier usage.

Since changing to the latest dispersant in March 2004, the total dispersant used in 27 months was about equal to the annual used in years prior to that change. The lack of regular additions coupled with rising concentrations of other chemicals tended to indicate that there was increasingly insufficient dispersant to prevent loss of solubility in the spray pond systems.

The team assessed the history of differences in chemistry parameters between the units. The team noted that Unit 2 consistently had the highest levels of phosphates, and the pH was above 8.2 for long periods of time compared to the other units. Unit 1 had roughly double the calcium compared to the other units; this was known to be because of using well water in this unit.

7.0 SUMMARY OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE 1. Sequence of Events This topic is documented in Attachment 4.

2. Operating Experience Personnel from the Operating Experience Branch of NRR supported the team by performing searches of operating experience databases and other sources. The intent was to identify operating experience report of similar problems and other relevant information, both within and outside the nuclear industry. Palo Verde Nuclear Generating Station had one instance of tube blockage, which was documented in Licensee Event Report 50-528/95-005-00, "Corrosion Nodules Blocking EDG 1B Jacket Water Cooler Tubes." This involved chemistry challenges in the spray pond system, but did not involve precipitation or scaling. Only two reports of problems with similar circumstances were identified, although the causes were different. It was noted, however, that outside the nuclear industry, this type of condition would probably not be considered to be serious. However, these searches identified information that indicated that zinc-phosphate chemistry control programs similar to the one used by Palo Verde Nuclear Generating Station were in common use in similar systems, and were used effectively.

3. Generic Issues The chemical control program in use in the Palo Verde Nuclear Generating Station spray ponds and associated systems were generally compatible when used together in a coordinated program. This type of program was in common use and effectively employed in similar applications. However, the unique design of the spray pond system and the unique operating practices were not adequately reviewed to ensure compatibility.

The only generic issue identified was that AA/HPS-1 type dispersant is know to interact with polyquat amine-type biocide and form an acrylic polymer that can cause water problems, including possibly fouling. The acrylic acid in this product has a strong negative charge compared to other dispersants, and will combine with the biocide.

4. Corrective Actions During this inspection, the licensee corrected the limits for the spray pond in the chemistry control program, and restored the ponds to within the new limits by blowing down the ponds. They were also evaluating changing the dispersant to regain the ability to monitor it. The ponds were being vacuumed and inspected to remove chemical sludge and sediment, and to verify pond integrity. Part 50.59 reviews were planned for past changes to chemistry procedures. Emergency diesel generator intercoolers were promptly cleaned, and increased frequency monitoring of performance was implemented while chemistry improvements were being identified and implemented.

5. Root Cause Determination.

The licensees determination of the cause was broad and thorough in assessing the problems and the history. This effort included looking at both the technical issues and the organizational aspects, which was appropriate. Adequate resources were devoted this effort, including obtaining several sources of outside expertise in a number of important aspects of the evaluation. The draft report, although awaiting some technical information, thoroughly documented the facts available, the process used, the possible causes considered, and the conclusions. This effort also identified a number of peripheral issues that were entered into the corrective action program.

6. Design Basis and Technical Specifications Compliance

A number of problems were identified with the design basis and Technical Specification compliance. These are discussed in the Findings section.

7. Operability Assessments and Compensatory Measures

A number of problems were identified with operability assessments. These are discussed in the Operability Assessment section. The compensatory measures and initial corrective actions implemented by the licensee were appropriate and timely, with the exception of stopping dispersant additions. In retrospect, this was inappropriate based on the later understanding of the cause of the fouling; however, from the perspective of what was known at the time, this was not unreasonable. Dispersant additions were resumed in a prompt manner when the cause was better understood.

8. Heat Exchanger Testing

The test methodology was reviewed along with several years worth of test results. The methodology was in accordance with industry standards, although the calculations provided results which were not typical. Specifically, the heat exchanger performance was calculated for the existing conditions, and the acceptance criteria (reflecting design basis conditions) were extrapolated to the existing condition results. The normal practice is to extrapolate the results from the existing conditions to design basis conditions. The team concluded that, while the licensee's practice made comparison of results technically awkward, this did not affect the ability to demonstrate proper performance. The team also identified that the licensee's practice of having the test and the calculation as separate activities contributed to having untimely results from the performance tests in at least one case, and possibly others.

9. Risk Analysis

Information was gathered to support a preliminary risk assessment. However, key information was still needed to be able to perform a final risk analysis. This is further discussed in the Findings section under the Analysis heading.

10. Methods of Heat Exchanger Protection The team assessed the chemistry control program design and implementation, operating practices, inspection and cleaning methods and results, and the cathodic protection used in portions of the system. As discussed in various parts of this report, the chemistry control program was changed over time in ways that created heat exchanger fouling. Corrosion control was being effectively implemented, however.

11. Effectiveness of Heat Exchanger Performance Monitoring As discussed in various parts of this report, performance monitoring was not effective in identifying and correcting heat exchanger degradation. The tests and inspections were being done according to commitments, but the results that indicated increasing amounts of chemical residue, and the need for increasing frequent cleanings, were not followed up to determine and correct the cause. This was an organizational problem, rather than a problem with the monitoring methods or frequencies.

1R07 Biennial Heat Sink Performance

a. Inspection Scope

The team reviewed design documents (e.g., calculations and performance specifications), program documents, implementing documents (e.g., test procedures, maintenance procedures, and actual data), and corrective action documents. The team interviewed chemistry personnel, maintenance personnel, engineers, and program managers.

The team verified whether testing, inspection and maintenance, or the biotic fouling monitoring program provided sufficient controls to ensure proper heat transfer.

Specifically, the inspectors reviewed heat exchanger test methods, test results from performance testing, inspection results, and chemical controls to limit fouling.

For the ultimate heat sink and its subcomponents, the team reviewed the heat sink to determine if it was free from clogging because of macrofouling and provided sufficient controls to ensure proper heat transfer. The inspectors reviewed;

(1) heat exchanger test methods and test results from performance tests,

(2) heat exchanger inspection and cleaning methods and results, and

(3) chemical treatment for the spray ponds to control fouling. The team selected the following heat exchangers for this inspection:

• Essential cooling water heat exchanger

• Emergency diesel generator jacket water heat exchanger

• Emergency diesel generator inter-cooler The team completed 3 of the 2 to 3 required samples.

b. Findings

An apparent violation of Criterion XI, "Test Control," associated with EW heat exchanger testing was discussed in Section 5.1.2. A minor violation of 10 CFR Part 50, Appendix B, Criterion III, was identified for using an inappropriate fouling factor in design documents for the EDG intercoolers. The team identified that fouling factors for clean water (0.0005) was used on both sides of these heat exchangers, where fouling factors of 0.002 was appropriate for the application.

The latter value was correctly used in the other EDH heat exchangers. This was determined to be minor because the team calculated that this had a small impact (4 percent) to the point at which these heat exchangers would not be able to perform their intended function, compared to the large margins available in the EDG system. This minor issue was entered into the corrective action program as Action Request 051201123.

4OA6 Meetings, Including Exit

On August 18, 2006, the inspectors presented the inspection results to Mr. C. Eubanks, Vice President, Nuclear Operations, and Mr. D. Mauldin, Vice President, Engineering, and other members of his staff, who acknowledged the findings.

An additional exit was conducted on September 26, 2006, to provide the results of preliminary significance determination and to change the characterization of the associated issues to Apparent Violations. The findings were discussed with Mr J.

Levine and members of his staff, who acknowledged the findings.

The inspectors confirmed that proprietary information that was examined during this inspection was properly handled in accordance with NRC policy.

4OA7 Licensee-Identified Violation

The following violation of very low safety significance (Green) was identified by the licensee and was a violation of NRC requirements which meets the criteria of Section VI of the NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs.

• Part 50,59 of Title 10 Code of Federal Regulations allows licensees to make changes to procedures described in the Updated Final Safety Analysis Report without prior NRC approval provided that the change satisfies the criteria in Part 50.59.c. Contrary to this, Revisions 2, 6, 7, 8, and 9 to Procedure 74DP-0CY01, "Specification for Bulk Chemicals," were made between 1997 and 2004 without 50.59 reviews. This was identified in the licensees corrective action program under CRDRs 2864575 and 2683642. This violation is of very low safety significance because it did not actually impede the regulatory process or contribute to degraded heat exchanger performance.

ATTACHMENT 1:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

G. Andrews, Department Leader, System Engineering

P. Borchert, Director, Operations

D. Breckenridge, Consulting Engineer, Arizona Public Service

P. Carpenter, Unit Department Leader, Operations

C. Churchman, Director, Engineering

A. Dave, Senior Engineer, Design Engineering

C. Eubanks, Vice President, Nuclear Operations

T. Green, Team Leader, Chemistry

D. Hautala, Senior Compliance Engineer

P. Heinstein, Chemical Vendor Site Representative, GE Betz

J. Hughey, Senior Engineer, System Engineering

H. Hurley, Root Cause Team Sponsor

R. Jenkins, Senior Chemist, Chemistry

D. Kanitz, Senior Compliance Engineer

M. Karbasian, Department Leader, Design Mechanical Engineering

J. Levine, Executive Vice President, Generation

D. Mauldin, Vice President, Engineering

J. Proctor, Section Leader, Regulatory Affairs - Compliance

M. Radspinner, Section Leader, System Engineering

C. Seaman, General Manager, Regulatory Affairs and Performance Improvement

T. Selby, Chemistry Consultant

G. Sowers, Section Leader, PRA

D. Straka, Senior Consultant, Regulatory Affairs

D. Vogt, Section Leader, Operations/STA

NRC personnel

J. Melfi, Resident Inspector, Palo Verde Nuclear Generating Station

E. Owen, Reactor Inspector, Region IV

N. Sieller, Operations Experience Branch, NRR

W. Sifre, Senior Reactor Inspector, Region IV

R. Telson, Operations Experience Branch, NRR and Acting Resident Inspector, PVNGS

S. Unikewicz, Senior Engineer, NRR

G. Warnick, Senior Resident Inpspector, Palo Verde Nuclear Generating Station

G. Werner, Senior Project Engineer, RIV

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000529/2006011-01, AV EW Train 2B Inoperable Longer than Allowed Outage Time (Section 5.1.1)

05000528;

05000529; AV Inadequate Test Control to Promptly Identify

05000530/2006011-02 Unacceptable Performance Test Results (Section 5.1.2)

05000528;

05000529; AV 50.59 Reviews Not Performed or Inadequate for

05000530/2006011-03 Multiple Changes to Spray Pond Chemistry Control Procedure (Section 5.1.3)

05000528;

05000529; AV Inadequate Identification and Corrective Action for

05000530/2006011-04 Degraded EW Heat Exchanger Performance (Section 5.1.4)

05000528;

05000529; AV Inadequate Design Control to Ensure No EW Heat

05000530/2006011-05 Exchanger Scaling.(Section 5.1.5)

Opened and Closed

05000528;

05000529; NCV Two Examples of Failure to Translate Spray Pond

05000530/2006011-06 Design Assumptions Into Plant Procedures Control (Section 5.2)

05000528;

05000529; NCV Multiple Examples of Inadequate Operability

05000530/2006011-07 Assessments for Heat Exchanger Degradation (Section 5.3)