Text

IR 05000528-06-011, 05000529-06-011 & 05000530-06-011; on 06/6-9/26/2006; Arizona Public Service Company; Palo Verde Nuclear Plant; Special Inspection, 50.59, Operability Evaluations, Design Control, Corrective Actions, Biennial Heat Sink,
	ML062710607
Person / Time
Site:	Palo Verde
Issue date:	09/28/2006
From:	Chamberlain D D; Division of Reactor Safety IV
To:	Levine J M; Arizona Public Service Co
References
	EA-06-221 IR-06-011
	Download: ML062710607 (62)
	v • d • e

September 28, 2006

EA 06-221James M. Levine, Executive Vice President, Generation Mail Station 7602 Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034

SUBJECT: Palo Verde Nuclear Generating Station, Units 1, 2, and 3 -

NRC SPECIALINSPECTION REPORT 05000528/2006011; 05000529/2006011;

05000530/2006011

Dear Mr. Levine:

On September 26, 2006, the US Nuclear Regulatory Commission (NRC) completed a specialinspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. This inspectionexamined your staff's response to the high intake air temperature condition identified in Emergency Diesel Generator 2B on May 17, 2006, and the fouling of safety-related heatexchangers serviced by the emergency spray pond system. The NRC's initial evaluation metboth the risk criteria and several of the deterministic criteria for initiating a special inspection.

The basis for initiating this special inspection is further discussed in the Charter, which is included as Attachment 2 to this report. The enclosed special inspection report documents the inspection findings which were discussedon August 18, 2006, with Mr. C. Eubanks, Vice President, Nuclear Operations, Mr. D. Mauldin, Vice President, Engineering, and other members of your staff. A supplemental exit was conducted on September 26, 2006, with Mr. and other members of your staff to discuss the preliminary significance determination and the characterization of the performance deficiency. The determination that the inspection would be conducted was made by the NRCon June 6, 2006, and the inspection started on June 12, 2006.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel. The report documents two findings that were determined to be violations of very low safetysignificance. Also, a licensee-identified violation which was determined to be of very low safety significance is listed in this report. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If Arizona Public Service Company-2-you contest any noncited violation in this report, you should provide a response within 30 daysof the date of this inspection report, with the basis for your denial, to the U. S. Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington, DC 20555-0001; andthe NRC Resident Inspector at Palo Verde Nuclear Generating Station.In addition, the attached report discusses a finding that appears to have greater than very lowsafety significance. As described in Section 5.1 of this report, the failure to recognize that improperly implemented chemistry controls for the emergency spray pond systems in all threeunits caused degraded performance in all emergency diesel generators and emergency cooling water systems over a period of years. This finding was assessed based on the best available information, including influential assumptions, using the applicable Significance Determination Process (SDP) and was preliminarily determined to be a Greater Than Green Finding. The final resolution of this finding will convey the increment in the importance to safety by assigning the corresponding color i.e., [(white) a finding with some increased importance to safety, which may require additional NRC inspection; (yellow) a finding with substantial importance to safety thatwill result in additional NRC inspection and potentially other NRC action; (red) a finding of highimportance to safety that will result in increased NRC inspection and other NRC action]. Thisfinding has a preliminary greater than very low safety significance because it has been estimated that in Unit 2, Train B of essential cooling water was incapable of performing its safety function for 6.8 months in 2003, based on extrapolation of performance data for this heat exchanger. Accident sequences that would cause high pond temperatures, primarily loss-of-

coolant accidents, could challenge this system to the point where the essential chiller would fail,causing a loss of room cooling to important mitigating systems. In addition, there is uncertaintyassociated with the amount of scaling that could occur on any of the affected heat exchangersfor all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario.Due to the complexity of this issue, there was insufficient information for a number of importantaspects to complete a final Phase 3 significance determination evaluation in a timely manner.

During the inspection process, it was apparent that your staff is working to develop the information and calculations needed to complete an accurate assessment. Therefore, the preliminary significance was based on a Phase 2 evaluation in accordance with Inspection Manual Chapter 0609. In order to complete a final significance determination, additional information is needed. The specific information needed is described in the assessment portion of Section 5.1.This finding does not represent an immediate safety concern because of the immediate andlong term corrective actions that have been implemented to reduce chemical addition, remove excess impurities from the ponds, clean all the affected heat exchangers, and adjust control limits appropriately. You are also in the process of removing sediment and sludge from the ponds.There are five apparent violations associated with this finding being considered for escalatedenforcement action in accordance with the NRC Enforcement Policy. The current EnforcementPolicy is included on the NRC's web site at http://www.nrc.gov/reading-rm/adams.html

.In accordance with Inspection Manual Chapter 0609, we intend to complete our evaluationusing the best available information and issue our final determination of safety significance within 90 days of this letter.

Arizona Public Service Company-3-The significance determination process encourages an open dialog between the staff and thelicensee, however the dialogue should not impact the timeliness of the staffs final determination. Before we make a final decision on this matter, we are providing you an opportunity (1) to present to the NRC your perspectives on the facts and assumptions used bythe NRC to arrive at the finding and its significance at a Regulatory Conference or (2) submityour position on the finding to the NRC in writing. If you request a Regulatory Conference, itshould be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least 1 week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open forpublic observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of the receipt of this letter.Please contact Linda J. Smith at (817) 860-8137 within 10 business days of the date of thisreceipt of this letter to notify the NRC of your intentions. If we have not heard from you within10 days, we will continue with our significance determination and enforcement decision and youwill be advised by separate correspondence of the results of our deliberations on this matter.Since the NRC has not made a final determination in this matter, no Notice of Violation is beingissued for this inspection finding at this time. In addition, please be advised that the number and characterization of apparent violations described in the enclosed inspection report may change as a result of further NRC review. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letterand its enclosure will be available electronically for public inspection in the NRC PublicDocument Room or from the Publicly Available Records (PARS) component of NRC'sdocument system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,/RA/Dwight D. Chamberlain, DirectorDivision of Reactor SafetyDockets: 50-528; 50-529; 50-530Licenses: NPF-41; NPF-51; NPF-74

Enclosure:

Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011

w/Attachment:

Supplemental Informationcc w/enclosure:Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Arizona Public Service Company-4-Douglas K. Porter, Senior CounselSouthern California Edison Company Law Department, Generation Resources

P.O. Box 800 Rosemead, CA 91770ChairmanMaricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003Aubrey V. Godwin, DirectorArizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040Craig K. Seaman, General ManagerRegulatory Affairs and Performance Improvement Palo Verde Nuclear Generating Station Mail Station 7636 P.O. Box 52034 Phoenix, AZ 85072-2034Jeffrey T. WeikertAssistant General Counsel El Paso Electric Company Mail Location 167 123 W. M illsEl Paso, TX 79901John W. SchumannLos Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100John TaylorPublic Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224Thomas D. ChampSouthern California Edison Company 5000 Pacific Coast Hwy, Bldg. D1B San Clemente, CA 92672 Arizona Public Service Company-5-Robert HenrySalt River Project 6504 East Thomas Road Scottsdale, AZ 85251Brian AlmonPublic Utility Commission William B. Travis Building P.O. Box 13326 1701 North Congress Avenue Austin, TX 78701-3326Karen O'ReganEnvironmental Program Manager City of Phoenix Office of Environmental Programs 200 West Washington Street Phoenix, AZ 85003 Matthew BenacAssistant Vice President Nuclear & Generation Services El Paso Electric Company 340 East Palm Lane, Suite 310 Phoenix, AZ 85004 Arizona Public Service Company-6-Electronic distribution by RIV:Regional Administrator (BSM1)DRP Director (ATH)DRS Director (DDC)DRS Deputy Director (RJC1)Senior Resident Inspector (GXW2)Branch Chief, DRP/D (TWP)Senior Project Engineer, DRP/D (GEW)Team Leader, DRP/TSS (RLN1)RITS Coordinator (KEG)DRS STA (DAP)V. Dricks, PAO (VLD)J. Lamb, OEDO RIV Coordinator (JGL1)ROPreports PV Site Secretary (PRC)K. S. Fuller, RC/ACES (KSF)C. A. Carpenter, D:OE (CAC)OE:EA File (RidsOeMailCenter)SUNSI Review Completed: LJS ADAMS: # Yes G No Initials: LJS

Publicly Available G Non-Publicly Available G Sensitive

Non-SensitiveR:\_REACTORS\_PV\2006\PV2006011rp-nfo.wpdEB2/TLEB2NRRNRRSRAEB2/BCCFO'Keefe/larPAGoldbergYDiaz-CastilloKParczewiskiMFRunyanLJSmith/RA//RA/T=CFOT=CFODPL for/RA/

09/26/0609/27/0609/26/0609/26/0609/28/0609/28/06PBD/BCACESOENRREB2/BCD:DRSTPruettMSHaireRFretzMTschiltzLJSmithDDChamberlainT=CFOT=CFOT=CFOE=CFO/RA//RA/

09/27/0609/27/0609/27/0609/28/0609/28/0609/28/06OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax ENCLOSUREU.S. NUCLEAR REGULATORY COMMISSION REGION IV Dockets:50-528; 50-529; 50-530Licenses:NPF-41; NPF-51; NPF-74 Report No.:05000528/2006011; 05000529/2006011; 05000530/2006011 Licensee:Arizona Public Service Company Facility:Palo Verde Nuclear Generating Station, Units 1, 2, and 3Location:5951 S. Wintersburg Road Tonopah, Arizona Dates:June 6 through September 26, 2006 Inspectors:N. O'Keefe, Team LeaderP Goldberg, Reactor Inspector, Engineering Branch 1 Y. Diaz-Castillo, Chemical Engineer, Office of Nuclear Reactor Regulation (NRR)K. Parczewiski, Senior Chemical Engineer, NRR M. Runyan, Senior Reactor AnalystApprovedBy:Dwight D. Chamberlain, Director Division of Reactor Safety ii

SUMMARY

OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE.........341R07Biennial Heat Sink Performance (71111.07B)...............................364OA6Meetings, Including Exit...............................................37 4OA7Licensee-Identified Violation............................................37ATTACHMENT 1: SUPPLEMENTAL INFORMATION..............................38KEY POINTS OF CONTACT..........................................A1-1 LIST OF DOCUMENTS REVIEWED....................................A1-2ATTACHMENT 2: SPECIAL INSPECTION CHARTER...........................A2-1ATTACHMENT 3: SUMMARY OF HEAT EXCHANGER PERFORMANCE TEST RESULTSA3-1ATTACHMENT 4: SEQUENCE OF EVENTS..................................A4-1 1SUMMARY OF FINDINGSIR 05000528/2006011; 05000529/2006011; 05000530/2006011; 6/6-9/26/2006; Palo VerdeNuclear Generating Station, Units 1, 2, and 3: Special Inspection, 50.59, OperabilityEvaluations, Design Control, Corrective Actions, Biennial Heat Sink, Test Control. The report covered a 3-month period of inspection by two region-based inspectors and twochemistry subject matter experts from headquarters. In addition to performing a special inspection, this report includes the results of the announced biennial heat sink inspection, which was performed concurrently. One finding was identified with five apparent violations, which were determined to have potential safety significance greater than very low safety significance.

Additionally, two NRC-identified noncited violations and one licensee-identified noncitedviolation with very low safety significance were identified. The significance of most findings is indicated by its color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609,

"Significance Determination Process." Findings for which the significance determination process does not apply may be green or be assigned a severity level after NRC managementreview. The NRC's program for overseeing the safe operation of commercial nuclear powerreactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, datedJuly 2000.Summary of Event and Inspection ResultsThe NRC conducted a special inspection to better understand the circumstances surroundingthe heat exchanger fouling at Palo Verde Nuclear Generating Station. The fouling came to theNRC's attention as a result of unusual temperatures noted during a surveillance test ofEmergency Diesel Generator 2B conducted on May 17, 2006. After 2 days, it was determined that this potentially impacted the ability of this safety-related system to perform its safety function during a design basis accident. It also became apparent that the fouling that was experienced had existed for a long period of time. The cause of the fouling was not fully understood, and affected other equipment in all trains of all three units. In accordance with NRC Management Directive 8.3, it was determined that this event had sufficient risksignificance and met several deterministic criteria to warrant a special inspection. This inspection identified a pattern of behavior where heat exchanger performance data andinternal inspection results were not understood, entered into the corrective action program, or adequately assessed for operability impact. When the degradation increased and theappearance of the foulant changed, this did not cause an increased response, other than toclean heat exchangers more frequently. The chemistry control credited to prevent fouling was determined to have no mechanism to actively monitor fouling. Further, the chemistry control limits were changed over time to actually promote fouling and increase the potential for scaling during design basis accidents. This pattern of behavior existed for the last 10 years.A.

NRC-Identified and Self Revealing Findings

Cornerstone: Mitigating Systems

TBD. A finding with five apparent violations was identified associated withfouling of safety-related heat exchangers cooled by the emergency spray pond system. Between 1995 and May, 2006, the licensee failed to recognize that 2improperly implemented chemistry controls for the emergency spray pondsystem caused a significant condition adverse to quality which degraded theperformance of all emergency diesel generators and emergency cooling water systems. The degraded performance was primarily due to heat exchangerfouling caused by adding excessive amounts of chemicals. The conditions that existed also had the potential to cause scaling after an accident starts. In one instance, it is estimated that this resulted in degrading the performance of Emergency Cooling Water Heat Exchanger 2B to the point where it would not have been capable of performing its intended safety function for approximately 6.8 months in 2003.An apparent violation of Technical Specification 3.7.7 was identifiedbecause Train A of the Essential Cooling Water System in Unit 2 was not capable of performing its safety function for approximately 6.8 months ending on September 27, 2003. An apparent violation of 10 CFR Part 50, Appendix B, Criterion XI, "TestControl," was identified because the two procedures that were performed to measure essential cooling water heat exchanger performance were implemented in a way that was inadequate to ensure the timely determination that the requirements and acceptance limits contained inapplicable design documents were met. An apparent violation of 10 CFR 50.59 was identified for making ninerevisions to Procedure 74DP-9CY04, "System Chemistry Specification," a procedure described in the Updated Final Safety Analysis Report, between 1998 and 2004 without performing evaluations of the potentialimpact of the changes on the safety-related components in the spray pond system; the changes revised spray pond chemistry parameter limitswhich were subsequently determined to have contributed to heat exchanger fouling.An apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI,"Corrective Actions," was identified. On March 19, 2002, performance testing for Essential Cooling Water Heat Exchanger 2B indicated that the system would not be capable of performing its design function, but thissignificant condition adverse to quality was not promptly identified, the cause determined, or corrective actions taken to restore the required heat exchanger performance. The failure to correct this degraded performance contributed to the continued degradation and eventual lossof function for an estimated period of 6.8 months.An apparent violation of 10 CFR Part 50, Appendix B, Criterion III,"Design Control," was identified for failure to correctly evaluate the scaling potential of the safety-related heat exchangers cooled by the emergency spray pond during a design basis accident. An error in the SEQUIL calculation caused the licensee to incorrectly conclude that scaling would not occur under the conditions established in the chemistry control program.

3The performance deficiency associated with these apparent violations was morethan minor because it impacted the equipment performance attribute of the Mitigating Systems Cornerstone objective to maintain the availability andreliability of systems needed to mitigate accidents. Specifically, EssentialCooling Water (EW) Train B in Unit 2 was estimated to have been incapable of performing its function under existing conditions for approximately 6.8 months. A Phase 2 significance determination process concluded that this finding has potential safety significance greater than very low safety significance because some accident sequences, notably loss of coolant accidents, were expected to elevate the ultimate heat sink temperature to the point where the degradedessential cooling water heat exchanger would be challenged. This was expected to cause failure of the essential chiller, and the resulting loss of room cooling tosafety-related equipment increased the plant risk. In addition, there is uncertainty associated with the amount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario. Additional information was needed to perform a final Phase 3 assessment, due to the complexity of the issue. (Section 5.1)

Cornerstone: Mitigating Systems

Green.

Two examples of a noncited violation of 10 CFR Part 50, Appendix B,Criterion III, "Design Control," were identified involving the failure to adequatelytranslate the design basis of the spray ponds into procedures. Design Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis Analysis," Revision 7, which demonstrated that the spra y pond systemcould adequately limit spray pond temperature during a design basis accident did not account for any reduced heat capacity caused by sediment buildup.

However, this fact was not translated into procedures, so approximately 400 cubic yards of sediment had built up in each of the six spray ponds when the team questioned the impact to the heat removal function. Also, the same calculation demonstrated that sufficient water was available to provide adequate cooling during a design basis accident, but did not account for any leakage from the ponds. The team determined that the licensee did not translate this into aprocedure to ensure that the condition of the spray pond was maintained such that leakage did not occur. Procedure 81DP-0ZZ01, "Civil System, Structure, and Component Monitoring Program," Revision 11, was used to monitor the condition of the pond structures. The team identified that it examined only the exposed concrete surfaces, which constituted about 7 percent of the surface area and almost none of the water-containing volume. Cra cks had beenidentified and repaired in this area, but the inspections were not expanded to the underwater surfaces. This issue was documented in Condition Report/Disposition Requests 2906671 and 2910912.Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, the finding could become a more significant safety concern. This finding affected the Mitigating Systems Cornerstone. This performance deficiency screened as having very low safety 4significance in a Phase 1 significance determination process because thelicensee was able to demonstrate that the sediment would not have resulted in a loss of safety function, and that significant leakage did not exist. The licensee was able to revise the calculation to take credit for heat absorption by the concrete walls, and scheduled inspections by divers of underwater portions of the ponds to follow sediment removal. (Section 5.2)

Cornerstone: Mitigating Systems

Green.

A noncited violation of 10 CFR Part 50, Appendix B, Criterion V,"Instructions, Procedures and Drawings," with multiple examples was identified for failure to adequately assess the impact to operability of degraded heatexchangers in the emergency diesel generators and essential cooling water system. Specifically, the licensee failed to follow Procedure 40DP-9OP26,"Operability Determination and Functional Assessment," Revision 16, inassessing indications of degraded heat exchanger performance, an activity affecting quality. Key support organizations were not always involving operations personnel with questions that had a potential to affect the operability of safety-related equipment, or were informing operators only after the support organization had fully evaluated the condition, delaying actions that were required to be prompt. Also, operations personnel did not always insist on a rigorous evaluation. This issue was documented in Condition Report/Disposition Requests 2918892, 2901815, and 2898237. Failure to adequately implement the operability assessment process was aperformance deficiency. This finding was more than minor because it impacted the equipment performance attribute of the Mitigating Systems Cornerstone objective to maintain the availability and reliability of systems needed to mitigateaccidents. This finding screened as having very low safety significance in a Phase 1 significance determination process, because the examples used for this violation were confirmed not to involve any loss of safety function. This finding had cross-cutting aspects in the area of human performance because the licensee did not follow their systematic process for operability decision making when information was not brought to the right decision makers. (Section 5.3)B.Licensee-Identified FindingsA violation of very low safety significance, which was identified by the licensee, has beenreviewed by the inspectors. Corrective actions taken or planned by the licensee have been entered into the licensee's corrective action program. This violation and corrective actions are listed in Section 4OA7 of this report.

5

REPORT DETAILS

1.0SPECIAL

INSPECTION SCOPE

The NRC conducted this special inspection to better understand the circumstancessurrounding the heat exchanger fouling at Palo Verde Nuclear Generating Station. Thefouling came to the NRC's attention as a result of unusual temperatures noted during asurveillance test of Emergency Diesel Generator (EDG) 2B on May 17, 2006. Over the following several days, it was determined that this potentially impacted the ability of thissafety-related system to perform its safety function during a design basis accident. Italso became apparent that the fouling that was experienced had existed for a longperiod of time. The cause of the fouling was not fully understood, and affected other equipment in all trains of all three units. In accordance with NRC ManagementDirective 8.3, it was determined that this event had sufficient risk significance and metseveral deterministic criteria to warrant a special inspection. The team used NRC Inspection Procedure 93812, "Special Inspection Procedure," toconduct the inspection. The special inspection team reviewed procedures, corrective action documents, and design and maintenance records for the equipment of concern.

The team interviewed key station personnel regarding the event, the root cause analysis, and corrective actions. A list of specific documents reviewed is provided in

1. The charter for the special inspection effort is provided as Attachment 2. 2.0SYSTEM AND EVENT DESCRIPTION2.1Event SummaryOn May 17, 2006, Palo Verde Nuclear Generating Station noted elevated temperaturesin the intake air for EDG 2B while the engine was running for a surveillance test. Theengine intake air is compressed and heated by the turbocharger, and is then cooled by

two parallel intercoolers before entering the engine. The elevated temperatures were noted at the outlet of the intercoolers, which was indicative of inadequate cooling in the intercoolers.The elevated temperatures were discussed among operations and engineeringpersonnel, and it was concluded that this condition did not affect operability. EmergencyDiesel Generator 2B was returned to service, and the following day the EDG Train A and essential cooling water (EW) heat exchanger were removed from service for approximately 23 hours2.662037e-4 days 0.00639 hours 3.80291e-5 weeks 8.7515e-6 months . On May 19, following the return to service of Train A, the operability of EDG 2B was revisited. An immediate operability determination wascompleted at 9:55 a.m. which concluded EDG 2B was operable. After additional review, at 5:54 p.m., it was concluded that EDG 2B was not operable, and TechnicalSpecification 3.8.1 was entered. Because the condition was potentially a common modefailure concern for the EDGs, all EDGs were run to assess the performance of the intercoolers in accordance with Technical Specification 3.8.1.2.The licensee inspected the EDG 2B intercoolers and found they were fouled on thecooling water (spray pond system) side. The emergency spray pond system (SP)6provides the cooling water to both the EDG coolers (two intercoolers, a jacket watercooler, and a lube oil cooler), as well as cooling the large EW heat exchanger. The intercoolers' water side had a white slimy substance which was apparently reducing the ability to transfer heat.That same week, the licensee had taken each of the Unit 2 EW heat exchangers out ofservice for tube cleaning. This was in response to earlier heat transfer degradation which was judged to be degrading fast enough that the Unit 2 EW heat exchangers would not remain operable through the full operati ng cycle and needed to be cleanedbefore summer. The degradation was not fully understood, but was suspected to be related to a similar white slimy substance.As NRC inspectors got involved and asked key questions related to the operability ofthese components, it became apparent that the scope of the problem was more thanjust EDG 2B.2.2System DescriptionsThe spray pond system for each unit is comprised of two trains. Each train has aconcrete lined spray pond of 6.2 million gallons of water, which serves as the ultimateheat sink. The ponds are 345 feet by 172 feet, and are 15.5 feet deep. The spray pump for each train takes a suction on one corner of the pond, at a level 6 feet below the bottom of the main portion of the pond. The pump delivers 16,300 gpm through buried piping to the EW heat exchanger and the EDG coolers. The water then returns to the pond through over 300 spray nozzles spaced around the pond above the surface.

The spray nozzles atomize the water, allowing for evaporative cooling. The spray pond system is normally in a standby condition with no flow. The spray pond piping wasmade of carbon steel with a protective coating on the inside surfaces.Each EDG has five coolers which have heat removed by the spray pond system: thejacket water cooler; lube oil cooler: fuel oil cooler, and a pair of turbocharger intercoolers. While the EDG is normally in standby, the jacket water system and lube oilsystem continue to circulate and heat their process fluids. The fuel oil cooler has a verysmall flow, and is not required for the EDG to function. The intercoolers function at the highest temperature, approximately 290F air inlet temperature.The EW heat exchangers transfer heat from the systems it cools to the spray pondsystem. The loads supplied are the essential chillers, shutdown cooling heat exchanger,and spent fuel pool cooling system. The essential chillers provide cooling to the controlroom ventilation, safety-related switchgear rooms, and emergency core cooling pump rooms. The shutdown cooling heat load during an accident constitutes over 94 percent of the total EW heat load.The spray ponds were provided with a filtering system. Pond water was pumpedthrough a pair of sand filters to remove larger impurities at 600 gpm each (1200 gpm total). The sand filters were backwashed to remove the impurities about once per day under normal conditions.

7 3.0CHEMISTRY DESCRIPTION3.1Fouling MechanismsThe degraded heat transfer capabilities in the EDG coolers and the EW heatexchangers occurred in all trains in all units. The degree of degradation varied somewhat among the units. The EDG intercoolers and the EW heat exchangers were observed to have fouled the most, while the jacket water and lube oil coolers fouling was minor and had no impact to operation.The team determined that the licensee did not take photographs or have samples of thesubstances found in these heat exchangers analyzed. When the May 2006 event occurred, samples were analyzed, but the exact substances could not be determinedbecause it was an amorphous mix of a number of substances. The major constituents were determined to be zinc (38 percent), phosphorous (29 percent), and calcium (32 percent). Two different substances have been found in the heat exchangers, though this was notrecognized. In earlier years, a thin coating that turned white as it dried was observed in heat exchanger inspections. This is now believed to be zinc hydroxide. In more recent years, a thicker coating that has been described as white, slimy, and lotion-like was observed. It was believed for years that the coating was zinc hydroxide which the chemistry control program was intentionally trying to deposit on the system piping, so itwas largely accepted. The team noted that the licensee did not reassess what thematerial was when the appearance began to change.The licensee's root cause assessment concluded that improper chemical controlresulted initially in excessive zinc hydroxide precipitating out of solution, which is a heat transfer foulant. Later, zinc phosphate precipitated as a thicker mix. This precipitant then provided a localized seed site for other chemicals to also precipitate out of solution.

The combined effect was an amorphous lotion-like substance that coated heat exchanger tubes and degraded heat transfer by interrupting cooling water flow at the tube wall. Specifically, the fluffy film caused a low flow zone near the tube wall and forced the turbulent flow away from the tube wall; since efficient convective heat transfer required strong turbulent flow at the tube wall, heat transfer was significantly impacted even though the film was fluffy and not tightly adherent. It should be noted that the above fouling mechanisms were different than scaling. Scaling involved hard mineral deposits, such as calcium carbonate and calcium phosphate which occur at high temperatures. Scaling concerns are addressed in Section 5.1.5.3.2The Intended Chemistry Control ProgramThe licensee's program was intended to meet two goals: minimize corrosion in the spraypond piping and components, and minimize fouling. The team determined that the PaloVerde Nuclear Generating Station program was similar to that used in cooling water systems in the electric power industry, both nuclear and non-nuclear. Though complex,this type of program is typically effective. When initially implemented, the spray pond 8chemistry control program used the chemical supply vendor's recommended limits;however, over time these limits were changed numerous times, eventually departing from these recommendations. The Palo Verde Nuclear Generating Station program treated the two parts of the spray pond system separately. The spray ponds were treated only to prevent gross biologicalfouling. The spray pond system piping and components were treated with acomplicated combination of chemicals to meet the goals stated above. Chemicals were added at the pump suction during pump runs for piping treatment, and pond chemicals were added at the same location but were not effectively dispersed through the pond.The carbon steel piping was protected from corrosion by adding zinc and phosphoricacid. This combination was intended to form a very thin layer (a few molecules thick) of zinc, with phosphate in solution to protect any active corrosion sites. The root cause assessment team concluded that this film should not be thick enough to be visible on the piping. The zinc coating must be replaced periodically, so extra zinc is added to stay in solution to be available as replacement if needed. However, zinc has a low solubility and will precipitate out of solution unless the conditions are carefully managed.The makeup water for the system has a relatively high alkalinity and some potentiallyscale-forming impurities (such as calcium). Scale is more likely to form at higher temperatures (such as those expected in heat exchangers), and under higher pH values. The alkalinity from makeup water and evaporating of water caused pH to naturally increase. Therefore, the licensee added sulphuric acid to lower pH and improve the solubility conditions. Dispersant was added to keep the impurities and the chemicals that were intentionallyadded in solution and avoid precipitation and fouling. The dispersant was very important to control solubility and avoid fouling and scaling. However, the team determined thatthe dispersant in use at the time of the event could not be measured directly, and themethod used to estimate dispersant concentration was no longer reflecting the actual conditions.Finally, the licensee added biocide and hypochlorite to control biological growth. Thepiping was treated during pump runs; pond treatments were started during pump runs, but continued for hours after the pump was secured. All chemicals were added to the ponds near the pumps.The team identified that the chemistry program objectives to minimize fouling was notbeing monitored. The effectiveness of fouling prevention was not being tracked by chemistry personnel.3.3How The Chemistry Control Was Actually ImplementedA summary of the chemistry control program and changes made over time is provided inSection 6.1.The team determined that the chemistry control program was being implemented in away that caused fouling of safety-related heat exchangers. Chemical additives (zinc and 9phosphate) were being added in excess of what was needed and above the solubilitylimits for the existing conditions. The pH was also being maintained too high, reducing the solubility limits. Also, dispersant was not being accurately being monitored, and as aresult, was not being added in sufficient quantities to effectively control the solubility ofthe existing chemicals. These conditions combined to cause the fouling.The team reached the following conclusions about the actual implementation of thespray pond chemistry control program:The Chemistry personnel implementing the program did not fullyunderstand how the spray pond chemistry control program was supposed to work. As a result, they made a series of inappropriate changes to chemistry limits which negatively impacted solubility without assessing orunderstanding the potential impacts.The specified pH band was too high to maintain zinc hydroxide insolution, so it precipitated out of solution and formed a foulant.The zinc control band was too high, resulting in injecting too much andprecipitating significant quantities out of solution. The precipitate collected in the EW heat exchangers and intercoolers, and may have also precipitated into the sediment at the bottom of the ponds. The ideal control concentration was determined to be about 0.5 ppm by the root cause team, while the actual control band was 1.0 - 3.0 ppm at the timeof the event. Excess phosphate was being added, contributing to the precipitationproblems. Phosphoric acid was added in a 4:1 ratio with the zinc; zinc was constantly removed from solution, while phosphate built up in the pond and piping. The ideal phosphate level was determined by the root cause team to be about 4.0 ppm, while at times, samples recorded over 20 ppm phosphates in Unit 2. The high phosphate levels exceeded the maximum solubility for the bulk water conditions, and precipitated out ofsolution in the EW heat exchangers and intercoolers, and probably also precipitated into the sediment at the bottom of the ponds.Calcium and other impurities from makeup water were allowed toconcentrate excessively over time, and under the relatively high pH conditions they contributed to precipitation problems. The root cause team concluded that the zinc phosphate precipitant provided a seed sitefor other compounds containing zinc, calcium, and phosphate to attach. Dispersant was intended to improve the solubility conditions, but there was too littledispersant present to be effective. The dispersant in use could not be measured directly, so a "tag" chemical (molybdate) which could be measured was added in a known proportion to the dispersant. The licensee did not recognize that this tag chemical was not subject to the same loss and use mechanisms as the dispersant. As a result, the tag chemical analyses registered incorrectly high results. The team determined that Chemistry personnel reduced or stopped adding dispersant for long 10periods of time because the molybdate concentrations indicated dispersant levels werehigh. Contributing to this, the tag chemical was only checked once per month. Also,when the licensee allowed the high levels of impurities to build up, they did not recognize that this should have required raising the amount of dispersant to maintain the same effectiveness.The team identified that the chemistry control practices for this system were not typicalof other plant systems with the same importance. Chemicals were added based onthumbrules and past additions, rather than basing them on recent sample results. Also, many key samples were taken only once per month, but the associated chemical additions were made several times per week. Some important samples had no controls associated with undesirable levels. For example calcium, an impurity, which is a scale-forming concern, had no action level to trigger action to reduce the concentration. The spray pond system was a concentrator for chemicals, so as calcium levels accumulatedover the years, Chemistry raised the limits to stay ahead of the actual levels, rather than reducing the existing concentrations. When sample results indicated that molybdate levels were high and it was presumed that this meant dispersant levels were high, dispersant additions were stopped. The molybdate samples were taken monthly, but the analyses were sometimes delayed by up to 2 to 3 weeks (Condition Report/

Disposition Request (CRDR) 2800653). This created an untimely feedback mechanismfor adjusting the chemical addition.The team identified that chemistry personal had focused on treating the pipes and didnot effectively manage the chemistry condition in bulk pond water. This approach was viewed as economical, and is a common industrial practice. However, this did not account for the unique nature of the spray ponds at Palo Verde Nuclear GeneratingStation. Unlike typical systems, the spray pond system was not designed or operated ina way that regular water turnover occurred, such that chemicals and impurities would be removed. It was not recognized that the chemistry control program in use typically involved a system that ran continuously and had a much higher water turnover. Theturnover should result in removal of impurities and cause the tag chemicals to more closely reflect dispersant concentrations. The continual buildup of chemicals and impurities at Palo Verde Nuclear Generating Station caused a departure from the conditions assumed when the vendor recommendation for dispersant levels was made. Most sampling was done during pump runs by taking samples from the pipes, with verylittle sampling of the bulk water in the ponds. As a result, the ponds were allowed to accumulate chemicals and impurities. This was further complicated by allowing sediment to accumulate at the bottom of the ponds, where precipitants could be concealed but still affect water chemistry. Biological fouling in the ponds was not being effectively controlled. The team concludedthat the biocide treatments and monitoring of the ponds were ineffective. During theonsite portion of the inspection, contract divers began vacuuming sediment from the Unit 1 ponds. Condition Report/Disposition Request 2905162 was written on June 21, 2006, to document that the diving operations were dislodging "sheets" of algae from thespray pond walls. The licensee was unaware that algae existed in the ponds prior to this time. The team reviewed the pond treatment practices and concluded that the biocide addition point was in one corner of the ponds, insufficient mixing was provided to 11distribute the chemicals. This was, in part, because much of the biocide chemicals wereadded after securing the spray pond pumps. The licensee's evaluation concluded that the existing algae would not impact operability, and the team concluded that this was areasonable conclusion.Finally, the water chemistry control program being used was challenged by thelicensee's practice of leaving the systems idle, without constant flow, for most of thetime. Typical cooling water systems run pumps continuously, which have the benefitsfor flushing pipes and heat exchangers out and mixing the chemicals. The Palo Verde Nuclear Generating Station spray pond systems were normally run only for chemistrycontrol (adding chemicals 3 times per week), infrequent testing, and during plant outages. This resulted in the system having no flow most of the time.The team noted that the chemistry control program documents stated that dispersantadditions were intended to be based on makeup water addition. However, that had never been the actual practice. The team identified that the licensee did not trackmakeup water additions to the spray ponds, and had no way of measuring the quantity of water flowing to any one pond. In an idle pond, evaporative losses ranged from 2,200 to 10,000 gallons per day. When the system was run, evaporation was much higher,and "drift" losses occurred when water droplets were carried out of the pond berm by wind. Despite these large and highly variable losses which required makeup and caused loss of dispersant, the team identified that the licensee had stopped addingdispersant for long periods, in part because they were not adding dispersant based on makeup water addition.The team identified that the chemical supply vendor site representative visited the siteseveral times per month, but this source of expertise was not utilized. The vendor wasnot asked to assess the performance of the program, and was not asked to observe heat exchanger foulant or analyze the material. The site representative was involved in trying to identify the lotion-like substance found in the spray pond sand filters, though he was not provided with a sample as a basis for the discussion.The team concluded that the licensee was no longer maintaining the spray pondchemistry control program within the bounds that were originally intended. They had made changes without understanding the effect the changes would have on the conditions and without understanding the complex interactions that existed. As a result, the fouling was self-induced by the chemistry control program.

4.0HEAT EXCHANGER PERFORMANCE MONITORING4.1Essential Cooling Water (EW) Heat Exchanger Performance TestingThe licensee committed to conducting heat exchanger performance testing on the EWheat exchangers. This was typically done for one train at the beginning of each refueling outage during the primary plant cooldown, when decay heat load allowed enough heat load to be able to get accurate results. Later, the licensee began to also conduct EW heat exchanger performance testing during the primary plant heatup at the end of refueling outages. These tests had a much lower heat loading, so the accuracy 12of the results was potentially lower, but the results would still be expected to provide anindication of the relative effectiveness of any tube cleaning during that outage. The team performed a detailed review of the licensee's performance testing methodsand results, including internal inspections and cleaning practices. This included reviewing documented results since 2001. This is further discussed in Section 1R07.The results of the performance testing conducted on the EW heat exchangers issummarized in Attachment 3. It can be seen that the data for any one heat exchangeris very limited, since it might be tested only once every three years. It can also be seen that significant reductions in heat exchanger performance were indicated by test results, yet very few instances were found where this degradation was documented in the corrective action program. This is further discussed in Section 5.4.4.2Emergency Diesel Generator (EDG) Performance TestingThe licensee did not conduct performance testing of any of the EDG heat exchangers. The primary reasons are briefly summarized:Lube oil coolers and jacket water coolers: The shell side fluids (lube oil andjacket water) have automatically controlled flows that adjust to the conditions to maintain a constant temperature. The thermostatic control valves cannot bemanually overridden or their positions measured. Also, the systems do not haveinstalled flow measurement devices, and the pipe configuration prevents an accurate flow measurement using an external device. Intercoolers: These heat exchangers have a relatively small heat load, and useair as a working fluid. Both these facts make it difficult to accurately measure heat exchanger performance.Instead of performance testing, Palo Verde Nuclear Generating Station's commitment intheir Generic Letter 89-13 response was to regularly clean and inspect these heat exchangers to ensure there was not excessive fouling. Since these inspections identified that the intercoolers in each unit experienced fouling, the licensee wastrending the relative performance of the intercoolers during EDG surveillance testing. This was done by assessing the difference between air outlet temperature and water inlet temperature over time. Cleanings were scheduled when this difference became high, but the condition was documented as a degraded condition in the corrective action program only once, although the degraded condition occurred on numerous occasions.

This is further discussed in Section 5.3.

13 5.0FINDINGS5.1Performance Deficiencies that Directly Related to Fouling to the Point of Inoperability a.Description

.In 1994, Palo Verde Nuclear Generating Station implemented a new chemistry controlprogram for the emergency spray pond system to control corrosion and prevent foulingof the safety-related components. This system provides the ultimate heat sink function. Throughout the period of 1994 through May 2006, the licensee made a series of changes to this program which created a chemical environment that was progressively more conducive to fouling of the heat exchangers which were relied upon to transfer heat from the reactor, containment, EDGs, and safety-related equipment rooms to the ultimate heat sink. The foulant was determined to be a buildup of excess chemicals which were added as part of the chemistry control program, but which were subsequently determined to have exceeded the limitations of solubility. Despite years oftest results which showed degraded heat exchanger performance, numerous heat exchanger inspections which documented chemical buildup, and an increasing need to clean the heat exchangers more frequently, the licensee failed to recognize the safety significance of the problem or determine and correct the cause. Degraded performance was observed in all trains in all three units. Because of significant inherent design margins, only one example was identified where fouling may have been sufficient to cause a loss of safety function. Specifically, the Unit 2 Train B EW heat exchanger was potentially inoperable between about April and October 2003.Also, during the inspection it was recognized that calculations which demonstrated thatheat exchangers cooled by the spray pond system would not be subject to scalingduring a design basis accident contained an error. When re-performed, the calculations indicated that scaling was possible. This would have the effect of further reducing the heat transfer capability of these heat exchangers during certain accident scenarios. This had the potential to affect components cooled by the spray pond in all units.The team reviewed the degraded performance and the remaining capabilities of thesecomponents over the period 1994 through 2006. The licensee performed computer modeling and analysis that adequately demonstrated that the EDG systems werecapable of performing under the degraded conditions. This was due to considerable margins inherent in the design, and because it could be shown that the fouling inEDG 2B that triggered this inspection was clearly the worst case that had been experienced. For the EW heat exchangers, the team identified the following fourperiods where the EW heat exchangers had less than the design basis heat removal capability, a degraded condition which constituted a significant condition adverse toquality. Of these, it appears that only 2B was incapable of performing its function under the existing, less challenging conditions in 2003.EW HX 1B was possibly as low as -20.8 percent margin entering 1R11 in 2004.

EW HX 2B was possibly as low as -49.78 percent margin entering 2R11 in 2003.

14EW HX 3B was possibly as low as -14.4% percent margin entering a midcycleoutage in 2004. .

EW HX 3A was measured at -22 percent margin entering 3R10 in 2003.

These estimates do not include the possible effect due to scaling discussed in subsection

(5) below. The impact due to fouling will be reassessed in all units duringthe final significance determination.Note that throughout this report, heat exchanger capacity is discussed in terms of"percent margin." This term is meant to refer to margin above or below the design capacity of a heat exchanger. In this context, "0 percent margin" means the heat exchanger would have exactly the capacity required to transfer the design heat load under limiting design conditions. Positive or negative margin expresses how much more or less than that capability actually existed. This was used to be consistent withlicensee documentation. b.Statement of the Performance Deficiency

.Between 1995 and May, 2006, the licensee failed to recognize that improperlyimplemented chemistry controls for the emergency spray pond system c aused asignificant condition adverse to quality which degraded the performance of all emergency diesel generators and emergency cooling water systems. The degradedperformance was because of heat exchanger fouling from chemical additives, and potentially because of scaling after an accident starts. This resulted in degrading the performance of EW Heat Exchanger 2B to the point where it would not have been capable of performing its intended safety function for approximately 6.8 months in 2003.

In addition, there is uncertainty associated with the amount of scaling that could occuron any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario. c.Enforcement

. (1)Technical Specification 3.7.7 requires that two trains of essential cooling water beoperable in Mode 1, with an allowed outage time for one train of 72 hours8.333333e-4 days 0.02 hours 1.190476e-4 weeks 2.7396e-5 months . Contrary to this, Train A of the EW system in Unit 2 was not capable of performing its safetyfunction for approximately 6.8 months ending on September 27, 2003, when the plantshut down for Refueling Outage 11. This will be treated as an apparent violationpending a final significance determination: AV 05000529/2006011-01, EW Train 2B Inoperable Longer than Allowed Outage Time. This issue was entered into the licensee's corrective action program under CRDR 2905161.The period during which the train was incapable of performing its intended safetyfunction was determined through straight-line extrapolation of the two prior heat exchanger performance tests for this train. The minimum acceptable performance was determined by licensee calculation dated June 22, 2006, which revised Design Basis Calculation 13-MC-SP-307 to remove all known margins. The minimum acceptable performance was 204 Btu/hr-ft2-F, or about -35 percent margin compared to the full design basis value. This calculation included the most limiting values for spray pond 15temperature and air temperature actually recorded during the exposure period, andaccounted for the existing number of tubes with plugs installed. The team reviewed this calculation and determined that appropriate values and assumptions were used for the purpose of determining heat exchanger performance capability.The licensee showed documentation that indicated that eddy current testing of this heatexchanger was performed during 2R10 in March, 2002. The licensee stated that the probes were known to push out the majority of the foulant and improve heat exchanger performance. The team acknowledged this fact, but could not quantify any potentialimprovement because no data existed to quantify this for Unit 2 EW heat exchangers.

The Unit 2 EW heat exchangers were the only ones with sleeves at each end; the sleeves necessitated using a smaller probe than was used in the other units, and thus had a bigger gap between the probe and the majority of the tube wall length. This factprecluded using test data which was available for the other units, particularly because the heat transfer characteristics of the foulant were not known. (2)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XI, "TestControl," requires that a test program shall be established to assure that all testing required to demonstrate that structures, systems, and components will performsatisfactorily in service is identified and performed in accordance with written test procedures which incorporate the requirements and acceptance limits contained in applicable design documents. Test results are required to be documented and evaluated to assure that test requirements have been satisfied. Test Procedure 70TI-9EW01, "Thermal Performance Testing of Essential Cooling Water Heat Exchangers," contained instructions for conducting performance testing.

Procedure 73DP-9ZZ10, "Guidelines for Heat Exchanger Thermal Performance Analysis," contained instructions for calculating the results of EW heat exchanger performance testing and verifying them against the design basis requirements in Calculation 13-MC-SP-0307, "SP/EW System Performance Design Bases Analysis,"

Revision 007. Contrary to this, the results of performance testing for EW Heat Exchanger 2B conducted on March 19, 2002, did not meet the design basisrequirements, but this was not correctly evaluated to determine whether the systemwould be capable of performing its design function until August 22, 2002. As a result, Unit 2 was allowed to restart and run an entire operating cycle without correcting thedegraded performance. This will be treated as an apparent violation pending finalsignificance determination: AV 05000528; 05000529;05000530/2006011-02,Inadequate Test Control to Promptly Identify Unacceptable Performance Test Results.

This issue has been entered into the licensee's corrective action program under CRDR 2928230.The team determined that the performance test was scheduled and conducted under amaintenance order. However, the calculation of the results and comparison to acceptance criteria was not scheduled. Further, the test results were not administratively tracked to ensure that the unit was not restarted before acceptableperformance was documented. As a result, the March 19, 2002, test results were determined after Unit 2 started up from Refueling Outage 2R10. The test results were unexpectedly low, but this was determined to involve a calculation error. When the error was corrected with results, which indicated low performance, the test method was evaluated. On August 22, 2002, the test package was completed and verified to 16demonstrate -9.5 percent capability. This untimely determination of unacceptableperformance did not trigger a review of operability, nor was continued degradationconsidered. As a result, this component continued to degrade for 18 months after demonstrating unacceptable performance. Had this determination been made prior to the end of the outage, corrective actions could have been taken to clean the tubes.

Technical Specifications require that both trains be operable prior to making plant mode changes, so the unit should not have be restarted with this condition, had it been identified. (3)Part 50.59 of Title 10 of the Code of Federal Regulations allows licensees to makechanges to procedures as described in the Updated Final Safety Analysis Report without obtaining a license amendment only if the change satisfies the criteria in 50.59.c. It goes on to specify that changes made to procedures must include written evaluations which provide the bases for the determination that the change does notrequire a license amendment. The Palo Verde Nuclear Generating Station Updated Final Safety Analysis Report,Section 9.2.1.2.F, specified that the spra y pond system would be protected from organicfouling and inorganic buildup by proper water treatment. Procedure 74DP-9CY04, "System Chemistry Specification," implemented the licensee's program, including chemical limits and addition frequencies, in order to control corrosion and fouling in the emergency spra y pond system.Contrary to the above, between 1998 and 2004, the licensee made multiple changes tothe chemistry control program specified in Procedure 74DP-9CY04, "System Chemistry Specification," without adequately assessing the impact to the safety-related systemsaffected by the changes. Revisions 3, 6, 8, 10, 12, 24, 28, 32, and 36 toProcedure 74DP-9CY04 made changes between March 1998 and February 2006, which affected spray pond chemistry parameter limits which were subsequently determined to contribute to heat exchanger fouling. Specifically:*Revisions 3, 6, 8, 12, 24, 28, and 32 were made without any documented 50.59review. The team determined that these revisions contained changes to chemistry limits in the spray pond which had the potential to negatively impact fouling, and therefore were required to be evaluated per 50.59. These included changing pH, phosphate, and zinc limits, and instructions on the frequency of dispersant additions. *The screening performed for Revision 10 was determined to be inadequatebecause the revision increased the maximum allowable pH in Unit 1, but the screening dated 6/30/99 did not evaluate this aspect of the revision. This limithad been lowered in Revision 6 to account for the decision to use well water as the makeup source for the Unit 2 spray pond. This water source had a significantly higher alkalinity, and the lower pH was determined to be necessary to avoid scaling at the time the modification was implemented. However, this aspect of the change was not evaluated in this screening.

17*The screening performed for Revision 36 was determined to be inadequatebecause the revision increased the maximum allowable zinc concentration by a factor of two, but Screening Number S-06-0049, Revision 1, did not address this aspect of the revision. Excessive zinc additions were subsequently determined to be a primary cause of fouling. This will be treated as an apparent violation pending final significance determination: AV 05000528; 05000529;05000530/2006011-03, 50.59 Reviews Not Performed or Inadequate for Multiple Changes to Spray Pond Chemistry Control Procedure.This issue was documented in CRDR 2902498. The team concluded that there weretwo prior opportunities to have identified that chemistry limit changes were being made without 50.59 reviews which did not identify this set of examples (CRDRs 2683642 and2864575); these are further discussed in Section 4OA7. (4)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XVI,"Corrective Actions," requires that significant conditions adverse to quality shall be promptly identified, and that the cause shall be determined and corrective action shall betaken to preclude repetition. Contrary to this, on March 19, 2002, performance testing for EW Heat Exchanger 2B indicated that the system would not be capable ofperforming its design function, but this significant condition adverse to quality was not promptly identified, the cause determined, or corrective actions taken to restore the required heat exchanger performance. Specifically, the unacceptable performance was not promptly identified, because the test results were not correctly calculated untilAugust 22, 2002, which was after operating mode changes and returning the unit to power following the outage. When the test results were finalized, the fact that thedesign basis capability was not met was not recognized or entered into the correctiveaction program. These failures to correct this degraded performance contributed to the continued degradation and eventual loss of function for a period of approximately 6.8 months. This will be treated as an apparent violation pending final significancedetermination: AV 05000528; 05000529;05000530/2006011-04, InadequateIdentification and Corrective Action for Degraded EW Heat Exchanger Performance.

This issue was entered into the corrective action program under CRDR 2897810. (5)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion III, requiresthat design control measures be established to verify the adequacy of design of structures, systems and components. A SEQUIL calculation was intended todemonstrate that the heat exchangers cooled by the emergency spray pond systemwould be capable of transferring the required heat load to the spray pond because scaling would not occur during a design basis accident. Contrary to this, the calculation used an improper setting which caused the calculated result to incorrectly show that scaling would not occur. This issue was documented in CRDR 2913430. This violationhas the potential to increase the exposure time when EW Heat Exchanger 2B was incapable of performing its safety function, since the 6.8 month exposure time did not 18account for any impact because of scaling. There is uncertainty associated with theamount of scaling that could occur on any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario. This will be treated as an apparentviolation pending final significance determination: AV 05000528; 05000529;05000530/2006011-05, Inadequate Design Control to Ensure No EW Heat Exchanger Scaling. This finding does not represent an immediate safety concern because immediate andlong term corrective actions that have been implemented to reduce chemical addition, remove excess impurities from the ponds, clean all the affected heat exchangers, and adjust control limits appropriately. The licensee was also in the process of removing sediment and sludge from the ponds. d.Assessment

.The performance deficiency associated with these violations was more than minorbecause it impacted the equipment performance attribute of the Mitigating Systemscornerstone objective to maintain the availability and reliability of systems needed tomitigate accidents. Specifically, EW Train B in Unit 2 was estimated to have been incapable of performing its function under existing conditions for approximately 6.8 months. This was determined through straight-line extrapolation of heat exchanger performance test data. The minimum required performance for the EW heat exchanger was determined by the licensee by modifying the design basis calculation to credit the use of all known margin and evaluated the condition for the worst case pond temperature known to exist during the period of concern. This performance deficiency affected the Mitigating Systems functions of short-termprimary heat removal (high and low pressure safety injection) and long-term heat removal (emergency core cooling system recirculation). Because this performance deficiency involved the loss of function for a single train for longer than its Technical Specification allowed outage time, Phase 1 of the significance determination process required a Phase 2 evaluation.To challenge the safety function of the EW heat exchanger, it was determined that theremust be both a high heat load and a high pond temperature. The sequences of concern were determined to be small, medium and large break loss-of-coolant accidents, loss-of-offsite power, plus, stuck open primary relief valve sequences. Loss of dc Bus B was also evaluated as a consequential failure.The performance deficiency was assumed to cause a loss of function of the EW heatexchanger. As a consequence, this has the following assumed impacts:Loss of cooling to shutdown heat exchanger - reduced containment heat removalfunction credit from multi-train to single train.Loss of cooling to the essential chiller, which fails and causes loss of roomcooling and eventual failure of the following equipment:

19Motor driven auxiliary feedwater pump - reduced credit frommulti-train to single trainContainment spray pump - reduced credit from multi-train tosingle trainHigh pressure safety injection pump - reduced credit frommulti-train to single trainLow pressure safety injection pump - reduced credit frommulti-train to Single TrainEW pump - reduced credit from multi-train to single train Control room complex - no Phase 2 impact because of redundanttrain and non-safety cooling supplyDirect current equipment room - required evaluation of the loss ofdc Bus B special initiatorRecovery credit was considered for the EW heat exchanger, because thecondition was a degradation in capability, not a complete loss of function. Therequirements in Manual Chapter 0609 for recovery credit were not explicitly met, but it was judged that normal operator training and procedures would be sufficient to maintain containment/core heat removal.Recovery credit was considered for mitigating the effects of loss of room cooling. The licensee had procedure steps to open doors for the affected rooms, and meet the requirements for recovery credit. It was uncertain whether this action would be effective, so this is an area where additional information is needed to make a final significance determination.Manual Chapter 0609 directs that findings of this type be evaluated for largeearly release frequency (LERF) contribution. For large, dry type containment buildings (e.g., Palo Verde Nuclear Generating Station) Appendix H and associated basis document indicates that Type A findings at pressurized-water reactors that affect containment heat removal do not contribute to LERF,because they involve late containment failure probability only. Therefore, the riskof this performance deficiency is best represented by CDF.The results of the Phase 2 SDP determined that this condition represented a riskthat was potentially greater than very low risk significance (potentially greaterthan Green). The Phase 2 results and assumptions were reviewed by Senior Risk Analysts from Region IV and Headquarters.

20A preliminary Phase 3 SDP analysis was performed. However, it was determined thatseveral important aspects and assumptions contained considerable uncertainty or could not yet be verified. While the licensee was working on a number of evaluations and calculations to resolve the uncertainty, these were not complete. The preliminary Phase 3 analysis and modified Phase 2 using the same assumptions as the Phase 3 analysis indicated that the risk could be lower than the Phase 2 analysis, but the resultswere highly dependent on the uncertainty of some of the information.In accordance with Manual Chapter 0609, NRC senior management determined that thepreliminary significance would be based on the Phase 2 SDP results, pending a final significance determination which would account for additional information to be provided by the licensee. This was intended to improve the timeliness of regulatory decision-making in situations where the complexity of the significance determination would otherwise cause an excessive delay.Information on the following aspects was still needed from the licensee in order tocomplete a final significance determination:The effect on heat exchanger heat transfer capability that scaling may haveplayed in the first 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of a design basis loss-of-coolant accident for the worst chemistry conditions that existed in each of the six spray ponds. Also, whether this impact would cause other periods of degraded performance because of fouling to challenge the capability of the system.Whether the peak spray pond temperature would be significantly lower forloss-of-coolant accident break sizes less than analyzed in the large break design basis accident. If so, what would the resulting limiting overall heat transfer coefficient values be for medium and small break sizes given the lower peak pond temperatures.Best available calculations (technical basis and results) for room temperatureprofiles during loss-of-coolant accidents with loss of room cooling to safety-related equipment because of a failed essential chiller. Also, best availableinformation of the failure probability and basis associated with the safety relatedpumps affected by the loss of room cooling.An assessment of human error probabilities for important operator actions underthe specific conditions expected to be present if operator action is to be credited in reducing the risk estimation. Specifically, if the licensee plans to credit controlling EW flow through the shutdown cooling heat exchanger as a way of delaying or eliminating a loss of the essential chiller. The technical basis for any "inadvertent cleaning" effect the licensee intends tocredit. This should include an analysis of how the any test data to be used relates to the geometry and conditions in the Unit 2 EW heat exchangers (i.e.

eddy current probe size, the heat exchanger tube sleeves, normal tube diameter, thermal conductivity of remaining precipitant, etc.).

21An analysis of the contribution to risk for external events associated with thisissue.5.2Design Control IssuesThree examples of a violation of 10 CFR Part 50, Appendix B, Criterion III, "DesignControl," were identified. The example related to scaling calculations is discussed in Section 5.1.5.(1) Spray Pond Allowed to Build Up Sediment

Introduction.

An example of a noncited violation of Criterion III was identified for failureto adequately translate the design basis of the spray ponds into procedures. The design basis calculation that demonstrated that the spray pond system could adequately limitspray pond temperature during a design basis accident did not account for reduced heat capacity caused by sediment. However, this was not translated into procedures to ensure that sediment did not build up in the spray ponds.Description. During early discussions of the design and condition of the spray ponds, itbecame apparent that the spray ponds had sediment along most of the bottom of each pond. It was also determined that the licensee had never drained or cleaned the ponds,nor did they monitor or quantify the sediment. The team questioned the impact the sediment had on the ponds' ability to perform their intended functions.Engineering and chemistry personnel initially stated that the ponds needed 18 inches ofat the bottom of the pond to assure adequate spray pond system pump net positivesuction head. This was therefore an unusable water volume. By this reasoning, this volume of water could be replaced by sediment with no impact, since sediment did not affect the net-positive suction head (NPSH). Informal reports from contract divers had indicated that sediment was about 4 inches thick on average, and did not exist near thepump suctions. The licensee determined that a preventive maintenance basis document from the early1990s had specified that a preventive maintenance item should be created to verify thatsediment did not accumulate and impact system capability. However, this was neverimplemented.Additionally, engineering personnel reviewed Calculation 13-MC-SP-0307, "SP/EWSystem Thermal Performance Design Basis Analysis," Revision 7, and determined that this calculation relied on both the usable and unusable water volume contained in the ponds to absorb the heat rejected during a design basis accident. Using the entire water volume yielded the result that under limiting conditions, the maximum pond temperature was maintained just below the maximum allowable temperature of 110F.

Further, the heat capacity of the sediment was about 40 percent compared to that of thewater that it displaced. Therefore, the team concluded that the spray ponds were alloutside the analyzed condition of the design basis.

22Engineering revised the calculation to account for the quantity of sediment believed toexist in each pond. In doing so, it was necessary to credit heat transfer to the concrete in the spray pond walls, which was not previously credited. The team reviewed the calculation and determined that it used reasonable assumptions and demonstrated adequate margin.As part of the corrective actions, the licensee began a campaign to remove and quantifythe sediment from each spray pond. By the time of this report, three of the six spray ponds had been vacuumed, removing approximately 400 cubic yards of sediment from each pond. This was within the volume assumed in the calculation.Analysis. Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, the finding could become a more significant safety concern. This finding affected the Mitigating Systems cornerstone. This performance deficiency screened as having very low safety significance (Green) in a Phase 1 SDP because the licensee was able to demonstrate that the sediment would not haveresulted in a loss-of-safety function.

Enforcement.

Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion III, requires that design control measures be established to ensure that designbasis information is translated into instructions, procedures, and drawings. The Calculation 13-MC-SP-0307 was intended to demonstrate that the heat exchangerscooled by the emergency spray pond system would be capable of transfering therequired heat load to the spray pond because scaling would not occur during a design basis accident. This calculation relied upon the entire volume of water available at the Technical Specification minimum water level to demonstrate that the pond could perform its intended safety function. Contrary to this, the licensee did not ensure that the watervolume was maintained as assumed in the calculation through instructions, procedures, or drawings. This issue was documented in CRDR 2906671. This example of a violationis being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000528; 05000529; 05000530/2006011-05a, InadequateDesign Control to Ensure Spray Pond Sediment Would Not Impact Operability.

(2) Spray Pond Integrity

Introduction.

An example of a noncited violation of Criterion III was identified for failureto adequately translate the design basis of the spray ponds into procedures. The design basis calculation, which demonstrated that sufficient water was available to provide adequate cooling during a design basis accident did not account for any leakage from the ponds. The team determined that the licensee did not translate this into a procedureto ensure that the condition of the spray pond was maintained such that leakage did not occur.

23Discussion. The team reviewed all CRDRs associated with the spray pond systeminitiated since 2000. In CRDRs 114987, and 2886930, and CORs 00-1-001, 00-2-001,and 00-3-001, the licensee documented cracks that were observed in the accessibleportions of the spray pond walls. In some instances, minor weepage through the wall was observed. The conditions were repaired. The team reviewed the licensee's treatment of the system within their maintenance ruleprogram, and determined that it was subject to condition monitoring. Condition monitoring was performed using Procedure 81DP-0ZZ01, "Civil System, Structure, and Component Monitoring Program, Revision 11. At Palo Verde Nuclear Generating Station, structural monitoring was performed in one unit every 10 years, rather than in each unit. The program required that if problems were identified in one unit's structure, the similar structures in the other units would be inspected.The team was concerned that licensee's Maintenance Rule Program only requiredinspections of the accessible portions of the pond structures. The licensee considered that the pond bottom and inner walls were inaccessible because of the presence of water. The team concluded that the portions of the spray pond liners considered by the licensee to be accessible represented a small fraction of the structure (less than seven percent). Almost the entire portion that was subject to inspections was above the water line, were structural integrity was not needed, whereas the uninspected portion was critical to the safety function.The team identified that the licensee had never drained any the spray ponds orconducted formal or complete inspections using divers. Further, the pond bottom constituted the majority of the pond liner surface, and this could not be inspected because the licensee had allowed approximately 4 inches of sediment to buildup on the pond bottom. Also, the licensee was not able to measure spray pond makeup, nor were they able to estimate spray pond water loss because of the large daily evaporative losses, which were highly variable. The licensee calculated that there was sufficient water maintained above the Technical Specification minimum water level to be able to tolerate a 5 gpm leak, which was small compared to the daily losses.The licensee acknowledged the need to inspect the pond liners. This was scheduled aspart of CRDR 2901589 to be performed by divers trained to perform this inspection uponcompletion of vacuuming in each pond. Inspections of the three completed ponds indicated no integrity problems.The licensee performed an assessment which concluded that no significant leakageexisted. This was based largely on the judgement that the geology and hydrology of thesite would cause significant leakage to be observable on the surface, and no such indication had ever been observed. This was supported by recent efforts to identify the source of detectable tritium in ground water, which included drilling some holes in thevicinity of the spray ponds. Many of the holes collected no water, and none of the water that was collected had chemical results which would indicate that it included spray pond water. Also, the construction methods ensured that there were no joints that could provide leakage paths through the walls or bottom. The observed cracking was judged to be too tight to allow significant leakage.

24The team reviewed the photographs from the inspections, the design and constructionof the liner, and the results of ground water studies, and concluded that it was reasonable to conclude that no significant leakage existed.Analysis. Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be more than minor because, if left uncorrected, this finding would be a more significant safety concern. This finding affected the Mitigating Systems cornerstone. This performance deficiency screened as having very low safety significance (Green) in a Phase 1 SDP because it did not result in a loss-of-safety function.

Enforcement.

Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion III, requires that design control measures be established to ensure thatdesign basis information is translated into instructions, procedures, and drawings.

Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis Analysis," Revision 7, which demonstrated t hat the system contained sufficient water toprovide adequate cooling during a design basis accident did not provide any margin for leakage from the ponds. Contrary to this, the team determined that the licensee did nottranslate this information into an instruction, procedure or drawing to ensure that the condition of the spray pond was maintained such that leakage did not occur. This issue was documented in CRDR 2910912. This example of a violation is being treated as anoncited violation, consistent with Section VI.A of the NRC Enforcement Policy:NCV 05000528; 05000529; 05000530/2006011-06b, Inadequate Design Control toEnsure Spray Pond Integrity.5.3Inadequate Implementation Of Operability Assessment ProcessIntroduction. A Green noncited violation with multiple examples was identified for failureto follow Procedure 40DP-9OP26, "Operability Determination and FunctionalAssessment," Revision 16. Key support organizations were not always involving operations personnel with questions that had a potential to affect operability of safety-related equipment, or were informing operators only after the support organization had fully evaluated the condition, delaying actions that were required to be prompt by days or weeks. Also, operations personnel did not always insist on a rigorous evaluation.

This violation was determined to be of very low safety significance because it was determined to not involve a loss-of-safety function in a Phase 1 SDP evaluation. This finding has Human Performance Cross-cutting aspects associated with decision making.Description. Charter Item 7 required a review of operability evaluations associated withheat exchanger fouling and an assessment of the compensatory actions and initial corrective actions to address the issues. During this review, the team noted a number of examples where the assessment ofoperability did not meet the NRC's expectations as delineated in Regulatory IssueSummary 2005-20 or follow the operability assessment process inProcedure 40DP-9OP26, "Operability Determination and Functional Assessment." More specifically, the team concluded that key support organizations (engineering, chemistry, and maintenance) were not always involving operations personnel with questions that had a potential to affect operability. In a few cases, operations was 25informed only after the support organization had fully evaluated the condition, delayingactions that were required to be prompt by days or weeks. In a larger number of cases, support organizations decided not to involve operations. Also, operations personnel became involved but did not always insist on a rigorous evaluation. The following are the more significant examples that illustrate these conclusions:*During a surveillance test of EDG 2B on May 17, 2006, high out of specificationintercooler air temperatures were noted. The licensee failed to adequately assess this degraded condition or formally enter the operability determinationprocess until May 19. Emergency diesel generator operability was informallydiscussed. The EDG system engineer assured the operations shift manager thatthe emergency diesel was operable based on a call with a consultant to the Cooper-Bessemer Owners Group (inappropriately represented as a representative of the vendor). The team determined that the scope of thediscussions had been limited to the impact to the engine and had failed to evaluate other aspects, such as the effect of higher exhaust temperatures on turbocharger blade creep, increased fuel consumption, and load control. While these were eventually addressed after NRC inspector involvement, this initialinformal and undocumented operability assessment allowed the licensee toremove the opposite train from service when EDG 2B was degraded and the impact was not properly understood. When operability was properly addressedon May 19, it was decided that EDG 2B should not have been considered to have a reasonable expectation of operability.*On May 19, 2006, Work Order 2896333 was initiated to inspect and clean theEDG 2B intercoolers. During review of this work order following EDG 2A restoration, the on-shift operators determined the high temperatures warrantedan operability determination. Condition Report/Disposition Report 2896661initially concluded that the condition was isolated to EDG 2B. The NRC inspectors identified that this ignored fouling in both EW heat exchangers on the common spray pond system side that had necessitated cleaning that same weekto be able to assure operability through the coming hot summer months. However, the team noted that operations did correctly decide to run all EDGs in accordance with Technical Specification Surveillance Requirement 3.8.1.2 toassess their operability. Additionally, the inspectors noted that the initialevaluation assessed performance using the existing pond and air temperatures, rather than using the more limiting design basis conditions; when this was included, the degradation was shown to potentially exceed the design basistemperature limit during accident conditions in two additional EDGs. Following prompting from NRC inspectors, on May 24, 2006, the licensee appropriatelyexpanded the scope of their evaluation. Additionally, the team noted that thelicensee initiated CRDR 2898120 that states, in part, "It appears that trendingand monitoring and/or extent of condition considerations of the essential water heat exchanger fouling condition should have provided for identification of the intercooler fouling condition and prevented the emergent condition discovered on May 17, 2006."

26*The Prompt Operability Determination that was issued on June 3, 2006,concluded that heat exchangers were adversely affected by a low temperature fouling mechanism that was already known. The team concluded that this initial conclusion was based on an incorrect understanding of the situation that datedback to 2003, and was the basis of the decision to stop adding dispersant to the spray ponds. This was eventually shown to be an inappropriate compensatory action. However, when a technical evaluation of facts progressed, the cause assessment correctly triggered a reassessment of operability and moreappropriate compensatory measures. *On June 10, 2006, the licensee identified the failure to implement routinepreventative maint enance tasks to remove biological fouling agents, corrosionproducts, and sedimentation from the essential water cooling spray ponds. The concern was documented in CRDR 2901737 and was determined to not meetthe procedural requirements for contacting the control room operators as anoperability concern based on the incorrect conclusion that the failure toimplement this task only affected the preventative maintenance process, even though it was known that the ponds had several inches of sediment buildup.

This issue should have been reviewed by the control room operators in accordance with procedural requirements because the failure to remove debris from the ponds could adversely affect the safety function of the ultimate heat sink based on water displacement and heat transfer considerations. The team concluded that the reviewers of this CRDR prevented a timely review ofoperability. This technical issue is further discussed in Attachment 4.*On June 10, 2006, the NRC inspectors discussed with the licensee concerns thatsedimentation buildup of the essential water spray ponds could potentially affectoperability of the ultimate heat sink. On June 21, 2006, licensee engineeringpersonnel provided the inspectors a calculation revision demonstrating that any sedimentation accumulation had placed the ultimate heat sink outside its analyzed condition for assuring the peak pond temperature limits could be met.

The inspectors noted that engineering should have recognized that all three units were outside their analyzed condition, but chose to evaluate the condition before discussing the operability concern with operations (CRDR 2906487 was writtenwith June 28 listed as the discovery date, and CRDR 2906671 was written for theuntimely identification of the issue). The team concluded that site personnellacked a questioning attitude and technical rigor while evaluating this degraded condition since the accumulation of sedimentation had been known for years at the site without documenting or evaluating the possible impact to operability untilquestioned by the NRC. *The team questioned whether the spray ponds had leakage which couldchallenge their ability to maintain enough water to perform their intended safetyfunction during a design basis accident. This question was raised when it was identified that there was a history of cracking in the ponds' concrete liners, and that the ponds were receiving inadequate inspection and monitoring (see Section 4.2.3). The team concluded that an operability assessment was requiredby the licensee's procedure for this condition. The licensee wrote

CRDR 2910912 and documented many of the points which would have been 27required in an operability assessment, although a discovery checklist wasinexplicably used to make the case that an operability determination was notrequired. This document reiterated the conclusion of a 2000 assessment in addressing the structural integrity function of the concrete, rather than the specific concern raised by the NRC team, which was water loss. The team was told by the engineering supervisor that performed the evaluation that a hole ofconsiderable size would be required to allow enough leakage to cause a loss of operability, so his evaluation had not addressed this aspect. The team notedthat this evaluation was based on engineering judgement, was not complete in addressing the functions affected, and was based on a perception of lack of evidence of leakage, rather than a solid body of facts. The team found that operations had exhibited a questioning attitude, and when it was noted that theevaluation had not addressed the leakage aspect, and operations had performed their own calculation to show that only 5.3 gpm leakage could be tolerated without losing operability, and this small margin was made possible by creditingthe amount of water normally maintained above the Technical Specification minimum water level. The team noted that this practice was reasonable for an operability assessment, but did not address the design problem of having nodesign margin reserved for realistic leakage values.Assessment. Failure to adequately implement the operability assessment process wasa performance deficiency. This finding was more than minor because it impacted the equipment performance attribute of the Mitigating Systems cornerstone objective to maintain the availability and reliability of systems needed to mitigate accidents. Thisfinding screened as having very low safety significance in a Phase 1 SDP, since it was confirmed not to involve any loss-of-safety function. This finding had cross-cutting aspects in the area of human performance because the licensee did not follow their systematic process for operability decision making when information was not brought tothe right decision makers.Enforcement. Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion V, "Instructions, Procedures and Drawings," requires that activities affecting quality shall be prescribed by instructions, procedures, or drawings, and shall be accomplished in accordance with those instructions, procedures, and drawings. The assessment of operability of safety-related equipmnet needed to mitigate accidents wasan activity affecting quality, and was implemented by Procedure 40DP-9OP26, "Operability Determination and Functional Assessment," Revision 16. Failure to followProcedure 40DP-9OP26, "Operability Determination and Functional Assessment,"Revision 16, to properly and promptly evaluate operability constituted a violation ofCriterion V. This issue was documented in CRDRs 2918892, 2901815, and 2898237. This violation is being treated as a noncited violation, consistent with Section VI.A of the NRC Enforcement Policy: NCV 05000528; 05000529;05000530/2006011-07, MultipleExamples of Failure to Properly Implement Operability Assessment Process.5.4Examples of Inadequate Response Under the Corrective Action ProgramIntroduction. Section 5.1.4 briefly discusses an apparent violation of 10 CFR Part 50,Appendix B, Criterion XVI, "Corrective Actions." This section provides more details and specific examples which demonstrate a pattern of behavior which did not recognize and 28address degraded heat exchanger performance between 1995 and May, 2006. Foulingand degradation of these heat exchangers and cracking in the spray pond structure was apparent from inspections, performance tests, and surveillance tests, but the licenseefailed to enter most instances into the corrective action program. Of the instances that were documented, the licensee failed to recognize that some instances constituted a significant condition adverse to quality, failed to determine the cause, and failed to implement corrective actions to prevent recurrence.Description. The Palo Verde Nuclear Generating Station corrective actions program isdocumented in Procedure 90DP-0IP10, "Condition Reporting," with Revision 28 being the current revision at the time of the inspection. Section 5.1.4 documented exampleswhere the licensee failed to properly implement the corrective action program that directly contributed to a loss of safety function in one train that had potentially greaterthan very low safety significance. The team identified the following additional examples of a violation where the safety significance was determined to be very low.Specific examples where the licensee's corrective action program was ineffectiveinclude:*Emergency diesel generator intercooler degradation was evident from availabledocumentation starting in 1995, but was not recognized until January, 1999 when it was documented in CRDR 36287. After that, the team identified noother problem reporting in the corrective action program associated with this ongoing EDG intercooler fouling until May, 2006. Instead, engineering attempted to manage the problem outside the corrective action program by repeatedly scheduling cleanings when degradation reached a threshold which was not formally established. This was determined to be an example of inadequate problem identification for a condition adverse to quality.*Problem reporting for EW heat exchanger fouling was incomplete and appearsto have been based on a relative departure from "expected fouling" instead of from an absolute reference. In the early years after implementing the zinc-phosphate chemistry control program, the team noted that EW heatexchanger performance tests measured up to 75 percent excess capacity, and on paper, these components had more than 100 percent excess capacity. By 2004, engineering was considering 30 percent excess capacity to be "expected."

This amounted to tolerating significant performance loss without reporting it into the corrective action program. The following examples associated with the EW heat exchangers' degraded performance illustrate this performance deficiency:HeatPerformanceDateExchangerMarginComments4/962B3.2%Untimely identification (CRDR 80645initiated 4 months after test).9/021B3.6% No CRDR written.

294/033A -22.0%Condition Report/Disposition Request2598216 failed to identifying that this was a significant condition adverse to quality, address the cause, or implement corrective actions to preclude repetition.10/032A1.0%Condition Report/DispositionRequest 2653867 addressed fouling and the informally attempted to determine the cause, but failed to conclude it was a significant condition adverse to quality when it was recognized that the problem affected multiple systems in all trains and all units.2/041B-17%No CRDR written, no cleaning or correctiveaction, so condition was allowed to further degrade until refueling outage in 5/04. This constituted a significant condition adverse to quality that was not entered into the corrective action program, corrected, or a cause determined.3/042A28.6%No CRDR written. This constituted acondition adverse to quality, if considered individually.5/041A9.9%No CRDR written. This constituted acondition adverse to quality, if considered individually.4/052B25.8%No CRDR written. This constituted acondition adverse to quality, if considered individually.5/052A29%No CRDR written. This constituted acondition adverse to quality, if considered individually.10/052B11.1%No CRDR written. This constituted acondition adverse to quality, if considered individually.10/053B19.3%No CRDR written. This constituted acondition adverse to quality, if considered individually.

3010/051B1.4%Condition Report/Disposition Request2835865 written. In 11/05, CRDR 2860763was written and took 3 months to address EW heat exchanger fouling in all units. Not classified as significant condition, and did not consider EDG impact. Operability Determination performed for performance test results in 10/05 was not completed until

1/06.Note that 30 percent margin was used as the threshold by the team in assessingwhen sufficient degradation had occurred to constitute a condition adverse to quality, to be consistent with the licensee's stated expectations. It could be argued that a higher threshold could be used, since no degradation should be present if the chemistry control program was performing as intended. However, it was considered sufficient to document the above examples to illustrate theextensive pattern of behavior that is being addressed in this violation.*In June, 2006, engineering personnel reviewed historical performance testingresults and noted that EW Heat Exchanger 2B may have been significantly degraded in 2002-2003. This was not entered into the corrective action program for a week, and not until the team asked. Condition Report/Disposition Request 2905161 was written to document this issue.*The corrective action process was ineffective in determining the significance ofthe fouling. The team determined that there was enough evidence to haverecognized that the fouling problems affected both trains and all units as early as 1999, which should have raised the issue to a "significant" classification, but this did not occur because the licensee did not adequately consider the extent ofcondition or identify the cause.*In December 2005, when it was recognized that the fouling was occurring fasterin EW heat exchangers to the point where their capacity would be assured only during cooler weather, some of these heat exchangers were taken out of service while the associated plant was on line to perform midcycle cleanings. However,the fact that the EDG coolers should also be expected to be experiencing the same degradation was not considered. The worsening of the degradation rate did not trigger a reassessment of the cause or significance.*In May 2005, chemistry wrote CRDR 2800653 to report that molybdenumanalyses yielded erratic results. They concluded that the analysis method was not sufficiently accurate (actual accuracy was estimated at 30-50 percent) to maintain it in the desired control band (225 - 270 PPB). Chemistry began to question whether the molybdenum tag was accomplishing what they wanted it to and requested an evaluation of alternatives. The licensee also identified that samples were untimely because of lack of qualified analysts (would sometimes wait 2 to 3 weeks to analyze samples). The CRDR also documented that theyhad stopped adding dispersant in November 2005, but still recorded increases inmolybdenum or other unexpected results in over half of the samples. The team 31concluded that this represented a missed opportunity to recognize that they werenot adding dispersant, and that dispersant concentration was no longer coupled with the tag chemical. The evaluation of alternatives considered other tag chemicals, but did not consider other dispersants which could be measured. The team noted that this CRDR was closed with the only change being to qualifymore analysts to perform the molybdate analysis.Assessment. The significance of this finding will be assessed as part of the finalsignificance determination. See Section 5.1 for the assessment of this finding.Enforcement. See Section 5.1.4 for the enforcement discussion associated with thisfinding.6.0ADDITIONAL INFORMATION6.1Summary of Chemistry Control ChangesThere were three distinct periods of interest in this inspection. From 1994 to 2001, thelicensee used Nalco (later bought by Calgon) chemicals. In 2001, the Arizona Public Service corporate system changed to Betz chemicals. Palo Verde Nuclear GeneratingStation changed to what was believed to be equivalent chemicals, but maintained essentially the same chemistry control program for the spray ponds. In 2004, the dispersant was changed because it was thought that the dispersant was reacting withthe biocide and forming an acrylic polymer and fouling heat exchangers.The licensee implemented zinc-phosphate chemistry control in the SP systems in 1994. The proprietary specialty chemicals used were:Dispersant:CALGON PCL-401 (AA/AMPS)Corrosion inhibitor:CALGON MSW-109 (zinc chloride in phosphoric acid (3:1)

Biocide: BULAB 6002 Chemical supplier: NALCO/CALGONInitially, zinc limits were 1.0 - 1.5 ppm, the pH band was 7.6-8.2, and dispersant wasadded three times per week.Between 1995 and 1999, some degraded performance was noted in EDG intercooleroutlet temperatures. This was not documented until 1999. Also, in 1995, the spray pond system began to have corrosion problems where localized corrosion nodulesformed and broke off, blocking heat exchanger tubes.On 4/24/96, the pH band was lowered from 7.6 - 8.2 to 7.4 - 8.0.In 1997, the licensee attempted to chemically passivate spray pond piping bysignificantly lowering pH to remove corrosion products, then reestablish a protective zinc layer. This was unsuccessful.On 3/3/98, pH specification was raised from 7.4 - 8.0 to 7.6 - 8.6. On 8/13/98, thedispersant addition time was increased from 2.5 to 3 hours3.472222e-5 days 8.333333e-4 hours 4.960317e-6 weeks 1.1415e-6 months .

32On 9/18/98, the licensee split the requirements so that Unit 1 had different limits, sincethey were changing Unit 1 so makeup water came from wells. Since this water had higher alkalinity and thus a higher scaling potential, Unit 1 was given a lower pH band to improve solubility and avoid scaling. Unit 1 had a pH specification of 7.8 - 8.2, whileUnits 2 and 3 had 8.0 - 8.4. However, on 7/6/99, this was reversed, and all units had apH specification of 8.0 - 8.4. No explanation was given why the increased potential forscaling was acceptable.On 7/11/00, the upper zinc specification was changed from 1.0 to 1.5 ppm.In 2001, the corporate chemistry group for Arizona Public Service changed chemicalsupply companies. On 5/30/01, the licensee began adding the following new chemicals:Dispersant:BETZ PY5200 (AA/HPS-1)Corrosion inhibitor:BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002 Chemical supplier: BETZIn October 2003, the licensee performed the first EW heat exchanger tube cleaning in2R11 in response to fouling. Since then, 100 percent of the EW heat exchanger tubesare cleaned every refueling outage.On 1/29/04, the licensee concluded from informal lab testing that EW heat exchangerfouling was being caused by an undesirable interaction between the dispersant and the biocide. This was based on discussions with the chemical supplier's site representative, and included informal lab tests which showed that a white substance could be formed by mixing the chemicals, although the conditions of this test were not documented and the substance formed was not analyzed. The team noted that the conclusions drawn fromthis informal test dominated the licensee's response through June, 2006.In March 2004, the dispersant was changed to address EW heat exchanger fouling. The chemicals were now:Dispersant:BETZ DN2317 (AA/APES)Corrosion inhibitor:BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002 Chemical supplier: BETZThe target dispersant concentration was 12.5 - 15 ppm. The previous dispersant wasnot removed from the ponds through feed and bleed, but was expected to continue to react and be removed by spray pond sand filters. There was no evaluation of adding a new dispersant with the previous dispersant still present, or an analysis of theacceptability fo the new dispersant for the system (LIST CRDR). Also, the zinc specification was doubled from 0.5 - 1.5 ppm to 1.0 - 3.0 ppm.In November 2004, a white "lotion-like" substance identified in Spray Pond 2C sandfilter. The cause was believed to be having the old dispersant and the new dispersant, polymerizing and coagulating into larger particles (essentially doing what they were supposed to do, but at a higher rate). It was not believed to be a problem for heat 33exchangers because of the mobility of the substance and most recent EW heatexchanger performance test showed improving trend (the team noted that this was actually after a midcycle cleaning with only 6 months of operation, so it was not useful toconsider this data). The licensee implemented more frequent backwashing of spray pond sand filters, believing that this would remove the problem chemical. Chemistry explained that this took 10 months to happen after changing dispersants by saying it was because of cooler temperatures in the pond. However, the team noted that this didnot take into consideration that the dispersant change took place in January when theponds were also cold. In May 2005, chemistry wrote CRDR 2800653 to report that molybdenum analysesyielded erratic results. They concluded that the analysis method was not sufficiently accurate (actual accuracy was estimated at 30-50 percent) to maintain it in the desired control band (225 - 270 PPB). Chemistry began to question whether molybdenum tag was accomplishing what they wanted it to and requested an evaluation of alternatives.

The licensee also identified that samples were untimely because of lack of qualified analysts (would sometimes wait 2 to 3 weeks to analyze samples). The CRDR alsodocumented that they had stopped adding dispersant in November 2005, but stillrecorded increases in molybdenum or other unexpected results in over half of the samples. The team concluded that this represented a missed opportunity to recognize that they were not adding dispersant, and that dispersant concentration was no longercoupled with the tag chemical. The evaluation of alternatives considered other tag chemicals, but did not consider other dispersants which could be measured. The teamnoted that this CRDR was closed with the only real change being to qualify moreanalysts to perform the molybdate analysis.On 10/17/05, the pH specification was changed from 8.0 - 8.4 to 7.8 - 8.4 in order toimprove chlorine effectiveness and zinc solubility.The team noted that between October 2005 and May 2006, both spray ponds in Units 2and 3 had molybdate results that were above the target level of 200-300 ppb almost continuously, even though dispersant was largely not being added. The validity of theseresults was not questioned. The team considered that this was imprudent, since evaporation required adding 2,200 to 10,000 gallons of makeup water per day if the pond was idle, more if the sprays were run. Of note, Unit 1 spray ponds were maintained within the target band during most of this period. The team reviewed the dispersant usage by assessing tank levels in each unit. Between October 2004 and August 2005 little or no dispersant was added to any of the ponds. From August 2005 until May 2006, dispersant was added fairly regularly in Unit 1, less regularly in Unit 3, and infrequently in Unit 2. This was known to the chemistry line organization, but not to management or the root cause team until the inspection team requested theinformation. The team also compared the recent dispersant usage to earlier usage.

Since changing to the latest dispersant in March 2004, the total dispersant used in 27 months was about equal to the annual used in years prior to that change. The lack of regular additions coupled with rising concentrations of other chemicals tended to indicate that there was increasingly insufficient dispersant to prevent loss of solubility in the spray pond systems.

34The team assessed the history of differences in chemistry parameters between theunits. The team noted that Unit 2 consistently had the highest levels of phosphates, andthe pH was above 8.2 for long periods of time compared to the other units. Unit 1 had roughly double the calcium compared to the other units; this was known to be because of using well water in this unit.

7.0SUMMARY OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE 1.Sequence of EventsThis topic is documented in Attachment 4.

2.Operating ExperiencePersonnel from the Operating Experience Branch of NRR supported the team byperforming searches of operating experience databases and other sources. The intent was to identify operating experience report of similar problems and other relevant information, both within and outside the nuclear industry. Palo Verde Nuclear Generating Station had one instance of tube blockage, which was documented inLicensee Event Report 50-528/95-005-00, "Corrosion Nodules Blocking EDG 1B Jacket Water Cooler Tubes." This involved chemistry challenges in the spray pond system , butdid not involve precipitation or scaling. Only two reports of problems with similar circumstances were identified, although the causes were different. It was noted, however, that outside the nuclear industry, this type of condition would probably not be considered to be serious.

However, these searches identified information that indicated that zinc-phosphate chemistry control programs similar to the one used by Palo Verde Nuclear Generating Station were in common use in similar systems, and were usedeffectively.

3.Generic IssuesThe chemical control program in use in the Palo Verde Nuclear Generating Stationspray ponds and associated systems were generally compatible when used together ina coordinated program. This type of program was in common use and effectively employed in similar applications. However, the unique design of the spra y pond systemand the unique operating practices were not adequately reviewed to ensure compatibility.The only generic issue identified was that AA/HPS-1 type dispersant is know to interactwith polyquat amine-type biocide and form an acrylic polymer that can cause water problems, including possibly fouling. The acrylic acid in this product has a strong negative charge compared to other dispersants, and will combine with the biocide.

4.Corrective ActionsDuring this inspection, the licensee corrected the limits for the spray pond in thechemistry control program, and restored the ponds to within the new limits by blowing down the ponds. They were also evaluating changing the dispersant to regain the abilityto monitor it. The ponds were being vacuumed and inspected to remove chemical 35sludge and sediment, and to verify pond integrity. Part 50.59 reviews were planned forpast changes to chemistry procedures. Emergency diesel generator intercoolers were promptly cleaned, and increased frequency monitoring of performance was implemented while chemistry improvements were being identified and implemented.

5.Root Cause Determination.The licensee's determination of the cause was broad and thorough in assessing theproblems and the history. This effort included looking at both the technical issues and the organizational aspects, which was appropriate. Adequate resources were devoted this effort, including obtaining several sources of outside expertise in a number of important aspects of the evaluation. The draft report, although awaiting some technical information, thoroughly documented the facts available, the process used, the possiblecauses considered, and the conclusions. This effort also identified a number of peripheral issues that were entered into the corrective action program.

6.Design Basis and Technical Specifications ComplianceA number of problems were identified with the design basis and Technical Specificationcompliance. These are discussed in the Findings section.

7.Operability Assessments and Compensatory MeasuresA number of problems were identified with operability assessments. These arediscussed in the Operability Assessment section. The compensatory measures andinitial corrective actions implemented by the licensee were appropriate and timely, with the exception of stopping dispersant additions. In retrospect, this was inappropriate based on the later understanding of the cause of the fouling; however, from the perspective of what was known at the time, this was not unreasonable. Dispersant additions were resumed in a prompt manner when the cause was better understood.

8.Heat Exchanger TestingThe test methodology was reviewed along with several years worth of test results. Themethodology was in accordance with industry standards, although the calculations provided results which were not typical. Specifically, the heat exchanger performance was calculated for the existing conditions, and the acceptance criteria (reflecting design basis conditions) were extrapolated to the existing condition results. The normal practice is to extrapolate the results from the existing conditions to design basis conditions. The team concluded that, while the licensee's practice made comparison of results technically awkward, this did not affect the ability to demonstrate pr operperformance. The team also identified that the licensee's practice of having the test andthe calculation as separate activities contributed to having untimely results from the performance tests in at least one case, and possibly others.

9.Risk Analysis 36Information was gathered to support a preliminary risk assessment. However, keyinformation was still needed to be able to perform a final risk analysis. This is furtherdiscussed in the Findings section under the Analysis heading.

10.Methods of Heat Exchanger ProtectionThe team assessed the chemistry control program design and implementation,operating practices, inspection and cleaning methods and results, and the cathodic protection used in portions of the system. As discussed in various parts of this report,the chemistry control program was changed over time in ways that created heat exchanger fouling. Corrosion control was being effectively implemented, however.

11.Effectiveness of Heat Exchanger Performance MonitoringAs discussed in various parts of this report, performance monitoring was not effective inidentifying and correcting heat exchanger degradation. The tests and inspections were being done according to commitments, but the results that indicated increasing amountsof chemical residue, and the need for increasing frequent cleanings, were not followed up to determine and correct the cause. This was an organizational problem, rather than a problem with the monitoring methods or frequencies.

1R07 Biennial Heat Sink Performance (71111.07B)

a. Inspection Scope

The team reviewed design documents (e.g., calculations and performancespecifications), program documents, implementing documents (e.g., test procedures, maintenance procedures, and actual data), and corrective action documents. The team interviewed chemistry personnel, maintenance personnel, engineers, and program managers. The team verified whether testing, inspection and maintenance, or the biotic foulingmonitoring program provided sufficient controls to ensure proper heat transfer.

Specifically, the inspectors reviewed heat exchanger test methods, test results from performance testing, inspection results, and chemical controls to limit fouling. For the ultimate heat sink and its subcomponents, the team reviewed the heat sink todetermine if it was free from clogging because of macrofouling and provided sufficient controls to ensure proper heat transfer. The inspectors reviewed;

(1) heat exchanger test methods and test results from performance tests,

(2) heat exchanger inspection and cleaning methods and results, and

(3) chemical treatment for the spray ponds to control fouling. The team selected the following heat exchangers for this inspection: *Essential cooling water heat exchanger*Emergency diesel generator jacket water heat exchanger

Emergency diesel generator inter-coolerThe team completed 3 of the 2 to 3 required samples.

b. Findings

An apparent violation of Criterion XI, "Test Control," associated with EW heatexchanger testing was discussed in Section 5.1.2. A minor violation of 10 CFR Part 50, Appendix B, Criterion III, was identified for using an inappropriate foulingfactor in design documents for the EDG intercoolers. The team identified that fouling factors for clean water (0.0005) was used on both sides of these heat exchangers, where fouling factors of 0.002 was appropriate for the application.

The latter value was correctly used in the other EDH heat exchangers. This was determined to be minor because the team calculated that this had a small impact(4 percent) to the point at which these heat exchangers would not be able toperform their intended function, compared to the large margins available in the EDG system. This minor issue was entered into the corrective action program as Action Request 051201123.4OA6Meetings, Including ExitOn August 18, 2006, the inspectors presented the inspection results to Mr. C. Eubanks,Vice President, Nuclear Operations, and Mr. D. Mauldin, Vice President, Engineering, and other members of his staff, who acknowledged the findings.An additional exit was conducted on September 26, 2006, to provide the results ofpreliminary significance determination and to change the characterization of the associated issues to Apparent Violations. The findings were discussed with Mr J.

Levine and members of his staff, who acknowledged the findings.The inspectors confirmed that proprietary information that was examined during thisinspection was properly handled in accordance with NRC policy.4OA7Licensee-Identified ViolationThe following violation of very low safety significance (Green) was identified by thelicensee and was a violation of NRC requirements which meets the criteria of Section VIof the NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs.*Part 50,59 of Title 10 Code of Federal Regulations allows licensees to makechanges to procedures described in the Updated Final Safety Analysis Report without prior NRC approval provided that the change satisfies thecriteria in Part 50.59.c. Contrary to this, Revisions 2, 6, 7, 8, and 9 to Procedure 74DP-0CY01, "Specification for Bulk Chemicals," were made between 1997 and 2004 without 50.59 reviews. This was identified in the licensees corrective action program under CRDRs 2864575 and 2683642. Thisviolation is of very low safety significance because it did not actually impede the regulatory process or contribute to degraded heat exchanger performance. ATTACHMENT 1:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

G. Andrews, Department Leader, System Engineering

P. Borchert, Director, Operations

D. Breckenridge, Consulting Engineer, Arizona Public Service

P. Carpenter, Unit Department Leader, Operations

C. Churchman, Director, Engineering

A. Dave, Senior Engineer, Design Engineering

C. Eubanks, Vice President, Nuclear Operations

T. Green, Team Leader, Chemistry

D. Hautala, Senior Compliance Engineer

P. Heinstein, Chemical Vendor Site Representative, GE Betz

J. Hughey, Senior Engineer, System Engineering

H. Hurley, Root Cause Team Sponsor

R. Jenkins, Senior Chemist, Chemistry

D. Kanitz, Senior Compliance Engineer

M. Karbasian, Department Leader, Design Mechanical Engineering

J. Levine, Executive Vice President, Generation

D. Mauldin, Vice President, Engineering

J. Proctor, Section Leader, Regulatory Affairs - Compliance

M. Radspinner, Section Leader, System Engineering

C. Seaman, General Manager, Regulatory Affairs and Performance Improvement

T. Selby, Chemistry Consultant

G. Sowers, Section Leader, PRA

D. Straka, Senior Consultant, Regulatory Affairs

D. Vogt, Section Leader, Operations/STANRC pers onnel

J. Melfi, Resident Inspector, Palo Verde Nuclear Generating Station

E. Owen, Reactor Inspector, Region IV

N. Sieller, Operations Experience Branch, NRR

W. Sifre, Senior Reactor Inspector, Region IV

R. Telson, Operations Experience Branch, NRR and Acting Resident Inspector, PVNGS

S. Unikewicz, Senior Engineer, NRR

G. Warnick, Senior Resident Inpspector, Palo Verde Nuclear Generating Station

G. Werner, Senior Project Engineer, RIV

1A1-2

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

05000529/2006011-01,AVEW Train 2B Inoperable Longer than AllowedOutage Time (Section 5.1.1)

05000528;

05000529;05000530/2006011-02AVInadequate Test Control to Promptly IdentifyUnacceptable Performance Test Results (Section

5.1.2)05000528;

05000529;05000530/2006011-03AV50.59 Reviews Not Performed or Inadequate forMultiple Changes to Spray Pond Chemistry Control

Procedure (Section 5.1.3)

05000528;

05000529;05000530/2006011-04AVInadequate Identification and Corrective Action forDegraded EW Heat Exchanger Performance

(Section 5.1.4)

05000528;

05000529;05000530/2006011-05AVInadequate Design Control to Ensure No EW HeatExchanger Scaling.(Section 5.1.5)

Opened and Closed

05000528;

05000529;05000530/2006011-06NCVTwo Examples of Failure to Translate Spray PondDesign Assumptions Into Plant Procedures Control

(Section 5.2)

05000528;

05000529;05000530/2006011-07NCVMultiple Examples of Inadequate OperabilityAssessments for Heat Exchanger Degradation

(Section 5.3)

LIST OF DOCUMENTS REVIEWED

Procedures30DP-9MP08, Preventive Maintenance Program, Revision 12

40OP-9SP01, Essential Spray Pond Train A, Revision 35

40DP-9OP26, Operability Determination and Functional Assessment, Revision 1670TI-9EW01, Thermal Performance testing of Essential Cooling Water Heat Exchangers,Revision 570DP-9SP01, Monitoring of ESPS Piping Integrity, Revision 3

73DP-9ZZ10,

Guidelines for Heat Exchanger Thermal Performance Analysis, Revision 4

1A1-373DP-9ZZ11, Heat Exchanger Condition Monitoring,

Revision

474DP-9CY04, Systems Chemistry Specifications, Revision 37 (other revisions were alsoreviewed)90DP-0IP10, Condition Reporting, Revision 28

Corrective Action Documents

2474313 2484639

21395

2591826

2598216

2653867

2654231

2683642 2699739

2748191

2750856

2789993

2819671

24784

2824865 2825469

28929

2835865

2837696

2859430

2860763

2864575 2870799

2886930

2896661

2897810

2898120

2898237

2896661 2901500

2901737

2901815

2902498

2902504

2904101 2905161

2905162

2906671

2908351

2987810Work Orders

2537300 2723814

23815

23818

2723819 2724390

24394

2805525

2805526 2886543

2886547

2896333

2897078 2897080

2897128

2897130 2898676

2898679Deficiency

Work Orders

2564416 270028650.59 DocumentsScreening S-06-0049, Revision 1, "Add Boron-10 Specifications to 74DP-9CY04 (Rev 36)," dated 2/15/06Calculations13-MC-DG-206,"Corrosion Allowance For DG Inter-cooler Water Boxes End Plates, datedJanuary 15, 199113-MC-DG-411, "DG Heat Exchanger Minimum Flow Rate vs. Inlet SP Water Temperature,"dated may 7, 199613-MC-DG-410, "Spray Pond Minimum Flow Rates To Diesel Generator Heat Exchangers," Revision 1 13-MC-DG-206, "Corrosion allowance For DG Inter-cooler Water Boxes End Plates," datedJanuary 15, 199113-MC-DG-403, "Diesel Intake Exhaust and Drain Pipe P/T," Revision 0

1A1-413-MC-SP-0307, "SP/EW System Performance Design Bases Analysis," Revision 00713-NS-B098, At-power PRA Study for the HVAC Room Coolers, Revision 2

MiscellaneousGE Betz Memo, "SEM/EDXA Analysis of Deposit Samples from Arizona Public Service - PaloVerde Nuclear Generating Station," Dated December 1, 2003Preventive Maintenance Basis Document 248948

Updated Final Safety Analysis Report Section 9.2, Cooling Water Systems, and Section 9.5.5,Diesel Generator Cooling Water System, Revision 13Regulatory Guide 1.27, "Ultimate Heat Sink for Nuclear Power Plants," Revision 2VTD-P162-0002, "Perfex Industrial Products Data Sheets and Drawings for Diesel Generator Inter-Coolers" Revision 113-NS-C076, "MSPI Data," Revision 00

RCTS

010359,

IEC 78-13 Perform Periodic Inspection of Spray Pond Intake Structure forSand/Silt Buildup dated 6/22/90.LDCP Number 2LM-EW-036, Sleeving of Unit 2 Trains A and B Essential Cooling Water HeatExchangers, 9/9/93PVNGS Design Basis ManualsAuxiliary Building HVAC System, Revision 13

Control Building HVAC System, Revision 13

Essential Cooling Water System, Revision 18

Essential Chilled Water System, Revision 10

Essential Spray Pond System, Revision 15

Diesel Generator, Class 1E Standby Generation, Fuel Oil Storage and Transfer System, Revision 016Operating ExperiencePalo Verde Nuclear Generating Station Licensee Event Report 50-528/95-005-00, CorrosionNodules Blocking EDG 1B Jacket Water Cooler TubesBraidwood Operating Event Report, "Lake Chemistry Trend, Calcium Carbonate Issue," dated

2/17/04Braidwood Operating Event Report on Calcium Carbonate Precipitation on the Non-essentialService Water Strainers, dated 4/18/02

1A1-5NRC Information Notice 94-79, Microbiologically Influenced Corrosion of Emergency DieselGenerator Service Water Piping, 11/23/94NRC IE Circular 78-13, Inoperability of Multiple Service Water Pumps, 7/6/78RegulatoryCommitment Tracking System Number

010359, Perform Periodic Inspection of Spray Pond Intake Structure for San/Silt Buildup

2A2-1ATTACHMENT 2SPECIAL INSPECTION CHARTERJune 7, 2006MEMORANDUM TO:Neil O'Keefe, Senior Reactor InspectorEngineering Branch 2

Division of Reactor SafetyFROM:Dwight D. Chamberlain, DirectorDivision of Reactor Safety SUBJECT:SPECIAL INSPECTION CHARTER TO EVALUATE PALO VERDE NUCLEARGENERATING STATION, UNITS 1, 2, AND 3 HEAT EXCHANGER FOULING You are hereby designated as the Special Inspection Team leader.

Your team members arePaula Goldberg, Yamir Diaz-Castillo, Krzysztof Parczewiski, and Mike Runyan.A Special Inspection Team is being chartered in response to the discovery of fouling of theemergency diesel generator (EDG) intercooler heat exchangers and essential cooling water heat exchangers.

Fouling of each of the heat exchangers cooled by this system could lead to afailure of the EDGs to supply rated electrical power to safety-related components and a reduction in cooling to essential chilled water (room cooling for emergency core cooling systemcomponents), shutdown cooling, fuel pool cooling, and nuclear cooling water.

The licensee implemented actions to inspect and clean the affected heat exchangers in order to restore margin for safe ty system operability.A.BasisOn May 17, 2006, during surveillance testing of the Unit 2 B Train EDG, the licenseeobserved an abnormal increase in turbocharger air temperature.

The cause of the increase is associated with fouling of the EDG intercooler heat exchanger.

On May 19, during a review of the increased turbocharger air temperature, the licensee determined that a reasonable expectation of operability no longer existed and declared the EDGinoperable.

The licensee subsequently cleaned the intercooler heat exchanger and declared the EDG operable.

The licensee also initiated work packages to clean and inspect the remaining five EDG intercooler heat exchangers.

Each of the EDG

intercooler heat exchangers had experienced an increase in turbocharger air temperature, but not to the same magnitude as the Unit 2 Train B

EDG.On May 25, 2006, a teleconference was conducted with the licensee to discuss theextent of the condition and cause of the intercooler heat exchanger fouling.

The licensee indicated that in 2001, a chemical additive (dispersant) to the spray pond reacted with a biocide to create the chemical substance that was found on the EDGintercooler heat exchangers and the essential cooling water heat exchangers in all three units.

Dispersant is used to prevent ions (e.g., iron) from plating out on spray

pond system surfaces.

The dispersant coats the ions and aids in the filtration of theions from the spra

y pond system.

The licensee indicated that the fouling of the heatexchangers was initially discovered in 2003.

2A2-2In March 2004, the licensee changed the type of dispersant and implemented morefrequent (once per refueling outage) inspections and cleaning of the essential cooling water heat exchangers.

The licensee indicated that heat exchanger fouling decreased following the decision to change the type of dispersant.

However, recent performance testing of the essential cooling water heat exchangers, EDG testing, and heat exchanger inspections have shown that the rate of fouling of the heat exchangers is increasing.

The licensee indicated that degradation of the EDG intercooler heat exchanger wouldnow be detected during surveillance testing.

The resident inspectors noted that theUnit 2 Train B EDG had operated at higher temperatures for several months without being questioned by the licensee.

The licensee also indicated that a performance test of the essential cooling water heat exchanger is performed whenever they initiate shutdown cooling operations and that the essential cooling water heat exchangers arecleaned every 18 months.

However, the inspectors noted that the rate of fouling of the essential cooling water heat exchangers may result in a reduction of essential cooling water heat exchanger performance to below design margins during the operating cycle. The inspectors also noted that the licensee's actions were focused on cleaning degraded heat exchangers and not on eliminating the source of the chemical fouling.The preliminary risk assessment for this condition determined that the increase inconditional core damage probability was 1E-6.

The risk assessment assumed that thefouling of the EDG intercooler heat exchangers and essential cooling water heat exchangers may affect the reliability of the following components, either directly orthrough a loss of room cooling:- emergency diesel generators- motor and turbine-driven auxiliary feedwater pumps- high pressure injection pumps

- low pressure injection pumps

- containment spray pumps

- shutdown cooling heat exchangersTo assess the risk of the condition, the analyst performed a sensitivity analysis todetermine the amount of change in unreliability of these components that would cause achange in core damage probability of 1.0E-6, considering an exposure period of 1 year.Using the Palo Verde Nuclear Generating Station SPAR model, Revision 3.2.1, andassuming average test and maintenance, the analyst determined that an increase inunreliability of 30 percent of all affected components would result in an increase in coredamage probability of 1.0E-6.

To attain this result, the fail-to-run individual and commoncause basic events were adjusted by the same amount for all components except for the shutdown cooling heat exchangers (a basic failure event was adjusted in this case).

2A2-3B.ScopeThe team is expected to address the following:1.Develop a complete sequence of events related to the discovery of the chemicalfouling of the EDG and essential cooling water heat exchangers.

2.Compare operating experience involving fouling of heat exchangers to theidentified issues at Palo Verde Nuclear Generating Station.

3.Determine if there are any generic issues related to the use of multiple chemicalsand their subsequent impact on operating systems.

Promptly communicate anypotential generic issues to regional management.4.Review the extent of condition determination for this condition and whether thelicensee's previous and planned corrective actions are comprehensive. 5.Review the licensee's determination of the cause of any deficiencies and/oroperating practices that allowed the chemical fouling to occur and continue.

Independently verify key assumptions and facts.

If available, determine if the licensee's root cause analyses and corrective actions have addressed the extent of condition.6.Determine if the design basis and technical specifications were met for theeffected systems.

7.Determine if the operability assessments, supporting analyses, andcompensatory measures for heat exchanger fouling were made in accordance with

RIS 2005-20.8.Evaluate the adequacy of heat exchanger testing and calculations that wereperformed to evaluate the effect of heat exchanger fouling. 9.Collect data necessary to support a risk analysis.

Specifically obtain informationassociated with the degree to which the EDG and essential cooling water systems would be affected during an event.10.Assess the method(s) used by the licensee to protect the heat exchangers fromcorrosion, scale formation, fouling, and other biological impacts.11.Assess the effectiveness and appropriateness of the tests and inspections usedto determine heat exchanger performance and capability, including frequencyand acceptance criteria.C.GuidanceInspection Procedure 93812, "Special Inspection," provides additional guidance to beused by the Special Inspection Team.

Your duties will be as described in InspectionProcedure 93812.

The inspection should emphasize fact-finding in its review of the

2A2-4circumstances surrounding the event.

It is not the responsibility of the team to examinethe regulatory process.

Safety concerns identified that are not directly related to theevent should be reported to the Region IV office for appropriate action.The Team will report to the site, conduct an entrance, and begin inspection no later thanJune 19, 2006.

While on site, you will provide daily status briefings to Region IVmanagement, who will coordinate with the Office of Nuclear Reactor Regulation, toensure that all other parties are kept informed.

A report documenting the results of theinspection should be issued within 45 days of the completion of the inspection.This Charter may be modified should the team develop significant new information thatwarrants review.

Should you have any questions concerning this Charter, contact me at

(817) 860-8180.

3A3-1ATTACHMENT 3SUMMARY OF HEAT EXCHANGER PERFORMANCE TEST RESULTSEW 1AEW 1BEW 2AEW 2BEW 3AEW 3B3/92 1R2;tube leak identifiedAll tubessleeved in

2/93 (8" at inlet and outlet)

becasue of cracking in

2R4All tubessleeved in

2/93, (8" at inlet and outlet)

because of cracking in

2R44/94, 3R4:Perf test

54.1%

margin 12/94 BeganZinc-

Phosphate chemistry control program.12/94 BeganZinc-

Phosphate chemistry control program.4/94 Unit 2Bpond begins trial use of Zinc-

Phosphate chemistry control program.12/94 BeganZinc-

Phosphate chemistry control program.4/95, 1R4:perf test 56%

and 33.8%

margin at outage beginning and end.4/95, 1R4perf test

39.8%

margin2/95, 2R5:perf test 56%

margin.3/95, 2R5:perf test

41.9%

margin.11/95, 3R5;

end of outage perf test 24.9%

margin.10/95, 3R5;beginning of outage perf test 42.8%

margin.10/96, 1R6;

end of outage perf test 34.3%

margin.9/96, 1R6;early outage perf test

55.7%

margin.2R6 - No testor cleaning.4/96, 2R6;beginning of outage perf test 3.2%

margin.

No cleaning,

CRDR

written 4

months later.3R6 - No testor cleaning.3/97, 3R6;early outage perf test

37.1%

margin.

EW 1AEW 1BEW 2AEW 2BEW 3AEW 3BAttachment 3A3-23/98, 1R7;early outage perf test

58.7%

margin.4/98, 1R7;

end of outage perf test 43.5%

margin.9/97, 2R7;early outage perf test

37.8%

margin.10/97, 2R7;chemical passivation

(cleaning).

End of outage perf test 38.6%

margin.10/98, 3R7;beginning of outage perf test 55.5%

margin.10/98, 3R7;

end of outage perf test 71.7%

margin10/99, 1R8 -No test or cleaning.10/99, 1R8;beginning of outage perf test 37.8%

margin.4/99, 2R8;early outage perf test

2.6%

margin.4/99, 2R8;late outage perf test 64.7

% margin.4/00, 3R8;early outage perf test

40.8%

margin.4/00, 3R8 -No test or cleaning.4/01, 1R9;early outage perf test

31.5%

margin.1R9 - No testor cleaning.10/00, 2R9;early outage perf test

27.6%

margin.10/00, 2R9 -No test or cleaning.3R9 - No testor cleaning.10/01, 3R9;early outage perf test 24%

margin.9/02, 1R10 -No test or cleaning.9/02, 1R10;early outage perf test

3.6% margin.

No CRDR

written, no cleaning.9/02, 2R10 -No test or cleaning.3/02, 2R10;early outage perf test

-9.5%

margin. No cleaning. 4/03, 3R10;early outage perf test

-22.0%

margin before cleaning,

26.8% after.

All tubes inspected with boroscope.4/03, 3R10 -No test or cleaning.10/03, 2R11;early outage perf test

1.0% margin.

Post-

cleaning margin

2.8%.10/03, 2R11;EW HX 2B

was cleaned

but not tested.CRDR

2653867 ACE concluded dispersant was interacting with biocide in Jan 04.

Changeddispersant starting with a batch add to each pond in 2/04.

EW 1AEW 1BEW 2AEW 2BEW 3AEW 3BAttachment 3A3-3 2/04,midcycle outage perf test -17.1%

margin.3/04,midcycle outage perf test 28.6%

before cleaning.

3/04,midcycle outage, cleaning performed but no test.

3/04,midcycle outage;

cleaning performed but no test.

3/04,midcycle outage;

cleaning performed but no test.5/04, 1R11;9.9% margin before cleaning,

34.5% after.5/04, 1R11 -EW HX 1B

cleaned but not tested.10/04, 3R11EW HX 3A

cleaned but not tested.10/04, 3R11;

34.6%

margin before cleaning,

26.8% after."Lotion-like" substance reported in water from Unit 2 sand filter.

CRDR 2750856.5/05, 2R12;post-cleaning margin 29%.4/05, 2R12;margin

25.8% before cleaning.1R12 - EWHX 1A

cleaned but not tested.

Fouling present.10/05, 1R12;1.4% marginbefore cleaning,

41.4% after.

Fouling present.10/05,midcycle outage; perf test 11.1%

margin.

No cleaning performed.

10/05,midcycle outage; perf test 19.3%

margin.

No cleaning performed.

6/06,midcycle outage; perf test 28.6%

before cleaning,

36.3% after.

3/06,midcycle outage; perf test 32.9%

margin before cleaning.5/06 -midcycle cleaning performed.

5/06 -midcycle cleaning performed.4/06, 3R12;

30.3%

margin before cleaning,

36.3% after.3R12; cleaned but not tested.

4A4-1ATTACHMENT 4SEQUENCE OF EVENTS1994Licensee implemented zinc-phosphate chemistry control program for spray pondsystem.1995Preventive Maintenance Basis for spray ponds revised to specify removingsediment every two cycles.

No preventive maintenance item was ever written toimplement this.4/96Essential Cooling Water 2B performance measured 3.2 percent margin.

CRDR 80645 written 8/27/96.9/98Unit 1 spray pond makeup source changed to well water.

Lower pH rangespecified for Unit 1 to account for higher alkalinity to avoid scaling.1/99Degraded performance in EDG intercoolers first noted.

It was later determinedto have been evident from records starting in 1995.

Condition Report/Disposition Request 36287 written.7/99Unit 1-specific pH specification removed, so it was effectively raised withoutevaluating the increased scaling potential.10/00Essential cooling water heat exchanger performance begins to showperformance results below 30 percent margin.

Prior to this, results were typically to 70 percent margin.5/01Licensee changed chemical vendors for water treatment of spray ponds.3/02Essential Cooling Water 2B performance measured significant negative margin. Condition Report/Disposition Request

2521395 written 5/30/02 to evaluate calculational problems.

Revised 7/10/02 when another calculation error identified.

September 2002 finally concluded -9.5 percent margin was the correct result.

No corrective action for degraded equipment performance.9/02Essential Cooling Water 1B performance measured at 3.6 percent margin.

NoCRDR written.

4A4-24/03Upper zinc specification raised to 1.5 ppm.4/03Essential Cooling Water 3A performance measured at -22 percent margin.

Theheat exchanger was cleaned to restore performance.

Condition Report/Disposition Request

2598216 written.10/03Essential Cooling Water 2A performance measured at 1.0 percent margin.

Bothtrains' heat exchangers cleaned, and routine cleanings were initiated.

11/03Declining performance trend identified in EW heat exchangers in all units. Condition Report/Disposition Request

2653867 written.

Also recognized EDG

intercoolers had experienced similar fouling since 1995.

Zinc precipitation identified as "the primary factor."

Operability determination completed thatshowed this and the 4/03 result in 3A were both capable of performing its safety function under existing conditions (but not under the more limiting design basis conditions).

Compensatory measure established to monitor spray pond temperature to ensure continued operability.1/04Apparent cause for

CRDR 2653867 concluded that dispersant was combiningwith biocide, forming sticky substance that was coating tubes.

2/04Essential Cooling Water 2B perfromance measured at -17.1 percent.

No CRDRwritten.

Operability was considered to be bounded by a previous operabilitydetermination.2/04Self-assessment identified that chemistry changed spray pond chemical evndorand chemicals without evaluating impact.

Condition Report/Disposition Request

2683642 written.3/04Changed dispersant.

No action taken to remove old dispersant through feedingand bleeding the ponds.3/04Mid-cycle cleaning performed for Unit 3 EW heat exchangers.4/04Essential Cooling Water 1A performance measured at 9.9 percent margin.10/04Upper zinc specification doubled to 3.0 ppm.

4A4-311/04White lotion-like substance identified in Unit 2 sand filter, CRDR 2750856written.

It was concluded that this was the old dispersant reacting and being removed by sand filter, which would help remove the remaining old dispersant.10/05Essential Cooling Water 1B performance measured at 1.4 percent margin.10/05Units 2 and 3 entered unscheduled outages.

Essential Cooling Water 2Bperformance measured at 11.1 percent and 3B measured at 19.3 percent margin.11/05Condition Report/Disposition Report

2860763 written to address loss ofperformance of Unit 1 EW heat exchangers.1/06 The operability determination for

CRDR 2860763 completed.1/06Condition Report/Disposition Report

2835865 written to address degradedperformance in all three units' EW heat exchangers in 10/05 tests.

This was the first time that licensee addressed continuing degradation and need for cleaning to assure heat exchangers would remain operable through end of the cycle. Unit 3 was expected to be operable until cleaning in spring outage.

Unit 1 was cleaned in 10/05.

Unit 2 was scheduled for mid-cycle cleanings before summer(5/06).4/06Unit 3 EW heat exchangers cleaned.5/17/06Essential Cooling Water 2B cleaned.

Emergency Diesel Generator 2Bintercooler fouling noted during surveillance run, when air intake temperature jumped up 29F from last test to 145F.5/18/06Unit 2 Train B removed from service for EW cleaning.5/19/06Train B returned to service.

Operability of EDG 2B revisited.

Concl uded thatthere was not a reasonable expectation of operability, so it was declaredinoperable.

All other EDGs run to check performance to comply with Technical Specification 3.8.1, Action B.3.2.

Emergency Diesel Generator 2B intercoolerscleaning initiated, CRDR 2896661.5/20/06At 7:40 p.m., engineering issued a white paper that set a 160F limit ofconfidence that EDGs would remain operable if they knew air temp would stay below this level.

4A4-45/23/06Operations lifted compensatory measure of monitoring pond temperatures basedon having completed cleaning in all EW heat exchangers.5/24/06Condition Report/Disposition Request

2897810 written to document significantcondition of fouling caused by incompatible chemicals.

This reiterated the earlier conclusion from

CRDR 2653867 in 2003.

Root cause assigned.5/20/06EDG 2B declared operable following cooler cleaning and surveillance testing. New cleaning method used brushes and significantly improved HX performance over previous method of chemical cleaning.5/22/06Inspectors point out that the promptly operability determination concluded EDGswould remain operable at least up to 160F intercooler outlet temperature, but used existing pond temperature plus the temperature increase observed in last surveillance, rather than the maximum pond temperature.

By doing so, EDG 3B

would have 162F intercooler outlet temp, and EDG 3A would be 158F.

Condition Report/Disposition Request

2897266 written.5/23/06White paper written at 8:42 p.m. to describe a method for adjusting as-foundspray pond temperatures to design basis accident conditions in the spray pond and show that EDGs were currently operable.5/23/06Inspectors pointed out that the discovery checklist was not created for this issuein

CRDR 2896661 as specified in procedures.

Checklist was created, but it haderrors that under-rated the significance.5/24/06Inspectors determine that the licensee's transportability review concl uded thatonly EDG 2B was affected, but all EDGs' intercoolers showed elevated delta-T's during previous monthly runs.

Following this, the licensee decided to re-evaluate the operability determination, assess whether both Unit 2 EDGs were inoperableat the same time on May 18, and evaluate the potential for fouling in otherEDGs.

Significant

CRDR 2897810 written on chemical incompatibilities in thespray pond system, based on 6-train impact.5/25/06Conference call between NRC Region IV and licensee about the condition andcause of the EDG 2B issue.

Condition Report/Disposition Request 2899237

written on apparent delay in making an immediate operability determination forthe elevated temperatures seen in EDG 2B on May 17.

Condition Report/Disposition Request

2898120 written to document that EDG intercooler

HX's were not considered in extent of condition review for previous EW heat exchanger fouling.

4A4-55/25/06Licensee began increased frequency testing and trending on all EDGs ascompensatory measure for fouling mechanism.6/22/06NRC questioned the impact of the accumulated sediment in the spray ponds onthe design basis accident capabilities of the pond.

Licensee calculated that itreduced the ability to accept heat rejected, and no longer met the design basisfor the pond.

A revision to the design basis calc was prepared which took credit for rejecting heat to the concrete pond liner which demonstrated operability, butno CRDR or operability determination was written.

Condition Report/DispositionRequest

2906671 was written on 6/29 when the team asked why there was no

CRDR.

IR 05000528/2006011

Text

Dear Mr. Levine:

Enclosure:

w/Attachment:

SUMMARY

NRC-Identified and Self Revealing Findings

Cornerstone: Mitigating Systems

Cornerstone: Mitigating Systems

Cornerstone: Mitigating Systems

REPORT DETAILS

INSPECTION SCOPE

Introduction.

Enforcement.

Introduction.

Enforcement.

1R07 Biennial Heat Sink Performance (71111.07B)

a. Inspection Scope

b. Findings

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened

Opened and Closed

LIST OF DOCUMENTS REVIEWED

Work Orders

Navigation menu

Search