Text

September 28, 2006

EA 06-221James M. Levine, Executive Vice President, Generation Mail Station 7602 Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034

SUBJECT: Palo Verde Nuclear Generating Station, Units 1, 2, and 3 - NRC SPECIALINSPECTION REPORT 05000528/2006011; 05000529/2006011; 05000530/2006011

Dear Mr. Levine:

On September 26, 2006, the US Nuclear Regulatory Commission (NRC) completed a specialinspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3. This inspectionexamined your staff's response to the high intake air temperature condition identified in Emergency Diesel Generator 2B on May 17, 2006, and the fouling of safety-related heatexchangers serviced by the emergency spray pond system. The NRC's initial evaluation metboth the risk criteria and several of the deterministic criteria for initiating a special inspection.

The basis for initiating this special inspection is further discussed in the Charter, which is included as Attachment 2 to this report. The enclosed special inspection report documents the inspection findings which were discussedon August 18, 2006, with Mr. C. Eubanks, Vice President, Nuclear Operations, Mr. D. Mauldin, Vice President, Engineering, and other members of your staff. A supplemental exit was conducted on September 26, 2006, with Mr. and other members of your staff to discuss the preliminary significance determination and the characterization of the performance deficiency. The determination that the inspection would be conducted was made by the NRCon June 6, 2006, and the inspection started on June 12, 2006.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel. The report documents two findings that were determined to be violations of very low safetysignificance. Also, a licensee-identified violation which was determined to be of very low safety significance is listed in this report. However, because of the very low safety significance and because they are entered into your corrective action program, the NRC is treating these findings as noncited violations consistent with Section VI.A.1 of the NRC Enforcement Policy. If Arizona Public Service Company-2-you contest any noncited violation in this report, you should provide a response within 30 daysof the date of this inspection report, with the basis for your denial, to the U. S. Nuclear Regulatory Commission, ATTN.: Document Control Desk, Washington, DC 20555-0001; andthe NRC Resident Inspector at Palo Verde Nuclear Generating Station.In addition, the attached report discusses a finding that appears to have greater than very lowsafety significance. As described in Section 5.1 of this report, the failure to recognize that improperly implemented chemistry controls for the emergency spray pond systems in all threeunits caused degraded performance in all emergency diesel generators and emergency cooling water systems over a period of years. This finding was assessed based on the best available information, including influential assumptions, using the applicable Significance Determination Process (SDP) and was preliminarily determined to be a Greater Than Green Finding. The final resolution of this finding will convey the increment in the importance to safety by assigning the corresponding color i.e., [(white) a finding with some increased importance to safety, which may require additional NRC inspection; (yellow) a finding with substantial importance to safety thatwill result in additional NRC inspection and potentially other NRC action; (red) a finding of highimportance to safety that will result in increased NRC inspection and other NRC action]. Thisfinding has a preliminary greater than very low safety significance because it has been estimated that in Unit 2, Train B of essential cooling water was incapable of performing its safety function for 6.8 months in 2003, based on extrapolation of performance data for this heat exchanger. Accident sequences that would cause high pond temperatures, primarily loss-of-

coolant accidents, could challenge this system to the point where the essential chiller would fail,causing a loss of room cooling to important mitigating systems. In addition, there is uncertaintyassociated with the amount of scaling that could occur on any of the affected heat exchangersfor all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario.Due to the complexity of this issue, there was insufficient information for a number of importantaspects to complete a final Phase 3 significance determination evaluation in a timely manner.

During the inspection process, it was apparent that your staff is working to develop the information and calculations needed to complete an accurate assessment. Therefore, the preliminary significance was based on a Phase 2 evaluation in accordance with Inspection Manual Chapter 0609. In order to complete a final significance determination, additional information is needed. The specific information needed is described in the assessment portion of Section 5.1.This finding does not represent an immediate safety concern because of the immediate andlong term corrective actions that have been implemented to reduce chemical addition, remove excess impurities from the ponds, clean all the affected heat exchangers, and adjust control limits appropriately. You are also in the process of removing sediment and sludge from the ponds.There are five apparent violations associated with this finding being considered for escalatedenforcement action in accordance with the NRC Enforcement Policy. The current EnforcementPolicy is included on the NRC's web site at http://www.nrc.gov/reading-rm/adams.html.In accordance with Inspection Manual Chapter 0609, we intend to complete our evaluationusing the best available information and issue our final determination of safety significance within 90 days of this letter.

Arizona Public Service Company-3-The significance determination process encourages an open dialog between the staff and thelicensee, however the dialogue should not impact the timeliness of the staffs final determination. Before we make a final decision on this matter, we are providing you an opportunity (1) to present to the NRC your perspectives on the facts and assumptions used bythe NRC to arrive at the finding and its significance at a Regulatory Conference or (2) submityour position on the finding to the NRC in writing. If you request a Regulatory Conference, itshould be held within 30 days of the receipt of this letter and we encourage you to submit supporting documentation at least 1 week prior to the conference in an effort to make the conference more efficient and effective. If a Regulatory Conference is held, it will be open forpublic observation. If you decide to submit only a written response, such submittal should be sent to the NRC within 30 days of the receipt of this letter.Please contact Linda J. Smith at (817) 860-8137 within 10 business days of the date of thisreceipt of this letter to notify the NRC of your intentions. If we have not heard from you within10 days, we will continue with our significance determination and enforcement decision and youwill be advised by separate correspondence of the results of our deliberations on this matter.Since the NRC has not made a final determination in this matter, no Notice of Violation is beingissued for this inspection finding at this time. In addition, please be advised that the number and characterization of apparent violations described in the enclosed inspection report may change as a result of further NRC review. In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letterand its enclosure will be available electronically for public inspection in the NRC PublicDocument Room or from the Publicly Available Records (PARS) component of NRC'sdocument system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,/RA/Dwight D. Chamberlain, DirectorDivision of Reactor SafetyDockets: 50-528; 50-529; 50-530Licenses: NPF-41; NPF-51; NPF-74

Enclosure:

Inspection Report 05000528/2006011; 05000529/2006011; 05000530/2006011

w/Attachment:

Supplemental Informationcc w/enclosure:Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Arizona Public Service Company-4-Douglas K. Porter, Senior CounselSouthern California Edison Company Law Department, Generation Resources P.O. Box 800 Rosemead, CA 91770ChairmanMaricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003Aubrey V. Godwin, DirectorArizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040Craig K. Seaman, General ManagerRegulatory Affairs and Performance Improvement Palo Verde Nuclear Generating Station Mail Station 7636 P.O. Box 52034 Phoenix, AZ 85072-2034Jeffrey T. WeikertAssistant General Counsel El Paso Electric Company Mail Location 167 123 W. MillsEl Paso, TX 79901John W. SchumannLos Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100John TaylorPublic Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224Thomas D. ChampSouthern California Edison Company 5000 Pacific Coast Hwy, Bldg. D1B San Clemente, CA 92672 Arizona Public Service Company-5-Robert HenrySalt River Project 6504 East Thomas Road Scottsdale, AZ 85251Brian AlmonPublic Utility Commission William B. Travis Building P.O. Box 13326 1701 North Congress Avenue Austin, TX 78701-3326Karen O'ReganEnvironmental Program Manager City of Phoenix Office of Environmental Programs 200 West Washington Street Phoenix, AZ 85003 Matthew BenacAssistant Vice President Nuclear & Generation Services El Paso Electric Company 340 East Palm Lane, Suite 310 Phoenix, AZ 85004 Arizona Public Service Company-6-Electronic distribution by RIV:Regional Administrator (BSM1)DRP Director (ATH)DRS Director (DDC)DRS Deputy Director (RJC1)Senior Resident Inspector (GXW2)Branch Chief, DRP/D (TWP)Senior Project Engineer, DRP/D (GEW)Team Leader, DRP/TSS (RLN1)RITS Coordinator (KEG)DRS STA (DAP)V. Dricks, PAO (VLD)J. Lamb, OEDO RIV Coordinator (JGL1)ROPreports PV Site Secretary (PRC)K. S. Fuller, RC/ACES (KSF)C. A. Carpenter, D:OE (CAC)OE:EA File (RidsOeMailCenter)SUNSI Review Completed: LJS ADAMS: # YesG No Initials: LJS # Publicly Available G Non-Publicly Available G Sensitive# Non-SensitiveR:\_REACTORS\_PV\2006\PV2006011rp-nfo.wpdEB2/TLEB2NRRNRRSRAEB2/BCCFO'Keefe/larPAGoldbergYDiaz-CastilloKParczewiskiMFRunyanLJSmith/RA//RA/T=CFOT=CFODPL for/RA/

09/26/0609/27/0609/26/0609/26/0609/28/0609/28/06PBD/BCACESOENRREB2/BCD:DRSTPruettMSHaireRFretzMTschiltzLJSmithDDChamberlainT=CFOT=CFOT=CFOE=CFO/RA//RA/

09/27/0609/27/0609/27/0609/28/0609/28/0609/28/06OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax ENCLOSUREU.S. NUCLEAR REGULATORY COMMISSION REGION IV Dockets:50-528; 50-529; 50-530Licenses:NPF-41; NPF-51; NPF-74 Report No.:05000528/2006011; 05000529/2006011; 05000530/2006011 Licensee:Arizona Public Service Company Facility:Palo Verde Nuclear Generating Station, Units 1, 2, and 3Location:5951 S. Wintersburg Road Tonopah, Arizona Dates:June 6 through September 26, 2006 Inspectors:N. O'Keefe, Team LeaderP Goldberg, Reactor Inspector, Engineering Branch 1 Y. Diaz-Castillo, Chemical Engineer, Office of Nuclear Reactor Regulation (NRR)K. Parczewiski, Senior Chemical Engineer, NRR M. Runyan, Senior Reactor AnalystApprovedBy:Dwight D. Chamberlain, Director Division of Reactor Safety ii

SUMMARY OF FINDINGS

....................................................1

REPORT DETAILS

..........................................................5 1.0SPECIAL INSPECTION SCOPE..........................................52.0SYSTEM AND EVENT DESCRIPTION.....................................52.1Event Summary.................................................5 2.2System Descriptions.............................................63.0CHEMISTRY DESCRIPTION............................................73.1Fouling Mechanisms.............................................73.2The Intended Chemistry Control Program.............................73.3How The Chemistry Control Was Actually Implemented..................84.0HEAT EXCHANGER PERFORMANCE MONITORING........................114.1Essential Cooling Water (EW) Heat Exchanger Performance Testing......114.2Emergency Diesel Generator (EDG) Performance Testing...............125.0FINDINGS..........................................................135.1Performance Deficiencies that Directly Related to Fouling to the Point ofInoperability...................................................135.2Design Control Issues...........................................21 5.3Inadequate Implementation Of Operability Assessment Process..........245.4Examples of Inadequate Response Under the Corrective Action Program...276.0ADDITIONAL INFORMATION...........................................316.1Summary of Chemistry Control Changes.............................317.0SUMMARY OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE.........341R07Biennial Heat Sink Performance (71111.07B)...............................364OA6Meetings, Including Exit...............................................37 4OA7Licensee-Identified Violation............................................37ATTACHMENT 1:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

..........................................A1-1

LIST OF DOCUMENTS REVIEWED....................................A1-2ATTACHMENT 2: SPECIAL INSPECTION CHARTER...........................A2-1ATTACHMENT 3: SUMMARY OF HEAT EXCHANGER PERFORMANCE TEST RESULTSA3-1ATTACHMENT 4: SEQUENCE OF EVENTS..................................A4-1

Enclosure1SUMMARY OF FINDINGSIR 05000528/2006011; 05000529/2006011; 05000530/2006011; 6/6-9/26/2006; Palo VerdeNuclear Generating Station, Units 1, 2, and 3: Special Inspection, 50.59, OperabilityEvaluations, Design Control, Corrective Actions, Biennial Heat Sink, Test Control. The report covered a 3-month period of inspection by two region-based inspectors and twochemistry subject matter experts from headquarters. In addition to performing a special

inspection, this report includes the results of the announced biennial heat sink inspection, which

was performed concurrently. One finding was identified with five apparent violations, which

were determined to have potential safety significance greater than very low safety significance.

Additionally, two NRC-identified noncited violations and one licensee-identified noncitedviolation with very low safety significance were identified. The significance of most findings is

indicated by its color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609,

"Significance Determination Process." Findings for which the significance determination

process does not apply may be green or be assigned a severity level after NRC managementreview. The NRC's program for overseeing the safe operation of commercial nuclear powerreactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, datedJuly 2000.Summary of Event and Inspection ResultsThe NRC conducted a special inspection to better understand the circumstances surroundingthe heat exchanger fouling at Palo Verde Nuclear Generating Station. The fouling came to theNRC's attention as a result of unusual temperatures noted during a surveillance test ofEmergency Diesel Generator 2B conducted on May 17, 2006. After 2 days, it was determined

that this potentially impacted the ability of this safety-related system to perform its safety

function during a design basis accident. It also became apparent that the fouling that was

experienced had existed for a long period of time. The cause of the fouling was not fully

understood, and affected other equipment in all trains of all three units. In accordance with

NRC Management Directive 8.3, it was determined that this event had sufficient risksignificance and met several deterministic criteria to warrant a special inspection. This inspection identified a pattern of behavior where heat exchanger performance data andinternal inspection results were not understood, entered into the corrective action program, or

adequately assessed for operability impact. When the degradation increased and theappearance of the foulant changed, this did not cause an increased response, other than toclean heat exchangers more frequently. The chemistry control credited to prevent fouling was

determined to have no mechanism to actively monitor fouling. Further, the chemistry control

limits were changed over time to actually promote fouling and increase the potential for scaling

during design basis accidents. This pattern of behavior existed for the last 10 years.A.NRC-Identified and Self Revealing FindingsCornerstone: Mitigating Systems*TBD. A finding with five apparent violations was identified associated withfouling of safety-related heat exchangers cooled by the emergency spray pond

system. Between 1995 and May, 2006, the licensee failed to recognize that

2improperly implemented chemistry controls for the emergency spray pondsystem caused a significant condition adverse to quality which degraded theperformance of all emergency diesel generators and emergency cooling water

systems. The degraded performance was primarily due to heat exchangerfouling caused by adding excessive amounts of chemicals. The conditions that

existed also had the potential to cause scaling after an accident starts. In one

instance, it is estimated that this resulted in degrading the performance of

Emergency Cooling Water Heat Exchanger 2B to the point where it would not

have been capable of performing its intended safety function for approximately

6.8 months in 2003.An apparent violation of Technical Specification 3.7.7 was identifiedbecause Train A of the Essential Cooling Water System in Unit 2 was not

capable of performing its safety function for approximately 6.8 months

ending on September 27, 2003. An apparent violation of 10 CFR Part 50, Appendix B, Criterion XI, "TestControl," was identified because the two procedures that were performed

to measure essential cooling water heat exchanger performance were

implemented in a way that was inadequate to ensure the timely

determination that the requirements and acceptance limits contained inapplicable design documents were met. An apparent violation of 10 CFR 50.59 was identified for making ninerevisions to Procedure 74DP-9CY04, "System Chemistry Specification,"

a procedure described in the Updated Final Safety Analysis Report,

between 1998 and 2004 without performing evaluations of the potentialimpact of the changes on the safety-related components in the spray

pond system; the changes revised spray pond chemistry parameter limitswhich were subsequently determined to have contributed to heat

exchanger fouling.An apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI,"Corrective Actions," was identified. On March 19, 2002, performance

testing for Essential Cooling Water Heat Exchanger 2B indicated that the

system would not be capable of performing its design function, but thissignificant condition adverse to quality was not promptly identified, the

cause determined, or corrective actions taken to restore the required heat

exchanger performance. The failure to correct this degraded

performance contributed to the continued degradation and eventual lossof function for an estimated period of 6.8 months.An apparent violation of 10 CFR Part 50, Appendix B, Criterion III,"Design Control," was identified for failure to correctly evaluate the

scaling potential of the safety-related heat exchangers cooled by the

emergency spray pond during a design basis accident. An error in the

SEQUIL calculation caused the licensee to incorrectly conclude that

scaling would not occur under the conditions established in the chemistry

control program.

Enclosure3The performance deficiency associated with these apparent violations was morethan minor because it impacted the equipment performance attribute of the

Mitigating Systems Cornerstone objective to maintain the availability andreliability of systems needed to mitigate accidents. Specifically, EssentialCooling Water (EW) Train B in Unit 2 was estimated to have been incapable of

performing its function under existing conditions for approximately 6.8 months. A

Phase 2 significance determination process concluded that this finding has

potential safety significance greater than very low safety significance because

some accident sequences, notably loss of coolant accidents, were expected to

elevate the ultimate heat sink temperature to the point where the degradedessential cooling water heat exchanger would be challenged. This was expected

to cause failure of the essential chiller, and the resulting loss of room cooling tosafety-related equipment increased the plant risk. In addition, there is

uncertainty associated with the amount of scaling that could occur on any of the

affected heat exchangers for all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident

scenario. Additional information was needed to perform a final Phase 3

assessment, due to the complexity of the issue. (Section 5.1)Cornerstone: Mitigating Systems

Green. Two examples of a noncited violation of 10 CFR Part 50, Appendix B,Criterion III, "Design Control," were identified involving the failure to adequatelytranslate the design basis of the spray ponds into procedures. Design

Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design

Basis Analysis," Revision 7, which demonstrated that the spray pond systemcould adequately limit spray pond temperature during a design basis accident did

not account for any reduced heat capacity caused by sediment buildup.

However, this fact was not translated into procedures, so approximately

400 cubic yards of sediment had built up in each of the six spray ponds when the

team questioned the impact to the heat removal function. Also, the same

calculation demonstrated that sufficient water was available to provide adequate

cooling during a design basis accident, but did not account for any leakage from

the ponds. The team determined that the licensee did not translate this into aprocedure to ensure that the condition of the spray pond was maintained such

that leakage did not occur. Procedure 81DP-0ZZ01, "Civil System, Structure,

and Component Monitoring Program," Revision 11, was used to monitor the

condition of the pond structures. The team identified that it examined only the

exposed concrete surfaces, which constituted about 7 percent of the surface

area and almost none of the water-containing volume. Cracks had beenidentified and repaired in this area, but the inspections were not expanded to the

underwater surfaces. This issue was documented in Condition

Report/Disposition Requests 2906671 and 2910912.Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be

more than minor because, if left uncorrected, the finding could become a more

significant safety concern. This finding affected the Mitigating Systems

Cornerstone. This performance deficiency screened as having very low safety

Enclosure4significance in a Phase 1 significance determination process because thelicensee was able to demonstrate that the sediment would not have resulted in a

loss of safety function, and that significant leakage did not exist. The licensee

was able to revise the calculation to take credit for heat absorption by the

concrete walls, and scheduled inspections by divers of underwater portions of

the ponds to follow sediment removal. (Section 5.2) Cornerstone: Mitigating Systems*Green. A noncited violation of 10 CFR Part 50, Appendix B, Criterion V,"Instructions, Procedures and Drawings," with multiple examples was identified

for failure to adequately assess the impact to operability of degraded heatexchangers in the emergency diesel generators and essential cooling water

system. Specifically, the licensee failed to follow Procedure 40DP-9OP26,"Operability Determination and Functional Assessment," Revision 16, inassessing indications of degraded heat exchanger performance, an activity

affecting quality. Key support organizations were not always involving operations

personnel with questions that had a potential to affect the operability of safety-related equipment, or were informing operators only after the support

organization had fully evaluated the condition, delaying actions that were

required to be prompt. Also, operations personnel did not always insist on a

rigorous evaluation. This issue was documented in Condition Report/Disposition

Requests 2918892, 2901815, and 2898237. Failure to adequately implement the operability assessment process was aperformance deficiency. This finding was more than minor because it impacted

the equipment performance attribute of the Mitigating Systems Cornerstone

objective to maintain the availability and reliability of systems needed to mitigateaccidents. This finding screened as having very low safety significance in a

Phase 1 significance determination process, because the examples used for this

violation were confirmed not to involve any loss of safety function. This finding

had cross-cutting aspects in the area of human performance because the

licensee did not follow their systematic process for operability decision making when information was not brought to the right decision makers. (Section 5.3)B.Licensee-Identified FindingsA violation of very low safety significance, which was identified by the licensee, has beenreviewed by the inspectors. Corrective actions taken or planned by the licensee have

been entered into the licensee's corrective action program. This violation and corrective

actions are listed in Section 4OA7 of this report.

Enclosure5REPORT DETAILS1.0SPECIAL INSPECTION SCOPEThe NRC conducted this special inspection to better understand the circumstancessurrounding the heat exchanger fouling at Palo Verde Nuclear Generating Station. Thefouling came to the NRC's attention as a result of unusual temperatures noted during asurveillance test of Emergency Diesel Generator (EDG) 2B on May 17, 2006. Over the

following several days, it was determined that this potentially impacted the ability of thissafety-related system to perform its safety function during a design basis accident. Italso became apparent that the fouling that was experienced had existed for a longperiod of time. The cause of the fouling was not fully understood, and affected other

equipment in all trains of all three units. In accordance with NRC ManagementDirective 8.3, it was determined that this event had sufficient risk significance and metseveral deterministic criteria to warrant a special inspection. The team used NRC Inspection Procedure 93812, "Special Inspection Procedure," toconduct the inspection. The special inspection team reviewed procedures, corrective

action documents, and design and maintenance records for the equipment of concern.

The team interviewed key station personnel regarding the event, the root cause

analysis, and corrective actions. A list of specific documents reviewed is provided in

1. The charter for the special inspection effort is provided as Attachment 2. 2.0SYSTEM AND EVENT DESCRIPTION2.1Event SummaryOn May 17, 2006, Palo Verde Nuclear Generating Station noted elevated temperaturesin the intake air for EDG 2B while the engine was running for a surveillance test. Theengine intake air is compressed and heated by the turbocharger, and is then cooled by

two parallel intercoolers before entering the engine. The elevated temperatures were

noted at the outlet of the intercoolers, which was indicative of inadequate cooling in the

intercoolers.The elevated temperatures were discussed among operations and engineeringpersonnel, and it was concluded that this condition did not affect operability. EmergencyDiesel Generator 2B was returned to service, and the following day the EDG Train A

and essential cooling water (EW) heat exchanger were removed from service for

approximately 23 hours2.662037e-4 days 0.00639 hours 3.80291e-5 weeks 8.7515e-6 months . On May 19, following the return to service of Train A, the

operability of EDG 2B was revisited. An immediate operability determination wascompleted at 9:55 a.m. which concluded EDG 2B was operable. After additional review,

at 5:54 p.m., it was concluded that EDG 2B was not operable, and TechnicalSpecification 3.8.1 was entered. Because the condition was potentially a common modefailure concern for the EDGs, all EDGs were run to assess the performance of the

intercoolers in accordance with Technical Specification 3.8.1.2.The licensee inspected the EDG 2B intercoolers and found they were fouled on thecooling water (spray pond system) side. The emergency spray pond system (SP)

Enclosure6provides the cooling water to both the EDG coolers (two intercoolers, a jacket watercooler, and a lube oil cooler), as well as cooling the large EW heat exchanger. The

intercoolers' water side had a white slimy substance which was apparently reducing the

ability to transfer heat.That same week, the licensee had taken each of the Unit 2 EW heat exchangers out ofservice for tube cleaning. This was in response to earlier heat transfer degradation

which was judged to be degrading fast enough that the Unit 2 EW heat exchangers

would not remain operable through the full operating cycle and needed to be cleanedbefore summer. The degradation was not fully understood, but was suspected to be

related to a similar white slimy substance.As NRC inspectors got involved and asked key questions related to the operability ofthese components, it became apparent that the scope of the problem was more thanjust EDG 2B.2.2System DescriptionsThe spray pond system for each unit is comprised of two trains. Each train has aconcrete lined spray pond of 6.2 million gallons of water, which serves as the ultimateheat sink. The ponds are 345 feet by 172 feet, and are 15.5 feet deep. The spray

pump for each train takes a suction on one corner of the pond, at a level 6 feet below

the bottom of the main portion of the pond. The pump delivers 16,300 gpm through

buried piping to the EW heat exchanger and the EDG coolers. The water then returns

to the pond through over 300 spray nozzles spaced around the pond above the surface.

The spray nozzles atomize the water, allowing for evaporative cooling. The spray pond

system is normally in a standby condition with no flow. The spray pond piping wasmade of carbon steel with a protective coating on the inside surfaces.Each EDG has five coolers which have heat removed by the spray pond system: thejacket water cooler; lube oil cooler: fuel oil cooler, and a pair of turbocharger

intercoolers. While the EDG is normally in standby, the jacket water system and lube oilsystem continue to circulate and heat their process fluids. The fuel oil cooler has a verysmall flow, and is not required for the EDG to function. The intercoolers function at the

highest temperature, approximately 290F air inlet temperature.The EW heat exchangers transfer heat from the systems it cools to the spray pondsystem. The loads supplied are the essential chillers, shutdown cooling heat exchanger,and spent fuel pool cooling system. The essential chillers provide cooling to the controlroom ventilation, safety-related switchgear rooms, and emergency core cooling pump

rooms. The shutdown cooling heat load during an accident constitutes over 94 percent

of the total EW heat load.The spray ponds were provided with a filtering system. Pond water was pumpedthrough a pair of sand filters to remove larger impurities at 600 gpm each (1200 gpm

total). The sand filters were backwashed to remove the impurities about once per day

under normal conditions.

Enclosure73.0CHEMISTRY DESCRIPTION3.1Fouling MechanismsThe degraded heat transfer capabilities in the EDG coolers and the EW heatexchangers occurred in all trains in all units. The degree of degradation varied

somewhat among the units. The EDG intercoolers and the EW heat exchangers were

observed to have fouled the most, while the jacket water and lube oil coolers fouling was

minor and had no impact to operation.The team determined that the licensee did not take photographs or have samples of thesubstances found in these heat exchangers analyzed. When the May 2006 event

occurred, samples were analyzed, but the exact substances could not be determinedbecause it was an amorphous mix of a number of substances. The major constituents

were determined to be zinc (38 percent), phosphorous (29 percent), and calcium

(32 percent). Two different substances have been found in the heat exchangers, though this was notrecognized. In earlier years, a thin coating that turned white as it dried was observed in

heat exchanger inspections. This is now believed to be zinc hydroxide. In more recent

years, a thicker coating that has been described as white, slimy, and lotion-like was

observed. It was believed for years that the coating was zinc hydroxide which the

chemistry control program was intentionally trying to deposit on the system piping, so itwas largely accepted. The team noted that the licensee did not reassess what thematerial was when the appearance began to change.The licensee's root cause assessment concluded that improper chemical controlresulted initially in excessive zinc hydroxide precipitating out of solution, which is a heat

transfer foulant. Later, zinc phosphate precipitated as a thicker mix. This precipitant

then provided a localized seed site for other chemicals to also precipitate out of solution.

The combined effect was an amorphous lotion-like substance that coated heat

exchanger tubes and degraded heat transfer by interrupting cooling water flow at the

tube wall. Specifically, the fluffy film caused a low flow zone near the tube wall and

forced the turbulent flow away from the tube wall; since efficient convective heat transfer

required strong turbulent flow at the tube wall, heat transfer was significantly impacted

even though the film was fluffy and not tightly adherent. It should be noted that the above fouling mechanisms were different than scaling. Scaling involved hard mineral deposits, such as calcium carbonate and calcium

phosphate which occur at high temperatures. Scaling concerns are addressed in

Section 5.1.5.3.2The Intended Chemistry Control ProgramThe licensee's program was intended to meet two goals: minimize corrosion in the spraypond piping and components, and minimize fouling. The team determined that the PaloVerde Nuclear Generating Station program was similar to that used in cooling water

systems in the electric power industry, both nuclear and non-nuclear. Though complex,this type of program is typically effective. When initially implemented, the spray pond

Enclosure8chemistry control program used the chemical supply vendor's recommended limits;however, over time these limits were changed numerous times, eventually departing

from these recommendations. The Palo Verde Nuclear Generating Station program treated the two parts of the spraypond system separately. The spray ponds were treated only to prevent gross biologicalfouling. The spray pond system piping and components were treated with acomplicated combination of chemicals to meet the goals stated above. Chemicals were

added at the pump suction during pump runs for piping treatment, and pond chemicals

were added at the same location but were not effectively dispersed through the pond.The carbon steel piping was protected from corrosion by adding zinc and phosphoricacid. This combination was intended to form a very thin layer (a few molecules thick) of

zinc, with phosphate in solution to protect any active corrosion sites. The root cause

assessment team concluded that this film should not be thick enough to be visible on

the piping. The zinc coating must be replaced periodically, so extra zinc is added to stay

in solution to be available as replacement if needed. However, zinc has a low solubilityand will precipitate out of solution unless the conditions are carefully managed.The makeup water for the system has a relatively high alkalinity and some potentiallyscale-forming impurities (such as calcium). Scale is more likely to form at higher

temperatures (such as those expected in heat exchangers), and under higher pH

values. The alkalinity from makeup water and evaporating of water caused pH to

naturally increase. Therefore, the licensee added sulphuric acid to lower pH and

improve the solubility conditions. Dispersant was added to keep the impurities and the chemicals that were intentionallyadded in solution and avoid precipitation and fouling. The dispersant was very important

to control solubility and avoid fouling and scaling. However, the team determined thatthe dispersant in use at the time of the event could not be measured directly, and themethod used to estimate dispersant concentration was no longer reflecting the actual

conditions.Finally, the licensee added biocide and hypochlorite to control biological growth. Thepiping was treated during pump runs; pond treatments were started during pump runs,

but continued for hours after the pump was secured. All chemicals were added to the

ponds near the pumps.The team identified that the chemistry program objectives to minimize fouling was notbeing monitored. The effectiveness of fouling prevention was not being tracked by

chemistry personnel.3.3How The Chemistry Control Was Actually ImplementedA summary of the chemistry control program and changes made over time is provided inSection 6.1.The team determined that the chemistry control program was being implemented in away that caused fouling of safety-related heat exchangers. Chemical additives (zinc and

Enclosure9phosphate) were being added in excess of what was needed and above the solubilitylimits for the existing conditions. The pH was also being maintained too high, reducing

the solubility limits. Also, dispersant was not being accurately being monitored, and as aresult, was not being added in sufficient quantities to effectively control the solubility ofthe existing chemicals. These conditions combined to cause the fouling.The team reached the following conclusions about the actual implementation of thespray pond chemistry control program:The Chemistry personnel implementing the program did not fullyunderstand how the spray pond chemistry control program was supposed

to work. As a result, they made a series of inappropriate changes to

chemistry limits which negatively impacted solubility without assessing orunderstanding the potential impacts.The specified pH band was too high to maintain zinc hydroxide insolution, so it precipitated out of solution and formed a foulant.The zinc control band was too high, resulting in injecting too much andprecipitating significant quantities out of solution. The precipitate

collected in the EW heat exchangers and intercoolers, and may have also

precipitated into the sediment at the bottom of the ponds. The ideal

control concentration was determined to be about 0.5 ppm by the root

cause team, while the actual control band was 1.0 - 3.0 ppm at the timeof the event. Excess phosphate was being added, contributing to the precipitationproblems. Phosphoric acid was added in a 4:1 ratio with the zinc; zinc

was constantly removed from solution, while phosphate built up in the

pond and piping. The ideal phosphate level was determined by the root

cause team to be about 4.0 ppm, while at times, samples recorded over

ppm phosphates in Unit 2. The high phosphate levels exceeded the

maximum solubility for the bulk water conditions, and precipitated out ofsolution in the EW heat exchangers and intercoolers, and probably also

precipitated into the sediment at the bottom of the ponds.Calcium and other impurities from makeup water were allowed toconcentrate excessively over time, and under the relatively high pH

conditions they contributed to precipitation problems. The root cause

team concluded that the zinc phosphate precipitant provided a seed sitefor other compounds containing zinc, calcium, and phosphate to attach. Dispersant was intended to improve the solubility conditions, but there was too littledispersant present to be effective. The dispersant in use could not be measured

directly, so a "tag" chemical (molybdate) which could be measured was added in a

known proportion to the dispersant. The licensee did not recognize that this tag

chemical was not subject to the same loss and use mechanisms as the dispersant. As

a result, the tag chemical analyses registered incorrectly high results. The team

determined that Chemistry personnel reduced or stopped adding dispersant for long

Enclosure10periods of time because the molybdate concentrations indicated dispersant levels werehigh. Contributing to this, the tag chemical was only checked once per month. Also,when the licensee allowed the high levels of impurities to build up, they did not

recognize that this should have required raising the amount of dispersant to maintain the

same effectiveness.The team identified that the chemistry control practices for this system were not typicalof other plant systems with the same importance. Chemicals were added based onthumbrules and past additions, rather than basing them on recent sample results. Also,

many key samples were taken only once per month, but the associated chemical

additions were made several times per week. Some important samples had no controls

associated with undesirable levels. For example calcium, an impurity, which is a scale-

forming concern, had no action level to trigger action to reduce the concentration. The

spray pond system was a concentrator for chemicals, so as calcium levels accumulatedover the years, Chemistry raised the limits to stay ahead of the actual levels, rather than

reducing the existing concentrations. When sample results indicated that molybdate

levels were high and it was presumed that this meant dispersant levels were high,

dispersant additions were stopped. The molybdate samples were taken monthly, but

the analyses were sometimes delayed by up to 2 to 3 weeks (Condition Report/

Disposition Request (CRDR) 2800653). This created an untimely feedback mechanismfor adjusting the chemical addition.The team identified that chemistry personal had focused on treating the pipes and didnot effectively manage the chemistry condition in bulk pond water. This approach was

viewed as economical, and is a common industrial practice. However, this did not

account for the unique nature of the spray ponds at Palo Verde Nuclear GeneratingStation. Unlike typical systems, the spray pond system was not designed or operated ina way that regular water turnover occurred, such that chemicals and impurities would be

removed. It was not recognized that the chemistry control program in use typically

involved a system that ran continuously and had a much higher water turnover. Theturnover should result in removal of impurities and cause the tag chemicals to more

closely reflect dispersant concentrations. The continual buildup of chemicals and

impurities at Palo Verde Nuclear Generating Station caused a departure from the

conditions assumed when the vendor recommendation for dispersant levels was made. Most sampling was done during pump runs by taking samples from the pipes, with verylittle sampling of the bulk water in the ponds. As a result, the ponds were allowed to

accumulate chemicals and impurities. This was further complicated by allowing

sediment to accumulate at the bottom of the ponds, where precipitants could be

concealed but still affect water chemistry. Biological fouling in the ponds was not being effectively controlled. The team concludedthat the biocide treatments and monitoring of the ponds were ineffective. During theonsite portion of the inspection, contract divers began vacuuming sediment from the

Unit 1 ponds. Condition Report/Disposition Request 2905162 was written on June 21,

2006, to document that the diving operations were dislodging "sheets" of algae from thespray pond walls. The licensee was unaware that algae existed in the ponds prior to

this time. The team reviewed the pond treatment practices and concluded that the

biocide addition point was in one corner of the ponds, insufficient mixing was provided to

Enclosure11distribute the chemicals. This was, in part, because much of the biocide chemicals wereadded after securing the spray pond pumps. The licensee's evaluation concluded that

the existing algae would not impact operability, and the team concluded that this was areasonable conclusion.Finally, the water chemistry control program being used was challenged by thelicensee's practice of leaving the systems idle, without constant flow, for most of thetime. Typical cooling water systems run pumps continuously, which have the benefitsfor flushing pipes and heat exchangers out and mixing the chemicals. The Palo Verde

Nuclear Generating Station spray pond systems were normally run only for chemistrycontrol (adding chemicals 3 times per week), infrequent testing, and during plant

outages. This resulted in the system having no flow most of the time.The team noted that the chemistry control program documents stated that dispersantadditions were intended to be based on makeup water addition. However, that had

never been the actual practice. The team identified that the licensee did not trackmakeup water additions to the spray ponds, and had no way of measuring the quantity

of water flowing to any one pond. In an idle pond, evaporative losses ranged from 2,200

to 10,000 gallons per day. When the system was run, evaporation was much higher,and "drift" losses occurred when water droplets were carried out of the pond berm by

wind. Despite these large and highly variable losses which required makeup and

caused loss of dispersant, the team identified that the licensee had stopped addingdispersant for long periods, in part because they were not adding dispersant based on

makeup water addition.The team identified that the chemical supply vendor site representative visited the siteseveral times per month, but this source of expertise was not utilized. The vendor wasnot asked to assess the performance of the program, and was not asked to observe

heat exchanger foulant or analyze the material. The site representative was involved in

trying to identify the lotion-like substance found in the spray pond sand filters, though he

was not provided with a sample as a basis for the discussion.The team concluded that the licensee was no longer maintaining the spray pondchemistry control program within the bounds that were originally intended. They had

made changes without understanding the effect the changes would have on the

conditions and without understanding the complex interactions that existed. As a result,

the fouling was self-induced by the chemistry control program.4.0HEAT EXCHANGER PERFORMANCE MONITORING4.1Essential Cooling Water (EW) Heat Exchanger Performance TestingThe licensee committed to conducting heat exchanger performance testing on the EWheat exchangers. This was typically done for one train at the beginning of each

refueling outage during the primary plant cooldown, when decay heat load allowed

enough heat load to be able to get accurate results. Later, the licensee began to also

conduct EW heat exchanger performance testing during the primary plant heatup at the

end of refueling outages. These tests had a much lower heat loading, so the accuracy

2of the results was potentially lower, but the results would still be expected to provide anindication of the relative effectiveness of any tube cleaning during that outage. The team performed a detailed review of the licensee's performance testing methodsand results, including internal inspections and cleaning practices. This included

reviewing documented results since 2001. This is further discussed in Section 1R07.The results of the performance testing conducted on the EW heat exchangers issummarized in Attachment 3. It can be seen that the data for any one heat exchangeris very limited, since it might be tested only once every three years. It can also be seen

that significant reductions in heat exchanger performance were indicated by test results,

yet very few instances were found where this degradation was documented in the

corrective action program. This is further discussed in Section 5.4.4.2Emergency Diesel Generator (EDG) Performance TestingThe licensee did not conduct performance testing of any of the EDG heat exchangers. The primary reasons are briefly summarized:Lube oil coolers and jacket water coolers: The shell side fluids (lube oil andjacket water) have automatically controlled flows that adjust to the conditions to

maintain a constant temperature. The thermostatic control valves cannot bemanually overridden or their positions measured. Also, the systems do not haveinstalled flow measurement devices, and the pipe configuration prevents an

accurate flow measurement using an external device. Intercoolers: These heat exchangers have a relatively small heat load, and useair as a working fluid. Both these facts make it difficult to accurately measure

heat exchanger performance.Instead of performance testing, Palo Verde Nuclear Generating Station's commitment intheir Generic Letter 89-13 response was to regularly clean and inspect these heat

exchangers to ensure there was not excessive fouling. Since these inspections

identified that the intercoolers in each unit experienced fouling, the licensee wastrending the relative performance of the intercoolers during EDG surveillance testing. This was done by assessing the difference between air outlet temperature and water

inlet temperature over time. Cleanings were scheduled when this difference became

high, but the condition was documented as a degraded condition in the corrective action

program only once, although the degraded condition occurred on numerous occasions.

This is further discussed in Section 5.3.

Enclosure135.0FINDINGS5.1Performance Deficiencies that Directly Related to Fouling to the Point of Inoperability a.Description.In 1994, Palo Verde Nuclear Generating Station implemented a new chemistry controlprogram for the emergency spray pond system to control corrosion and prevent foulingof the safety-related components. This system provides the ultimate heat sink function. Throughout the period of 1994 through May 2006, the licensee made a series of

changes to this program which created a chemical environment that was progressively

more conducive to fouling of the heat exchangers which were relied upon to transfer

heat from the reactor, containment, EDGs, and safety-related equipment rooms to the

ultimate heat sink. The foulant was determined to be a buildup of excess chemicals

which were added as part of the chemistry control program, but which were

subsequently determined to have exceeded the limitations of solubility. Despite years oftest results which showed degraded heat exchanger performance, numerous heat

exchanger inspections which documented chemical buildup, and an increasing need to

clean the heat exchangers more frequently, the licensee failed to recognize the safety

significance of the problem or determine and correct the cause. Degraded performance

was observed in all trains in all three units. Because of significant inherent design

margins, only one example was identified where fouling may have been sufficient to

cause a loss of safety function. Specifically, the Unit 2 Train B EW heat exchanger was

potentially inoperable between about April and October 2003.Also, during the inspection it was recognized that calculations which demonstrated thatheat exchangers cooled by the spray pond system would not be subject to scalingduring a design basis accident contained an error. When re-performed, the calculations

indicated that scaling was possible. This would have the effect of further reducing the

heat transfer capability of these heat exchangers during certain accident scenarios. This had the potential to affect components cooled by the spray pond in all units.The team reviewed the degraded performance and the remaining capabilities of thesecomponents over the period 1994 through 2006. The licensee performed computer

modeling and analysis that adequately demonstrated that the EDG systems werecapable of performing under the degraded conditions. This was due to considerable

margins inherent in the design, and because it could be shown that the fouling inEDG 2B that triggered this inspection was clearly the worst case that had been

experienced. For the EW heat exchangers, the team identified the following fourperiods where the EW heat exchangers had less than the design basis heat removal

capability, a degraded condition which constituted a significant condition adverse toquality. Of these, it appears that only 2B was incapable of performing its function under

the existing, less challenging conditions in 2003.EW HX 1B was possibly as low as -20.8 percent margin entering 1R11 in 2004.

EW HX 2B was possibly as low as -49.78 percent margin entering 2R11 in 2003.

Enclosure14EW HX 3B was possibly as low as -14.4% percent margin entering a midcycleoutage in 2004. . EW HX 3A was measured at -22 percent margin entering 3R10 in 2003. These estimates do not include the possible effect due to scaling discussed in

subsection (5) below. The impact due to fouling will be reassessed in all units duringthe final significance determination.Note that throughout this report, heat exchanger capacity is discussed in terms of"percent margin." This term is meant to refer to margin above or below the design

capacity of a heat exchanger. In this context, "0 percent margin" means the heat

exchanger would have exactly the capacity required to transfer the design heat load

under limiting design conditions. Positive or negative margin expresses how much more

or less than that capability actually existed. This was used to be consistent withlicensee documentation. b.Statement of the Performance Deficiency.Between 1995 and May, 2006, the licensee failed to recognize that improperlyimplemented chemistry controls for the emergency spray pond system caused asignificant condition adverse to quality which degraded the performance of all

emergency diesel generators and emergency cooling water systems. The degradedperformance was because of heat exchanger fouling from chemical additives, and

potentially because of scaling after an accident starts. This resulted in degrading the

performance of EW Heat Exchanger 2B to the point where it would not have been

capable of performing its intended safety function for approximately 6.8 months in 2003.

In addition, there is uncertainty associated with the amount of scaling that could occuron any of the affected heat exchangers for all three units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident

scenario. c.Enforcement. (1)Technical Specification 3.7.7 requires that two trains of essential cooling water beoperable in Mode 1, with an allowed outage time for one train of 72 hours8.333333e-4 days 0.02 hours 1.190476e-4 weeks 2.7396e-5 months . Contrary to

this, Train A of the EW system in Unit 2 was not capable of performing its safetyfunction for approximately 6.8 months ending on September 27, 2003, when the plantshut down for Refueling Outage 11. This will be treated as an apparent violationpending a final significance determination: AV 05000529/2006011-01, EW Train 2B

Inoperable Longer than Allowed Outage Time. This issue was entered into the

licensee's corrective action program under CRDR 2905161.The period during which the train was incapable of performing its intended safetyfunction was determined through straight-line extrapolation of the two prior heat

exchanger performance tests for this train. The minimum acceptable performance was

determined by licensee calculation dated June 22, 2006, which revised Design Basis

Calculation 13-MC-SP-307 to remove all known margins. The minimum acceptable

performance was 204 Btu/hr-ft2-F, or about -35 percent margin compared to the full

design basis value. This calculation included the most limiting values for spray pond

Enclosure15temperature and air temperature actually recorded during the exposure period, andaccounted for the existing number of tubes with plugs installed. The team reviewed this

calculation and determined that appropriate values and assumptions were used for the

purpose of determining heat exchanger performance capability.The licensee showed documentation that indicated that eddy current testing of this heatexchanger was performed during 2R10 in March, 2002. The licensee stated that the

probes were known to push out the majority of the foulant and improve heat exchanger

performance. The team acknowledged this fact, but could not quantify any potentialimprovement because no data existed to quantify this for Unit 2 EW heat exchangers.

The Unit 2 EW heat exchangers were the only ones with sleeves at each end; the

sleeves necessitated using a smaller probe than was used in the other units, and thus

had a bigger gap between the probe and the majority of the tube wall length. This factprecluded using test data which was available for the other units, particularly because

the heat transfer characteristics of the foulant were not known. (2)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XI, "TestControl," requires that a test program shall be established to assure that all testing

required to demonstrate that structures, systems, and components will performsatisfactorily in service is identified and performed in accordance with written test

procedures which incorporate the requirements and acceptance limits contained in

applicable design documents. Test results are required to be documented and

evaluated to assure that test requirements have been satisfied. Test

Procedure 70TI-9EW01, "Thermal Performance Testing of Essential Cooling Water

Heat Exchangers," contained instructions for conducting performance testing.

Procedure 73DP-9ZZ10, "Guidelines for Heat Exchanger Thermal Performance

Analysis," contained instructions for calculating the results of EW heat exchanger

performance testing and verifying them against the design basis requirements in

Calculation 13-MC-SP-0307, "SP/EW System Performance Design Bases Analysis,"

Revision 007. Contrary to this, the results of performance testing for EW Heat

Exchanger 2B conducted on March 19, 2002, did not meet the design basisrequirements, but this was not correctly evaluated to determine whether the systemwould be capable of performing its design function until August 22, 2002. As a result,

Unit 2 was allowed to restart and run an entire operating cycle without correcting thedegraded performance. This will be treated as an apparent violation pending finalsignificance determination: AV 05000528; 05000529;05000530/2006011-02,Inadequate Test Control to Promptly Identify Unacceptable Performance Test Results.

This issue has been entered into the licensee's corrective action program under CRDR2928230.The team determined that the performance test was scheduled and conducted under amaintenance order. However, the calculation of the results and comparison to

acceptance criteria was not scheduled. Further, the test results were not

administratively tracked to ensure that the unit was not restarted before acceptableperformance was documented. As a result, the March 19, 2002, test results were

determined after Unit 2 started up from Refueling Outage 2R10. The test results were

unexpectedly low, but this was determined to involve a calculation error. When the error

was corrected with results, which indicated low performance, the test method was

evaluated. On August 22, 2002, the test package was completed and verified to

Enclosure16demonstrate -9.5 percent capability. This untimely determination of unacceptableperformance did not trigger a review of operability, nor was continued degradationconsidered. As a result, this component continued to degrade for 18 months after

demonstrating unacceptable performance. Had this determination been made prior to

the end of the outage, corrective actions could have been taken to clean the tubes.

Technical Specifications require that both trains be operable prior to making plant mode

changes, so the unit should not have be restarted with this condition, had it been

identified. (3)Part 50.59 of Title 10 of the Code of Federal Regulations allows licensees to makechanges to procedures as described in the Updated Final Safety Analysis Report

without obtaining a license amendment only if the change satisfies the criteria in

50.59.c. It goes on to specify that changes made to procedures must include written

evaluations which provide the bases for the determination that the change does notrequire a license amendment. The Palo Verde Nuclear Generating Station Updated Final Safety Analysis Report,Section 9.2.1.2.F, specified that the spray pond system would be protected from organicfouling and inorganic buildup by proper water treatment. Procedure 74DP-9CY04,

"System Chemistry Specification," implemented the licensee's program, including

chemical limits and addition frequencies, in order to control corrosion and fouling in the

emergency spray pond system.Contrary to the above, between 1998 and 2004, the licensee made multiple changes tothe chemistry control program specified in Procedure 74DP-9CY04, "System Chemistry

Specification," without adequately assessing the impact to the safety-related systemsaffected by the changes. Revisions 3, 6, 8, 10, 12, 24, 28, 32, and 36 toProcedure 74DP-9CY04 made changes between March 1998 and February 2006, which

affected spray pond chemistry parameter limits which were subsequently determined to

contribute to heat exchanger fouling. Specifically:*Revisions 3, 6, 8, 12, 24, 28, and 32 were made without any documented 50.59review. The team determined that these revisions contained changes to

chemistry limits in the spray pond which had the potential to negatively impact

fouling, and therefore were required to be evaluated per 50.59. These included

changing pH, phosphate, and zinc limits, and instructions on the frequency of

dispersant additions. *The screening performed for Revision 10 was determined to be inadequatebecause the revision increased the maximum allowable pH in Unit 1, but the

screening dated 6/30/99 did not evaluate this aspect of the revision. This limithad been lowered in Revision 6 to account for the decision to use well water as

the makeup source for the Unit 2 spray pond. This water source had a

significantly higher alkalinity, and the lower pH was determined to be necessary

to avoid scaling at the time the modification was implemented. However, this

aspect of the change was not evaluated in this screening.

Enclosure17*The screening performed for Revision 36 was determined to be inadequatebecause the revision increased the maximum allowable zinc concentration by a

factor of two, but Screening Number S-06-0049, Revision 1, did not address this

aspect of the revision. Excessive zinc additions were subsequently determined

to be a primary cause of fouling. This will be treated as an apparent violation pending final significance determination: AV 05000528; 05000529;05000530/2006011-03, 50.59 Reviews Not Performed or

Inadequate for Multiple Changes to Spray Pond Chemistry Control Procedure.This issue was documented in CRDR 2902498. The team concluded that there weretwo prior opportunities to have identified that chemistry limit changes were being made

without 50.59 reviews which did not identify this set of examples (CRDRs 2683642 and2864575); these are further discussed in Section 4OA7. (4)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion XVI,"Corrective Actions," requires that significant conditions adverse to quality shall be

promptly identified, and that the cause shall be determined and corrective action shall betaken to preclude repetition. Contrary to this, on March 19, 2002, performance testing

for EW Heat Exchanger 2B indicated that the system would not be capable ofperforming its design function, but this significant condition adverse to quality was not

promptly identified, the cause determined, or corrective actions taken to restore the

required heat exchanger performance. Specifically, the unacceptable performance was

not promptly identified, because the test results were not correctly calculated untilAugust 22, 2002, which was after operating mode changes and returning the unit to

power following the outage. When the test results were finalized, the fact that thedesign basis capability was not met was not recognized or entered into the correctiveaction program. These failures to correct this degraded performance contributed to the

continued degradation and eventual loss of function for a period of approximately

6.8 months. This will be treated as an apparent violation pending final significancedetermination: AV 05000528; 05000529;05000530/2006011-04, InadequateIdentification and Corrective Action for Degraded EW Heat Exchanger Performance.

This issue was entered into the corrective action program under CRDR 2897810. (5)Part 50 of Title 10 of the Code of Federal Regulations, Appendix B, Criterion III, requiresthat design control measures be established to verify the adequacy of design of

structures, systems and components. A SEQUIL calculation was intended todemonstrate that the heat exchangers cooled by the emergency spray pond systemwould be capable of transferring the required heat load to the spray pond because

scaling would not occur during a design basis accident. Contrary to this, the calculation

used an improper setting which caused the calculated result to incorrectly show that

scaling would not occur. This issue was documented in CRDR 2913430. This violationhas the potential to increase the exposure time when EW Heat Exchanger 2B was

incapable of performing its safety function, since the 6.8 month exposure time did not

Enclosure18account for any impact because of scaling. There is uncertainty associated with theamount of scaling that could occur on any of the affected heat exchangers for all three

units during 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of an accident scenario. This will be treated as an apparentviolation pending final significance determination: AV 05000528; 05000529;05000530/2006011-05, Inadequate Design Control to Ensure No EW Heat Exchanger

Scaling. This finding does not represent an immediate safety concern because immediate andlong term corrective actions that have been implemented to reduce chemical addition,

remove excess impurities from the ponds, clean all the affected heat exchangers, and

adjust control limits appropriately. The licensee was also in the process of removing

sediment and sludge from the ponds. d.Assessment.The performance deficiency associated with these violations was more than minorbecause it impacted the equipment performance attribute of the Mitigating Systemscornerstone objective to maintain the availability and reliability of systems needed tomitigate accidents. Specifically, EW Train B in Unit 2 was estimated to have been

incapable of performing its function under existing conditions for approximately

6.8 months. This was determined through straight-line extrapolation of heat exchanger

performance test data. The minimum required performance for the EW heat exchanger

was determined by the licensee by modifying the design basis calculation to credit the

use of all known margin and evaluated the condition for the worst case pond

temperature known to exist during the period of concern. This performance deficiency affected the Mitigating Systems functions of short-termprimary heat removal (high and low pressure safety injection) and long-term heat

removal (emergency core cooling system recirculation). Because this performance

deficiency involved the loss of function for a single train for longer than its Technical

Specification allowed outage time, Phase 1 of the significance determination

process required a Phase 2 evaluation.To challenge the safety function of the EW heat exchanger, it was determined that theremust be both a high heat load and a high pond temperature. The sequences of concern

were determined to be small, medium and large break loss-of-coolant accidents,

loss-of-offsite power, plus, stuck open primary relief valve sequences. Loss of dc Bus B

was also evaluated as a consequential failure.The performance deficiency was assumed to cause a loss of function of the EW heatexchanger. As a consequence, this has the following assumed impacts:Loss of cooling to shutdown heat exchanger - reduced containment heat removalfunction credit from multi-train to single train.Loss of cooling to the essential chiller, which fails and causes loss of roomcooling and eventual failure of the following equipment:

Enclosure19Motor driven auxiliary feedwater pump - reduced credit frommulti-train to single trainContainment spray pump - reduced credit from multi-train tosingle trainHigh pressure safety injection pump - reduced credit frommulti-train to single trainLow pressure safety injection pump - reduced credit frommulti-train to Single TrainEW pump - reduced credit from multi-train to single train

Control room complex - no Phase 2 impact because of redundanttrain and non-safety cooling supplyDirect current equipment room - required evaluation of the loss ofdc Bus B special initiatorRecovery credit was considered for the EW heat exchanger, because thecondition was a degradation in capability, not a complete loss of function. Therequirements in Manual Chapter 0609 for recovery credit were not explicitly met,

but it was judged that normal operator training and procedures would be

sufficient to maintain containment/core heat removal.Recovery credit was considered for mitigating the effects of loss of room cooling. The licensee had procedure steps to open doors for the affected rooms, and

meet the requirements for recovery credit. It was uncertain whether this action

would be effective, so this is an area where additional information is needed to

make a final significance determination.Manual Chapter 0609 directs that findings of this type be evaluated for largeearly release frequency (LERF) contribution. For large, dry type containment

buildings (e.g., Palo Verde Nuclear Generating Station) Appendix H and

associated basis document indicates that Type A findings at pressurized-water

reactors that affect containment heat removal do not contribute to LERF,because they involve late containment failure probability only. Therefore, the riskof this performance deficiency is best represented by CDF.The results of the Phase 2 SDP determined that this condition represented a riskthat was potentially greater than very low risk significance (potentially greaterthan Green). The Phase 2 results and assumptions were reviewed by Senior

Risk Analysts from Region IV and Headquarters.

20A preliminary Phase 3 SDP analysis was performed. However, it was determined thatseveral important aspects and assumptions contained considerable uncertainty or could

not yet be verified. While the licensee was working on a number of evaluations and

calculations to resolve the uncertainty, these were not complete. The preliminary

Phase 3 analysis and modified Phase 2 using the same assumptions as the Phase 3

analysis indicated that the risk could be lower than the Phase 2 analysis, but the resultswere highly dependent on the uncertainty of some of the information.In accordance with Manual Chapter 0609, NRC senior management determined that thepreliminary significance would be based on the Phase 2 SDP results, pending a final

significance determination which would account for additional information to be provided

by the licensee. This was intended to improve the timeliness of regulatory decision-

making in situations where the complexity of the significance determination would

otherwise cause an excessive delay.Information on the following aspects was still needed from the licensee in order tocomplete a final significance determination:The effect on heat exchanger heat transfer capability that scaling may haveplayed in the first 24 hours2.777778e-4 days 0.00667 hours 3.968254e-5 weeks 9.132e-6 months of a design basis loss-of-coolant accident for the

worst chemistry conditions that existed in each of the six spray ponds. Also,

whether this impact would cause other periods of degraded performance

because of fouling to challenge the capability of the system.Whether the peak spray pond temperature would be significantly lower forloss-of-coolant accident break sizes less than analyzed in the large break design

basis accident. If so, what would the resulting limiting overall heat transfer

coefficient values be for medium and small break sizes given the lower peak

pond temperatures.Best available calculations (technical basis and results) for room temperatureprofiles during loss-of-coolant accidents with loss of room cooling to safety-

related equipment because of a failed essential chiller. Also, best availableinformation of the failure probability and basis associated with the safety relatedpumps affected by the loss of room cooling.An assessment of human error probabilities for important operator actions underthe specific conditions expected to be present if operator action is to be credited

in reducing the risk estimation. Specifically, if the licensee plans to credit

controlling EW flow through the shutdown cooling heat exchanger as a way of

delaying or eliminating a loss of the essential chiller. The technical basis for any "inadvertent cleaning" effect the licensee intends tocredit. This should include an analysis of how the any test data to be used

relates to the geometry and conditions in the Unit 2 EW heat exchangers (i.e.

eddy current probe size, the heat exchanger tube sleeves, normal tube diameter,

thermal conductivity of remaining precipitant, etc.).

21An analysis of the contribution to risk for external events associated with thisissue.5.2Design Control IssuesThree examples of a violation of 10 CFR Part 50, Appendix B, Criterion III, "DesignControl," were identified. The example related to scaling calculations is discussed in

Section 5.1.5.(1) Spray Pond Allowed to Build Up Sediment Introduction. An example of a noncited violation of Criterion III was identified for failureto adequately translate the design basis of the spray ponds into procedures. The design

basis calculation that demonstrated that the spray pond system could adequately limitspray pond temperature during a design basis accident did not account for reduced heat

capacity caused by sediment. However, this was not translated into procedures to

ensure that sediment did not build up in the spray ponds.Description. During early discussions of the design and condition of the spray ponds, itbecame apparent that the spray ponds had sediment along most of the bottom of each

pond. It was also determined that the licensee had never drained or cleaned the ponds,nor did they monitor or quantify the sediment. The team questioned the impact the

sediment had on the ponds' ability to perform their intended functions.Engineering and chemistry personnel initially stated that the ponds needed 18 inches ofat the bottom of the pond to assure adequate spray pond system pump net positivesuction head. This was therefore an unusable water volume. By this reasoning, this

volume of water could be replaced by sediment with no impact, since sediment did not

affect the net-positive suction head (NPSH). Informal reports from contract divers had

indicated that sediment was about 4 inches thick on average, and did not exist near thepump suctions. The licensee determined that a preventive maintenance basis document from the early1990s had specified that a preventive maintenance item should be created to verify thatsediment did not accumulate and impact system capability. However, this was neverimplemented.Additionally, engineering personnel reviewed Calculation 13-MC-SP-0307, "SP/EWSystem Thermal Performance Design Basis Analysis," Revision 7, and determined that

this calculation relied on both the usable and unusable water volume contained in the

ponds to absorb the heat rejected during a design basis accident. Using the entire

water volume yielded the result that under limiting conditions, the maximum pond

temperature was maintained just below the maximum allowable temperature of 110F.

Further, the heat capacity of the sediment was about 40 percent compared to that of thewater that it displaced. Therefore, the team concluded that the spray ponds were alloutside the analyzed condition of the design basis.

2Engineering revised the calculation to account for the quantity of sediment believed toexist in each pond. In doing so, it was necessary to credit heat transfer to the concrete

in the spray pond walls, which was not previously credited. The team reviewed the

calculation and determined that it used reasonable assumptions and demonstrated

adequate margin.As part of the corrective actions, the licensee began a campaign to remove and quantifythe sediment from each spray pond. By the time of this report, three of the six spray

ponds had been vacuumed, removing approximately 400 cubic yards of sediment from

each pond. This was within the volume assumed in the calculation.Analysis. Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be more than

minor because, if left uncorrected, the finding could become a more significant safety

concern. This finding affected the Mitigating Systems cornerstone. This performance

deficiency screened as having very low safety significance (Green) in a Phase 1 SDP

because the licensee was able to demonstrate that the sediment would not haveresulted in a loss-of-safety function. Enforcement. Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion III, requires that design control measures be established to ensure that designbasis information is translated into instructions, procedures, and drawings. The

Calculation 13-MC-SP-0307 was intended to demonstrate that the heat exchangerscooled by the emergency spray pond system would be capable of transfering therequired heat load to the spray pond because scaling would not occur during a design

basis accident. This calculation relied upon the entire volume of water available at the

Technical Specification minimum water level to demonstrate that the pond could perform

its intended safety function. Contrary to this, the licensee did not ensure that the watervolume was maintained as assumed in the calculation through instructions, procedures,

or drawings. This issue was documented in CRDR 2906671. This example of a violationis being treated as a noncited violation, consistent with Section VI.A of the NRC

Enforcement Policy: NCV 05000528; 05000529; 05000530/2006011-05a, InadequateDesign Control to Ensure Spray Pond Sediment Would Not Impact Operability. (2) Spray Pond Integrity Introduction. An example of a noncited violation of Criterion III was identified for failureto adequately translate the design basis of the spray ponds into procedures. The design

basis calculation, which demonstrated that sufficient water was available to provide

adequate cooling during a design basis accident did not account for any leakage from

the ponds. The team determined that the licensee did not translate this into a procedureto ensure that the condition of the spray pond was maintained such that leakage did not

occur.

23Discussion. The team reviewed all CRDRs associated with the spray pond systeminitiated since 2000. In CRDRs 114987, and 2886930, and CORs 00-1-001, 00-2-001,and 00-3-001, the licensee documented cracks that were observed in the accessibleportions of the spray pond walls. In some instances, minor weepage through the wall

was observed. The conditions were repaired. The team reviewed the licensee's treatment of the system within their maintenance ruleprogram, and determined that it was subject to condition monitoring. Condition

monitoring was performed using Procedure 81DP-0ZZ01, "Civil System, Structure, and

Component Monitoring Program, Revision 11. At Palo Verde Nuclear Generating

Station, structural monitoring was performed in one unit every 10 years, rather than in

each unit. The program required that if problems were identified in one unit's structure,

the similar structures in the other units would be inspected.The team was concerned that licensee's Maintenance Rule Program only requiredinspections of the accessible portions of the pond structures. The licensee considered

that the pond bottom and inner walls were inaccessible because of the presence of

water. The team concluded that the portions of the spray pond liners considered by the

licensee to be accessible represented a small fraction of the structure (less than

seven percent). Almost the entire portion that was subject to inspections was above the

water line, were structural integrity was not needed, whereas the uninspected portion

was critical to the safety function.The team identified that the licensee had never drained any the spray ponds orconducted formal or complete inspections using divers. Further, the pond bottom

constituted the majority of the pond liner surface, and this could not be inspected

because the licensee had allowed approximately 4 inches of sediment to buildup on the

pond bottom. Also, the licensee was not able to measure spray pond makeup, nor were

they able to estimate spray pond water loss because of the large daily evaporative

losses, which were highly variable. The licensee calculated that there was sufficient

water maintained above the Technical Specification minimum water level to be able to

tolerate a 5 gpm leak, which was small compared to the daily losses.The licensee acknowledged the need to inspect the pond liners. This was scheduled aspart of CRDR 2901589 to be performed by divers trained to perform this inspection uponcompletion of vacuuming in each pond. Inspections of the three completed ponds

indicated no integrity problems.The licensee performed an assessment which concluded that no significant leakageexisted. This was based largely on the judgement that the geology and hydrology of thesite would cause significant leakage to be observable on the surface, and no such

indication had ever been observed. This was supported by recent efforts to identify the

source of detectable tritium in ground water, which included drilling some holes in thevicinity of the spray ponds. Many of the holes collected no water, and none of the water

that was collected had chemical results which would indicate that it included spray pond

water. Also, the construction methods ensured that there were no joints that could

provide leakage paths through the walls or bottom. The observed cracking was judged

to be too tight to allow significant leakage.

24The team reviewed the photographs from the inspections, the design and constructionof the liner, and the results of ground water studies, and concluded that it was

reasonable to conclude that no significant leakage existed.Analysis. Failure to adequately translate the design basis of the spray ponds intoprocedures was a performance deficiency. This finding was determined to be more than

minor because, if left uncorrected, this finding would be a more significant safety

concern. This finding affected the Mitigating Systems cornerstone. This performance

deficiency screened as having very low safety significance (Green) in a Phase 1 SDP

because it did not result in a loss-of-safety function. Enforcement. Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion III, requires that design control measures be established to ensure thatdesign basis information is translated into instructions, procedures, and drawings.

Calculation 13-MC-SP-0307, "SP/EW System Thermal Performance Design Basis

Analysis," Revision 7, which demonstrated that the system contained sufficient water toprovide adequate cooling during a design basis accident did not provide any margin for

leakage from the ponds. Contrary to this, the team determined that the licensee did nottranslate this information into an instruction, procedure or drawing to ensure that the

condition of the spray pond was maintained such that leakage did not occur. This issue

was documented in CRDR 2910912. This example of a violation is being treated as anoncited violation, consistent with Section V

I.A of the NRC Enforcement Policy:NCV 05000528; 05000529; 05000530/2006011-06b, Inadequate Design Control toEnsure Spray Pond Integrity.5.3Inadequate Implementation Of Operability Assessment ProcessIntroduction. A Green noncited violation with multiple examples was identified for failureto follow Procedure 40DP-9OP26, "Operability Determination and FunctionalAssessment," Revision 16. Key support organizations were not always involving

operations personnel with questions that had a potential to affect operability of safety-related equipment, or were informing operators only after the support organization had

fully evaluated the condition, delaying actions that were required to be prompt by days

or weeks. Also, operations personnel did not always insist on a rigorous evaluation.

This violation was determined to be of very low safety significance because it was

determined to not involve a loss-of-safety function in a Phase 1 SDP evaluation. This

finding has Human Performance Cross-cutting aspects associated with decision making.Description. Charter Item 7 required a review of operability evaluations associated withheat exchanger fouling and an assessment of the compensatory actions and initial

corrective actions to address the issues. During this review, the team noted a number of examples where the assessment ofoperability did not meet the NRC's expectations as delineated in Regulatory IssueSummary 2005-20 or follow the operability assessment process inProcedure 40DP-9OP26, "Operability Determination and Functional Assessment." More specifically, the team concluded that key support organizations (engineering,

chemistry, and maintenance) were not always involving operations personnel with

questions that had a potential to affect operability. In a few cases, operations was

25informed only after the support organization had fully evaluated the condition, delayingactions that were required to be prompt by days or weeks. In a larger number of cases,

support organizations decided not to involve operations. Also, operations personnel

became involved but did not always insist on a rigorous evaluation. The following are

the more significant examples that illustrate these conclusions:*During a surveillance test of EDG 2B on May 17, 2006, high out of specificationintercooler air temperatures were noted. The licensee failed to adequately

assess this degraded condition or formally enter the operability determinationprocess until May 19. Emergency diesel generator operability was informallydiscussed. The EDG system engineer assured the operations shift manager thatthe emergency diesel was operable based on a call with a consultant to the

Cooper-Bessemer Owners Group (inappropriately represented as a

representative of the vendor). The team determined that the scope of thediscussions had been limited to the impact to the engine and had failed to

evaluate other aspects, such as the effect of higher exhaust temperatures on

turbocharger blade creep, increased fuel consumption, and load control. While

these were eventually addressed after NRC inspector involvement, this initialinformal and undocumented operability assessment allowed the licensee toremove the opposite train from service when EDG 2B was degraded and the

impact was not properly understood. When operability was properly addressedon May 19, it was decided that EDG 2B should not have been considered to

have a reasonable expectation of operability.*On May 19, 2006, Work Order 2896333 was initiated to inspect and clean theEDG 2B intercoolers. During review of this work order following EDG 2A

restoration, the on-shift operators determined the high temperatures warrantedan operability determination. Condition Report/Disposition Report 2896661initially concluded that the condition was isolated to EDG 2B. The NRC

inspectors identified that this ignored fouling in both EW heat exchangers on the

common spray pond system side that had necessitated cleaning that same weekto be able to assure operability through the coming hot summer months. However, the team noted that operations did correctly decide to run all EDGs in

accordance with Technical Specification Surveillance Requirement 3.8.1.2 toassess their operability. Additionally, the inspectors noted that the initialevaluation assessed performance using the existing pond and air temperatures,

rather than using the more limiting design basis conditions; when this was

included, the degradation was shown to potentially exceed the design basistemperature limit during accident conditions in two additional EDGs. Following

prompting from NRC inspectors, on May 24, 2006, the licensee appropriatelyexpanded the scope of their evaluation. Additionally, the team noted that thelicensee initiated CRDR 2898120 that states, in part, "It appears that trendingand monitoring and/or extent of condition considerations of the essential water

heat exchanger fouling condition should have provided for identification of the

intercooler fouling condition and prevented the emergent condition discovered on

May 17, 2006."

26*The Prompt Operability Determination that was issued on June 3, 2006,concluded that heat exchangers were adversely affected by a low temperature

fouling mechanism that was already known. The team concluded that this initial

conclusion was based on an incorrect understanding of the situation that datedback to 2003, and was the basis of the decision to stop adding dispersant to the

spray ponds. This was eventually shown to be an inappropriate compensatory

action. However, when a technical evaluation of facts progressed, the cause

assessment correctly triggered a reassessment of operability and moreappropriate compensatory measures. *On June 10, 2006, the licensee identified the failure to implement routinepreventative maintenance tasks to remove biological fouling agents, corrosionproducts, and sedimentation from the essential water cooling spray ponds. The

concern was documented in CRDR 2901737 and was determined to not meetthe procedural requirements for contacting the control room operators as anoperability concern based on the incorrect conclusion that the failure toimplement this task only affected the preventative maintenance process, even

though it was known that the ponds had several inches of sediment buildup.

This issue should have been reviewed by the control room operators in

accordance with procedural requirements because the failure to remove debris

from the ponds could adversely affect the safety function of the ultimate heat

sink based on water displacement and heat transfer considerations. The team

concluded that the reviewers of this CRDR prevented a timely review ofoperability. This technical issue is further discussed in Attachment 4.*On June 10, 2006, the NRC inspectors discussed with the licensee concerns thatsedimentation buildup of the essential water spray ponds could potentially affectoperability of the ultimate heat sink. On June 21, 2006, licensee engineeringpersonnel provided the inspectors a calculation revision demonstrating that any

sedimentation accumulation had placed the ultimate heat sink outside its

analyzed condition for assuring the peak pond temperature limits could be met.

The inspectors noted that engineering should have recognized that all three units

were outside their analyzed condition, but chose to evaluate the condition before

discussing the operability concern with operations (CRDR 2906487 was writtenwith June 28 listed as the discovery date, and CRDR 2906671 was written for theuntimely identification of the issue). The team concluded that site personnellacked a questioning attitude and technical rigor while evaluating this degraded

condition since the accumulation of sedimentation had been known for years at

the site without documenting or evaluating the possible impact to operability untilquestioned by the NRC. *The team questioned whether the spray ponds had leakage which couldchallenge their ability to maintain enough water to perform their intended safetyfunction during a design basis accident. This question was raised when it was

identified that there was a history of cracking in the ponds' concrete liners, and

that the ponds were receiving inadequate inspection and monitoring (see

Section 4.2.3). The team concluded that an operability assessment was requiredby the licensee's procedure for this condition. The licensee wrote

CRDR 2910912 and documented many of the points which would have been

27required in an operability assessment, although a discovery checklist wasinexplicably used to make the case that an operability determination was notrequired. This document reiterated the conclusion of a 2000 assessment in

addressing the structural integrity function of the concrete, rather than the

specific concern raised by the NRC team, which was water loss. The team was

told by the engineering supervisor that performed the evaluation that a hole ofconsiderable size would be required to allow enough leakage to cause a loss of

operability, so his evaluation had not addressed this aspect. The team notedthat this evaluation was based on engineering judgement, was not complete in

addressing the functions affected, and was based on a perception of lack of

evidence of leakage, rather than a solid body of facts. The team found that

operations had exhibited a questioning attitude, and when it was noted that theevaluation had not addressed the leakage aspect, and operations had performed

their own calculation to show that only 5.3 gpm leakage could be tolerated

without losing operability, and this small margin was made possible by creditingthe amount of water normally maintained above the Technical Specification

minimum water level. The team noted that this practice was reasonable for an

operability assessment, but did not address the design problem of having nodesign margin reserved for realistic leakage values.Assessment. Failure to adequately implement the operability assessment process wasa performance deficiency. This finding was more than minor because it impacted the

equipment performance attribute of the Mitigating Systems cornerstone objective to

maintain the availability and reliability of systems needed to mitigate accidents. Thisfinding screened as having very low safety significance in a Phase 1 SDP, since it was

confirmed not to involve any loss-of-safety function. This finding had cross-cutting

aspects in the area of human performance because the licensee did not follow their

systematic process for operability decision making when information was not brought tothe right decision makers.Enforcement. Part 50 of Title 10 of the Code of Federal Regulations, Appendix B,Criterion V, "Instructions, Procedures and Drawings," requires that activities affecting

quality shall be prescribed by instructions, procedures, or drawings, and shall be

accomplished in accordance with those instructions, procedures, and drawings. The

assessment of operability of safety-related equipmnet needed to mitigate accidents wasan activity affecting quality, and was implemented by Procedure 40DP-9OP26,

"Operability Determination and Functional Assessment," Revision 16. Failure to followProcedure 40DP-9OP26, "Operability Determination and Functional Assessment,"Revision 16, to properly and promptly evaluate operability constituted a violation ofCriterion

V. This issue was documented in CRDRs 2918892, 2901815, and 2898237. This violation is being treated as a noncited violation, consistent with Section VI.A of the

NRC Enforcement Policy: NCV 05000528; 05000529;05000530/2006011-07, MultipleExamples of Failure to Properly Implement Operability Assessment Process.5.4Examples of Inadequate Response Under the Corrective Action ProgramIntroduction. Section 5.1.4 briefly discusses an apparent violation of 10 CFR Part 50,Appendix B, Criterion XVI, "Corrective Actions." This section provides more details and

specific examples which demonstrate a pattern of behavior which did not recognize and

28address degraded heat exchanger performance between 1995 and May, 2006. Foulingand degradation of these heat exchangers and cracking in the spray pond structure was

apparent from inspections, performance tests, and surveillance tests, but the licenseefailed to enter most instances into the corrective action program. Of the instances that

were documented, the licensee failed to recognize that some instances constituted a

significant condition adverse to quality, failed to determine the cause, and failed to

implement corrective actions to prevent recurrence.Description. The Palo Verde Nuclear Generating Station corrective actions program isdocumented in Procedure 90DP-0IP10, "Condition Reporting," with Revision 28 being

the current revision at the time of the inspection. Section 5.1.4 documented exampleswhere the licensee failed to properly implement the corrective action program that

directly contributed to a loss of safety function in one train that had potentially greaterthan very low safety significance. The team identified the following additional examples

of a violation where the safety significance was determined to be very low.Specific examples where the licensee's corrective action program was ineffectiveinclude:*Emergency diesel generator intercooler degradation was evident from availabledocumentation starting in 1995, but was not recognized until January, 1999

when it was documented in CRDR 36287. After that, the team identified noother problem reporting in the corrective action program associated with this

ongoing EDG intercooler fouling until May, 2006. Instead, engineering

attempted to manage the problem outside the corrective action program by

repeatedly scheduling cleanings when degradation reached a threshold which

was not formally established. This was determined to be an example of

inadequate problem identification for a condition adverse to quality.*Problem reporting for EW heat exchanger fouling was incomplete and appearsto have been based on a relative departure from "expected fouling" instead of

from an absolute reference. In the early years after implementing the

zinc-phosphate chemistry control program, the team noted that EW heatexchanger performance tests measured up to 75 percent excess capacity, and

on paper, these components had more than 100 percent excess capacity. By

2004, engineering was considering 30 percent excess capacity to be "expected."

This amounted to tolerating significant performance loss without reporting it into

the corrective action program. The following examples associated with the EW

heat exchangers' degraded performance illustrate this performance deficiency:HeatPerformanceDateExchangerMarginComments4/962B3.2%Untimely identification (CRDR 80645initiated 4 months after test).9/021B3.6% No CRDR written.

294/033A -22.0%Condition Report/Disposition Request2598216 failed to identifying that this was a

significant condition adverse to quality,

address the cause, or implement corrective

actions to preclude repetition.10/032A1.0%Condition Report/DispositionRequest 2653867 addressed fouling and

the informally attempted to determine the

cause, but failed to conclude it was a

significant condition adverse to quality when

it was recognized that the problem affected

multiple systems in all trains and all units.2/041B-17%No CRDR written, no cleaning or correctiveaction, so condition was allowed to further

degrade until refueling outage in 5/04. This

constituted a significant condition adverse

to quality that was not entered into the

corrective action program, corrected, or a

cause determined.3/042A28.6%No CRDR written. This constituted acondition adverse to quality, if considered

individually.5/041A9.9%No CRDR written. This constituted acondition adverse to quality, if considered

individually.4/052B25.8%No CRDR written. This constituted acondition adverse to quality, if considered

individually.5/052A29%No CRDR written. This constituted acondition adverse to quality, if considered

individually.10/052B11.1%No CRDR written. This constituted acondition adverse to quality, if considered

individually.10/053B19.3%No CRDR written. This constituted acondition adverse to quality, if considered

individually.

Enclosure3010/051B1.4%Condition Report/Disposition Request2835865 written. In 11/05, CRDR 2860763was written and took 3 months to address

EW heat exchanger fouling in all units. Not

classified as significant condition, and did

not consider EDG impact. Operability

Determination performed for performance

test results in 10/05 was not completed until

1/06.Note that 30 percent margin was used as the threshold by the team in assessingwhen sufficient degradation had occurred to constitute a condition adverse to

quality, to be consistent with the licensee's stated expectations. It could be

argued that a higher threshold could be used, since no degradation should be

present if the chemistry control program was performing as intended. However,

it was considered sufficient to document the above examples to illustrate theextensive pattern of behavior that is being addressed in this violation.*In June, 2006, engineering personnel reviewed historical performance testingresults and noted that EW Heat Exchanger 2B may have been significantly

degraded in 2002-2003. This was not entered into the corrective action program

for a week, and not until the team asked. Condition Report/Disposition

Request 2905161 was written to document this issue.*The corrective action process was ineffective in determining the significance ofthe fouling. The team determined that there was enough evidence to haverecognized that the fouling problems affected both trains and all units as early as

1999, which should have raised the issue to a "significant" classification, but this

did not occur because the licensee did not adequately consider the extent ofcondition or identify the cause.*In December 2005, when it was recognized that the fouling was occurring fasterin EW heat exchangers to the point where their capacity would be assured only

during cooler weather, some of these heat exchangers were taken out of service

while the associated plant was on line to perform midcycle cleanings. However,the fact that the EDG coolers should also be expected to be experiencing the

same degradation was not considered. The worsening of the degradation rate

did not trigger a reassessment of the cause or significance.*In May 2005, chemistry wrote CRDR 2800653 to report that molybdenumanalyses yielded erratic results. They concluded that the analysis method was

not sufficiently accurate (actual accuracy was estimated at 30-50 percent) to

maintain it in the desired control band (225 - 270 PPB). Chemistry began to

question whether the molybdenum tag was accomplishing what they wanted it to

and requested an evaluation of alternatives. The licensee also identified that

samples were untimely because of lack of qualified analysts (would sometimes

wait 2 to 3 weeks to analyze samples). The CRDR also documented that theyhad stopped adding dispersant in November 2005, but still recorded increases inmolybdenum or other unexpected results in over half of the samples. The team

Enclosure31concluded that this represented a missed opportunity to recognize that they werenot adding dispersant, and that dispersant concentration was no longer coupled

with the tag chemical. The evaluation of alternatives considered other tag

chemicals, but did not consider other dispersants which could be measured. The

team noted that this CRDR was closed with the only change being to qualifymore analysts to perform the molybdate analysis.Assessment. The significance of this finding will be assessed as part of the finalsignificance determination. See Section 5.1 for the assessment of this finding.Enforcement. See Section 5.1.4 for the enforcement discussion associated with thisfinding.6.0ADDITIONAL INFORMATION6.1Summary of Chemistry Control ChangesThere were three distinct periods of interest in this inspection. From 1994 to 2001, thelicensee used Nalco (later bought by Calgon) chemicals. In 2001, the Arizona Public

Service corporate system changed to Betz chemicals. Palo Verde Nuclear GeneratingStation changed to what was believed to be equivalent chemicals, but maintained

essentially the same chemistry control program for the spray ponds. In 2004, the

dispersant was changed because it was thought that the dispersant was reacting withthe biocide and forming an acrylic polymer and fouling heat exchangers.The licensee implemented zinc-phosphate chemistry control in the SP systems in 1994. The proprietary specialty chemicals used were:Dispersant:CALGON PCL-401 (AA/AMPS)Corrosion inhibitor:CALGON MSW-109 (zinc chloride in phosphoric acid (3:1)

Biocide: BULAB 6002

Chemical supplier: NALCO/CALGONInitially, zinc limits were 1.0 - 1.5 ppm, the pH band was 7.6-8.2, and dispersant wasadded three times per week.Between 1995 and 1999, some degraded performance was noted in EDG intercooleroutlet temperatures. This was not documented until 1999. Also, in 1995, the spraypond system began to have corrosion problems where localized corrosion nodulesformed and broke off, blocking heat exchanger tubes.On 4/24/96, the pH band was lowered from 7.6 - 8.2 to 7.4 - 8.0.In 1997, the licensee attempted to chemically passivate spray pond piping bysignificantly lowering pH to remove corrosion products, then reestablish a protective zinc

layer. This was unsuccessful.On 3/3/98, pH specification was raised from 7.4 - 8.0 to 7.6 - 8.6. On 8/13/98, thedispersant addition time was increased from 2.5 to 3 hours3.472222e-5 days 8.333333e-4 hours 4.960317e-6 weeks 1.1415e-6 months .

2On 9/18/98, the licensee split the requirements so that Unit 1 had different limits, sincethey were changing Unit 1 so makeup water came from wells. Since this water had

higher alkalinity and thus a higher scaling potential, Unit 1 was given a lower pH band to

improve solubility and avoid scaling. Unit 1 had a pH specification of 7.8 - 8.2, whileUnits 2 and 3 had 8.0 - 8.4. However, on 7/6/99, this was reversed, and all units had apH specification of 8.0 - 8.4. No explanation was given why the increased potential forscaling was acceptable.On 7/11/00, the upper zinc specification was changed from 1.0 to 1.5 ppm.In 2001, the corporate chemistry group for Arizona Public Service changed chemicalsupply companies. On 5/30/01, the licensee began adding the following new chemicals:Dispersant:BETZ PY5200 (AA/HPS-1)Corrosion inhibitor:BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002

Chemical supplier: BETZIn October 2003, the licensee performed the first EW heat exchanger tube cleaning in2R11 in response to fouling. Since then, 100 percent of the EW heat exchanger tubesare cleaned every refueling outage.On 1/29/04, the licensee concluded from informal lab testing that EW heat exchangerfouling was being caused by an undesirable interaction between the dispersant and the

biocide. This was based on discussions with the chemical supplier's site representative,

and included informal lab tests which showed that a white substance could be formed by

mixing the chemicals, although the conditions of this test were not documented and the

substance formed was not analyzed. The team noted that the conclusions drawn fromthis informal test dominated the licensee's response through June, 2006.In March 2004, the dispersant was changed to address EW heat exchanger fouling. The chemicals were now:Dispersant:BETZ DN2317 (AA/APES)Corrosion inhibitor:BETZ MS-6209 (zinc oxide in phosphoric acid in 4:1 ratio)

Biocide: BULAB 6002

Chemical supplier: BETZThe target dispersant concentration was 12.5 - 15 ppm. The previous dispersant wasnot removed from the ponds through feed and bleed, but was expected to continue to

react and be removed by spray pond sand filters. There was no evaluation of adding a

new dispersant with the previous dispersant still present, or an analysis of theacceptability fo the new dispersant for the system (LIST CRDR). Also, the zinc

specification was doubled from 0.5 - 1.5 ppm to 1.0 - 3.0 ppm.In November 2004, a white "lotion-like" substance identified in Spray Pond 2C sandfilter. The cause was believed to be having the old dispersant and the new dispersant,

polymerizing and coagulating into larger particles (essentially doing what they were

supposed to do, but at a higher rate). It was not believed to be a problem for heat

Enclosure33exchangers because of the mobility of the substance and most recent EW heatexchanger performance test showed improving trend (the team noted that this was

actually after a midcycle cleaning with only 6 months of operation, so it was not useful toconsider this data). The licensee implemented more frequent backwashing of spray

pond sand filters, believing that this would remove the problem chemical. Chemistry

explained that this took 10 months to happen after changing dispersants by saying it

was because of cooler temperatures in the pond. However, the team noted that this didnot take into consideration that the dispersant change took place in January when theponds were also cold. In May 2005, chemistry wrote CRDR 2800653 to report that molybdenum analysesyielded erratic results. They concluded that the analysis method was not sufficiently

accurate (actual accuracy was estimated at 30-50 percent) to maintain it in the desired

control band (225 - 270 PPB). Chemistry began to question whether molybdenum tag

was accomplishing what they wanted it to and requested an evaluation of alternatives.

The licensee also identified that samples were untimely because of lack of qualified

analysts (would sometimes wait 2 to 3 weeks to analyze samples). The CRDR alsodocumented that they had stopped adding dispersant in November 2005, but stillrecorded increases in molybdenum or other unexpected results in over half of the

samples. The team concluded that this represented a missed opportunity to recognize

that they were not adding dispersant, and that dispersant concentration was no longercoupled with the tag chemical. The evaluation of alternatives considered other tag

chemicals, but did not consider other dispersants which could be measured. The teamnoted that this CRDR was closed with the only real change being to qualify moreanalysts to perform the molybdate analysis.On 10/17/05, the pH specification was changed from 8.0 - 8.4 to 7.8 - 8.4 in order toimprove chlorine effectiveness and zinc solubility.The team noted that between October 2005 and May 2006, both spray ponds in Units 2and 3 had molybdate results that were above the target level of 200-300 ppb almost

continuously, even though dispersant was largely not being added. The validity of theseresults was not questioned. The team considered that this was imprudent, since

evaporation required adding 2,200 to 10,000 gallons of makeup water per day if the

pond was idle, more if the sprays were run. Of note, Unit 1 spray ponds were

maintained within the target band during most of this period. The team reviewed the

dispersant usage by assessing tank levels in each unit. Between October 2004 and

August 2005 little or no dispersant was added to any of the ponds. From August 2005

until May 2006, dispersant was added fairly regularly in Unit 1, less regularly in Unit 3,

and infrequently in Unit 2. This was known to the chemistry line organization, but not to

management or the root cause team until the inspection team requested theinformation. The team also compared the recent dispersant usage to earlier usage.

Since changing to the latest dispersant in March 2004, the total dispersant used in

months was about equal to the annual used in years prior to that change. The lack

of regular additions coupled with rising concentrations of other chemicals tended to

indicate that there was increasingly insufficient dispersant to prevent loss of solubility in

the spray pond systems.

Enclosure34The team assessed the history of differences in chemistry parameters between theunits. The team noted that Unit 2 consistently had the highest levels of phosphates, andthe pH was above 8.2 for long periods of time compared to the other units. Unit 1 had

roughly double the calcium compared to the other units; this was known to be because

of using well water in this unit.7.0SUMMARY OF HOW THE TEAM ADDRESSED THE CHARTER SCOPE 1.Sequence of EventsThis topic is documented in Attachment 4. 2.Operating ExperiencePersonnel from the Operating Experience Branch of NRR supported the team byperforming searches of operating experience databases and other sources. The intent

was to identify operating experience report of similar problems and other relevant

information, both within and outside the nuclear industry. Palo Verde Nuclear

Generating Station had one instance of tube blockage, which was documented inLicensee Event Report 50-528/95-005-00, "Corrosion Nodules Blocking EDG 1B Jacket

Water Cooler Tubes." This involved chemistry challenges in the spray pond system, butdid not involve precipitation or scaling. Only two reports of problems with similar

circumstances were identified, although the causes were different. It was noted,

however, that outside the nuclear industry, this type of condition would probably not be

considered to be serious. However, these searches identified information that indicated

that zinc-phosphate chemistry control programs similar to the one used by Palo Verde

Nuclear Generating Station were in common use in similar systems, and were usedeffectively. 3.Generic IssuesThe chemical control program in use in the Palo Verde Nuclear Generating Stationspray ponds and associated systems were generally compatible when used together ina coordinated program. This type of program was in common use and effectively

employed in similar applications. However, the unique design of the spray pond systemand the unique operating practices were not adequately reviewed to ensure

compatibility.The only generic issue identified was that AA/HPS-1 type dispersant is know to interactwith polyquat amine-type biocide and form an acrylic polymer that can cause water

problems, including possibly fouling. The acrylic acid in this product has a strong

negative charge compared to other dispersants, and will combine with the biocide. 4.Corrective ActionsDuring this inspection, the licensee corrected the limits for the spray pond in thechemistry control program, and restored the ponds to within the new limits by blowing

down the ponds. They were also evaluating changing the dispersant to regain the abilityto monitor it. The ponds were being vacuumed and inspected to remove chemical

Enclosure35sludge and sediment, and to verify pond integrity. Part 50.59 reviews were planned forpast changes to chemistry procedures. Emergency diesel generator intercoolers were

promptly cleaned, and increased frequency monitoring of performance was

implemented while chemistry improvements were being identified and implemented. 5.Root Cause Determination.The licensee's determination of the cause was broad and thorough in assessing theproblems and the history. This effort included looking at both the technical issues and

the organizational aspects, which was appropriate. Adequate resources were devoted

this effort, including obtaining several sources of outside expertise in a number of

important aspects of the evaluation. The draft report, although awaiting some technical

information, thoroughly documented the facts available, the process used, the possiblecauses considered, and the conclusions. This effort also identified a number of

peripheral issues that were entered into the corrective action program. 6.Design Basis and Technical Specifications ComplianceA number of problems were identified with the design basis and Technical Specificationcompliance. These are discussed in the Findings section. 7.Operability Assessments and Compensatory MeasuresA number of problems were identified with operability assessments. These arediscussed in the Operability Assessment section. The compensatory measures andinitial corrective actions implemented by the licensee were appropriate and timely, with

the exception of stopping dispersant additions. In retrospect, this was inappropriate

based on the later understanding of the cause of the fouling; however, from the

perspective of what was known at the time, this was not unreasonable. Dispersant

additions were resumed in a prompt manner when the cause was better understood. 8.Heat Exchanger TestingThe test methodology was reviewed along with several years worth of test results. Themethodology was in accordance with industry standards, although the calculations

provided results which were not typical. Specifically, the heat exchanger performance

was calculated for the existing conditions, and the acceptance criteria (reflecting design

basis conditions) were extrapolated to the existing condition results. The normal

practice is to extrapolate the results from the existing conditions to design basis

conditions. The team concluded that, while the licensee's practice made comparison of

results technically awkward, this did not affect the ability to demonstrate properperformance. The team also identified that the licensee's practice of having the test andthe calculation as separate activities contributed to having untimely results from the

performance tests in at least one case, and possibly others. 9.Risk Analysis

Enclosure36Information was gathered to support a preliminary risk assessment. However, keyinformation was still needed to be able to perform a final risk analysis. This is furtherdiscussed in the Findings section under the Analysis heading. 10.Methods of Heat Exchanger ProtectionThe team assessed the chemistry control program design and implementation,operating practices, inspection and cleaning methods and results, and the cathodic

protection used in portions of the system. As discussed in various parts of this report,the chemistry control program was changed over time in ways that created heat

exchanger fouling. Corrosion control was being effectively implemented, however. 11.Effectiveness of Heat Exchanger Performance MonitoringAs discussed in various parts of this report, performance monitoring was not effective inidentifying and correcting heat exchanger degradation. The tests and inspections were

being done according to commitments, but the results that indicated increasing amountsof chemical residue, and the need for increasing frequent cleanings, were not followed

up to determine and correct the cause. This was an organizational problem, rather than

a problem with the monitoring methods or frequencies.1R07Biennial Heat Sink Performance (71111.07B) a.Inspection ScopeThe team reviewed design documents (e.g., calculations and performancespecifications), program documents, implementing documents (e.g., test procedures,

maintenance procedures, and actual data), and corrective action documents. The team

interviewed chemistry personnel, maintenance personnel, engineers, and program

managers. The team verified whether testing, inspection and maintenance, or the biotic foulingmonitoring program provided sufficient controls to ensure proper heat transfer.

Specifically, the inspectors reviewed heat exchanger test methods, test results from

performance testing, inspection results, and chemical controls to limit fouling. For the ultimate heat sink and its subcomponents, the team reviewed the heat sink todetermine if it was free from clogging because of macrofouling and provided sufficient

controls to ensure proper heat transfer. The inspectors reviewed; (1) heat exchanger

test methods and test results from performance tests, (2) heat exchanger inspection and

cleaning methods and results, and (3) chemical treatment for the spray ponds to control

fouling. The team selected the following heat exchangers for this inspection: *Essential cooling water heat exchanger*Emergency diesel generator jacket water heat exchanger

Emergency diesel generator inter-coolerThe team completed 3 of the 2 to 3 required samples.

Enclosure37 b.FindingsAn apparent violation of Criterion XI, "Test Control," associated with EW heatexchanger testing was discussed in Section 5.1.2. A minor violation of 10 CFR Part 50, Appendix B, Criterion III, was identified for using an inappropriate foulingfactor in design documents for the EDG intercoolers. The team identified that

fouling factors for clean water (0.0005) was used on both sides of these heat

exchangers, where fouling factors of 0.002 was appropriate for the application.

The latter value was correctly used in the other EDH heat exchangers. This was

determined to be minor because the team calculated that this had a small impact(4 percent) to the point at which these heat exchangers would not be able toperform their intended function, compared to the large margins available in the

EDG system. This minor issue was entered into the corrective action program

as Action Request 051201123.4OA6Meetings, Including ExitOn August 18, 2006, the inspectors presented the inspection results to Mr.

C. Eubanks,Vice President, Nuclear Operations, and Mr.

D. Mauldin, Vice President, Engineering,

and other members of his staff, who acknowledged the findings.An additional exit was conducted on September 26, 2006, to provide the results ofpreliminary significance determination and to change the characterization of the

associated issues to Apparent Violations. The findings were discussed with Mr

J.

Levine and members of his staff, who acknowledged the findings.The inspectors confirmed that proprietary information that was examined during thisinspection was properly handled in accordance with NRC policy.4OA7Licensee-Identified ViolationThe following violation of very low safety significance (Green) was identified by thelicensee and was a violation of NRC requirements which meets the criteria of Section VIof the NRC Enforcement Policy, NUREG-1600, for being dispositioned as NCVs.*Part 50,59 of Title 10 Code of Federal Regulations allows licensees to makechanges to procedures described in the Updated Final Safety Analysis

Report without prior NRC approval provided that the change satisfies thecriteria in Part 50.59.c. Contrary to this, Revisions 2, 6, 7, 8, and 9 to

Procedure 74DP-0CY01, "Specification for Bulk Chemicals," were made

between 1997 and 2004 without 50.59 reviews. This was identified in the

licensees corrective action program under CRDRs 2864575 and 2683642. Thisviolation is of very low safety significance because it did not actually impede the

regulatory process or contribute to degraded heat exchanger performance. ATTACHMENT 1: SUPPLEMENTAL INFORMATION

1SUPPLEMENTAL INFORMATIONKEY POINTS OF CONTACT

Licensee Personnel

G. Andrews, Department Leader, System Engineering

P. Borchert, Director, Operations

D. Breckenridge, Consulting Engineer, Arizona Public Service

P. Carpenter, Unit Department Leader, Operations

C. Churchman, Director, Engineering

A. Dave, Senior Engineer, Design Engineering

C. Eubanks, Vice President, Nuclear Operations

T. Green, Team Leader, Chemistry

D. Hautala, Senior Compliance Engineer

P. Heinstein, Chemical Vendor Site Representative, GE Betz

J. Hughey, Senior Engineer, System Engineering

H. Hurley, Root Cause Team Sponsor

R. Jenkins, Senior Chemist, Chemistry

D. Kanitz, Senior Compliance Engineer

M. Karbasian, Department Leader, Design Mechanical Engineering

J. Levine, Executive Vice President, Generation

D. Mauldin, Vice President, Engineering

J. Proctor, Section Leader, Regulatory Affairs - Compliance

M. Radspinner, Section Leader, System Engineering

C. Seaman, General Manager, Regulatory Affairs and Performance Improvement

T. Selby, Chemistry Consultant

G. Sowers, Section Leader, PRA

D. Straka, Senior Consultant, Regulatory Affairs

D. Vogt, Section Leader, Operations/STA

NRC personnel

J. Melfi, Resident Inspector, Palo Verde Nuclear Generating Station

E. Owen, Reactor Inspector, Region IV

N. Sieller, Operations Experience Branch, NRR

W. Sifre, Senior Reactor Inspector, Region IV

R. Telson, Operations Experience Branch, NRR and Acting Resident Inspector, PVNGS

S. Unikewicz, Senior Engineer, NRR

G. Warnick, Senior Resident Inpspector, Palo Verde Nuclear Generating Station

G. Werner, Senior Project Engineer, RIV

1A1-2

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened05000529/2006011-01,AVEW Train 2B Inoperable Longer than AllowedOutage Time (Section 5.1.1)05000528;

05000529;05000530/2006011-02AVInadequate Test Control to Promptly IdentifyUnacceptable Performance Test Results (Section

5.1.2)05000528;

05000529;05000530/2006011-03AV50.59 Reviews Not Performed or Inadequate forMultiple Changes to Spray Pond Chemistry Control

Procedure (Section 5.1.3)05000528;

05000529;05000530/2006011-04AVInadequate Identification and Corrective Action forDegraded EW Heat Exchanger Performance

(Section 5.1.4)05000528;

05000529;05000530/2006011-05AVInadequate Design Control to Ensure No EW HeatExchanger Scaling.(Section 5.1.5)

Opened and Closed

05000528;

05000529;05000530/2006011-06NCVTwo Examples of Failure to Translate Spray PondDesign Assumptions Into Plant Procedures Control

(Section 5.2)05000528;

05000529;05000530/2006011-07NCVMultiple Examples of Inadequate OperabilityAssessments for Heat Exchanger Degradation

(Section 5.3)

LIST OF DOCUMENTS REVIEWED

Procedures30DP-9MP08, Preventive Maintenance Program, Revision 12

40OP-9SP01, Essential Spray Pond Train A, Revision 35

40DP-9OP26, Operability Determination and Functional Assessment, Revision 1670TI-9EW01, Thermal Performance testing of Essential Cooling Water Heat Exchangers,Revision 570DP-9SP01, Monitoring of ESPS Piping Integrity, Revision 3

73DP-9ZZ10,

Guidelines for Heat Exchanger Thermal Performance Analysis, Revision 4

1A1-373DP-9ZZ11, Heat Exchanger Condition Monitoring,

Revision

474DP-9CY04, Systems Chemistry Specifications, Revision 37 (other revisions were alsoreviewed)90DP-0IP10, Condition Reporting, Revision 28

Corrective Action Documents24743132484639

21395

2591826

2598216

2653867

265423126836422699739

2748191

2750856

2789993

2819671

2478428248652825469

28929

2835865

2837696

2859430

286076328645752870799

2886930

2896661

2897810

2898120

289823728966612901500

2901737

2901815

2902498

290250429041012905161

2905162

2906671

2908351

2987810Work Orders25373002723814

23815

2381827238192724390

24394

280552528055262886543

2886547

289633328970782897080

289712828971302898676

2898679Deficiency

Work Orders2564416270028650.59 DocumentsScreening S-06-0049, Revision 1, "Add Boron-10 Specifications to 74DP-9CY04 (Rev 36),"dated 2/15/06Calculations13-MC-DG-206,"Corrosion Allowance For DG Inter-cooler Water Boxes End Plates, datedJanuary 15, 199113-MC-DG-411, "DG Heat Exchanger Minimum Flow Rate vs. Inlet SP Water Temperature,"dated may 7, 199613-MC-DG-410, "Spray Pond Minimum Flow Rates To Diesel Generator Heat Exchangers," Revision 1 13-MC-DG-206, "Corrosion allowance For DG Inter-cooler Water Boxes End Plates," datedJanuary 15, 199113-MC-DG-403, "Diesel Intake Exhaust and Drain Pipe P/T," Revision 0

1A1-413-MC-SP-0307, "SP/EW System Performance Design Bases Analysis," Revision 00713-NS-B098, At-power PRA Study for the HVAC Room Coolers, Revision 2

MiscellaneousGE Betz Memo, "SEM/EDXA Analysis of Deposit Samples from Arizona Public Service - PaloVerde Nuclear Generating Station," Dated December 1, 2003Preventive Maintenance Basis Document 248948

Updated Final Safety Analysis Report Section 9.2, Cooling Water Systems, and Section 9.5.5,Diesel Generator Cooling Water System, Revision 13Regulatory Guide 1.27, "Ultimate Heat Sink for Nuclear Power Plants," Revision 2VTD-P162-0002, "Perfex Industrial Products Data Sheets and Drawings for Diesel Generator Inter-Coolers" Revision 113-NS-C076, "MSPI Data," Revision 00

RCTS

010359,

IEC 78-13 Perform Periodic Inspection of Spray Pond Intake Structure forSand/Silt Buildup dated 6/22/90.LDCP Number 2LM-EW-036, Sleeving of Unit 2 Trains A and B Essential Cooling Water HeatExchangers, 9/9/93PVNGS Design Basis ManualsAuxiliary Building HVAC System, Revision 13

Control Building HVAC System, Revision 13

Essential Cooling Water System, Revision 18

Essential Chilled Water System, Revision 10

Essential Spray Pond System, Revision 15

Diesel Generator, Class 1E Standby Generation, Fuel Oil Storage and Transfer System, Revision 016Operating ExperiencePalo Verde Nuclear Generating Station Licensee Event Report 50-528/95-005-00, CorrosionNodules Blocking EDG 1B Jacket Water Cooler TubesBraidwood Operating Event Report, "Lake Chemistry Trend, Calcium Carbonate Issue," dated2/17/04Braidwood Operating Event Report on Calcium Carbonate Precipitation on the Non-essentialService Water Strainers, dated 4/18/02

1A1-5NRC Information Notice 94-79, Microbiologically Influenced Corrosion of Emergency DieselGenerator Service Water Piping, 11/23/94NRC IE Circular 78-13, Inoperability of Multiple Service Water Pumps, 7/6/78RegulatoryCommitment Tracking System Number

010359, Perform Periodic Inspection of Spray Pond Intake Structure for San/Silt Buildup

2A2-1ATTACHMENT 2SPECIAL INSPECTION CHARTERJune 7, 2006MEMORANDUM TO:Neil O'Keefe, Senior Reactor InspectorEngineering Branch 2

Division of Reactor SafetyFROM:Dwight D. Chamberlain, DirectorDivision of Reactor Safety SUBJECT:SPECIAL INSPECTION CHARTER TO EVALUATE PALO VERDE NUCLEARGENERATING STATION, UNITS 1, 2, AND 3 HEAT EXCHANGER FOULING You are hereby designated as the Special Inspection Team leader.

Your team members arePaula Goldberg, Yamir Diaz-Castillo, Krzysztof Parczewiski, and Mike Runyan.A Special Inspection Team is being chartered in response to the discovery of fouling of theemergency diesel generator (EDG) intercooler heat exchangers and essential cooling water heat exchangers.

Fouling of each of the heat exchangers cooled by this system could lead to afailure of the EDGs to supply rated electrical power to safety-related components and a reduction in cooling to essential chilled water (room cooling for emergency core cooling systemcomponents), shutdown cooling, fuel pool cooling, and nuclear cooling water.

The licensee implemented actions to inspect and clean the affected heat exchangers in order to restore margin for safety system operability.A.BasisOn May 17, 2006, during surveillance testing of the Unit 2 B Train EDG, the licenseeobserved an abnormal increase in turbocharger air temperature.

The cause of the increase is associated with fouling of the EDG intercooler heat exchanger.

On May 19, during a review of the increased turbocharger air temperature, the licensee determined that a reasonable expectation of operability no longer existed and declared the EDGinoperable.

The licensee subsequently cleaned the intercooler heat exchanger and declared the EDG operable.

The licensee also initiated work packages to clean and inspect the remaining five EDG intercooler heat exchangers.

Each of the EDG

intercooler heat exchangers had experienced an increase in turbocharger air temperature, but not to the same magnitude as the Unit 2 Train B

EDG.On May 25, 2006, a teleconference was conducted with the licensee to discuss theextent of the condition and cause of the intercooler heat exchanger fouling.

The licensee indicated that in 2001, a chemical additive (dispersant) to the spray pond reacted with a biocide to create the chemical substance that was found on the EDGintercooler heat exchangers and the essential cooling water heat exchangers in all three units.

Dispersant is used to prevent ions (e.g., iron) from plating out on spray pond system surfaces.

The dispersant coats the ions and aids in the filtration of theions from the spray pond system.

The licensee indicated that the fouling of the heatexchangers was initially discovered in 2003.

2A2-2In March 2004, the licensee changed the type of dispersant and implemented morefrequent (once per refueling outage) inspections and cleaning of the essential cooling water heat exchangers.

The licensee indicated that heat exchanger fouling decreased following the decision to change the type of dispersant.

However, recent performance testing of the essential cooling water heat exchangers, EDG testing, and heat exchanger inspections have shown that the rate of fouling of the heat exchangers is increasing.

The licensee indicated that degradation of the EDG intercooler heat exchanger wouldnow be detected during surveillance testing.

The resident inspectors noted that theUnit 2 Train B EDG had operated at higher temperatures for several months without being questioned by the licensee.

The licensee also indicated that a performance test of the essential cooling water heat exchanger is performed whenever they initiate shutdown cooling operations and that the essential cooling water heat exchangers arecleaned every 18 months.

However, the inspectors noted that the rate of fouling of the essential cooling water heat exchangers may result in a reduction of essential cooling water heat exchanger performance to below design margins during the operating cycle. The inspectors also noted that the licensee's actions were focused on cleaning degraded heat exchangers and not on eliminating the source of the chemical fouling.The preliminary risk assessment for this condition determined that the increase inconditional core damage probability was 1E-6.

The risk assessment assumed that thefouling of the EDG intercooler heat exchangers and essential cooling water heat exchangers may affect the reliability of the following components, either directly orthrough a loss of room cooling:- emergency diesel generators- motor and turbine-driven auxiliary feedwater pumps- high pressure injection pumps

- low pressure injection pumps

- containment spray pumps

- shutdown cooling heat exchangersTo assess the risk of the condition, the analyst performed a sensitivity analysis todetermine the amount of change in unreliability of these components that would cause achange in core damage probability of 1.0E-6, considering an exposure period of 1 year.Using the Palo Verde Nuclear Generating Station SPAR model, Revision 3.2.1, andassuming average test and maintenance, the analyst determined that an increase inunreliability of 30 percent of all affected components would result in an increase in coredamage probability of 1.0E-6.

To attain this result, the fail-to-run individual and commoncause basic events were adjusted by the same amount for all components except for the shutdown cooling heat exchangers (a basic failure event was adjusted in this case).

2A2-3B.ScopeThe team is expected to address the following:1.Develop a complete sequence of events related to the discovery of the chemicalfouling of the EDG and essential cooling water heat exchangers.

2.Compare operating experience involving fouling of heat exchangers to theidentified issues at Palo Verde Nuclear Generating Station.

3.Determine if there are any generic issues related to the use of multiple chemicalsand their subsequent impact on operating systems.

Promptly communicate anypotential generic issues to regional management.4.Review the extent of condition determination for this condition and whether thelicensee's previous and planned corrective actions are comprehensive. 5.Review the licensee's determination of the cause of any deficiencies and/oroperating practices that allowed the chemical fouling to occur and continue.

Independently verify key assumptions and facts.

If available, determine if the licensee's root cause analyses and corrective actions have addressed the extent of condition.6.Determine if the design basis and technical specifications were met for theeffected systems.

7.Determine if the operability assessments, supporting analyses, andcompensatory measures for heat exchanger fouling were made in accordance with

RIS 2005-20.8.Evaluate the adequacy of heat exchanger testing and calculations that wereperformed to evaluate the effect of heat exchanger fouling. 9.Collect data necessary to support a risk analysis.

Specifically obtain informationassociated with the degree to which the EDG and essential cooling water systems would be affected during an event.10.Assess the method(s) used by the licensee to protect the heat exchangers fromcorrosion, scale formation, fouling, and other biological impacts.11.Assess the effectiveness and appropriateness of the tests and inspections usedto determine heat exchanger performance and capability, including frequencyand acceptance criteria.C.GuidanceInspection Procedure 93812, "Special Inspection," provides additional guidance to beused by the Special Inspection Team.

Your duties will be as described in InspectionProcedure 93812.

The inspection should emphasize fact-finding in its review of the

2A2-4circumstances surrounding the event.

It is not the responsibility of the team to examinethe regulatory process.

Safety concerns identified that are not directly related to theevent should be reported to the Region IV office for appropriate action.The Team will report to the site, conduct an entrance, and begin inspection no later thanJune 19, 2006.

While on site, you will provide daily status briefings to Region IVmanagement, who will coordinate with the Office of Nuclear Reactor Regulation, toensure that all other parties are kept informed.

A report documenting the results of theinspection should be issued within 45 days of the completion of the inspection.This Charter may be modified should the team develop significant new information thatwarrants review.

Should you have any questions concerning this Charter, contact me at

(817) 860-8180.

3A3-1ATTACHMENT 3SUMMARY OF HEAT EXCHANGER PERFORMANCE TEST RESULTSEW 1AEW 1BEW 2AEW 2BEW 3AEW 3B3/92 1R2;tube leak identifiedAll tubessleeved in

2/93 (8" at inlet and outlet)

becasue of cracking in

2R4All tubessleeved in

2/93, (8" at inlet and outlet)

because of cracking in

2R44/94, 3R4:Perf test

54.1%

margin12/94 BeganZinc-

Phosphate chemistry control program.12/94 BeganZinc-

Phosphate chemistry control program.4/94 Unit 2Bpond begins trial use of Zinc-

Phosphate chemistry control program.12/94 BeganZinc-

Phosphate chemistry control program.4/95, 1R4:perf test 56%

and 33.8%

margin at outage beginning and end.4/95, 1R4perf test

39.8%

margin2/95, 2R5:perf test 56%

margin.3/95, 2R5:perf test

41.9%

margin.11/95, 3R5;end of outage perf test 24.9%

margin.10/95, 3R5;beginning of outage perf test 42.8%

margin.10/96, 1R6;end of outage perf test 34.3%

margin.9/96, 1R6;early outage perf test

55.7%

margin.2R6 - No testor cleaning.4/96, 2R6;beginning of outage perf test 3.2%

margin.

No cleaning,

CRDR

written 4

months later.3R6 - No testor cleaning.3/97, 3R6;early outage perf test

37.1%

margin.

EW 1AEW 1BEW 2AEW 2BEW 3AEW 3BAttachment 3A3-23/98, 1R7;early outage perf test

58.7%

margin.4/98, 1R7;end of outage perf test 43.5%

margin.9/97, 2R7;early outage perf test

37.8%

margin.10/97, 2R7;chemical passivation

(cleaning).

End of outage perf test 38.6%

margin.10/98, 3R7;beginning of outage perf test 55.5%

margin.10/98, 3R7;end of outage perf test 71.7%

margin10/99, 1R8 -No test or cleaning.10/99, 1R8;beginning of outage perf test 37.8%

margin.4/99, 2R8;early outage perf test

2.6%

margin.4/99, 2R8;late outage perf test 64.7

% margin.4/00, 3R8;early outage perf test

40.8%

margin.4/00, 3R8 -No test or cleaning.4/01, 1R9;early outage perf test

31.5%

margin.1R9 - No testor cleaning.10/00, 2R9;early outage perf test

27.6%

margin.10/00, 2R9 -No test or cleaning.3R9 - No testor cleaning.10/01, 3R9;early outage perf test 24%

margin.9/02, 1R10 -No test or cleaning.9/02, 1R10;early outage perf test

3.6% margin.

No CRDR

written, no cleaning.9/02, 2R10 -No test or cleaning.3/02, 2R10;early outage perf test

-9.5%

margin. No cleaning. 4/03, 3R10;early outage perf test

-22.0%

margin before cleaning,

26.8% after.

All tubes inspected with boroscope.4/03, 3R10 -No test or cleaning.10/03, 2R11;early outage perf test

1.0% margin.

Post-

cleaning margin

2.8%.10/03, 2R11;EW HX 2B

was cleaned but not tested.CRDR

2653867 ACE concluded dispersant was interacting with biocide in Jan 04.

Changeddispersant starting with a batch add to each pond in 2/04.

EW 1AEW 1BEW 2AEW 2BEW 3AEW 3BAttachment 3A3-3 2/04,midcycle outage perf test -17.1%

margin.3/04,midcycle outage perf test 28.6%

before cleaning. 3/04,midcycle outage, cleaning performed but no test.3/04,midcycle outage;

cleaning performed but no test.3/04,midcycle outage;

cleaning performed but no test.5/04, 1R11;9.9% margin before cleaning,

34.5% after.5/04, 1R11 -EW HX 1B

cleaned but not tested.10/04, 3R11EW HX 3A

cleaned but not tested.10/04, 3R11;34.6%

margin before cleaning,

26.8% after."Lotion-like" substance reported in water from Unit 2 sand filter.

CRDR 2750856.5/05, 2R12;post-cleaning margin 29%.4/05, 2R12;margin

25.8% before cleaning.1R12 - EWHX 1A

cleaned but not tested.

Fouling present.10/05, 1R12;1.4% marginbefore cleaning,

41.4% after.

Fouling present.10/05,midcycle outage; perf test 11.1%

margin.

No cleaning performed.10/05,midcycle outage; perf test 19.3%

margin.

No cleaning performed.6/06,midcycle outage; perf test 28.6%

before cleaning,

36.3% after.3/06,midcycle outage; perf test 32.9%

margin before cleaning.5/06 -midcycle cleaning performed.5/06 -midcycle cleaning performed.4/06, 3R12;30.3%

margin before cleaning,

36.3% after.3R12; cleaned but not tested.

4A4-1ATTACHMENT 4SEQUENCE OF EVENTS1994Licensee implemented zinc-phosphate chemistry control program for spray pondsystem.1995Preventive Maintenance Basis for spray ponds revised to specify removingsediment every two cycles.

No preventive maintenance item was ever written toimplement this.4/96Essential Cooling Water 2B performance measured 3.2 percent margin. CRDR 80645 written 8/27/96.9/98Unit 1 spray pond makeup source changed to well water.

Lower pH rangespecified for Unit 1 to account for higher alkalinity to avoid scaling.1/99Degraded performance in EDG intercoolers first noted.

It was later determinedto have been evident from records starting in 1995.

Condition Report/Disposition Request 36287 written.7/99Unit 1-specific pH specification removed, so it was effectively raised withoutevaluating the increased scaling potential.10/00Essential cooling water heat exchanger performance begins to showperformance results below 30 percent margin.

Prior to this, results were typically to 70 percent margin.5/01Licensee changed chemical vendors for water treatment of spray ponds.3/02Essential Cooling Water 2B performance measured significant negative margin. Condition Report/Disposition Request

2521395 written 5/30/02 to evaluate calculational problems.

Revised 7/10/02 when another calculation error identified.

September 2002 finally concluded -9.5 percent margin was the correct result.

No corrective action for degraded equipment performance.9/02Essential Cooling Water 1B performance measured at 3.6 percent margin.

NoCRDR written.

4A4-24/03Upper zinc specification raised to 1.5 ppm.4/03Essential Cooling Water 3A performance measured at -22 percent margin.

Theheat exchanger was cleaned to restore performance.

Condition Report/Disposition Request

2598216 written.10/03Essential Cooling Water 2A performance measured at 1.0 percent margin.

Bothtrains' heat exchangers cleaned, and routine cleanings were initiated.

11/03Declining performance trend identified in EW heat exchangers in all units. Condition Report/Disposition Request

2653867 written.

Also recognized EDG

intercoolers had experienced similar fouling since 1995.

Zinc precipitation identified as "the primary factor."

Operability determination completed thatshowed this and the 4/03 result in 3A were both capable of performing its safety function under existing conditions (but not under the more limiting design basis conditions).

Compensatory measure established to monitor spray pond temperature to ensure continued operability.1/04Apparent cause for

CRDR 2653867 concluded that dispersant was combiningwith biocide, forming sticky substance that was coating tubes.

2/04Essential Cooling Water 2B perfromance measured at -17.1 percent.

No CRDRwritten.

Operability was considered to be bounded by a previous operabilitydetermination.2/04Self-assessment identified that chemistry changed spray pond chemical evndorand chemicals without evaluating impact.

Condition Report/Disposition Request

2683642 written.3/04Changed dispersant.

No action taken to remove old dispersant through feedingand bleeding the ponds.3/04Mid-cycle cleaning performed for Unit 3 EW heat exchangers.4/04Essential Cooling Water 1A performance measured at 9.9 percent margin.10/04Upper zinc specification doubled to 3.0 ppm.

4A4-311/04White lotion-like substance identified in Unit 2 sand filter, CRDR 2750856written.

It was concluded that this was the old dispersant reacting and being removed by sand filter, which would help remove the remaining old dispersant.10/05Essential Cooling Water 1B performance measured at 1.4 percent margin.10/05Units 2 and 3 entered unscheduled outages.

Essential Cooling Water 2Bperformance measured at 11.1 percent and 3B measured at 19.3 percent margin.11/05Condition Report/Disposition Report

2860763 written to address loss ofperformance of Unit 1 EW heat exchangers.1/06 The operability determination for

CRDR 2860763 completed.1/06Condition Report/Disposition Report

2835865 written to address degradedperformance in all three units' EW heat exchangers in 10/05 tests.

This was the first time that licensee addressed continuing degradation and need for cleaning to assure heat exchangers would remain operable through end of the cycle. Unit 3 was expected to be operable until cleaning in spring outage.

Unit 1 was cleaned in 10/05.

Unit 2 was scheduled for mid-cycle cleanings before summer(5/06).4/06Unit 3 EW heat exchangers cleaned.5/17/06Essential Cooling Water 2B cleaned.

Emergency Diesel Generator 2Bintercooler fouling noted during surveillance run, when air intake temperature jumped up 29F from last test to 145F.5/18/06Unit 2 Train B removed from service for EW cleaning.5/19/06Train B returned to service.

Operability of EDG 2B revisited.

Concluded thatthere was not a reasonable expectation of operability, so it was declaredinoperable.

All other EDGs run to check performance to comply with Technical Specification 3.8.1, Action B.3.2.

Emergency Diesel Generator 2B intercoolerscleaning initiated, CRDR 2896661.5/20/06At 7:40 p.m., engineering issued a white paper that set a 160F limit ofconfidence that EDGs would remain operable if they knew air temp would stay below this level.

4A4-45/23/06Operations lifted compensatory measure of monitoring pond temperatures basedon having completed cleaning in all EW heat exchangers.5/24/06Condition Report/Disposition Request

2897810 written to document significantcondition of fouling caused by incompatible chemicals.

This reiterated the earlier conclusion from

CRDR 2653867 in 2003.

Root cause assigned.5/20/06EDG 2B declared operable following cooler cleaning and surveillance testing. New cleaning method used brushes and significantly improved HX performance over previous method of chemical cleaning.5/22/06Inspectors point out that the promptly operability determination concluded EDGswould remain operable at least up to 160F intercooler outlet temperature, but used existing pond temperature plus the temperature increase observed in last surveillance, rather than the maximum pond temperature.

By doing so, EDG 3B

would have 162F intercooler outlet temp, and EDG 3A would be 158F.

Condition Report/Disposition Request

2897266 written.5/23/06White paper written at 8:42 p.m. to describe a method for adjusting as-foundspray pond temperatures to design basis accident conditions in the spray pond and show that EDGs were currently operable.5/23/06Inspectors pointed out that the discovery checklist was not created for this issuein

CRDR 2896661 as specified in procedures.

Checklist was created, but it haderrors that under-rated the significance.5/24/06Inspectors determine that the licensee's transportability review concluded thatonly EDG 2B was affected, but all EDGs' intercoolers showed elevated delta-T's during previous monthly runs.

Following this, the licensee decided to re-evaluate the operability determination, assess whether both Unit 2 EDGs were inoperableat the same time on May 18, and evaluate the potential for fouling in otherEDGs.

Significant

CRDR 2897810 written on chemical incompatibilities in thespray pond system, based on 6-train impact.5/25/06Conference call between NRC Region IV and licensee about the condition andcause of the EDG 2B issue.

Condition Report/Disposition Request 2899237

written on apparent delay in making an immediate operability determination forthe elevated temperatures seen in EDG 2B on May 17.

Condition Report/Disposition Request

2898120 written to document that EDG intercooler

HX's were not considered in extent of condition review for previous EW heat exchanger fouling.

4A4-55/25/06Licensee began increased frequency testing and trending on all EDGs ascompensatory measure for fouling mechanism.6/22/06NRC questioned the impact of the accumulated sediment in the spray ponds onthe design basis accident capabilities of the pond.

Licensee calculated that itreduced the ability to accept heat rejected, and no longer met the design basisfor the pond.

A revision to the design basis calc was prepared which took credit for rejecting heat to the concrete pond liner which demonstrated operability, butno CRDR or operability determination was written.

Condition Report/DispositionRequest

2906671 was written on 6/29 when the team asked why there was no

CRDR.

IR 05000528/2006011

Text

Dear Mr. Levine:

Enclosure:

w/Attachment:

SUMMARY OF FINDINGS

REPORT DETAILS

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

NRC personnel

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

LIST OF DOCUMENTS REVIEWED

Navigation menu

Search