ML110040409

Draft, Regarding Review of Root Cause Analysis and Actions Addressing the Underground Pipe Damage to the Condensate Storage Tank Return Line

ML110040409
Person / Time
Site:	Oconee
Issue date:	12/23/2010
From:	NRC/RGN-II
To:
References
FOIA/PA-2011-0021
Download: ML110040409 (4)

Text

.4 Review of Root Cause Analysis and Actions Addressing the Underground Pipe Damage to the Condensate Storage Tank Return Line

a. Inspection Scope The inspectors reviewed the Root Cause Analysis (RCA) for damaged underground piping to the condensate storage tank that was identified on February 15, 2009. The inspectors reviewed the report and pertinent documents and interviewed station personnel to determine if the RCA adequately addressed the issue and provided for adequate corrective actions.

Background

Entergy excavated two sections of condensate storaige tank (CST) piping in an area between the CST and the Auxiliary Feed Pump B'iild;ing (AFPB) in October and November 2008 to address recommendations dqdumented in an Indian Point Independent Safety Evaluation (ISE) Report,,, d July 31, 2008. l-1tle first excavation location, Entergy determined that five areas-,of piping required coating repair, that is, the pipe coating was damaged or missing. Ultrasonic pipe thickness measurements of these areas provided information that the r"alls(remined at or near their original manufactured thickness. In the second excavation,t*here was one area of pipe with degraded coating and some cdrrugated piping th6athad some coating material missing from the top of a few crests indicatingthat excavation may have damaged the coating.

Based on the observations and rep*airs made, Entergy concluded that the pipes did not exhibit significant pipe degradation that wouid warrant further inspection of these same locations in the future.

During the excavations in 2008, Entergy employees noted water coming up through the CST returnie pipe sleeve V here the buried piping comes into the AFPB through the floor slab. Entergy did not enter this condition into their corrective action program.

Entergy performed a chemnistry analysis of the water and determined that there was no hydrazine in the water and therefore gidid ot come from the CST piping. Additionally, theI pH ofite water was measured as 7.5'2, not consistent with CST water which is

, typically 8.5 - 9.5 due to the presence of hydrazine. The chemistry sample also detected 700 pCi/L of tritium. Entergy determined that the amount of tritium in the

• sample was consistent withlground water readings. Specifically, tritium levels for water taken from manhole #5 (within 20 feet of the pipe sleeve) on October 17, 2008 (the last recorded sample for mnhole #5) resulted in approximately 800 pCi/L. Samples taken from the CST on November 12, 2009 resulted in 800 pCi/L as well. Entergy concluded the leakagewas groundwater due to the open excavations and heavy rains.

On February 15, 2009 at approximately 3:00 p.m. an Operations watchstander again noticed water coming up through the CST return pipe sleeve in the AFPB floor. Entergy took chemistry samples of the water and determined that 54 parts per billion (ppb) hydrazine was present, indicating that the water was from the condensate system. At 1:30 a.m. on February 16, Entergy declared the CST inoperable. By February 18, Entergy determined that the leak rate from the CST was approximately 17 gallons per minute (gpm), began excavating the probable location of the leak, and determined that the leak was coming from the CST return piping and not the CST supply pipe. On February 19, the excavation of the CST piping exposed the CST return pipe and the leak in the pipe. Entergy engineers removed the pipe coating from the surrounding 133/

areas and performed ultrasonic testing of the pipe walls and determined that the pipe was structurally sound and the metal loss causing the leaks was very localized to areas where the pipe coating was apparently damaged. Entergy, in addition to the hole identified in the horizontal run of the return pipe, also discovered areas of metal loss on a pipe elbow in the same line due to pipe coating degradation. Entergy cut out the damaged piping and welded a new pipe portion in place on February 20. The CST was declared operable at 6:56 a.m. on February 21.

Entergy performed a root cause analysis to determine to cause of the CST pipe leak.

Entergy also contracted a vendor to analyze the portion of piping that was cut out to determine the failure mode. Entergy identified during excavation that many rocks were used in the backfill of the pipe. The rocks ranged in size from small pebbles to 8-inch rocks. A review of the backfill specifications at the time of plant construction did not provide detailed information on what size rocks could be used for the fill in this area.

This particular area had a concrete slab poured on top of the fill and was not meant to be a load bearing surface and thus did not have strong requirements for the type of fill to be used. The area was primarily to be used as a walkway. The vendor analysis of the pipe concluded that the leak was caused by external corrosion in areas where the pipe coating was degraded. Although the exact type of external corrosion could not be definitively determined, the observations support the corrosion was the result of exposure to a range of ground water characteristics and/or microbiologically influenced corrosion. The root cause documented that the large rocks found in the backfill likely damaged the pipe coating during installation of the pipe or backfill allowing the corrosion mechanisms described above to act on the bare metal surfaces. The pipes were found to be in good condition where the coating was intact.

The root cause team examined Indian Point's capabilityIo track water usage to determine whether it was reasonable for the station to identify the leakage prior to February 15.;Entergy determined that it was not feasible for Operators to detect the leakage from~ mnain condenser hotwell level indications or CST level indications because the rate of leakage (10-17g'pm)*as too small, to detect with the installed instrumentation. Furthermore, Enterg*described that steam generator blowdown (100 gpmp), auxiliary steam heating during the winter (20 gpm), and condensate system leakage (10'-20 gpm) contribute to losses of inventory in the hotwell that require replenishment from the drain collecting tank and/or the CST. Water used for the secondary plant and Unit 2 CST is supplied from the city water system, processed through a water conditioning unit, then sent to a Unit 1 CST. From the Unit 1 CST, water can be sent to the drain collecting tank (DCT) or the Unit 2 CST. Makeup to the hotwell is normally from the DCT, but deviations between the makeup rate from the DCT and the loss- rate from the secondary plant is made up from the unit 2 CST inventory.

Control room operators monitor the decrease in CST level periodically and make up to the CST as necessary to avoid receiving a low-level alarm.

The RCA described the Indian Point Buried Piping and Tank Inspection Program (BPTIP) that is under development. The program began development in late 2007 under a corporate-wide initiative to develop these programs at all Entergy sites. The program identifies underground pipes at the site and assigns an impact assessment level based on safety impact, public risk impact, and economic impact of failure. The high impact systems are also corrosion risk assessed by considering: soil resistivity, drainage, material, and existing coatings or cathodic protection. The scheduling of the

examination of the pipes is determined by the impact and corrosion risk assessments.

The site plans to have the program fully developed by the end of 2009.

The RCA developed a corrective action plan to address the root and contributing causes of the pipe degradation. Some of the corrective actions are listed below:

• Update the buried piping backfill and excavation specification for the site

• Implement improved inspection techniques for buried piping

• Evaluate the need for cathodic protection systems for select buried piping systems

• Evaluate the need for a drainage system near the CST pipes

• Evaluate the use of existing monitoring wells for buried pipe and tank leaks for early leak detection

• Perform pipe inspections in the near future on the following pipes at specific locations:

• Condensate return line to CST (2 different locations) 0 Condensate supply line from CST (l2different Iocaitios)

• Service water line 408 (2 different locations)

• Remainder of underground pipes to be inspected in accordance with Buried Piping Program schedule

b. Assessment and Observations The inspectors reviewed activities surrounding the CST return line leak and the associated root cause analysis arnd determined that Entergy adequately responded to the leak and performed or created'adequate corrective actions to address the issue.

The inspectors also concluded that there was*noperformance deficiency associated with Entergy's inability to d!etect the pipe and leakage prior to it manifesting

,dgradation itself on February 15, 2009 as leakage throigh a ground-level pipe sleeve. Additional observations and assessmenftfollows beow.

The inspectors reviewed plant drawings and the backfill specifications provided by the engineer/architect at the tirrme *fplant construction and determined that the drawings and specification did no detail the type of backfill required and specifically did not rocks from beiused

• prohibit in the backfill.

The inspectors verified that Entergy performed required testing in accordance with the American Society of Mechanical Engineers Boiler & Pressure Vessel (ASME BPV)

Code Section Xl and 10 CFR 50.55a. Section Xl requires that pipes similar to the CST returnline be tested 3 times over the 10-year inspection interval by verifying no obstructiondo flow trorugh the pipe while in service. Entergy meets this requirement by verifying unobstructed flow by monitoring auxiliary feed water (AFW) pump recirculation flow parameter during quarterly Technical Specification Surveillance Tests. The AFW recirculation linies feed water back through the CST return line back to the CST.

The inspectors identified that Entergy's' Operations personnel, on a daily basis, log the processed water sent from the city water system to the Unit 1 CSTs such that they know the amount of water used daily by secondary plant operations. The inspectors plotted the data and identified a sharp increase in water consumed by Unit 2 from September 2008 to November 2008 (from 5282960 gallons per month to 7246210 gallons per month). In the months October through March, water usage for Unit 2 increases. In 2006, the winter month average usage was approximately 5.9 million gallons per month,

in 2007 it was approximately 6.2 million gallons per month, and in 2008 it was approximately 6.8 million gallons per month. Entergy did not recognize the increase in usage. When questioned, Entergy communicated that the log reading is used to verify billing from the water conditioning vendor and not to trend and track the usage of water in the plant. Furthermore, Entergy concluded that because there were many loads using the water source from the Unit 1 CST it would be very difficult to identify a leak of 10-20 gpm using the installed instrumentation. The Unit 1 CST supplies water to the following loads: Unit 2 CST, Unit 2 hotwell, primary water storage tank, make-up to the Unit 1 deaerator, Unit 1 CST continuous chemistry sample flow, water factory continuous flow for chemistry, main generator stator head tank turbine hall closed cooling system head tank, and the instrument air closed coliing head tank.

The station procedure governing the recording of operatorlgs is OAP-017, "Plant Surveillance and Operator Rounds," that describes its purpose'."...to ensure plant operations and equipment are systematically observed to determine if equipment and instrumentation are operating properly. Methods described enable identification of abnormalities immediately through recognition of data deviations and over the long term by recognition of developing data/log reading trends." The procedure recuires that watch-standers be cognizant of the configuation and condition of equipmientand systems and when system parameters are trending out of, or have exceeded, normal bands the control room supervisor is notified. hipese6 bands are defined with maximum and/or minimum values or normal operating bands in the logs so that the watchstander can compare log readings with them. For the case of the treated city water to the Unit 1 CSTs, no such bands are defined, as Opeations logs thisnumber to aid in the verification of billing information fromn the water treatment ndor.

The inspectors concluded that Entergy adequately assessed conditions surrounding the water leakage identified in the AFPB fin 2008; however, the inspectors concluded that the condition s[ould have been entered inito the corrective action program (CAP) with corrective actions, operability determinations, and trending data documented. Because the chemistry samples at the time documented pH, hydrazine, and tritium levels consistent with thatof ground water,itis ,not likely that Entergy could have detected a leak i uridergroun ing with this data alone.. The inspectors concluded that, a;lthough it w*ld, have been a good practice, it was not required to monitor input streams to important tanks described earlier to detect abnormal makeup rates that may rindicate leakage. 'Entergy's procedure OAP-017 describes that the station will trend parameters that it lousing dmethodologies described in the procedure. However, Entergy did not provid normal bands, or maximum/minimum values for the log reading associated with the aamount of processed city water being sent to the Unit 1 CST. The inspectors deterrmin*d that this represented a situation where Entergy was utilizing Operationswatch-standers to log data for business purposes vice monitoring plant operations because it was convenient, and this particular parameter was not intended to be monitored per OAP-01 7. Because Entergy did not trend makeup water usage into Unit 2 via the Unit 1 CST it was not likely that Entergy could have detected the underground pipe leakage based on water consumption. The inspectors noted that Entergy's root cause did not address the aspects of the 2008 leakage not being documented in the CAP and also did not address the ability to monitor water usage through the water treatment facility and existing plant logs.