ML103440449

Document: Apparent Cause Evaluation Template Document: from Robert Montgomery

ML103440449
Person / Time
Site:	Salem
Issue date:	04/05/2010
From:	Montgomery R Public Service Enterprise Group
To:	Office of Information Services
References
FOIA/PA-2010-0334 LS-AA-126-1003, Rev 10
Download: ML103440449 (27)

Text

LS-AA-125-1003 Revision 10 Page 1 of 27 ATTACHMENT 4 Apparent Cause Evaluation Template Page I of 2 ACE

Title:

Auxiliary Feedwater (AF)/Control Air (CA) Buried Piping Degradation CR: 20458568 170109108 Event Date: 04/05/10 Station: Salem Unit 1 Evaluator: Robert Montgomery - Buried Pipe Program Engineer Sponsor: Kerry Colville - Salem Engineering Programs Manager

Description:

On April 5, 2010, during the planned Salem 1R20 Buried Pipe Program inspection of the buried portions of the No. 12 and No. 14 Auxiliary Feedwater (AF) lines,. per procedure ER-AA-5400-1002, it was identified that the excavated section of both lines were externally corroded, and lacked the existence of any protective external coating. These two lines run underground around the outside of the south west side of the Unit 1 Containment Building. There are two (2)

Station Air (SA) and two (2) Control Air (CA) lines that run in the same buried area as the No. 12 and No. 14 AF lines around the Containment Building wall.

During the extent of condition inspection for the upstream portion of the buried AF lines, inside the Fuel Transfer Tube Area (FTTA), a buried section of 1-inch diameter Control Air (CA) piping was identified as having a section of coating damaged and a hole in the pipe itself.

The charter established for this evaluation focuses the Apparent Cause Evaluation (ACE) on the following. two aspects of the event,,1. To evaluate why the No. 12 and the No. 14 buried AF pipe degraded and needed to be replaced, and, 2. To evaluate why one portion of CA piping had a thru wall leak in the FTTA and needed to be replaced.

The subject AF piping is 4-inch diameter, Schedule 80 (Nominal wall of 0.337"),

A106 Gr. B seamless carbon steel, and it is classified as Nuclear 3, Seismic Category I, and Safety Related. The Tmin is calculated as 0.278". The Salem Pipe Specification S-C-MPOO-MGS-0001 SPS 54E and SPS 28, the Arrangement Drawing, and the Pipe Spool Details for the buried portions of the Auxiliary Feedwater piping specify that the buried AF piping should be coated with X-Tru-Coat, (an adhered polyethylene protection system), and Bitumastic 505 (applied at the welded joints). The subject CA piping is 1-inch diameter, Schedule 40 (Nominal wall of 0.133"), A106 Gr. B seamless carbon steel, and it is classified as Nuclear 2, Seismic Category I, and Safety Related, The Salem Pipe Specification S-C-MPOO-MGS-0001 SPS 38A and the Arrangement Drawing specify that the buried CA piping should also be coated with X-Tru-Coat, (an adhered polyethylene protection system), and Bitumastic 505 (applied at the welded joints).

hdommum int" rewrd was deleted in

.07

/ftp

LS-AA-1 25-1003 Revision 10 Page 2 of 27 Work Order 60084266 was planned and scheduled for the 1R20 Outage to perform a Guided Wave and visual inspection of the No. 12 and No. 14 Auxiliary Feedwater lines as they travel underground around the outside of the Unit 1 Containment Building. An excavation was performed along the containment building wall between pipe hangers 12 & 14AFWA-11 and 12 & 14AFWS-13.

The visual examination that was performed, along with the Guided Wave inspection and subsequent Ultrasonic Thickness (UT), inspections identified that both of these excavated lines were, corroded, and, lacked the existence of any external coating. Visually, (and later confirmed with UT), it was apparent that there was pipe wall loss on both AF lines for their entire length when the corrosion products were mechanically removed from the pipes outside diameter (approximately 0.100 inch thick scale / corrosion was removed).

The engineering extent of condition inspection was continued into the FTTA, since both AF lines (and both of the SA and CA lines) are also buried in there.

Visual inspection inside the FTTA revealed the same AF pipe conditions as that of the underground Auxiliary Feedwater piping that was excavated around the Containment Building wall outside. During this visual inspection in the FTTA, the 1-inch No. 12 (B) Control Air Header was identified as having a severely degraded area that was leaking air. This small section of Control Air line appeared to have its tape wrap and plastic coating mechanically damaged some time in the past.

Based on the results of the extensive UT inspections that were performed on the excavated portions of the No. 12 and No. 14 AF lines, and on the condition of the same lines inside the FTTA, it was determined that both lines would be replaced in kind. The inaccessible portions of the AF buried piping in the Fuel Transfer Tube Area were rerouted above ground,, and, over the top of the Fuel Transfer Tube. Additionally, the 1-inch No. 12 (B) Control Air header section of pipe that was damaged inside of the FTTA was also replaced and recoated. All of this work was completed in the 1R20 Outage.

Due to the original design specifications for the buried portions of the No. 12 and No. 14 Auxiliary Feedwater piping requiring protective coating, it was not anticipated that the entire run of both buried lines. would have no signs of external coating at all, for the entire pipe run. The 1-inch Control Air line in the FTTA that was damaged and externally corroded was also not expected, due to the properly coated condition of the same header outside the FTTA.

Failure Mechanism:

The most probable cause for the general corrosion of the No. 12 and the No. 14 buried Auxiliary Feedwater buried pipe is the manual (by mistake) removal of the originally installed protective external coating (X-Tru coat) prior to burial, during original plant construction. Original plant documentation points to the piping being received with X-Tru coat. A single remaining piece found inside of a pipe sleeve shows that the coating was indeed installed, and, prior to burial of the

LS-AA-125-1003 Revision 10 Page 3 of 27 piping outside of this sleeve, the coating was ripped off of the pipe, leaving it exposed to the soil for 36 plus years, allowing the general corrosion to occur on the pipe. The fact that this single piece of X-Tru coat was found in the penetration sleeve supports the most probable cause that the X-tru coat was removed from the auxiliary feedwater piping outside of the penetration sleeve prior to burial.

The most probable cause for No. 12 (B) Control Air header piping section having, a through wall leak in the Fuel Transfer Tube Area is due to the accidental damage to the Bitumastic 505 tape wrap system on this section of pipe from stepping on or climbing on the pipe. Closer examination of this damaged section shows the tape wrap bunched up below the elbow, indicative of it being pushed off of the elbow and pipe. The bunching of the tape wrap at the base of the elbow supports the most probable cause that this section of pipe tape wrap was accidentally stepped on, thus sliding the tape wrap off of the elbow and pipe.

Sequence of Events:

Date Event 08/11/71 Arrangement Drawing issued for construction 01/72 Pipe Spool Details drawn by M.W. Kellogg Co.

06/72 to 12/72 Receipt of Unit 1 Auxiliary Feedwater spools 11/73 Approx. Period of Installation of Unit 1 Aux Feedwater piping 04/73 to 01/74 Picture showing Aux Feed pipe with Yellow Herculite 08/76 Salem Unit 1 Operating License 12/76 Salem Unit I Initial Criticality.

07/77 Salem Unit I Commercial Operation 03/08 Salem Buried Pipe Program procedures issued 02/09 Salem Buried Pipe Program Risk Ranking completed 06/09 Aux Feedwater buried piping ID'd for inspection 12/09 Work Order planned/scheduled for 1R20 vs. On-Line 04/05/10 First Location excavated for G-Wave Collar installation 04/05/10 No. 12 & No. 14 AF lines identified. as being corroded 04/05/10 G-Wave insp. ID'd pipe upstream/downstream also corroded 04/05/10 UT of both lines started - continues through 04/12/10.,

04/07/10 Two additional Locations excavated for extent of condition 04/07/10 Locations 2 & 3 excavations show both No.12 & No.14 lines are corroded 04/09/10 Extent of condition walk down in Fuel Transfer Tube Area 04/07-10/10 Remainder of AF pipe excavated around Unit 1 Containment Building.

04/10/10 Total length of both No. 12 & 14 AF lines corroded - one small area of tar found near, Location 2 dig.

04/10/10 Identified 1"Control Air line had damaged section of coating and hole in pipe.

04/15/10 Replacement Work Order (WO) SCRF'd into outage. (WO written on 04/08/10)

LS-AA-1 25-1003 Revision 10 Page 4 of 27 Causal Factors (CF):

The causal factor technique "Cause and Effect Analysis", from the Root Cause Manual LS-AA-125-1001 Att. 10, was utilized to identify the following two causal factors (CF1 and CF2) and contributing cause below:

CF 1: During original plant construction, the yellow herculite type material that was covering the Auxiliary Feedwater piping prior to burial looked similar to the pipe's normal protective yellow coating, X-Tru-coat, underneath.

Causes:

Apparent Cause 1:

Utilizing the Causal Factor Analysis, LS-AA-125-1003 Att. 1, and the Equipment Apparent Cause Evaluation Guide, LS-AA-1 25-1003 Att. 2, (see attached), the most probable cause identified that the protective pipe coating, X-Tru coat, was mistakenly removed during original installation. Prior to burial of the Auxiliary Feedwater pipe, during original plant construction,. it was covered with a yellow herculite type material, which was wrapped on top of the pipe's original protective coating. There may have been worker confusion during the removal of the herculite that also allowed the yellow X-Tru-coat protective coating underneath to be mistakenly removed as well, leaving bare pipe.

Basis 1: A photograph from Salem Unit 1 construction, taken some time during the period April 1973 to January 1974, shows the Unit 1 Auxiliary Feedwater piping installed prior to burial. In this photo, the AF piping is covered in a yellow herculite type of material, wrapped with what appears to be a black electrical tape. The X-Tru-coat is yellow as well. During removal of the yellow herculite /

black electrical tape covering it may have been misunderstood by workers to remove everything that is yellow on the Auxiliary Feedwater pipe in the burial area. This would be substantiated by the fact that the. piece of X-Tru-coat found in the No. 12 AF pipe wall spool, that had the outside portion ripped off, was left in place because it was already installed in the wall spool, and could not be cut off when the yellow herculite material was removed. The Control Air and Station Air lines that run right next to the subject Auxiliary Feedwater lines also had the yellow herculite covering it, as seen in the same photograph. Both the CA and SA lines have the yellow X-Tru-coat on them, as well as an extra Bitumastic 505 tape wrap system on top, covering the yellow X-Tru coat. This information supports the discussion on accidental removal of everything yellow, since these CA and SA lines had the tape wrap system covering their yellow X-Tru coat, the tape wrap itself, and hence, the yellow X-Tru coat underneath, was not touched on these lines. Since this is a Legacy issue from original construction, Corrective Action No. 1 fixes the degraded piping versus trying to fix a pre-job brief from over 30 years ago.

Cause Code: 4UL - Worker Knowledge or skill deficiency

LS-AA-1 25-1003 Revision 10 Page 5 of 27 Contributing Cause 1a:

Utilizing the Causal Factor Analysis, LS-AA-125-1003 Att. 1, and the Equipment Apparent Cause Evaluation Guide, LS-AA-125-1003 Att. 2, (see attached), the most probable contributing cause identified that the lack of Cathodic Protection for the AF buried piping contributed to the general corrosion of the bare AF piping.

Basis ta: Original plant design for Salem Unit 1 did not include Cathodic Protection for buried piping segments. Cathodic protection is known to protect steel pipes that are buried, even without coating. The absence of any Cathodic Protection for the extended life of buried piping, coincident with the protective X-Tru-coat being removed left the No. 12 and No. 14 AF piping exposed to the soil for 36 plus years, and corroding away. Soil Analysis was performed in the excavation areas during this 1R20 work. As such, the results of the analysis indicated the following: The soil pH in the area ranged from 6.61 to 7.63 - this is a Neutral pH range, with corrosion ratesbeing independent of pH. The soil resistivity in the area ranged from 12,666, ohm-cm to 79,000ohm-cm, which, according to NACE International Classifications for Corrosivity, with the degree of corrosivity being above 10, 000 ohm-cm, it can be considered negligible.

Therefore, with the soil having negligible corrosivity and being close to Neutral pH, this would help explain why the AF pipe corrosion was as slow as it was across approximately 36 years of being in the ground. Cathodic Protection is recognized as an industry best practice by EPRI's Buried Pipe Integrity Group (BPIG),. and it is a performance indicator in the Buried Pipe Program Health Report, as well as being part of the Buried Pipe Risk / Ranking process.

Recommendations No. 1 and. No. 2 were written to present request for Cathodic Protection system for Buried piping to PHC for Salem Units 1 and 2 respectively.

Cause Code:. 3S - Component / part design application deficiency CF 2: The 1-inch Control Air line is in the FTTA, which is dark and not well kept, giving the impression that there is no need to use caution when working in there.

Causes:

Apparent Cause 2:.

Utilizing the Causal Factor Analysis, LS-AA-125-1003 Att. 1, and the Equipment Apparent Cause Evaluation Guide, LS-AA-125-1003 Att. 2, (see attached), the most probable cause for the damage to the 1-inch Control Air pipe is due to a worker stepping on the elbow, pushing. down the protective tape wrap, while performing work in the FTTA. Because this line sticks out of the sand the highest in the FTTA,. a worker may have stepped on the elbow, pushing down the tape wrap and coating, leaving that area of bare pipe exposed.

LS-AA-1 25-1003 Revision 10 Page 6 of 27 Basis 2: Based on initial entry pictures into the FTTA during 1R20, it was evident that this area is not a commonly traversed area, with the Styrofoam insulation laying all over the place, and a rough, sandy floor, this area could easily be conceived as not being important to the plant, thereby creating the appearance of an area where proper plant care and caution for equipment in the area may not be so rigid. This could set up. for a worker not being careful when initially installing the insulation, by standing on this 1-inch pipe, not knowing its significance. Therefore, the most probable cause for the No. 12 (B) Control Air header piping section having. a through wall leak in the Fuel Transfer Tube Area may be due to the accidental movement of the tape wrap system on this section of pipe from being stepped on or climbed on, and pushing the tape wrap. system down, away from the elbow and pipe. The bunching of the tape wrap at the base of the elbow supports this most probable cause. CRCA# 2 fixes the degraded piping in the FTTA prior to 1R20 restart.

Cause Code: 4UL - Worker Knowledge or skill deficiency Organizational &Programmatic Issues:

The following O&P issues were important to this event:

OP 1: Inadequate communication within the organization - since this is a Legacy Issue from original construction, this is an educated guess. as to why the X-Tru-coat was removed,. as stated earlier, possibly with the yellow herculite material cover.

OP 2: Inadequate job skills, work practice, or decision making. This is a Legacy issue regarding the FTTA during original construction, and not having a good work practice when working in the plant, by accidentally stepping on piping.

PSEG Nuclear's current day work practices differ from. the work practices used during original construction; there are processes in,place to help prevent either of these two 0 & P issues from occurring during. a project's installation. These processes include use of the Pre-Job Brief, (HU-AA-1 211), used to ensure. that worker/supervisor communication occurs prior, to starting a job task, a Design Change Package closeout checklist, (CC-AA-103-1001 Form 11), to be used after project installation, the use of the STAR Principal, (Stop, Think, Act, Review), to ensure the worker is mentally engaged with their task, and the use of the 2-Minute Drill Card, to be used at the job site to help with proper task engagement.

Extent of Condition:

Apparent Cause 1: With the Auxiliary. Feedwater pipe being covered with yellow herculite type material prior to burial it may have created confusion during removal, and may have allowed the yellow X-Tru-coat underneath to be mistakenly removed as well, leaving bare pipe. During the 1R20 Outage the two (2) Station Air Lines and two (2) Control Air Lines were also examined for

LS-AA-125-1003 Revision 10 Page 7 of 27 missing/removed coating. The Picture reviewed from the period April 1973 to January 1974, which shows the Unit 1 Auxiliary Feedwater piping installed prior to burial covered in the yellow herculite material also shows the SA and CA lines covered in the yellow herculite material as well. These four (4) lines that run right next to the AF lines all had theii coating, applied and intact during the 1R20 inspection. The Salem Unit 2 AF lines were also checked in the more easily accessible FTTA of Unit 2,. after some minor sand removal / excavation (Notification 20457987 created for Unit 2 inspection). Inspections were performed and the original coating was present and intact. Additionally, an Op Eval 70109482 was performed for Unit 2 AF. The, Buried Pipe program includes all buried piping types for inspection, (i.e. coated or non-coated). Since this was a one time Legacy Issue regarding the yellow X-Tru-coat being removed along with the yellow herculite, the current extent of condition satisfactorily covers the lines in the same immediate area as the Unit 1 Aux Feedwater piping (i.e. CA and SA lines), and the same piping at Unit 2. Additional inspection of the Unit 2 Auxiliary Feedwater lines, in addition to the extent of condition inspections already performed during the 1R20 Outage timeframe (Notification 20457987),

are currently scheduled for 2R1 8 to include an excavation along the Unit 2 Containment Building, similar to the area excavated on the Unit 1 Containment Building, (Work Order 60084161). With the Auxiliary Feedwater system being classified as Safety Related, the extent of condition also included a review of any other Safety Related buried piping. In the Buried Pipe program the only other Safety Related buried piping systems include the Control Air piping and the Nuclear Service Water Headers. As discussed earlier, the Control Air piping was looked at in the existing trench with the Auxiliary Feedwater piping, with no issues noted (except as noted in the FTTA). The Safety Related Service Water piping is comprised of Pre-Stressed Concrete Pipe (PCCP), and not considered similar to the subject AF piping's carbon steel material.

Apparent Cause 2:

The 1-inch Control Air pipe sticks out of the sand the highest in the FTTA, and during original plant construction, a worker stepped on the elbow, pushing down the tape wrap, leaving: that area of pipe. exposed. The No. 11 CA header is physically located right next to the No. 12 CA header that was damaged. This pipe coating was inspected and found to be intact. Outside of the FTTA the. CA lines, (11. and 12 Headers) were clearly visible in the excavated area around the Containment Building. The majority of this Tape-Wrap style of pipe coating was in very good condition considering its age under ground, however, some damage has occurred that required repair. Notification 20458761 was written to document and repair. Additionally, the CA headers at Unit 2, in the FTTA, uncovered during the Unit 2 AF extent of condition inspection (Notification 20457987), were also inspected as an extent of condition, and found to be in very good condition, with no torn or removed sections of coating. As discussed in the extent of condition for Apparent Cause 1 above, the only other Safety Related buried piping system, in addition to this apparent cause's CA and AF piping, is the Nuclear Service Water headers, which are comprised of PCCP, and not considered similar to the subject CA piping's carbon steel material.

4 4' LS-AA-1 25-1003 Revision 10 Page 8 of 27 Extent of Cause:

Apparent Cause 1; With the, Auxiliary, Feedwater pipe being covered with yellow herculite type material prior to burial it may have created confusion during removal, and may have allowed the yellow X-Tru-coat underneath to be mistakenly removed as well, leaving bare pipe.. The extent of cause for this apparent cause identifies that the removal of the yellow herculite and the X-Tru-coat was most likely related to a poor pre-job briefing for that job. The SA and CA piping located in the same, supports as the AF pipe were found to be properly coated, thus the extent of cause was limited to the.AF piping, Also, the only other Safety Related buried piping system, in addition to the AF and CA systems, is the Nuclear Service Water Headers. These headers are not comprised of the X-Tru-coated carbon steel, and therefore not considered similar to this apparent cause extent of cause.

Apparent Cause 2: The 1-inch Control Air pipe sticks out of'the sand the highest in the FTTA,. and during original plant construction, a worker stepped on the elbow, pushing down the tape wrap, leaving that area of pipe exposed. The extent of cause for this apparent cause identifies that the FTTA for Unit 1 provided an area where there could have been a tendency to perform work with less regard than working in the plant area itself. The Unit 2 FTTA was in a slightly better shape upon visual examination. These are the only areas of either plant that have sandy floors, giving them an unfinished look, which further indicates that there could have been a tendency to perform work with less regard than working in the plant area itself. However, inspection in the Unit 2 FTTA found the CA lines to be in very good condition, with no torn or removed sections of coating. As stated above, the only other Safety Related buried piping system, in addition to the AF and CA systems, is the Nuclear Service. Water Headers.

These headers are not comprised of the X-Tru-coated carbon steel, and therefore not considered similar to this apparent cause extent of cause.

Previous Events:

LS-AA-1 25-1003 Revision 10 Page 9 of 27 AFW and condensate water lines in the area of this leak were uncovered, inspected and found to be correctly coated in accordance with the original installation specification for buried piping.

Use:

This failure is similar to the pipe degradation found at Salem. The cause identified was very similar to this Salem issue with AF; the original pipe installation was not coated as required per installation specifications. This review was used to evaluate if there should be newer corrective actions or extent of conditions than those already implemented. Surry had relocated the line, and inspected the other piping in the area of leak. Part of the extent of condition review for Salem's issue included review of the coating in the immediate area of the piping degradation, (i.e. CA and SA lines), and at Unit 2. Our extent of condition is similar to those of Surry's for this OE.

(b)(4)

Use:

This failure describes an air pipe in a very wet environment, thereby setting up for pitting and wet environment corrosion. Although there is no mention of coating, this is a sample of bare piping in the soil that failed. This review was reviewed as an example, and to add supportive information to the fact that bare pipe in the ground in a wet environment will corrode.

(b)(4)

Use:

This failure describes incorrect pipe material used for burial, and not including the line in the Buried Pipe program. This OE was reviewed based on it's applicability to material use in a buried application. The corrective actions identify that Brunswick redesigned their piping, and had to add it to their program. This OE is in line with Salem's redesign of the AF pipe, although for a different reason than

LS-AA-125-1003.

Revision 10 Page 10 of 27 Brunswick's improper material, it still highlights the use of a pipe re-run as a corrective action. The Salern AF pipe was already in the Buried Pipe program.

Use:

This OE included the identification of non-functional cathodic protection, in addition to general corrosion due to coating damage. While this Salem issue was a result of missing coating, the use of Cathodic Protection was identified as a Contributing Cause. Use of this OE for support of Cathodic Protection system corrective action.

Use:

This OE was reviewed for its backfill being identified as an issue, This OE was reviewed for its identification of the importance of proper backfill. The ECP

LS-AA-1 25-1003 Revision 10 Page 11 of 27 80101381 was written to replace the AF buried piping outside, as it runs around the Containment Building wall, and called for approved flowable fill to be used as backfill, versus dirt. No actions are take from this OE since the ECP utilized a backfill that was appropriate for the newly installed AF pipe/coating, without rocks or other coating-damaging items.

Corrective Actions (CRCA):

Corrective Action #1 -Action: 60089561 For Apparent Cause 1 - The No. 12 and No. 14 Auxiliary Feedwater Lines were corroded due to improper coating removal during original construction. Replace both lines and properly coat prior to reburial.

Owner: Pat Ayers - LRE Due: COMPLETE Both the No. 12 and 14 AF lines were replaced in kind going around the Unit 1 Containment Building, and coated with CERAMALLOY CL+, prior to burial with flowable fill.

Corrective Action #2 -.Action: 60089575 For Apparent Cause 2 - The No. 12 Control Air header in the FTTA is damaged and needs replacement.

Owner: R. Swartzwelder Due: COMPLETE The subject damaged No. 12 Control Air Header section in the Unit 1 FTTA has been replaced in kind, and recoated.

Recommendations (ACIT):

Recommendation #1 - Action: 70106150 Op. 10 For Contributing Cause Ia - The Salem Unit 1 Buried Piping systems have never been protected by Cathodic Protection. ACIT to present Cathodic Protection system implementation recommendations to Plant Management - PHC. This was a previously self identified action for the Buried Pipe program.

Owner: R. Montgomery Due: 08/29/10 Recommendation #2 - Action: 70106150 Op. 10 For Contributing Cause Ia - The Salem Unit 2 Buried Piping systems have never been protected by Cathodic Protection. ACIT to present Cathodic Protection system implementation recommendations to Plant Management - PHC. This was a previously self identified action for the Buried Pipe program.

Owner: R. Montgomery Due: 08/29/10 Supporting Information:

On April 5, 2010, during the planned 1R20 Buried Pipe Program inspection of the buried portions of the No. 12 and No. 14 Auxiliary Feedwater (AF) lines that run around the outside of south west side of the Unit 1 Containment Building, it was identified that the excavated section of both lines were corroded, and appeared to lack the existence of any external coating. During the extent of condition inspection for the upstream portion of buried AF piping in the Fuel Transfer Tube

LS-AA-1 25-1003 Revision 10 Page 12 of 27 Area (FTTA) a section of buried 1-inch diameter Control Air (CA) piping was identified as having a section of coating damaged and a hole in the pipe itself.

The charter established for this evaluation focuses the Apparent Cause Evaluation (ACE) on the following two aspects of the event, 1. To evaluate why the No. 12 and the No. 14 buried AF pipe degraded and needed to be replaced, and, 2. To evaluate why one portion of CA piping had a thru wall leak in the FTTA and needed to be replaced.

Based on the Buried Pipe Program's (BPP) Risk/Ranking of buried piping systems the Auxiliary Feedwater system's two buried trains, (No. 12 & No. 14),

were ranked High (based on their Total Risk Ranking). An Outage Work Order was planned and scheduled to perform a Guided Wave (G-Wave) inspection and visual inspection of these two lines as they travel underground, around the outside of the Unit 1 Containment Building.

The No. 12 and No. 14 AF lines originate in the Auxiliary building, downstream of the Auxiliary Feedwater pumps. Both of these lines leave the Mechanical Penetration area of the Auxiliary Building through a wall. penetration, and into the sand filled Fuel Transfer Tube Area (FTTA). The FTTA is an enclosed area, approximately 5-feet wide, located between the Containment Building Wall and the Fuel Handling Building Wall. The FTTA houses the Fuel, Transfer Tube, used for moving fuel during refueling. The two AF lines then exit the FTTA, going through wall penetrations underground, to the outside area, hugging the Unit 1 Containment building wall for approximately 90' around, before going into the Outer Penetration Area (building), and tying into their respective Feedwater/Steam Generator trains. The subject AF piping is 4-inch diameter, Schedule 80 (Nominal wall of 0.337"), Al 06 Gr. B seamless carbon steel. It is classified as Nuclear 3, Seismic Category I. The Tmin is calculated as 0.278".

Per the Salem Pipe Specification, the Arrangement Drawings, and Pipe Spool Details, the specified coating for the buried AF piping is X-Tru-Coat, an adhered polyethylene protection system, and Bitumastic,, which was specified per drawings and pipe specifications to be applied at the welded joints.

Work Order 60084266 was planned and scheduled for the 1R20 Outage to perform a Guided Wave (G-Wave) and visual inspection of the No. 12 and No. 14 AF lines as they travel underground around the, outside, area of the Unit 1 Containment Building. In order to. provide the best location for application of the Guided Wave collar, and to perform a direct visual inspection of the pipe, an excavation was performed midway along the containment building wall between pipe hangers 12 & 14AFWA-11 and 12 & 14AFWS-13. This midway excavation point was made so the G-Wave signal could. be sent in both, directions, (upstream and downstream of the collar), capturing the condition of as much of both buried trains as possible. The excavation area was approximately 5'- 6' wide, and approximately 5' deep. In addition to the original excavation area, a second G-Wave collar was applied inside of the Outer Penetration building, to get a reading on both lines, just before they enter the building. This second location was

LS-AA-125-1003 Revision 10 Page 13 of 27 planned due to the added coverage that could be obtained on the buried piping by taking readings inside the building. However, due to the piping geometries inside, only the No. 14 line was able to be inspected with the G-Wave collar, On April 5, 2010, prior to application of the G-Wave collar in the excavation area it was identified that both of the lines that were uncovered were corroded, and appeared to lack the existence of any external coating. In an effort to determine the extent of the corrosion,. both pipes were cleaned of their corrosion surface for a length of 12" to facilitate installation of the G-Wave collar, which must be applied to solid pipe (i.e. No corrosion).. Visually, it was apparentthat there was pipe wall loss when the corrosion product was removed (approximately 0.100").

This was confirmed with Ultrasonic Thickness (UT). on the collar installation area of both lines. Notification 20456999 was initiated for the corrosion conditions.

Although the Guided Wave inspection technique is only used as a screening tool, and designed to determine corrosion locations on coated piping, the G-Wave technicians were requested to provide an indication of the piping wall condition for remaining buried piping in both directions (upstream and downstream) from the installed G-Wave collar. The G-Wave inspection identified that the pipe in both directions was also. corroded. The technicians were then requested to indicate the location of the worst corroded areas on both lines. The worst pipe wall loss areas were located upstream of the G-Wave collar, approximately 13' North around the Containment Building wall for both lines. On April 7, 2010, an additional excavation was performed for extent of condition at this second location. A third excavation was also performed in order to get yet another direct visual inspection of both lines, and to apply the G-Wave collar on the No, 12 line just before it enters the Outer Penetration Building, since it could not be inspected from inside, as mentioned above.

The second and third excavations revealed that both lines in these areas also had the same external general corrosion as that identified in the first excavation area, with no evidence of coating. A small portion of a coal tar like product, approximately 9 inches in length and 7 inches in the circumference was found near the second excavation site. This piece of coating was in the shape of 4 inch piping and conformed to that same pipe surface.

On April 7 - 8, 2010, excavation of the remainder of the Auxiliary Feedwater piping around Unit 1 Containment Building was performed. This excavation revealed that the total lengths of both the No.12 and No. 14 AF lines were externally corroded. Visual inspections of this piping after excavation showed no recognizable coating system. In an effort to support Past Operability (70108698

- Op. 20), the newly excavated piping was cleaned of its corrosion covering with needle guns. The piping was then gridded, and UT was performed on both lines, to determine the lowest wall thickness., Over 17,000 UT readings were recorded between both trains. The minimum wall thickness for the No. 12 line showed a 55 percent loss (0.152 inches remaining wall), For the No. 14 line, the greatest loss was approximately 78 percent (0.077 inches remaining wall). This low area on the No. 14 line was discovered underneath a pipe support bracket. For the

LS-AA-125-1003 Revision 10 Page 14 of 27 No. 14 line 13 percent of the UT readings were below the Tmin of 0.278". For the No. 12 line only 2 percent of the UT readings were below Tmin.

During the total excavation of the No. 12 and No. 14 AF lines, there were two (2)

Station Air (SA) lines and two (2) Control Air CA) lines that were also uncovered.

All six (6) of these lines, (2 Aux Feed, 2 CA, & 2 SA) travel around the:

Containment Building, buried, on the same pipe support hangers. Both the Control Air lines and the Station Air Lines had coating throughout their entire run around the Containment Building wall. Coating. on these lines consisted of X-Tru Coat (an adhered polyethylene coating) and a Tape Wrap system. Minor coating damage areas were noted on the CA and SA lines, and appeared to mostly be damage from the excavation effort.

An extent of condition inspection was continued into the FTTA, since both AF lines (and both of the SA and CA lines) are also buried in there. Notification 20457854 was written for the extent of condition inspection. Although these lines in the FTTA are in.a covered area, they are indeed buried under sand, and extend down, under the Fuel Transfer Tube. Visual inspection inside the FTTA revealed the same AF pipe conditions as that of the underground Auxiliary Feedwater piping that was excavated around the Containment Building wall outside. On April 10, 2010, during this visual inspection in the FTTA,, one of the 1-inch Control Air lines (the B Header or 12 Header) was identified as having a severely degraded area that was leaking air. This section of Control Air line appeared to have its tape wrap and plastic coating mechanically damaged some time in the past. There was evidence that the tape wrap was pushed off of the pipe section just below an elbow, as if someone had accidentally stepped on it, and pushed a section of coating off,. thereby allowing the bare carbon steel pipe to corrode. The initial walk down of the FTTA showed a poorly maintained. area, with Styrofoam wall insulation scattered about in this small, enclosed area.

Notification 20457869 was written to document this CA line corroded area, and to have it replaced in kind, with coating applied.

Based on results of the, extensive UT inspection performed on the buried portions of the No. 12 and No. 14 AF lines, it was determined that both lines would be replaced in kind, and coated prior to re-burial. ECP 80101381 was written to replace the AF buried piping outside, as it runs around the Containment Building wall. DCP 80101382 was written to replace the inaccessible sections of AFW buried piping near and below the Fuel Transfer Tube in kind. However, they were run above ground in the FTTA area, and over the top of the Fuel Transfer Tube. Therefore, the AF buried piping for Unit 1 was replaced.

During the removal of the No. 12 and No. 14 AF lines during replacement, a small section (approximately 12-inches in length) of X-Tru coat was found inside of the No. 12 AF line wall penetration as it passes into the Outer Penetration building, through penetration W-19401-008. This was the only recognizable existence of any coating product, other than the small section of a tar like substance.

LS-AA-125-1003 Revision 10 Page 15 of 27 Salem Arrangement Drawing 207483 was released for construction on 08/11/71.

Revision 0 to this original drawing includes the following note - NOTE: "Where Buried Pipe requires an anchor guide or other attachment, that section of Plastic coat is to be removed prior to welding. Protection of that section is to be done in field by application of two coats of Bitumastic 505 applied cold." Additionally, Pipe Spool Details were drawn by M.W. Kellogg Company in January 1972.

These spool. details, VTD's 157755 through 157763 for the. No. 12 AF line from the inside FTTA to the Outer Penetration and VTD's 157778 to 157786 for the No. 14 AF line from inside the FTTA to the Outer Penetration, and. all state that the piping spools are to be X-Tru coated. Quality Control (QC) Reports (QC Reports 3624, 3840, and 4673) for the subject Auxiliary Feedwater spools being received on site during the June 1972 to December 1972 timeframe all indicate that the pipe was received matching the documentation that the spools were shipped with, the M.W. Kellogg spool details indicating the X-Tru coat requirement.

A review of an original revision to the Salem Piping Specification for Auxiliary Feedwater, Revision R-4d from October 13, 1972, reveal the same original piping note for underground piping, which states the following: "For protection of underground piping in the yard see page 28f of Piping Schedule No. 28 of this specification. "X-Tru-coat" may be used."

PSE&G Lab Report #61565-J, written January 30, 1974 is a documentation of pipe coating inspections that were made during 1973. This report idenitifies the Auxiliary Feedwater piping located on the West side of No. 1 Containment. It states that there are 200 feet of 4-inch piping, and that is was inspected on 12 73. However, under the status, there is no indication given, where as the other pipe inspections listed give a status of OK or Unknown. An accompanying memorandum, dated March 11, 1974, states that "You will notice from the "Status Column" that certain coated / lined piping was found to contain Holidays for which the repair status is unknown. We request you review these unknown status situations to assure repairs have. been made to linings or coatings. Please advise us on your findings.". The undetermined status of the Auxiliary Feedwater piping coating on this coating inspection report provides a possibility of the coating having holidays that have been partially removed, and the coating thus removed, thereby giving a "nothing" in the Status column of the report.

A photograph from Salem Unit 1 construction, taken some time during the period April 1973 to January 1974, shows the Unit 1 Auxiliary Feedwater piping installed prior to burial. In this photo, the piping is covered in a yellow herculite type of material, wrapped with what appears to be a black electrical tape. This type of yellow herculite / black electrical tape cover was used to cover other piping during this timeframe as evidenced by other photos (Image 236) from M:\Photos\SALEM OUTAGES & ARCHIVE\SALEM CONSTRUCTION\FOLDER 7 - APRIL 1973-JAN 1974. The X-Tru-coat is also yellow in color. A probable cause can be made that during removal of the yellow herculite / black electrical

LS-AA-1 25-1003 Revision 10 Page 16 of 27 tape covering it may have been misunderstood by workers to remove everything that is-yellow on the Auxiliary Feedwater pipe in the burial area., This would be substantiated by the fact that the piece of X-Tru-coat found in the No. 12 AF pipe wall spool, that had the outside portion ripped off, was left in place because it was already installed in the, wall penetration, and could not be cut off when the yellow herculite material was removed. Image 315 on M:\Photos\SALEM OUTAGES & ARCHIVE\SALEM CONSTRUCTION\FOLDER 7 - APRIL 1973-JAN 1974 shows the yellow covering on the Auxiliary Feedwater piping before it enters the Outer Penetration. It is noted that the Station Air and Control Air lines are also visible in this same picture, and are also wrapped with the yellow covering. However, both the SA and CA lines have an obvious, clearly different type coating, (a tape wrap), then what was on the Auxiliary Feedwater piping originally (yellow X-Tru-coat). Thus, the most probable cause of the No. 12 and the No. 14 buried Auxiliary Feedwater buried pipe degradation is the manual (by mistake) removal of the originally installed protective external coating (X-Tru coat) prior to burial, during original plant construction.

Finally, the original plant design for Salem Unit 1 did not include Cathodic Protection for buried piping segments. Cathodic protection is known to protect steel pipes that are buried, even with out coating. The absence of any Cathodic Protection for the extended life of buried piping, coincident with the protective X-Tru-coat being removed left the No. 12 and No. 14 AF piping exposed to the soil for 36 plus years, and corroding away. Soil Analysis was performed in the excavation areas during this 1R20 work. As such, the results of the analysis indicated the following: The soil pH in the area ranged from 6.61 to 7.63 - this is a Neutral pH range, with corrosion rates being independent of pH. The soil resistivity in the area ranged from 12,666 ohm-cm to 79,000ohm-cm, which, according to NACE International Classifications for Corrosivity, with the degree of corrosivity being above 10, 000 ohm-cm, it can be considered negligible.

Therefore, with the soil having negligible corrosivity and being close to Neutral pH, this would help explain why the AF pipe corrosion was as slow as it was across approximately 36 years of being in the ground.

Regarding the 1-inch Control Air, this line is Schedule 40 (Nominal wall of 0.133"), A106 Gr. B seamless carbon steel, and it is classified as Nuclear 2, Seismic Category I. The Salem Pipe Specification S-C-MPOO-MGS-0001 SPS 38A and the Arrangement Drawings 218278 and 218277specify that the buried CA piping should also be coated with X-Tru-Coat, and Bitumastic 505 (applied at the welded joints). The reason for the application of the additional tape wrap system could not be identified. Based on initial entry pictures into the FTTA during 1R20, it was evident that this area is not a commonly traversed area, with the Styrofoam insulation laying all over the place, and a rough, sandy floor, this area could easily be conceived as not being important to the plant, thereby creating the appearance of an area where. proper plant care and caution for equipment in the area may not be so rigid. This could set up for a worker not being careful when initially installing the insulation, by standing on this 1-inch pipe, not knowing its significance. Therefore, the most probable cause for. the.

LS-AA-1 25-1003 Revision 10 Page 17 of 27 No. 12 (B) Control Air header piping section having a through wall leak in the Fuel Transfer Tube Area may be due to the accidental movement of the protective coating and tape wrap system on this section of pipe from stepping on or climbing on the pipe.

Finally, an OPEX review consisted of utilizing the INPO website, and performing searches on applicable words for this evaluation, such as "buried", "Auxiliary Feedwater", "leak", "underground", etc. The following OE showed relevance to this evaluation for review, and are documented in the OPEX Review section entitled "Previous Events" above..

LS-AA-1 25-1003 Revision 10

LS-AA-125-1003 Revision 10 (b)(4)

LS-AA-1 25-1003 Revision 10 Page 20 of 27 Causes: The Root Cause is the installation specification in effect at the time of plant construction. There is evidence, that sections of the pipe coating were damaged by rocks that were present in the backfill for the CST lines. The pipe coating material is fiber-based saturated. with coal-tar. The material is then applied to the pipe. Since it is a fiber, the. coating is susceptible todamage from the various size rocks found in close proximity to and in some cases, up against the pipes themselves.

The water table in the area of the leak is between eight to ten feet with the pipe elevation at approximately ten feet. The backfill specification did not specify the use of clean sand and gravel under the pipe that would have limited the wicking of the ground water to the soil surrounding the pipes. This kept the soil in the area moist, and at times wet. These soil conditions would find its way into defects in the coating causing corrosion external to the pipe.

For this ACE's subject Auxiliary Feedwater buried pipe degradation finding, 0E31083 was issued. It was originally issued as OE30927, and then updated.

The OE topic is Corrosion on Buried, Externally Coated, Safety Related Pipe.

The OE's abstract reads as follows: During a planned Buried Pipe program inspection of the auxiliary feedwater (AFW) system, the station identified general corrosion on portions of externally coated carbon steel underground pipe. Extent of condition excavation and non-destructive examination identified areas that did not meet initial acceptance criteria and required replacement. The likely cause is failure to apply the pipe protective coating per design during original construction.

References:

1. P&ID - 205236 Sh. 1 (Aux Feedwater)

2. P&ID - 205243 Sh. 1 (Control Air)

3. Arrangment Drawing - 207483 (Aux Feedwater)

4. Arrangment Drawing - 207482 (Aux Feedwater)

5. Arrangment Drawing - 218278 (Control Air)

6. Arrangment Drawing - 218277 (Control Air)

7. Construction Isometric Drawing -AF-1-3A Sh. 1

8. VTD's - Pipe Spools - 157755 to 157763 & 157779 to 157786

9. Salem Pipe Spec. - S-C-MPOO-MGS-0001 (SPS 54, SPS 28, SPS38A)

10. Quality Control (QC) Receipt Inspection Reports-June 1972 to December 1972

a. QC 3624

b. QC 3840 C. QC 4673

LS-AA-1 25-1003 Revision 10 Page 21 of 27

11. Photos - Located on M:\Shared\Buried Piping\ACE 2010 Aux Feed

12. 10/13/72 - Internal Memorandum from A. D'Ambra to D. D'Fiore regarding revisions to Pipe Schedules 1S16 & 1S54.

13. 03/11/74 - Internal Memorandum from H. C. Warden & A, D'Ambra to C.

Bradish discussing PSE&G Lab Report #61565-J Regarding Pipe Coating Inspection dated 01/30/74.

14. 05/17/10 - Salem Soil Study Results - Tables 2 - 4, showing results of Maplewood Testing. Services Soil analysis that was performed per WO 60084266 Op. 60.

15. Technical Evaluation 70108698 - OE, Past Operability, Max Press.

16. Work Order 60089561 - Replaces Auxiliary Feedwater piping

17. Work Order 60089575 - replaces 1-inch Control Air section

18. Work Order 60084266 - Original Aux Feedwater G-WaveNisual Insp.

19. Notification 20458554. - Control air in FTTA

20. Notification 20457854 - UT AF piping in FTTA

21. Notification 20456999 - Aux Feedwater pipe corroded in Excavation area

22. Notification 20457987 - Unit 2 Aux Feed extent of condition inspection 23 Op Eval 70109482 - performed for Unit 2 Auxiliary Feedwater piping

24. ECP 80101381 - written to replace the Aux Feedwater buried piping outside, as it runs around the Containment Building wall.

25. DCP 80101382 - written to replace the inaccessible sections of Aux Feedwater buried piping near and below the Fuel Transfer Tube in kind.

26. Work Order 60084161 - Unit 2 2R1 8 Buried Pipe inspection / excavation of the buried No. 22 and No. 24 Auxiliary Feedwater piping along the Containment Building Wall.

LS-AA-125-1003 Revision 10 Page 22 of 27 LS-AA-125-1003 - ATTACHMENT 2 Question 1:

Run to failure (RTF) classification Check Is the component incorrectly classified as Critical, Non-Critical or Run to Failure per MA-AA-716-2110?

No. The Auxiliary Feedwater buried piping and the Control Air buried piping are correctly classified as non-RTF per Att. 4 of MA-AA-716-21 10 and is therefore considered critical for this evaluation.

No for Cathodic Protection, since it was not part of the original plant design.

Question 2:

PM/PDM Review Has the past PM/PDM not been performed in accordance with the PCM template? (For PM's performed on this component.)

No. The Auxiliary Feedwater buried piping and the Control Air buried piping are not part of a PM program. The Auxiliary Feedwater buried piping and the Control Air Buried Piping are part of the Buried Pipe Program, and as such, this was the first required inspection.

No for Cathodic Protection, since it was not part of the original plant design.

Question 3:

Maintenance Performance Assessment Is there a deficiency with the performance of the most recently performed maintenance?

No. Maintenance is not required on the Auxiliary. Feedwater buried piping or on the Control Air buried piping.

No for Cathodic Protection, since it was not part of the original plant design.

Question 4:

Performance Monitoring Assessment Has system/component monitoring been deficient in identifying normal or abnormal equipment degradation?

No. Performance monitoring is not required on the Auxiliary Feedwater buried piping or the Control Air Buried Piping.

No for Cathodic Protection, since it was not part of the original plant design.

LS-AA-1 25-1003 Revision 10 Page 23 of 27 Question 5:

Operating Experience Review Is there a deficiency in how past operating experience (OPEX) applicable to this component has been addressed?

No. There was no past OE that was improperly addressed. Although there is OE concerning buried piping failures or non-coated buried piping, the Buried Pipe Program itself was recently established to inspect buried piping based on industry experiences.

No for Cathodic Protection, since itwas not part of the original plant design.

However, industry recommendations TODAY emphasize the benefits of installing a Cathodic Protection system for buried pipe protection.

Question 6:

PCM Template Review Is there a deficiency in any PCM template applicable to this component?

No. There is no deficiency in the PCM template for the Auxiliary Feedwater buried piping or the Control Air buried piping. There are no PCM templates for these lines.

No for Cathodic Protection, since it was not part of the original plant design.

Question 7:

Operational Performance Review Are the operating procedures or practices for this component inappropriate or unacceptable?

No. The Auxiliary Feedwater buried piping and the Control Air. buried piping are not controlled by operating procedures.

No for Cathodic Protection, since it was not part of the original plant design.

Question 8:

Maintenance Practice Review Are there problems with the maintenance practices, behaviors or training for this component?

Yes. The Auxiliary Feedwater buried piping and the Control Air buried piping were installed over 30 years ago, when worker knowledge or skill deficiencies may have contributed to the removal of the yellow herculite and the yellow X-Tru-coat on the Auxiliary Feedwater piping, as well as not truly having the knowledge

LS-AA-125-1003 Revision 10 Page 24 of 27 base to not stand on plant equipment in the isolated and unfinished FTTA. Both of these practices occurred during original construction, and were not considered to be prevalent today. Proper Pre-Job Briefs, the Pocket Safety Manual, and increased worker training are the norm today.

For the lack of Cathodic Protection for the Auxiliary Feedwater piping, the answer to this question is No. This was not a maintenance practice.

Question 9:

Design Review Is the design configuration for this component incorrect?

No. The Auxiliary Feedwater buried piping and the Control Air buried piping design configurations are correct, and original plant documentation required these lines to be coated when buried. Review of the evidence gathered, (ie. QC Receipt inspections, field observations) indicated that the Auxiliary Feedwater piping was coated as required by design, and then had the coating accidentally removed, thus allowing the general corrosion to set up on the piping. The, Control Air buried piping was correctly coated as design required, however, it was accidentally damaged at some point early in the life of the plant, and this allowed corrosion to set up.

Yes, for Cathodic Protection, since it was not part of the original plant design.

Question 10:

Manufacture/Vendor Quality Check Is there a concern with the quality of parts, shipping or handling?

No. Review of the evidence gathered, (ie.. QC Receipt inspections, field observations) indicated that the Auxiliary Feedwater piping was coated as required, by design, and then had the coating accidentally removed, thus allowing, the general corrosion to set up on the piping. The Control Air buried piping was correctly coated as design required, however, it was accidentally damaged at some point early in the life of the plant, and this allowed corrosion to set up.

No for Cathodic Protection, since it was not part of the original plant design.

Question 11:

Problem/Issue Management Review Have previous issues not been adequately addressed including but not limited to aging, obsolescence, chronic problem, scheduling, or business planning?

No. The Buried Pipe Program was recently established, and inspection of the Auxiliary Feedwater buried piping, and subsequently, the Control Air buried

LS-AA-125-1003 Revision 10 Page 25 of 27 piping in the same location identified the general corrosion, caused by accidental removal of the coating. This was the first Buried Pipe program inspection of these lines.

No for Cathodic Protection, since it was not part of the original plant design.

Question 12:

Unknown or Different Cause Did the equipment fail due to an unknown cause or other cause than listed in steps 1 through 11 above?

No. The most probable cause for the general corrosion of the No. 12 and the No.

14 buried Auxiliary Feedwater buried pipe is the manual (by mistake) removal of the originally installed protective external coating (X-Tru coat) prior to burial, during original plant construction.

The most probable cause for No. 12 (B) Control Air header piping section having a through wall leak in the Fuel Transfer Tube Area is due to the accidental damage to the Bitumastic 505 tape wrap system on this section of pipe from being stepped on or being climbed on.

No for Cathodic Protection, since it was not part of the original plant design.

Question 9 above was answered yes.

LS-AA-125-1003 Revision 10 Page 26 of 27 (b)(4)

LS-AA-125-1003 Revision 10 Page 27 of 27 ATTACHMENT 5 Apparent Cause Evaluation Quality Checklist Page 2 of 2 (b)(4)