ML20154M625

Rev 0 to, Engineering Rept,Station Svc Water Sys Coating Removal Project Evaluation

ML20154M625
Person / Time
Site:	Comanche Peak
Issue date:	09/21/1988
From:	Guldemond W, Ronaldo Jenkins, Nace L TEXAS UTILITIES ELECTRIC CO. (TU ELECTRIC)
To:
Shared Package
ML20154M624	List: ML20154M619 ML20154M625
References
ER-ME-19, ER-ME-19-R, ER-ME-19-R00, NUDOCS 8809290047
Download: ML20154M625 (61)

Text

_ _ _ _ _ _

TU ELECTRIC 1 CPSES l

l ENGINEERING REPORT STATION SERVICE WATER SYSTEM C0ATING REMOVAL PROJECT EVALUATION ER-ME-19 REVISION 0 SEPTEMBER 21, 1988 PREPARED BY  % 8 W. G. tM. ND ('- --

PREPARED BY , /_ YL/ xx R. T' JEN(/NS APPROVED BY L. D. NAct 000 ado $K O 45 A PNU

Enclosure to T?'.. 88699 September 23, 1988 Page 1 of 46 TABLE OF CONTENTS Page Title 2 Executive Summary 4 I Introduction 8  !! Process Evaluation 8 A. Background "

8 1. Problem Characterization 8 a. Activities Prior to September 1985 11 b, Problem Characterization Af ter September 1985 13 2. Evaluation of Alternatives and Selection of Solution 14 3. Background Conclusions 15 B. Project Implementation 15 1. Prerequisite Decisions ,

15 a. Work Group Selection 15 b. Acceptance Criteria Definition 16 c. Codes and Standards Applicability 17 2. Procurement 18 a. Requisitions 21 b. Contract 22 c. Conclusion 24 3. Development of Procedures and Criteria 24 a. TU Electric Activities 26 b. Contractor Activities 27 c. Procedures and Criteria Conclusions 28 4. Solution Implementation 28 a. Mobilization 29 b. Process Tests 32 c. Coating Removal 35 d. Nuclear Regulatory Commission Interface 38 5. Project implementation Conclusio!-

39  !!!. Ter.hnical Evaluation 39 A. Introduction 40 B.. Assessment of Cause and Extent 40 1 Methodology 40 2. Cause 40 3. Extent of Condition 42 4. Corrective Action 45 IV. Summary and Conclusions '

Appendix A - Chronology Figures (1 through 8)

P

, . _ . -.,._.,_.,m _

Enclosure to TXX-88699 September 23, 1988 Page 2 of 46 EXECUTIVE SUf41ARY The Comanche Peak Station Service Water System (SSWS) is a safety-related ASME Code Class 3 cooling water system. During normal operation it provides cooling to the Component Cooling Water Heat Exchangers. During postulated accidents, it cools certain equipment necessary to mitigate the consequences of those accidents.

The SSWS is constructed of carbon steel and stainless steel. Carbon steel piping four inches in diameter and larger was originally coated with Plasite 7122, an epoxy-based protective coating. The coating was a) plied to rt tard  ;

general corrosion, thereby prolonging the service life of tie system. ,

in September 1985, significant deterioration of the Plasite coating was identified. This discovery set in motion a major effort to determine the  ;

extent of coating failure and corrosion in system piping. Extensive  !

examinations of the SSWS were conducted, the results of which were factored l into engineering analyses and a comprehensive plan of action to improve the  ;

reliability and longevity of the system, it was determined that the coating  !

was failing due to a combination of corrosion and erosion and, because the '

coating was improperly applied. Further, it was determined that patches of coating could detach from pipe surfaces and cause unacceptable blockage of SSWS flow to the Component Cooling Water Heat Exchangers and other vital equipment. While corrosion damage to system piping did not represent an immediate threat to system reliability, action was needed to monitor and  :

retard its progression, l Based on this information, the decision was made to remove the coating from i SSWS piping 10 inches and larger in diameter. Dry sandblasting was chosen as j the meth.od of removal. A specialty contractor was selected to remove the t coating and a number of tests of the sandblasting process were performed.

Coating removal began in early April 1988 and was completed in early July 1988. During system refill following coating removal, a hole approximately one-half by three-eighths inch was discovered in one o' the 10 inch supply lines to one of the Emergency Diesel Generators. A r u very plan was prepared consisting of two parts. The first part consisted of a technical evaluation  !

of the hole and other damage in SSWS piping and those actions necessary to return the SSWS to its design condition. The second part consisted of an .

evaluation of the administrativ] controls applied to the sandblasting process, !

Based on the technical evaluation, it was determined that the hole in the pipe i occurred when the sandblast apparatus stalled for several minutes in one j location. The stall occurred due to unanticipated buildup of blasting grit in  ;

the piping. The extent of damage to the system due to sandblaster malfunction  ;

was limited to a small portion of 10 inch piping in one train of SSWS.  :

Stalling was identified during the early stages of coating removal and action was taken which successfully resolved the problem. This was confirmed by i reviewing video tapes taken of the interior surfaces of the piping following i sandblasting.  ;

Enclosure to TXX-88699 September 23, 1988 Page 3 of 46 Three sections of pipe, which appeared to have the most extensive damage, were removed from the system and examined to assess the seriousness of the damage.

Based on the examinations it was determined that, with the exception of the damage associated with the hole, the pipe was acceptable from a wall thickness standpoint.

A safety evaluation was performed which showed that the damage, including the hole, would not have compromised the safety function of SSWS, even had the damage remained uncorrected.

With regard to the evaluation of administrative controls applied to the sandblasting process, it was determined that those activities leading up to and including selection of sandblasting were properly supported with approp.'iate technical and management resources. Appropriate decisions were made and concern was expressed for the impact of sandblasting on pipe wall thickness. Pre-implementation activities appropriately identified the tasks which needed to be accomplished and the organizations responsible for completing those tasks.

A number of shortcomings occurred during the implementation phase of the project. Most notable in this regard was that the pre-qualification sandblast i

tests did not encompass the conditions under which the process would be performed in t m plant. A number r,f other shortcomings contributed directly to problems encountered during coating removal. The most significant of these were not adequately establishing the relationship between the coating removal activities; the TV Electric Quality Assurance Program; and the groups i participating in the project during the procurement process, and the lack of oversite on the par t of task management.

The extent of these problems is limited to those cases where specialty contractors not having an a) proved quality assurance program are employed to perform quality related wor ( under the auspices of the TV Electne Quality l Assurance Program. Therefore, corrective actions, including procedural i enhancements and training, focus on such activities.

j 4

l i

Enclosure to 1XX-88699 September 23, 1988 Page 4 of 46

1. INTRODUCTION During refill of the Unit 1 Station Service Water System (SSWS) following an cutage for removal of coating from portions of system piping and complet.on of other activities, a hole was discovered in a 10 inch section of Train A pipe.

Preliminary investigation revealed that the hole was caused by the l sandblasting precess utilized to remove the coating from the pipe. A recovery plan was developed consisting of two parts. One part involved an evaluation of the administrative processes and activities associated with coating removal. The other part involved a technical assessment of the impact of sandblasting on SSWS piping and a course of action to address discrepant conditions. This report documents the results of the recovery plan effort.

The Comanche Peak Station Service Water System (SSWS) is an ASME Code Class 3 cooling water system. During normal operations, it provides cooling to the Component Cooling Water Heat Exchanger (CChHX). During postulated accidents, it serves to cool certain vital equipment necessary to mitigate the consequences of those accidents. The system consists of two trains per unit.

Each train contains a pump and associated piping and equipment necessary to deliver cooling water to the associated train of safety related equipment.

These portions of the system are safety related. Additional piping and equipment is provided for non-safety support functions. Safety-related  ;

components serviced by SSWS include the Emergency Diesel Generators, the '

Safety Injection Pumps, the Centrifugal Charging Pumps, the Containment Spray Pumps, and the Component Cooling Water Heat Exchangers. In addition, the SSWS serves as the backup water supp.y for the Auxiliary Feedwater System.

The SSWS is constructed of carbon steel and stainless steel. Piping up to 3 1/2 inches in diameter is made of 304 or 316 stainless steel. Piping larger than 3 1/2 inches in diameter is either carbon steel or st, nless steel.

Carbon steel piping in the flow path to critical safety com onents r is 10 inches and larger in diameter. In accordance with design specifications,  ;

carbon steel piping four inches in diameter and larger was supplied by the l vendor with an internal coating of Plasite 7122, an epoxy-based protective coating. During system construction, Plasite 7122 was applied to joints and other internal surfaces not coated by the vendor on 4 inch and larger diameter carbon steel piping. The coating was installed to retard corrosion and erosion of the pipe, thereby prolonging its service life. As such, the coating served no safety function.

In September 1985, CPSES set in motion a major effort to evaluate tne material condition of the SSWS coatings and piping following the discovery of system protective coating adherence problem > and a number of system leaks. Extensive examinations of the SSWS were performed, the results of which were factored i into engineering analyses and a comprehensive plan of at ion to improve the l reliability and longevity of the SSWS. This plan consin ed of near-term ,

actions focusing on coating deterioration issues and the susceptibility of the '

SSWS to corrosion / erosion leaks, and longer term actions consisting of monitoring and testing activities to provide a 5 asis for deciding whether more i extensive system modifications were warranted.

O o

Enclosure to TXX-88699 September 23, 1988 Page S of 46 From the outset, this effort was guided by two questions. First and most importantly, could the problems identified compromise the safety function of the SSWS or other equipment? Second, to what extent could the identified problems result in loss of plant production capacity?

It was determined that two conditions could exist which might represeN safety concerns. First, the potential existed for sheet mode failure of .',w coating.

In this mode, patches of coating might detach from the pipe surface and block cooling flow to critical components. Second, localized accelerated corrosion I

could occur. It was determined that small holes had developed in the coatings. These holes allowed system fluid to come in contact with the base piping material and establish corrosion cells of various sizes. The corrosion rate at these locations was such that, if it were allowed to continue, local areas of pipe thinning could occur, i

In the near tera, three options existed to address these issues: 1) remove the existing coatings; 2) replace the coated pipe; 3) repair / replace the coatings.

Coating removal would eliminate the potential for sheet mode failure and

. accelerated local corrosicn. It would, however, expose the system to general corrosion. Although no general corrosion allowance had been assumed in system design based on coating application, considerable pipe wall margin existed in the as-built pipe. Thus, coating removal would require a complementary engineering evaluation to determine system expected life, and a monitoring '

program to determine the rate of system corrosion / erosion. Pipe replacement would accomplish the same results as coating removal, but at potentially

, greater cost and schedule impact. Coating repair / replacement would not

, address either issue entirely and would be difficult to implement on inaccessible pipe. Consequently, the decision was made to remove the coating.

Given that coating removal of the type ensisioned for the SSWS was unique, and i that other near-term actions to reduce SSWS usceptibility to

corrosion / erosion required the same plant conditions to implement as coating J

removal, a task force was established to develop and coordinate these actions, and to manage the SSWS outage necessary to implement them. The task force

, consisted of a Project Manager and representatives from Comanche Peak Engineering (CPE), Stone and Webster (SWEC) acting as the lead engineering group, Construction, Projects, Results Engineering (Operations), and Startup.

! A number of coating removal Utions were considered. Ultimately, a combination 4

of automatic and manual dry sandblas+.ing was selected and approved by Senior Management as the best method available to remove the coatings without causing

damage to the piping. A specialty contractor was selected and a series of tests of the automatic sandblasting process was conducted. Coating removal in Unit I was implemented during the period April-July,1988. Verification of i

Coating removal was made by direct Visual examination of accessible pipe and review of video tapes made of the interior of inaccessible pipe.

! As noted above, a hole in a 10 inch pipe was discovered following coating

! removal. This resulted in the development of a two part recovery plan. The

{ first part involved an evaluation of the administrative pror, esses and l activities associated with coating removal. The second part involved a l technical assessment of the impact of sandblasting on SSWS piping and a course of action to address discrepant conditions.

'. Enclosure to TXX-88699 September 23, 1988 Page 6 of 9 ,

i The objective established for the first part of the recovery plan was to determine whether the coating removal process was controlled in a manner commensurate with its importance to safety. First, a review was conducted of those activities leading up to and including the decision to use sandblasting as the method of coating removal. The purposes of this review were to assess the adequacy of those actions leading to the decision to remove the coatings, to determine why sandblasting was chosen, and to determine what constraints were to be applied to its use. Second, a review was conducted of those l

activitiet associated with the proceduralization of the sandblasting process j and the oversight to be provided to determine if identified constraints and 1 existing programmatic requirements were adequately incorporated. Finally, a review was conducted of the implementation ard oversight that occurred in the l field to determine if the identified procedures and programmatic requirements were adhered to. These reviews were conducted using available documentation i and by conducting interviews of personnel involved in the various phases of i the project.  !

At the outset of this evaluation, it was recognized that the SSWS coating 4

removal task involved a number of unique attributes. First, the coating t

removal itself was a first-of-a kind activity. Pre-packaged and qualified [

coating removal processes did not txist. Consequently, the task involved i considerable developmental work in an area where little experience existed in the nuclear industry. Second, the work involved removal of non-safety  !

coatings from a safety-related system. ThPd, the SSWS piping falls under the auspices of the ASME Boiler and Pressure Vesul Code and, as such, is subject 1 to ASME Section XI requirements. While sandblasting in general has not been i considered by the industry to be a process subject to ASME Code requirements, 1

it was recognized that sandblasting could remove metal from the pipe and thereby reduce pipe wall thickness. Fourth, this was a one time, narrowly

focused, short duration task. Finally, the decision was made to utilize a specialty contractor to perform the coating removal task. The chosen '

. contractor did not have an approved quality assurance prograrr. or a qualified

process. This combination of attributes is not typically encountered at CPSES. Consequently, a unique relationship needed to be defined between the l activities of the contractor and the TV Electric procedures and quality i assurance requirements. i j The objectives of the second part of the recovery plan, technical assessment, 1 were to determine the extent of damage to SSWS pipe caused by sandblasting, to l l assess the safety significance of the damage, and to develop a program to r

! resolve the problems caused by the damage. {

It was recognizcd that all of the activities associated with assessment of the  !

j identified conditions and actions taken to .orrect any discrepant conditions must be conducted in accordance with the requirements of the CPSES QA Program and ASME XI procedures.  !

l l

i l t

l

'. Enclosure to TXX-88699 September 23, 1988 Page 7 of 46 The methodology employed was to remove and replace a three foot section of pipe containing the hole. Video tapes taken for verification of coating removal were reviewed to determine the extent of potential damage to the remainder of the inaccessible 10 inch pipe. Based on this review, three spools appearing to have the worst damage were removed from the system, cut into test specimens, and examined to quantify the extent of the damage. This information was then used to train personnel and assess the significance of damage to the remainder of the 10 inch pipe. Based on this information, a safety evaluation of the damage was performed to determine the impact of the damage on the ability of SSWS to perform its safety function. Finally, the results of all these efforts were fattored into a corrective / preventive action program.

'. Enclosure to TXX-88699 September 23, 1988 Page 8 of 46

11. PROCESS EVALVATION II.A Background II.A.1 Problem Characterization II.A.1.a Activities Prior to September 1985 '

Problems with the Plasite 7122 coatings which had been applied to the SSWS were first identified in 1980. During disassembly of flanged connections, the coating was observed to flake off. It was also identified that the coating applied on site had not been procured or applied under the quality assurance program. Based on this information, concern developed that sufficient quantities of the coating could detach from the piping surfaces to plug heat exchangers and strainers. This, in turn, could compromise the ability of the SSWS to accomplish its safety function of removing heat from certain safety related equipment.

The problems were reported to the NRC (SOAR CP-80-07. TXX-3218) and the information was forwarded to the Architect / Engineer, Gibbs and Hill (G&H), for ;

evaluation (TWX-12,523). The G&H response (GTT-7277) concluded that the problems identified could not lead to a condition adverse to safety. Two i bases were stated for this conclusion. First, it was not considered probable i

that enough coating could break away at one time to result in unacceptable blockage of both CC&"iXs simultaneously. Second, the coatings were applied to j reduce corrosion. Corrosion was not considered to be a safety issue.

Excessive corrosion would be detectable by routine inservice inspection.

Two additional pieces of information were provided in the G&H response.

First, detached coatin? carried by system flow to the SSWS screen wash nozzles would be broken up and flushed out by the differential pressure across the l nozzles. Second, degradation caused by small pieces of coating flaking off j the piping und slowly accumulating in the CCWHXs would be detectable by monitoring system thermal performance.

1 Following receipt of this informatun, three actions were taken. First, the  !

NRC was notified that the issue was not reportable under 10CFR50.55(e)

(TXX-3229). Second, the piping specifications were revised to clearly indicate that the SSWS coating was not safety-related. Finally, the FSAR was amended to clearly indicate that the coating was not quality assured.

I During this evaluation of the G&H response, information was identified which supported the conclusion drawn in 1980 that large scale coating failure was  :

not credible. First, large scale coating failures nad not been experienced at CPSES, nor was there information avaliable to suggest that such failures had, f been experienced elsewhere in the industry. Second, the Plasite 7122 coating was designed for use in the SSWS environment. Third, the coating had been I applied by the piping supplier and in the field in accordance with approved J procedures.  ;

Specifically, procedures have existed at CPSES for the application of Pla5ite i

7122 since at least 1977. A number of these procedures were examined, including procedure 35-1195-PFP-9 Revisions 0, 1, and 2 dated February 25, 1977, December 18, 1980, and April 12, 1984, respectively. While

'. l

'. Enclosure to TXX-88699 September 23, 1988 i Page 9 of 46 this procedure notes that the coating is non-safety related and does not include Quality Control (QC) hold points, it does provide specific instructions for application of the coatings including film thickness and curing time requirements; post-application verifications for coating thickness and voids, and requires the verifications to be witnessed by a Test Engineer; requires that the verifications be documented; and requires the documentation to be permanently retained. A number of these records were examined which establish that required verifications were performed and identified problems were reviewed and dispositioned.

Similarly, it was determined that information was available to support the position that excessive system corrosion would be detectable by inservice inspection (ISI). Specifically, 10CFR50.55a required operating plants to have an ISI program in accordance with Section XI of the ASME Boiler and Pressure Vessel Code. For Class 3 piping the Code requires periodic visual inspection (external) of system piping to detect leakage. While this program was not required to be implemented at the +ime G&H performed its evaluation, implementation is required as a coidition of an operating license. In addition, the system was periodically opened for maintenance, thereby allowing detection of discrepant conditions.

The revisions to the piping specifications, the FSAR amendment, and closure of the report with the NRC were appropriate based on the fact that the coatings were applied for corrosion resistance only, and the conclusion that coating failures did not represent a safety issue.

During this review of the G&H evaluation, however, two other observations were made. First, the evaluation did not explicitly address the lack of formal quality controls for procurement and installation of the coating. Second, the initial problem was based on observed flaking of coating from flange fcces.

Additional inspections were not performed to determine the extent of the problem.

The lack of formal quality controls carries with it three potential problems.

First, absent other controls, the quality of the coating applied is indeterminate. Second, the coating applied might not be compatible with the material to which it was applied. Finally, if formal quality controls were required but were not applied, a quality program problem could be indicated.

As noted above, s .? coating application at CPSES was procedurally controlled and verified. This, combined with the fact that the type of coating was defined by the piping specificatior.s, provides nece,,ary confidence as to the quality and compatibility of the applied coatings. Based on a review of the piping specifications, it was concluded that no clear requirement for fonnal quality controls existed. This was clarified by the subsequent specification and FSAR changes which confirmed that formal quality controls were not required. Based on these changes, the NRC closed this issue in Inspection Report 50-445/8109; 50-446/8109.

The fact that the G&H evaluation concluded that coating problems of the type encountered did not represent a significant problem provided a basis for not performing additional inspections.

Enclosure to TXX-88699 September 23, 1988 Page 10 of 46 Overall, it is concluded that the G&H evaluation was adequate given the information available at the time, and provided an adequate basis for the subsequent specification and FSAR changes.

The issue of significant coating failure was next officially raised in October 1983. INP0 SER 68-83 documented degradation and delamination of Plasite 7122 H in the cooling water system at the Palo Verde Nuclear Plant, irdicating that the coating debris could have plugged heat exchangers/ strainers. The causes cited for the failures were excessive film thickness and inadequate curing.

In accordance with established procedures, the Plant Evaluation (PE) Group determined the SER to be applicable to CPSES, and recommended that responsible groups make changes as necessary to application processes to address the identified causes.

TV Electric Engineering was requested to respond to PE concerning the SER.

Based on its review, Engineering concluded that no action was required because: 1) the issue had been formally addressed earlier by G&H and it had been concluded that coating failures could not adversely impact plant safety;

2) existing specifications provided adequate controls of coating application; and 3) the coating was non-quality assured.

The SER was followed in March 1985 by NRC Information Notice (IEN) 85-24.

This IEN restated the situation at Palo Verde. In addition, the IEN noted that the Plasite failures did not challenge the adequacy of Plasite as a cooling water system coating provided that application and curing were  ;

correctly completed. It further noted that Plasite had been successfully used at other nuclear facilities. PE closed the IEN on April 19, 1985, based on the evaluation performed in response to INP0 SER 68-83.

In retrospect, it appears that the Engineering evaluation did not fully

{ consider that one of the bases for the G&H conclusion was that there was no credible failure mechanism which could lead to sufficient quantities of coating being released into the flow stream simultaneously to cause a )roblem.

The Palo Verde experience showed that there was a credible failure meclanism.

On the other hand, no such failures had been identified at CPSES and the application process had been the subject of both in-process controls and post application verifications. There was no reason to believe that the coating had been improperly applied, i

Notwithstanding concerns raised as to the adequacy of the 1980 G&H evaluation, no further action need be taken since the Plasite coatings have been removed from the carbon steel piping in the flow path to critical safety equipment.

In addition, no generic action is needed other than the ongoing and extensive review of desiga and hardware performed under the Comanche Peak Response Team program and the Corrective Action Program.

In discussions with PE, it was determined that wher. INPO SER 68 83 was processed, PE was not formally tasked with evaluating the responses to its requests. Since that time, the role and responsibility of PE have evolved significantly, and PE has become much more aggressive with respect to the quality of information it receives in response to inquiries. An INP0 "Good Practice" was received as a result of the PE program. Additionally, PE has undertaken a comprehensive review of previous evaluations of generic comunications for which it had and has responsibility, the objective of which l

j

Enclosure to TXX-88699 September 23, 1988 Page 11 of 46 is to ensure that those evaluations meet its current standards. It should also be noted that generic communications were factored into the Corrective Action Program. Overall, the need for thorough evaluation o' industry reinforced by the issuance of a Nuclear operating Engineering experience was and Operations recently(NED) procedure NE0 2.29, on this subject.

Group II.A.1.b Problem Characterization Af ter September 1985 In September 1985 deterioration of the Plasite 7122 coating was noted in the SSWS outlet from the Unit 1 Train A CCWHX during scheduled maintenance. This condition was formally documented in a plant Problem Report (PR85-532) to ensure that a thorough evaluation would be conducted. The initial concern with regard to the coating deterioration was loss of corrosion protection.

Given that the 1980 G&H evaluation concluded that corrosion was not a safety concern, the deterioration was not viewed as reportable to the NRC. However, a work request was issud which required additional inspection of SSWS Train A, 24 and 30 inch pipe to detennine the extent of deterioration.

A number of leaks in system welds were subsequently identified and documented on a Problem Report (PR85-699) in December 1985. Initially, it was believed that there might be a causal link between the coating deterioration and the leaks. This combination of problems prompted a timely notification to the NRC in January 1986. On February 24, 1986, a letter (TXX-4711) was sent to the NRC confirming the January notification but noting that weld and coating

~

problems were not related. On April 11, 1986, a followup letter was sent to the NRC (TXX-4762) which concluded that the weld and coating problems were reportable.

These discoveries set off a series of events which ultimately resultea in a detailed characterization of the coating and other concerns with SSWS piping.

Between September 1985 and November 1987, CPSES, with the assistance of Ebasco, Stone and Webster (SWEC), and others, performed a series of inspections and evaluations on both trains of Unit 1 SSWS. One of these inspections was that required for PR85-532. These activities included visual observations by CPSES personnel, SWEC personnel, an Ebasco coatings expert, dnd the Plasite vendor; laboratory analyses of samples of the Plasite by the vendor; coating thickness measurements; location and characterization of areas of erosion, corrosion, and pitting; ultrasonic testing of high stress points and areas of extensive pitting; and repairs to both coating and piping.

These activities were properly documented and controlled. One example of the controls applied to these activities involved the pipe crawls performed by Ebasco, in August 1987. These pipe crawls were performed in accordance with Field Verification Method CPE-EB-FVM-ME-099 titled, "Engineering Walkdown of Coating / Corrosion of Equipment and Components Subject to Immersion Service".

This FVM describes the means by which quantitative and qualitative data on the coating and corrosion conditions of immersion service areas may be obtained through visual inspection, photographs, dimensional and depth measurement of corrosion pits, ultrasonic measurement of pipe wall thickness where significant corrosion is present, and collection of corrosion samples. The task description for the Ebasco Protective Coatings Program, CPE-TD-EB-008 Revision 3, was revised to reference applicable qualification and training requirements for personnel performing the pipe crawls and to reference the FVM for the performance of the pipe crawls.

t '.

'. Enclosure to TXX-88699 September 23, 1988 Page 12 of 46 It was noted during review of certain coating repair activities conducted in 1906 and 1987 using a Belzona coating material, that a limited number of ultrasonic pipe wall thickness measurements were made without first removing the paint on the exterior of the pipe. This can introduce an error in the results obtained. Because of this error, the affected areas were measured again with satisf actory results. It was subsequently verified that all ultrasonic data used to determine the acceptability of SSWS pipe wall thickness was obtained with the paint removed from the exterior surface of the pipe.

Information continued to be compiled as a. result of the ongoing investigations. Interim reports were sent to the NRC on a periodic basis. In July 1987, SWEC issued its first draft report on corrosion and coating  ;

deterioration in the SSWS. This report concluded that the Plasite 7122 i coating at CPSES was vulnerable to sheet mode failure. Coating failures were ,

judged to be occurring by two mechanisms: erosion / corrosion in areas of high turbulence and degradation due to improper installation of the coating.

Additional conclusions presented in the draft report included the following:

o Any coating system is temporary and will require monitoring, o The depth of corrosion at the weld joints of the underground pipe needs to be better defined to determine if adequate minimum wall is available, o While additional confirmation is required, it appears that there is no excessive corrosion occurring in the carbon steel pipe that would affect its function over the next two years, o Microbiologically induced corrosion is likely to occur in stainless steel ,

sections of the SSWS. ,

The draft report recommended more extensive inspections of the 24 and 30 inch pipe to quantify the extent of corrosion, particularly in weld zones on buried pipe; repair of coating defects; planning and procurement for pipe replacement with a more erosion / corrosion resistant piping material; replacement of water retaining gaskets with non-water retaining gaskets to minimize crevice corrosico; insulation of bimetallic joints to minimize the potential for galvanic corrosion; and maintaining water flow through all system branches to minimize microbiologically induced corrosion. '

On November 16, 1987, SWEC issued the SSWS corrosion evaluation report. This issued report reached the same general conclusions as the draft report with the exception that it concluded the SSWS pipe coatings should be removed, thereby precluding sheet mode failure and localized accelerated corrosion. it was recognized that coating removal would expose the SSWS to general i corrosion. As general corrosion had not been assumed in original system design, analyses were performed which shoned that general corrosion could be tolerated. In addition, a commitment was made to perform periodic pipe wall thickness measurements beyond the inservice inspection requirements of ASME Section XI to monitor the corro: ion rate.

The recommendations described above were adopted by TU Electric.

l

4 Enclosure to TXX-88699 September 23, 1988 Page 13 of 46 It is concluded that the characterization of the problem was handled professionally. Management devoted extensive resources to characterizing the problems in the SSWS, The activity was directed by Engineering. Appropriate avenues of investigation were identified and necessary technical expertise was brought to bear. The results of the investigatory activities were documented and appropriately summarized in a final technical report which provided root causes for both the problems identified in PR 85-532 and those identified later, and a well defined course of action to resolve those problems. The NRC was appraised of the coating problems in a timely manner and made aware of progress through a series of written reports.

II.A.2 Evaluation of Alternatives and Selection of Solution following issuance of the SWEC SSWS Corrosion Evaluation Report, Senior Management established a task force to address coating removal and other SSWS enhancements. The Unit 2 Assistant Project Manager was assigned as the task force Project Manager. Adaitional task force members were identified from TV Electric Projects, SWEC, Startup, CPE Mechanical Engineering, Operations Results Engineering, and Construction.

Management's decision to adopt a task force / project management approach to the Plasite removal issue was reflective of both the importance of and complexities associated with the coating removal efforts and represented a commitment to project continuity. The diverse nature of the resources needed to complete Plasite removal demanded a coordinated, multi-discipline effort.

Although the Engineering studies showed that the conditions in the SSWS would not demand immediate corrective action, Senior Management established an aggressive schedule for implementing the recommendations of the SWEC Corrosion Evaluation Report. Specifically, the decision was made to implement those actions necessary to resolve concerns with coating integrity and reduce the susceptibility of the Unit 1 SSWS to corrosion prior to the Unit 1 Hot functional Test. This decision reinforced the priority to be given to task force activities.

The task force considered a wide variety of techniques for removing the Plasite coating including gas blasting, sand blasting, hydrolasing, chemical cleaning, pipe sleeving, and pipe replacement. Each of these techniques was explored to determine whether it was capable of removing the coating and the impact it would have on both SSWS and the remainder of the plant.

With regard to sandblasting, the task force identified two potential problems which had to be addressed. ^irst, sandblasting could result in removal of metal, raising questions about preservation of minimum pipe wall thickness.

Second, unless properly controlled, dust could cause a significant plant cleanliness problem. Additionally, it was recognized that a verification of coating removal would have to be accomplished. For 24 inch, 30 inch, and accessible 10 inch diameter pipe, this could be done by direct visual observation. For inaccessible 10 inch diameter pipe, an alternative technique would have to be developed.

With regard to the first issue, minimum wall impact, the task force decided that a demonstration would be performed to determine if sandblasting would remove the Plasite without causing unacceptable metal damage.

Enclosure to TXX-88699 September 23, 1988 Page 14 of 46 With regard to dust control, it was decided that tents and filters would be r

set up at locations where spread of dust was likely. Airflows would be ,

controlled such that dust would be captured. Additionally, in-process '

monitoring would be conducted to detect any dust / grit breakthrough during blasting and to assess integrated dust / grit accumulation outside the SSWS.

4 With regard to verification of coating removal from inaccessible 10 inch pipe, it was decided that video tapes would be made of the pipe following sandblasting. Review of these tapes would be used to verify coating removal.

A 3reliminary demonstration of the sandblasting process was conducted on fe3ruary 19, 1988. The demonstration determined that a sandblasting process {

using an apparatus consisting of a movable cart with a set of rotating nozzles (spinblaster) could successfully remove Plasite from a coated 10 inch pipe without causing significant base metal removal. It was directly supervised by l Engineering. The actual blasting operation was performed by 0. B. Cannon

Inc., one of the bidders for coating removal. Ultrasonic test (UT) measurements of pipe wall thickness were made and recorded before and after sandblasting operations. Two sets of data were obtained. The first set characterized the amount of base metal removed in the process of removing the coating. The second set characterized the amount of metal that could be removed if the blaster was left at one point for four minutes.

The demonstration showed that sandblasting would remove the Plasite coating and that only one to five mils (1 mil =.001 inch) of metal were removed in the process. Only 12 mils of metal were removed if the blaster was left in one place for four minutes. Two other observations were mad.e by the task force.

First, the grit used in the blasting created an undesirable debris removal problem and would have to be replaced. Second, the 60 psi air pressure was significantly below that which could be encountered in the plant. Based on these observations, the task force recognized that additional testing was necessary; however, the suitability of the sandblasting process had been established. Engineering, recognizing that additional testing was required, believed that the amount of metal removed was sufficiently small such that no post-work verification of pipe wall thickness was required. This was based on the results of previous engineering studies which showed that a significant margin existed between nominal pipe wall thickness and that thickness at which piping would be challenged.

Based on information provided by the task force, Senior Management selected sandblasting as the most viable process for removing the coating in a safe, timely and cost effective manner. Senior Management directed the task force to assure that sandblasting would not cause unacceptable pipe wail thinning.

!!.A 3 Background Conclusions As of March 1988, significant progress had been made in addressing the problems of corrosion, erosion, and coating deterioration in the SSWS.

Extensive inspections and engineering evaluations had been performed which quantified the problems and their safety significance. A corrective action plan had been established consisting of near term and long term actions to address the issues. A task force had been created to develop and implement these actions. A schedule had been adopted for prompt implementation of corrective actions. A process had been selected and demonstrated effective for removing the coating,

Enclosure to TXX-88699 September 23, 1988 Page 15 of 46 The task force properly identified the potential risks associated with sandblasting and established an approach to characterize and control those risks. A preliminary process demonstration was performed which indicated that sandblasting could remove the coating with little risk to the pipe material.

It was recognized that additional developmental work, including tests, was needed to finalize process parameters.

The degree of management oversight exercised during this phase of the project was good. Senior Management took the aggressive step of establishing a task force to develop and implement a solution to the problem of coating removal and monitored task force activities through a series of meetinos. The task force Project Manager held weekly meetings of the task force to follow progress end provide a forum for resolving identified problems.

!!.B. Project Implementation

!!.B.1. Prerequisite Decisions This phase of the project involved those activities necessary to translate the management decision to employ sandblasting into a process containing the controls and quality attributes necessary to ensure that the SSWS coating could be removed without unacceptable impact on the system or the remainder of the plant.

Before the selected methodology could be implemented, the following questions had to be answered:

o Who will perform the work?

o What constitutes an acceptable product?/How is product acceptability to be verified?

o What Codes and Standards are applicable?

The scope of work for coating removal was defined by Design Modification Request - Construction Phase (CtiRC) 88-1-020, which was translated into Design Change Authorization (DCA) 61219, Revision 1. This DCA defined the scope of work as removal of coatings from 10, 24, and 30 inch safety related SSWS pipe.

!!.B.1.a. Work Group _ Selection With regard to who would perform the work, due consideration was given to utilizing internal resources and specialty contractors. Given that the work was very specialized, and that little experience existed internally to execute the selected process, a decision was made to employ the services of a specialty contractor.

!!.B.1.b. Acceptance Criteria Definition Two acceptance criteria were defined. The Design Change Authorization established that the coating be removed to resolve the outstanding issues of blockage of flow to critical components and localized accelerated corrosion.

Initially, this criterion was to be satisfied by removing all of the coating from the 10, 24, and 30 inch SSWS pipe. The second criterion established by

Enclosure to TXX-88699 September 23, 1988 Page 16 of 46 Senior Management was that the process not cause damage to the piping or other plant equipment which would compromise design bases. This criterion was to be met by ensuring that sandblasting would not remove unacceptable amounts of metal from the SSWS pipe and by establishing a dust control program as discussed above.

!!.B.1.c Codes and Standards Applicability The work was designated as quality related. First, the work was being performed to address safety issues. Consequently, adequate quality assurance / quality control measures were necessary to ensure that the product of the process adequately resolved those safety issues. Second, the work could affect a safety related s As defined in the TV Electric Quality Assurance Program (QAP)ystem, and 10CFR50 SSWS, Appe.tdix B, such work must be performed using quality controls.

The Codes and Standards applicable to SSWS are defined by Specification, 2323-MS-100, which was referenced in the Design Change Authorization for coating removal, included among these Codes and Standards are the ASME Boiler and Pressure Vessel Code, the Steel Structure Painting Council Standards, applicable ANSI Standards, and the 2323-MS-100 imposed cleanliness standards.

All of these standards were imposed on the sandblasting process with the exception of ASME Section XI. Given that the current CPSES ASME Section XI -

Program requires evaluation of metal removal activities for inclusion under the auspices of that Program, the decision to not apply ASME Section XI to coating renoval was explored during this evaluation.

Based on this effort, it was determined that the primary reasons for not identifying coating removal as an activity to be conducted under the auspices of ASME Section XI were that sandblasting is not normally considered by the l industry to be an ASME Section XI activity, and that the amount of metal to be ,

! removed by sandblasting was expected to be insignificant. Given that the  ;

l sandblasting process had not been fully developed, this decision was  !

l premature. .

Further review revealed that Engineering procedures for implementing ASME Section XI activities require that documents pertaining to ASME components which direct activities that are not ASME Section XI repairs or replacements I should so indicate and state that Onner's form NIS-2 is not required. A number of such documents were generated during the coating removal project: 1)

DMRC88-1-02v; 2) DCA 61219 Revision 1; 3) the requisition for 0. B. Cannon services; 4) the purchase order / contract with 0. B. Cannon; and, 5) the

0. B. Cannon process procedure. Only the requisition explicitly identified the non-applicability of ASME Section XI. None of the documents referenced Form NIS-2.

With respect to the issues of metal removal and documenting ASME non- .

applicebility, two actions will be taken. First, procedure changes will be j made and training will be provided to appropriate personnel on when metal j removal activities on ASME components should be conducted under the auspices of tha CPSES ASME Section XI Program. Second, the requirements to clearly indcate the applicability of ASME Section XI in appropriate documents will be i reinforced through training, t l

l l

l a

Enclosure to TXX-88699 i September 23, 1988  ;

Page 17 of 46 l 11.B.2. Procurement The TV Electric QAP requires procurement documents to specify:

o Scope of work.

o Technical requirements including procedures, Codes, regulations, test requirements, inspection requirements, and acceptance requirements.

o 0A requirements to be imposed on the contractor.

o Documentation requirements.  ;

o Nonconforming conditions which require disposition.

ECE 6.02, "Preparation and Review of Procurement Documents", is the governing  !

procedure for procurement of parts and services by Engineering and j Construction (E&C). This procedure requires the Responsible Engineer (RE) to .

prepare a requisition in accordance with Nuclear Purchasing Group Procedure [

NPP 5.0. NPP 5.0 directs the requisitioner to Nuclear Furchasing Group Instruction NPI-5.0-2, "Control of TU Electric Requisition cn Purchasing l Department."

NPI-5.0-2 reqaires the originator to provide the following basic information

! on the requisition:

4 o A description of what is wanted and the technical and quality

requirements which are applicable, i d

t a o Suggested sources of the item or service. L i ,

j o The procurement Quality Code of the item or service.

< There are four procurement quality codes: A; V; C; and N. These codes define i the pertinent quality requirements to be applied to the vendor / supplier. Code .

A is applied to quality-related procurements, it requires the vendor / supplier l to have a TV Electric-approved 10CFR50 A)pendix B Quality Assurance Program, I accept 10 CFR Part 21 reporting responsi)ilities, and be on the approved l j vendur list. Code V would be applicable to those cases where safety-related l hardware or quality-related services were needed but the vendor selected could i i It acknowled l not or would not satisfy Code A requirements. applicationit of all or part of 10CFR50 A allons ,

application to be made either through the vendor's 0A program, the TV Electric QA program, or a combination of these programs. The vendor need j

, not be on the approved vendor list. The intent of Code V is to ensure that ,

1 appropriate QA requirements are satisfied where they are not otherwise imposed j on or by the vendor. Code C is applied to those products where the required ,

quality level is "Commercial Grade'. Code C would not be used for quality- l related procurements unless the product was to be subsequently L

, dedicated / qualified by TV Electric. Code N is for non-quality related l q procurements which are not subject to formal 10CFR50 Appendix B quality  ;

i constraints. .

I l

l I

\  !

+

Enclosure to TXX-88699  !

September 23, 1988 l Page 18 of 46 i Assignment of the quality code by those organizations defining and perfonning l work is appropriate given that primary responsibility for quality rests with j those organizations. Assignment of the quali y code by those organizations  :

rather than by the Quality Assurance Department assures that the assignments are made by personnel most familiar with and knowledgable of product requirements, and avoids any possibility of compromising needed independence i between production and quality assurance organizations. [

The next step in the process, Engineering review, is dictated by a combination of NP!-5.0-02, ECE 6.02-03, "Engineering Review of Procurement Documents for 1 Services", and ECE 6.02. ECE 6.02 requires Engineering to review the '

requisition. For Code V services procurements, ECE 6.02-03, requires the reviewer, whether that person is the RE or another individual, to specify ,

those critical characteristics and verification activities necessary to satisfy applicable quality assurance (QA) requirements, who is to perform them, and when they are to be performed.

Once Engineering completes its revien, the requisition is sent to Quality Assurance for review. This review is to be conducted in accordance with NQA 6.02, "Quality Review of Procurement Documents".

NQA 6.02 requires the Quality Assurance Department reviewer to verify that the procurement documents identify or provide for later identification of test, inspection, and acceptance requirements and any special instructions, and, for Quality Code V procurements, that critical characteristics be verified by inspection and/or tests performed prior to, upon, or after receipt of the item or service. Finally, the procedure requires that any discrepancies or exceptions found during review shall be resolved with the document originator.

These procedures adequately define the tasks necessary to ensure a complete requisition; however, for Code V services they could be enhanced by inclusion of additional guidance. Specifically, existing procedures do not provide detailed guidance as to the m% by which the TV Electric QAP will be imposed on the work activities. In contrast, in Code A procurements, the relationship between the TU Electric QAP and the work activities are clear and unambiguous.

In such procurements, the contractor applies its own TU Electric approved QA program in accordance with regulatory requirements, while TU Electric QA fulfills an auditing and surveillance function.

These enhancements are under development, in the interim, the Vice President, Engineering and Construction has issued direction (NE 21514) that specifications or task descriptions will be prepared for all Code V procurements which will define how the TU Electric QAP will be imposed on the activities of the contractor, ll.B.2.a Requisitions Two requisitions were processed to support the SSWS coating removal.

The first requisition (6R-343877), approved on September 25, 1987, requested that Nuclear Purchasing obtain bids for SSWS coating removal. The requisition was issued for the limited purposes of identifying interested specialty contractors and obtaining general pricing information. This requisition wAs not intended to serve as the final basis for a contract.

N Enclosure to TXX-88699 Se,ntember 23, 1988 Page 19 of 46 Although there were some shortcomings in this requisition, overall it properly identified the scope of work and identified it as Code V. Given its limited purpose, the shortcomings are not viewed as significant determinants of the eventual proper implementation of the coating removal process.

The second requisition, number 6R-350338, was initiated for the procurement of the services of 0. B. Cannon. The following cbservations were made with regard to this requisitions o The stope of specialty contractor wnrk was clearly defined, o The Procurement Quality Code was Code V, o The requisition identified that the process to be employed was dry sandblasting and that the blast material had to be approved. Engineering subsequently provided approval for #3 Flintbrasive as part of the process sequence discussed later in this report.

o The final cleanliness criteria were clearly defined and exceeded those contained in specification 2323-MS-100.

o The Services Review Summary identified that the work was not subject to l either ASME Section !!! or ASME Section XI requirements, o The requisition established Steel Structure Surface Preparation Specification No. 6 as the industrial standard to be applied to blasting.

In addition, applicable ANSI Standards were referenced, o The requisition addressed the issue of release of blast material / debris to general plant areas by noting that dusting of the plant must be kept to acceptable levels, o The requisition contained a number of attributes which should have resulted in a well controlled process. Spccifically, the requisition required that the specialty contractor submit detailed procedures for

, Engineering approval. Specific attributes to be included in these procedures were Engineering hold points, details concerning equipment i

operating pressures, blast material and sire, and dust suppression and removal methods. In addition, the requisition required that an initial acceptance criteria test be included in the procedures, o The Critical Characteristics Evaluation prepared for this requisition identified complete removal of coatings as the critical characteristic.

Neither the Critical Characteristics tvaluation, nor the requisition, explicitly mentioned pipe wall thickness as a parameter of concern.

o The verification plan specified three activities nhich required l verification. The first nas the removal of coating in service aater pipe as per the technical ano quality assurance requirements. These requirements are contradictory as written and should have been clarified.

Originally, verification responsibility was assigned to TV Electric OA.

j lt was later assigned to Engineering's coatings expert. The second l activity was verification of cleanliness of pipe as defined in commercial specifications. This verification was to be performed in-process by TV l

I

N

'o Enclosure to TXX-88699 i September 23, 1988 '

j Page 20 of 46 l

l Electric QA. This responsibility was later assigned to Engineering's [

coatings expert. The final activity was, "Initial Acceptance Criteria

Test." Verification was to be performed by TV Electric QA and [

Engineering prior to beginning work. This responsibility was later i 1 assigned to Engineer ' coatings expert.

4

) While these verification activities lacked detail, they did reflect the need to ensure that the planned sandblast tests were accomplished prior to beginning work, that the piping, system was left in an acceptable 1

condition of cleanliness upon work completion, and that the coatings were l removed.

TV Electric QA reviewed this requisition and suggested two changes. As l originally written, the requisition stated that the entire blasting operation r

would be under TV Electric's QAP and that the initial acceptance criteria test f 1 would be witnessed by TV Electric QA representatives. Engineering deleted l l both statements based on QA's recommendation by making "pen and ink" changes l I as allowed by procedure. The first statement was judged unnecessary given i I that the requisition was designated as Procurement Quality Code V. By  !

j definition, the TV Electric QA program was applicable. Thus, this deletion  ;

4 did not authorize or represent a reduction in the quality requirements to be ,

j imposed on the work. The second statement was deleted, because it was QA's l view that the test did not require direct QA involvement. In exploring these  ;

changes, it was determined that the RE could not recall being contacter. either '

by QA or the Engineering reviewer concerning the changes. Consequently, the i

, intent of the originator in including the statements was not determined prior (

i to their deletion. This sequence of activities is recognized as a weakness. [

To this end, the need to contact originating organizations of requisitions l' prior to making changes to quality requirements will be reinforced with ,

requisition reviewers.

l

it should be recognized that changes of this type would not impact the quality [

of work performed under a Code A procurement insofar as, by definition, the  !

contractor's QAP would govern quality activities. Therefore, the impact of  !

{ this weakness is limited to Code V procurements. l i

l Overall, the requisition to obtain the services of 0. B. Cannon could have  !

! been tubstantially improved. The requisition did not include detailed test i objectives, provisions for inspection / acceptance of wall thinning, detailed .

I documentation requirements, or sufficient controls to identify nonconforming  !

! conditions. The requisition did not reference the originating DCA/0MRC. t i Detail was lacking in the verification plan. Finally, the review process for '

) this requisition was not adequately managed. The Engineering review was not  !

j performed by the appropriate engineering group and there is no evidence of  ;

a task force management review. A project management charter is under

development which will address these weaknesses, j l Had this requisition been processed as a Code A procurement, the deficiencies  !

j in the quality requirements would have been avoided. Specifically,  ;

documentation and nonconforming condition requirements would be prescribed by  ;

) the contractor's QAP and the contractor would clearly have been obligated to i l comply with those requirements. In addition, verification activities would l have been accomplished during the review and inspection activities required by the contrac'.or's QAP. Finally, the contractor and its personnel would have l been familiar with its quality obligations as specified in its QAP. ,

I

1

• Enclosure to TXX-88699 September 23, 1988  !

Page 21 of 46  ;

Had this requisition been processed as Code A, the technical requirements I would have received an additional level of review by the contractor as it j applied its QAP to those requirements. This review would, in all likelihood, ,

i have resulted in clarification of those requirements, especially with regard

. to process testing where the contractor would have had to prepare detailed j

test procedures.

j Overall, the procurement process enhancements discussed above with respect to 4

the level of detail required of Code V services procurements and the need to ,

contact originating organizations prior to making changes to requisitions, l

)

will prevent recurrence of problems of the type encountered in this i requisition. The letter (NE 21514) issued by the Vice President, Engineering j and Construction, requiring specifications or task descriptions for Code V

services procurements provides adequate interim assurance that such '

, procurements will contain necessary technical and quality information.

i l l  !!.B.2.b Contract  ;

! The next step of the procurement process examined was the generation of the  !

purchase order (P0) resulting in the contract with 0. B. Cannon. This review }

1 focused on whether the process correctly translated the technical and quality  ;

] requirements contained in requisition 6R-350338 into the P0. -

This review was based on P0 CP-0794 and the final contract signed by TV l Electric with 0. B. Cannon. The following observations were made

'

o The scope of work was translated from the requisition to the P0.

i o The contract included the requisition-specified requirement that the e contractor submit procedures for ventilation, blasting, and acceptance [

i criteria for approval, although it included less detail than that j specified in the requisition.  !

o The contract established the right of TU Electric to stop any work which  !

3 it considered unsatisfactory and obligated the contractor to receive TU '

) Electric asproval prior to recommencing work. The contract did not i obligate tie contractor to notify TV Electric of any problems or i discrepancies so that remedial measures may be taken in accordance with >

j the QAP. -

1 I 1

o One of the contract conditions indicated that it was the result of the i work, not the work process itself, that was of interest to TV Electric.

Although, this condition could be read to contradict one of the (

principles of the QAP, i.e., the need to perform work in accordance with  ;

approved procedures, in the context of the overall contract and its  !

implementation, it is apparent that this condition did not create any l 4 misunderstandings. In fact, this condition did not influence either the  !

l conduct of the work in accordance with approved procedures or the degree [

] of oversight provided to the work process by TV Electric. Hoeever, to (

l minimize ambiguities in future contracts, a NE0 Policy Statement is being l

] prepared which will requ,re that TV Electric policies, commitments, and (

, obligations be clearly conveyed through contract language.

I l

l i

, t i Enclosure to TXX-88699 l September 23, 1988 i Page 22 of 46  !

Overall, it is concluded that the contract signed with 0. B. Cannon was less  !

specific than the requisition with respect to critical technical and quality  !

attributes. In this regard, however, this contract was executed as part of a {

i Code V procurement, and it required that the work be performed under the i auspices of the TV Electric QA program. Consequently, the lack of specificity {

4 in the contract did not decrease the level of quality assurance required to be t j applied to the coating removal process. [

i in the course of assessing the contract, a review of applicable procurement I

3rocedures was conducted to determir.e to what extent contracts are required to I j 3e reviewed prior to issuance. From this review it was determined that the .

only review explicitly required to ensure the proper translation of quality I i requirements from the requisition is to be performed by the Quality Assurance  !

j Department (QAD). For Code V services procurements, where Engineering is  !

1 required to specify the detailed quality requirements applicable to the  !

i requisitioned activities and to establish the relationship between the  !

} contractor and the TV Electric QAP, this review is not viewed to be  !

j sufficient. Consequently, procurement procedures will be revised to require l

that the technically responsible roup review such contracts prior to issuance i I to ensure that they adequately re lect appropriate technical and quality l requirements.  !

1 >

I II.B.2.c Conclusion l l

l The purposes of the procurement process are to obtain the services of a t contractor and to explicitly define the relationship between that contractor I i and the technical and quality requirements applicable to the task to be j d performed. I l

i The first purpose of the procurement process was achieved. The services of [

! 0. B. Cannon were acquired to remove the coatings from the SSWS piping. The  ;

} second purpose, defining the relationship between the contractor and the  !

! technical and quality requirements applicable to the coating removal task, was ,

not fully achieved. l The quality level of the task was properly and formally identified. The need ,

l for a detailed procedure to control the process and the need to adhere to that l 4 procedure was also formally identified. However, steps were not adequately [

l taken to ensure that the identified quality level and the need for procedures  !

were translated into specific instructions. The most notable shortcomings  !

I were that the verification plan did not identify all of those activities which

{ would have to be accomplished and by whom, and tnat the requisition did not i

include all of the technical parameters required to ensure that the i sandblasting process would not cause unacceptable damage to the SSWS piping. ,

i I i The absence of some technical parameters in the requisition is explainable by l the fact that the process itself was still under development. The need for j additional tests was identified. Honever, a needed tie between the testing j and the coating removal procedure c.as not established. As it happened, this ,

tie was informally made. The results of tests were translated into non- f proceduralized process controls. [

l h

l  !

, i

Enclosure to VXX-88699

September 23, 1988 Page 23 of 46 4

i Existfng procedures require that applicable cuality requirements be defined in I the procurement documents. Revien of other cocumentation associated with the j coating removal project reveals that the quality requirements were discussed, l This is reflective of the fact that involved personnel were concerned with the J quality of the coating removal task.

I With regard to whether the shortcomings in implementation of the procurement

process for the coating removal project had generic implications for other 4 Code V procurements or for CPSES procurements of other types, it was concluded

that the implications did not extend beyond Code y procurements. The explicit j weakness involved in the coating removal procurement was not identifying the relationship between the requisitioned commodity and the TU Electric QAP. The i

problem occurred because personnel involved in the procurement process had little or no experience in Code V service procurements and were not

] sufficiently sensitive to the need to clearly and unambiguously impose the i

TV Electric QAP on the contractor's activities. The potential for this problem exists only for Code Y procurements. For Code A procurements, the relationship is clearly defined. As a result, in Code A procurements the outside cor. tractor understands the importance of quality assurance and, through his QA program, will assure formal proceduralization of processes and testing, and strict adherence to documentation and non-conformance reporting j requirements. Code C procurements do not normally involve safety or quality-I related comodities that demand oversight by a formal QAP. However, when it

! is desired to use a Code C comodity in such an application, a separate and i nell defined dedication process must be implemented. Code N procurements are j not quality assured.

l There are onl

> for services.y two Thetypes of Code technical V procurements, and those quality attributes for for hardware hardware areand those generally l well-defined in existing Engineering Specifications, as are the physical attributes of the hardnare which can be used to verify its quatity prior to 4

use. Therefore, weaknesses of the type encountered in *N HW5 tating l removal procurement are not expected in the area of t wdware procurement.

) This limits the problems identified to Code V services procurements.

' To determine whether there might be other Code V services procurements which suffered from the problems identified herein, a review of current and previous Code V services procurements has been undertaken.

Three actions nere developed to minimize the potential for recurrence of the problems associated with the SSWS Code y services procurement. The first action, which is already in progress, is to require that specifications or task descriptions be prepared to support all requisitions for such services.

These documents will provide clear definition of and responsibility for implementation of technical and quality requirements. The second action is to modify e(isting procedures to provide additional guidance on the process by which th? TV Electric QAP is to be applied for Code V services procurements.

The third action is training of all personnel responsible for preparing and reviening procurement documents in their areas of responsibility and the requirements of the revised procedures.

Enclosure to TXX-88699 September 23, 1988  ;

Page 24 of 46 II.B.3 Development of Procedures and Criteria  ;

i i The signing of the contract with 0. B. Cannon completed the procurement *

p rc' es s . What remained was process proceduralization as it related to the activities of the contractor and the associated activities of TV Electric. l

!!.B.3.a _TV Electric Activities l l t The first portion of this discussion will focus on the associated activities j of TV Electric. Those activities consisted of the following: [

1 o Control of pre-work test activities f

) o Definition of in-process work authorization.

i ,

j o verification of coating removal.  ;

Ouring this revien the following observations were made: l

o As noted in the previous section of this report a series of process tests l 1 was planned to establish the ability to remove coating without -

unacceptable impact on the piping. Procedures were not prepared for  !

these tests; however, they were performed under the direct supervisson of 6 l Engineering utilizing the 0. B. Cannon process procedure, i o A Startup Work Authorization (SWA) was used to authorize coating removal. j l All quality-related work to be performed is under a SWA is required to be  !

l performed in accordance with approved site procedures. Such documents l l are required to be reviewed by the Quality Assurance Department to l l determine the necessary inspection scope and attributes, identify l

] required inspection documentation, and verify or establish necessary l inspection points. i Engineering reviewed and approved the 0. B. Cannon process procedure,  !

QCP 1. Since the requisition which procured the services of 0. B. Cannon j nas quality-related, the Quality Assurance Department should also have i reviewed QCP-1 prior to issuance of the SWA. Responsibility for ensuring proper and complete reviews of the work procedure rested with the task l force generally and the Construction Department specifically. This ,i review was not obtained. That the review was not obtained stems largely .

from the earlier shortcoming in not unambiguously establishing a  !

relationship between the work activities, the TV Electric QAP, and the '

involved TV Electric organizations during the procurement process. The [

necessity for complying with all applicable requirements prior to l obtaining on SWA will be reinforced with appropriate organizations, in  !

addition, the SWA procedure will be revised to provide more explicit  :

guidance on what constitutes an approved site procedure. '

o At the time the work began in the plant, it nas recognized that coating removal verifications were required, and. as discussed above. that they would be conducted by a combination of direct visual observation and review of video tapes. Initially, it was assumed that the coatings would be removed in their entirety. Engineering, therefore, concluded that a I

i

% Enclosure to TXX 88699 September 23, 1988 Page 25 of 46 special inspection procedure was not warranted. Subsequently, it was realized that small quantities of coating material would remain on the piping. This necessitatad changes to the coating removal criterion. An evaluation was performed to establish the amount of coating which could remain. The resulting criteria were placed in existing procedure, EME 3.21-08, "Engineering Verification of Protective Coating Applied to Steel Surfaces Subject to Imersion Service (Coating Removal From Station Service Water Pising)." Difficulty was enccuntered initially in determining whet 1er the criteria were met from the video tapes taken of the 10 inch pipe due to distortion of the video image. However, modificationt to the verification technique and the procedure were made.

Verifications were completed which satisfied the requirements of EME 3.21-08 and which demonstrated that the coating was removed to the extent dictated by the acceptance criteria.

Coatings removal verifications were perfonned on approximately 30 spools as part of developing EME 3.21 08 and were documented on special forms.

This effort was undertaken with the knowledge that reverification would be required if the activities did not satisfy the recuirements of EME-3.21-Of Subsequently, these results were determinec to satisfy the requirt its of the final procedure end credit was taken for them.

It must be recognized that the purpose of EME 3.21 08 was to verify that the coating had been removed to the extent that the remaining coati',g could not block SSWS flow to critical safety components. This procedure did not, nor was it intended to, require evaluation of other indications in the piping observed during the coating removal verifications.

Further, it was not intendt.d to require verification of the manner in nhich sandblasting occurred. Evaluation of indications other than coating removal was intended to be the subject of existing nonconformance procedures. A separate activity had been scheduled to assess wall thickness reduction due to corrosion as part of the overall SSWS project.

o One last observation was made concerning proceduralization of TV Electric activities. Section 6.1 of NQA 1.01, "TU Electric NEO Quality Assurance Department Organization and Responsibilities", specifies that the Quality Department is responsible for assessing the critical attributes associated with quality related equipment or work activities. To this end, the Quality Assurance Department developed a surveillance checklist based on QCP-1 and the 0. B. Cannon work plan. The checklist covered compressed air, blasting equipment, abrasive, vacuum procedures, preoperational requirements, and documentation. The surveillance attributes adequately reflected the content of QCP-1 and the 0. B. Cannon work plan.

In addition, the Quality Assurance Department developed a surveillance checklist for coating removal verifications performed under EME 3.2108.

Review of this checklist shoned that it appropriately covered the coating removal verification requirements.

Enclosure to TXX-88699 September 23, 1988 ,

Page 26 of 46 II.B.3.b Contractor Activities The next portion of this evaluation focused on the proceduralization of

0. B. Cannon's work activities. The content of the Cannon procedure was defined in the 7equisition soliciting Cannon's services. Review and approval was conducted in accordance with the approved Engineering procedures.

The adequacy of the Cannon procedure was assessed by comparing the content of that procedure with the requirements in requisition 6R-3! W 8. This requisition specified that QCP-1 should contain the fcilcwing informations o Engineering hold points, o Blast material and size.

o Details concerning equipment, operating pressures, removal methods,  ;

and dust suppression methods,

]

o An initial acceptance criteria test.

i Revien of thc approved Revision 2 of QCP-1 revealed the folloning:

o No Engineering hold points were established.

l o Blast material and size were not explicitly specified. However, QCP 1 l

required that blast material be capable of removing the coa'ing, that it 2

be selected by O. 8. Cannon supervision, that it be approvou by the Onner, and that the abrasive not contain coal slag derivatives, aluminum oxide, steel grit, or steel shot. Consequently, adequate provisions were l inrorporated to ensure that the blast material used was appropriate. I

o The types of equipment specified nere automatic spinblasters and standard 1 blasting nozzles. Air was to be supplied by TV Electric. Dust suppression methods were addressed in terms of ventilation flons and directions and through the use of dust collectors and blast houses.

4 l o The need for conducting an initial acceptance criteria test was included, '

i The procedure specified that an Onner Representative would be present and

) that QCP 1 would be used to perform the test. ,

k Given that Engineering's i'itent was to control the blasting process such that

• blast impact on pipe wall need not be separately verified, the following attributes, as a minimum, should ha've been included in the 0. B. Cannon procedure:

i o The envelope of acceptable blasting parameters for metal removal  ;

purposes.

i i

o In pror.ess data taking to assure adherence to the envelope of blasting l

, parameters.

o imediate cessation of work and Onner notification upon disco,ery of unexpected conditions or difficulties.

\

\ Enclosure to TXX-88599 September 23, 1988 Page 27 of 46 o Prior Ownef approval of enanges to procedures cr equipment.

Some blasting parameters were established as a result of tests and were the subject of separate written instructions and t.4ining provided by the contractor to its personnel. Specifically, these instructions addressed spinblaster travel rate and the amount of blasting allowed in any given location. These instructions were supplemented by an informal agreement between the contractor and involved TV Electric personnel that TV Electric would be promptly appraised of difficulties encountered in the Ilasting operation. Moreover, as discussed below, as problems were encountered, work was susper.ded and enhancements were made with the knowledge of TV &lectric.

Similarly, changes in QCP 1 were cpproved by TV Electric and it was kept ir. formed of changes in instrucUons and equipment. However, these controls should have been formally defined, imposition of the informal controls is reflective of the fact that involved personnel were concerned with the potential impact of alasting on pipe wall thinning. Not formally incorporating these control; into QCP-1 impacted subsequent in process monituring activities incofar as these contrnis were not included in the Quality Control surveillance cbcklists, nor were they highlighted to the Construction Engineers as part of their oversight responsibilities.

II.B.3.c Procedures and Criteria Conclus'..ns Process proceduralization was accomplish 6 nith ouly limited success. An adequate coating removal verification pre > dure was developed, A Quality Control surveillance checklist was developed which reflected the requirements of the verification procedure. The contractor's process procedure wa:,

reviewed and approv?d by Engineering in accordance with approved procedures.

However, int procedures nere not preparei ce;) trolling the performance of process tests, the resu..s of the tests were not ir.corporated into the contractor's process procedure, and the contrattor's process procedure was not reviewed ade p tely to satisfy the pierequisites for the selected work authorization process. Quality Control did develop a surveillance checklist which reflected the requirements of the contrrctor's process procedure; honever, the adequacy of this checklist was limited by the inadequate incorporation of needed detati into the process procedure.

The observed weaknesses in proceduralization were largely the result of not clearly defining the relationship between contractor activities, the TV Electric QAF, and the responsible TV Electric er anizations, It is noted

that where those responsibilities were clear (surve llance, coating removal

verification, ant contractor process procedure review), the associated i activities were conducted in accordance with pre existing procedures. Based i on this observation, it is concluded that actions being t3 ken to enhance the level of detail associated nith Code V servici s prccurement activities nill

]

, also address the shortcomings identified in this area l

l i

Enclosure to TXX-88699 September 23, 1988 Page 28 of 46 II.B.4 Solution Implementation II.B.4.a Mobilization Prior to actually beginning work, the contractor had to be mobilized. In order to understand how mobilization occurred, the SSWS task force Project Manager and representatives from the Construction Department were interviewed.

From these interviews it was determined that the mobilization of 0. B. Cannon was to be accomplished in accordance with procedure EC 6.11, Revision 0, "Post

- Contract Award Indoctrination Meetings." While no regulatory requirements exist for this procedure, it was voluntarily imposed by E&C as an enhancement to previously existing procedures on the subject of controlling the work of f il contractors. A similar procedure exists ia the Nuclear Operations 4 organization. The purpose of this procedure is to establish a standard method .

for conducting meetings which apprise an awarded contractor of specific TU '

Electric policies, procedures and goals, through a kick-off meeting held at  :

the site, prior to the Contractor's site mobilization date. Responsibility for coordinating this meeting is procedurally assigned to the Contracts Administrator.

• The following topics are to be discussed at the meeting as appropriate:

o How TV Electric Engineering and Construction (E&C) expects t'siness to be j conducted by its own and by contractor personnel to obtain successful completion of the work.

o The control measures to be applied by TJ Electric ESC including work progress monitoring and interfaces to nsure the work is accomplished in accordance with contract requirements, approved design documents, and is completed as scheduled, o Correspondence control and TV Electric furnished items.

o Plant layout. [

o The need to complete the Contractor's Work Release Authorization Form and obtain Owaer's approval on this form prior to beginning work.

o The work completion schedule and the progress reporting process.

o Limitations on scope of work. L o Safety program's and training.

o The contractor's responsibility regarding requisitioning, receipt, l inspection, and storage of materials.

The Contractor's Work Release Authorization Form i 3 quires the contractor to 6 certify that the contract /P0 has been signed, necessary insurance is in  !

effect, procedures are completed and incorporated, contractor personnel performing the work have been trained in the applicable work and/or quality i related procedures and records of this training have been forwarded to the applicable IV Electric organization, and materials and/or special tools supplied by the contractor have been received in the TV Electric warehouse and  ;

1 Enclosure to TXX-88699 September 23. 1988 Page 29 of 46 have been accepted by Quality Assurance as applicable. The form requires a TU Electric Contracts representative to certify that the Owner has signed the ,

contract /P0 and that the contractor's insurance is in effect and is acceptable. It requires a TV Electric E&C representative to certify that the procedures required by TV Electric are in effect by the contractor and approved by the Owner, contractor personnel training in the contractor and/or Owner developed procedures related to the scope of the contract has been  ;

accomplished, and contractor sup) lied materials and/or special tools have been j received by the TV Electric ware 1ouse and accepted by Quality Assurance. A Quality Assurance Department representative is also required by EC 6.11 to make the same verifications regarding procedures, training, and 9 materials /special tools. Finally, a TV Electric representative is required to 4

certify that the Contractor's Work Release Authorization Form has been  :

completed and approved by the Owner, and the contractor is released to begin i work.

• Based on a review of available documentation, it was determined that the required meeting was conducted on April 6, 1988. The meeting was attended by l l the 0. B. Cannon Area Superintendent and General Foreman. TV Electric '

representatives included the Construction Engineering Mechanical ,

! Superintendent, the Contract Administrator, Engineering, and Safety. Three of the attendees were from the task force. It was noted that the mobilization i procedure does not define the minimum level of attendance at contractor

]

mobilization meetings. This will be resolved by making necessary procedure l l

changes.

! The Contractor's Work Release Authorization Form was completed by the contractor several days after the meeting; however, TV Electric did not

! complete the sections of the form for which it was responsible. In following up on this oversight, which was documented on Deficiency Report (DR) C-88-03361, it was determined that the procedure lacked specificity regarding whether the Contracts Administrator or the E&C representative had i lead responsibility for ensuring that the form was completed. A procedure

revisior, has been prepared which identifies that Projects Contract Management l

and the Department Manager of the department for whom the services are to be performed will both sign the Contractor Work Authorization Form. It was also .

determined that the verification responsibilities contained in EC 6.11 for the  !

Quality Assurance Department are not reflected in either a NE0 level procedure '

or a Quality Assurance Department procedure. This will be corrected by j establishing the appropriate procedural requirements.

!!.B.4.b Process Tests Implementation of coating removal actually involved two separate activities.

l' The first was the conduct of a number of process f osts whose objectives were

! to show that the 0. 8. Cannon procedures, person' { and equipment could l successfully remove the SSWS coatings without ca. .ng undue damage to the SSWS l pipe, and to establish appropriate parameters for controlling the sandblasting process. Oversight of these tests was provided by Fngineering. Copies of the documentation associated with these tests were obtained and reviewed and

! discussions were held with Engineering to explore how the tests were managed / conducted and the results obtained.

i f

'. Er, closure to TXX-88699 September 23, 1988 Page 30 of 46 The first series of tests was performed on April 6, 8, and 11,1988. The documentation associated with this series of tests consists of documents summarizing the tests and a set of two diagrams showing pre and post-blast Ultrasonic Test (UT) readings taken on the piping spools used for the tests.

From this documentation it was determined that the April 6 demonstration was performed only to establish that the coating could be removed and was successful in that regard.

The April 8 test involved pulling the spinblaster manually through a coated 10 inch spool as necessary to remove all of the coating. UT readings were taken before and after the blasting to determine the extent of metal removal. The maximum removal was 4 mils.

It should be recognized that, to this point, the acceptance criterion for coating removal was that all of the coating be removed. Test conditions were controlled to the extent that this criterion was met. The amount of metal removal was measured as a dependent variable.

The April 11 test was conducted by using the spinblaster to blast in one circumferential location for one minute in a 10 inch uncoated pipe at a nozzle pressure of 98 psi. Comparison of pre and post blast UT results documented on the diagram su plied show that the amount of metal removed was insignificant (1 mil or less .

The conclusion reached by Engineering based on this series of tests was that the Plasite lining could be completely removed. The accompanying amount of metal removed was not significant provided that the spinblaster was not held in place for longer than one minute. This supported the Engineering belief that no post-blast pipe wall thickness measurements were necessary to assess the impact of sandblasting. 0. B. Cannon was instructed regarding the one minute limitation and incorporated this direction in written instructions and training provided to its personnel. In addition, Engineering approved the use of Flintbrasive #3 as a blast material.

The second set of tests occurred on April 19, and was performed on 10 inch uncoated pipe using the same blast material used in the previous series of tests at a nozzle pressure of 98 psi. Three pulls of the spinblaster were performed of one inch each with a one minute stall at each pull point.

Results showed that cumulative metal loss was 12 mils. This was used as the basis for post-blast Engineering minimum wall analysis. The contractor was instructed informally to avoid successive, closely-spaced blasts.

Two key points were identified during the evaluation of these testing activities. First, the tests nere a quality-related activity. They were conducted to establish the envelope of process parameters necessary to support coating removal without pipe damage. As such, special procedures should have been prepared, reviewed, and approved in accordance with existing requirements. Further, the results of the testing should have been formally documented. Neither of these activities occurred.

'. Enclosure to TXX-88699 September 23, 1988 Page 31 of 46 The second key point identified was that the test conditions were not representative of the conditions that would be encountered when the process was employed in the plant. Specifically, it was known that in-plant blasting would occur on long sections of piping and that grit / debris removal equipment would be necessary to assure that excessive g-it/ blaster debris would not accumulate in the pipe. Use of this equipment during testing was not required because the tests were performed on short pipe sections which allowed the grit and debris to be blown out of the pipe by the action of the spinblaster alone.

It is, however, noted that the tests did accurate'y encompass those conditions which would be experienced during planned blasti n of plant piping removed from the plant.

The process tests performed were not fully representative of the conditions which would be encountered during remote spinblasting in the plant. Not recognizing this condition was a oversight which permit *.ed sandblaster-induced damage to the SSWS piping. As such, it is the root cause of the damage. Had this oversight been recognized, either the test conditions could have been modified to make them more representative of in-plant conditions, or compensatory control mechanisms could have been established to accommodate the differences.

Engineering is directly responsible for this oversight. However, the effects of the oversight were not detected until after problems were encountered with the remote spinblasting process in the plant. Two formal opportunities existed to detect / avoid this oversight before sandblasting occurred in the plant. Not taking advantage of these opportunities is viewed as a significant contributor to the damage which occurred to the pipe.

The first opportunity occurred during the procurement process. As noted above, during that process, the objectives of the tests were not clearly defined. Consequently, the requisition originator's intentions with regard to the reasons for performing the tests were not clear. Neither the intentions nor the lack of clarity were challenged. Rather, the assumption was made that Engineering would, as a result of the required tests, deliver a "qualified" process.

The second opportunity was QAP-required proceduralization of the tests. Had formal test procedures been prepared which explicitly identified test objectives, method of conduct, acceptance criteria, and documentation requirements, at laast two opportunities would have been created to detect the  ;

oversight. The first opportunity would have occurred during the test procedure review and approval cycle where independent parties could critically examine the objectives, methodology, and acceptance criteria. The second opportunity would have occurred during the results review cycle.

Thus, it is concluded that, while Engineering did recognize the need for process testing and did perform that testing, it was not conducted adequately dnd Was not subjected to required formal quality assurance requirements, it is not anticipated that these problems would extend to Code A services procurements as the vendor would apply its QAP to all procured activities, including process testing. Internal testing is subject to the formal controls of the TV Electric QAP.

(

k Enclosure to TXX-88699 September 23, 1988 Page 32 of 46 With regard to these conclusions, three actions will be taken. First, more explicit procedural controls will be established governing testing associated with Cede V services procurements. Second, a review of testing associated with other Code V services procurements will be conducted to determine whether it was properly conducted. Finally, a review of existing controls on process testing generally will be conducted to confirm that improvements are not required.

II.B.4.c Coating Removal The second activity associated with the completion of the coating removal project was the coating removal itself. Oversight of this activity was performed by Construction, Engineering, and the Quality Assurance Department.

Field activities were assigned to specific Construction Engineers. One of these engineers was interviewed to determine the manner in which these activities were conducted. From these discussions it was ddermined that Construction provided continuous coverage of 0. B. Cannon in-plant work.

Specifically, procedure implementation was monitored along with dust control activities for the process and adjacent plant areas, and coordination with other interfacing groups such as Operations and Startup was provided. Also, Construction participated in task force meetings to report on job status and problems.

A number of process problems was identified including the following:

r o Stalling of the spinblaster during coating removal on the early sections  ;

of 10 inch pipe was identified by 0. B. Cannon and reported to the assigned Construction Engineer. The stalling problem was brought to the ,

attention of the task force. In-plant remote spinblasting of the 10 inch t pipe was discontinued until enhancements as discussed in Section III of ,

this report were made to the process. j o Difficulty was being experienced in achieving complete coating removal. [

Multiple passes of the spinblaster were required to remove all of the i coating. Given that the process in the 10 inch diameter pipe involved a [

blast cycle followed by a video camera inspection, the process was L proceeding slowly. Additionally, concern developed that multiple blaster l passes, which had not been modeled during the pre-work demonstrations,  !

might result in removal of too much metal. i i

The concern over the effect of multiple passes was addressed by i performing another pronss +.est under the direct supervision of f Engineering, but, without a test procedure. This test was performed on I June 14, 1988, using a blaster supplied with 150 psi air on a 10 inch t uncoated pipe. The higher air pressure was selected as being b conservative with respect to metal removal. l The demonstration involved measuring the amount of metal removed under three conditions: a one minute stall in blaster longitudinal movement; a

)

two minute stall in blaster longitudinal movement; and a two minute stall  !

in blaster longitudinal moveme.t plus a superimposed blast at a three f inch per minute blaster movement rate. The acceptance criterion was that !

neither a one minute longitudinal stall no. a thrse inch per minute pull j rate would remove 5 mils of metal or more. The metal removal results r V

Enclosure to TXX-88699 September 23, 1988 Page 33 of 46 were 4 mils, 8 mils and 12 mils respectively. This data showed that metal would be removed at the rate of 4 mils per minute in either a stalled condition or with a three inch per minute pull rate. Thus, the acceptance criterion was met, provided that the minimum blaster transit speed was 3 inches per minute or that the blaster was not left in a given position for more than one minute. These restrictions had already been imposed on and by the contractor, although not in QCP-1.

Complementing the in-process work oversight activities of Construction were surveillance activities performed by the Quality Assurance Department. A total of seven surveillances were conducted on two topics. The first topic was 0. B. Cannon's implementation of QCP-1. Surveillances of this topic were performed on April 13, 18, 21; May 12; and June 22. The remaining two surveillances, performed on June 1 and 14, cover.d Engineering verification of coating removal. The surveillance reports for all seven surveillances were revieued. With respect to those focusing on the implementation of QCP-1, the following observations were made:

o The checklist described earlier in this report was used for all five surveillances.

o As noted earlier in this report, spinblaster speed needed to be controlled to ensure that excessive base metal was not removed. The restrictions on this attribute were not imposed by QCP-1 but by other 0.

B. Cannon documents. These documents were not included in the scope of the surveillance activities.

o No observation was made of vacuuming to remove blaster debris from in-plant pipe.

o The first four surveillances showed that 0. B. Cannon was acceptably implementing QCP-1. The final surveillance, performed on June 22, identified two discrepant conditions. The first of these concerned the blast material being used. Documentation could not be produced at the time that Flintbrasive Grades 3 and 4 had been approved for use.

Flintbrasive #3 had been approved as part of the demonstration process.

This discrepant condition was appropriately documented on Deficiency Report C-88-03435. Engineering independently generated a separate Deficiency Report C-88-03428, which also documented the use of unapproved blast material.

The second problem was that incorrect documentation forms were being used. This was also documented on a Deficiency Report (C-88-03434).

Both issues were evaluated by Engineering, and it was determined that neither impacted the acceptability of the work performed.

Enclosure to TXX-88699 September 23, 1988 Page 34 of 46 With respect to the surveillances performed on Engineering verification of coating removal, the following observations were made:

o A checklist was developed for this surveillance based on change EDCN-03 to procedure EME 3.21-08 Rev. O. During this time frame, change EDCN-04 was issued revising the cov'ng removal verification acceptance criteria to reflect the changes to tre removal criterion discussed above.

Although the surveillance, checklist was not modified, verification of EDCN-04 implementation was satisfied by the method of checklist implementation.

o The surveillances indicated that the verifications were adequately conducted and discrepant conditions were appropriately documented.

in addition to the surveillance activities noted above, the Quality Assurance Department became involved in performing pipe wall thickness and corrosion pit depth measuremeits as the coating removal progressed. These activities were initiated in early June at the request of Engineering. In addition, Engineering rescheduled Quality Assurance Department pipe wall thickness measurements on other locations where initial data was needed to form the basis for future corrosion evaluations. During the course of these measurements, indications of minimum wall problems were received and nonconformance reports (NCR) were generated.

Ultimately, it was determined that the indications of minimum wall problems were not, in fact, minimum wall violations. However, as a result of one of the NCRs documenting an apparent minimum wall violation, TV Electric QC was requested by Engineering to perform inspections of Train B, SSWS 24 and 30 inch pipe in any area of localized or general wall thickness reduction and record the maximum depth of wall loss beyond 40 mils in 30 inch pipe and 50 mils in 24 inch pipe. The 40 and 50 mil criteria were based on a pipe stress analysis performed as part of the SSWS corrosion evaluation which showed that loss of this much wall material would not challenge pipe integrity.

Additionally, QC was requested to map any areas of closely knit clusters of localized corrosion and UT areas adjacent to corroded surfaces to provide a baseline uncorroded wall thickness.

To evaluate the safety significance of indications identified by these inspections, pipe wall thickness was established by first measuring the depth i of the indication using a depth gage. Next, the pipe wall thickness in the four quadrant locations immediately adjacent to the indication was measured j ultrasonically. Finally, the depth of the indication was subtracted from the wall thickness adjacent to the discontinuity. No minimum wall violations were identified as a result of these activities.

This technique was evaluated with respect to the requirements of the applicable edition and addenda of ASME Section XI. Subarticle IWA-2200(a) of Section XI states that the examination method to be used for Code Class 3 piping, such as SSWS piping, is specified in Table IWD-2500-1. This table does not require the conduct of a visual examination or inspection of the interior surfaces of piping. Thus, the inspections performed go beyond the requirements of ASME Section XI and the methodology used was acceptable to evaluate the indications.

Enclosure to TXX-88699 September 23, 1988 Page 35 of 46 The coating removal work proceeded effectively, except for the problems discussed above. The equipment problems were brought to the attention of .

TV Electric and were resolved appropriately from a technical standpoint; however, these problems were not documented in one of the formal CPSES prob'....,

reporting systems. Consequently, root cause evaluations, corrective actions, dnd implications of the problem were not documented. The issue of the threshold for reporting problems encountered during conduct of work activities is the subject of initiatives separate from SSWS coating removal. This ,

situation represents another example which will be addressed by that initiative.

The number and nature of the problems encountered early in the spinblasting process warranted a formal work stoppage. This action was not taken. Not taking this action represents a shortcoming in the perurmance of Task Force management. To address this situation, Project Manager duties and responsibilities will be defined in a specific charter.

The final activity associated with coating removal was inspection to verify that the coatings had been adequately removed. As noted, above, this was completed successfully in accordance with approved procedures.

II.B.4.d Nuclear Regulatory Commission Interface As part of this evaluation, a review was conducted of the exchange of information between TV Electric and the Nuclear Regulatory Commission (NRC) on issues relating to the SSWS coating removal project. From this review it was determined that information was exchanged primarily by two mechanisms. The first mechanism was a series of reports submitted to the NRC pursuant to 10CFR50.55(e). The second mechanism was routine interface with the onsite NRC inspection staff.

II.B.4.d.1 10CFR50.55(e) Reports As noted earlier in this report, coating degradation and weld leaks were communicated to the NRC as potentially reportable issues in January 1986. -

This notification was confirmed in writing in a letter dated February 24, 1986 (TXX-4711), which noted that the coating degradation and weld leaks were not related, and that further testing and evaluation was necessary to resolvt.

i these issues.

On April 11, 1986, another interim report (TXX-4762) was provided to the NRC.

i This report provided the corrective action plan for the weld leaks. With respect to coating degradation it noted that repairs would be made using a Belzona lining. The extent of repairs to Train A was to be based on recently

completed Train A piping inspections. The extent of Train B repairs would be based on a scheduled Train B piping inspection.

On September 19, 1986, a supplemental repurt (Txx-5026) was provided to the NRC wherein it was identified that the inspections of Train A revealed no generic failures of the Plasite 7122 coating. Further, it statea that several pitted areas were repaired and that other areas containing minor defects not warranting repair were identified for use as baseline data for future i

inspections.

I

Enclosure to TXX-88699 September 23, 1988 Page 36 of 46 Interim reports continued to be submitteo and the NRC was notified that SWEC was performing a corrosion study on SSWS, On June 22, 1988, the final report (TXX-88476) for Unit i was submitted. This report noted that the SWEC corrosion evaluation report had been issued and that the Plasite lining was experiencing sheet mode and blistering failures.

Further, it provided commitments to the corrective actions discussed earlier in this report, including coating removal, relative to corrosion, erosion, and coating degradation.

Based-on a review of this series of its reports it is concluded that TV Electric kept the NRC appropriately informed of the status, and ultimately, the corrective actions for corrosion, erosion, and coating deterioration in SSWS. Further, through reference to the SWEC corrosion evaluation report, extensive information on the root causes and impact of these problems is provided, including sheet mode failure, localized corrosion / pitting, and general erosion / corrosion.

It is noted that neither the SWEC report nor the final 50.55(e) report to the NRC address those activities which occurred prior to 1986; however, given that these activities were generically addressed by the Corrective Action Program which had been fully communicated to the NRC, this situation is acceptable.

II.B.4.d.2 Interface With The Onsite NRC Inspection Staf_f As noted above, the NRC was kept appraised of SSWS corrosion, erosion, and coating problems via a series of reports. Copies of these reports were provided to the onsite inspection staff as the reports were issued. However, there was little additional contact with the onsite inspection staff with regard to the SSWS project until af ter coating removal actually began.

As documented in NRC Inspection Report 50-445/88-34, NRC inspection of the coating removal was initiated on May 2, 1988. During the course of this inspection and a subsequent inspection documented in Inspection Resort 50-455/88-40 the inspector raised a number of question regarding tie coating removal project.

In report 50-445/88-34, which documented inspections primarily occurring in May and early June 1988, the inspector expressed concerns over the 1980 G&H evaluation of flaking of coating from flange faces and the lack of formal quality controls applied to the coatings; the actions taken based on the G8H evaluation including the FSAR and specification changes, and closure of INP0 SER 68-83 and IEN 85-24; whether pitting in the piping identified during the system inspections performed in 1986 an1 1987 was adequately evaluated and mapped; the adequacy and inspectability of the paint removal criteria; and potential for sandblaster-induced wall thinning. The exit meeting for this inspection was conducted on June 7, 1988. These concerns were updated in Report 50-455/88-40 issued in July 1988. The exit meeting for this inspection was conducted July 6, 1988.

1 Enclosure to TXX-88699 September 23, 1988 Page 37 of 46 Based on interviews with a number of individuals involved in the coating removal project it was determined that direct contact between the onsite inspection staff and coating removal project personnel occurred with individuals at the working level and their supervision. There was no contact with the Task Force Project Manager, although the Task Force was made aware of the inspection staff's concerns through its membership. After becoming aware of the concerns near the close of the first inspection, the Task Force Project Manager initiated contact with the onsite inspection staff and was advised that the concerns were being addressed. In addition, frequent contact with the onsite inspection staff was initiated and maintained by the TV Electric Licensing Department to obtain specific information pertaining to SSWS concerns and communicate that information to responsible TV Electric Management.

Based on the number and the nature of the concerns expressed during the June 7 exit meeting, the Comanche Peak Manager, Mechanical Engineering, was directed to review the status of the coating removal project. As a result of this effort, presentations were made to the inspection staff on June 20, June 27, and July 13 on the technical concerns expressed over the ongoing coating removal work. In addition, on July 20, the Executive Assistant to the Vice President of Engineering and Constructicn met with representatives of the onsite inspection staff to assure that all concerns were understood by TV Electric. One additional presentation was scheduled for the end of July.

This presentation was postponed when the hole in the SSWS pipe was discovered on July 29. These actions were taken to assure that the concerns expressed by the NRC were understood and being addressed, and the TV Electric evaluations of the concerns were being communicated to the NRC.

In addition to the actions described above, interviews with involved personnel indicated that specific actions were taken to address a number of the concerns. Supervisors responsible for evaluation of industry operating experience initiated rereviews not only of IEN 85-24, but of all IENs received prior to 1986 to determine whether prior reviews were adequate. Action to improve the inspectability of the coating removal criteria, already in progress, was accelerated. Piping inspections to further characterize the status of corrosion in the SSWS piping, planned as part of the SSWS Erosion / Corrosion Program, were conducted in parallel with coating removal activities.

Explicit reference to the NRC concerns does not appear in the documentation of task force activities; however, that documentation does include references to a number of the same issues. For example, in memorandum NP-6966 dated April 22, 1988, it is noted that a SWEC evaluation was in progress to determine the criteria for the amount of Belzona and Plasite that can remain in the pipe. In a subsequent memorandum, NP-7135 dated May 6, 1988, it was noted that inspection criteria discrepancies have been resolved. In memorandum NP-7537 dated June 17, 1988, it is noted that Ebasco's review cf the 10 inch pipe video inspections revealed areas requiring further evaluation and that final disposition is restraining pipe restoration. In followup memorandum NP-7602 dated June 24, 1988, it is noted that the inspections are in progress and that the results will be forwarded to SWEC for analysis to determine if repairs are necessary.

\ Enclosure to TXX-88699 September 23, 1988 Page 38 of 46 II.B.4.d.3 Conclusions Based on this information it is concluded that TV Electric was appropriately r responsive to NRC concerns expressed during the coating removal project.

Actions were taken to investigate those issues which had not been addressed previously. Actions planned to address issues identified internally were accelerat1d when those issues were reinforced by NRC concerns. Management i became aggressively involved in assuring that NRC concerns were understood f fully and being addressed, and that the results of those actions were 4

communicated to the NRC.

i  !

It must be recognized that a number of the concerns identified by the NRC ,

during coating removal activities had been explicitly addressed internally t prior to work beginning in the plant. For example, pitting is explicitly addressed in the SWEC corrosion evaluation report. It notes that pitting is

, an operational concern rather than a safety concern and that its progress will L be monitored periodically. Sandblaster-induced pipe wall thinning and QA/QC l involvement were likewise discussed and dispositioned in Task Force meetings.

j Initial coating removal criteria were also established and actions taken to 4

! change them when it was recognized in April 1988 that those criteria could not be met. Each of these issues were considered resolved. Notwithstanding, each ,

was reexamined in light of concerns expressed by the NRC. '

i in spite of these activities it is recognized that the quality of

communications with the NRC could have been improved had the Task Force

! generally and the Task Force Project Manager specifically been more proactive

in initiating contact with the NRC, and more aggressive in ensuring that NRC t

! concerns were fully understood, addressed, and resolved with the NRC. To this

, end, direction will be provided in the Project Manager Charter discussed

earlier in this report regarding NRC interface during special projects.

!!.B.5 Project Implementati" Conclusions

! Overall, the project implementation phase was marginally successful in ,

I accomplishing the objective of removing coatings from SSWS pipe without t i

unacceptable impact on either the SSWS or other plant equipment. In general, .

! the tasks which were prerequisites to beginning in-plant work were defined and  !

! responsibilities for those tasks were assigned. However, those tasks were not [

, uniformly completed because the details required for implementation were not i

! adequately established. The result was that formal controls were frequently 1 found to be lacking on critical activities. These problems derive directly

! from ineffective implementation of the procurement process such that the j relationship between the contractor's activities, the TV Electric QAP, and the ,

I responsible TV Electric organizations was not adequately defined. This lack l of formal controls contributed directly to not employing mechanisms which  ;

I would have created the opportunity to detect a technical oversight in process  !

! testing and which ultimately allowed damage to occur to SSWS pipe. It is l l concluded that the type of procurement, Code V services, was a unique l contributor to the implementation shortcomings. It is only with Code V that l l the relationship discussed above must be defined solely as part of the -

l procurement process. For Code A procurements the relationship is  !

l automatically defined for the contractor through his own QAP. Consequently, f l the corrective and preventive actions in this section of the report focus on ,

l enhancing the controls on Code V services procurements. i i

Enclosure to TXX-88699 September 23, 1988 Page 39 of 46 Ill. TECHNCR EVALUATION

!!!.A Introduction As noted above, on July 29, 1988, a leak was discovered in a Unit 1 Train A 10 inch pipe. The leak was documented on Nonconformance Report NCR-88-12705 Rev. O. It consisted of an oval hole, approximately 1/2 by 3/8 inch, located in the piping which supplies water to the Emergency Diesel Generator, Centrifugal Charging Pump Lube Oil Cooler, Containment Spray Pump Bearing Cooler, and the Safety Injection Pump Lube Oil Cooler. The system normally operates at a pressure of 50 psig and a temperature range of 40' to 102* F (Design Pressure 150 psig, Design Temperature 170'F).

A three foot section of pipe containing the hole was replaced per NCR 88-12705 Rev. O. Subsequent inspection of the removed section of pipe showed that it had no visible discontinuities other than the hole and an indentation (dent) of similar configuration approximately 180 degrees opposite the hole. On the inside of the pipe, the hole and the dent were approximately 1-1/2 inches in diameter with a very smooth taper. Piping exterior conditions at the location of the hole were characterized b a thin protruding circu'.nferential lip which seemed to have been pushed out b internal forces.

Assessment of the removed piping section indicated that the hole and opposing dent could have been caused by the spinblaster used to remove the coating from the 10 inch pipe. Stone and Webster Engineering Corporation (SWEC) performed an immediate review of the video tapes for the remainder of the 10 inch pipe and concluded that other indications, originally thought to have been corrosion, may also be spinblaster damage.

A TV Electric Project Manager was assigned (Reference NE-21251) to perform a technical assessment of the damage to the SSWS piping. SWEC assigned (ReferencJ SWEC 1072) a project engineer to support the technical assessment.

The objective of the technical assessment was to determine whether or not sandblasting had caused damage to the piping system and to implement corrective actions to allow recovery of the system from any damage that may have occurred. This technical assessment was divided into two phases, assessment of cause and extent of damage, and corrective actions. The following questions had to be answered:

o What caused the hole in the 10 inch pipe?

o Is other damage present in the system?

o Is the system acceptable for operation?

o Does the system meet the requirement of the ASME Code?

o What hardware changes, Engineering evaluations and inspections are necessary to return the system to an operating status?

The technical assessment Project Manager directed that all activities associated with the recovery for the SSWS be conducted under the controls imposed by CPSES ASME Section XI procedures (Reference NEO 2.26) and applicable portions of the CPSES QA Plan.

Enclosure to TXX-88699 i September 23, 1988 Page 40 of 46 III.B Assessment of Cause and Extent III.B.1 Methodology <

t A series of interviews was initiated with personnel who had been directly involved with the coating removal project. The 0. B. Cannon field superintendent returned to CPSES to participate in these interviews. The

0. B. Cannon equi) ment was brought back to the site to help explain what could have occurred wit 1 the coating removal process. A spool by spool summary of the coating removal process was created by review of the 0. B. Cannon daily logs kept during the work at CPSES by 0. B. Cannon personnel.

A complete review of the video tapes made of the inside surface of the 10 inch pipe was also initiated to look for the presence of the hole on the tapes and to see if similar indications existed elsewhere. High resolution monitors were used to review the tapes.

III.B.2 Cause The indications (dents) were caused by stalling of the rotating head of the spinblaster which was used for coating removal in the inaccessible portions of the 10 inch SSWS piping. This conclusion is based on the following:

) o The physical configuration of the spinblaster nozzle assembly is consistent with the subject opposing dents, assuming a stalled head condition (i.e., two opposing 180 degree nozzles are utilized).

o Video tape evaluation shows opposing (180 degree) indications in a limited portion of the 10 inch SSWS piping. Comparison of an interim

coating removal verification tape with the final verification tape for the same piping run shows that several areas of denting, which had not i initially existed, clearly occurred during blasting operations performed subsequent to filming the interim tape.

Long straight runs of 10 inch pipe were remotely cleaned by use of the s)inblaster. These piping runs ranged from 60 feet to 140 feet in length.

T1e operator pushed or pulled the spinblaster from one end of the )ipe while the other end was connected to an air blower to remove sand from tie pipe and control dust generated by the coating removal operation. Consequently, the operator could not maintain constant visual contact with the spinblaster to assure that it was properly functioning. In addition, the spinblaster t

carriage movement was also not uniform as it was being pushed with air hose or pulled on a flexible cable. These problems were not anticipated and the j spinblaster tests did not model these conditions.

III.D.3 Extent of Problem l The Unit 1 SSWS coated piping consists of appnuimately 4000 feet of 24 inch

! and 30 inch piping and approximately 3000 feet of 10 inch piping. The coating was removed from this piping by one of two methods, grit blasting with a hand

, held sand blaster nozzle or passing a spinblaster through the piping.

1

Enclosure to TXX-88699 September 23, 1988 Page 41 of 46 Eighty-four of the 10 inch spools were removed from the plant and grit blasted in the sand blast yard. These spools are short sections of piping less than 20 feet in length, and their associated elbows. They were grit blasted using the spinblaster but due to their short length the o)erator could observe the spinblaster at all times. Due to their short lengt1, no air blowing was necessary to remove sand. The spinblaster discharged the sand with enough i force that sand and removed coating were ejected from the far end of the pipe for the short piping spools cleaned in the blast yard. Spinblaster malfunction due to sand buildup did not occur. Also, this rethod of removal ,

was successfully tested by the process tests described earlier in this report.  !

Therefore, probability of damage is very low with this type of manual operation of the spinblaster. Further evaluation of these spools is not '

necessary.  ;

The coating in the 24 and 30 inch piping was removed by manual operation of a  :

standard hand held sandblaster nozzle. The operator cculd observe the coating removal process at all times. The spinblaster was used on a short run of piping, but did not effectively remove the coating. Coating in this short run of piping was later manually removed by the hand held sandblaster nozzle. The probability of damage is very low with this type of manual operation.

Therefore, further evaluation of the 24 and 30 inch piping is not necessary..

Numerous enhancements were tried following identification of spinblaster problems in May 1988, including hand pulling the spinblaster and installation of a strobe light to allow the operator to monitor spinblaster head stall.

These enhancements were only partially successful in preventing spinblaster trouble and were finally rejected in favor of more successful enhancements, i These enhancements include:

o Installation of 18,000 cfm blowers, replacing the 5000 cfm blowers, to I improve sand removal in long runs of piping.

o Use of an airborn with the 5000 cfm blower to improve sand removal in short runs of pipe.

o The installation of an air drive motor on the spinblaster to ensure continued rotation of the spinblaster head.

o The installation of a remote video camera on the spinblaster to allow the operator to continuously monitor the operation and movement of the ,

spinblaster.

o Use of rigid pipe to control spinblaster movement more precisely. i These enhancements can be seen on Figure 1. The spinblaster prior to enhancements is shown on Figure 2, The locations where remote cleaning occurred in the Train A 10 inch supply piping prior to spinblaster enhancements are shown on Figure 4 by a dashed line. Similar areas of the Train A 10 inch return piping were also spinblasted prior to process enchancements. The remainder of Train A 10 inch piping, and all of Train B 10 inch piping, were cleaned after spinblaster enhancements were made.

Enclosure to TXX-88699 September 23, 1988 l Page 42 of 46  !

l Review of the video tapes reveals that no significant damage occurred in the l 82 spools of 10 inch pipe spinblasted remotely in the plant in Train A and l Train B after process enhancements had been made to the blaster operation. '

Some of the spools have minor indications that may have been made by the spinblaster, but are clearly not indications of damage that could violate ASME Code stress minimum wall. Indications as shallow as .017 inch are readily  ;

visible on the video tapes. Engineering review of the video tapes grouped '

indications as light, medium and heavy which were defined as follows:

Light - less than .040 inch deep Medium .040 to .100 inch deep Heavy - greater than .100 inch deep Significant damage is, therefore, limited to 36 spools of 10 inch diameter piae (approximately 650 feet) which were spinblasted in the plant prior to enlancements being made to the spinblaster process. These s first worked on by the contractor w th the in-plant (remote) use pools of were the the spinblaste.. Spin blaster stall and sand buildup were reported by the contractor in the first three weeks of work and in-place spinblasting of the 10 inch pipe was discontinued until enhancements could be made in the process.

Priblems encountered by the contractor included:

o Buildup of sand in pipe.

o Difficulty in controlling spinblaster by cable pulling with automatic tugger.

o Inability to remove coating in some areas.

o Inability of operator to observe process, o Sticking of spinblaster head due to sand buildup.

Additional review of the video tapes reveals that there were three types of spinblaster damage to the 10 inch pipe. These are: denting as characterized by the hole found in the pipe; grooving, where the blaster head stalled while the spinblaster was moved through the pipe producing a dent which is elongated instead of being rounded; and ribbing, where the spinblaster head was turning without corresponding carriage movement, producing circumferential rings in the pipe.

Ill.B.4. Corrective Action All 10 inch pipe remotely spinblasted in the plant was video taped after coating removal. Based on an engineering review of this video tape, three 20 foot spools were selected for removal from the plant. These three spools contained the most extensive and most significant indications that had been observed in the 10 inch piping. They also contained examples of the three types of spinblaster indications; ribbing, denting, and grooving. These spools were sectioned into 32 piping coupons as shown in Figures 5 through 8.

Quality Control personnel took depth gauge and pipe wall thickness measurements. This data, documented on Inspection Report 88-0136, was

1 Enclosure to TXX-88699 September 23, 1988 Page 43 of 46 evaluated by Stone and Webster Engineering. SWEC reported that 611 of the damaged areas in the three spools were acceptable from minimum wall requirements as calculated in an ASME Code stress calculation.

An Engineering review of the video tapes revealed that other indications of spinblaster damage were bounded by the indications contained in the 32 piping coupons, except for 12 indications. These additional indications were measured by Quality Control personnel for pipe wall thickness except for two which were inaccessible due to piping system support configuration. This data, documented on Inspection Report 88-0141, was evaluated by SWEC. SWEC reported that the two indications not measured were similar in configuration and depth to the ten measured and the ten measured were found acceptable for minimum wall requirements as calculated in an ASME Code stress calculation.

A total of approximately 80 spinblaster marks were identified in the original Engineering evaluation of the video tapes after the hole was found in the 10 inch pipe. A total of 27 have been measured by Quality Control personnel using approved NDE procedures. Only eight of those measured were greater than

.100 inch deep and only four of those had a projected corrosion lifetime of less than 20 years. Those four are in the three spools removed from the plant and replaced with new pipe. Spinblasting indications remaining in the system that have not been measured are bounded by the samples that have been measured, therefore additional NDE of spinblaster indications is not planned. ,

The SSWS is being added to the CPSES Erosion / Corrosion Monitoring Program as recommended in November 1987 in the SWEC SSWS Corrosion Evaluation Report.

Wall thickness measurements will be made to baseline the system and will be repeated periodically to gather data on corrosion / erosion rates in the system.

Data points will include areas of probable corrosion / erosion, selected areas of known spinblaster damage, and selected areas of known corrosion denting.

Data points will be selected to allow monitoring of representative locations from the outside of the pipe while the system is in service. Measurements will be taken by Quality Control personnel using UT equipment in accordance with approved procedures. Data from this program will be used by Engineering personnel to make system life predictions and piping repair / replacement decisions.

A safety evaluation was performed as part of the corrective actions taken.

The safety evaluation considered the effect that each type of spinblast indication could have had on the safe operations of the plant had they gone uncorrected. Also considered was the unlikely event that the hole could have gone undetected and therefore uncorrected. The effects of both loss of heat removal capability and spray and flooding on the surrounding components and structures were also considered. The conclusions of this safety evaluation are discussed belon.

The hole would have negligible effect on the heat removal capabilities of the SSWS. The flow out of the hole was conservatively calculated to be 39.3 gpm.

This represents approximately .23 percent of the total system flow and 1.7 percent of branch locp flow respectively. The distribution of the loss of fluid between downstream components is well within their respective flow margins.

k Enclosure to TXX-88699 September 23, 1988 Page 44 of 46 The other spinblaster damage conditions would have had no immediate effect on the system since all were acceptable based on an ASME Code stress evaluation.

Generalized or accelerated corrosion might create a minimum wall condition in the SSWS at the damage locations in the future. Two failure mechanisms were i evaluated.

The first is through wall pitting at the damage location. Through wall ,

pitting would result in system leakage from pin hole leaks. These leaks would '

be detected, not occur simultaneously, and would not prevent the system from performing its safety function.

The second failure mechanism is characterized by more generalized corrosion at the damage locations. This could create an ASME Code stress minimum wall ,

condition that might result in a crack during a seismic event, producing ,

through wall leaks. This failure mechanism was evaluated by SWEC and '

determined not to be safety significant. These postulated leaks occur only in areas of minimum wall and do not propagate through the thicker adjacent pipe wall.

Engineering analysis demonstrates that these postulated leaks would not prevent the system from performing its safety function. Flow rates from the i hole or from the postulated leaks would be less than that calculated for a  :

Moderate Energy Pipe Crack in accordance with SRP 3.6.2. As such, the effects i of temperature, spraying, and flooding are bounded by analysis performed as a  :

result of the program delineated in Design Basis Document, DBD-ME-007, "Pipe f Break Postulation and Affects." '

Each of the types of damage, ribbing, grooving and denting were evaluated for i their safety implications. l Four cases of ribbing were included in the 32 piping coupons. Ribbing l resulted in a reduction of wall thickness of less than .025 inch. The >

required wall thickness to satisfy stress requirements for the piping system l ranges from .100 to .188 inches. Multiple UT readings on the 32 piping coupons indicate an average wall thickness of approximately .390 inches and a minimum wall thickness of .363 inches in the areas of ribbing, it is I concluded that ribbing is not safety significant. i i

Two cases of grooving were included in the 32 piping coupons. Grooving i resulted in a reduction of wall thickness of less than .045 inch. For the  ;

reasons discussed above it is concluded that grooving is not safety [

significant.

l t

Sixteen dents (8 pairs) were included in the 32 piping coupons. These dents ranged in size from less than .030 inch to .307 inch. The maximum measurement was identified 180 degrees from the existing hole. Each dent, with the exception of the hole and its opposing dent, has been evaluated, and remaining l wall thickness in each case is above the stress requirements for the piping  !

system. A spool specific wall stress was calculated for the maximum dent anu (

the remaining wall thickness was determined to be greater than required. '

Based on the above, the observed damage is considered not to be safety l significant.

l t

5

Enclosure to TXX-88699 September 23, 1988 ,

Page 45 of 46 4

IV.

SUMMARY

AND CONCLUSIONS As noted at the beginning of this report, the evaluation of the SSWS coating removal process had .vo primary objectives: first, it was to determine whether the coating removal process was controlled in a manner commensurate with its ,

importance to safety; and second, it was to determine the extent and significance of the damage to the SSWS piping caused by coating remova! and to i develop a plan to resolve the problems caused by the damage. In considering these objectives and arriving at the conclusions set forth below, TV Electric ,

carefully reviewed the circumstances leading to the damage to the SSWS in 1 order to identify the root cause and contributing factors and evaluate their generic implications.

The cause of the physical damage to the SSWS which occurred during coating '

removal was non-conservatism in the sandblast pre-qualification test.

Specifically, the are-qualification tests did not fully model actual in-plant  !

conditions, with tie result that the potential for blaster grit / debris buildup '

and spinblaster malfunction was not identified. .

A number of factors contributed significantly to the damage to the SSWS.

First, the prequalification test was not conducted in a sufficiently formal ,

manner. Second, the implementation of procurement controls did not result in i formally identifying all relevant information concerning the process to the l specialty contractor. Third, implementation of project controls did not  ;

result in a formal stop work order during the initial stages of the remote spinblasting operations. Finally, there was inadequate coordination between -

engineering, quality assurance, construction, projects, and the specialty contractor.

l These contributing factors share a common cause, that being the failure to unambiguously define the relationship between the contractor's activities, the -

TV Electric QAP, and the responsible TV Electric organizations. This l relationship should have been defined during the procurement process and enforced by the Task Force Project Manager. l l

Concerns identified during this evaluation were carefully reviewed, including  !

those not considerd to be root causes or significant contributing factors to  !

the damage to the pipe. Each was evaluated for generic implications and  !

corrective and preventive actions were developed to preclude their recurrence, j The specific conclusions reached by TV Electric as a result of this evaluation are described below.  !

o The findings of this evaluation are not significant with respect to the I safety functioning of the SSWS in that:

- The extent of damage was limited and would not have prevented SSWS from performing its safety function.

- The damage did not represent a threat to the functioning of other safety related equipment.

k Enclosure to TXX-88699 September 23, 1988 Page 46 of 46 i Appropriate corrective action was taken in a timely manner and controlled in accordance with the TV Electric QAP and the CPSES ASf4E Section XI Program.

The SSWS was to have been and will be monitored for future degradation as part of the corrosion monitoring program.

o The tasks necessary to support successful implementation of coating removal were defined by management and appropriate assignments were made.

TV Electric administrative procedures utilized during the coating removal project contain adequate provisions to ensure that a quality product is obtained; however, available checks and balances were not consistently and formally employed to assure effective controls were applied to the coating removal.

The interface between the various TV Electric and contractor organizations participating in the SSWS coating removal project was not well defined.

A critical technical characteristic, wall thinning due to sandblasting, was not sufficiently addressed.

Corrective action development and implementation lacked formality, o This project possessed a number of unique attributes which contributed significantly to the identified implementation shortcomings. The most significant was that a Code V services procurement was involved. The number of Code V services procurements is small and the experience level in processing such procurements is correspondingly low. A number of factors complicated this development.

The sandblasting process required significant development before it was capable of being effectively utilized.

Virtually no experience relative to the spinblasting process employed existed in the CPSES organization.

The work activity was to occt.r at the interface between the safety (piping) and the non-safety (coating) portions of a system.

The project involved extension of a process not normally considered an ASf4E Section XI process into an area withoet obvious precedents.

o The uniqueness of the project limits the implications of the weaknesses identified to those infrequent cases were Code V services procurements were conducted. Thus, the findings are not significant relative to the overall quality of construction at CPSES nor do they represent a significant progrannatic breakdown, o Appropriate corrective actions have been identified and are being taken to remedy identified weaknesses. Interim measures taken provide assurance that the weaknesses will not recur before identified corrective actions have been fully implemented.

~,

APPENDIX A CHRON0 LOGY Based on a review of available documentation and personnel interviews, the following chronology was developed relative to disposition of SSWS coatings issues:

DATE EVENT 09/18/80 A potentially reportable deficiency regarding possible indeterminate quality of SSWS coatings is reported to NRC Region IV.

10/02/80 TV Electric Engineering requests Gibbs and Hill to evaluate the safety impact of the possible indeterminate quality of the SSWS coatings.

10/17/80 Gibbs and Hill responds to the 10/02/80 inquiry, concluding that loss of coating is not a safety concern.

10/20/80 Piping fabrication and erection specifications are revised to indicate that the SSWS coatings are not safety-related.

10/29/80 TV Electric informs the NRC that the coatings issue is not reportable.

05/21/81 Amendment 21 to the CPSES FSAR is issued stating that the SSWS coating is non-quality assured.

07/23/81 NRC Inspection Report 81-09 closes the SSWS coating issue.

10/11/83 Institute of Nuclear Power Operations (INP0) Significant

, EventReport(SER)68-83isreceiveddocumentingsheet l mode failure (rapid large scale separation) of Plasite 7122 H coating in the cooling water system at Palo Verde Nuclear Plant.

11/07/83 TV Electric Engineering completes its evaluation of INP0

. SER 68-83.

04/25/84 The INP0 SER discussed above is closed by Plant Evaluation.

03/26/85 NRC Information Notice (IEN) 85-24 is received presenting the same informaiton as the 1983 INP0 SER.

09/10/85 Operations personnel are informed that significant deterioration of the Plasite coating on the SSWS outlet from the Unit 1 Train A CCWHX has been observed.

09/23/85 The coating deterioration noted on 9/10/85 is documented

on CPSES Problem Report (PR)85-532.

.N DATE EVENT 10/04/85 Work Request #1363 is issued to perform walkdowns of portions of the 24 and 30 inch Unit 1 Train A SSWS piping to determine the location and extent of coating deterioration.

12/20/85 CPSES PR 85-699 is issued documenting leaks in welds in the SSWS. The problem is judged to be potentially '

reportable. Work orders are prepared to repair the leaks.

12/23/85 A TU Electric internal memorandum is issued stating that PR 85-599 is potentially reportable and suggests that there may be a causal link with the coating failures.

• 01/10/86 Work Request #1363 is implemented in the 24 inch line downstream of SSWS Pump 1A.

01/20/86 TV Electric Engineering issues its preliminary evaluation of the potentially reportable issues identified in PR's  !85-699 and 85-532 which concludes that the weld leaks and '

the coating failures are unrelated. Additional SSWS ,

inspections are planned to determine the extent and cause  !

of coating failures, t 01/21/86 StoneandWebster(SWEC)personnelconductlimited .

I observations of the SSWS and evaluate the informat'on  ;

assembled for the preliminary evaluation of PR's85-699 '

and 85-532. ,

01/24/86 The NRC is notified of the potentially reportable deficiency involving weld and coating failures.

(SDAR CP-86-07)

I 01/29/86 SWEC issues a report to CPSES documenting its findings  ;

i relative to SSWS based on 1/21/86 activities. -

4 L j 02/03/86 Accessible SSWS piping is inspected by a Plasite service j representative.

02/14/86 Eased on the results of the coating inspections performed [

on 1/10/86, Operations develops a plan to determine the i extent of coating failures in SSWS. <

02/18/86 The Plasite manufacturer forwards a copy of the laboratory r analysis report on samples taken during the 2/3/86  !

inspection, i 02/24/86 The NRC is notified in writing of the SSWS weld and  !

coating deficiencies identified in PR e5-532 and PR 85-699.

03/12 - Inspections of the Unit 1 Train A SSWS piping are 04/21/86 performed in accordance with the 2/14/86 Operations plans.

'# DATE EVENT 04/11/86 The NRC is notified that the weld and coating failures which were identified on 01/24/86 as being potentially reportable are reportable.

04/24/86 TV Electric Memorandum TIM-860454 documents the results of the 3/12-4/21 inspections of SSWS piping.

05/07/86 Operations completes its engineering review of PR 85-532 based on the findings and conclusions documented in TIM-860454.

03/17/87 An interim report is issued to the NRC on 50AR CP-86-07 documenting corrective actions in progress and noting that SWEC is performing a SSWS corrosion study.

03/24/87 SWEC receives verbal authorization to proceed with inspection and evaluation of SSWS Train B.

04/87 SWEC performed an inspection of SSWS piping.

05/22/87 SWEC provides its preliminary recommendations for addressing SSWS corrosion / erosion based on the previously completed Train A inspections and the more recent Train B inspections.

07/14/87 SWEC provides a draft report on the results of their SSWS corrosion evaluation.

08/87-09/87 Ebasco performs 100% inspection of Unit 1 24 and 30 inch SSWS piping. Inspection reveals sheet mode failure and pipe wall thinning due to corrosion denting.

09/25/87 A requisition is approved by CPE soliciting bids for SSWS coating removal.

10/21/87 Ebasco coating inspection report issued.

11/16/87 SWEC issues the final SSWS corrosion evaluation report.

11/19/87 Bids for SSW3 coating removal are received from three vendors including 0. B. Cannon.

11/87 A task fore.e is assembled to develop alternatives and schedules for implementing corrective actions for SDAR CP-86-07. The first task force meeting occurs on 11/23/87. Weekly meetings start on 12/24/87.

12/03/87 SWEC issues its final report on SOAR CP-86-07.

12/87-2/88 The task force explores various techniques for Plasite removal.

01/88 Ebasco performs inspections of selected 10 inch SSWS pipe spools removed from the plant.

DATE ' EVENT 02/19/88 A demonstration of the 0. B. Cannon process is performed.

03/08/88 Senior management selects sandblasting as the method of coating removal from the options presented by the task force.

03/14/88 TU Electric and O. B. Cannon personnel meet onsite to discuss the sandblasting process. SWA issued to allow Construction to prepare SSWS Train A for coating removal.

03/23/88 0. B. Cannon submits its procedure, QCP-1, for coating removal for review / approval.

03/25/88 A requisition is approved for 0. B. Cannon to remove Plasite coatings from Unit 1 SSWS 10, 24 and 30 inch piping.

03/28/88 SWEC comments on QCP-1 are forwarded to 0. B. Cannon for incorporation.

03/30/88 0. B. Cannon sandblast equipment arrives on site.

04/05/88 A contract is finalized and signed with 0. B. Cannon.

04/06/88 TV Electric conducts contractor mobilization meeting with

0. B. Cannon.

04/06-11/88 A series of sandblast tests is performed. O. B. Cannon personnel trained in spinblaster utilization and procedures.

04/11/88 0. B. Cannon begins SSWS sandblasting operations in blast yard and on Train A in the diesel generator building.

04/13/88 Revision 1 to QCP-1 is approved by Engineering.

04/18/88 Remote spinblaster coating removal started in Train A 10 inch piping in the yard tunnel.

04/19/88 A third series of tes.5 of the sandblast process is performed.

04/19/88 ' Problems with sand accumulation in the pipe during remote spinblasting are identified.

04/20/88 Additional 0. B. Cannon personnel are trained in spinblaster operations and procedures.

05/02/88 NRC inspection of coating removal activities is initiated.

05/03/88 Commenced spinblasting of SSWS Train A at elevation 810 in the Safeguards Building (are where hole was subsequently located).

J

I DATE EVENT 05/06/88 3pinblasting suspended except for touch-up work due to spinblaster problems.

05/18/88 Initial use of gear driven spinblaster and video camera.

06/07/88 NRC exit meeting covering results from coating removal inspections.

06/14/88 A final sandblast test shows metal removal of 4 mils per minute with the blaster in one place for two minutes and 4 mils at a 3 inch per minute forward blaster speed.

06/17/88 NRC Inspection Report 50-445/88-34; 50-446/88-30 issued documenting coating removal inspection results.

06/20/88 TV Electric and NRC Inspection Staff hold a meeting to discuss technical issues on cnating removal.

06/22/88 Final Report for Unit 1 on SDAR CP-86-07 issued by TV Electric.

06/24/88 TV Electric Quality Control initiates a 100'5 inspection of SSWS Unit 1 Train B internal pipe surfaces to qantify corrosion-induced wall thinnirig.

06/27/88 TV Electric provides additional information to the NRC inspection staff requested on 6/20/88.

07/06/88 NRC exit meeting covering 6/88 coating removal inspection results.

07/07/88 SSWS sandblasting operations are completed.

07/10/88 Quality Control inspection of SSWS Train B 24 and 30 inch pipe completed.

07/13/S8 0. B. Cannon is demot'lized and departs the site.

07/14/88 TV Electric meets with the NRC and the State of Texas, Department of Labor and Standards, to status coating removal activities and address inspection findings.

07/15/88 'NRC Inspection Report 50-445/8840; 50-446/8836 issued documenting 6/88 inspection results.

07/20/88 TV Electric representative meets with the NQC Inspection Staff to assure that all NRC concerns with coating removal were understood by TV Electric.

07/29/88 The SSWS is refilled. A piping leak is identified.

08/01/88 TV Electric meets with NRC Inspection Staff to discuss pipe leak. A previously scheduled meeting for 8/4/88 to discuss inspection issues is cancelled.

• DATE i- EVENT ,

i 08/02/88 NRC exit meeting covering 7/88 coating removal inspection f results.

t 08/02/88 Potential spinblaster damage to SSWS 10 inch pipe is identified for the first time from the review of video tapes made to assess coating removal.

08/03/88 Three foot section of pipe containing the hole is removed I and replaced.  :

i 08/15/88 . Project Manager assigned for technical assessment of coating removal project.  ;

08/24/S8 Three spools of 10 inch SSWS pipe removed from the plant for examination.

t 08/31/88 Safety Evaluation completed for spinblaster damage to r SSWS.

i 09/02/88 NRC Inspection Report 50-445/8847 and 50-446/88-42 issued i on SSWS coating removal inspection results.  !

09/13/88 TV Electric presentation to NRC on the SSWS coating l removal project.  !

l i

i i

I i

I r

[

f V

h I

i I

---I

o ENGINEERING REPORT ER-NE-019 FICURE 1 4 A/d4/NZ LtdJ/f G .. .. .. .

.. __ _\

,a nsu 50 . .,,

@ - e y2 =

l O "

p -q '.

y m

• C

~

_j d_ AY _

l

/ r _ _ a n '

A _._ T

• y~h E N I \ i i , : , , ,

__ .. 2 .. .. .. ..

Q. g

\

ALAGt J/651

$s YA  %

• Al< MM

$6Cf/0HAL Vl8V TH<m%H F176 cCM/AL capts AV SCALE fl.ECftlC LINE -

Alt LINE SMNRJ$11T(--1 AIKM0fCK

/+

wia ,,a smeusrgg nWA%,.x BW6au#?r

- ,.7.n , ceses l

I

. l ENGINEERING REPORT ER-ME-019 FIGURE 2

--+

4 I I m

ho rLou W M

=

~hT f g n 6 a $- '

3 So,

~

+ ..

_ _a_

N:

smygu i k wwa/ -

urad -

SECDOWL VIEV Tuta4H nn? ~

M? SCALK SflNdLA$11?Afm l

/*N I

\

s P

$Pl4AASTER j,j ,

"O *

art s 7M l cnes

ENGINEERING REPORT ER-ME-019 FIGURE 3 I

Ny 30'SW PlPING I

L

\

DIESEL GENER ATOR UNIT 1 CONTAINMENT o

/ FCOG -

[ (

k AUXIL I ARY

\ i

• ~

C '

BLOG SAFEGUARD BLOG '

CORR 100Rg

,f y . . .

T

,0. s, g ,g ,sc./ - eiet res~ct I

CPSES STATION SERVICE WATER SYSTEM TRAIN A PIPING

s e

ENGINEERING REPORT ER-ME-019 FIGURE 4

' gM, e s EI. ' ISM 77/.}.'Tu'U.A ** 'd "**'

%N gn:

p 4

f .

Y ~ '@

e o

%. JACsET sATER

[ COMER r 44 -)

s Oh ,, g 4f*N k

• < co ~

,%,'%,%s* {> <>

s

• y, %

f

• Q3s O

i ST A110N SERvict e&ith 1R AIN ' A' 5LS Pt t 70 CutSIL CEPdRATOR JACa[7 n4T(R COMER l

l l

l 4

1 i

1 EIICINEERIllC REPORT ER-ME-019 FICURE 5 l

N

/

/

/

/

/

,/

/

/ ISB06-3A-1

/

i

[

[

s i

10' SERVICE WATER SUPPLY PIPING I ERP-SW-1-SB-006 i SPOOL NO. SW-1-SB-06-3  !

COUPON NO.1SB06-3 t

i f

I i,

5

~

O ENGINEERING REPORT ER-ME-019 FICURE 6 ISB06-4A-3 N

/

e ISB06-4A-2 /

j 1SB06-4A-1 15806-46 j

/ 15806-48-1 l

j D ISB06-48-4 10' SERVICE WATER SUPPLY PIPING BRP-SW-1-SB-006 SPOOL NO. SW-1-SB-06-4 C0bPON NO. 15806-4

o ENGINEERING REPORT ER-ME-019 FIGURE 7 15807-11A-13 -

N

/

/

15807-11A-2 /

/ 15807-11A-1 2'FLG. /

15807-118-2 j

/ 15807-118-1 l

/

D \ 15807-118-12 10' SERVICE WATER RETURN PIPING BRP-SW-1-SB-007 SP0OL N0. SW-1-SB-07-11 COUPON N0.1SB07-11 1

l'

.3 -

g; ,

ENGINEERING REPORT

ER-ME-019 FIGURE 8 ISB07-12A-14 N

/

/

ISB07-12A-2 /

j ISB07-12A-1

/

15807-128-2 j

/ 15807-128-1 l

/

/

D 15807-128-13 10' SERVICE WATER RETURN PIPING BRP-SW-1-SB-007 SPOOL N0. SW-1-SB-07-12 COUPON NO.15B07-12