ML20092G743

Affidavit of H Hudson Re QA Problems at Facility

ML20092G743
Person / Time
Site:	Diablo Canyon
Issue date:	06/05/1984
From:	Hudson H PULLMAN POWER PRODUCTS CORP. (FORMERLY PULLMAN, INC.)
To:
Shared Package
ML20092G713	List: ML20092G712 ML20092G728 ML20092G743 ML20092G761 ML20092G777 ML20092G778 ML20092G787 ML20092G791 ML20092G798
References
NUDOCS 8406250222
Download: ML20092G743 (47)

Text

, , ,

+ .

ff/($f AFFIDAVIT ag4 M'y N3:48 My name_is Harold Hudson. I have worked for 5 years at -

Diablo Canyon, as a Pullman Power Products pipefitter, QA/QC inspector, QA program Internal Auditor and Lead Auditor. I am about to resign my job, in large part because of my family's fears about the safety of the plant if it begins commercial operation. I am providing this statement to answer a myth --

that quality assurance at Diablo Canyon was acceptable because problems were identified -- through a case study'-- pipe suopost rupture restraints. Problems indeed were identified, which is one of the three steps necessary for a good audit or QA program.

But it is not sufficient. The problems kept recurring. That is be-

_cause the QA program failed in its second and third responsibil-ities -- identifying any similar deficiencies that exist; and identifying and addressing the cause of the problem, to prevent recurrence. -

Repetitive cases of previously identified violations represent a deliberate quality assurance breakdown, ng a success. .The history of the pipe rupture restraint program is a series of repetitive violations.

A History of the Pullman Power Produrts (M.W. Kelloco)

Pine Ruoture,Restr_aint Construction Procram at the Diablo Canyon Nuclear Plant.mCalifornia.

Prepared by Harold Hudson 5/26/84 Pullman Power Products W.W. Kellogg Co.) was contracted'by the Pacific Gas and Electric Ccepar.y to install piping, pipe supports

&$$?0$$3h5 g PDR 1

lloth

!v

• 8' l

and pipe rupture restraints at the Diablo Canyon Nuclear Plant.

Pipe Rupture Restraints are used to insure that if a pipe filled i with steam or pressurized water rupture, surrounding equipment would be protected by restraini*ng the pipe at critical points.

t If not restrained, the steam or pressurized water flowing from a broken pipe would cause the pipe to whip back and forth i

damaging surrounding equipment. Pipe Rupture Restraints take on i

i .

4

special importance at the Diablo Canyon Plant due to the close i

proximity of the Hosgri Earthquake Fault and the effect an i

earthquake would have in piping systems at the plant.

In May 1970, M.W. Kellogg (PPP) would sign PG&E Contract i

Specification #8711 for erecting Main Systems Piping and furnishing, fabricating, and erecting the balance of power plant piping.

I C.S. #8711 covered piping, valves, hangers and pipe supports.

Actual on site construction would begin in 1971. In'1971 PG&E i

would issue contract Specification #8833XR to furnish and erect j

Otructural steel for Units 1 and 2. M.W. Kellogg's'(PPP)'

criginal work under this contract was to erect containment i

6 structure pipe rupture restraints for Units 1 and 2 and the re-1 Cetor coolant loop, cross over pipe restraints for Units 1 and 2.

The C.S.#8833XR construction schedule called for Unit C1 Pipe Rupture Restraint erection to start on 7-8-72 and Unit #2 erection to start 3-8-73. The framing for Pipe Rupture

, Restraints would be subject to a Quality Assurance Progeam in s cecordance with section 3 of the contract. In addition all I

j Pipe Rupture Restraint welding procedures were to be prepared and 2

lhk

g. _ _ _ . _ . ~ _ _ _._ ._ _ . _ _ . . _ . __ . . . . . . _ . _ . .

O' p, '

e qualified in accordance with the American Welding Society (AWS)

D1.0-69 or D1.1-72 Codes. PG4E would designate Pipe Rupture Restraints as Design Class I work requiring full Quality Assur-ance corrpilance. But it should be noted - at neither C.S. #8833XR

.. 1. . . .. , . .. . .

or C.,5.#8711,made any reference to or ma:.

any commitment to comply with.10 CFRCO Appendix B, the Codt f Federal. Regulations concerningJQuality Assurance requirements.

.L W...Kellogg (Pullman) would' erect'Pipt Rupture Restrainks l with little attention to Quality Assurance. On 9-19-73, the PG&E Project Superintendent sent to Kellogg a letter concerning Kellogg's Quality Assurance Program. This Letter stated that past audits conducted both py the Atomic Energy Commission and PG&E Quality Assurance Dept ihad disclosed nunarous QA deficiencies.

! These deficiencies \ usually fell'into two categories.

f,'

l 1. Failure to follow existing Quality Assu'rance procedures.

2. Failure to upgrade Quality Assurance procedures.

j "G&E requested Kellogg to place mere emphasis on their QA audit *

?

l program to eliminate most deficiencies before the next AEC and i

PG&Z audits.

On 10-24-73 Kellogg reported the results of their first .

r audit of the Rupture Restraint QA Program. One of the areas audited was " Adherence to Correct Installation Procedures."

Per the report all aspects of Rupture Restraint installation l were-checked to insure compliance to a letter (unavailable for review) approved by PG&E's A.G. Walters on 10-19-72. Tha audit

report stated that "it appears that Spec 8833XR and 8711 as i

stated in the body of the letter are being complied with completely 4

3

____$__-___-__----__-_-_____-1_---. - - - - _ - - - - W

e <

but, it would scem to be beneficial if all the references stated in the letter were condensed into a single procedure to be jp used a this complex." The initial Rupture Restraint construction

.did u.... no_t..have.gn approved Engineering.. Specification to direct the work but merely a letter referencing regt.rements for erection and Quality Assurance.

But PG&E would come to a different ccnclusion about Kellogg's Pipe Rupture Restraint QA program.

During October and November 1973, PG&E conducted an audit to verify that Pipe Hangers and Pipe Rupture Restraints were fabricated, furnished and erected in accordance with Spec 8711, PG&E and Kellogg QA manuals. It should be noted that PG&E did 3 not audit Restraints against the correct Contract Specification, Spec. 8833XR but against Spec 8711 which covered Pipe Supports and not Pipe Rupture Restraints.

s The same mistake was made in the Kellogg audit of 10-24-73. Why PG&E did not include Spec 8833XR ya which had placed Rupture Restraints under specific QA requirements is unknown. This would be a reccurringCproblem in the early years of construction. Rupture Restraint and Pipe Supports Sc,3 L g,

would often be confused as one and the same. They wouldgauditEb the same Spec, and share the same construction and QA requirements.

1 The audit disclosed that Kellogg (Pullman) and PG&E's General Construction Dept. departed significantly from the require-ments of the Specification and PG&E's Quality Assurance Manual.

Kellogg's(Pullman) Quality Assurance program did not comply with Section 4 of Spec 8711 and PG&E's Procedure PRP-4. It also l disclosed that the PG&E Mechanical Department's surveillance o A1[Bl%

program did not comp 1.y with Proceduro PRC-7 As a result, PG 's Project Superintendent stopped work o

on the installation.of the pipe hangers and rupture restraints and directed that,. corrective action be, initiated,to r,esol,ve , ,,

all; deficiencies and preclude recurrence.

.. The audit.Jeviewed Kellogg's (Pullmt i Quality Assurance Manual, with respect to the pipe hangers d restraints, for adequacy and compliance to Spec 8711 and Cu Procedure PRP-4.

Section 4 of Spec 8711 set forth the requirements of the standard '

" Supplementary Specifications for Contractor's Quality Assurance Program" included in Procedure PRP-4.

Kellogg's (Pullman) QA Manual copplied with Section 4 of the Specifications but the Manual did not specifically address itself to, nor completely apply to the control of pipe hangers and restraints. Because of this Kellogg (Pullman) had written an

" Engineering Specification", ESD223, establishing a QA program applicable to the control of hangers and restraints. The intent of ESO223 was to set forth procedures and instructions to the field i QA inspectors, engineers and foreman implementing the policy

stated in the QA Manual. The audit revealed that ESD223 esta- .

i blished QA policy in' stead of providing instructions on how to implement the policy. stated in the Manual.

L ESD223 did not meet all the requirements of Section 4 of l

the Spec. Deficiencies were noted in the areas of document i

i review and control, qualification of special processes and l personnel, work procurement control, receipt inspection of material identification control and status of material, nonconforming material l control, inspection and test records and inspection and test plans. The hanger and restraint QA program was found to be in

. _ _ - _ , . , . , . . . - . - - - - - -- -- - =-- - '

c .

violation of Procsdure PRP-4.

~

A separate QA Manual /QA Program was established for Pipe Supports and Rupture Restraints. This program was based on Contract Spec 8711 QA Requirements. Again C.S.#8833XR was ignored..... Rupture Restraint QA requirements were referenced in

\ CS #8,833Xf, not.C.S. #8711. No commitment was made to 10CFR50 5 45 Appendix B and/or ANSI Nap.2 QA Requirements. ANSI Nes.2 had recently come into being to provide QA ccverage for areas that fell outside ASME code QA requirements which Pipe Supports and Rupture Re-straints did. Alsoi Discrepancy Reports identifying and dis-positioning the discrepant item existing in work completed were to be initiated, and steps to preclude recurrence implemented.

Another item audited was the receipt, storage and installation i

of pipe hangers and rupture restraints.

The audit revealed:

1. Kellogg's (Pullman) receipt inspections were only chechs for road damage and completeness of material only.

Kellogg did perform surveillance inspections of stored assemblies.

2. PG&E Civil Dept, provided the inspection and documenta-tion to assure that procurement requirements had been met. Several receiving inspection forms which noted contingencies had not been completed. These items had not been placed on " hold" or withheld'from installation.

The Resident's Instructions did not require identification and segregation of non-conforming items. <

receiving reports for all restrainst could not beAdditionally,

!_ located.

3. Kellogg (Pullman) had not determined or received a written release from PG&E stating that the procurement i

requirements had been met.

4. Except for ultrasonic inspection, Kellogg documented their inspections on " marked-up" erection drawings.

The method of recording inspections and acceptance criteria were not set forth in an instruction, and the auditor had difficulty determining the inspection status.

The auditor found that not all in-process inspection of workmanship and technique required by the AWS Code were being performed.

5. Some. welders were welding materials of greater thickness than they were qualified.

6. Welding was not in complete accordance with the assigned weld procedures.

Several of the non-essential variables had been altered or were not being complied with.

m . _ _ - _ _ _ _ M ^d

7 Provisions for the installation and inspection of high strength steel bolts were not in accordance-with the AISC Code.

The recommended corrective action for these findings was the same'as for Audit Item No. I. stated as above.

- Another item audited was PG&E's Resident Mechanical Engineers surveillance system of'the fabricating,' furnishing and

installing of piphthangers~ and rupture restraints. '

The audit revealed that surveillance of the receipt and installation of pipe hangers and rupture restraints were performed by Power Plant Piping Group. The Resident's written instructions k to this group were set 6crth. in MF5-2. But MFZ-2 instructions did not specifically address surveillance of pipe hangers and restraints.

Corrective action was to issue written instructions for surveillance of pipe hangers and restrain ts Thus this audit revealed that containment rupture restraint erection was in -

noncompliance to Spec 8711 and presumably Spec E833XR, which had similar QA requirements.

It was during this same time frame that other problems l were identified in the Kellogg's QA Program. A Kellogg l Internal Audit dated 9-6-73 revealed that the N.D.E. Personnel '

Qualification Program was not included in the engineering specifications, . thereby making it part of the Kellogg QA Program and thus requiring PG&E approval of each page and each revision..  !

As a result of this audit NDE Personnel Qualifications Requirements l

were incorporated into ESD 235 and ESD 237, making these- i I

requirements part of the QA Program and subject to PG&E review l and approval.

In 1973 the American National Standards Institute (ANSI) would issue ANSI N4S.2.6, which defined an acceptable methed

Ih 6 i for ihplementing 10CFRf0 Appendix 3 requirements for "Qualifi- l cation of Inspection, Examination, and Testing Personnel for the Construction Phase of Nuclear Power Plants". Kellogg's  !

Corporate QA Manual would be revised in 3-19-74 and its Corporate l

FieldInstallationMhnualwouldberevisedon4-1-74toliplement

..,2.. r,_. < . - -- . . , . 2.

j j

ANSI N45.2.6.- A 12-12-74 Kellogg Interoffice Correspondence  !

I i

from the Corporate Director of QA, E.F. Gerwin, would only suggest/ recommend to the Diablo Canyon site QA/QC Manager that he implement ANSI N45.2.6 requirements. A subsequent Interoffice

~

Correspondence from the Kellogg Corporate QA Dept., dated 12-17-74, would direct the site QA/QC Manager to put into effect ANSI N45.2.6 "at your earliest possible convenience".

A Kellogg Corporate Management Audit of the Diablo Canyon job site on April 3,4, and 5, 1975, revealed nonconforme-ities in the area of " updating of Certificate of Qualification

(

Records"andrecommendedcompletereviewofpersonnelrecords[by the Field QA/QC Manager. Field QA/QC Manager J.P. Runyan t

responded to the Corporate Audit on an I.O.C. dated 5-13-75, stating, " Personnel records review has been performed and updated.

\

l We have also updated our records in an attempt to comply with ANSI N45.2. 6". Runyan,--on 6-15-75 would re'ise the ESD 237 Certificate of Qualification card for Quality Assurance Techni-l cians and Inspectors to read " qualified in accordance with SNT-TC-I l'K and/or ANSI N45.2.6." As a result, I believe that Field QA/QC canager J.P. Runyan deliberiately falsified QA Personnel Certification Records to give the appearance of compliance to p dhd

ANSI N45.2.6 requirements when no such compliance was implemented.

Runyan, in his 5-13-75 response to the Corporate Audit stated:

"It should be noted that it is virtually impossible to comply

' toe [lky to N45.5.6 because of experience requirements. We cannot hire personnel that meet the experience requirements for the salary sc~ ale we off'er. Even if the money was available, it would be difficult to find qualified people. We are taking the approach of qualification based on performance in a specific ,,

job." This was a nonconformance to ESD235 and ESD 237 QA/QC personnel qualification requirements, both ANSI N45.2.6 d SNT- Okh TC-IA qualification requirements and the intent of 10CFR/O App..B.

Criteria II, IX and IVII. As a result, the Kellogg attempt to upgrade its QA Program was a dismal failure resulting in falsified records. It should be noted that Kellogg did not revise its QA Manual-to reflect the attempted ANSI N45.2.6 com-pliance and that PG&E did not revise C.S. #8711 or C.S. #8833XR to direct compliance tb ANSI N45.2.6.

In August 1973., the U.S. Atomic Energy Commission issued j Regulatory Guide 1.29, which indicated that " nuclear power plant structures, syctems, and components important to safety be designed to withstand the effects of earthquakes without loss of capability to perform their safety functions". It also i indicated that pertinent requirements of Appendix B to 10CFR50 (Quality Assurance Criteria for Design, Construction and Operation of Nuclear Power Plants) would apply to all activities affecting the safety related functions of the identified structures, systems, and components, including their foundations and supports.

The discovery of the Hosgri earthquake fault off the enac- 110N -

_ _ __._. _.__.m_ . _ _ _ . . _ _. _ _ .u . _ . _ . _ _ _ . _

of the Diablo Canyon Plant placed the power plant within the Seismic Design classification established by the U.S. AEC in its Regulatory Guide 1.29 and made 10CFR O Appendix B .

QA Criteria a necessary part of PG&E's design and construction .

program. But PG&E did not revise its C.S. #8711 or C.S.

1. 883.3.XR to require,Kellogg's construction program to comply with the QA requirements of 10CFRSO, Appendix B. PG&E and Pullman have contended that the Piping construction program which was based on ASME Section III Code requirements meet the intent 1 h of 10CFR O, App. B. But the Pipe Support and Pipe Rupture Restraint construction programs were not based on ASME SEction III, b

and were not required by Contract Spec to meet 10CFR$0, App. B. Y LO The result was that pipe support and rupture restraint QA programs were not based on nor did they comply with the QA ^

g requirements of 10CFR O, Appendix B. -

x The seismic analysis and reanalysis to withstand a major earthquake resulted in redesign and additional construction of hangers, supports, and rupture restraints in an ongoing process.

With the confirmation of the Hosgri Fault in 1973/1974, there C

was an upgrading program instituted to beef up existing hangers and rupture restraints. This program was called the "Hosgri Rework Program." The reanalysis and subsequent work granted to Kellogg, including the Hosgri Program was performed by l Kellogg/ Pullman in 1975, 1976 and 1977. The erection of Pipe  !

Rupture Restraints expanded to piping systems in all areas of the  ;

power plant.

In 1974, PG&E contracted Nuclear Services Corporation -

to design the additional pipe Rupture Re traints s which were '

ret @irad 9ra M19fmlWwA . Crwuar

s,.a - _ _ _ _ . . . . _ , . - _ . . - _ _ . .-. ._ _ . . . _ _ . . _ . . - .

kh Bo5dr6P1 2;_ L ;n Bergen Metal Products to fabricate the restraints.

Kellogg/ Pullman would perform the erection of these Pipe Rupture Restraints. '

- On 1-14-74, Engineering Specification Diablo (ESD) 243, Pipe' Rupture Restraints, was~ issued by Kellogg and on 2-1-74 a revision.to the ESD was approved and published. Most of the ESD requirements were copied from PG&E Spec 8833XR and the AWS Code D1.0-69. The 2-1-74 revision to ESD 243 required all Rupture Restraint welds to be made with weld prodedure Code 7/8, preheat of 50 F minitum with welder verification only, no preheat check by QA other than periodic monitoring during welder audits, and no documentation of preheat or interpass temperature. Visual inspection of fit up and final inspection with ultrasonic examination of all full penetration welds was required. For over a year'these were-the only QA/QC requirements for welding on Rupture Restraints.

A ' problem which arose in these. years was QA/QC directing production work. The Kellogg (Pullman) QA/QC Manager issued i

l an Interoffice Correspondence on 1-31-74 stating that superinten-dents had complained of QA Inspectors talking to and giving work instructions to foremen and pipefitters. He stated that

'fiQ4 *from now on, no support or rupture restraint QA Inspector shall discuss any rework, defective support problem or engineering spec. requirements with foremen, general foremen, or pipefitters.

It was necessary on 6-17-74 for the QA/QC Supervisor to l issue an Interoffice Correspondence further clarifying the

, role of QA. He stated that QA is not an engineering service l

l 11 '

[ aL~J

y- - -. . - _ . . . . . . - . - - . . - . . . _ . - . . . . . . . - a.. .: =.-. . .... .

and then went on to state what functions QA would perform.

QA/QC was not to direct production work or to provide engineering services at the Diablo Canyon Plant. It is ques-tiened just how much of this type work QA/QC did. What functions QA/QC would perform would develop as the Rupture Restraint erection program progressed. -- - .c .

~

On 3-27-74 a Kellogg (Pullman) internal audit of the pipe support documentation of completed supports sub ncnmenc 4A> bb revealed several deficiencies. The corrective action for one of these deficiencies would later play a role in a rupture restraint documentation problem. The audit revealed that some process sheets did not have the proper amount of insp.ection poinhIEh8h. The audit's corrective action directed that "any inspection points that do not apply to a particular support shall be noted with a "N/A"." Thus inspectors were given the authority on pipe support process sheets to check N/A "not applicable" for inspection hold

~

points that they felt did not apply. The problem of N/Aing inspection hold points would arise in rupture restraints in the future.

l A problem in the rupture restraint weld documentation i

program would be revealed in a Kellogg (Pullman) internal audit of pipe rupture restraints on 5-13-74. The audit revealed inspector's " Daily inspection Log" whien showed field welds in rupture restraints. Their status was in compliance with kght ESD 24 . But the actual field weld process sheet used to document the individual weld did not show a date when the welding operation was completed nor whether a final visual inspection was performed. The audit also revealed that most RR field welds in cum M

the Unit i Auxiliary, Building showed poor workmanship. The conclusions of the audit were that "if possible, a date should be shown on the process sheet when an operation is completed" and that " field welds on th_e Unit 1_ Rupture Restraints in the Aux'iliary Building should be reinspected, and a modified Process She,et shoul,d be.made up to show 100% or final inspecticn of these weld's." No mention was made of the condition of Rkstraintsinotherpartsoftheplant. It would not be until May 1975 that these conclusions concerning weld process sheet documentation would be incorporated into the requirements of ESD 243. Rupture Restraints erection would continue with only cursory QA/QC participation.

On 12-24-74 Discrepancy Report #2654 was written on RR (QD0' 1031-5RT, Unit 1, W area. " Cracks were reported in base material 6" long at FW2C and FW2F. Ultrasonic examination revealed indechtions to be laminar in nature, 1" below the flange face.

Indentations were ground to remove and new weld metal was added. FIES 'I h This was the m& heel indication that a cracking problem was developing in rupture restraints. Many more similar situations would arise. .

In February 1975, PG&E would perform Audit No. 75-2 on Kellogg (Pullman) to verify that' piping supports and rupture restraints were installed per PG&E and Kellogg QA Manuals, Specs 8711 and 8833XR and the FSAR. The audit discovered de-parture from prescribed quality procedures in the areas of drawing control, weld electrode control, ultrasonic equipment calibration, and PG&E surveillance inspection documentation.

The audit stated, " individually, the departures were not of 13

_ W$

3- _

major significance; however, collectively the departures  ;

indicate the need for a more comprehensive internal audit system."

Since the beginning of construction in 1972, Kellogg (Pullman) had performed only two internal audits on rupture restraints. This was in October 1973 and March 1974. Kellogg had been performing internal audits but mainly on the erection of piping with occasional audits on hanger supports. Because of this PG&E audit, Kellog would begin to audit rupture restraint work more often.

As a result of a.Kellogg Internal Audit of drawing control for rupture restraints on 3-24-75, which discovered out of revision drawing being used for erection, the QA/QC supervisor issued an Interoffice Correspondence dated 4-3-75 directing all R.R.

drawing to be audited once a month by QA inspectors; that the Pipe Support Dept. (rupture restraints were included in this department) be,added to the Chief Field Engineer's drawing distribution list for R.R. revision update; and that out-of-revision drawings discovered be updated by the inspectors responsible. Thus it became the Inspector's responsibility -

to control drawing for rupture restraints.

By the spring of 1975, it was becoming apparent to Kellogg (Pullman) QA management that a lack of preheat for welding was becoming a problem.

On 4-25-75, the Kellogg (Pullman) QA/QC Manager issued an l

Interoffice Correspondence to all support inspectors stating that the A.W.S. Code required preheat when welding structural members if the material thickness exceeded 3/4". He stated that weld procedure 206 indicated preheat requirements for different matnRial thicknesses _and that khasa recni a-=-*e -- '*-4 "- - ihu

t welding processes. Thus in many cases two welding procedures  !

. t would have to be used to make a weld.

~

He added that the temperature should be maintained during the welding process; and that inspectors should note on the process sheet that preheat was checked and give th'e ' approximate temperature.

~'

' Revision #5 to ESb 243,' dated 5-6-75, added author 1zation

'tduseweldcode205o'r206forverticalbuttwelds,'d5f

~

angle g'usset plates, 30' groove welds with backing, and 45 0 gyggy, welds with backing. This revision also clarified and expanded welding inspection and documentation requirements to include seven sequential steps with six QC hold points (production could not proceed until the hold point was signed by QC) as follows:

1. Verify material, clean and fit up. (H.P.)

2. P. reheat temperaturg (H.P.)

Nod 3. Ec_ Lm Fama Ro o -T F A,55 (H.P.)

4. 10% inspect multiple pass fillets (H.P.)

5. Weld comulete

6. Final visual ( H . P, . )

7. N.D.E. completed weld (H.P.)

~

With revision #5 to ESD 243, verification of preheats became a QC function instead of a production function. Process sheets would now be issued detailing the operation sequences for each weld and specifying where QC Insgections were required.

. But this revision would not be fully implemented. Process sheets for rupture restraint #148, would have welding performed as late as April 1976 which did not comply with the requirements of revision #5 to ESD 243.

Another problem that arose was the fact that the process sheets listed field weld numbers but did not indicate the t

j(696 type of weld being made (fille /, groove, etc). This would cause problems at a later date when process sheets, field 15 d6N

_._m.. _.. . . - _ _ . . _ . -

r . .

)R M CS

.jo b layout cnd A actual field conditions would not match and the QA/QC dept. would perform a restamping program to make them match.

Even with revision #5 to ESD 243, preheating of welds remained a problem. On 9-17 and 19, 1975, PG&E performed a f quality control audit of the Kellogg (Pullman)~ company's welding on pipe rupture restraints.

This audit found that QA '

personnel allowed welders to weld without. verifying minimum  ;

preheat and interpass temperatures. As a result of this audits l

?

the Kellogg (Pullman) QA/QC Manager issued an Interoffice [

Correspondence, date 9-22-75, stating that welders were not preheating and that Inspectors were required to monitor preheat and interpass temperatures. He pointed out that these temperatures must be maintained during the welding process and when checked, recorded on the process sheet.

• Also in response to the PG&E audit, the QA/QC Manager i

sent~'a~ letter to PG&E, dated 10 75, stating corrective action had been taken to assure that preheat requirements were being followed and applied in compliance with established procedures.

This letter also stated a meeting had been held with the Superintendent in charge of Rupture Restraints to establish O.

production responsibilities with regard to preheating.

An official response to the PG&E audit was made by the i Kellogg (Pullman) QA/QC Manager on 10-9-75, when he issued DescrepancyeR port #2969 stating that rupture restraints in i

the field had welds completed without proper prehea't. PG&E's  !

official recommended disposition was to " accept as is based on i

acceptance of ultrasonic testing."  !

16

Tha problem of inadequate preheating of walds had now been l

officially. recognized by both Kellogg (Pullman) and PG&E, with corrective action promised. Up to this time Kellogg (Pullman  !

Field Engineers' had been'providing minimal engineering services i for welding rupture restraints.. These engineers had been pri- [

marily concerned with .the erection of piping and pipe hanger ,

supports. Because of the' continuing problems with weld I

[

cracking in restricted joints, the QA/QC Mana er on 10-2 -75, isQOd  ;

(QEST11tNktT sued an Interoffice Correspondence to QC Support Inspectors which j i

in effect ordered inspectors to perform engineering duties.

He stated that inspectors should take the following action in an effort to avoid the cracks:

1. Suggest to the production personnel that they use more heat, preferably 300' or more. He notes that this p is not required but is highly recommended.

2. Check to assure that the temperature is maintained r during the complete welding cycle. _ _ _ . _ _ _

~~~~

3. Recommend a welding sequence which will induce less {

stress.

4. After weld is complete let it cool completely before i final visual inspection then examine closely for  !

tight cracks. 1

5. Make sure that there are no visible cracks before calling for .T. inspection.

ph

. Suggesting to production personnel that more heat be  !

applied to welds and recommending welding sequences should 'I l

l l have been a designated engineering function. It was not, I

' and as a result of this correspondence.it became the QC .

inspector's responsibility in direct contradiction to the QA/QC Manager's directions of 1/31/74. QC Inspectors were now to assume engineering duties. This correspondence also l would tentatively identify additional reasons for the cracking j problem, welds in restricted joints and welding sequences. gg 17

- .~........---.:z. .;p -- :; -. . - .

= =  :- =--

?. -

l t

The weld cracking problem in Rupture Restraints would continue. Beside the weld cracking problem there would also be a problem in identifying welds after they were made. PG&E ,

during the week of Oct. 27, 1975, conddcted Audit No. h 75-4  ;

to verify compliance to PRP-4, Suppliers'/ Contractors Quality

~

" siiurance Prdgramd and'EsD~2df.Fdur items were audited with one f discrepancy found. The audit disclosed that the procedures  !

for identifying welds were not being implemented uniformly.

"5ou5 riiptur[ restraints were $udited for workmanship and on f two of them weld identification inconsistencies were noted.  !

Restraint No. 1047.4R7 had three welds stamped with a welders ID letters, but the process sheets did not ' '

{

reflect the welder's ID letters. On one of 'the above welds the j process sheet indicated that the ultrasonic examination had been completed, but the weld had not been stamped with the inspector's (Y) stamp per ESD 243. Restraint No. 1047,14Rt had two welds whichwerenotsgampedwiththewelder'sID. One weld process

$0h sheet indicated XT inspection but the weld was not stamped j to reflect this. The corrective action recommended by PG&E was for all welders and inspectors to be instructed on >

the requirements for stamp'ing and inspecting completed welds.

This problem of weld identification and documentation  !

was not an isolated case but effected almost all rupture restraints '

erected up to this time. The problem was not just failure of ,

welders to stamp their welds and inspectors to record the informa- ,

tion on the process sheets. On many of the restraint erections there were joint connections involving as many as 3 to 10 or

( more welded connections. All the welded connections in the j joint were given a single identification number. Then later j it was decided that each welded connection had to be identified, '

sotheprocesssheetswereamendedtoreadFWnumberA-for however many joints were involved. But the process sheet i did not necessarily reflect the correct welder for each welded joint. Then to compound the problem, Kellogg would initiate a stamp program as part of their corrective action to the PG&E I l audit.

I Kellogg's (Pullman) response to the audit, dated 12-1-75 l was that a field inspector had been assigned to review all  !

28 AcA

~~~ ~ ^^'

m___.__ r field records against completed work to assure correlation

~

Field records used were " Daily Inspection Logs" l between the two.

T nese records di'd not record what type and process sheets.

of weld was made (fillet, groove, etc.). So the field inspector

~ ..

could not accurately ^ match field records with welds and.sub-cequently many welds were, misidentified and misstamced.

This problem of weld identification would resurface on several future occasions and reveal that welds were not correctly identified and stamped.

The problem of prop.erly, filling out GA documentation was The QA/QC Manager issued an Interoffice a continuing problem.

Correspondence on 4/14/76 giving instructions on how to make changes to QA documents. " White out" was not to be used to A line through the incorrect entry and a correct entries.

All changes new entry for the correct information was directed.

had to be initialed and dated.

~

On 4/22/76 an Interoffice Correspondence issued to all field inspectors gave instructions concerning the proper filling It stated that process sheets will be out of Process sheets.

Lines drawn down the

' signed and dated in each required block.

column with initial and date at the top and bottom is not accepted. Any changes including N/A on the process sheet will be initialed. If a weld is cut out you will state the reason, initial and date. QA documents would be of little value if the documents were not filled out properly or the information provided did not include all data or provide accurate data. This problem would keep reoccuring.

On June 4, 1976, PG&E Engineering Research sent a letter kD es

to PG&E management at Diablo Canyon Plant concerning an investigation into the cahhh of cracking adjacent'to beam-to-column flange welds in Unit i pipe rupture restraints  !

4 (Df#315,8). A fai1ure analysis was performed on a portion of cracked welds and residual stress measurements were made

+

on the beam .the welds came ,from as well as in areas . adjacent '

to where the cracks were found. The results of these inves-tigations were:

1. The fracture is brittle in nature.

2. The fracture results from flame cutting of the welding relief hole in the weld.

3. There are high, up to yield stress level, residual stresses in the vicinity of the beam-to-column weld

, joints. These stresses are a result of the beam-to- e.

column weld.

4. Higher residual stresses, and cracks, appear to be associated with wide, greater than 3/4" wide weld passes.

The letter stated that the failures appear to be the result of a number of minor materials property, fabrication details, and construction sequence details that combined to cause these

. cracks. The letter then gave recommendations for repair and modification of welding and manufacturing procedures to to alleviate these problems. These recommendations were:

C

1. Preheat before all thermal cutting operations according to the welding preheat schedule for the thickness of material being cut.

2. Remove, by grinding or other mechanical means, a minimum 'of 1/16 inch from all flame cut or are gouged surfaces not to be incorporated in the weld.

3. The welding procedure should be modified to limit the weld bead width to 5/8" maximum *F or 2h" and thicker material in beam-to-column joints and other restrainted joints, the minimum preheat temperature should be raised to 3005 F, and a maximum interpass temperature of 8000 F should be imposed.

20 gcq

-^ - - -

~ ~ ~ ~ -

4. Where possible the weld joint detail should bemodified t '

This can be accomplished by using anarrower groove, a

- double-V weld preparation, or both, instead of the 45 single-V weld preparation presently used. '

Weld Code 7/8, a primary R.R. welding procedure, did joint not..: include in'its Weld procedure Specifications (WPS),

But Code 7/8 would now be used dataili for a double'V weld.

to make double V groove welds in nonconformance to the WPS.

Revision #7 to ESD 243, dated 6-10-76 was a direct result of the PG&E invest'igation of a cracking problem on rup ure restra 126, Unit 1 turbine Building. The revision added tdbular data for preheat and interpass bem gerature requirements during welding and thermal cutting. It added a requirement to clean by 3RWDry SunsMne a minimum of 1/16" from thermal cut surfaces whic not to be incorporated into a weld. The revision added minimal guidelines to dimension weld access re' lief holes.

Prior to this time a specific preheat and interpass te=pera-The weld procedure specification ture was not included in ESD 243.

was the control document, however, reference to ESD 243 was not included in the weld specification until October 1976.

There were four weld procedure specifications for rupture Weld code restraints with weld Code 7/8, the main procedure.

7/8 was originally two separate procedures identified as weld These procedures were approved Code 7 and weld Code 8.

on 11/25/69. Both codes were for welding carbon steel pipe On 12-10-73, using E7018 shielded metal arch welding process, the codes were combined and added carbon steel plate to the specifications. Weld code 7/8 was identified for use on h 21

- ~ ~ ~ ' ~ - ~ ~ ~

__ ...___ [____ _ - -

c. .

ructure restraints groove and fillet welds. Preheat requirements ,

ware changed to 50' F minimum with 175' F minimum for material that had a carbon content in excess of 0.30% and 1" thickness.  :

Interpass temperature was indicated to be SO C F minimum. A 10-15-76 revision to weld Code 7/8 stated See ESD 243 for AWS Welding",

raferring to structural steel welding (which rupture restraints -

. This revision also stated that the procedure was qualified was).

to allow welding of unlimited thickness on structural members I

under AWS requirements.

i Weld Code 92/93 was similar to , weld code 7/8 in that some r

?

i of the welding techniques were the same. Code 92/93 was qualified for open butt welding but was used to weld groove welds

-with a backing strip. This weld code was used during peak  ;

i workload periods because there was' no requirement to re-qualify ,

i welding personnel. A problem would arise with process sheets l

i RCD referencing Code 7/8 but eet requisition referencing Code 92/93.  !

+

Production and QC substituted Code 92/93 for Code 7/8 to f expedite the construction process.  ;

t Weld Code 205 was developed and approved for flux cored arc welding of carbon steel to carbon steel for structural steel i

[

only.

I

' Weld Code 206 was developed and approved for gas, metal arc  !

l l

welding carbon steel to carbon steel for structural steel only. ,

l i

Kellogg (Pullman) established ESD 219 for " weld procedure ,

monitoring"in 1973. This procedure was originally established to monitor Class I pipe welding. Revision 5, dated 6-17-76, added l rupture restraint welding as Class I welding and directed that r l l i

ESD 243 would be the applicable procedure for preheat monitoring f . - - - - - - . _ - - - - - _ - . - . - - _ _ _ _ . _

for structural walding. ESD 219 also stated that Walders and inspectors shall monitor the interpass temperature of all Class I Paragraph 3.3 of walds for compliance with the weld procedure.

ESD 219 concerning " Pre-Heat temperature" states "the mini 6um ..:.s

-.a... .

If the air or F.

pra-heat temperature on this project is 508 matal temperature is below 50c F, pre-heating is required. -

.a. .. . . . Pull-Air temperature shall be monitored by wall thermometers."

12-24-80, found there man Internal Audit report #80, date of ware no wall thermometers evident in the powerhouse which could ba used to monitor air temperature to determine if pre- heating ,

was required.

Corrective action was to order thermometers and, upon receiving them to implement ESD 219.3.3.

Nine months later on 9-22-81, Internal Audit Report #94 QC, would report that wall thermoieters were received by Pullman f .

calibrated but never issued to the field for implementation o ESD 219.3.3. It was not until November 1981 that wall thermometers  :

monitor air temperature.

were placed in the power house to It took. 8 years for the requirements of ESD 219.2.2 to be in-

.It took 11 months af ter the noncompliance _was found plemented.

In my pro-before corrective action was actually itplemented. I fessional opinion, this was inadequate. implementation of I

Quality Assurance requirements.

3-23-7f -fo '7 'Jo '75

',.^N From G-22 72, a total of twenty four discrepancy reports ~

t i ts.

were generated which involved cracking in Pipe Rupture Res ra n ,

i On 7-22-76, ESD 243 was revised to authorize field modificat on ,

of weld joint detail during weld repairs and/or new weld preps.

This was done to reduce the volume of weld metal deposited, 23

, k

C 2

1.e. narrower grooves, double bevel grooves versus single bevel grooves, thereby helping resolve the weld cracking problem.

The continuing problem,of weld cracking raised the question On 9-9-76, an of when,the final visual exam,should take place.

IOC was issued to all rupture restraint inspectors instructing

~ '

them To' sign process sheet sthpi #5 I weld complete (not a QC hold point) when welding was complete. Step #6 - Final Visual, was not to be signed until the weld had cooled to ambient temperatures and then the inspector was to check and see that the weld area was clean of slag, scale and smoke, and that it U i The inspector was then to complete his heb was smooth for #T exam.

final inspection and sign the process sheet. This would help ,

~

T inspectors to more readily detect cracks in the_ welds. ,

On 10-7-76, DR#3295 and PG&E DR#M-3192 would report x

1200 welds to " rupture restraint structure members" without the  ;

application of the required preheat. The welds' involved attachments such as temporary lifting eyes, nuts and bolts,' shims, rod eyes and hinges. The weld sizes ranged from single pass 1/8" fillets to 1/2" fillets. Base material thickness, which governed preheat .

There were no process requirem.nts, ranged from 3/4" t6 6".

sheets issued to contro.1 the welding or any other QA/QC documentation.

DR#3295 required that these welds be examined by magnetic Three particle testing to determine if they were acceptable.

Based hundred fourteen welds were examined and found acceptable.

i on the acceptance of these welds, the remaining welds were accepted as is without being tested.

Rod

~

24

m -

- = - - -

Up to August 1977, there are no records of Nuclear Regulatory C:mmission Inspectors involving themselves specifically with rupture i

Then on 8-2-77 an NRC inspector made-an inspection rastraints.

t The inspector fb of the Benf 9 rupture restraint:en' Unit I piperack.

.The inspector found what he believed to be undercut on JW40.

He found on process sheets clso found documentation problems.

for FW40 and 41 that the final inspection was dated one day prior to the fit up of the weld joints. He also found another process cheet with the final inspection hold point "N/A" by Kellogg Inspector Mullis.

Kellogg (Pullman) issued DR#3449 to report and resolve the The follhwing corrective action findings of the NRC inspector.

was taken:

1. FW40 had weld metal added to fill the low area at the weld

~~

edge.

2. FW 40 and 41 were reinspected and the dates correcte and a random reinspection of a minimum of 20% of all welds accepted by Inspector Lindell was performed.

3. All process sheets reviewed in #2 above which had "N/A" inserted in inspection points were reinspected and if(This action i required, repairs made.was more suspected of N/Aing process sheet op Mullis who was caught N/Aing by the NRC.)

4. Errors found in stamping of welds duringThis reinspection requisitions and process sheet documentations.

would involve 43 welds on Bent 4.

- Inspector Lindell had not been employed by Kellogg (Pullman)

Inspector Mullis s cince 9-3-76 so no action was taken against him.

had no explanation for entering "N/A" in the process sheets.

Eullis was then fired for failure to comply with established procedure.

On 9-12-77, an:IOC was issued by the QA/CC Manager to report field inspectors on meeting with Pullman Power Products (Kellogg) ikh r q,

g._____._______-.-~~:-.,-

f.

r i

These meetings pointed out that field r

on August 8 and 9,1977. f it  !

inspsetors did not have the authority to N/A inspection po n s.'

They were to advise welders and fitters of the hold point

• ' l jaqulFements 'andTo~perYorm' r'e'q'uiredin' spec't' ion's' Es soon Also discussed was the termination t Epossible after notification. i ation and the work j

of. Inspector Mullis, the reason for that term n i rcquired to correct the situation.  !

showed the NRC that the company had  !

By this Pullman (Kellogg) ,

implemented corrective action for QC inspector's failures j Inspector Mullis was a scapegoat f to comply with procedures.  !

to cover up bigger prob 11ms.

d for N/Aing a final  :

. Inspector Mullis cannot be excuse l inspection point, but what about extenuating circumstances?

k l Inspector Mullis was doing more than just QC inspection wor . l 6 6 ~

fb' In the Unit #1 /E, /W and piperack ar'eas, Inspector val Mullis l was performing engineering and drafting work with the.appro f n

QA/QC Management and Production Management.

of Pullman (Kellogg) f 10-23-75 from the QA Q/C An Interoffice Correspondence dated duties l Manager had directed QC inspectors to assume engineering j t than required to of telling production personnel to use more hea ,

their make welds and to tell production personnel how to make walds by recommending welding sequences which would induce l Inspector Rullis assumed the engi- l less stress in the welds.

l

-neering duties. This instruction was in direct contradiction to earlier QA/QC correspondence dated 1-31-74 which stated foremen and l

-inspectors were not to give work instructions to td ,

t pipefitters, and to correspondence dated 6-17-74 which sta e Why was it l F

Quality Assurance was not an engineering service. \DYk ,

26 ^ - - ~ - - - - ~ _ _ _ _ _ ,

~

..a__ .~

. 3 n:cassary for QC inspectors to perform engineering duties? ern f

Yh;ra were field engineers on the jobsite, but their main conc '

They gave little if was tho erection of piping and hangers. straints.

any engineering direction to the erection of rupture rePullman

. = - . - =. --

Rupture restraints had low engineering priority because lis to l

u. ,.  ;

(Ksilogg) management had instructed inspectors like Mul ..

.. .- . l Af ter the NRC incident provide the engineering services needed.

i cnginsering would take a more active role.

In addition to performing QC and engineering duties, Inspector he worked on.

Mullis did As-Built drawings of the rupture restraints f the rupture Thoss drawings showed the as-built field conditions o tion rostraint as well as numbers assigned to each welded connec l Inspector Mullis for documentation identification purposes. i  ;

drcw many of these as-built drawings and they are the baslayout field s I

+

for the current rupture restraint documentation packages - .. {

i drawings.  ;

Inspector Mullis was fired for NAing an inspection k y for an point, ,

'yat QA/QC Management on two occasions stated it was o a Interoffice Correspondence dated 3-27-74 inspector to do-so. f stated "any inspection points that do not apply to *

(attachment 3A) Interoffice a particular support shall be noted with S a "N/A"."

kg A 120) stated "any changes Correspondence dated 4-22-76 (et*=chmea4r NAing including N/A on the process shee t will be initialed..." I i

inspection points on process sheets was an accepted in rupture pract ce on i

supports which inspector Mullis decided to imp ement

~

rastraints. d documenta-  ;

So the first NRC audit of rupture restraints reveale ize .

e tion problems and field welding problems but failed to recogn hk f

" N the QA program, quality control inspectors - - - - _ _ _ _ ,

doing engineering and drafting work. i Inspector Mullis assumed duties and responsibilities outsid  !

I Pullman (Kellogg) management knpw his assigned QC functions.

Maybe  ;

cnd approved of it until Mullis was caught by the NRC.

l 3.

the reason Inspector Mullis NAed the inspection point was that he was so busy doing engineering and drafting that he didn't have time for quality control.

It should be noted that on 5-17-77 an interoffice correspon- '

' dance issued by the QA/CC Manager stated that Inspector Mullis f SNT-TC- l "through daily demonstrations meets the requirements o .

I IA..., ESD 235, ESD 237 and KFP6 " Evidence of Continuing Satis-i j

factory Performance"." Two and a half months later he was fired for failure to comply with established procedures.  !

ESD 243 was revised on 1-19-78 to add the requirements for

• the Field Engineer to review all drawings and h itiate all Field t

It added a requirement for QA review of process t Process sheets.

sheets prior to issue for work and revised the field process sheet to include the weld symbol, thickness of material and QA review

[

ontries.

QOb On 7-20-78, DR#3683 reported a lam lar tear which opened t Subsequent l during repair of a weld in the Unit #1 piperack. -

l NDE and metallurgical studies by PG&E revealed a generic problem On 10-3-78, PG&E cssociated with highly restrained joints.

issued non-conformance report #DC1f-78-RM-008 which identified g#

I that welds for pipe rupture restraints in materials greater than

/

! 11f' thick had developed cracks.

On 3-23-79, PG&E issued non-conformance report #DC1-79-RM-006, kON

,. =

-- , -=--=='

which identified numerous welds that developed cracks after On 5-7'-79, NCR#

completion of welding and final examination. '

DC1-79-RM-007 was issued, which identified that further inves- l j

.tigation.had found rejectable linear. indications in.other rupture ractraint weld joints.

On 6-6-79 PG&E issued NCR #.DC1-79-RM-010, I

i which identified that. nondestructive and destructive testing had ,

[

found the existance of rejectable defects in field welds. This [

NCR resulted in an extensive program of investigation, evaluation cnd repair of rupture restraint welds. On 6-21-79 PG&E issued l NCR # DC2-79-RM-011 which identified welds in Unit #1 with pML f f

rajsetabledefects,andthatthesameors4gilarfonditionsmay i i

cxist in Unit II. i j

The major problems causing rupture restraint weld cracking- r.

as determined by PG&E and Pullman were:  !

i

1. Joint A. Massive Desienweldments, 5" deep x 4-5/8" wide with 45, single bevel grooves that would shrink unrestrained '

about h" in a transverse direction, instead wereAll 1 i

- totally restrained by high columns and beams.

g dk. ppotentialshrinkageistransformedintoresidual  ;

stress and/or cracks.

Highly restrained joints with heavy sections attached to relatively thin sections. Lateral reinforcement b.

stiffers, 2" to 3" gusset plates, were welded exactly i l opposite, both pulling on h" to 3/4" thick webs / and "

flanges.

c. PG&E Department of Engineering Research would develop their investigation around four additional welded l

connection joints classified by degree of restraint.  ;

i t

1 f 12. Base

a. Almost Material all cracks originated as lamellar tears in A441-l and A588 steels used in highly restrained joints.

I

b. Some materials had excessive rolled laminations.

c. PG&E supplied base material'that was inadequately d identified prior to implementation of QA verification of base material.

d. Low melting point alloys formed with copper (in A441) and sulfides triggering tears.

[) I

3. Indiscriminate Material Removal

a. Large dtstructive test samples were removed.

______'e' b Some sections were essentially destroyed chaSt@ cracks. imm

other joints in the sama structuro.

4. Inadecuate Preheat a.. Material type beingand Interrass welded was not Temeerature included as an Control As element of planning for rupture restraint work.

a result, sufficient controls were not established for preheat and interpass temperatures. +

b2. Ambiguous terms and phrases.were copied from PG&E specifications, with inadequate implementation of AWS code requirements regarding preheat and interpass

- . temperatures.

. both A major crack repair program would be initiated in Units of the power plant in March of 1979. The Pullman Field QA/QC Manager stated in an IOC dated 8-28-79 that an estimated 40,000 man hours had been expended to date and that only approx-Rework would imately 50% of the work in Unit I was completed.

continue'in Unit I & II until 1981/1982.

Therupturerestraintcrackrecairorogramwouldresultin't1AQ copRuc T fo N rN In May 1979, major changes in the Pullman inst;uctIbn program.

Pul16an would-issue a special welding procedure to make the wald repairs. Welding technique Specification #AWS1-1 was formulated to clarify the technique for application of weld code 7/8 AUS1-1 and other similar.

procedure as applied to AWS welding only. B edeb Cos go4 their t:chniques were based on PG&E recommended procedures *L4hg The technique gave very i

analysis of the cracking problems. .

But detailed parameters for making the crack repair welds.

these techniques were not applied to the general rupture restraint construction program. Weld Code 7/8 would continue to be the l

l Prior-l primary welding procedure for general RR construction.

4g to1979rupturerestrainfweldershadbeenqualifiedtothe As a result of the crack repair program ACME Section IX code. "f C welders would now be required to qualify for the AWS Code d,cvk requirements.

l 30 Nd

a _.

. . Al .: Q===:=:  :

h to l A' number

< f of changes were made concerning the NDE requireman f

C.S. #8833XR was revised to require l lfer rupture restrainte. # ~ MON t r welds -

that all completed full penetration and partialyenetra i inspected.

i il j

and fillet. weld h".and. larger shall be magnetic part c e these re-l L' ,J

-Pullman would~ prepare a QA Instruction #143 to implement  !

l

1. 243.  :

quircments which wouldi eventually be incorporated hinto ~

ESD i al j

Whnn the instruction was submitted to the PG&E

~

d Resident l

{

Engineer for his approval he would. amend the instruction.to rea .

, f y

"cIl partial penetration welds h"'rjid larger" would require f Pullman would implement the PG&E n;gnstic particle examination. NDE l rovised QAI#143 and for the.next two years would perform ts.  ;

i whichdidnot.,complywith/t.herevfEedC.S.#8833XRrequremen f ired QAI#143 to In August 1581, PG&E recognizedc its %rror and requ ,

f p  ;

be. revised to include all partial penetration welds to be A reinspection program was initiated magnetic particle examined. f to identify the welds not magnetic particle tested. f PG&E provided to Pullman tihe NDE procedures to be used  ;

I However, Pullman Internal Audit for magnetic particle testing.

CLXXVII, dated 9-25-80 identified that PG&E had provided con- '

flicting procedures for Pullm'an to use./ ,PG&E had directed that

'011 rupture restraint magnetic particle, exams were to be perform This procedure stated that to PG&E's DER NLE procedure #3212.

the preferred examination was the Yoke method per PG&E DER N But EG&E had provided Pullman with a DER NDE procidure #3204.

, e , PG&E had stated one procedure #3205 which was a prod. method. dif-m3thod was to be used but had provided a procedure for a l Audit, PG&E would forent method. ,As a result of the Interna v ,

direct' Pullman to use the Prod. method.$ O

' man to use a PG&E ultrasonic proce- -. .-

e,.

, , v

$dra #3523 to examine only full penetration welds 9/16 This would'not comply with C.S.

Bront;r effective throughout. l1 connections

1. 8833XR requirements to ultrasonically inspect a. -

This conflict between C.S.

utilizing- full penetrations-welds.

V be identified 2

C8833XR requireiiaents and PG&E #29,iTdated procedure would July 1982, in' Pullman's Unscheduled Internal the Audit non-conformance but both Pullman and PG&E refused Not untilto address 1984 when '

to Centract Specification requirements. d hallcgationsofnon-conformancetocontract/Trequire uld PG&E revise mada to the Nuclear Regulatory Commission wo C.S. #8833XR.

g_.'

\

s O

t i

I l

32 r

i l .

In 1982 I identified in Pullman's Internal Audit

1. 101 that ESD234 Ultrasonic procedure had not been properly ESD234 had been used prior to 1979 to examine

. qualified.

IA#101 iden-all full penetration Rupture Restraint welds. i i tified Ehat ESD234 did not have ProcedureS.8833XR Qualif cat on R: cords documenting a Procedure Qualification Test.C. ,

(including ^NDE) to have qualifica-rGquired all'proceduresThis problem may have contributed to the weld tion records.

cracking problems. g,gg The Rupture Restraint 1eek Repair Program was not h' There the only major problem with Pipe Rupture Restraints. ifi-would be a significant Quality Assurance breakdown ident By 1977 ed in the Rupture Restraint Construction Program. i lties PG&E was concerned that Pullman was experiencing diff i cu in performing work, that was constantly changing per requ re-to qualify standards that monts at the direction of PG&E, i h the would allow PG&E to enter into the later hearings w t be NRC with complete confidence that Units I and II would PG&E requested Pullman to have acceptable for licensing. Pullman an independent audit performed of its QA Program. lif.

contracted Nuclear Services Corporation of Campbell, Ca to perform this audit. 20, 1977, Nuclear From August 22 to September t Services Corp. audited the Pullman Construction Program a the Diablo Canyon job site. The basic conclusion reached 0 .

by NSC was that the Pullman QA Program did not meet 10CFR5 NSC summarizes Pullman's problems, Appendix Requirements.

as follows:

1. Prior to early 1974, there is little evidence available to verify the adequacy of the work per- lkU Theavailableevidenceindicate,[sthat formed.

only a rudimentary quality control program existed and that control over the production organization was minimal. NSC concluded that there was no con-fidence that welding done prior to early 1974 was performed in accordance with welding specification requirements .

E _ f.._..._______.-,

• e 2.From early 1974 to late 1974, there is evidence available to verify the adequacy of the work per-formed. The available evidence indicat,es that con-9 trol was achieved,of.the materials control program s

and the welding control program.

.. .3. Fr'om late 1974 to the present, an increasing amount of documentation and records has been gen-erated to verify the adequacy of the work performed.

The available evidence d'emonstrated that an increas-ingly more stringent quality program has been placed into effect and increasing greater control of the work effort has been achieved. However, the present _

program and controls still do not meet 10CFR50 Appendix B requirements.

As a result of the 1977 Nuclear Service Corp. audit, t PG&E's QA Department would perform Audit #80422, issued 6-13-78. PG&E's conclusion was that the QA Program knplemen-i ted by Pullman essentially fulfilled contract requirements -

and meets requirements of the ASME Boiler and Pressure Vessel

', Code, 1971 edition. PG&E stated that the 1971 code was consistent with the requirements of 10CFR50 Appendix B. What PG&E and Pullman failed to recognize was that only Pullman's Piping Construction program was ba ed on the ASME Code QA lh[MV requirements. The Pipe Support and Pipe Rupture Restraint S N h*T 10 N % -

QA pr grams were n t baped on a -acecri:1 code or standard OL Y and there was no commitment to 10CFR50, Appendix B for these programs.

One of PG&E's audit findings was that Pullman audits performed to verify Unit II hardware items in early 1978 did not effectively evaluate the quality of their work.

Pullman had audited 122 hangers, restraints, and snubbers and 77 sometric drawing packages and found no discrepencies, l Yet when PG&E re-audited half of the items inspected by l

l Pullman, several discrepencies were noted. The result was l

that PG&E ordered Pullman's corporate staff to perfrom I another audit in the summer ofl978.

The PG&E Audit #80422 would generate two Non-ctnformance Reports. NCR#DC-78-RM-004 identified that There Pullman's QA Program was not adequately defined. h Q A requirements of t e w;re procedures whi,ch implemented contracc but are not identified as part of the program end revisions are not ,c,ontrolled by the program.,PG&E

. hich manuals ,and procedures found that it wa,s not clear w The corrective w;re applicable to specific activities.

cetion was to write a program discription that would the documents to be considered part of the cicar1Y identify i hy total quality assurance program and establish the he rarc of the documents.

The second part of the NCR addressed Pullman'sThe scope inadequate corporate and Internal Audit Program.

of both types of audits had not been established, and there t was no detailed schedule developed to show that all aspec s Audit records indicated of the program had been audited. No that all aspects of the program had not been audited. d management audits h,ad been performed on pipe supports an An unofficial, unapproved internal audit rupture restraints.

cchedule existed, butCs) it had

%e.

not been followed consistantly and few ESD's appeared +1th schedule. P.G.&E's qQ(

A second NCR #DC-78-RM-005 was also issued.

review of prodedures and work in progress indicated that i

Quality Control inspectionst\SSo indeoend ace A G D the from scheduling program as written. and .

iMW l W a production pressures was not accured by i

Procedures did not clearly indicate that it was the-Pro-ductions Department's responsibility to read and use the pro-cess sheet insuring that steps were performed in the required sequence and that hold points were observed.

Four Minor Variation Reports would be issued to It should be noted that deal with specific discrepancies.

identified some Pullman inspectors who werelnot P.G. & E.

qualified to ANSI N452.6 and recommended that the Pul man

[

l inspector certification card should be amended l to2.6,eliminate l

the claims that inspectors are qualified to ANSIt45.

or inspectors should be qualified in accordance with its h I

L remirements . _

~ --

-w...--..--

> e InNovember1978anfIOCfromPullman'sCorporate Sanior QA Engineer to'the Director of QA confirmed P.G.&E.'s audi,t findings concerning Pullman's Corporate audit program.

~

The IOC stated that tihe.."Diablo Canyon Project has been s.

. .. . . , . - . , . . ... .. . . . . . . . . . .., The entire auditid extensiv61y only in hardware areas.

The IOC would also state, program has'not'been eva3.uated."

"In the past, Pullman Power Products did not conduct audits

~

br~prac~tices'tb ASME or 10'CFR 50,' Appendix'B."

A'lso in November 1978 Pullman implemented corrective action to NCR#DC-78-RM-004 by issuing a QA Program Description.

Pullman deleted the Pipe Support / Pipe Rupture Restraint QA Manual from its QA Program.

In its place there would only be The QA Program Description stated, "The basic one QA Manual.

document for the QA Program is the Pullman Power Products QA Manual.

This manual was written to conform to the requirements of ASME Section III 1971 for piping fabrication and installation. Organization, Many of the requirements of the piping manual Weldsuch heat as:

treatment, Welders M NDEj Calibration, Weld Rod Control, Not all qualification und audits are applicable to other work.

the requirements of the piping manual are applicable to the full scope of work. Where these exceptions exist they are indicated by subtier documents such as separate QA plans, ESD's'or QA instruction."

'The QA Program Description listed a number of subtier documents as applicable to Pipe Supports and Pipe Rupture Restraints.

ific list of But nowhere ib the.QA Program Discription is there a spec the piping manual requirements which are applicable to Supports -

and Restraints.

The Discription states that many of the require-ments of the piping manual are applicable to other work but it

,Also there is no i fails to specify which requirement for which work, committment in the QA Program Description to 10 CFR 50 Appendix ASME B

for the other work areas which fall outside'the scope of the The result is that to this day Pipe Rupture Section III QA Manual.

Restraints still do not have an adequately defined QA Program which I

10 CFR 50, Appendix BOA AN9 07M A h>Miou A( cong,. AgAug, is based

%9c As a result . the P.G.& E. QA Department Audit #80422, g dated.6/13/78, which found that Pullman's Corporate audit performed in early 1978 "did not effectively evaluate the quality of their

& E. to send adlitional staff work", Pullman.was required by P.G.

N f@rm "an overall assessment of the situation"

e l

to determin.e whether additional reinspection should be performed and l

tha scope thereof.

Pullman's Corporate Management performed the The purpose of Audit #7177-3-78 eite audit from,7/10 to 7/20/78.

astoverifyandfvaluatefieldinitiatedcorrectiveactionthat I rgsulted from the Nuclear, Service Corporation Audit of Pullman, ,

f.o verify th,e adequacy of'the Quality Assurance Program implemented cndthe quality of hardware installed y primarily in Unit 1.

~

l Pullman 5' Aud'it #717'7--3-78 would result in 43 Audit Action

' ^

i R quests requiring corrective a'ction to improve the adequacy of the QA Program.

Criterion I of the audit would verify 24 of the Nuclear Service Corporation Audit findings that had been or would There findings included:

require corrective action. responsibilities h6b 1. Description of individual , O_

are inadequate.

2. Hydrostatic testing interface between P.G.&E, and PPP lacks adequate control.

3. Interf ace between PPP Corporate Organization and Field Organization is not described with respect to Field Purchases and Corporate QA Auditing of those suppliers.

4. Indoctiination and training program requiremerits for personnel involved in quality related activities are inadequate.

5. QA Document Control Procedure does not have provisions for training and familiarity in the implementation of pro-cedures.

6. Activit'ies affecting quality are not described in pro-cedures.

7. No control exercised over ESD procedures

8. No procedure for control of QA. instructions.

9. Procedure detailing review of Hangers and Pipe Supports is lacking.

10.No procedure exists prohibiting the changing or alteration of key documents.

11. No procedure specifying who is responsible for 90 Day Welders Log.

12. Random sampling of welding in process not documented.

13. There is no procedure for preheating of weld joints.

" - . ~ _ . - _

14. ESD 231 does not provide enough information for HOT f f

and COLD bending small bore pipe.

.. $. h ,15. Lagk.of identity.of.

.e Hydr Ntatic and heat-treated-

...t. .... .

r. .-.

guages with applicable inspection reports. ., l m

16. ESD 213 does. not contain provisions for reporting

,. . g preandpfstcalibrationvalues.

., N "

pik 17.'Hydrbatic Test Procedures did not cross reference l each other.

18. No procedure ,

for filing, storing and protection of records.

gob 19. No prhcedure or checklist to define scope of field conducted internal audits.

Criterion II of Audit 7177-3-78 reported .c a significant problem Information referenced 9

4 in the evaluation of the Piping Ipo's.

on the Field Installation Instruction (Drawings) did not agree with information published on the Process Sheets.

Criterion III of' Audit 7177-3-78 reported numerous individual dcscrepancies of Hahger assemblies but did not report any program daficiencies. Of the 43 Criterion IV dealt with Rupture Restraints.

Audit Action Request generated ~by Audit 7177-3 ',8, 20 were written cgainst Rupture Restraints. A' significant QA Program deficiency The was identified in the Rupture Restraint construction program.

corporate auditor concluded:

"The rupture restraints documentation package cannot '

be used for an adequate audit. It was pointed.out that additional drawings are available. The only way some of these restraints could have been installed is by the referenced design drawings, however we were in-formed by site personnel that other drawings exist that could effect the final installation. These additional It drawings are not referenced within the RR package.

is obvious, and site personnel agree, that this is a definite problem in regards to drawing referencing. 0A 40E site personnel also have problems getting documentation to properly match final erection due to lack of "as built" drawings. It was pointed out that there is a lack of proper

" &;cnRR/Y-n PCMB P.G. & E. and site PP/QA."

O - .

was Hanger Drawing Control Criteria V of Audit 7177-3-78 endAfBuiltingProgram,andtheauditconcludedthereis .

cvidence that adequate control is being exercised.

Criterian VI of the audit was a review of Non- ht Conformance-Reports and concluded that there was evidence t a chs uc ti:n of the DR"S were " generally" followed tha recommended Mi;fossmcawithh theturenecessar e

4

-n of Ehe work performed.

Criterian VII concerned Management Audits and found that cudits were not performed in accordance with the QA Program rcquirements of every six months. concluded that the area of main concer Audit 7177-3-78 It was recommended that was associated with Rupture Restraint. g4 cFieldInspectionProgrambeinitiatedintheareaofRup{ure A.A. Eck, who was the head Rcstraints for both Unitconcludedthatthg"QgalityAssuranceProgram I and II.

cuditor for this audit, l ca implemented basically meets the ASME Ec.tcr and Pressure Vess Code Requirements, 1971 edition."

h Although significant QA problems were identifiedtinclaimed te Rupture Restraint Construction Program, PullmanSME Managemen code re-the QA Program as implemented basically meets the A ht A possible reason for this could have been had theno fact t a quirements.

piping, which was based on the ASME code QA requirements, s l cignificant problems idsntified. or 10 CFR 50 Appendix B or were not based on the ASME code, It was ANSI N45.2 QA requirement, had significant QA problems. d national

• their absence of committment to the federal code an Rupture standards which resulted in a deficient QA program for Restraint.

P.G. & E. now was acutely aware that Pullman'sP.pipe On 10/26/78, G. &rupture E.

~

restraint program had been out of control. This NCR was concerned

' issued Nonconformance Report #DCl-78-RM-009. i of withPullman's documentation for the erection The NCRandwould inspect on identify:

rupture restraints inside Containment I. d Work "1. Documentation shows work complete, correct is not correct.

records are incomplete or nonexistent."& E. would find th But P.G.

NCR#DCI-78-RM-009 beyond Containment I and documentation problems.

L._

.=. -

was cancelled and in its place P.G. & E. issued NCR# DCI-79-RM-003 on 1/24/79 for all Unit I Rupture Restraint work, and NCR# DC2-80-RM-002 Both NCR's en 11/19/80 for all Unit II Rupture Restraint Work. t _. ; ,,

1. DC,I-79--RM- 0,03, and #DC2-80-RM .002 would identify:.

"1.a. Documentation shows acceptable bolted connections.

3.

However, there are cases of out of tolerance gaps

,n existing under base plates, nuts not bearing against splice plates properly and nut not engaged per re-quirements.

b. Documentation.shows acceptable welded connections.

However, there are cases of materials and welds not conforming to the specifications.

c.

There are bolts that have " torque seal" which.indi-cates tensioning and inspection, however, inspection records do not exist."

PG&E would identify the cause of the Nonconformances to bein- the fact that " Pullman Power Products Rupture Restraint Program has had odequate design change control, inspection performance and control."

Another cause not identifie'd by PG&E was the fact that Pullman's

'RuptureRestraintconstructionprogramwasnotcommittggotheQA 10CFR50 Appendix B or ANSl::C.1 codes, requirements of the ASME, for the result being a totally inadequate Quality Assurance Program the erection and inspection of Rupture Restraints.

The corrective action required by PG&E was that " Pullman chall perform a documented inspection of all bolted and welded connections and applicable documentation, required by the Specifica- '

tion, as set forth in approved contractors ESD8s, in order to:

1. Identify connections which do not conform to specifica-l i

tion requirements and gg

2. identify connections which do art require documentation."

h It IdentifieddfficientconditionswouldberesolvedpertheNCR's. he Nuclear iid not report these NCR's to

'should be noted that PG&E j kON Regulatory Commision as a "bCFR Part 21 Report ble item.

Pullman would issue on 2/16/79, ESD 273 "0A Final Walkdown cnd Documentation Review-Rupture Restraints" as the procedure to The final walkdown ect the reinspection of Rupture Restraint work.

.e l inspection and documentation review commenced shortly thereafter

~ 'in Unit I and continued into the summer of 1980.

Unit I Final Walkdown Inspections were performed in non-Pullman compliance to Eh 273 and other procedure requirements.

~

D::ficient Condition' N6tice (DCN's) ~)476-027 (4/1/80), #476-028

~

(4/21/80)'and #476v0'29'{5/1/80) identified that Final QA Walkdo Inspections did nit conform tio"QA instructions #13i and' #148, which Otated that ESD 268 and ESD 273 would be used g dentify and document dificiencies discovered during final twndware walkdown.

The following ESD 273 and ESD 268 procedure requirements were not implemented during the Unit I Final Walkdown inspections.

1. QC Inspectors did not initiate Deficient Condition Notices during the walkdown process but merely noted (ESD 273).

dok dfficienciesonaQC/EngineeringWalkdownSheet,

2. A D.C.N. was not initiated for each dificient condition detected. Deficient conditions were taken from the QC/ Engineering Walkdown sheet and listed on a punch Representative list and then assigned a single DCN number.

Punch li'st DCN#381-215 for construction induced defects, This did had 98 separate deficient conditions listed.

not conform. to ESD 273 procedure requiring a DCN for" dc4 each dificient condition noted.

3. ESD 273 required that " documentation of all deficient conditions noted shall be in accordance with ESD268".

hented (joA The following ESD 268 procedures were not impl during Final Walkdown Inspections of Unit I Rupture Restraints.

A. Field QC Inspectors did not generate DCN's as required I

' by ESD 268. Instead QC Inspectors noted deficiencies on QC/Eng. Walkdown Sheets.

B. ESD 268 required that "each DCN shall be assigned This was a number by the Field QC Inspector concerned."

i not done. Engineering reviewed the QC/Eng. Walkdown sheet l

and then requestg a DCN number from the QC Inspector (lod Supervisor, not pfi Field QC Inspector noting the aeficient condition. The orginator was squeezed out of the picture.'

j I

I