ML20134B226

From kanterella
Jump to navigation Jump to search
QA Case Study Rept
ML20134B226
Person / Time
Site: 05000000, Marble Hill
Issue date: 12/17/1982
From:
NRC
To:
Shared Package
ML20132B505 List:
References
FOIA-84-293 NUDOCS 8508150555
Download: ML20134B226 (41)
v • d • e


Text

. _ _ _ _ _ _ _ _ _ _ _ _ _ _ -

. ss QUALITY ASSURANCE CASE STUDY REPORT To: W. D. Altman - NRC Project Team Leader Date: December 17, 1982 I. GENERAL DATA A. Utility Assessed Public Service of Indiana (PSI), Marble Hill Nuclear Facility, I located near Madison, Indiana. l B. Dates / Places of Fieldwork November 15, 1982 - NRC Region III Office, Chicago (Glen Ellyn), I Illinois.

November 16-19, 1982 - PSI's Marble Hill Nuclear Facility, Madison, Indiana.

C. Assessment Team Members W. D. Altman L. D. Kubicek Project Team Leader EG&G Idaho, Inc.

NRC Headquarters H. Harty R. M. Kleckner Battelle Pacific N. C. Kist and Associates horthwest Laboratory K. C. Carroll M. G. Patrick EG8G Idaho, Inc. Battelle Columbus D. Persons Contacted (1) Interviews or Presentations Made to Entire Team NRC Region III Office:

D. H. Danielson 11. D. Shaffer Chief, Materials and NRC Region III Processes Section Chief, Midland Section R. A. Haase T. M. Tamling NRC, Regicn III NRC, Region III Management Progrant Chief, Program Support Section Section

~

Fot A-h-293 8508150555 850610 GH -293 PDR 1

,,9,.

, s NRC Region III Office (continued):

J. J. Harrison R. F. Warnick NRC, Region III NRC, Region III Senior Resident Director, Office of Inspector Special Cases Marble Hill D. R. Hunter H. M. Wescott NRC, Region III NRC, Project Inspector .

Chief, Zimmer Section Escott Company F. J. Jablonski C. E. Williars NRC . Region III NRC, Region III Inspector, Project

~

Chief, Plant Systems Section 18 Section J. E. Klinklin NRC, Region III Project Section Chief Marble Hill Nuclear Facility:

E. D. Aimone J. J. Harrison Nuclear Division Senior Resident Inspector Personnel thnager USNRC PSI C. Beckham D. Harold .

Quality Engineering Design Document Control Manager PSI PSI J. V. Bott D. V. Menscer Nuclear Regulations President & Chief and Affairs Manager Operating Officer PSI PSI T. Gross M. L. Morris Records Management Division Administration Manager Manager PSI PSI D. Harold B. Morrison PSI Supervisor, Audit Manager Document Control PSI S. 0. Harris W. M. Petro Division Manager of Vice President of Training & Development Nuclear Services PSI PSI 2

L

Marble Hill Nuclear Facility (continued):

L. O. Ramsett D. L. Shuter J. Roberts Quality Assurance Civil Senior Quality Contract Manager -

Office, PSI Engineer, PSI Materials, PSI R. Simmons G. Rosier J. P. Thomas Operations Quality Quality Assurance Assistance Project Assurance Manager, Cherne Director, PSI Supervisor, PSI Contracting Corp.

R. S. Sallee W. A. Wogsland S. W. Shields Materials Quality Executive Director - Senior Vice l Engineering Manager Nuclear Operations President - Nuclear PSI PSI PSI (2) Persons Interviewed Individually By Subteam and Their Respective Companies Note: Abbreviations of companies listed herein.

CLJV = Commonwealth Lords Joint Venture '

Cherne = Cherne Contracting Corporation Westinghouse NSSS = Westinghcuse Electric Corporation S & L = Sargent and Lundy (Architect Engineer) l Newburg = Newburg Construction Company Team 1 Team 2 Team 3 Basic Re-sponsibility QA Project Management Construction & Contractors Membars W. Altman & H. Harty & R. K. Carroll & M.

L. Kubicek Kleckner Patrick Persons C. Beckham W. Petro N. Riechel Inter- Quality Engineering Vice President, PSI Construction view;d Manager, PSI Nuclear Services, PSI Manager D. Shuter J. Thomas M. Linn Sr. Quality Engr. - Assistant Project PSI, Project Civil, PSI Director, PSI Contract Manager L. Ramsett H. Hoge D. Stegemoller QA Officer, PSI Purchasing Manager, Newberg, VP for PSI Power Construction R. Simmons T. Burns L. Elliot Operations QA Project Engineer Newberg, Project Supervisor, PSI Manager, PSI Manager n .-

. s Persons Interviewed Individually By Subteam (continued):

W. Wogsland S. Brewer D. Maxwell Executive Director Nuclear Safety & Newberg, Asst. Project Nuclear Opt..ations, Licensing Manager, Mgr. for Engineering PSI PSI M. Morris C. Togni A. Pepesh Nuclear Division Chief Civil Newberg, Structural Administration Mgr., Engineering, PSI Supt.

PSI B. Morrison L. Nicodemus D. Melin Audit Manager Engineering Administr- Newberg, Asst. Project PSI ation Services Mgr. General Supervisor, PSI Construction S. Shields A. Varney J. Coffman Sr. Vice President - Resident Engineer Newberg, Lead Area Nuclear, PSI (Civil), PSI Engineer, Cont. Bldg.

1 and 2 S. Harris B. Spence R. Cramer Training Director Project Controls PSI, Senior PSI Manager, PSI Constr. Engr.

B. Morrison R. Bailey M. Martin Audit Manager, PSI Mechanical NSSS Newberg, Labor Supervisor, PSI Supt.

S. Sallee R. Hill B. Muensterman Materials Quality Supervising Engineer PSI, Project Contract Engr. Mgr., PSI ASME Piping, PSI Mgr., Fechanical J. Friend C. Markum M. Centennial Audit Supervisor, Westinghouse NSSS PSI, Asst. Project PSI Supervising Engr. Contract Mgr.,

Electrical B. Renwick G. Dobbs S & L Site CLJV, Construction Project Engineer Manager A. Dolgaard Cherne, Sr. Quality Engineer i J. McDaniel i CLJV, Area Mgr.

Auxiliary Building C. Sward S & L Structural Engineer n .

E. Literature Reviewed in Conjunction with Assessment

1. D. M. Hunnicutt, Construction and Engineering Support Branch, NRC Region III, letter to H. D. Thornburg, Office of Inspection and Enforcement, RL/IE/ Utility Meeting - Substantive CA Findings, August 12, 1975.
2. Chapter 17, Quality Assurance, Marble Hill Preliminary Safety Assessment Report, through Amendment 18, May 1977.

l

3. R. S. Boyd, Office of Nuclear Reactor Regulation, letter to Dr.

J. Coughlin, Public Service of Indiana, Inc., Issuance of Construction Permits for Marble Hitt Nuclear Generating Station, April 4,1978.

4. Order Confirming Suspension of Construction, Docket Nos. 50-546, 50-547, August 15, 1979.
5. Management Analysis Company Report, Public Service Company of Indiana, Inc., Marble Hill Pro,iect Diagnostic, October 2, 1979.
6. J. G. Keppler, NRC Region III, letter to S. W. Shields, Public Service of Indiana, transmitting IE Inspection Reports and Notice of Violation, October 4,1979.
7. Construction Probtains at Marble Hitt Nuclear Facility: Nuclear Regulatory Conmission Oversight, proceedings of hearings before a t__ 5

Subcomittee of the Comittee on Government Operations, House of Representatives, November 27-28, 1979.

8. S. W. Shields, Public Service of Indiana, letter to V. Stello, USNRC Office of Inspection and Enforcement, transmitting a description of Licensee Activities Addressing Order Confirming ,

Suspension of Construction, February 28, 1980.

9. Quality Assurance Program Description, Public Service Company of Indiana, Inc., Marble Hill Nuclear Generating Station, Units 1 and 2, April 1980. .
10. L. S. Rubenstein, Acting Chief, Light Water Reactors, 8 ranch No.

4, letter to S. W. Shields, Public Service of Indiana, 3hrbIs Eil!

Quality Assurance Program, April 28, 1980.

11. J. G. Keppler memorandum to N. C. Mosely, SATE Board Nesults for Marbte Hill Nuclear Generating Station Units i and 2, December 1980.
12. M. H. Schwarts, Attorney at Law, Bedford, Kentucky, letter to G.

M. Snyder, Member of Congress, concerning Report on Concrete Assessment, April 21, 1981. '

13. H. J. Wong, Division of Resident and Regional Reactor Inspection, letter to J. H. Snfezek, Direction or RRRI, Summary of January 8, l985, Meeting with Public Service Company of Indiana, Inc.,

February 9, 1981.

6 _ _ _ . _ . _ __ __

14. W. J. Dircks, Executive Director for Operations, letter to the Honorable G. M. Snyder, House of Representatives, concerning Resumption of Concrete Work at Marble Hill, June 25, 1981.
15. J. G. Keppler, NRC Region III, letter to S. W. Shields, Public
Service of Indiana, transmitting report on the Review of the Evaluation of Concrete at Marble Hill Nuclear Genenting Station units I and 2, Report No. IE-124, July 1, 1981.

l

16. SALP-1, Details on "Below Average" Facilities in NUREG 0834, prepared for ED0 and Comission Briefing in Septernber 1981.

, 17. C. E. Norelius, Division of Engineering, RIII, memorandum to J. H. Sniezek Division of Resident and Regional Reactor

, Inspection Program Staff, M:rble Hitt-Bradford/Dircks Memo of October 23, ISSE, November 18, 1981,

18. Special Management Report by Public Service Indiana, Marble Hill Nuclear Project, January 1982.
19. J. J. Harrison, Senior Resident Inspector, Congressional Briefing for Marble Hili Ruclear Genenting Station, February 25, 1982.
20. J. G. Keppler, NRC Region III, letter to S. W. Shields, Public Service of Indiana, Transmittal of Systemtic Assessment of Licensee Performnce (SALP) Report, July 20, 1982.

. - . _7 _- -_ ___ _ _ _ _

21. Transcript of Public Meeting, Briefing on Qualic- gAssurance, SECY-82-352, September 29, 1982.

i

22. Transcript of Remarks by Commissioner J. K. Asselstine, USNRC, to ANS Executive Conference on Government and Self-Regulation of Nuclear Power Plants, Williamsburg, Virginia, October 18, 1982.

F. Description of Plant The Marble Hill Nuclear Reactor is being constructed by Public Service of Indiana (PSI) at a location approximately 10 miles south of Madison, Indiana. PSI's corporate headquarters are located at Plainfield Indiana. However, the utility has located corporate management personnel that are assigned responsibility for construction and operation of the nuclear reactor at the construction site.

This action was taken.to assure that PSI management fulfills its responsibilities to construct a safe and operable nuclear facility in compliance with NRC requirements. The following data is considered significant for inclusion in this report:

1. The project consists of two 1150 megawatt reactors of the Pressurized Water Reactor (PWR) type with attendant auxiliary systems. Construc-tion is approximately 40 percent complete for Unit 1 and 20 percent complete for Unit 2.
2. Major contractors are as follows:

' ~

a. Architect Engineer - Sargent and Lundy.
b. NSSS Supplier - Westinghouse Electric Corporation. i l
c. Construction Contractors - Newberg Construction Company for civil structural; Cherne Contracting Corporation for mechanical work; . Commonwealth Lords Joint Venture for electrical work, i -

a

i

d. The construction permit was issued April 4,1978.
e. Present schedules provide for operation of Unit 2 in 1986 and operation of Unit 1 in 1988.

II.

SUMMARY

OF FINDINGS' During the spring of 1979, recurrent problems with the placement and repair of concrete for safety related structures were being experienced which led to questioning of the effectiveness of the overall quality assurance program at the Marble Hill Nuclear Facility. Subsequent -

investigations by the NRC Region III office confirmed that the proc lems in the quality assurance program, which earlier had been identified as pertaining to concrete construction activities, extended to other areas as well. Pursuant to this finding, Public Service of Indiana (PSI) issued a stop work order on August 7,1979, which was later confinned by the NRC on August 15, 1979.

A. Root Causes of Quality Problems Based upon the results of the assessment process, the primary root cause of the breakdown of quality assurance at Marble Hill Facility is attributed to the utility's inexperience with a nuclear project.

Derivation of this root cause is based upon the utility's failure to recognize the complexity of a nuclear project at the outset, use of techniques and organizational structures which had served well for-construction of fossil type plants but were inappropriate for nuclear work, excessive dependency and reliance on the experience of the prime contractors, failure to recognize symptoms of QA program failures and misunderstanding of the NRC's role in the construction process.

O

I Secondary causes which contributed to the OA program failure can be summarized as management's lack of understanding of the merit of an institutionalized quality program including fear of a QA organization as an empire builder, failure to manage the project from the outset, and reliance on a false sense of security through dependency on contractors, the NRC, and the replication process to avoid trouble.

Problems within the NRC are also seen to be contributidg causes in

. , that the licensing process is too oriented to technical issues. The presence of NRC inspectors is untimely (irregular presence of inspection teams, assignment of resident inspector too late in construction cycle) and-ineffective communications exist within the NRC such that messages from the field do not get through for action in a timely fashion.

B. Remedial Actions Taken to Correct Quality Problems As a result of the shutdown order, PSI underwent extensive rework of their quality assurance program which finally resulted in approval j by the NRC in March 1981 for the resumption of safety related work.

Of cardinal importance to the dramatic turnaround was PSI management's recognition there was a serious problem, and a conscious decision to face it openly and substantively. As a result:

3

1) PSI has gone to great lengths to obtain, retain.and further train highly qualified managerial and technical people. The project staff has been increased to about 750 personnel from a preshutdown level of about 75.
2) Extensive reorganization has taken place including relocation of upper management at the construction site.
3) A complete and very positive attitude change towards quality assurance (both as a technical program management philosophy and as a specific organization) has taken place which is rooted in top management and endorsed throughout all levels of the organization.
4) Emphasis is placed on doing the job right the first time and making the necessary resources available to cause this to happen.

, 5) PSI now.very firmly manages all aspects of the nuclear construction process going on at Marble Hill.

C. Site Findings with Generic Implications

1) Management must have and demonstrate a firm comitment to quality including the assignment of sufficient resources to a_ccomplish the task. The unquestionable reed for totally safe cperation of nuclear plants demands extra controls and documentation to demon-strate compliance with regulatory and design requirements which results in higher construction costs.
2) The pressures of schedule cannot be pennitted to supersede quality assurance program requirements.

I

_l _ _ . . _ _ _

3) Utilities must have or obtain personnel with the education, qualificationsand experience that are required to construct complex nuclear reactor facilities.
4) The Utility should manage the project.
5) A team attitude must exist and the QA/QC organization should be an effective member of that team.
6) Qualified NRC personnel should be available at the construction site from the very beginning of construction work.
7) The application of nuclear codes, standards and the regulatory requiremen1Mi is confusing and inconsistency of application within the various NRC Regions was voiced by contractor personnel inter-viewed who had worked at other reactor projects.
8) The Architect Engineer must prepare specifications that are unambiguous and interpretable with clearly defined tolerances to reduce conflict between QA/QC and construction personnel on how strictly the-specification must be interpreted and enforced.
9) NRC Region III personnel openly admit that in the early going they were inexperienced and did not fully understand what was needed for a successful-QA program. They were minimally qualified at that time and badly understaffed. This problem appears to be rectified as far as Marble Hill is concerned.

_. _ 12 - _ __ . _ , _ _

III. GENERAL TOPICS A. Licensee 0A Program Prior to the stop work order, the QA organization was treated as a suspect organization and a newcomer in the project team that had not earned its place and trust. Early in the project the QA organization only had approximately 19 personnel. These personnel were not, in all casesi, fully qualified. Efforts to obtain additional personnel during the 1977,1978 and 1979 period (until stop work order in August) were mostly unsuccessful. Stop work authority was limited.

Presently, the QA staff of 130 people report to the Corporate QA Officer who is located at the construction site, and reports to the Senior Vice President Nuclear Division. This places quality on equal reporting status with other engineering organizations assigned to the project. A quality engineering and Quality Audit Manager report to the Corporate QA Officer. The quality engineering organization has six sections re-porting to the manager representing all work / engineering disciplines; e.g., mechanicai, electrical, a materials and a records section. These sections provide surveillance over the contractors programs in their assigned disciplines. The audit organization has organizational inde-pendence and audits all PSI and contractor activities including suppliers of equipment / materials.

The following are indicators that the QA program at Marble Hil1 is, effective with full management support:

1) QA has full stop work authority.

13

/

2) The project staff says that the message they receive from management is to build the plant right; management meetings stress doing the job right, and not expendiency.
3) The staff exhibits and expresses esprit de corps with respect to completing a quality project.
4) There was no expressed complaint that any other component of the project team was not doing its job, was interfering, was defensive, was not carrying its load, was disruptive, etc.
5) It was frequently st&ted that "the QA organization is honest and
easy to communicate with"; "QA and engineering now discuss problems, previously $e were antagonists".
6) The authority for the Marble Hill project was vested in a single person whose only responsibilities are for that project.
7) The project management team understands how to achieve quality in construction.
8) There is no expressed uncertainty about job responsibilities, which appear to be clearly defined.
9) Subordinates frequently spoke highly of their management at all levels, but especially the higher levels.

l

.- _4 ___ . . _ .

s.

10) Improved pr) ject management practices have been incorporated into project organization.

Comprehensive procedures at all levels of organization and for all activitics have been place'd in effect and are monitored carefully for compliance.

Team building efforts have resulted in (a) common overall objectives that all agree to, (b) an uncommonly high degree of integration of the QA staff into project activities, (c) an openness in organizational interactions, (d) a cooperative, non-defensive attitude in highlighting and resolving nonconformance reports, and (e) a high degree of communication'.

Contract (construction) managers have been established with raised authorities; contract management teams are comprised of construction, scheduling, engineering, and QA personnel focused in a coordinated fas.iion to achieve common objectives. Previously the approach had been " discipline-oriented with much finger pointing".

  • The licensee has rewritten (or is in the process of rewriting) all construction contracts to define quality requirements (and to change to cost-type contracts to permit the licensee more flexibility in quality-related matters). Fixed fee contracts -

caused schedule to override quality matters and resulted in many additional cost claims when quality requirements were enforc-ed.

15

11) Record keeping, document control and material procurement and storage programs have been computerized and appear highly effective.
12) Staffing levels project an acceptable ratio of QA/QC personnel as indicated by the following data:

PSI expanded their total site staff from a preshutdown level of 75 personnel to about 750 presently. QA increased from about 19 personnel to 130 personnel.

The major contractor on site is Newberg which is responsible for civil / structural work. Their present staff of about 1700 personnel includes 80 QA/QC personnel. There were only 16

~

QA/QC personnel in August of 1979 when the stop work order was issued. Comonwealth-Lords Joint Venture (electrical contractor) estimates staffing to level out as work progresses to 500 craft, 200 support, 60 QC and 30 QA personnel. ,

B. Description of OA Problems at Marble Hill The QA problers at Marble Hill were manifested mostly in concrete placement problems exhibited by honeycombing being evident when forms were removed and noncompliance with ASME Code requirements for materials. PSI was acting as an approved ASME Materials Supplier without authorization. This resulted in inadequate materials documentation and traceability.

16 -

i NRC personnel were aware of concrete problems but were unsuccessful in getting PSI to acknowledge the severity and full implications of the problem. A fomer contractor employee reported the inadequate corcrete placement practices and caused the situation to be fully reviewed by NRC which led to the stop work order. ASME Code inspectors reported the material problems and code violations which were a surprise to the NRC Region III Inspectors. This indicates that NRC was not as forceful or effective as it should have been in the early stages of the project.

C. Problem Analysis and Justification of Findinas The Assessment Team performed an extensive literature search (reference Section I.E. of this report) and arrived at postulcted root causes of QA program failures at the Marble Hill Reactor Facili ty. The root causes of failure are sumarized in Section II of this report. The fieldwork only caused slight modification to the postulated root causes of failures and substantiated those listed in this report as accurate. Substantiation of the root causes was accomplished by the following:

1) Presentaticn of the postulated root causes to NRC Region III personnel responsible for Marble Hill. NRC personnel were most helpful and openly shared their knowledge of past and present conditions at Marble Hill. They confimed the accuracy of the Assessment Teams findings. Mr. Cordell Williams and Jay Harrison were exceptionally knowledgeable and cooperative in assisting the Assessment Team. Mr. Harrison, the NRC Site Inspector for 17 l

+

for Marble Hill, is very competent and respected by all PSI and contractor personnel that were interviewed. He worked closely with the Assessment Team for the four day visit at the site providing valuable insight to the complicated and numerous organizational interfaces involved in this project and also 1

concurred with the team's conclusions.

2) PSI Management was open, candid and forthright on the problems that led to the stop work order. They also concurred with the Assessment Team's conclusion.
3) An extensive interview program was undertaken. Personnel inter-viewed and their positions are listed in Section I.D of this report. Irtterviews with personnel of PSI and all contractors from Senior Management to working level personnel did not reveal any contradictions as to what caused the early QA ' problems or the effectiveness of remedial actions taken. Personnel were highly qualified, motivated and comitted to doing a quality job.

The change over to cost plus incentive fee contracts has aided the contractors in their willingness to meet PSI's quality standards.

Some personnel voiced concern that PSI had overcorrected and was 4

spending too much money, to achieve the high level of quality that is now PSI's objective. Some contractor personnel expressed concern that tolerances are more rigidly enforced than the designer anticipated, reflecting that perhaps tolerances are unrealistic

and tighter than necessary. Concerns were voiced that some code requirements and/or design requirements conflict within them-selves and creates a suspicion that the designer took the easy way out without recognizing the practical realities of construct.on.

The above is listed to assure the reader that many viewpoints were sought and that while some concerns were expressed the overall message was consistent. QA problems were serious prior to the stop work order, schedule was the motivating factor, quality was understaffed and suppressed, inexperience of the utility and contractor personnel in nuclear work was a serious problem and these conditions have been effectively corrected.

4) The audit process was utilized to some degree to substantiate conclusions-and to assess effectiveness of the QA program. .

Documents obtained as objective evidence of compliance with requirements have been retained and incorporated into the Assessment Teams Site Visit Historical File.

D. Remedial Action Supoortive Data The success of remedial actions taken were mainly dependent upon PSI Corporate Management openly addressing the problem of their inexperience and errors, seeking the help of qualified consultants to gain a full understanding of the problens and then committing sufficient resources to obtain qualified personnel and sufficient

~

staff to do the job correctly.

4 k

19 , _ .

l The QA manual that existed before the stop work order had essentially the same requirements as are now contained in the various company manuals. The implication that may be drawr. from this is that manage-ment understanding, attitude, and commitment are keys to a successful QA program.

It should be noted that without vigorous action on NRC's part the l turn around that resulted in effective remedial action would not have occurred. It took the stop work order.by NRC to get the utilities attention. Safety related work was suspended from

August 15, 1979, to March of 1981. This caused a significant 4

delay in schedule. Actual costs are not known; however, the following facts are of interest:

1) PSI increased staff from 75 to about 750 personnel. The QA/QC staff increased from 19 to the present leve.1 of 130. During reinspection efforts, the staff peaked at 150 personnel.
2) Cost of concrete repairs was reported as' about 8 million dollars.
3) One of the NRC stipulations before safety related work could be

~

resumed was a reinspection of all completed work. This reinspection 1

effort resulted in rework of 75% of the items reinspected. Actual cost of this effort is not known.

41 The cost estimate in 1976 was 1.8 billion dollars for completion i of both units. The present cost estimate at completion is 5.1 20

billion dollars. It is not possible to segregate out of this the i

actual cost of quality assurance program efforts.

4

5) The contractors on site presently have a total of 210 QA/QC personnel. Prior to stop work actual numbers are not

?-

known but PSI reports the QA/QC total contractor staff as very minimal at that time.

6) NRC personnel were lacking in experience and there was a definite shortage of qualified personnel when the NRC investigation was started prior to the shutdown order. Inspectors were brought in
from other regions to assist in the investigation that was required f to determine the full scope of the problems including those revealed by.ASME Inspectors. After the full investigation the stop I work order was issued. Actual manpower impact to NRC is not known, but it did impact an already less than satisfactory situation for l
NRC.

e

, E. S_pecific Findings / Observations Resulting From Review of Problems and Remedial Actions I

ll A constant theme echoed by both the NRC and the utility was I

fossil plant " cookbook" type construction, most of which could I

be left to contractors, was inappropriate for nuclear work.

With this approach, the fossil plant construction mind-set mani-

! fested itself and led all concerned into trouble because this 21

approach was simply not up to controlling the complexities and rigorous documentation aspects required in the nuclear industry.

2) QC/QA is too new of a concept for most utility management to readily accept. It appears that management's attitude towards QA is like a self-fulfilling prophecy. If recognition of quality assurance as a management tool and a positive attitude is not there, adequate resources will not be provided and the QA

~

organization itself will be hamstrung with too many controls to be effective.

3) Communications within the NRC, from field offices to headquarters, are not effpirtive. Problems and suggested resolutions are identified and escalated, but it appears as though not much of anything happens as a result.

l

4) Both utility and the NRC field office were in agreement that a competent resident inspector should be assigned at the start of construction. Under current policy, the utility's ways are too i

set at the 15 percent mark and bringing about needed changes can l

prove to be extremely difficult. It was also recognized that i the NRC is probably too pressed for manpower now to be able to instigate this kind of program and have it be effective.

l

! 5) There is not any one indicator that can be looked at to determine l whether a QA program is effective or not. PSI had methods for i

k m

measuring performance prior to the shutdown, but they were crude and a good picture on performance could not be formulated.

Problems were treated-as individual entities into themselves.

It was only after the problems kept repeating that trends were looked at and further investigations conducted that the program was recognized as being ineffective end existing primarily on paper.

6) Prior to the shutdown, there was no one individual who could be pointed to, and the statement made, "He is the one with the ultimate responsibility for qua11ty assurance." Responsibilities were too spread out and no one could really be held accountable. Problem areas were escalated and resulted in a high level stalemate,,such that nothing really effective was done to get at the root problem.

s

7) The existence of a QA program on paper and the utility's verbal sales-manship to the NRC are not sufficient to demonstrate the adequacy l of construction management capabilities. A suggestion by PSI is to have first time applicants present their case to a panel of experienced peers from the industry and get their approval before approaching the NRC. The presentation would have to be by, and to, senior management personnel to ensure the commitment can be properly evaluated.

1 i 8) PSI has gone to great lengths to assemble a first rate organization to handle nuclear work. Extensive recruiting was done to obtain the most knowledgeable and experience people available. Pay scales

were adjusted to attract and hold qualified people and necessary resources in terms of manpower, materials, and management backing have been provided to assure a job can indeed be performed correctly the first time.

9) PSI personnel, prior to the shutdown, really were not aware they were headed for serious trouble. Even when the shutdown order came many were not convinced. This perception prevailed from senior management down to the working levels.
10) All individuals interviewed reflected that PSI is now very assertive in managing and having contractors follow PSI requirements. This is in opposition to the pre-shutdown situation where undue confidence and relianct was palced on the subcontractors to keep the utility out of trouble. To some degree this also applied to the NRC, the utility seemed to believe that the NRC (through its licensing i and inspection processes) would catch them before their program i

went too far astray.

J 111 Progressive attitudes toward quality assurance are very visible i

through the company's Hi-Q program; there is a strong team spirit and a seemingly low level of adversative relationships at the managerial / supervisory positions. PSI also stresses employees are i to be open about problems and that there will be no recrimination.

l 24

12) Replication of design was generally seen to be a good philosophy.

If it is managed appropriately it can lead to significant cost savings; to follow replication blindly, though, is asking for trouble.

13) Subcontractors perceive overkill with respect to procedures and docurentation requirements, but are encouraged that PSI is taking a firm approach to management.
14) There is a perceived difference of how NRC regions interface with the utilities. These variations, real or perceived, could affect how utilities do things.

~

15) A perception offered by senior management likened the current QA program of the NRC to the one that existed at Marble Hill -

prior to the shutdown order. Basically, the program is not seen as very mature or effective for reasons of:

a. The organization is to fractionated;
b. is subject to too many controls;
c. does not report high enough in the management chain;
d. suffers from lack of good inspection personnel and other resources to do an adequate job;
e. inconsistency between inspectors because of no real guidance coming from headquarters.

25

IV. SPECIFIC LESSONS LEARNED A.

Implications of This Case Study For Improvement.of QA Programs For Design

1) Tolerances are more rigidly enforced than the designer antici-pated, reflecting that perhaps tolerances are unrealistic and tighter than necessary.

2)

Concerns were voiced that some code requirements and/or design requirements conflict within themselves and creates a suspicion that the designer took the easy way out without recognizing the practical realities of construction.

3) Replication of designs cannot be utilized without problems but is a good concept.

B.

Imolications of This Case Study For Imorovement of QA Programs For Construction The text of this report fully covers this subject. One concern is re-emphasized due to its importance - Utilities are experienced in constructing fossil plants and they must be made aware that nuclear plants are far more complex and that a fossil plant mentality will not suffice for nt. clear construction. Personnel experienced in nuclear facility construction must be in key positions for both the utility and its contractors. The regulatory requirements for nuclear plants are extensive and result in high construction costs,

~~ -

l I

but they must be met and compliance documented if public confidence in the safety of nuclear power is to be recovered.

C. Implications of This Case Study For Imorovement of QA Programs For Startup, Testing, Maintenance and Operations Early preparation was seen to be necessary for the smooth transition from the construction to the operations phase. PSI has recognized that considerable lead time is needed to adequately prepare personnel, procedures and plant systems for the 1986 startup date and have created an organizational structure which functions to ensure readiness. Training programs have been established, operational a

and maintenance procedures are being developed and personnel are involved with the testing program. These activities have the benefit of gettjng people who will be involved with plant operation first hand knowledge of the plant systems and insures that significant operational type problems are addressed and corrected before startup is initiated.

D. Implications of This Case Study For Improvement of OA Programs For Contractin'g and Procurement

1) Fixed price contracts tie the utility up in settling additional cost claims, create pressure to meet schedules thus compromising compliance with QA program requirements and implies that such contractors should not be allowed to utilize their own QC organi-zation because of potential for intimidation.

v- a,-

2) Cost plus incentive fee contracts add significantly to the cost of the utility, but create an atmosphere where quality objectives can be met. The licensee has rewritten (or is in the process of rewriting)'all construction contracts to define quality requirements (and to change to cost-type contracts to permit the licensee more flexibility in cuality-related matters).
3) The utility should manage the project and take an active roll in establishing and enforcing the QA requirements rather than sluffing this responsibility onto its contractors.
4) Contract (construction) managers have been established with raised authorities; contract management teams are comprised of construction, scheduling, engineering and QA personnel focused in a coordinated fashion to achieve comon objectives. Previously

~ the approach had been " discipline-oriented with much finger pointing."

E. Implications of This Case Study For Improvement of QA Programs With Regard to Manacement The need for management understanding and comitment to quality has been demonstrated through this case study as the most significant issued involved in the Marble Hill QA problems. This report clearly documents the problems caused by la'ck of management experience in nuclear QA and the successes achieved when the remedial action was taken.

gp1

i Implications of This Case Study For Improvement of OA Programs F.

With Regard to Training

1) PSI has developed and implemented a comprehensive and thorough training program with one level of training established for management level personnel and a lower level (less detail on 10CFR50 Appendix B requirements) for craft or non-manigement level personnel. This program is detailed in a " QTS" Manual -

" Quality Training System Modules" and a " Training and Development Guide" manual. The concensus of all personnel interviewed was that the program was effective and very beneficial in helping PSI create the spirit that exists at the site which is, "we are doing a quality job here."

2) The principal contractor for civil / structural work, "Newberg,"

has established an extensive training program for all personnel.

They use video tape cassettes to demonstrate to craftsman how certain work is to be performed and the performance requirements expected of them on work at this utility. A journalist and professional in commercial television advertising was hired )

to put the training video programs together. Some of the ,

l video programs are animated. They report excellent success from this training effort. -

l l

G. Imolications of This Case Study for Improvement of QA Programs With Regard to Organization Structure, Project Management and Flow of Information l 1

1

_ cm _ _ _ _ __ _ __ _ J

1) Organizational changes were rede which resulted in the following:

Contract (construction) managers have been established with raised authorities; contract management teams are comprised of construction, scheduling, engineering, and QA personnel focused in a coordinated fashion to achieve common objectives.

Previously the approach hao been " discipline-oriented with much finger' pointing." A construction manager is as:igned to a portion of the facility; e.g., containment for Unit 1 or Auxiliary Building with all attendant disciplines required reporting to him. This acticn was reported as having improved all aspects of coordinating and prioritizing work in a given facility leading to improved utilization of available craftsman.

It also helped to resolve QA/QC interface and specification interpretation probism between OC and the project prior to starting work which has allowed them to meet schedules while attaining the high quality objectives set by PSI.

2) Relocation of corporate management personnel responsible for nuclear work at the construction site proved very beneficial.
3) One person was made clearly visible as responsible for the entire project and one clearly visible as responsible for QA for the entire project.

M

V. IMPLICATIONS OF THIS CASE STUDY WITH RESPECT TO SUGGESTED APPROACHES FOR IMPROVEMENT OF QA Implications with respect to Ford Amendments:

A. Alternative 1: More prescriptive approach in defining principal architectural and engineering criteri,a.

The following comments were received when this question was presented:

"It's not procedures that count in achieving QA, but individuals qualified in project activities that counts."

"QA is no more difficult than having qualified people in key posi-tions in the organization."

"No problem if criteria focus on the right aspect of the problem."

B. Alternative 2: Issuance of cps contingent upon licensee demonstration of QA/QC competence:

The following statement represents a composite of opinions expressed by personnel interviewed as interpreted by the Assessment Team:

For utilities without nuclear experience and smaller size utilities, NRC should appoint a panel of utility CEOs who have built nuclear plants to review applicant's organization, staffing, and competence for nuclear plant construction, and which would recommend to NRC Commissioners whether CP should be granted.

-o

e C. Alternative 3: Evaluations, inspections and audits by independent association of professionals.

Mixed responses were received to this question as follows:

"If this type of inspection is needed, something is seriously

! wrong. As long as NRC and the ~ licensee work together, there 1 is no need for designated representatives. They would lead to a cops-and-robbers situation and increased emphasis on paperwork."

" Aerospace quality does not arise from QA as much as from long experience."

" Worth the try."

"Would be divisive." -

D. Alternative 4: Improvement in NRC's organization, methods and programs.

The perception that "one can only tie NRC, not win" has many implications for NRC in its decision-making processes and regulatory requirements (opinion expressed by PSI Management).

i The following are direct quotations from personnel interviewed:

"NRC should have taken action at Marble Hill earlier but they lacked confidence" speaks to the need for special qualifications . l for NRC personnel. The NRC staff at Marble' Hill prior to shutdown

~

32

and at other sites familiar to the interviewee were generally not qualified to make judgments on construction matters.

"NRC should look more at what licensee management is doing to manage a nuclear project."

"NRC inspectors should get out more."

"NRC is requiring the licensee to do its work for thea."

"NRC is like a trafff; cop with a quota - sees only the trees, not the forest."

"The NRC mindset is to find something during the audit, or at least be thorough. Thus they focus on the small items."

Corment on preshutdown condition.

"NRC inspectors are concerned with forms and procedures, crossing t's and dotting i's. Most audits relate to procedures, not QA" 4 The following represents a composite of opinions expressed by personnel interviewed as interpreted by the Assessment Team:

NRC should not allow small utilities to proceed with nuclear plants without an experience / qualified staff and indepth know-how. NRC does not have the capability (e.g., personnel) to evaluate a utility's ability to construct / operate nuclear plants.

NRC was not forceful enough in the early stages of Marble Hill, but is very constructive now. NRC inspectors should be at the site when construction begins, not at 15%, because the QA philosphy is set by then. Clearly there is a role for NRC at construction sites. .

Burdensome paperwork required by NRC QA requirements is a deterrent to QA. The QA paperwork is excruciating and nitpicking.

E. Alternative 5: Issuance of CP contingent upon licensee contracting with independent inspectors.

The following statement represents a composite of opinions expressed by personnel interviewed as interpreted by the Assessment Team:

The licensee / contractor inspectors are far better qualified to judge construction quality because they are in the process all the time.

VI. NRC INITIATIVES -

A. QA Holdpoints Ample holdpoints or opportunites for inspectors to come in and witness work functions already exist (composite of opinions expressed by personnel interviewed).

One gets as much information from the present snapshot method m

(inspectors covering the job full time) as you would from formalized QA holdpoints (composite of opinions expressed by personnel interviewed).

" Field inspectors are far more qualified than those brought on the job for a short time."

B. Feedback Into National Standards Practice "NRC's blind acceptance of industry standards for nuclear plant construction is a big problem." This comment applies to NRC use of ASME, ANSI, IEEE, etc., standards which address other technologies besides nuclear, and NRC lacks ability to determine which requirements should apply to a nuclear facility (comment by PSI ManagementT.

C. TMI Action Plan No comments received.

D. NRC Qualification of Licensee / Contractor QA Programs See coments under V.D.

VII. IMPLICAT*0NS OF THIS CASE STUDY WITH RESPECT TO OTHER SECY-82-352 INITIATIVES Which initiatives mighthave made a difference at Marble Hill.

A. NTOL

1) Self Evaluation No

_ - , . _ , . - - . , - - + , _ , ,- - - , , -- - . . .

2) Regional Evaluation No
3) Integ. Design No B. Industry Initiatives
1) INP0 (including managment audits) Yes
2) Utility evaluation (with outside help) Yes

~

C. Construction Inspection Prog.

1) Revise procedures and increase resources Yes
2) Cat (with INPO) Yes
3) Integrated design inspection No
4) Evaluation of reported information Maybe D. Designated Repnesentatives More no than yes E. Management
1) Seminars Yes
2) Qualification / certification of QA personnel No
3) Craftsmanship No F. Management Audit (suggested by PSI) Yes l

m

' ' lor'Revie w ant! Approval s,,,,,,, _ in. . is_ i n.,i.- '

4^3 r

! : ? ?o t No.

SD # 82-234

. originaior MG PATRICK Ctru urrent e Internal Distribution

, . Patrick / File N meused WH MCINTOSH [^#>f7

#/ b /W Patrick /LB SD Files November 30, 1982 Mr. Ken C. Carroll Branch Manager Quality Procurement, Programs and Construction EG&G, Idaho, Inc.

P. O. Box 1625 Idaho Falls, Idaho 83415

Dear Ken,

Attached is a compilation of what I think I gleaned from our interviews at Marble Hill, November 16-19, 1982, including my list of those whom we talked with. Your coments of this will be appreciated, and I will be happy to discuss it with you at your convenience.

Hopefully this will be of some help to you in writing the case study report.

Sincerely, M. G. Patrick Attachments MGP:cjc xc: H. Harty, Battelle PNL o t A -gy.g3 m,,,...,.,,,,...,,,;o,,...,,,.......m.,,,,,,,,,,,.o,,.. Y Ice i , i . . ,,, ,, ,, , . . , , . n m s .. 4 ._ , . . , . . , . ., .. .. ... .,

i

,.. t Marble Hill Results ofS 'ite Visit

! 1. Findings e

1.1 Pre-August 1979 Problems The interviews conducted, in general, confinned the root causes of pro-

. blems at Marble Hill postulated from material reviewed before the site

! visit. At the outset, the approach of the utility management to the

, project reflected a serious lack of understanding of the major differ-i ences between a nuclear plant and a conventional fossil fuel power

plant. This lack of understanding included at least three (3) major

! areas:

, o Nuclear plants require a higher level of quality than is required in fossil plants, o The NRC requires a very high degree of assurance, or proof, i

through documentation that the requisite quality has been j attained.

! o Nuclear plants are vastly more complex than fossil plants.

i i There are indications that the executive levels of management in Public Service of Indiana (PSI) recognized their lack of knowledge about nuclear.

plants. Some of these are:

! o The decision to replicate the design of the Byron Plant.

o PSI engaged an A-E, Sargent and Lundy, with nuclear experi-ence.

l_

! o PSI hired a few staff members with nuclear experience.

i o The construction contractors, in general, had prior nuclear i

experience.

There are other indications, however, that reflect the lack of a full

, understanding of the problem by senior management, and the ramifications I

of this lack. These include:

i 1. A high level of reliance on the construction contractors to coor-I dinate their efforts and control quality.

2. Nonacceptance of the recommendations from the field for additional
, staff of qualified QA/QC people.

! 3. In the case of at least one major contractor, prior nuclear work consisted primarily of simply providing construction craft services, i.e., functioned as a " labor broker". Which does not provide the

, complex work coordination and QC background that was probably l expected. ,

m.,..

!-3 '

a u .

, 2

4. The extensive use of firm, fixed-price construction contracts.

! 5. Minimal attention to records retention as indicated by the des-i truction of about 250,000 documents after they had been micro-j filmed.

4

6. Construction materials on-site, awaiting installation were not adequately protected.

There were also some indications of NRC 's inspection practices, prior to the establishment of a resident inspector, contributing to the confusion.

These included the following:

1. Different inspectors on different occasions generated an impression of inconsistent and less than rigorous inspections.
2. . Producing an impression of adversarial gamesmanship by not in-j forming PSI of problems encountered at other plant sites until j after the same problem was shown to exist at Marble Hill and using i such knowledge to choose inpection points.
1.2 CurrentSituatio[i 4

It was apparent from tiie interviews that PSI has mounted a major effort to overcome the deficiencies which led to the August 1979 stop work i situation. Numerous decisions, actions, and changes have obviously ,

j taken place that reflect a dramatic change in approach and apparent results. Some of the indicators of such changes are:

1. PSI has increased its staff at the site by adding large numbers.

l 2. The people interviewed give strong evidence of being well quali-

fied, dedicated to the project and endorsing a collaborative approach rather than adversarial.
3. A strong team spirit seems to prevail, particularly at the higher 1

levels of management.

4

4. The QA/QC emphasis seems to be firmly endorsed at the higher levels i of management and seems to be at least accepted at the lower levels in the contractor organization.
5. All of'the construction contracts have been converted to some type i

of cost reimbursement, therefore, increasing the utility 's level of

, control.

6. 'There were repeated' statements to the effect that maintaining-j schedule means, "do it right the first time".

l 7. Very good housekeeping was evident throughout the site.

I s

4 e

, i 3

8. Excellent warehousing facilities have been provided for material storage.
9. At least one major contractor has implemented an extensive training program focused on "do it right the first time".
10. Even those interviewed that considered their primary responsi-bilities to be productivity and production were cognizant of meeting quality requirements.
11. Very extensive records management and document control systems and organizations in place.
1.3 Other Observations and/or Concerns During the interviews there were a number of concents and/or concerns that cannot, at this time, be fully evaluated as basis for conclusions.

Some of these are included here for whatever follow-on value they may represent:

1. Marble Hill has over-corrected its problem and has created a bureaucratic, paper work case of overkill.
2. The extensive paper work and quality emphasis is resulting in very high costs.
3. Tolerances are more rigidly enforced than the designer anticipated, reflecting that perhaps tolerances are unrealistic and tighter than necessary.
4. Too much ambiguity in the requirements.
5. In:pections rigidly controlled by check lists may cause the QC inspectors to function as zombies and not think. (This was voiced from the standpoint of causing a focus on the nitpicking kinds of items and overlooking something important.)
6. Concerns were voiced that some code requirements and/or design requirements conflict within themselves and creates a suspicion that the designer took the easy way out without recognizing the practical realities of construction.
7. Ratio of manual to non-manual in total on-site work force is 1.45 (as of October 2, 1982).

e

, e

4' Marble Hill Site Visit November 16-19, 1982 Persons Interviewed Individually by Subteam Norman Riechel PSI, Construction Manager Mick Linn PSI, Project Contract Manager - Civil Don Stegemoller Newberg, V.P. for Power Construction Les Elliott Newberg, Project Manager Dave Maxwell Newberg, Asst. Project Manager for Engineering Al Pepesh Newberg, Structural Superintendent Dick Melin Newberg, Asst. Project Manager, General Construction John Coffman Newberg, Lead Area Engineer, Containment Bldgs 1 and 2 Roger Cramer PSI, Senior Construction Engineer Max Martin Newberg, Labor Superintendent Bill Muensterman PSI, Project Contract Manager, Mechanical Mark Centennial PSI, Asst. Project Contract Manager, Electrical Gary Dobbs CLJV, Construction Manager Al Dolgaard Cherne, Senior Quality Engineer Jim McDaniel CLJV, Area Manager, Auxiliary Building Chris Sward S&L, Structural Engineer Interviews or Presentations to Entire Team Bill Shields PSI, Executive V.P. for Nuclear Darohld Menscer PSI, President and Chief Operating Officer Tom Gross PSI, Supervisor, Records Management Dave Harhold PSI, Supervisor, Document Control i Jeff Roberts PSI, Project Contract Manager, Materials

Jack Bott PSI, Manager, Nuclear Regulations and Affairs Jay Harrison NRC, Resident Inspector e

O d

.. - _ . , . _ - . . _ _ ._ ._,,_

Retrieved from "https://kanterella.com/w/index.php?title=ML20134B226&oldid=2876654"