ML20085K488
| ML20085K488 | |
| Person / Time | |
|---|---|
| Site: | 05000000 |
| Issue date: | 09/19/1983 |
| From: | AFFILIATION NOT ASSIGNED |
| To: | NRC |
| Shared Package | |
| ML20085K448 | List: |
| References | |
| NUDOCS 8310200459 | |
| Download: ML20085K488 (43) | |
Text
,.
. REVISED c'
DRAFTWORKINGPAPER
/
E d
QUALITY ASSURANCE CASE STUDY WORKING PAPER
^
CASE C
=
PREPARED FOR NUCLEAR REGULATORY COMMISSION WASHINGTON, D. C.
20014
/
September 19, 1983
~
83yogggg9Jggo07
. REVISED CORRESPONDENCE PDR DRAFT WORKING PAPER
O DRAFT. WORKING PAPER
^
s
{
TABLE OF CONTENTS I -,
SUMMRY OF FINDINGS A.
Introduction 1
B.
Background
1 C.
Sumary 5
~
II.
ROOT CAUSES OF THE LICENSEE'S PROBLEMS WITH QUALITY IN DESIGN A.
Primary Root Cause 9
13 B.
Secondary Root Cause III.
REMEDIAL ACTIONS TAKEN TO CORRECT QUALITY PROBLEMS 15 IV. GENERIC IMPLICATIONS 17 V.
IMPLICATIONS OF THE CASE STUDY FOR NRC QA INITIATIVES 22 A.
Measures for Near-Term Operating Licenses (NTOL) 23 B.
Industry Initiatives 24 C.
NRC Construction Inspection Program 24 D.
Designated Representatives 25
.E.
Management Initiatives 26 VI.
IMPLICATIONS OF THIS CASE STUDY FOR THE FORD AMENDMENT ALTERNATIVES 26 A.
More Prescriptive Architectural and Engineering Criteria 27 B.
Conditioning the Constructiort Pennit on the Applicant's Demonstration of Its Ability to Manage an Effective Quality Assurance Progran, 27 l
C.
Audits, Inspections, or Evaluations by Associations of Professionals Having Expertise in Appropriate Areas -
Management Audits 28 D.
Improvement of NRC's QA Program 28 E.
Conditioning the Construction Pennit on the Applicant's Comitments to. Submit to Third-Party Audits of Its QA Program 29 Appendix A Appendix U
~
DRAFT WORKING PAPER
3 DRAFT WORKING PAPER
/
r I
QUALITY ASSURANCE CASE STUDY WORKING PAPER
~
CASE C H
I.
SUMMARY
OF FINDINGS A.
Introduction The Nuclear Regulatory Commission (NRC) has undertaken a study of selected nuclear reactor construction projects to determine the important factors or root causes that underlie effective and ineffective assurance-of-quality programs.
Several nuclear projects which have experienced major quality-related problems and several which have not will comprise the study popula-tion.
Data and findings from these site-specific studies will be used by the NRC in the fonnulation of generic policies and programs related to assurance of quality and in responding to the. Congressional Amendment to the FY 1983 Authorization Bill. This working paper summarizes the findings from the third case study.
B.
Background
The Licensee of the Case C Study had established its own in-house engineering and construction management capability in the 1930s.
During the late 1940s and early 1950s, outside architect-engineer (A-E) firms were utilized because of unusually large (post-WWII) system expansion requirements.
In the mid-1950s, the Licensee's earlier practice of doing its own engineering and construction management was resumed.
s i
During the late 1950s and early 1960s, the Licensee planned an ambitious j
program to construct several nuclea' power stations.
Nuclear power was recognized as a'new technology and the Licensee took actions to prepare itself for entry into this field, including having observers at the construction sites of some early nuclear power plants, participating in the design of a test reactor, and studying A-E's designs of proposed nuclear i
plants. The Licensee decided to build its first nuclear plant -- a small
(<100MWe)fpower reactor -- througt! a " turn-key" contract fo'r design ano constructfon.
The plant was completed in the early 1960s, and the Licen'see operated it successfully for about 15 years until it was retired.
The DRAFT WORKING PAPER
L DRAFT WORKING PAPER
~
's 2
Licensee capitalized on the turn-key design and construction activity to familiarize its staff with nuclear activities to enable it to engineer and construct subsequent nuclear plants.
The Licensee had been succes ful in engineering and construction activities on a variety of generating technologies and related electrical transmission systems.
During the early and mid-1960s, t.he Licensee announced plans for several nuclear plants.
Environmental and/or seismic problems, coupled with intense intervention, political factors, load growth changes, and other considerations, resulted in all but the Case C nuclear station being cancelled. :4any of these factors were also present in the Case C project, resulting in significant delays and cost increases.
~
The Case C nuclear station is comprised of two large (>1000MWe) units.
The Licensee announced Uni'ts i and 2 in 1966 and 1968, respectively. "
Construction permits were issued in 1968 and 1970.
Unit 1 of the nuclear station was largely completed by the mid-1970s and fuel was received onsite for both units in 1975 and 1976.
Then occurred a series of required modifications to the nuclear station which delayed its completion.
Included in these were NRC regulations relatedtopipe-break-outside;containmentwhichnecessitated,amongother
' things, relocation of a number of conduits (1973-75); identification and/or g
reconsideration'of a seismic fault which required such modifications as column stiffening, tank bracing, revising piping hangers and equipment supports, diaphragm stiffening, buttressing and foundation chances (1978-79); the Brown's Ferry incident which required modifications related to cable spreading, inerting atmosphere, new decking, and extensive cc,ncrete anchorboltinitallation(1980);theTMIaccidentwhichrequiredinstallation of extensive ' additional wiring, sub-cooled monitors, hydrogen recombiners, and other modifications (1981).
DRAFT WORKING _ PAPER __________
DRAFT WORKING PAPER J'I 3
r
,' l It is important to note that, over the time span of about eight years, at j
least one of the two units had been within a few months of being completed I
on a number of occasions.
Thus far, Unit 1 has undergone three hot functional tests and three containment leak tests.
Unit 2 has undergone one containment leak test.
In September 1981 the Licensee received operating licenses for its two units.
These were suspended two months later following notification by the Licensee to NRC that the diagrams used to locate the vertical seismic floor response spectra in the Unit 1 containment annulus area were in error. Briefly, the error occurred as follows:
the Licensee had transmitted to its seismic consultant a ske,tch of the vertical loadings from which the cons.uTtant was, to determine the seismic response spectra. There was no indication on the sketch which unit the loadings applied to, though the consul' tant understood (correctly) that they were'for Unit 2.
Th'e consultant thought that Un'it 1 was a slidealong unit (instead of a mirror-image unit) and perfonned the analysis on Unit 1 based on that assumption.
The information returned to the Licensee was marked as " Unit 1" (in fact, the analysis applied to Unit 2, not Unit 1). The Licensee accepted the data at face value as being for Unit 1 and, because it knew the plants to be mirror-image plants, flipped the data so as to be applicable to Unit 2 (in fact, the data in the flipped condition were correct for Unit 1, not ynit.2).
The seismic response spectra were now incorrect for both Units. I and 2.
Upon confinnation that wrong diagrams were used in the development of Unit 1 design requirements, the Licensee reanalyzed the design requirements for Unit i using the a,ppropriate containment annulus frame orientation diagrams and determined that, as a result of the error, m'difications were required to be made on $1 Unit 1 pipe supports.
These moaifications involved such actions as ad' ding snubbers, changing the snubber size, adding braces, replacing, structural members, and stiffening base plates.
6 DRAFT WORKING PAPER
DRAFT WORKING PAPER c
4 In an inspection report of seismic-related errors, the Nuclear Regulatory Comission stated that the basic cause of this problem appeared to be the informal manner in which the subject data were developed by the Licensee and transmitted to its seismic consultant, and the lack of independent review of the data within the Licensee's organization prior to submittal to that consultant.
j The Licensee had been the architect-engineer / construction manager for the Case C nuclear power station. One of the major actions that the Licensee took as a result of the aforementioned error was the formation of a Project Completion Team comprised of the Licensee's engineering / construction personnel and personnel from a newly hired architect-engineering finn.
4 AnextensiveIndependentDesignVerificationP[ogram(IDVP)wasinitiated in early 1982 in response 'to the seismic errors discovered in 1981. The Project Completion Team is also conducting a concurrent design verification program.
ip As of January 1983, it was reported that an estimated 90% of the design and 40% of the construction required for modifications as a result of a wide range of reviews spawned by discovery of the seismic diagram error had been completed.
The Licensee has, applied for reinstatement of the operating
. licenses.
At the time of the case study visit, neither the Independent Design Verifica-
~
tion Program nor the Licensee's design verification program had revealed significant further deficiencies in the design or construction of the nuclear station.
The design errors which were identified were not considered to have prevented the affected systems from performing their functions satisfactority.
DRAFT WORKING PAPER
DRAFT WORKING PAPER I(r 5
The objective of this case study was to determine the underlying root causes for the above design problem and to determine the generic implica-tions this experience may have for the nuclear industry. The Case C study r
team was comprised of six personnel; two assigned to concentrate on the project engineering / design aspects, two on construction, and two on quality assurance programs. Unlike other case studies, the case study team was unable to divide into sub-teams to. pursue these separate topics. All interviews of licensee and contractor personnel were conducted by the entire team.
Thus, these working papers do not contain separate subteam inputs.
Prior to, during, and following the site visit, the team reviewed several dozen documents and reports related to this plant and its history, including 3
[
licensing correspondence, inspection investigation reports, and third-party reviews of the Licensee's QA, design, and construction programs." The team spent three and one-half days with the Licensee, including a one-day plant visit. Prior to the Licensee discussions,' two of the team spent one day
}
with the NRC regional staff, and during the Licensee discussions, the entire team spent a day with the regional staff. The site visit culminated (.at a later time) in a briefing for Licensee staff by the team leader in which the findings of the team were reviewed and the Licensee staff had an opportunity 1
i
' to coment on them.
l' C.
Sumary s
Based on review of the background documentation and the interviews described above, the Case'C Study Team identified the following factors which it considered significant in contributing to the quality problem experienced by the Licensee:
1.
The primary root cause of the design-related quality problem was the Licensee's' failure to plan, establish, and effectively implement a managemen't system which provided adequate control and oversight over all aspects of the project. The Licensee failed to fully control the flow -df information across all the interfaces inherent in the engineer-ing/ design process and te provide appropriate reviews of the information transmitted.
m=
DRAFT WORKING PAPER
t._
?
DRAFT WORKING PAPER
~
e 6
There appear to be several factors which contributed to this failure.
Using the experience gained from their earlier turn-key plant and participation of the staff in other nuclear projects, the Licensee after considerable evaluation assumed the role of architect-engineer for this nuclear project. As previously stated, the Licensee had good success with various types of generating projects it had engineered an'd managed over the years.
The nuclear project was fitted into a design, engineering, and management system that may not have been adequately modified to handle all aspects of nuclear work, including the control of quality at design interfaces. As a general rule, it has been more difficult to apply QA to the engineering process than to the construction process, and the Licensee found this to be the case.
Even though QA was apparently rigorously applied to the constr~uction of the project in question (andgrowinginstrengthasNRCrIquirementsandguidance evolved) the Licensee 'did not implement NRC quality requirements for engineering as intensely as they did for construction. Their attitude seemed to be that the engineering organization was comprised of
~
professionals capable of doing what is right without overlaying a stringent formal quality assurance program beyond the normal controls considered part of good engineering practice.
.Anotherfactorintheproblem;ofassuringqualityinengineering relatedtochangesjnNRCrequirementsthatoccurredbetweenthelate l
1960s and l' ate 1970s.
It appears that the Licensee did not completely understand the implications of the changes as they occurred; hence, a QA program for engineering that the AEC might have found acceptable early in the project might not pass NRC scrutiny in the late 1970s.
l 0
9 6
DRAFT WORKING PAPER
DRAFT WORKING PAPER 7
2.
Secondary root causes included the following:
j a.
Failure to understand and appreciate the potential merit of a formal institutionalized QA program. This is suggested by the fact that the project Completion Team adopted the A-E's quality assurance program, even though they were concerned about imposing.
a new system on the project at a late date (the Licensee's engineering procedures were maintained, however).
Examples of program deficiencies (drawn from various reports on the project and discussions with NRC inspectors) which had occurred during the project and the key indications of these deficiencies were as follows:
Design Control The Licensee's engineering staff did not always document important data transmitted to subcontractors Verbal transfer of design information to subcontractors occurred Assigned cognizant engineers were sometimes bypassed in the information or approval processes Adequate internal comunications among the disciplines did not always exist within the Licensee's organization l
Requirer,nents ?or independent reviews were not always
~
l
.followed l
Control of Instructions, procedures, and Drawings / Document Control l
The Licensee's engineering did not develop and/or implement formalized procedures to comply with early QA program
, requirements j
In some cases, outdated drawings were used to establish seismic criteria In some cases, diagrams in lieu of release' drawings were e'
used -- a contributing factor to the seismic problem l
l t
1 DRAFT WORKING PAPER
~~
DRAFT WORKING PAPER
~
8 Control of Service Contracts Proceduralized activities for services contracts were lacking.to control all interfaces with some subcontractors Informal " letter-type" contracts and documents were used Service contracts.were not treated as formally as hardware-contracts Formal quality requirements were not placed on some subcontractors until the late 1970s b.
NRC's failure to Sell QA as a flanagement Tool The NRC requirement for quality assurance seemed to cone across as just anothe,r requirement.
The emphasis from NRC seemed to be on externals; the trappings of a QA program, rather than its substance -- develop a QA manual, set up a QA organization, make the QA manager report fligh in the organization, etc. NRC tended to lose sight of what it was trying to achieve and failed to provide adequate guidance on what a quality assurance program should be. NRC failed to inspect against QA requirements in the engineering area to the extent they inspected against QA requirements for construction.
c.
Long Period of Time Between Inception of the Project and Operation As previously stated,I.the: period of time between the issuance of a construction" pennit and the present has been about 15 years. This long period of time greatly increased the exposure to changes in technology, to changing regulatory requirements, and to changing state of the art in technical matters with the attendant opportunities for quality failure.
O e
h p
- s e
DRAFT WORKING PAPER
DRAFT WORKING PAPER
/
4 9
f e
II.
ROOT CAUSES OF THE LICENSEE"S PROBLEMS WITH QUALITY IN DESIGN
.x Based on the study team's evaluation of NRC's files and other documentation regarding the Case C Project, discussions with cognizant NRC personnel and Licensee and contractor personnel, the following are proposed as the primary and secondary root causes of the Licensee's quality problems in the design and construction of its nuclear station:
A.
Primary Root Cause The primary root cause of the design-related quality problem was the Licensee's failure to plan, establish, and effectively implement a management system which provided adequate control and oversight over all aspects of the project. The primary root cause emerged out of several factors which, taken together, increased the likelihood of a. design error.
These factors include the pressure (whether real or felt) to complete.the nuclear station, informal communications across important interfaces, an inadequate application of quality assurance / quality control to all aspects of the design process, and the resistance by engineering of the application u
of fonnal quality assurance procedures.
Reviews and audits of the poject indicate that the station, as it existed in 1975, had been properly and correctly designed and constructed. The seismic and other analyses (at the then state-of-the-art) had been performed j
to the correct configurations and bases.
It was in the reanalysis after the essential completion of Unit 1 in 1975, prompted by new seismic assumptions, that the design error previously described occurred.
From the issuance of construction permits for the Licensee's nuclear station to the present.< time was approximately 13-15 years, making this station one of the longes.t -- if not the longest -- in the construction process.
The large amount of rework resulting from changing regulatory requirements, coupled 4th turnover in personnel and increased facility costs lengthened the construc[ ion period and increased the real (or felt) pressure to complete the facility. As a facility nears completion or is in a prestartup condition (as the Licensee's station was,in the mid-1970s) and new or changed requirements
~
~
~
arise, there is a tendency to accomplish the activity and to formalize action later. Such conditions, coupled with infonnal interface procedures, increase DRAFT WORKING PAPER
n 1
DRAFT WORKING PAPER
-y 10 I
~
possibility of error.
i Another factor which contributed to the problem was the need for additional exp'ertise, especially in the seismic area, resulting in greater use of consultants and engineering service contractors than had been customary on the Licensee's other generating projects. This increased the possibility'of
~
interface problems and required changes from the customary operation of the Licensee's engineering staff. There seems to have been a tendency to extend the informality common in close-knit engineering organizations to some of these outside groups.
Customary controls and review processes for dealing with them were not always effectively applied. These interfacing problems were increased by the proximity of the consultants and engineering service contractors to the Licensee (a greater distance-might have required more formalization of communications).
A well-developed engineering team which relies heavily on informal comunication among its members has both advantages and disadvantages from a quality assurance standpoint. Such close contact generally contributes greatly to the quality of the engineering work. At the same time, it can create practices which are not appropriate in dealing outside the organization. Geographical separation generally requires a higher degree of formalization in corrrnunication. Geographical proximity (working
' n the same office or bSilding) can result in items being discussed i
sufficiently that a comon understanding is reached between the parties involved.
In the Case C project, the practice of utilizing informal comunications with key consultants located in the imediate area (city and suburbs) developed; however, in some cases, the distance was probably great enough that the' level of comunication required to reach full understanding of key points.was probably not achieved.
E 4
DRAFT WORKING PAPER
f DRAFT WORKING PAPER 11 1
One of the comments made by the Licensee's staff was " engineering viewed j
their consultants as an extension of themselves."
(It should be noted that the Licensee's staff assigned to the project Completion Team has been physically relocated to the A-E's facility). While the problem of inter-face control cannot be considered the primary cause of the diagram error that occurred, it was a contributing factor.
The error is indicative of less then adequate procedures for design reviews and comunications.
During much of the project, the application of quality assurance / quality control (QA/QC) to the design process was not well understood by the Licensee or emphasized by NRC as much as construction QC.
Changing require-ments resulting in redesign, coupled with turnover in design per,sonnel not, familiar with all the ramifications of the original design, make the application of QA/QC to the design process increasingly impo'rtant.
The matter was further complic'ated by the developing nature of the 10CFR50' Appendix B criteria and their implementation.
During the meetings with the Licensee, its A-E, and the regional NRC staff, there were repeated comments with respect to ongoing problems in interpreting 10CFR50 Appendix B criteria and their application to the engineering process, as well as concerns about infringing on " professionalism" and " creativity." These considerations, plus the fact that the Licensee's engineering organization was very j
independent, contributed to their, resisting application and/or understanding
'of formal quality assurance procedures.
There also seems to have been a tendency to require, or at least receive, more stringent quality assurance from contractors than was applied to in-house efforts.
NRC investigations in late 1981 and early 1982 found that design and engineering QA practices in consulting contractors' organizations were better than those in.the Licensee's engineering organization.
It is not entirely clear whether this reflected a difference in the Licensee's requirements or a difference in practices.
NRC inspectors made the observation that the Licensee is
" tougher o,n its contractors than on itself." This attitude may have contributed to the apparent success in assuring quality in the construction efforts, since essentially all of that work was done by contractors.
e=
DRAFT WORKING PAPER
L DRAFT WORKING PAPER 12 The significance attached to this finding is the possible reflection of i
an attitude in engineering -- an attitude of reflecting some degree of professional arrogance that "we do no wrong, but we sure have to watch out f' r those other guys." The major quality problems identified to date have o
been within that organization.
It appears that the application of rudimentary quality assurance practices for design document control should have preven'ted the error that occurred.
l The engineering deficiencies discovered in September 1981 occurred during a time, ironically, when quali+y assurance appeared to be undergoing significant strengthening within the Licensee's organization.
In September 1976, the Licensee hired a new corporate director of QA who was qualified,,
knowledgeable, and aggressive.
During late 1976 and 1977, the QA program was restructured and a new QA program was established in 1978.
The case
~^
study team was unable to e'stablish the att'itudes and relationships between engineering and the new quality assurance director during those years.
i To summarize, the primary root cause was the failure to manage completely a project that is large and complex, and the failure to plan and effectively implement a management system embodying all of the controls necessary to ensure correct completion of such a project.
There were several factors that contributed to this primary root cause.
The Licensee had a high degree l
of confidence with respect to its engineering capability. The Licensee had been successful'with various types of generating projects. What had worked for those projects was assumed to work for its first (in-house) nuclear project and, thus,'the project was fitted into an existing structure which probably carried with it practices not appropriate to nuclear work.
e
~
/
c' DRAFT WORKING PAPER
DRAFT WORKING PAPER
~
13 The engineering function in the Licensee's organization was very strong; strong enough politically to resist the imposition of management controls that were required elsewhere in the company or for contractors. Many management personnel had come from the engineering function; the appreciated its capability, had been part of its good performance, and had not seen a need to enforce additional, more stringent quality controls over it.
Concurrently, the atmosphere and regulations for construction of a nuclear power plant were changing significantly. The Licensee may not have completely understood the implications of the changes as they were occurring.
B.
Secondary Root Causes P.
Based on a review of referenced materials, discussions and inteFviews with the Licensee, the Regional NRC office, and analysis, the study team has identified three secondary root causes of.the design problems experienced at the Licensee's plant.
They are:
- 1) failure to understand and appreciate the potential merit of a formal institutionalized QA program, 2) NRC's failure to sell quality assurance as a management tool, and 3) the long period of time between inception of the project and completion.
Each is discussed in more detail:
1.
Failure to understand and appreciate the potential merit of a formal QA program. The Licensee had a highly capable organization and had successfull.y comple'ted many projects.
It had started to crganize a quality assurance program before the requirements of 10CFR50 Appendix B became mandatory.
It is believed that the Licensee's perception was that good quality was achieved in their projects and that, while the new requirements might change some things, it would not affect the under-lying bases for their good quality performance.
Consequently, the early program could be characterized as a documented or proceduralized process of meeting the requirements.
It did not significantly affect the way that the L4censee had been doing its engineering / design woric. When seismic (or other) problems arose, the Licensee reacted as any concerned or conscientious organization would.
If a mistake had been made, it was totally willing to mak.e it right.
DRAFT WORKING PAPER
m DRAFT WORKING PAPER 14 A member of the Project Completion Team (an A-E employee) said that he had reviewed the Licensee's quality assurance program in great depth prior to fonnation of' the project Completion Team. He noted that the Licensee's early program had weaknesses, but had improved greatly during the project.
He was concerned about imposing a new quality assurance system (the A-E's),on the project at such a late date.
In the end, however, the A-E's quality assurance program was adopted, even though the Licensee's engineering procedures were maintained --
perhaps implying that the Licensee's QA program did not fully satisfy what the A-E considered riecessary for a nuclear plant.
2.
NRC's failure to sell QA as a management tool. As far as the Licensee, was concerned, the requirement for quality ~ assurance came across as just another NRC requirement. The emphasis from NRC seemed to be on tne trappings of a QA program -- develop a QA manual, set up a QA
organization -- rather than its substance.
NRC appeared to lose sight of what it was trying to achieve.
NRC inspection emphasis seemed to focus first on operations, then construction.
The message conveyed
~~~
was that the most important area was not design and engineering, because NRC did not effectively provide guidance for regulations or inspect extensively in the design and engineering area.
NRC failed to provide
. guidance on what constituted a design quality assurance program.
It did not have sufficjent technical strength to provide effective inspection oversight of design and engineering QA programs.
3.
Length of time ~ between inception of the project and operation. The period of time.between the issuance of a construction permit and the case study totals about 15 years for Unit 1.
This long period of time greatly intreased the project's exposure to the normal occurrence of events,,to changing regulatory requirements, to changing state-of-the-art in technical matters, and to changing political climates and public perception.
Some of these factors required portions' of the facility to be redesigned.
Redesigns may not be subject to as thorough analysis as original design efforts because the persongel have changed (employees retire or are_rnoved to new jobs) and the scope of review may be less.
_ DRAFT WORKING PAPER
~
DRAFT WORKING PAPER 15
.s One of the major causes of redesign on this project was the changing j
seismic picture.
Initially, experts with impressive geological and seismological experience postulated the kinds of earthquakes that might occur. That information was given to the Licensee's consultants to describe the loads and seismic response criteria.
The seismic field was developing very rapidly during this period and new data were developed on faults in the plant area.
Data from a seismic event in the region resulted in the Licensee's plant being designed to two different types of earthquakes; a design earthquake and a double-design earthquake. Other major causes of redesign were the Brown's Ferry fire, ti.e TMI accident, and other changes in NRC requirements. This frequent retrofitting,affec,ted morale and contributed to a climate ec,nducive to, errors and quality failures.
~
III.
REMEDIALACTIONSTAXENTOCORRECT(TURNAROUND)QUALITYPROBLEMS The major remedial action taken by the Licensee was the formation of the Project Completion Team. Approximately 275 of the Licensee's staff was merged with about 600 of the A-E's staff to form a new project engineering organization.
The design and licensing functions were merged into the Project Completion Team.
The project engineer for Unit 1 is a licensee staff member; the project engineer for Unit 2 is an A-E staff member. The Licensee's chief engineer's stamp still j
appe,ars "un drawings and its discikline engineers can ask for documents to approve, though the discipline engi'nders appear to be involved in an overview function.
It was understood th3t as the plants become operational, the A-E's staff will
_ phase out and the Licensee's staff will again resume responsibility for engineering.
l 1
o
-O DRAFT WORKING PAPER
h
.. ~ -
t DRAFT WORKING PAPER 16 The Licensee's personnel comented that the transition to the Project Completion Team was difficult but that it was a good learning experience for his staff which had "become kind of hide bound."1 Now that they are working with the A-E's staff, they see a much more alive and responsive organization, one in which decisions are made at lower levels, reviewed by appropriate management, and work is pushed forward with considerable aggressivness.
He said these thi'ngs were good for his staff to experience and it will be better for it when the project is completed.
The Licensee has also learned that it is important to review consultants' work.
The Licensee had retained a large number of consultants -- perhaps 100 or more --
and the Licensee did not have adequate manpower to totally review all of the 4
work done by consultants. The need for the License'e (or Project Completion Team) to carefully scrutinize consultants' work has now been clearly established.
The Licensee now realizes that quality assurance is a total envelope of management-controlled procedures.
If they were to start a new nuclear plant, they would
~~~
ensure that the entire quality system was in place before starting. All organizations involved in the project would have similar quality systems.
1 Quotations are not vefbatim, but they are believed to convey the meaning intended.
e B
/
e DRAFT WORKING PAPER
DRAFT WORKING PAPER 17 IV. GENERIC IMPLICATIONS Based on the information reviewed and analyzed by.the Case C Study Team, several possible generic implications, or lessons, emerge. These are high-lighted for each case study to provide input and to help form overall con-clusions concerning factors which constitute important elements in nuclear plant construction quality. The first four address licensee implications; the last three NRC implications:
A.
Nuclear power plants are complex facilities and licensee management must appreciate that fact.
Design and construction practices normally applied to fossil fueled plants are not adequate to assure quality in nuclear plants.
Licensees which have designed and constructed fossil fueled power.
plants only should not expect experience and technology alone to be adequate for undertaking nuclear plant construction under the present regulatory climate. One difference is that the licensee's management must be knowledg-able about how to achieve quality in nuclear plant design and construction.
In this Case, the Licensee seemed to lack a full understanding of how to institute a quality assurance program to adequately control the design process, even though (or perhaps because) much of the management came out of the engineering organization. There is no question that the Licensee's management wanted a quality facility. At the time of the Case C visit, all
. indications were that it wasichieved as far as construction was concerned and, apart from the seismic design error, it appears to have been substantially achieved in the design.1 All this seems to have been achieved more by previously learned good practices than by the application of a formalized
. lit was reported by the. Project Completion Team staff that the Independent Design
~
Verification Program (IDVP) which has involved about 50 personnel, had examined the containment and other systems in considerable detail.
Perhaps 40,000-50,000 l
different items had been looked at.
Only 63 needed a more detailed analysis and, I
of that number, only eight to ten were classified as legitimate design errors. Of l
the legitimate design errors, none were considered to have prevented the affected
-systems from perforTning their functions satisfactorily.
am e o DRAFT WORKING PAPER
DRAFT WORKING PAPER 18 approach to quali,ty.
The NRC Regional Office also indicated that in the early 1970s there had beena problem in interpreting 10CFR50 Appendix B in its application to the design process.
Licensee personnel noted that it was not until the 1973-74 period that quality assurance was actively considered for application to the design process and, by that time, much of the design was completed.
Since quality was already thought to be part of the design (and apparently it was) it was considered unnecessary to put in a more substantive quality assurance program for the remaining design 'ork (which proved to be far more extensive than thought w
at the time). The fact that the Project Completion Team adopted the A-E's quality assurance program may be indicative of the judgment that the Licensee's methods of applying QA to the desigri process for nuclear plants needed improvement.
B.
A licensee needs to understand its own corporate limitations as it under-takes a nuclear power oroject, and set up a project management structure in which its role is consistent with its capabilities and complements the roles of its contractors. The capabilities of its contractors must augment the licensee's lack of experience er expertise in engineering, procurement, construction, and management.
The Licensee recognized its limitations in
,certain areas and made use of;a large number of consultants (Section IIA).
What apparently went unrecognized were the evolving requirements for engineer-ing support ove'r the life of the project, necessitated in part by changing regulatory requirements. A-E personnel stated that in earlier nuclear projects, engineering staffs (A-E or licensee) did not document the design process in the manner done today, and that the Licensee's practice was typical of the earlier practices.
Further, the types of problems experienced in the design $f the Licensee's nuclear station have also occurred to some extent in sorne other plants in which the A-E has been involved. A licensee's engineering staff involved in a single plant may have difficulty staying current with the state of the art'in nuclear technology and regulation, and is not as likely to assimilate advanced procedures that the industry as a whole has developed.
DRAFT WORKING PAPER
DRAFT WORKING PAPER 19 Whereas the A-E's staff was able to gain appropriate experience because of involvement in a variety of plants, the Licensee's engineering staff was not.
Those organizations which were involved in several plants were able to staff appropriately.
Those with single plants had greater difficulty in doing so and had to rely on consultants or other contractors.
Another facet of the same problem is the evolution of the understanding necessary to incorporate new criteria.
As an example, it was pointed out that when introduced,10CFR50 Appendix B was a new language to many engineers.
Later projects were able to apply Appendix B requirements to the design process more readily because personnel had a better understanding of the requirements and process.
~
A-E personnel also stated that the length of the project can have detrimental
^
effects due to the turnove'r of personnel over a long period of time. A-E personnel said that the length of a project is a common thread in all projects that have gotten into trouble, as far as quality assurance is concerned.
Long exposure opens the project to many potential changes and delays.
Successful design activities require experienced personnel, but doe to. promotions, retirements, etc., over the course of the project, the possibility for error is introduced, because newer employees are often unaware of all of the earlierconsiderationsthathkdgoneintoadecision.
C.
A licensee needs to manage the nuclear project and ensure that interfaces between the project participants ( A-E, construction contractors, etc.) are
' properly maintained and monitoried. A total project system that imposes effective controls'and checks over all key aspects of the project is recuired, including records management and document control, as well as design, construc-tion, procurement, cost, schedule, etc.
The system must also be able to accomodate change; for example, the changing regulatory environment has presented,the Licensee's engineering staff with moving targets that required'dhange, but which may not always have been fully recognized or-quickly accomodated by them. The large number of consultants used for this project was different from previous projects; however, the consulting DRAFT WORKING PAPER
DRAFT WORKING PAPER roles were of limited scope and, in some cases, a collegial relationship developed. There was a lack of formality in the processes for passing information across some interfaces. This occurred in part because some of the consultants also consulted for the Atomic Energy Comission and it was assumed that they knew what the requirements were.
a The geographical proximity of a number of the consultants also helped erode a formal interface control system.
It was noted that procedural matters would not have been handled with the same informality if the subcontractors had been 50 miles away, rather than across town.
For example, it was stated that there was much more formality in procedures with a seismic consultant located about 40 miles from the Licensee's offices than.
with one in the same city. When the project started, there were no require-ments or regulations for c,ontrol of contractors which would provide fqr an auditable trail. As the project developed, regulations became more detailed and complex, but in-house relationships and procedures did not evolve rapdily enough to fully accomodate all changes. The Licensee did not have adequate manpower or expertise to fully review all of the work done by its consultants / contractors.
The interfaces between engineering functions or operations must be minimized and~ carefully monitored.
That the Licensee recognized this problem was apparent from a,senioi \\taff's comment that moving the Project Completion Team together on three floors in the A-E's building was imensely helpful in the comunication process. He also stated that there was no substitute for good procedures to monitor interfaces.
I D.
The licensee must be committed to cuality from too management down, and it must be effectively comunicated by top management and manifested in procedures and controls.
It is helpful when the licensee recognizes that an assurance of quality program properly conceptualized, structured, and implementId can be an effective management tool that can be cost effective.
If management attempts to implement a " canned QA" program rather than an assurance-of-quality program, it can be seen as threatening to some, and as an artificial l'aying on of another system by others -- a system with which one must contend, but one which has no useful purpose or function.
L DRAFT WORKING PAPER
DRAFT WORKING PAPER
~
/
21 E.
NRC needs to treat QA as a management tool, not as just another j
requirement. As another requirement, the concept of quality assurance is treated as just another system laid on the licensee. As a management tool, the concept of QA assumes a much more important and useful role in the eyes of management.
It tells them something about the amount of rework and project cost, about the projected reliability and safety of the operating '
plant. NRC needs to understand and stress this aspect to gain better acceptante of its QA programs.
F.
NRC needs to pay more attention to ensuring quality in the design process.
During the Case C project, there was no effective in-depth evaluation by NRC of the Licensee's design process. The Licensee had nearly completed
~
the engineering work in the early 1970s when the quality assurance require-ments of 10CFR50 Appendix B were brought into the picture. Part of the problem was the imprecise nature of 10CFR50 Appendix B, and this factor did not encourage the Licensee to install a QA program to handle the remaining design work to be done. The NRC did not insist on it, either,
~
perhaps because Unit 1 was already in process and 10CFR50 Appendix B requirements were to be applied "as practicable " as far as Unit 1 was concerned.
It can be expected that engineering organizations in general will resist l
the introduction of qual,ity assurance into the design process.
There is little acknowle'gment from the Licensee's engineering that, had better QA l
d procedures been adopted, it would have avoided the design diagram error (this attitude does not apply to the Project Completion Team).
J 0
(
l I
e l
DRAFT WORKING PAPER
DRAFT WORKING PAPER 22 G.
NRC needs to focus more on the effectiveness of implementation of the quality assurance program and less on the trappings of licensee programs; e_.g., less on the QA manual, organization charts, where the QA manager reports, and paperwork per se. There appears to be a lack of understanding of how to effectively apply quality assurance to the day-to-day design process, and additional guidelines are needed for application of QA to design. The design process may inherently contain a high degree of assurance of quality.
Perhaps for this reason, it has been difficult to fonnalize an acceptable QA program for design. Licensee personnel stated that the early implementation of 10CFR50 Appendix B was manufacturing oriented.
That orientation, together with the perception that QA can't be applied to the engineering process, are barriers to its adoption.
NRC+ ~
needs to address the issue of assurance of qua.lity in the design / engineering process.
s V.
IMPLICATIONS OF THE CASE STUDY FOR NRC QA INITIA.TIVES NRC has underway or under study a number of initiatives which are designed to establish additional confidence in the quality of design and construction activities, to improve the management control of quality, and/or to improve the NRC capability to evaluate the implementation of licensee programs. The initiatives are described in the NRC Staff Paper SECY 82-352, " Assurance of Quality," and subsequent corresponden'ce between the Commission and the NRC 1
staff. One of the purposes ~of this Case Study is to provide feedback regarding the relevance of the various initiatives to the Case C Licensee's nuclear construction project.. Subsequent paragraphs take each initiative in turn and discuss whether the initiative, had it been an ongoing activity at the time of the Licensee's design' error, would have made a difference; i.e., would the initiative have prevented or at least mitigated the design error that has been discussed earlier. A more complete discussion of the scope and details of the various NRC QA initiatives may be found in SECY 82-352 and SECY 83-32, "First-Quarterly Rept)rt on Implementati'on of the Quality Assurance Initiative." Most of these initiatives were discussed with the senior management of the Licensee.
- - - - - - - - - - - - - - - - - -DRAFT WORKING PAPER
DRAFT W0RKING PAPER
/
t s
23 E
~
A.
Measures for Near-Term Operating Licenses (NTOL) l.
Licensee self evaluation - maybe s
s This initiative applies to action that would take place when the licensee is in the process of receiving its operating license.
It requires that the licensee examine selected portion's of the engineering design or construction. A licensee self evaluation pennits an!
evaluat7on cf the project from beginning to end and wodid permit the
Chief Executive Officer to state that the station had'been built according to its comitments.
Had this been a requi:ement, it is quite possible that one of the design areas audited would have ra6 ted to seismic considerations, since that has b'een such'a. major ccnsidera-tion in the d'esign and construction of this particular station.
s Although it is.unlikely that the review would have identified the error that actually, occurred, it should have identified the problem of design document control.
2.
Regional evaluation - no The licensee regional evaluation is an action $ hat would take place when the licensee is in the process of receivisguits operating license.
The effect of the regional evaluation could be similar to that described
.in (1) above. The scope of the regional evaluation would have to be expandedtoinclude{detaiiedbesignreviewforittobeapplicable to the problea in, Case C.
3.
Independent Design Verification Program (IDVP) - yes
,,The licensee IDVP is an action that takes place when t1e licensee is
' n the process of receiving its operating license.
Tne IDVP would i
' h've appited.in the case of this Licensee's plant in which the design a
and construc(ion are essentially completed.
Design verifications can be perfonned at any stage in the design, but the most productive period
" is whe'n the design is essentially completed.
k w
DRAFT WORKING PAPER l
~
DRAFT WORKING PAPER 24 It is likely that an IDVP would address one or more-of the sensitive issues relating to the plant under review. This would have included the seismic problem as stated under (1) above and, since an IDVP should be more thorough in the design area than either of the evalua-tions in (1) or (2) above, there is an increased probability that the diagram error and design document control deficiencies would have been
.found.
B.
Industry Initiatives 1.
INPG "Constructian" audits - maybe This initiative is applicable because phase 2 part of INP0 " construction" audits now consid'rs design.
It is possible that an INP0 " construction" e
audit would have detected the problem that' occurred in this licensee's plant. However great-the scope of these audits, it is not likely -that the specific error would have been detected; it would have probably identified the design document control deficiency.
2.
Utility Evaluation Using INP0 Method - not applicable; not being done now.
C.
NRC Construction Inspection Program
~
1.
Revised procedures and incre sed resources - yes (if included design)
This particular initiative applies to the construction program. The deficiency found in the Licensee's plant related to design and not to construction.
If this initiative were expanded to include design, then it would probably have detected the design document control deficiency.
2.
Construction Appraisal Team (CAT) Inspection - no This initiative applies to the construction phase; the Licensee's qualtjyproblemsoccurre'dinthedesig phase.
DRAFT WORKING PAPER
f DRAFT WORKING PAPER j
25
- s 3.
Integrated Design Inspection - maybe The integrated design inspection is an action that would take place when the licensee is in the process of receiving -its operating license, though it could be done before.
For the same reasons given for the effectiveness of measures for Near-Tenn Operating Licenses, the integrated design inspection would likely have uncovered the design document deficiency.
It is possible, but unlikely, that it would have detected the error.
4.
Evaluation of Reported Infonnation - not likely This initiative would computerize 10CFR50.55e and Part 21 reports, facilitating' trend and other analyses of these event reports. This analysis provides an additional cross-check on the quality and operations at a licensee's site. ' The type. of aua'lity failure that occurred a1 the Licensee's site is not unlike other errors that result from lack of interface control.
Possibly, the reporting of similar problems in other plants would-have been useful to either the NRC Inspection and Enforcement staff or to the Licensee's engineering staff in looking for errors of this nature.
D.
Designated Representatives maybe
'At the time of this case study, it was unclear how the designated repre-sentative syste' might be implemented by the NRC.
Generally, it has been m
considered to apply to the construction process, and not to the design process. However, the FAA uses designated engineering representatives (DER) who are employees of manufacturers, but are deputized by the.FAA to review and veri,fy certain elements of design.
(There are also designated manufacturing iepresentatives (DME) who verify that the assembly or fabrica-tion process'is acceptable).
The DER could be used to spot check the design or design, process. However,. this initiative, had it been in effect, may have uncovered the design document control deficiency that the Licensee -
experienced, and possibly the error itself, had seismic analyses been subject to DER r9 view.
- a e
DRAFT WORKING-PAPER
DRAFT WORKING PAPER 26 E.
Management Initiatives 1.
Seminars - yes
- Seminars similar to' those that the NRC Commissioners conducted in years past, as well as seminars by utility executives who had design-related problems would probably have been helpful in bringing the Licensee's management to an-increased awareness of the importance of quality control measures in the design process.
2.
Qualifications / Certifications of Quality Assurance / Quality Control Personnel - no The problem that the Licensee experienced did not relate to the qualif,ica-tions of the quality assurance / quality control personnel.
It related totheinstitutionofadequatequalityass$rancecontrolproceduresin the design process.
3.
Craftsmanship - no The quality problem experienced by the Licensee had nothing to do with the training or skill level of craftsmen.
VI.
IMPLICATIONS OF THIS CASE STUDY F0R THE CONGRESSIONAL AMENDMENT ALTERNATIVES Sect-ion 13 to NRC's FY 1983 Authoriza' tion bill requires NRC to conduct a 1
study of existing and alternative programs for improving quality assurance and quality control at nuclear power plants under construction.
This Section, called the Congressional Amendment, requires NRC to look in particular at the feasibility and efficacy of five specific alternative program concepts. As a part of this analysis', each alternative concept was evaluated with respect to whether it would have made a difference in the Licensee's construction program, had it been in place at the time of the Licensee's constructior. k rmit.
Each of the alternatives was discussed with senior utility personnel.
DRAFT WORKING PAPER
DRAFT WORKING PAPER x
27
~
_J A.
More Prescriptive Architectural and Engineering Criteria - maybe The Authorization Act requires NRC to evaluate the following alternative:
13(b)1 - adopting a more prescriptive approach to defining principal architectural and engineering criteria for the construction of commercial nuclear power plants that would serve as a basis for quality assurance and, quality control inspection and enforcement actions.
In the case of the Licensee's design error, more prescriptive architec-tural and engineering criteria by itself would not have affected the Licensee's problem.
No one could have foreseen tne seismic complications.
However, if more stringent criteria were expanded to cover the design process itself, then the design document control deficiency might not have' occurred.
B.
Conditioning the Construction Pennit on the Acolicant's Demonstration of Its Ability to Manage an Effective Quality Assurance Program - maybe The Authorization Act requires NRC to evaluate the following alternative:
13(b)2 - requiring as a condition of the issuance of construction permits for commercial nuclear plants that the Licensee demonstrate the capability of independently managing the effective performance of all quality assurance and quality control. responsibilities for the plant.
It should be noted that, at tne time the Licensee received its construction permit, it was among the better qualified utilities for undertaking nuclear plant construction. At the time the construction permit was issued,10CFR50 l
Appendix B was not.a requirement.
At that time, the Licensee could have pointed to its excellent record in the construction of other types of power plants.
It cod 1d have also pointed to its performance in the operation of a small nuclegr plant.
It is unlikely that the Licensee would have changed its procedures sufficiently to prevent, almost a decade later, the type of design erfor from occurring'that resulted in the withdrawal of its operating license; however, the requirement for a demonstration of ability, if done today, would presumably evaluate the procedures for the transfer of informa-tion acr'oss interfaces between an applicant and its contractors.
DRAFT WORKING PAPER
f DRAFT WORKING PAPER l-28 C.
Audits, Inspections, or Evaluations by Associations of Professionals Having Expertise in Appropriate Areas - Management Audits - yes The Authorization Act requires NRC to evaluate the following alternative:
13(b)3 - encouraging and obtaining more effective evaluations, inspections, or audits of commercial nuclear power plant construction by independent industry or institutional organizations based on best experience and practices.
It is likely that audits by independent or professional organizations look-ing at the design process would have identified the quality assurance deficiency that was inherent in the transmittal of infonnation between the engineering organization and its consultants that led to the design error.
If the design process was not audited,'it is unlikely that the deficiency would have been discovered.
D.
Improvement of NRC's QA Program - yes The Authorization Act requires NRC to evaluate the following activitiesi 13(b)4 - reexamining the Comission's organization and method for quality assurance development, review, and inspection with the objective of deriving improvements in the Agency's program.
'It is clear from previous sections of this report that NRC was part of the problem. The fellowing changes to NRC's programs would have mitigated and possibly prevented the development of the design quality problems
~
discussed earlier:
a.
Modify the licensing review process for a construction pennit to cover the applicant's ability to effectively manage a project as complex and technicaTly demanding as the construction of a nuclear reactor in accordance with NRC requirements.
The construction permit review e
would'need to have included a review of the design procedures that the applicant prooosed to use and its relationship with its consultants.
DRAFT WORKING PAPER
~
j DRAFT WORKING PAPER j
29
- 4 A thorough audit o' the applicant's implementation of its proposed s
procedures might have been sufficient. The focus of this type of review would be for the applicant to demonstrate its capability to effectively manage and/or overview all aspects of the project, including quality assurance and control of design documents.
b.
Revise i.he NRC inspection program to 1) focus more on the design and engineering aspect of nuclear plant corisbuction, and 2) increase NRC presence and capability in the regional offices to review and overview design practices and the design process.
Part of the reason for NRC's failing to recognize the problem was the lack of NRC
~
inspection e,ffort,in the design process. This was due largely to inspection resources that were limited in both number and technical expertise to overview the design process.
w t
a E.
Conditioning the Construction Permit on the Applicant's Commitments to Submit to Third-Party Audits of His Quality Assurance Program - yes The Authorization Act requires NRC to evaluate the following alternative:
13(b)5 - requiring as a conoition of the issuance of construction permits for comercial nuclear power plants that the applicant enter into contracts f
or make other arrangements wi,th an independent inspector for auditing quality
, assurance responsibilities for the purooses of verifying quality assurance
~
performance. An independent inspector is a third party who has no respon-l sibilities f6r the design or construction of the plant.
This alternative, as it applies to this case study, was discussed under l
Alternative C above.
It is believed that, had this initiative been in place, it could well have prevented the Licensee from making the error that occurred.
A. comprehensive review of the implementation of the quality assurance procedures that existed within the design organization should 4
have revealed the design document. control deficiency that resulted in the designp[oblem.
DRAFT WORKING PAPER
~
REVISED DRAFT WORKING PAPER b
- APPENDIX A EVALUATION OF GENERIC KEY INDICATORS REVISED T1 RAFT WnpyIE_pAppt
DRAFT WORKING PAPER APPENDIX A EVALUATION OF GENERIC KEY INDICATORS FOR CASE C STUDY d
1.0 Licensee is fully committed to a program for assurance of quality A.
Project management appears to firmly believe that their plant has been and is being built with adequate quality -- maybe excessive quality.
Certainly it is their intent to build a quality facility.
This Licensee, through its construction organization, appears to have supported a good QA/QC effort in the on-site construction activities; however, this diligence has not, in all cases, extended to service contractors and materials suppliers. Prior to 1982, an equal comitment was lacking with respect to the engineering activities on the project.
This is reflected in the deficiency in management follow-up that allowed the violations of procedures and inadequate management reviews in 1977 to remain undetected for four years.
B.
Corporate QA audits construction activities on a periodic basis, but there did not appear to be the same attention given to engineering activities.
The Licensee has taken considerable care to separate quality assurance from quality control. The QC function is the responsibili+.y of the construction organization. This was supplemented in manyinstancesbytheenginekrswhodidthedesignoverseeingconstruction
' and/or resolving cons'truction problems. At one time, QA appeared to be l
a term used to describe an organization required by regulations; now the Licensee has an appreciation of its importance e.nd the coat of failing
\\
to adequately document engineering actions.
The utilization of the A-E's QA program for the IDVP gives evidence that the Licensee now acknowledges the need to apply QA controls to the engineering process.
/
11RAET WORKINGJAEER
DRAFT WORKING PAPER s
2 2.0 Responsibility and authority are clearly defined and properly implemented A.
At present, there appears to be clearly defined policy guides with respect to responsibilities and authorities for nuclear power plant construction and operational quality. Apparently, good experience in designing, constructing, and operating other types of power generating facilities led the Licensee to ~a'ssume that similar procedures would be adequate for this nuclear station.
This project has been a long time in the design and construction phase.
Contractor responsibilities and authorities and changing interfaces failed to keep up with the formaliza-tion required by NRC regulations over the past decade. This failure occurred, at least in part, because the project has been on the verge of l
completion for about eight years, and the need to change was probably not pressing nor thought necessary. The Project Completion Team members interviewed seemed clear as to their understandings of responsibilities and authorities.
u-There was some uncertainty as to the role of the chief engineers in the utility relative to the project.
The Corporate Manager of QA expressed a clear understanding of his responsibilities and authority.
B.
There are no observations fo th present organization; the Licensee is aware that engineering QA should have been more fonnal in the early program.
/
DRAFT WORKING __ PAPER ___
.[
DRAFT WORKING PAPER 3
3.0 Qualified work force is utilized A.
Overall, the work force employed on the Licensee's project appear well qualified.
The Licensee's engineering staff had limited nuclear experience entering into the project, and the staff was not large as measured by present-day stancards. Where the engineering work force was-not qualified, liberal use of consultants or contractors was employed, apparently for the most part, quite successfully. To effectively manage them requires that adequate quality assurance procedures be in place and followed carefully. This does not appear to have been the case with seismic consultants and other early contractors. The construction work forces which were employed appear to have adhered to good construction practices. Sonfe reservations evolved relative to the corporats QA staff.'
These came, in part, from impressions in one of the group meetings and, in part, from opinions expressed by a regional inspector.
B.
The Licensee apparently did not fully appreciate the importance of staffing with experienced QA personnel in the beginning.
4.0 Instructions, procedures, and drawings are clear and adequate A.
Instructions, procedures, and drawings were not reviewed in detail apart from those associated with loadings for the seismic consultant's analysis.
While the drawings werp inadequate for this one case, there was no evidence l
of pervasive ihadequacies in design drawings.
l l
It should be noted that the engineering work currently being done by the Project Completion Team is guided by the utility's engineering procedures and the A-E's QA manual.
Since this hybrid. team has existed for less than one year, one'would be surprised, indeed, if there have not been communica-tions probl, ems.
It was not possible to probe deeply enough to identify any specific problems, however.
/
1 T) RAFT WORKING PAPFR
DRAFT WORKING PAPER 4
B.
Presently, tha quality assurance organization reviews drawings, but does not sign them. This is consistent with the Licensee's philosophy of engineering maintaining sole responsibility for design.
5.0 Quality /0A program deficiencies are sought out and reported promptly A.
This factor seems to be strongly and effectively supported at the construction site.
The " mirror image" problem indicates a weakness in this regard in the engineering activities.
It must be noted, however, that deficiencies once discovered, have been promptly reported and addressed.
+
6.0 Corrective action program is effective A.
Good, once a problem had been identified.
The Licensee has been very-responsove to identified needs for corractive action; however, QA progi deficiencies in engineering may not have been regarded with the same intensity as QA deficiencies in construction as far as corrective action 1:~
was concerned.
B.
No observations made.
7.0 Des.ign review activities detect and ' resolve design deficiencies A.
The procedures'for design review appeared to be similar to those success-fully used by the Licensee in the design and construction of other types of generating facilities. Among other things, the overall designs were reviewed by chief.(discipline) engineers.
No data were obtained on the numbers and types of field changes.
The " mirror ~ image" and related problems represent a unique deficiency in the area of design review. Although the problem was eventually discovered and is being resolved, the discovery was more fortuitous than the result of an orderly process. At the present time, of course, very extensive reviews are in progress.
DRAFT WORKING PAPER
"4
/
DRAFT WORKING PAPER 5
8.0 Design input data is adequately controlled A.
The look-back reviews have not discovered major design or construction problems with the original plant.
Performance in this area was probably satisfactory during.the original plant design process.
There was a major breakdown in 1977 in the handling of seismic diagrams.
Major efforts to revies design input data control are currently being applied, especially in the IDVP.
B.
No observationc made.
9.0 Organizational structure is conducive to attainment of quality A.
No fault was id'entified with the formal organization structure; however, the (informal) position of power on the part of engineering is suspected to have been a factor in resisting the application of an effective design QA process.
This level of informal power now appears to have been significantly eroded.
B.
QC functions are performed by the departments responsible for the task.
10.0 Planning, scheduling, and budgeting provide the resources to do the job A.
The engineering problems whib,h have been so costly are suspected to have
' resulted, at least in part, from very heavy schedule pressures. Whether l
these pressure were real or felt was not established.
There was no indication of lack of resources applied to the project.
B.
No observations made.
e p
DRAFT WORKING PAPER
DRAFT WORKING PAPER 4
6 11.0 Design control process A.
As identified in numerous earlier indicators, this is the major breakdown which occurred on this project.
During the past year, this has been corrected with a very extensive IDVP and an internal "look-back" program initiated by the Project Completion Team.
B.
In the early days of the project, there was not a formal design control process which was independent of the engineering organization.
12.0 Work package development and control A.
Not investigated.
+
B.
No observations made.
13.0 Procurement control A.
A satisfactory evaluation of this indicator is difficult because of the
~
time period of this project. The IDVP consultant auditing the QA program has conceded that the numerous, significant deficiencies that have been identified are deficient by today's standards, but not by the standards existing at the time the procurements were made.
B.'Theoriginalseismic.contrac$didnotspecifyaQAprogram.
In fact, the contractor'was not required to explain its QA program until 1977 --
l long after much work was completed.
14.0 Nonconformance control A.
Not investigated.
B.
No observations made.
/
DRAFT WORKING PAPER
C DRAFT WORKING PAPER
/
7 15.0 -Soecial process control e
A.
No significant QA/QC problems have occurred at the construction site.
B.
Welders are qualified today and all indications are that construction practices have always required welder qualification. No observations made on cther processes.
16.0 Examination-test, and inspection control A.
This point was not probed in depth, but NRC regional staff and expressions at the site indicated that the construction site efforts were excellent.
B.
No observations made.
17.0 Calibration control A.
Not investigated.
B.
No observations made.
18.0 Records A.
Although major problems have,resulted from weak documentation practices in engineering, the a'v&ilable records led to discovering the error about four years after it occurred.
Further, we were advised by an NRC regional inspector that the records and traceability relative to construction (materials, heats, location, etc.) were unusually extensive and accurate.
e j
em DRAFT WORKING PAPER
r DRAFT WORKING PAPER
.a 8
19.0 Audits A.
The use of audits in the early years of the project appears to have been Jimited to those typically done in projects involving other types of generating facilities.
The audit of the design process was probably not a strong emphasis or the design control procedure deficiency would have been noted.
The audit program has been very extensively strengthened during the cast year.
B.
The present program includes audit activities; however, they were not verified.
The Licensee had a QA/QC program, but the problems they have experienced would indicate that they did not have an aggressive system to verify implementation _in the design control area.
NRC audit repo_r,ts gave the Licensee good reports on construction quality program implementa-tion.
20.0 Identification and control of material items A.
This was only superficially investigated; however, a NRC regianal inspector specifically commented that the utility had been far above average in this regard.
t 6
f DRAELWORKING PAPER
REVISED DRAFT WORKING PAPER e.
r
-mm APPENDIX B DEFINITION OF LEVELS OF CIUALITY FAILURE -
e 0
e l
e 9
e e
REVISED M WQEEINGJAPR
r' DRAFT WORKING PAPER e.
~'
APPENDIX B DEFINITION OF LEVELS OF QUALITY FAILURE CAUSES 1.
The Deepest Sense of Quality Failure There.are basic underlying causes of quality failure, which clearly transcend QA and QA programs. They can be characterized as broadly philosophical. They are at the extremity of the chain of causes (e.g., building a nuclear power plant without knowing how -- which h'as as necessary conditions 1) the licensee does not know how, and 2) NRC permits them to build, even though they don't know how).
It is usually very difficult, if not impractical, to develop recomendations that address such philosophical issues. These are, nonetheless, root causes.
For our purposes, we are defining root causes at the following, more ooerative level.
2.
The Operative Sense of Quality Failure There are basic underlying causes of quality failure, which frequently --
~
transcend QA and QA programs, but not necessarily.
They can be characterized as general. They are near the end of the chain of causes, but are limited to where it is practical to bring about corrective action (e.g., lack of manage-ment commitment).
It is at this level that corrective actions often treat many symptoms of poor quality.
It is in this sense that the term " root cause" applies in this report.
The,re is yet another level which-we have defined as symptomatic / procedural.
i 1
l 3.
The Symptomatic / Procedural Sense of Quality Failure These are often the imediate causes of quality failures. These can transcend QA and QA programs, but it is unlikely.
They are characterized as detailed and specific.
They are intennediate in the chain of causes and, as such, are subcauses of (2) sbove.
Recommendations for corrective actions at this level are relatively easy, but are likely to treat without addressing underlying causes.
g I
DRAFT WORKING PAPER