ML20195D652

PERFORMANCE-BASED Inspections

ML20195D652
Person / Time
Issue date:	06/30/1988
From:	Hawkins F, Jerrica Johnson, Liner R, Putnam C NRC, SCIENCE APPLICATIONS INTERNATIONAL CORP. (FORMERLY, WEIRICH & ASSOCIATES
To:	Office of Nuclear Reactor Regulation
References
CON-FIN-B-8733 NUREG-CR-5151, SAIC-88-3014, NUDOCS 8806230162
Download: ML20195D652 (27)

Text

. . . .

NUREG/CR-5151 SAIC-88/3014 Performance-Based Inspections l

}

Prepared by F. Hawkins, J. Johnson, R. Liner, C. Putnam Science Applications international Corporation i

Prepared for U.S. Nuclear Regulatory Commission 8806230162 000630 hk$ 5t h "h PDR

• - 4 NOTICE This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, or any of their employees, makes any warranty, expressed or implied, or assumes any legal liability of re.

sponsibihty for any third party's use, of the results of such use, of any information, apparatus, product or process disclosed in this report, or represents that its use by such third party would not infringe privately owned rights.

NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publications will be available from one of the following sources:

1. The NRC Public Document Room,1717 H Street, N.W.

Washington, DC 20555

2. The Superintendent of Documents, U.S. Government Printing Of fice, Post Of fice Bos 37082 Washington, DC 20013 7082

3. The National Technical Information Service, Springfield, VA 22161 Although the listing that follows represents the majority of documents cited in NRC pubhcations.

it is not intended to be exhaustive.

Referenced documents available for inspection and copying for a fee from the NRC Public Docu ment Room incl' Je NRC correspondence and internal NRC memoranda; NRC Of fice of Inspection and Enforcement t,ulletins, circulars, information notices, inspection and investigation notices; Licensee Event Reports; vendor reports and correspondence; Commission papers; and apphcant and licensee documents and correspondence.

The following documents in the NUREG series are available for purchase from the GPO Sales Program: formal NRC staff and contractor reports, NRC-sponsored conference proceedings, and NRC booklets and brochures. Also available are Regulatory Guides, NRC regulations in the Code of federal Regulations, and Nuclear Regulatory Commission issuances.

Documents available from the National Technical Information Service include NUREG series reports and technical reports prepared by other federal agencies and reports prepared by the Atc<nic Energy Commission, forerunner agency to the Nuclear Regulatory Commission.

Documents available from public and special technical libraries include all open literature items, such as books, journal and periodical articles, and transactions. federal Register notices, federal and y

state legislation, and congressional reports can usually be obtained from these hbraries.

Documents such as theses, dissertations, foreign reports and translations, and non N RC conference proceedings are available for purchase from the organization sponsoring the publication cited.

Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the Division of Information Eupport Services, Distribution Section, U.S. Nuclear Regulatory Commission, Washington, DC 20555, Copies of industry codes and standards used in a substantive manner in the NRC regulatory process J are maintained at the NRC Library, 7920 Norfolk Avenue, Bethese. Maryland, and are available j there for reference use by the public. Codes and standards are usually copyrighted and may be j purchased from the originating organization or, if they are American National Standards, from the i American National Standards Institute,1430 Broadway, New York, NY 10018. l l

l i

i

l NUREG/CR-5151 SAIC-88/3014 Performance-Based inspections i

Manuscript Completed: March 1988 Date Published: June 1988 Prepared by F. Hawkins/NRC J.H. Johnson /Weirich and Associates R.T. Uner, Jr.. C.H. Putnam/SAIC Sci:nce Applications international Corporation 1710 Goodridge Drive McLean, Virginia 22102 Prcpared for Division of Licensee Performance and Quality Evaluation Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Wcshington, DC 20555 NRC FIN B-8733 I

ABSTRACT This report describes the concept of performance-baseo inspections that is being taught in NRC's training course, "Inspecting for Performance." This concept has been endorsed and is being implemented by the Nuclear Regulatory Commission (NRC). NPC performance-based inspections concentrate on activities that impact plant reliability and safety. The inspections begin with performance-based observations, and then the inspectors let discrepencies or uncertainties lead to inspection of other areas, such as quality verification organization effectiveness, training adequacy, and procedural controls. This inspection approach departs from past NRC practices that emphasized documentation and program review as a means to measure operational safety.

The first goal of performance-based inspections is to improve the NPC inspector's ability to accurately evaluate plant safety and reliability. This goal will be achieved by increasing the inspector's emphasis on actual observation of ongoing facility work activities and reducing the erphasis on document and program reviews. The secondary goal of performance-based level inspections is to encourage licensees to manage their facilities in a performance-based manner.

PERFORMANCE-BASED INSPECTIONS The NRC training course, "Inspecting for Perforrance," upon which this report is based, was sponsored by the Quality A:surance Branch of the Office of Nuclear Reactor Regulation. it is one of several NRC initiatives to improve NRC inspection effectiveness and encourage licensees to manage their facilities in a performance-based manner by focusing attention on activities important to reliability and safety. Appendix A presents the Policy Issue Memo, SECY-87-220, "Assurance of Quality," which endorses this concept.

The Inspecting for Performance course serves as the cornerstone of a performance-based NRC inspection program. It focuses NRC inspectors on liceO? performance to determine if the nuclear facility is operated in a safe and reliable manner. This training course was pilot tested at the NRC Technical Training Center in September 1987. It is a 20-hour program of lectures and workshops that present the concepts and demonstrate the general and specific applications of performance-based inspection techniques. It is currently being taught at NRC headquarters and regional offices to provide guidance to NRC inspectors. The NRC is encouraging the nuclear industry to adapt a suitable version of the course for their use.

This report briefly describes the concepts and principles addressed by the course. Its purpose is to disseminate the performance-based inspection concepts to the nuclear industry.

BACKGROUND The nuclear industry is changing. Early activities were concentrated on the design of facilities; construction gradually became the predominant activity; and now the operations phase is prevalent at most facilities. As a result of this evolutionary process, licen;ees have passed through the phases of site selection, construction, licensing, and startup. Similarly, the NRC has evolved in an attempt to ensure that the Commission operates effectively while accommodating the evolving activities and needs of the industry.

In the past, NRC inspections have emphasized documentation review to achieve a comprehensive overview of facility acceptability. Because adequate and valid documentation was a primary resource for licensing evaluation, emphasizing the review of documentation, coupled with some observation of inprocess work activities, was an appropriate way to judge acceptability.

Today this emphasis is changing. With acceptability of the facility established through the licensing process, the emphasis now is on ensuring ongoing quality execution of operational responsibilities.

A major goal of performance-based inspections is to increase the emphasis on actual observation of ongoing facility activities and to reduce the I

emphasis on document review as the primary resource for NRC inspections.

Facility activities, such as maintenance, engineering, and design control, involve many individual activities that are very important to the safe and reliable operation of the facility. Many of these activities can only be properly evaluated by someone observing the critical steps of the activity; the evaluation should stress the most important parts of the activity, and should not dwell on matters that have little significance. By concentrating the inspection effort on the areas that have the greatast effect on facility reliability and safety, the value of the inspection is greatly enhanced.

The concept of evaluating components' performance or work activities on the basis of their effect on the safe operation and reliability of the facility is called performance-based evaluation. Performance-based evaluation can be applied to all areas of nuclear operations, including management, maintenance, and modifications. A performance-based inspection program stresses the activities that impact facility reliability and safety.

The implementation of t. performance-based inspection program by the NRC is an attempt to focus NRC inspections on technical activities and reflects support of the performance-based approach to managing the operation of a nuclear facility. The course addresses issues and application of the performance-based approach for the inspectors. Such questions as "Will this make a difference in safety or reliability?" and "Where should I look to best determine a licensee's ability to perform this task?" will become part of the inspector's thought process.

THE CONCEPT OF QUAL.ITY ASSURANCE The image of someone inspecting or checking work being performed by someone else often comes to mind when people hear the term quality assurance.

Although partially correct, thi., image is not a complete picture. The person checking or inspecting work may be part of the quality assurance organization, but is only a small part of an effective quality assurance l program.

Quality assurance (QA) is defined as "all those planned and systematic actions necessary to provide adequate confidence that an item or facility will perform satisfactorily while in service." The key words of this definition are confidence and perform. This definition says nothing about inspectors, auditors, procedures, or testing. It speaks of confidence, which is not a document or an activity, but a certain state of mind. It also speaks of an ability to perform, not inspect or audit. The key to understanding the concept of quality assurance is understanding that f effective quality assurance leads to confidence in the safe and reliable performance of all components and activities.

The key to applying the concept of quality assurance is based upon determining how to achieve adequate confidence that an item or facility will perform satisfactorily in service. Can facilities apply quality assurance 2

concepts without developing a formal QA program? A formal QA program, as described in ANSI N18.7-1976, is an organizational structure whose purpose is the attainment and verification of the desired level of quality. The two functions noted--attainment and verification--are of equal imprtance, but often when quality assurance is brought to mind, only the verification function is envisioned. Frequently, people think of QA as a QA program, of a QA program as documentation, and of documentation as a way to meet all the i standards. It is true that the documentation is necessary, but the workers must attain quality before it can be verified. The first step has to be attainment (doing); then comes verification (checking).

In assigning responsibility for the attainment and verification functions of quality assurance for the operational phase of nuclear power plants, the American Nuclear Society issued ANSI N18.7-1976. In that standard it states ".. quality assurance should be recognized as an interdisciplinary function involving many organizational components and therefore should not be regarded as the sole domain of a single quality assurance group" (paragraph 3.2). This description of quality assurance covers all aspects of performance, including attainment and verification.

APPLICATION OF QUALITY ASSURANCE CONCEPTS It is interesting to compare the sizes of the work forces involved in attaining quality with those verifying quality. Usually there are at least

, 10 people who perform work for each one who inspects or audits it. Yet when the work does not pass inspection, it is often referred to as a "QA organization problem." Who controls the characteristics of the item? How is the concept of quality assurance applied? Workers should recognize the fact that they are the first line in quality control. They control many characteristics of the items produced.

There are excellent examples of applying quality assurance in companies that have no formal QA program. These companies put their emphasis upon l attainment so that verification requirements are minimized. Take as an

! example a successful restaurant. To achieve success it must provide a

, consistently high level of quality. Each person involved in the operation I has a part in the attainment of the required level of quality and does their part without a formal quality assurance program. The buyer must find and stock the required supplies, the chef must combine and prepare the ingredients for the meals, and the waiters must serve the meals properly and efficiently, if any one of these workers fails to achieve their level of quality, the reputation of the restaurant may suffer. In this example one l can identify the contributions, roles, and function of the workers that are l

involved in attainment; the customer provides the verification.

l An extension of this example can also be made by comparing restaurant activities with the QA criteria established in 10 CFR 50 Appendix B. Design 3

l

[ t

Control (Criterion 111) is established through the use of recipes to main-tain a consistent quality level. Since most recipes are written as instructions, a connection to Criterion V (Instructions, Procedures, and Drawings) can easily be made. The restaurant must maintain an inventory of supplies for preparation; those supplies that are perishable must be kept fresh and handled carefully. Procurement Document Control (Criterion IV),

i Control of Purchased Material, Equipment & Services (Criterion VII), and i

Handling, Storage, and Shipping (Criterion XIll) could be applied to this i area. Test Control (Criterion XI) is maintained in a restaurant through l tasting the food as it is prepared, and the final acceptance test is provided by the customer.

l Therefore, even though the restaurant does not use a formal or written QA program, one can demonstrate from a review such as the one above that basic quality requirements, including those in Appendix B, must exist wherever a consistently high quality product is produced. The key for success is not in the procedures, audits, and tests--success is dependent upon implementation and application. The performance of the cook following the recipe will determine the quality of the food. The best recipe can only produce high quality results if the cook's implementation is successful.

The ultimate test, with the customer, will be an assessment of the preparation, presentation, and service. Even though the cook prepares the meal prcperly, if either the presentation or the service is poor, the restaurant will not prosper. A total quality effort is needed for success.

The similarity between employees at nuclear facilities and in a restaurant is that both are skilled workers who strive to produce a high quality product. If tne workers are successful in attaining the desired level of quality, then the inspectors and auditors will be successful.

Up to this point, the examples used have addressed personnel in the process of producing an item or component, but the terms doers and workers can be applied to almost everyone. Quality assurance involves each person's performance at every function and level, be it administrative, operations, or engineering. Every function in a nuclear plant produces a product that has a direct bearing on the overall reliable and safe operation of that facility. Just as each person who works in a restaurant has an effect on the restaurant's success, each person's actions at a nuclear facility hav?

an effect on the operation of that facility.

In an effective quality assurance program, the workers are aware of quality requirements, and both the employees and their management are confident of their ability to perform to the required level of quality. They work with the quality verification (QV) personnel to be sure that what they do is checked at required intervals. These QV personnel are technically competent i and understand the requirements and acceptability standards for the work. I They track the progress through inprocess inspection and try to anticipate i difficulties before a reject or stop-work point is reached. In addition, they use an overview program to report trends and combine trend analysis with feedback from quality control personnel to provide management with regular 4

I

QV status reports. QV feedback is used by management to assist in planning similar work in the future. When the work is done, there are no surprises--

the attainment and verification have been coordinated and quality assured.

APPLICATION OF PERFORMANCE-BASED CONCEPTS Consider how quality is evaluated in the context of the restaurant example.

If a verification of quality is required, a normal method used is to go to the restaurant and taste the food. This onsite inprocess observation of the restaurant's performance provides a realistic evaluation. First the performance is observed (taste, service), then an evaluation could be made.

Was it good? What could be improved?

Simply stated, this illustrates the concept of performance-based inspection.

The process is simnle: observe, then evaluate. The key point is to observe and evaluate on te basis of performance. Observe the areas most important to performance; then perform an evaluation based on improving performance.

In nuclear operations the application of quality evaluations focuses on the meaning of the term performance. A facility that is a top performer does not necessarily run longer or have a higher megawatt rating. The NRC believes that a top-performing nuclear facility operates with more reliability and safety. Therefore, when the term performance-based is applied to the nuclear industry, it means based on reliable and safe operating performance. A performance-based inspection program is one that analyzes and evaluates with emphasis on safety and reliability:

Inspection should be performed on activities that make a difference.

Inspection should be performed in a manner that emphasizes safety and reliability, not trivia with no real impact on facility performance.

Inspection should be performed by qualified personnel who have the necessary technical capabilities to accurately observe and evaluate an activity.

With these three assertions, the concept of performance-based evaluations can be applied to NRC inspections that focus on activities with the greatest impact on safety and reliability. Appendix B presents an overview of the method and sequence used for a typical NRC performance-based inspection.

The preferred approach for performance-based inspections is direct observation of the activity in process. In many cases, however, the activity to be inspected will not be in progress at the time of inspection.

Since the evaluation must still be performed, the inspector must perform an evaluation without the benefit of direct observation of the activity.

Alternative methods include discussions with plant personnel and reviews of 5

i pertinent documents for performance-based information. The major shortcoming of these alternatives is that the inspector's activity takes place away from the point of occurrence. Direct observation is always the preferred inspection method and inspectors should make every effort to actually observe activities in the selected area. The inspectors' discussions with plant personnel and their document reviews should be used to enhance and verify performance-based observations.

When the inspector encounters problems during a performance-based inspection, he should investigate the cause of the problem. This evaluation should determine if it is a specific or widespread problem and if the root cause can be traced to procedures and controls, training, work control, or I

other programs. He should also check to determine if the problem has been identified by the facility quality verification organization. If the QV organization has identified the problem, the inspec'or should assess the planned corrective action. If the aroblem has not been identified, the inspector should try to determine w1y it was missed. This line of evaluation might then lead to an examination of the procedures being used by the QV organization.

Notice the change in sequence between the traditional inspection method and the performance-based method. The traditional method has personnel training and qualification or procedure approval verified up front as a separate function. In the performance-based method, a problem with work performance leads the inspector to other areas of investigation. The inspector should follow the trail of a problem until he finds the cause.

The inspector should use the same sequence when conducting a document inspection. The documents relating to the performance of the activity are inspected first. If problems are encountered, the inspector will need to investigate the background and supporting information to determine the cause of the difficulties.

SUMMARY

\

The NRC performance-based inspection program is based upon evaluating, j through direct observation of plant activities, whether quality has been achieved in plant operations. Performance-based evaluations and decisions emphasize action in areas that make a difference and are critical to plant reliability and safety. Addressing an issue that affects the safety or reliability of the facility focuses attention on activities that achieve reliable and safe facility operations.

Quality assurance is an interdisciplinary function that stresses, in its basic definition, confidence and performance as keys to success. Quality assurance is not a department, a manucl, or a program; it is a state of mind. The quality assurance function should not be regarded as the sole l domain of a single quality assurance group; it is everyone's responsibility. l 1

1 6

1 i

___._m_. _

The purpose of a quality assurance program is the attainment and verification of quality. But quality must be attained before verification can occur. Accurate documentation and correct procedures are not the measure of success in a quality assurance program. The success comes from an acceptable level of quality being attained and verified.

The application of performance-based concepts is being used by the NRC to broaden the scope and direction of NRC inspections through performance-oriented techniques that are based upon observation and evaluations of activities affecting reliability and safety. This approach reduces emphasis upon document reviews and attention to indicators that do not impact facility reliability and safety. Similarly, by focusing attention on activities important to reliability and safety of operations, NRC performance-based inspections will provide a model that encourages licensees to manage facility operations in a performance-based manner.

i 7 l

APPENDIX A

\

NRC POLICY ISSUE, SECY-87-220 .

4

fo%

August 31, 1987 POLICY ISSUE SEcy-87-220 fg: The Comissioners UnformatIOn)

From: Victor Stello, Jr.

Executive Director for Operations

Subject:

ASSURANCE OF QUALITY

Purpose:

To infom the Comission of the staff's shift in emphasis from "compliance-based" inspections of licensee's quality verification organizations to "perfomance-oriented" inspections of these organizations.

Background:

In May 1984, the staff issued NUREG-1055, "Improving Quality and the Assurance of Quality in the Design and Construction of Nuclear Power Plants." NUREG-1055 identified poor management as a major contributor to quality breakdowns. The report also addressed issues that went beyond traditional qualityassurance(QA)issuesandfocusedon(1)stepsthat could be taken in licensees' QA programs and (2) changes in the NRC oversight of such programs to improve the perfomance ofthelineorganizations("qualityachievers")andthe licensees' QA organizations ("quality verifiers").

Traditionally, a licensee established a QA program that was I

reviewed and accepted by the NRC; the NRC then "inspected" the implementation of the licensee's programs. These inspections were heavily oriented towards programatic reviews rather than reviews of actual day-to-day work in progress or actual plant hardware.

The Comission, in a December 12, 1985 response to Congress involving the issues raised in NUREG-1055, made it clear that it is the responsibility of licensees--not the NRC--to achieve and assure quality. The Comission i 'icated that improvements in quality must come from the industry itself, and that the key to achieving quality and assuring quality lies in line management. Therefore, the Comission concluded that the NRC programs for improving quality should focus on the ability of licensee management to ensure that significant safety problems are either prevented or detected early and adequately resolved.

CONTACT: A. Bill Beach, NRR 492-8413

The Comissioners Discussion: In the three years since the completion of NUREG-1055, many of the NRC programs it generated to focus on the performance of line management have been completed or are ongoing. Because the majority of the NRC's regulatory oversight and insp;ction activity is aimed at monitoring the perfonnance of the licensee's line organization in achieving quality, most of these programs were initiated in NRC organizations outside of the NRC Quality Assurance Regulatory Programs. These include, for example, the development of performance indicators and improved diagnostic inspection techniques. Therefore, with most of the agency's focus on the line organization, it is now appropriate for the NRC Quality Assurance Regulatory Programs to focus principally on initiatives aimed at evaluating the effectiveness of the quality verification organizations in identifying, reporting, and ensuring the correction of significant safety problems.

I As a result of the recent NRC reorganization, the Office of Nuclear Reactor Regulation (NRR) Quality Assurance (QA) Branch is responsible for developing and implementing a comprehen:1ve program for assuring quality and reliability of reactor licensee facilities. These activities were responsibilities of the fonner Office of Inspection and Enforcement (IE) QA Branch.

Other responsibilities of that Branch were transferred to the Offices of Nuclear Materials Safety and Safeguards and Research.

l These discussions are liinited to those functions transferred to the NRR QA Branch. The major emphasis of this branch is now on reactor operations. Thus, the staff believes it is crucial that NRC Quality Assurance Regulatory Programs be oriented to evaluating the implementation of licensees' quality verification programs in the identification and correction of safety-significant technical problems, in addition to merely evaluating the processes of licensees' quality assurance programs. Further, because there are essentially no new plants being licensed, the role of a licensee's quality j verification program has shifted from one of developing and monitoring programs to oversee contractors to one of being a more integral part of the licensee management feedback loop.

Consequently, repetitive reviews of licensee's quality assurance programs that have been in place for some time would not be an appropriate use of NRC's limited resources.

Quality verification is synonymous with systems of checks, audits, inspections, and other fonns of verification performed by a licensee's Quality Assurance organization, the Quality Control organization, the Quality Engineering organization, and/or other independent review organizations such as a Plant Operations Review Comittee and Safety Operation Review Comittee. These organizations monitor the overall

The Comissioners performance of the line organization and are responsible for ensuring that quality is achieved and that significant problems are avoided.

These organizations--in particular, the Quality Assurance organization--must have the ability to understand significant operational safety problems and be technically credible to ensure that NRC requirements are satisfied. It is imperative that licensees' verification organizations understand reactor operations and plant systems, and that they be involved in the day-to-day operations in the plant. Monitoring processes in a program review cannot be a substitute for the actual observation of safety-significant activities. The NRC staff expects the staffs of licensees' verification organizations to consist of qualified professionals who:

(1) Routinely monitor Control Room decorum and behavior, including monitoring of back-shif t activities.

(2) Understand fund 3 mental engineering concepts and design analyses as the concepts / analyses relate to plant system functions, s (3) Recognize the interrelationship of plant systems and

• components, and understand how the systems / components function.

Further, in addition to their technical capabilities, the staff also expects verification organizations to be perfonnance oriented so they have the capability to identify, report, and ensure the correction of safety-significant problems. As such, the staff emphasizes that quality cannot be inspected or audited into a work product. The assurance of quality rests with the line organization responsible for the work product / function. In this regard, verification involves the use of technically knowledgeable staff to provide feedback to line management. It is not acceptable for licensees to rely solely on the existence and audit capability of a QA/QC organization for verifying quality. In this regard, there ara many indicators that licensees can use in detennining the extent to which line managers rely on QA/QC for assuring quality. For example, the effectiveness of c'orrective actions in rest,1ving previously identified problems is a measure of properly dealing with the root cause.

The staff has recently completed the first inspection initiated by the NRR QA Branch to assess the effectiveness of licensees' quality verification organizations. Temporary Instruction 2515/78, "Inspection of Quality Verification Functions," (enclosed) describes the inspection guidance for

)

The Commissioners l focusing on the perfomance of quality verification organi-zations. The techniques assess whether quality verification organizations are, in fact, providing licensees with an effective and independent internal problem-detection capability.

Further, the staff expects to determine whether quality verification organizations are given adequate technical resources and support from management, whether management really relies on quality verification organizations to evaluate its operations technically as well as procedurally, and whether line management is taking corrective action on safety-significant issues raised by the quality verification organizations.

This type of inspection technique accomplishes two significant goals. First, it sends a clear message to the industry that the NRC expects licensees' quality verification organizations to be technically credible with the understanding necessary to monitor reactor operations. Second, it places the emphasis of NRC inspections on technical, safety-significant issues rather than on QA programatic reviews. This sends a clear message to the industry that the NRC is aggressively pursuing technical issues.

The staff in no way means to imply that the NRC does not consider programmatic inspection processes important; it is the focus of the inspection process that must be changed. The "perfomance-oriented" inspection technique focuses on significant safety problems first. Programs are then reviewed to evaluate deficiencies and to detemine the effects on the safe c'eration of the facility.

Further, the staff in no way means to imply that "perfomance-oriented" inspections de-emphasize the importance of enforcement. Strong enforcement actions must be taken when significant violations of NRC requirements are identified.

The staff believes that, although these types of inspections may be viewed as more subjective since the areas and scope of inspection can vary from site to site, they will result in improved enforcement actions that focus on the underlying causes of identified violations of NRC requirements. By focusing on performance of the quality verification organizations, the staff can better evaluate the significance of technical issues identified by the quality verification organizations and the ability of the organizations to comunicate these issues to line management, and ensure effective resolution.

l This reorientation of NRC inspection efforts requires training and increased interface with regional inspectors, by NRC

The Comissioners managers, at,d NRC staff. An upgraded QA training course is being developed to replace the one put in place following the Ford Study. It is being structured for a broader technical audience. The course will feature lessons learned from studies such as the Ford Study, and it will have as one of its major objectives improved staff assessment of licensees' quality and quality assurance programs through technical inspections. The staff intends to proceed to iraplement the shift in emphasis in the assurance of quality program described above.

f bj V ctor Stello, Jr.

Executive Director for Operations

Enclosure:

Temporary Instruction 2515/78 DISTRIBUTION:

Commissioners OGC (H ST)

OI OIA GPA OCA ACRS SECY l

ENCLOSURE C

p

,.. .,k UNITED STATES

. NUCLEAR REGULATORY COMMISSION 5 OFFICE OF INSPECTION AND ENFORCEMENT Washington, D.C. 20656

• g..... -

r.

INSPECTION AND ENFORCEMENT MANUAL QAVT TEMPORARY INSTRUCTION 2515/78 INSPECTION OF QUALITY VERIFICATION FUNCTIONS 2515/78-01 INSPECTION OBJECTIVE To assess the effectiveness of licensees' quality verifi;ation organiza- (

tions.

2515/78-02 BACKGROUND Quality verification is synonymous with systems of checks, audits, inspec-tions and other forms of verification. Licensee management has a number of organizations which perform those verifications. They include the Quality Assurance (QA) organization, the Quality Control (QC) organization, the Quality Engineering organization, and independent review groups, such as the Independent Safety Evaluation Group (ISEG).

Quality verification organizations act in a measurement and advisory func-tion, monitoring the overall performance of the plant; identifying substan-dard or anomalous performance, or precursors of potential probless; report-ing findings in an understandable form in a tienely fashion to a level of line management having the authority to effect corrective action; and promptly verifying the effectiveness of the corrective action and reporting those verification results back to line management. An effective quality verification organization is technically and perfomance oriented; it fo-cuses its efforts toward and products as opposed to being concerned only with processes and procedures. The organization should have technical re-sources available to it, and it should be aggressive in searching for, identifying, and following up on problems.

2515/78-03 INTRODUCTION TO THE INSPECTION This temporary instruction (TD provides a means (1) to ensure that quality verification organizations are effectively contributing to the identifica-tion, solution, and prevention of safety significar.t technical problems and deficiencies in plant systems and operations and (2) to measure the effec-tiveness of line management in ensuring that identified quality deficien-cies are responded to promptly and completely. This inspection will re-quire the inspector to make objective and subjective judgments based on information obtained through interviews, observations, and review of avail-able documentation.

Issue Date: 01/12/87

This inspection will use either significant plant proble2s that have re-cently occerred or significant NRC technical inspection findings as the vehicle for assessing quality v?rification effectiveness. This is appro-priate to attess the roles of the quality verification organizations and line managemeat before and in resolution of an issue. Quality verification organizations hsve responsibility for ensuring that quality is achieved and problems are avoided. Thus, the presence of problems reflects on quality verification organizations' performance as well as that of line management.

In performing an assessment of this type, the inspector sho01d understand

.the roles, responsibilities, and interactions of all quality v'erification l organizations. It is 1,mportant to examine all organizations that fcreally i conduct quality verifications. The cumulative capability of these crgan-izations in coritributing to safe plant operatinns is indcative of the overall quality verification program's effectiveness.

l This TI can be used alone or be integrated into an existing NRC team. It is anticipated that it will be used selectively and at specifically chosen sites.

2515/78-04 INSPECTION REQUIREMENTS f

This inspection assesses the effectiveness of the licensee's quality veri-fication organizations in identifying technical issues and problems having safety significance and in following up to ensure they are resolved in a 1 timely manner, as opposed to being procedure and process oriented in their '

activities.

In general, the approach is to' evalut.te why the licensee's quality verifi-cation organizations were not effective in identifying problems that have been identified by inspections of NRC or other third parties; or similarly, why licensee quality verification organizations have not been effective in identifying and addressing the precursors or root causes to events, huality verification organizations perform various types of verificatior.s, such .as audits, surveillances, and third party observations. They perform their verifications in a number of functional disciplines, such as mainten-ance, operations, and design changes.

04.01 In the area se*4e.:ted fer this assessment, the inspector should per- <

form a detailed analysis of existing problems. This should be eccomplished throegh direct observation of in process activities, interviews of plant personnel, and documentation reviews of a se-} active sample of completed verifications performed within the past 12 months.

04.02 Based on the results of these activities, the insy.itetor should as-sess and draw conclusions regarding how well the licensee accomplished the following:

a. planaed for each urification activity

b. provided individuals of adequate technical experien;:e and expertise to conduct the verification activity Issue Date: 01/12/87 2515n8

l 1

c. conducted the verification . activity in adequate' onth and with the appropriate emphasis on technical activities h

d. implemented its corrective action responsibilities 2515/78-05 Ill5PECTION GUIDANCE This guidance is meant only to p'rovide helpful ideas in various areas and is not meant to be a check list for the inspection; it is neither all-in-clusive nor a minimum requirement. Inspecto p, who are comfort.able with and experienced in this type of results-orient / A inspection are encouraged to direct their efforts where they will be most productive and to provide com-ments which can be used to improve this guidance for less experienced in-spectors.

J 05.01 NRC Inspector Preparation -

j

a. Goal. The primary goal of thi inspection is to establish whether the verification organizations a$re looking for and finding substan-tive technical problems. #

b. Method. The inspector should j perform a detailed analysis of the

, selected issue (i.e., problevareas and occurrences of events) to understand the technical problems and events, why they occurred and what the roles of the verification organizations and line manecement were. For selected problee and events, this analysis should in-clude (1) determining the chaipjof events leading to the problem's occurrence, (2) developing an iJnderstanding of the technical and work activities associated with the issue, (3) determining the in-femation that is needed to understand its generic implications, (4) determining the extent to which the licensee identified precursors to the problem and investigated the facts surrounding i ts occur-rence, and (5) identifying the licensee's corrective actions to cor-rect the problem and the remedial actions taken to preclude its re-currence. /

c. Information Base. Plant-specific information, including the find-ings of previous NRC inspectOns and information obtained from dis-cussions with the resident inspectors, sNuld be used to perform the analysis. Other sources, such as SALP reports, LERs, ir.fermation notices, IE bulletins, employee concerns, PRAs, and current internal licensee plant status or problem reports, clso should guide the in-spector,

d. Selective Samples. Selective samples are samples chosen to provide a quick picture of performance and, thus, provide for a more effec-tive inspection. Items should be selected in areas w'pe known on suspected. deficiencies exist. The size of the selective sample is '

establish 1 when the inspector is satisfied that sufficient data / ,I ) ;

have been reviewed and documented to support a conclusion about the area sampled.

e. Experience and Expertise. It is important that inspectors be ac-companied by individuals having the necessary technical or opera-tional experience and expertise when assessing areas other than ,

their own areas of expertise.

2515/78 Issue Date: 01/12/87

05.02 Inspection Requirement 04.02a

, )'1 ,

a. Preparation. The inspector should assess the licensee's planning to assure that it takes into account trends in performance and responds to events. The development of a written audit plan and the selec-tion and orientation of the auditors should be reviewed to assure that the effort is well organized and completed in an efficient man- 3 ner. 1

b. Depth. Time spent on a verification, when compared to the complex-ity into of the the subject depth of matter being verified, the verification itsshould and provide insight importance to the licensee. The level of the verification's detail should be deter-eined. For exunple, is the licensee's design verification activity only a check to verify that the calculations and analyses have been performed, or is it a check of the calculations' validity and a ver-ification of the engineering basis for them?

c. Scheduling. Scheduling should be dynamic. Successful verification programs use a "living" schedule that permits important plant ac-tivities and events to be verified and documented as they are oc-curring, or shortly thereafter. This provides for the greatest in-pact and most effective followup.

j05.03 Inspection Requirement 04.02b

a. Parsonnel Experience and Expertise. The importance of having per-sonnel with direct experience and expertise in the area they are reviewing cannot be overemphasized. Personnel interviews will pro-vide' useful inforsation. Likewise, qualification records, which should be current and readily available, are useful for evaluating an individual's capability to competently verify a specific area.

However, this should not be used as the sole basis for questioning an individual's ability. The fewer technically experienced perton-nel a quality verification organization has, the more closely the licensee's verifications should be assessed to determine its ability to detect operational and technical quality flaws.

b. Use of Outside Assistance. For members of the organization to function af fectively in areas other than their own area of expar-tise, it is essential that they be accompanied by individuals who ,

have the necessary technical or operational background to be able to find significant, but often subtle, quality conditions requiring corrective action.

'05.04 Inspection Requirement 04.02c 3

a. Types of Verifications. Organizations perform various types of ver-

~ifications. Aucits are the most ob/ious. The inspector should be aware of the various types of verifications and their relation to other parts of the quality verification program,

b. Achievement Orientation. The ifcensee's verifications should be in depth, not superficial, and should emphasize technical achievement more than programatic conformance. However, the results of an achievement-oriented quality verification may indicate that wider l

Issue Date: 01/12/87 251558

problems exist. In such a case, the quality verification organiza-tion should then examine the programmatic implications and identify other technical areas where review is warranted,

c. Verification Activity Report. It is important for the verification report to accurately reflect the collected data. Otherwise manage-ment would be receiving an incomplete or biased per,spective of the plant's quality achievement. A report that overstates the effort expended or draws positive conclusions from superfi.cial verifica-tions should be identified as a weakness in the verification pro-gram. Problems with timeliness, clarity, or accuracy in reporting may hamper the corrective action process and say indicate attitudinal problems as well. Listen for symptoms of these problems during personnel interviews.

d. Communication and Cooperation. Through personnel interviews, the inspector should evaluate the adequacy of communication and coopera-tion between the verification organization, the functional disci-plines and line management.

05-05 Inspection Requirement 04.02d

a. Corrective Action Response. The inspector should evaluate each re-sponse and the organization's actions subsequent to the response for adequacy. The actions should correct root causes, not merely the symptoms of those causes. The implications of the deficiency on other plant activities should be a consideration in the assessment of the response. Appendix B to 10 CFR 50 requires that the cause of the condition is deteMned and remedied to preclude repetition.

For each analysis re' - nd, the inspector should evaluate the re-ported root cause ano e adequacy of the steps taken to determine the actual root catm

b. Followup. The organization snould follow up corrective actions in an aggressive manner. It should have access to upper line manage-ment and the authority to effectively use that access. The inspec-tor should determine the extent to which the quality verification organizations are ectually meeting with line management at this is a key indicator of their effectiveness.

c. Escalation. The inspector should assess the effectiveness of the verification organization and line management to determine whether identified quality deficiencies are responded to promptly and ef-fectively. Although a separate escalation procedure may not be re-quired, the organization should have established a means to bring the attention of upper management to those problems which are not promptly and adequately corrected. Examples of significant correc-tive actions remaining open for excessive periods or being extended repeatedly without apparent reason are indicative of an ineffective corrective action followup program. Those examples also may be in-dicative of a line management problem.

d. Closure. Closures should not have been based on superficial reviews or completed without a detemination that the corrective action will be effective in preventing recurrence or emergence of similar prob-lems.

2E15/78 Issue Date: 01/12/87 7

e. Trending. The organization should have an established and effective program designed to identify significant trends.

f. Corrective Action Tracking Systems. Various deficiency tracking and corrective act;on systems exist inside and outside of the or';aniza-tion. The total extent of those systems and their overlapp' ng re-sponsibilities and requirements, if any, should be identified. The more overlapping and ambiguous the system control,-the more likely it is that problems will occur, such as a deficiency being tracked in a system which is not monitored by the group responsible for the corrective action.

2515/78-06 REPORTING REQUIREMENTS l Document compir: tion of this inspection in routine inspection reports.

1 2515n8-07 EXPIRATION This TI will remain in effect for twelve months after issuance.

l 2515/78-08 IE CON 1ACT Questions regarding this TI sho'1d be addressed to Frank Hawkins on FTS 492-8490 (or 301/492-8490) or Wayne Scott on 492-4220.

2515n8-09 MODULE TRACKING SYSTEM INPUT h cord actual inspection effort against Module Number 25578.

This TI is to be performed at licensee sites using the inspection hours allocated by IE for the performance of the following inspection procedures (IP). The requirements to perform the following IPs are held in abeyance during the effective period of this TI. However, those IPs may be useful for a more detailed review in their specific subject area and would be es-pecially useful if the inspector feels that an in-depth compliance- ori-ented inspection.is needed.

IP Title 35701 Quality Assurance Program Annual Review 37703 Tests and Experiments Program 38701 Procurement Control Program 38702 Receipt, Storage,and Handling of Equipment and Materials Program 39701 Records Program 39702 Document Control Program 40700 Onsite Review Committee 40701 Offsite Review Committee 40702 Audit Program 40703 Offsite Support Staff 40704 Implementation, Audit Program END Issue Date: 01/12/87 2515/78

1 l

APPENDIX B TYPICAL NRC PERFORMANCE-BASED INSPECTION i i

)

OBSERVATION OF PERFORMANCE

^

PROBLEMS WITH ' '

MANAGEMENT [

OBSERVE SIMILAR PERFORMANCE BY ANOTHER GROUP YES NO ROBLEMS IDE IFIE I

INVESTIGATE INVESTIGATE TRAINING, PROGRAMMATIC OUAUFICATION OR PROBLEM SELECTION OF 1ST GROUP FOL' OW EVIDENCE TO OTHER PROGRAMS AND DETERMINE CAUSE OF PROBLEM 0 "

fET N VE FICA ION CONTROL PR MS s NO YES fgyg Y WHY NOT?

INVESTIGATE CORRECTIVE INVESTIGATE ACTION OV EFFECTIVENESS l

v i sec a . . . i c s, *, o. , co=*,os i'a ' s eia =~ c a .. noc ~ . s. s..

s.c.

o.. ... a ff BIBUOGRAPHIC DATA SHEET I4UREG/CR-5151 72,*,'s' c.,osi os ,. . .

su ,si,.

SAIC-88/3014 2 ,i,65 . solv 3,4,Li J Liaki OL.st Performance-Base nspections

.o....o.,y..i,it .

ops,= ..

.l.

i a vi-oa + . March 1988 F. Hawkins/tiRC, J. Jo son /We1 rich and Associates,

. 3.y. ,0. , , i i e R. T. Liner, C. H. Putt m/SAIC -asey

, l

.i.-

Jonn & 1988

, . . . c . . a c . 3. s , . , ,0., s . . i . 0 . . . e s c . n ,,m. i. c . . ,a an c e, .y e . es , s... -

Science Applications Interr ;ional Corporation ,

1-2p3}67, ,93,0-00 1710 Goodridge Drive McLean, VA22102 M8733 c.;

i,c ,

,c .o ,;. .s 2 0. s.s z . . .os s.. .s: .. ...sa . o0. os , ~ .i,c jz ". 4..io.

Division of Licensee Performance , d Quality Evaluatipb Technical i

Office of Nuclear Reactor Regulati f ,,,,,,,,;,,_,,,_,

U.S. Nuclear Regulatory Commission Washington D.C. 20555 /g March 1988-June 1988 12 5.Pr.l. s'.m.ei;,ii e Based upon NRC course, "Inspecting for rfoptiance,"aqualityassurancetrainingcourse.

3 . . i v . . c , ,ix . ~. .. . ,,;

i This report describes the concept of p .ance-based inspections that is being taught in NRC's training cour,prfc, se, " specting for Performance." This concept has been endorsed and is Leitig imple nted by the Nuclear Regulatory Commission (NRC). NPC performance tions concentrate on activities thatimpactplantreliabilityang/,rhasedinspt 5ffety. The 'nspections begin with performance-based observations,/ 6Jd then the in ectors let discrepenties or uncertainties lead to inspectio /of other areas, uch as quality verification organization effectiveness, t@ ning adequacy, an procedural controls. This

.nspection approach departs /pom past NRC practice that emphasized docurrentation and program e9iew as a treans to meas re operational safety.

The first goal of perform Ee-based inspections is t improve the NPC inspector's ability to ac'durately evaluate plant saft y and reliability. This goal will be achieved b[' increasing the inspector's e basis on actual observation of ongoing $ facility work activities and re cing the erphasis on docurrent and program reviews. The secondary goal of pe ormance-based level inspections is to e. burage licensees to manage their f a "lities in a performance-based M nner.

.. oo:v iss .s. .i i . . o '. ;3:o: .,o., , , . . . . . . .

quality assur ce inspections 1 Unlimited

, si:.. . : . . .s . c . . :s

. r . ,.

• c 5 '"' "5 o' Unc1assified

/' ':"**5 s

,r.....,

, s Unclassified

.S t

• C. 8 0 1 6 ** C E

UNITED STATES '

NUCLEAR REGULATORY COMMISSION '"I'ONOcElttUSS"'

WASHINGTON, D.C. 20555 *

PER%T No G 67 OFFICIAL BUSINESS PENALTY FOR PRIVATE USE, $300 120555078877 1 1AN19R US NPC-0 ARM-ADM DIV 0F PUB SVCS r' O L I C Y 9 Pub MGT HR-PDR NUREG W-53/

WASHINGTON DC 20555 t

i