ML20128Q820

Insp Rept 99901037/96-01 on 960715-18.Nonconformance Noted. Major Areas Inspected:Implementation of Selected Portions of Licensee Quality Program

ML20128Q820
Person / Time
Issue date:	10/16/1996
From:	NRC (Affiliation Not Assigned)
To:
Shared Package
ML20128Q798	List: ML20128Q795 ML20128Q812 ML20128Q820
References
REF-QA-99901037 99901037-96-01, 99901037-96-1, NUDOCS 9610210182
Download: ML20128Q820 (14)

Text

. . _ . ._ _ _ . _ __ _ _ _ _ __ _ _ . _ _ . _ _ _ _ _

I U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION l

Report No: 99901037/96-01 Organization: Zetec, Incorporated Issaquah, Washington l

l l

Contact:

Stephen H. von Fuchs, Quality Assurance Manager (206) 392-5316 L

l Nuclear Industry Manufacturer and supplier of steam generator tubing, Activity: eddy current inspection instruments, probes, software, calibration standard.t. manipulators, training and services for the comr ecial nuclear power industry.

Dates: July 15-18,1996 l

l l

l Inspectors: Joseph J. Petrosino, Team Leader John M. Gallagher, Electrical Engineer John K. Ganiere, Electrical Engineer Michael J. Morga.i, Reactor Engineer Phillip Rush, Materials Engineer Approved by: Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs i

Enclosure 2 9610210182 961016 ~

PDR GA999 EECZETEC 99901037 PDR

1 INSPECTION

SUMMARY

I During this inspection, the NRC inspectors (team) reviewed the implementation i

of selected portions of the Zetec, incorporated (Zetec) quality program, which l

was expressed in the Zetec Quality Program Manual (Z-QA, Revision 14, January 1995), and specified controls to encompass the requirements contained in the criteria of 10 CFR Part 50, Appendix B, American National Standards Institute-American Society of Quality Cor, trol (ANSI /A5QC) Q9001 and International Organization for Standardization (IS0) Standard-9001. Through a review of NRC licensee's purchase orders (P0s) and discussions with Zetec staff, the team found that NRC licensees impose Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) and other unique nuclear requirements on Zetec for its nondestructive examination services (NCE), NDE analyst activities, EC test calibration standards, and other related

components.

Conversely, Zetec's Eddynete' computer software and associated testing probes i

were found classified by Zetec and procured by NRC licensees and NDE testing customers as commercial grade items (CGI's), as defined in G21.3 of 10 CFR Part 21. Even though Zetec considers its Eddynet software and associated l probes as CGI's, Zetec's quality manual policy stated that its manual defines

! corporate quality policies and requirements and applied to all Zetec products

! and services. The team found that although Zetec classified its Eddynet computer software program, probe design, manufacture, and supply as CGI's, it used some elements of its Z-QA quality program for control of these components.

Therefore, in addition to the below inspection bases, the team also considered industry acceptable guidance when it evaluated Zetec's program controls such as, the Institute of Electrical and Electronics Engineers, Incorporated (IEEE)

Standard 7-4.3.2-1993, "IEEE Standard Criteria for Digital Computers in Safety l Systems of Nuclear Power Generating Stations," as endorsed by NRC Regulatory i

Guide 1.152, " Criteria for Digital Computers in Safety Systems of Nuclear Power Plants," and Electric Power Research Institute (EPRI) NP-6201, "PWR I Steam Generator Examination Guidelines." The inspection bases were:

• Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50)

. 10 CFR Part 21, " Reporting of Defects and Noncompliance" l

Overall, the NRC inspection team found that the measures for your Eddynet i software development and control were generally sound, with the exception-of a I

few weaknesses that are discussed herein. The team observed strengths in Zetec's program, such as employee involvement, employee communication, and solicitation and disposition of potential issues from your software customers.

' eddyn.t i. r.gi.ter.d tr.d rk of z.t.c. inc.

(

The team also observed that your employees demonstrated an overall sense of ownership and pride in their jobs. The team also believed that the strengths noted contributed to the soundness of Zetec's program and the adequacy of the eddy current (EC) software.

During this inspection, one instance where Zetec, Incorporated failed to conform to NRC requirements imposed upon them by NRC licensees was identified.

This nonconformance is discussed in Section 3.7 of this report.

2 STATUS OF PREVIOUS INSPECTION FINDINGS

-2.1 Violation 99901037/85-01-01 (CLOSED)  :

NRC Inspection Report 99901037/85-01 stated that, contrary to Section 21.6 of 10 CFR Part 21, copies of Section 206 of the Energy Reorganization Act and Procedure ZAG-16 were not posted at the Zetec facility.

The team looked at Zetec's current posting and reviewed its content regarding the regulations. The team noted that the posted documents addressed the re" irements of 10 CFR Part 21.

2.E inlation 99901037/85-01-02 (CLOSED)

NRC :pection Report 99901037/85-01 stated that, contrary to Section 21.21 of 10 Ctx Part 21, Zetec did not communicate the requirements of 10 CFR Part 21 to Westinghouse Specialty Metals Division, which supplied Inconel tubing to Zetec in August 1982.

This issue concerned a single piece of Inconel tubing which had been fabricated into a calibration standard by Zetec for one licensee's steam generator tubing EC inspection. This matter was adequately resolved and corrective action was taken in 1985 when the finding was identified.

l 2.3 Nonconformances 99901037/85-01-03 throuah 85-01-13 (CLOSED)

Nonconformances 85-01-03 through 85-01-13 are considered -losed because adequate corrective action was taken and the applicable licensees were made 1 aware'of the issues when the inspection report w u issued. Additionally, in a '

letter dated June 9, 1986, regarding Inspection Report 99901037/85-01, the NRC l staff stated that it had reviewed Zetec's replies and found them responsive to the concerns raised in the 1985 notice of nonconformance.

l l 3 INSPECTION FIlWINGS AND OTHER CONNENTS 3.1 10 CFR Part 21 Proaram The team reviewed Procedure ZAG-16, " Reporting of Safety Hazards," which Zetec adopted pursuant to 10 CFR Part 21. The team noted that although the procedure generally met the intent of Part 21, it lacked some clarity and

specificity in certain areas. For example, the procedure was not very clear

about the vendor's responsibilities under 521.21(b) of Part 21 and it did not i define some key words, such as evaluation and discovery. The team discussed 3

l

potential changes to ZAG-16 with the QA Manager, which would appropriately clarify Zetec's procedure. The QA Manager committed to reviewing end ,

modifying the procedure as necessary within approximately 90 days from the i receipt of this report. No other concerns were noted in this area.

3.2 Ouality Assurance Proaram The inspection team selectively reviewed Zetec's Quality Manual Z-QA, Revision 14, dated January 1995, and found that it addressed 10 CFR Part 50, j Appendix B, ISO 9001, and ANSI /ASQC Q9001. Z-QA was divided into separate '

l sections which correlated to the different requirements and regulations. The l team noted that document Z-QA stated that: "The scope of the quality assurance program plan includes the total operation of Zetec, Incorporated. The ,

measures are to be implemented to the extent necessary to assure that '

services, equipment and other items supplied to or performed for customers by Zetec shall conform to the specified quality levels of applicable codes, standards, regulatory criteria and purchase order specifications...."

Further, the team noted that the cover page of Zetec's Quality Manual stated, ,

"This manual defines Corporate Quality Policies and Requirements and applies i to all Zetec products and services."

The team reviewed a sample of the licensee's purchase orders (P0s) and verified that Appendix B of 10 CFR Part 50 and other unique nuclear requirements had been imposed on Zetec for its nondestructive examination (NDE) services, EC test calibration standards, and related components. The team also found that Zetec's probe products and Eddynet software were typically procured by NRC licensees and NDE testing customers as commercially available products; consequently, Zetec's quality program would encompass Zetec's control of Eddynet and associated probes under its ISO 9001 aspects.

From its review of Quality Manual Z-QA and observations of Zetec's operation, the team concluded that the implementation of Zetec's program and establishment of its manual appeared to be adequate.

3.3 Eddynet* Software Develooment. Validation and Confiauration Control The team evaluated Zetec's software development validation and configuration l control process to determine whether Zetec was achieving and maintaining a consistent level of quality in its Eddynet software products. The Eddynet software is used in the NDE eddy current inspection of safety-related steam generator tubing and plays a critical role in the generation of the eddy current inspection data.

Zetec has a single manager, the Software Development Supervisor, in charge of the entire software development program. This person is responsible for all aspects of the development and configuration control of the Eddynet software.

The Software Development Group consists of six programmers. Three of the programmers have had extensive field experience in performing EC acquisition and analysis, and four of the six are considered to be experienced computer language C programmers. An associateo group, the Software Support Group, also reports to the Software Development Supervisor and is responsible for developing the software validation checklists.

4

i i

i The Eddynet 95 software product is written in computer language C for 4

i application in either of two different Hewlett Packard workstations: Series 300/400 (HP-UX Version 9.03 operating system); or Series 700 (HP-UX Version 9.05 operating system). Eddynet 95 comprises several independent programs,

such as acquisition programs,' analysis programs, and inspection management i programs. The team noted that the Eddynet series and associated firmware are offered as commercial products. Although Zetec is registered as an ISO 9001 a supplier, the team was told by Zetec staff that its software development does i not conform to the guidelines of ISO 9000-3, " Guidelines for the Application j of ISO 9001 For the Development, Supply and Maintenance of Software."

j 3.3.1 Software Requirement Specification The Software Development Supervisor told the team that Zetec had not

! established any formal software requirement specification for the performance i expectations of the Eddynet software. The original Eddynet software was

written in an effort to develop a digital acquisition system and consisted of

! relatively straightforward algorithms. As a result, the original development i process did not follow a formal software development life-cycle approach which

would encompass a software requirement specification. From its discussion t

with the Zetec software staff, the team understood that since it first

, developed its.Eddynet software, Zetec has continued to enhance Eddynet j software with strong emphasis on the man-machine interface characteristics of j- its software program. As a result, Zetec had focused much of its attention on j ensuring that the software product is validated before release.

The team discussed with other NRC staff its concern that no documented software requirement specification existed. Upon further consideration of

Zetec's utilization of its available resources for the development and i validation of the software, the team concluded that the lack of a detailed i specification was more a question of development efficiency than quality.

The team observed that the majority of software changes which had been

incorporated into new versions of Eddynet 95 were related to man-machine interface enhancements. These changes were derived from Zetec's Software

Change Request (SCR) Program which is described in Section 3.5 of this report.

j No other concerns were noted in this area.

l 3.3.2 Software Verification and Validation i The team reviewed Zetec's software verification and validation process. The

! team reviewed the methodology that Zetec had established to validate the

. Eddynet software program after Zetec made changes to the program. The i following three Zetec work instructions were reviewed to evaluate their 3

adequacy and appropriateness:

1

= New Version Product Disk Validation

= Version Update Disk Validation i

• Prototype Disk Validation i

J l

! 5 i

.i I-

t The team concluded that the major purpose of these work instructions was to ensure that new version releases, version update releases, and prototype releases of the Eddynet software product performed correctly. Additionally, the team found that these procedures. appeared to adequately control the applicable portions of the Zetec program for which they were written and were appropriate to the circumstances.

l The team found that the Software Support Group had developed a software checkout book that contained Eddynet validation checklists used by the Zetec i group respcnsible for the validation process. The software checkout book consists of multiple checkout groups and subgroups which contain specific validation checklists for each of the programs within Eddynet. The Software Support Group develops and maintains these validation checklists. A version l checkout coordinator from the Software Support Group was designated as being responsible for the overall coordination of the validation process.

l The l l validation is performed by employees within Zetec's Product Support Group and, )

l to a limited extent, by the Software Support Group. Zetec indicated that its validator personnel had extensive field experience with Eddynet.

The team concluded that there was sufficient independence between the software developers, the Software Support Group who develop the validation checklists, and the Zetec group that performs the validation process.

The software group stated that the intent of the validation process was to exercise all possible program functions and, therefore, verify the functionality of each of the programs in the software. The team found that i functionality problems encountered by Zetec during its validation process were l l recorded on version validation sheets. The problems were then compiled by the version checkout coordinator and were reviewed and handled by the Software Development Supervisor. The team observed that after the appropriate changes had been made, Zetec subjected its software to a second complete validation I process checkout. That is, Zetec repeats all of the actions that were done for the first validation process. l On the basis of conversations with the Software Support Group, the team concluded that the checkout process is continually revised to account for new features added to the Eddynet program, and the subgroup checklists are updated l to ensure that all new features of the software are exercised. In addition, based on procedural instructions and management instructions, it is the understanding of the individual validators that they are responsible to exercise all functions within the program, even if each function is not specified in the subgroup checklist.

Although the team found that the current validators are adequately performing their job functions, Zetec had not established documented controls or release signatu're requirements for ensuring that the validation checkout subgroup lists are fully updated to include functional tests for all new features or changes associated with a new version or version update before sending the checkout subgroups to the validation team. As a result, there is a potential that new features of the software may not be functionally tested before they are delivered to customers for safety-related applications. The inspectors classified this as a weakness in the validation process.

6

l The team noted that, in addition to the validation of the functional performance of the program, the validation process also served as feedback from the validators about the acceptability of new features. The program descriptions associated with software changes are not considered complete until the validation has been finished. This allows the validation team to evaluate and assess the man-machine interface characteristics and functionality associated with each software change. Overall, Zetec's method of performing validation appeared to be effective because of the added strength which comes from using personnel who have Eddynet field experience.

l The experience aspect also compensates somewhat for Zetec's lack of a documented software requirement specification, as discussed in Section 3.3.1 of this report. No other concerns were noted in this area.

3.3.3 Review of Eddynet* 95, Version 2.0, Validation The team reviewed the checkout group and subgroups completed for the validation of Eddynet 95, Version 2.0, HP-UX Version 9.03 (300/400 Series).

The team observed that all items within the subgroups appeared to be checked.

The team also noted that the employees performing the validation often changed terminology in the subgroup checklists and added comments within the margins of the checklists. In additior, checklist instructions were modified if the information in the. instruction was different from the information regarding the program functionality. For instance, the team found one checklist instruction that was changed because it incorrectly referred to a " button" rather than to a " pull-down menu." In most cases, these changes or comments were not recorded on the validation sheets since they did not affect the functionality of the scftware.

Further review of this area revealed that Zetec had not established any oocumented controls for ensuring that the written comments or changes made in the validation checklists are reviewed and handled by either a version checkout coordinator or by the Software Development Supervisor. Therefore, during discussions with Zetec staff, the team also identified this as a weakness because of the potentially inconsistent results that could occur during the validation process.

The team also reviewed the master version validation sheet'and verified that all problems encountered during the validation were reviewed and handled by the Software Development Supervisor. In general, the team concluded that the validation of Eddynet 95, Version 2.0, was performed in accordance with the work instruction "New Version Product Disk Validation," and appeared to be effectively controlled. No other concerns were identified in this area.

3.3.4 Review of Eddynet* 95, Version 3.0, Validation The team witnes' sed Zetec's validation activities associated with the inspection planning program of Eddynet 95, Version 3.0. A data manager / data analyst in the Product Support Group was performing the validation in accordance with the appropriate subgroup thecklist. The data analyst confirmed to the team that the intent of tne validation was to exercise all program functions to er..ture that the software performs as designed. To achieve this, validation personnel executed the steps written in the subgroup l

i checklist. The team found through its discussion and review that the ,

validators generally exercise more functions than are prescribed in the I checkout list in an effort to verify the functionality of all aspects of the  !

program and to check for the potential impact of incorrect key manipulations. l 1

The data analyst told the team that all new functions are not always incorporated into the checkout list they receive. As a result, the analysts may add supplemental checks to test new functions at their discretion. An analyst who encounters a new function that is not on the checklist may contact the Software Support Group for an explanation of the new function. As i described in Section 3.3.2 of this report, the team concluded that Zetec  ;

l appeared to lack adequate controls to ensure that all new features of the i software were incorporated into the validation subgroup checklist; however, it was viewed as a strength that the analysts may add supplemental checks to test new functions.

i The team determined that new versions of the Eddynet 95 software are validated  !

on both Series 300/400 and Series 700 Hewlett Packard workstations. In l witnessing the validation of Version 3.0, the team noted that the validation on the workstations was performed essentially in parallel by separate validators. The team believed that this parallel validation process is a i valuable asset to Zetec's process control and allowed for crosschecks and l real-time on-line discussions between validators regarding problems associated l

with the software or with the validation subgroup checklists. This could also be considered as a diverse implementation of the software programs and, as l such, an additional means for finding software errors.

3.3.5 Impact of Source Code Change The data analyst and Software Development Group indicated that changes to the source code that do not affect the high-level functionality of the program or cause a visible impact on the operator's workstation are transparent to the validator. As such, the team concluded that the independent validation process is not structured to assure that software errors which are transparent i to the user are detected. Specifically, there is no formalized independent I check of the source code that allows for detection of these types of l programming errors.

, This practice is not in accordance with acceptable nuclear and software development community-accepted quality assurance (QA) procedural guidance and i l practices that govern the development and validation of software, and would be l a considered as a noncompliance if Appendix B to 10 CFR Part 50 were applicable. The inspectors concluded that Zetec lacked criteria to allow for

, the determination of source code changes that require an independent check,  ;

and a process for performing the independent check cf the source code. '

3.3.6 Eddynet* Software Configuration Control The team reviewed Zetec's processes for controllino and reporting changes to software, including a review of procedures for revising Eddynet software programs. Zetec indicated that software changes were controlled by a software change notice (SCN) system. Each software change was documented in an SCN 8

I i

i form that described the changes, the proposed activities, and the personnel responsibilities for changing the software. The team reviewed QA Procedure

-QAP-8, " Control of Production Software," devision 11. This procedure controlled the revision, validation, and release of production software such as Eddynet 95, and gave specific details about SCN processing.

The team also reviewed Zetec's SCN log and associated SCNs. The team concluded that the software revisions appeared to be processed in accordance with QAP-8. Each SCN was uniquely identified and could be traced to a j specific Eddynet version. The team also reviewed a sample population of i j software revisions and noted they were validated before they were released.  !

l Zetec staff stated to the team that the software changes associated with a new version of Eddynet are typically described in a Zetec " announcement bulletin" that is sent to customers when new versions are released. Tne team reviewed l l the release announcement for Eddynet 95, Version 2.0, and confirmed that the  !

information in it pertained to significant software changes or enhancements.

The team concluded that Zetec maintains a good program for tracking and i controlling software changes and for notifying customers of the changes and enhancements associated with new versions of Eddynet.

3.3.7 Development, Validation, and Configuration Control As described above, the team reviewed Zetec's processes for software development, software validation, and software configuration control. The team concluded'overall that Zetec has an effectively implemented and l controlled software development and software validation program; however, some l

weaknesses were apparent. The team discussed the following two observations in detail with Zetec's staff:

• The team noted that high-quality man-machine interface characteristics-with respect to effective use of software products in field environment is a principal attribute of the Eddynet software. The development and validation processes support this attribute because they are performed by Software Development and Product Support personnel who have extensive field experience with Eddynet. However, the team also noted that field experience requirements for programmers and validation personnel are not controlled by QA procedures.

• The team concluded that there are na formal software requirement specifications for software changes. Instead, there is a heavy reliance on real-time on-line discussions in the performance of both software development and validation job tasks to maintain a high level of agreement during the development and validation activities. The team believed that this can result in undocumented bases for important decisions regarding the final software product.

3.4 Identifyina and Reoortina Software Deficiencies i

The team asked Zetec about how it addressed program errors in software that could potentially affect the quality of EC data being manipulated or analyzed.

Zetec indicated that significant software deficiencies attract immediate l 9 I

l

_ .__ _ _ _ _ . ~ . _ _ _____. _ .._. _ _ _ _ _ _ _ _ - . _

1 attention as they are processed in accordance with QA Procedure QAP-9,-

" Software Change Request," Revision 0,-and are reported to customers. Zetec provided three examples of how it notified customers about software problems or limitations. The team concluded that Zetec appeared to have addressed the

! problems promptly and efficiently. Although 10 CFR Part 21 is not applicable l to Zetec's Eddynet software, the Zetec process appeared to meet the intent of l 621.21(b) of 10 CFR Part 21.

3.5 Software Chanae Reauests The team reviewed QA Procedure QAP-9, " Software Change Request." This procedure defines the process for documenting, reviewing, and implementing SCRs. Zetec developed the SCR form for the use of customers and Zetec field persor.nel to report problems associated with Eddynet or to make i recommendations for software enhancements.

l The Software Development Supervisor reviews all SCRs and.is responsible for  !

deciding if an SCR will be approved (that is, if action will be taken to 1 address the customer's request) and for deciding the schedule for implementing 4 the change or enhancement. The Software Development Supervisor noted that l this decision may require speaking with the originator of the request to get  !

additional information or to modify the request, (to reduce the impact on l other parts of the program).

Upon approving an SCR, the Software Development Supervisor assigns the task to a programmer. The Software Development Supervisor discusses the details of 1 the change with the assigned programmer and, to implement the software change, '

relies on the programmer's experience and judgment. Preceding the validation phase, the Software Development Supervisor reviews the new or modified code developed by.the programmer to ensure that it meets its intended function.

t l -The team noted that the majority of the programmers and validators appear to have strong field experience with Eddynet. Although there is no formal ,

software requirements specification for software changes, formal discussions and reviews take place throughout the change process to ensure that software i changes are handled appropriately. In addition, the software changes are validated on two separate operating systems.

The team reviewed Zetec's process for handling SCRs by tracking several SCRs that had been initiated by Framatome Technologies, Inc. Specifically, the team reviewed Zetec's SCR log to ensure that the SCRs were reviewed upon receipt, assigned an SCR number, and appropriately dispositioned. The team also confinned that SCNs were implemented as appropriate and that the corresponding software changes were incorporated into new versions of Eddynet.

In general, the team concluded that Zetec reviewed and addressed,the SCRs promptly and in accordance with QAP-9.

Overall, the team concluded that the SCR process appears to be effective in identifying man-machine interface problems and customer requests for software

enhancements. Zetec receives a large number of SCRs. The team confirmed that

most of these SCRs request software enhancements rather than report software ,

i j 10 1

1 l

l l deficiencies Despite the large numbers of SCRs it receives, Zetec appears to l have a good program for tracking and addressing them promptly and-appropriately.

3.6 Data Analysis Procedures and Acouisition Techniaues The team reviewed the vendor's EC data analysis, " Eddy Current Data Analysis Procedure," FSP 301-EVAL, Revision 7, July 7, 1994, and an acquisition procedure, "Eddynet Eddy Current Acquisition Procedure," FSP-301-EN, Revision 3, October 20, 1994. The team evaluated Zetec's analysis procedures and site specific procedures to determine to what extent industry data analysis practices discussed in such industry documents as EPRI NP-6201 had been incorporated into Zetec's program. The team found that Procedure FSP 301-EVAL addressed general personnel, documentation, equipment, and procedural requirements for the analysis of EC inspection data. Procedure FSP 301-EN, Revision 3, addressed required actions needed to set up some of the hardware and software for acquiring EC inspection data.

The team found that t .e guidelines were clearly written. In addition, the i analysis guidelines contained such strengths as specific requirements on lead l analyst responsibilities, analyst independence, and noise and rejectable data criteria. By interviewing NDE technicians, the team confirmed that Zetec analysts were familiar with the details of the procedure.

l The team found that site-specific guidelines for examining steam generator l tubing are typically developed and revised through discussions between the utility and the NDE vendors involved in the steam generator tube inspections.

The team noted that a typical site-specific examination guideline did not contain all of the attributes found in FSP 301-EVAL. Zetec indicated that the NDE technicians generally focused on revising the finer details of the proposed site-specific guidelines rather than on the broader aspects of the procedure.

l The team concluded that Zetec's EC analysis and acquisition procedures l l contained sufficient detailed instructions for conducting inspection-related l l activities. The team noted several strengths in the analysis guidelines; 1 l however, these strengths were not always being considered by NDE technicians j l

for incorporation into site-specific examination guidelines. '

3.7 Comouter-Assisted Data Analysis The team reviewed the vendor's controls for computer data screening (CDS) techniques used in the analysis of EC inspection data. The review focused on assessing whether Zetec had been adequately managing its activities involving the use of CDS. , The team found that Zetec is often contracted to supply EC data analysis using a CDS algorithm. These computer algorithms use specific criteria which are inputted by a Zetec data analyst before a particular utility customer S/G tube EC inspection to identify expected EC indications.

The individual responsibility is significant for the data analyst because they ,

must develop these criteria based on expected modes of tube degradation for  !

each different utility S/G tube EC examination. However, the team noted that Zetec did not have a written or formal type of procedure for controlling or

11 I

r * *

1 4 i l

establishing the specific process that is to be used by its data analysts for developing these criteria. As a result, the potential exists that screening

! criteria developed by one analyst may be significantly different from that of j another analyst. To establish the paramaters for identifying indications in

the EC data, a data analyst must independently develop the screening logic.

Before a Zetec analyst starts EC inspection at a particular nuclear power plant site, the CDS is typically subjected to a site-specific examination,

! which it must pass in order to be used in an inspection.

The team's review of this activity identified that when developing CDS criteria, analysts wili typically address known degradation modes. However,

because of the different experience levels of the analysts, a variance in the

effectiveness of the developed criteria can occur. This can allow types of degradation previously undetected by a particular analyst to be missed by the i CDS algorithm. The team identified to the Zetec staff types of parameters, or j ranges, that may need to be delineated within procedures or instructions to 3 ensure compliance with the requirements, and to ensure that the widest ,

spectrum of degradation modes will be identified. These aspects were not .

required by Zetec's program. '

j The team also recognized that Zetec's data analysts have a broad knowledge of degradation mechanisms and may adapt well to any new degradation mechanisms i

that appear during inspections. The team noted that although Zetec analysts I

have not been working with written procedures or instructions regarding

, written guidance and instruction for the development of these criteria, Zetec s was using only experienced analysts for developing its CDS criteria.

l The team determined that the potential existed that the lack of such

procedures could lead to differing performance capabilitius for CDS techniques under actual inspection conditions and as a result, Zetec's CDS program could fail to identify potentially defective tubes. Therefore, although Section j 2.5, " Procedures," of Zetec's Z-QA manual indicated that every procedure and l all revisions are reviewed and approved by quality assurance to ensure that i

standards of acceptability for all requirements and features are-clear, no l requirements for CDS criteria development were found. The +eam identified the lack of written guidance to govern the development of CDS criteria as Nonconformance 99901037/96-01-01.

3.8 Eddy Current Probe Fabrication The team observed the EC probe manufacture and interviewed assembly-line personnel involved with the fabrication of the EC inspection probes. The team reviewed several of Zetec's procedures used to manufacture and inspect the EC probes. For example, Zetec's " Fabrication Procedure for Motorized Rotating 3 Coil Probe Head," Revision 0, dated March 13, 1996, specified the sequence for i constructing a certain type of EC probe, and the procedure clearly defined the j steps and essential variables of the fabrication process and required periodic 1 simple checks to verify coil / probe performance.

The team also observed that the assembiy-line personnel involved in coil construction were knowledgeable about the use of the various testers used to verify the performance of r. oils. The team reviewed an acceptance-related 12

l l

procedure, QAP-21, "ULC Check-Out Procedure," Revision 2, and noted that the procedure contained steps for visual inspections (including physical measurements of probe dimensions), a physical inspection of the overall integrity of the probe, and a functional inspection requirement to verify the electrical response of the probe. The team observed that Zetec's inspection personnel used consistent reference standards for verifying probe performance i and appeared knowledgeable in their areas of responsibility. l l

\

3.9 EC Personnel Trainina. Qualifications. and Resoonsibilities l The team reviewed Zetec's Procedure QAP-101, " Personnel Qualification &

Certification," Revision-4, and discussed the responsibilities of Zetec's Field Services (FC) Group with Zetec's Supervisor of Product Support and Technical Development. The team assessed training documentation and reviewed certification examinations and discussed the specifics of Zetec's certification and qualification program with Zetec's Senior Training Manager. 1 The team noted that Zetec EC personnel are qualified and certified to Zetec's j written Procedure QAP-101 and that Procedure QAP-101 addressed the i recommendations contained in ASNT SNT-TC-1A, " Personnel Qualification-Certification in Nondestructive Testing," 1984. The team also reviewed various Level I, II, and III certification examinations and noted that the l

l examinations appeared to be thorough and well-designed, and adequately test knowledge of basic theory and EC application. The team noted that Zetec's i

, certification and qualification records appeared to generally conform to the l requirements of ASME Code Section XI (1989) and met the intent of l l

Criterion IX, of 10 CFR Part 50, Appendix B. l The team found that Zetec's analyst training program experience requirements may vary because they are dependent upon the currently available job assignment and the collective amount of specific site-testing and 0JT

available for the recommended nine-month hands-on period. Therefore, Zetec's l current practice could have the potential to limit job variety, exposure to various steam generator designs, and differences in testing practices. The team also noted that Zetec's program lacked specificity relative to l delineating types of experience that would be required for individual qualification.

l During the team's review of this area, it also noted that Zetec's supplemental Level I steam generator tube inspection training requirements, that is, training performed after Level I certification for on-site, on-the-job-training (0JT), were typically followed by routine assignment to Zetec's Field Services Group. Individual training records that were reviewed in this area showed that some candidates accumulated more than 1,000 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> of site-specific training, while other records only reflected 400-500 hours of similar training. Consequently, Zetec's program could allow a 500-600 hour disparity between different Level II NDE examiner training hours. The team noted, in the records which it reviewed, that an individual's particular level of involvement with a specific team and uniqu2 job preparation, such as in-house

! mock-up work, EC equipment setups, and simulated data acquisition run-through, r

was typically the primary contributor to an individual's specific hours used for their experience and qualification.

13

.- . - ~ - _ - - - .. - . - -. -- - .

Although a concern regarding analyst experience requirements was noted, the team concluded that Zetec's current commitment to the ASNT recommendations and its current program meets the intent of ASNT-SNT-TC-1A by providing a methodology that appears to appropriately qualify and certify its eddy current testing personnel.

l 3.10 Internal Audits Internal auditing takes place within Zetec's Z-QA quality program. Section

XVIII, " Audits" of Z-QA, addresses the requirements contained in 10 CFR Part 50 Appendix B and ISO 9001. This document calls for a formal in-house auditing of the EC function, safety-related/ASME Sections III and XI l

Code activities, on at least a "once per each calendar year" basis. The team I

verified that internal third-party assessments of Zetec's QA Group is performed by product services personnel at least once per calendar year.

! On the basis of its review of documentation and discussions with Zetec's QA

Manager, the team concluded that Zetec's internal QA audits were adequate and met both the intent and scope of requirements in Criterion XVIII, " Audits," of 10 CFR Part 50 Appendix B.

The team also reviewed Zetec's Product Service Group's internal audits of QA

! for 1994 and 1995. On the basis of its review of documentation and

! discussions with the Product Services Manager, the team concluded that Zetec's audits of QA were adequate and that these audits met both the intent and scope of Criterion XVIII of 10 CFR Part 50 Appendix B requirements.

3.12 Entrance and Exit Meetinas l

At the entrance meeting on July 15, 1996, the NRC team leader discussed the scope of the inspection, outlined the areas to be inspected, and established contacts to be used during this inspection with Zetec management. At the exit meeting on July 18, 1996, the team discussed its findings and concerns.

l l

PERSONNEL CONTACTED l I

Howard E. Houserman General Manager i Jimmy Crittenden Product Support l l Doug Handy Electronic Supervisor j Laura Mcdonald Quality Assurance Hike 0'Laughlin

~

Validation-Data Manager / Data Analyst Bob Vollmer Product Support Manager j Stephen H. von Fuchs Quality Assurance Manager l Richard Warlick Software Development Supervisor Kem Wilson Software Administrative Assistant

Troy Woller Software Support Specialist 14 I

i