ML20129J716
| ML20129J716 | |
| Person / Time | |
|---|---|
| Issue date: | 11/04/1996 |
| From: | NRC (Affiliation Not Assigned) |
| To: | |
| Shared Package | |
| ML20129J706 | List: |
| References | |
| REF-QA-99901303 99901303-96-01, 99901303-96-1, NUDOCS 9611070206 | |
| Download: ML20129J716 (12) | |
Text
_ _ _ _ - _ _ _ _ _ - _ _ _ _
U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No:
99901303/96-01 Organization:
Panalarm Business Unit Process and Analytical Instruments Division AMETEK, Inc.
7401 North Hamlin Avenue Skokie, Illinois 60076
Contact:
A.H. Demerer, Quality Assurance Manager (847) 509-7196 Nuclear Industry Annunciator systems including temperature i
Activity:
monitoring modules DATES:
September 9-12, 1996 Inspector:
Richard C. Wilson, Senior Reactor Engineer APPROVED BY:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspectio, Branch Division of Inspection and Support Programs t
9611070206 961104 PDR GA999 ENVAMET
__ ___ __ _____:L __ __.
1 INSPECTION
SUMMARY
During this inspection, the NRC inspector reviewed the implementation of selected portions of the AMETEK, Inc., Panalarm Division quality assurance (QA) program, and reviewed activities associated with the supply of safety-grade replacement parts for annunciator systems supplied by Panalarm for nuclear power plants in the 1970s and early 1980s.
In the early 1970s the Panalit company produced Panalarm annunciator systems.
The company name was changed to Riley, and subsequent owners were US Filter and Ashland Oil, prior to acquisition by AMETEK.
Panaiarm has approximately 100 employees.
Nuclear activity amounts to about 0.1% of sales.
The audit bases were:
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Raprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" During this inspection, a violation of NRC requirements was identified and is discussed in Section 3.1 of this report.
During this inspection, three instances where Panalarm failed to conform to NRC requirements imposed upon them by NRC licensees were identified.
These nonconformances are discussed in Sections 3.2 and 3.3 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS This was the first NRC inspection of Panalarm.
3 INSPECTION FINDINGS AND OTHER COMMENTS 3.1 10 CFR Part 21 Proaram The inspector reviewed Panalarm's procedure for reporting in accordance with 10 CFR Part 21: Quality Control procedure QC-90-100, "(10CFR21)
Notification Procedure," Revision 0, dated March 6, 1990.
The procedure focused on evaluating a defect to determine if it was a reportable condition; it stated that if the defect was determined to not be a reportable c4.:6it ion, then no further action would be taken.
Procedure QC-90-100 incorrectly confused the terms " deviation" (which is not used in the procedure) and " defect" (which is incorrectly used). As a result, Panalarm personnel were not alerted to the need to identify deviations from purchase order requirements, and to evaluate the deviations to determine if they could become defects.
Further, the evaluation of defects as used in the procedure is not required, nor would it serve any useful purpose.
The procedure also did not address significant changes to Part 21 in the years since the procedure was 2
written, regarding reporting requirements and content of procedures. As a result, the procedure failed to comply with 10 CFR Part 21.
Failure to have procedures that address the reporting and procedural content requirements of 10 CFR Part 21 constitutes Violation 99901303/96-01-01.
3.2 Ouality Assurance Proaram a.
Inspection Scone The inspector selectively reviewed the "AMETEX, Inc., Panalarm Division Quality Assurance Manual," Document 900181, Revision 10, dated December 8,1993, and the "AMETEK Panalarm Division Quality Control Procedures Manual," Document 900004, dated May 2, 1986, which described the Panalara quality assurance (QA) program. During the inspection the inspector asked about a quality control (QC) procedure that was not included in the Procedures Manual, and the QA manager determined that an obsolete copy of the manual had inadvertently been provided. The inspector then reviewed the table of contents for the January 28, 1994, revision of the QC Procedures Manual, which identified 30 additional QC procedures. Two of these, covering 10 CFR Part 21 reporting and commercial grade item dedication, had already been identified and reviewed during the inspection.
The inspector selected and reviewed three others.
b.
Observations and Findinas Section 1 of the QA manual stated that the program is based on the American National Standards Institute standard ANSI N45.2-1971, and Section 2 stated that products supplied on purchase orders requiring compliance with Appendix B to 10 CFR Part 50 will be processed using the procedures described and referenced in the QA manual.
Revision 0 of the manual was dated May 5, 1977.
Panalarm management advised the inspector that Panalarm was considering changing the QA program to ISO format and was also considering ending their Appendix 8 program. Neither decision had been made by the conclusion of the inspection.
The inspector had two concerns with Panalarm's QA program.
The first relates to QA verification of the final acceptance testing of Model 86 modules, which is the principal basis for considering them to be safety-grade, as described in Section 3.3 below.
In Section 12, " Test Control," of the QA manual, paragraph 12.6 stated that documented test results are approved by engineering and/or QA.
Engineering review of the test results is a means of ensuring quality.
However, because there was little independent QA revt.w of other activities affecting quality, such a review of the final acceptance testing is necessary to satisfy Criterion I of Appendix B to 10 CFR Part 50, " Organization," which requires the independent QA organization to verify that activities affecting safety-related functions have been correctly performed.
3
n l
The inspector also determined that Panalarm procedures did not reflect the current organization structure. The organization chart dated July 1996 that was provided to the inspector differed from the chart included in Section 3, " Organization", in QA Manual Revision 10 dated December 8, 1993.
The QA manual chart shows the QA function reporting to the operations manager, with no alternate path specified; the July 1996 chart shows the QA manager reporting to the production manager, but with a dashed line re cesenting an administrative link directly to the s
division vice president and manager. Several procedures also contained confusing references to QA and QC positions.
For example, QC Procedure QC-76-47, " Training of Test and Inspection Personnel," dated April 25, 1986, referred to the QC Supervisor and the QC department, but the QA-manager stated that reference to his position was intended.
This clarification is significant because Panalarm's production department has QC personnel who lack the organizationel independence required by Criterion I of Appendix B to 10 CFR Part bO.
c.
Conclusions Panalarm's failure to Thermocouple Monitor require QA verification of final acceptance testing constitutes Nonconformance No. 99901303/96 02.
FAnelarm's failure to provide clear procedural references to the l
independent QA function where required constitutes Nonconformance No. 99901303/96-01-03, 3.3 Model 86 Thermocouple Monitor a.
Inspection Scope Prior to the inspection, the QA manager identified the Temp-Matic (or TempMatic--Panalarm literature uses both terms) Model 86 thermocouple monitor as the only safety-related component currently supplied.
The inspector reviewed test and dedication procedures and reports, and other pertinent documentation, and observed a final acceptance test demonstration.
b.
Observations and Findinos The Model 86 analog module accepts a thermocouple input, compares it with an adjustable setpoint, and provides status light tand contact outputs.
Variations include a differential unit accepting two thermocouple irputs, a " half-gain" version, and a burnout protection feature. The module consists of two small printed circuit boards with several interconnecting wires, joined by standoffs and bolts to form an assembly about the size of a pack of cigarettes.
The module has no case, and is intended for insertion into an annunciator grid structure in the control room.
Licensees commonly refer to the module as a temperature switch.
The Model 86 is not part of current Panalarm annunciator systems.
It is sup;. lied in small quantities as a replacement part.
4
The inspector reviewed three 1973-4 qualification type test reports and verified that they identified the test specimens clearly enough to permit determining the similarity of current units. The inspector also reviewed subsequent engineering change notices that documented design reviews. These documents appeared to be adequate for tracing design changes of the Model 86 modules.
Panalarm completed a dedication procedure for the Model 86 temperature modules in 1992 that provided the following:
Document 900396, "Model 868 Temp-Matic EPRI Conformity,"
Revision 0, dated November 5, 1992 (currently in Revision 3 dated January 30,1995), identified each part in the Model 86 versions as critical or non-critical, and tabulated critical characteristics and their verification methods for the critical parts.
Quality Control Procedure QC-92-110, " Commercial Grade Item Dedication for Safety Related Assemblies - Quality Control Procedures for Dedicated Items," Revision 0, dated July 30, 1992, specified batch receipt inspection and controlled handling practices for critical parts; it also required training for personnel performing these activities, and for testing personnel.
j Notes referencing these two new documents were added to the two assembly level drawings for Model 86 modules, 86800-P-2 and 86800-V-1.
The NRC inspector found that the Model 86 parts dedication activities did not address possible heterogeneity of commercial grade parts lots, and no audits of commercial grade parts suppliers were performed.
However, the receipt inspection did verify visual and performance characteristics for many parts, which were then subject to controlled handling.
The principal basis for considering the completed Model 86 modules to be safety grade was the final acceptance testing, which was relied on for dedication of some parts as well as performance of the assembled module.
The acceptance test procedure written for GE contracts--Document No.
108-8022-900035, " Final and Acceptance Test Procedures for Model 86 Tempmatic Modules for General Electric [GE] Nuclear Energy Division Jobs," Revision 16, dated December 29, 1989, was more comprehensive than the previous procedure, Document No. 900024, " Calibration and Final Test Procedures for Temp-Matic Model 86," Revision 2, dated July 11, 1975.
The new procedure repeats more of the in-process calibration points; provides more detail for model variations; includes a check for input loop resistance; tests the optional burnout protection; and requires 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> post-test operation.
Although the new procedure's title would appear to restrict its use to P0s from GE, this is apparently not the case.
Dedication procedure QC-92-Il0 requires use of procedure 900396 for dedicating Model 86 switches. This document in turn requires testing in accordance with the revised procedure to verify certain critical 5
s characteristics at the module level. As discussed below, an IES Utilities audit report stated that they would continue to invoke QC-92-Il0 and 900396 in P0s.
Although no safety-related work was in progress during the inspection, Panalare demonstrated final and acceptance testing for a Model 86 module using Procedure 108-8022-900035.
The NRC inspector identified a few concerns with the procedure:
steps 6 and 7 of Part A did not distinguish between test values for standard :nd half-gain modules; Figure 2 specified an incorrect wire type for type J thermocouples; and the procedure did not specify that high and low range, but not setpoint, potentiometers are sealed with glyptol.
In each case the Panalarm inspector was aware of the correct procedure, and explained it.
The NRC inspector did not consider these concerns significant; Panalarm agreed to correct them.
The NRC inspector noted that the Panalarm inspector performing the final acceptance test reported to a production supervisor, and there was no oversight by the QA organization. This practice was not in conformance with Criterion I of Appendix B, " Organization", which requires the QA organization to verify that activities affecting safety-related functions have been correctly performed.
The Panalarm QA manager agreed to immediately notify affected personnel that either the QA manager or the QC technician reporting to him must witness the final acceptance testing of safety-related equipment.
The schedule for a written procedural change depends on Panalarm's decision concerning continued supply of safety-related equipment.
The inspector found no purchase orders for non-GE procurement of Model 86 modules.
This was partially because the P0s initially identified by the QA manager did not constitute a complete set of recent safety-related orders.
Subsequently, an additional P0 from a licensee was traced through a file cabinet containing certificates of conformance to a separate set of P0 files, but time did not permit pursuit of possible additional safety-related P0s through this path.
The additional P0 was for the Entergy Operations procurement of isolators described below.
Since the Model 86 dedication program was performed in response to GE requirements, and the acceptance test procedure for GE orders was more extensive than the older Panalarm procedure, it would be necessary to examine specific licensee procurement requirements to determine whether
~
direct licensee procurements of Model 86 modules would be acceptable.
As discussed below, an audit report by IES Utilities stated that their safety-related P0s would continue to invoke Panalarm procedures that require the use of the GE acceptance test procedure, but no related 1
procurement records were available to the NRC inspector within the available time limits.
c.
Conclusions 6
The inspector concluded that Panalarm's QA program for Model 86 modules supplied to GE was acceptable.
The acceptability of direct licensee procurements would depend on the specific requirements of P0s.
Panalarm's failure to perform independent verification of final acceptance testing constitutes Nonconformance No. 99901303/96-01-04.
3.4 Purchase Orders a.
SCoDe The inspector reviewed three P0s from GE and two from Entergy Operations, Inc.
The P0s covered a variety of equipment and services, as described below.
b.
Observations and Findinoi SE: GE Nuclear Energy P0 No. 52896020591 dated April 19, 1996, ordered three Model 86-BVTFF-EGCUST temperature switches.
The PO stated that the switches were safety-related, invoked 10 CFR Part 21, and required the items to be either purchased safety-grade, dedicated, or evaluated as performing no safety-related function.
The switches were delivered to GE San Jose, California, for use at the Grand Gulf power station.
Panalarm provided acceptance test data sheets which identified the test instruments, and certification of conformance to contract requirements which identified customer drawing no. 164C5687P302, Revision 16.
The data sheets were signed as attested to by the QA manager two days after the production inspector's signoff.
Except for this latter instance of Nonconformance 99901303/96-01-04, the NRC inspector found the documentation for the P0 to be acceptable.
GE Nuclear Energy P0 No. 205-95L289 dated March 17, 1995, ordered two temperature switches, GE drawing no. 164C5687P320.
The switches were delivered to GE San Jose for use at the Nine Mile Point unit 2 (NMP2) power plant.
In this case the data sheets were attested to by the former quality control manager on April 27, 1995; there is no evidence that he actually witnessed the testing, although that was the same day 4
as the production inspector's signoff.
The inspector had no other concerns with the documentation for this P0.
The inspector also briefly reviewed GE P0 No. 205-95L600 dated July 13, 1995. This P0 called for a failure analysis of a temperature switch from NMP2, and required scrapping the switch after submittal of the report.
The P0 stated that, when the switch was placed in service on January 7,1991, a plant operator touched the switch and it spuriously al armed.
The P0 identified the switch as safety-related, and invoked 10 CFR Part 21.
Panalarm provided a certification of meeting the P0 requirements. A factory service repair report stated that the unit tested satisfactorily, and test points were verified; spurious alarms were noted only at the edges of the deadband, which is typical of all units. An acceptance test data sheet was also provided, with attestation by the QA manager three days after the production inspector's signoff.
The NRC inspector noted that Panalarm's GE test 7
procedure specified that all setpoints should be set for approximately mid-scale trip before shipping.
The inspector was aware of three LERs, but no 10 CFR Part 21 reports, submitted by NMP2 in recent years involving Panalarm temperature switch modules.
LER 92-001 dated February 3, 1992, addressed a temperature switch that generated a spurious high temperature alarm; it stated that from April 1987 to February 1992, NMP2 had 76 work requests for the repair / replacement of Panalarm temperature switches, with drift in and out of the alarm state as a predominant failure mode. The GE part number for the LER switch was 164C5687P001, whereas the part number for the switch returned to Panalarm for evaluation was 164C5687P301; the LER did not give a serial number, but evidently the switch addressed in the LER was not the one returned to Panalarm. Another NMP2 LER,92-024, reported a temperature switch with a trip relay coil that failed open, causing a downscale trip output.
Both of these LERs indicated plans for replacement of all Panalarm temperature switches with another manufacturer's products.
NMP2 LER 94-04 dated January 17, 1994, referred to replacement of at least 65 Riley temperature switches, but the LER did not identify the replacement equipment.
Enterov:
Entergy Operations, Inc. P0 No, MP519933 dated July 19, 1995, covered two GG81147001 optical isolators for the Grand Gulf nuclear station.
The P0 stated that the parts shall be qualified replacements for original equipment qualified to IEEE 323-74 and 344-75 by Wyle test report 4407411, and referenced the original Riley job no. 607-7002Q; it also required electrical tests "as accepta( 1 by Riley Co."
The P0 stated that the equipment performs a safet3-related function, invoked 10 CFR Part 21, and required the seller to comply with his QA program as approved by the purchaser.
The NRC inspector observed an Entergy letter to Panalarm dated February 23, 1994, which stated that based upon review of Revision 10 of the Panalarm QA manual dated December 8,1993, the Panalarm QA program was considered acceptable to Entergy.
The letter also called for Entergy review of any program revisions prior to implementation.
The Panalarm QA manager had no evidence of an Entergy audit, although he was not required to have any.
Panalarm provided a certificate of compliance with P0 requirements. A test report for each isolator showed input and output voltage levels for ten channels measured per procedure QC-77-53 dated Noverr 'r 23, 1981.
Entergy P0 No. WP061473 dated November 23, 1994, covered c0 type LP92268005 printed circuit boards for the Waterford 3 plant.
The P0 stated that the boards were for use on an annunciator panel originally furnished under Ebasco/Riley P0 #NY-403540 dated January 30, 1976, through Supplement #8 dated July 26, 1979.
(The NRC inspector found that a 1981 Ebasco audit of Riley found them acceptable for Appendix B listing.)
The P0 required the item manufacturer's certification that there were no changes in design. material, manufacturing, or interchangeability between January 30, 1976, and the date of manufacture of the replacement boards. Alternately, a list of changes could be provided for Entergy's approval.
The P0 invoked the latest version of 8
4 tt Pa; larm's QA manual, and the QA program which has been approved by Entergy's supplier audit supervisor. The P0 also stated that the provisions of 10 CFR Part 21 apply to the item included on the order.
The P0 called for certification of conformance to all P0 requirements, and required certification of an engineering equivalency evaluation (but not submittal of the evaluation itself) for any substitute parts, with revised technical manual pages or product information.
Panalarm certified that no changes had been made that affected qualification to IEEE 323-1974 and IEEE 344-1975.
A list of drawing revisions was provided, with engineering change notices attached that identified several parts changes.
Panalarm supplied an inspection report, but there was no evidence of testing.
The NRC inspector reviewed an Entergy letter dated February 23, 1994.
The letter stated that based on review of Revision 10 of Panalarm's QA manual, the QA program was considered acceptable for safety-related P0s.
The letter did not mention any audit, and the Panalarm QA manager had no evidence of an Entergy QA audit, or of Entergy surveillance of any activity on the P0.
The P0 did not specifically invoke Appendix B or state that the equipment was safety-related.
If Entergy did intend to perform a safety-related procurement, the NRC inspector did not observe documentation at Panalarm that supported that objective.
c.
Conclusions The inspector concluded that the Model 86 modules supplied to GE conformed to Appendix B of 10 CFR-Part 50.
The optical isolators supplied to Entergy were not demonstrated to be safety-grade by the Entergy and Panalarm documentation reviewed.
Although Entergy's P0 for printed circuit boards invoked 10 CFR Part 21 and Panalarm's QA manual, it was not clear that the P0 covered safety-related equipment.
Absent evidence of an audit or surveillance, there was also no basis to support Method 2 or 3 dedication of the boards for safety-related use.
3.5 Customer audits a.
Scope The inspector reviewed reports of audits by GE, Wolf Creek Nuclear Operating Corporation, and IES Utilities, and a Commonwealth Edison surveillance report.
b.
Observations and Findinas GE performed an audit of Panalarm cn March 7 and 8,1996.
Upon resolution of the audit findings, GE intended to retain Panalarm as an approved supplier of safety-related temperature switches and related commercial grade items.
The Panalarm QA manager stated that at the time of the NRC inspection, both GE corrective action requests remained open; 9
they involved Panalarm's failure to conduct triennial internal and external audits. All five observations were closed.
Wolf Creek scheduled an audit of Panalarm on May 23-24, 1996.
The intent of the audit was to extend the Appendix B scope of Panalarm to 1
include inverters and isolators, and to address Panalarm's failure to notify of a mandatory inspection hold point on a safety-related P0 (this finding was closed by a September 3, 1996, letter). While at Panalarm, the Wolf Creek representative determined that programmatic refinements would be necessary to maintain an Appendix B program, although he considered the procedures adequate to support a commercial grade program.
Panalarm did not plan to revise the QA program or documentation pending decisions concerning ISO and Appendix B commitments, so the Wolf Creek representative changed his audit to become a surveillance of activities on an existing P0.
Wolf Creek did not extend the safety-related scope of supply.
The NRC inspector considered the Wolf Creek representative's actions to be prudent.
IES Utilities, for the Duane Arnold Energy Center (DAEC), conducted an audit of Panalarm on June 13-15, 1995, as a triennial followup to a 1992 audit.
The audit had three findings and seven observations, the last of which were closed by an IES letter dated February 7,1996.
Based on the audit, Panalarm was retained as an approved Appendix B supplier of Model 868 switches, including their repair, for the DAEC.
The NRC inspector noted that the IES audit report and related correspondence were distributed to several other licensees.
Page 1 of the audit report states that it was performed to fulfil DAEC requirements, and there was no indication that it was a joint utility group audit.
The audit report also stated that Panalarm documents 900396 Revision 3 and QC-92-110 Revision 0 would continue to be invoked on DAEC P0s, and specified that subtier supplier documentation would have to be verified by testing, source inspection, and/or audit / survey.
Commonwealth Edison (Comed) conducted a survey of Panalarm on April 6, 1992.
The announcement letter stated that Comed intended to perform Method 2 dedication. Comed's May 21, 1992, letter stated that Panalarm was listed as an approved manufacturer of commercial grade items for Series 10, 50, and 70 annunciators and parts.
The Panalarm QA manager had no subsequent audit information regarding Comed.
The NRC inspector noted that some of the audits by Panalarm customers raised concerns with Panalarm's 10 CFR Part 21 program, yet after the i
concerns were resolved the Panalarm Part 21 procedure was still inadequate as described in Section 3.1.
c.
Conclusions The inspector concluded that the customer audits cf Panalarm were effective, apart from the findings of this inspection report.
3.6 Entrance and Exit Meetinas 10
l i
In the entrance meeting on September 9, 1996, the NRC inspector discussed the scope of the inspection, outlined the areas to be inspected, and established interfaces with Panalarm management.
In the exit meeting on September 12, 1996, the inspector discussed his findings and concerns.
LIST OF PERSONS CONTACTED Lawrence M. Froman, Division Vice President and Manager l
Arlin H. Demerer, Quality Assurance Manager David Minerd, Production Manager Lorenzo Garcia, Production QC Inspector.
Lou Granato, QA/QC Technician I
11
- k.,
a.
' ITEMS OPENED, CLOSED, AND DISCUSSED Onened I
99901303/96-01-01 VIO inadequate Part 21 procedure 99901303/96-01-02 NON inadequate provision for verifying safety-related activities 99901303/96-01-03 NON inadequate organizational definitions in QA manual i
99901303/96-01-04 NON inadequate inspection independence Closed None
' Discussed None 12