ML20155D301

Insp Rept 99900871/88-01 on 880125-28.Violation & Noncompliances Noted.Major Areas Inspected:Recurring Problems W/Company 25 Kv Electrical Inverters

ML20155D301
Person / Time
Issue date:	04/19/1988
From:	Baker E, Petrosino J Office of Nuclear Reactor Regulation
To:
Shared Package
ML20155D292	List: ML20155D288 ML20155D294 ML20155D298 ML20155D301
References
REF-QA-99900871 NUDOCS 8806150091
Download: ML20155D301 (11)

Text

.

. g OT.GANIZATION: ELGAR CORPORATION SAN DIEG0, CALIFORNIA REPORT INSPECTION INSPECTION 01/25-28/88 ON-SITE HOURK- O NO.: 99900671/88-01 DATES:

CORRESPONDEhCE ADDRESS: Elgar Corporation ATTN: Mr. P. A. Zecos President and Chief Executive Officer 9250 Brown Deer Road .

San Diego, California 92121 ORGANIZATIONAL CONTACT: Mr. Clyde B. McVicker, QA Manager TELEPHONE NUMBER: (619) 450-0085 NUCLEAR INDUSTRY ACTIVITY: Elgar manufactures electrical inverters, uninterruptible power supplies and associated products.

4 ASSIGNED INSPECTOR: sw ///' 8 888 J./J.fetrosino, Program Development and Reactive Date

, Inspection Section (PDRIS)

OTHER INSPECTOR (S): W. E. Gunther, Brookhaven National Laboratory APPROVED BY: . TT4 / fd7-$$

h e , PDRIS, Vendor Inspection Branch Date E T. Baker, Ac l

INSPECTION BASES AND SCOPE:

A. BASES: 10 CFR Part 21 and Appendix B to 10 CFR Part 50.

l l

B. SCOPE: This inspection was made as a result of recurring problems

! experienced with Elgar's 25 KYA electrical inverters by the Palo Verde Nuclear Generating Station (PVNGS) units 11 and III. The inspection was limited to the Elgar 25 and 7.5 KVA inverter design change activities.

l PLANT SITE APPLICABILITY: Clinton (50-461); Commanche Peak (50-445/446);

Crystal River (50-302); Dresden (50-237/249); Fitzpatrick(50-333); Hatch (50-321/366); Indian Point (50-286); Millstone (50-245/336/423); (continued) l 8806150091 880603 PDR QA999 EMVELGAR 99900S71 n r.n

n

~

. l ORGANIZATION: ELGAR CORPORATION SAN DIEG0, CALIFORNIA REPORT INSPECTION N0.: 99900871/88-01 RESULTS: PAGE 2 of 11 PLANT SITE APPLICABILITY: Monticello (50-263); Nine Mile Point (50-220/410);

Oconee (50-269/270/287); Palo Verde (50-528/529/530); Perry (50-440); Point Beach (50-266/301); RanchoSeco(50-312);RiverBend(50-458);SanOnofre (50-206/361/362); Seabrook (50-443); South Texas (50-498/499); St. Lucie (50-335/389); Vennont Yankee (50-271); Vogtle (50-424/425); WPPSS (50-392);

Waterford (50-382); and Yankee Rowe (50-029).

A. VJ0LATIONS:

Contrary to Section 21.21, "Notification," of 10 CFR Part 21, several deviations were revealed where Elgar neither performed its required evaluation of potentially reportable 10 CFR 21 issues nor infonned the end user so they could cause an evaluation to be performed. A number of the deviations are likely to constitute defects and would have been reportable if evaluated (88 01-01).

This is a Severity Level III violation (Supplement VII).

B. NONCONFORMANCES:

1. Contrary u. Criterion I, "Organization," Criterion II, "Quality Assuranct Program," Criterion V, "Instructions, Procedures and Drawings," of Appendix 6 to 10 CFR Part 50, Elgar has failsd to clearly establish the duties and authorities of its engineering l

personnel to assure its safety-related design and engineering

activities are satisfactorily accomplished (88-01-02).

2. Contrary to Criterion III, "Design Control," of Appendix B to 10 CFR

Part 50 and El ar's quality assurance manual (QAM), it was noted that (88-01-03

a. Elgar has failed to ensure that its design control measures provide for an independent verification of the technical adequacy of its design changes for at least its 7.5 and 25 KVA inverters.

A review of approximately 55 Elgar engineering design changes (ECN) revealed that 12 out of the 55 ECN's were prepared, reviewed and approved by the same person;

b. Elgar has failed to assure that the cumu'Jtive effect of multiple design changes on a given drawing does not affect the function-ality of the safety-related system or component in regard to the original design. A review of the 55 forementioned ECN's revealed that no design function reviews are performed by Elgar to assess the possible affects of the collective design changes on the original system design function;

I g

i ORGANIZATION: ELGAR CORPORATION SAN DIEGO, CALIFORNIA REPORT INSPECTION NO.: 99900871/88-01 RESULTS: PAGE 3 of 11

c. Elgar could not provide previous revisions of its current drawings to the NRC inspector for any of the nine safety-related inverter component drawings that were being reviewed to deter-mine the extent of design changes. This activity was being performed in parallel with an ECN review; and

d. Nine ECN's that were reviewed did not indicate whether an engineering evaluation had been performed.

3. Contrary to Criterion XVIII, "Audits," of Appendix B to 10 CFR Part 50, it was noted that the last two internal QA department audits used QA personnel as audit team members and the OA manager was the audit team leader for one of the two audits (88-01-04).

C. UNRESOLVED /0 PEN ITEMS:

1. Setpoints - Technical discussions were conducted with the cognizant Elgar engineers to determine whether or not generic problems may exist in regard to maintaining setpoint parameters on its logic system circuit cards. A problem with the setpoints was identified at the PVNGS units Il and III and has not been fully resolved by Elgar to date. The scope of the discussions between NRC staff and Elgar personnel included the collective functional affect on a system due to several minor design changes over a period of time. Additional discussions and evaluations are required. Therefore, this issue will be classified as an OPEN item (88-01-05).

2. Notification of end users - Numerous ECN documents were generated with "design error / deviation" stated by Elgar as the reason for the change.

The functionality of licensee inverter units with respect to the I original design parameters is in question. This issue is UNRESOLVED (88-01-06).

3. IC Sockets (ECN 1680) - This issue could effect the seismic qualifica-l tion of licensee inverters that are currently using the associated circuit card. Until further Elgar review is performed, this issue will remain UNRESOLVED (88-01-07).

4 Field Changes - Further review and discussions will be performed during a future inspection to determine the method in which Elgar controls its field design changes. This issue is an OPEN item l (88-01-08).

D. STATUS OF PREVIOUS INSPECTION FINDINGS:

Neither reviewed nor dispussed during this inspection, .

t

J .

1 ORGANIZATION: ELGAR CORPORATION SAN DIEGO, CAllFORNIA REPORT INSPECTION

PAGE 4 of 11 N0.: 99900871/88-01 RESULTS:

i E. INSPECTION FINDINGS AND OTHER COMMENTS:

1. Entrance and Exit Meetings .

The NRC staff informed the Elgar Corporation management representa-

' tives of the saope of the irspection during the entrance meeting on

,' January 25, 1980 and sunmar. zed the inspection findings, observa-tions, and NRC staff concerns during the exit meeting cn January 28, '

1988.

2. Background The NRC Region V staff revealed in May of 1987, that the Palo Verde Nuclear Generating Station (PVNGS) Units 11 and III had experienced numerous problems with its Elgar electrical inverters. However, the licensee was not able to adequately resolve the inverter problems and PVNGS continued to experience an undesirable number of outages due to problems such as blown fuses, failed silicon control rectifiers (SCRs) and printed circuit logic card problems.

The PVNGS facility uses four 25 KVA single phase inverters (model INV2b3-1-101) and two three phase 25 KVA inverters (model The four single phase INV253-3-101) on each of its three units.

inverters are energized during normal operation to supply power to the vital bus while the three phase inverters are in a standby sendition to supply emergency power to the shutdown cooling isolation

' valve motor.

An overall review of the problems that PVNGS experienced was performed by the staff, and an inspection at Elgar was deemed appropriate.

3. 10 CFR Part 21 Requirements Section 21.21, Notification," of 10 CFR Part 21 requires, in part, that each corporation adopt procedures to provide for either evalua-ting deviations identified in a basic component delivered to a nuclear power plant facility or to inform the customer of the deviation in order that the custnmer may cause the deviation to be evaluated.

Contrary to this, two examples were found where Elgar identified deviations or errors in its designed components and neither performed the required evaluation nor informed the applicable customers. Viola-tion itein 88-01-01 was identified in this area.

.,.._,-,,,o , , _ _ . . . - . . _ . , . ,. .r_ -- --

l I

ORGANIZATION: ELGAR CORPORATION SAN DIEGO, CALIFORNIA REPORT INSPECTION N0.: 99900871/88-01 RESULTS: PAGE 5 of 11 The first example was discovered during <2iscussions with Elgar personnel. Elgar determined that PVNGS was having a high number of silcon controlled rectifier (SCR) failures because they were not applying adequate torque when installing SCRs. Without the proper torque applied, the SCR will heat up beyond its nonnal operating temperature and prematurely fail.

However, further discussion determined that Elgar had omitted.the specific SCR torque requirement in its maintenance instructions for the 25 KVA inverters. Consequently, if a licensee replaced an SCR in an inverter, the required torque value may not be applied. For a l

nonnally energized inverter application, the resulting consequences I could be a loss of power to important instrument and control functions causing an electrical transient which could include a reactor trip.

! Conversely, if a SCR was replaceo on an inverter used for a standby l function, a premature SCR failure could cause a total loss of that i function during an actual transient when the inverter was 1ergized l

for emergency operations.

Consequently, since a premature SCR failure in an inverter performing a safe shutdown function coHd result in the '.oss of that function, this issue should have been rearted to all customers. .

The second example was discovered during a review of numerous engineering change notices (ECN's). Several ECN's were reviewed which indicated that they were generated because of a design error or l

deviation. Even though the Elgar ECN corrected the imediata problem, I notifications were not made to other customers that they may have equipment which would require modifications nor was a review performed by Elgar to determine what other equipment was involved. Examples of the errors / deviations include the following:

a. ECN 6379, dated April 1,,1987, "25 KVA Inverter PWM Analog logic Card," Dwg. 643-102-41.' The problem involved a .1 microfarad i

disccapacitor[partnumber 821-104-05] that was replaced because "the leads are weak and broken easily causing customer

~

complaints." No Part 21 eve'uation was performed;

b. ECH 6539, dated December 14, 1987, "7.5 KVA Inverter Filter Assembly," Dwg. 642-211-43. The ECH was issued to correct a documentation error in that a "non-nuclear" SCR had to be changed i

to "nuclear-grade" for eight assembly drawings. However, even though a large number of SCRs and circuit boards were involved, l

1 o

l . - , . .

I

{

4 ORGANIZATION: ELGAR CORPORATION SAN DIEG0, CALIFORNIA REPORT INSPECTION NO.: 99900871/88-01 RESULTS: PAGE 6 of 11 no review was performed to determine if the incorrect SCRs had been sold for use in nuclear power plants prior to the time period that the ECN was approved; .

c. ECN 1499, December 30, 1980, "25 KVA Inverter, Alarm logic Board," Dwg. 643-103-42, referenced the need to increase resistor wattage in order to "reduce the stress factor of resistors" in accordance with IEEE-650, which is the qualification standard for nuclear inverters. No Part 21 evaluation was performed;

d. ECH 4530, dated June 22, 1984, "Static Switch Logic Assembly,"

i l

Dwg. 549-000-2. Ibidem; l

e. ECH 5589, dated August 30, 1985, "Static Switch Logic Assembly."

! Ibidem; and

f. ECN 3530, dated February 14, 1983, "25 KVA Inverter Alarm Logic Board," Dwg. 643-103-42. This ECN was issued to change the value of a resistor on a printed circuit board in order to effect latching of an alann function, i.e., ensure that an out-of-6 tolerance condition remained annunciated. This change was indicated as "mandatory" requiring rework of all units in production; however, no determination was documented as to the affect on previously shipped units.

The above examples are issues that should have been reported to the end user as a minimum so that the end user could perform the evalua-tion for substantial safety hazard required by 10 CFR Part 21.

4 QA Program Establisment and Implementation In parallel with the inspector's ECN review, the applicable QA manual sections, procedures, and instructions that control Elgar's safety-related design and engineering activities were reviewed. The review identified numerous inconsistencies and an overall lack of specificity in the documents that are supposed to provide guidance for the engineering department's safety-related activities. Certain aspects of the regulations were not fully and/or correctly translated into the Elgar program requirements. Nonconformance item 88-01-02 was identified in this area.

As an example, for design changes, Elgar's QAM section 04, "Design,"

requires, in part, that engineering will maintain drawing and change control as prescribed in QAM Section 07. Section 07 requires, in

i l

i I

i ELGAR CORPORATION I

ORGANIZATION: '

SAN DIEGO, CALIFORNIA REPORT INSPECTION h0.: PAGE 7 of 11 99900871/88-01 RTLTS:

part, that engineering will assure that "drawing and change controls are maintained in accordance with this procedure." However, the procedure did not provide specific instruc.tions for the engineering personnel. The lower tier documents were'found to be written in the same manner. The documents were found to not clearly establish what the specific duties and authorities of Elgar's engineering personnel were.

Conversely, the Elgar "Engineering Design Review" prc,;edure, nomber EEP-20, delineates specific duties for the Design Review Board Members.

However, the duties, if performed as stated, do not meet the intent of an NRC Appendix B to 10 CFR Part 50 requirement which states, in part, that "measures shall be established for the selection and review for suitability of application of materials and parts that are essential to the safety-related functions [of a system]."

The section of EEP-20 that was found to be inconsistent with the regulation states, in part, "Design Review Board Members will...ask questions, probe...to sufficiently satisfy themselves that the design approach is sound...They should be on the lookout for over-designed hardware...giving the customer more than he asked for. The members should be mindful of product costs. Any process, material, or part type that appears too expensive should come under investigation to determine if a less expensive approach could suffice." Other Elgar documents that were found to not clearly establish its engineering duties and responsibilites include:

(a) Part 5.0, "Design," of Section 04 of Elgar's QAM.

(b) Section 17, "QA Records," of Elgar's QAM.

(c) Procedure EEP-1, "Engineering Release and Change."

(d) Procedure EEP-3, "Engineering Change Requests."

(e) Procedure 56007-01, "Flow Diagram" [ design changes).

(f) Procedure 60002-01, "Change Notices" [ design change notices).

(g) Procedure 60003-01, "Change Control Board."

5. Design Control The NRC inspectors reviewed numerous Elgar engineering design changes (ECNs), drawings, and change requests (ECRs) to evaluate the adequacy of the Elgar design change control in its 25 and 7.5 XVA electrical inverter areas. As a result of the design control review, three major areas of inadequate design change control were revealed. Noncon-formance item 88-01-03 was identified in this area. The three areas are as follows:

l I

ORGANIZATION: ELGAR CORPORATION SAN DIEG0, CALIFORNIA REPORT INSPECTION RESULTS: ) AGE 8 of 11 N0.: 99900871/88-01

a. Lack of Independence In The Preparation, Review and Approval Process for Design Control - A review of 55 ECNs revealed that 12 out of 55 were prepared, reviewed and approved by the same individual. The 12 ECNs are: ECN 1104, ECN 1397, ECN 1473, ECN 1493, ECN 1499, ECN 1687, ECH 1796, ECN 1836, ECN 2212, ECN 3800, ECN 5988, and ECN 6216.

b. Inadequate Technical Evaluation of The Collective Effect of Multiple ECNs - A review was performed of the majority of.ECN's associated with a particular drawing to verify whether or not an engineering evaluation was periodically performed to assess the cumulative effect of the design changes on the original design function of the system / component. While their individual importance varies for ECNs, even minor design changes, when considered cumulatively, may significantly change the ability of the equipment to operate within its original design specifi-cations.

No objective evidence could be provided to the inspectors that would indicate such a review was performed or that Elgar identi-fied and controlled its design interfaces. Additionally, discussions with the cognizant design engineers revealed that reviewing for the cumulative design change affect was not a

typical past or present engineering practice. Examples include I the following ECNs

(1) ECN 1680, dated March 10, 1981, 25 KVA Inverter, multiple drawings. This ECN was issued to add integrated circuit (IC) sockets to certain safety-related printed circuit boards. The documentation associated with this change does not include or address the added mass in regard to its seismic integrity, nor does it address the seismic qualifi-cation of the subassembly and inverter panel; (2) ECH 2790, dated May 19, 1982, 25 KVA inverter, multiple drawings. This ECN was issued to negate further implemen-tation of ECH 1680 and to revert back to soldering the ICs to the boards. However, it could not be determined if any boards shipped during the 14 month interval when IC sockets were used have been identified to the end users so they may either change out the cards or evaluate the impact on their inverter panel qualification; j

- h ORGANIZATION: ELGAR CORPORATION SAN DIEGO, CALIFORNIA l

INSPECTION REPORT > AGE 9 of 11 NO.: 99900871/88-01 RESULTS:

(3) ECN 3252, dated November 19, 1982, Dwg. 643-209-42 (INV253-1-101). This ECH was issued to change the existing model GE-364 SCR to SCR model GE-384. However, no docu-

~

I mentation was provided to indicate that an evaluation of the affect of the change on the original circuitry design and function was performed; (4) ECN 4031, dated November 30, 1983, Dwg. 643-209-42 (INV253-1-101). This ECH changed the size of fuse F1 from 200 amperes to 250 amperes. No calculations (or reference to) i were documented to indicate that adequate protection of l

internal components and circuitry would still exist. The l

design change was not referenced back to the previously shipped inverters (INV253-1-101) even though the ECN indicated "Mandatory" action and "Rework" was required; (5) ECN 4812, dated January 7,1985, Dwg. 643-102-41 (INV253 101). This ECN changed resistor and capacitor values for i the Analog logic Board. No documented evidence existed to indicate that an evaluation was performed to ensure that l

the original design criteria and functions were within tolerances; j

(6) ECN 5988, dated January 30, 1986, Dwg. 549-000-2. This ECN for the Static Switch logic Assembly required the replacement of circuit board resistors because the SETPOINTS on the overvoltage and undervoltage trip and reset points were too close together. This change was indicated as mandatory for new units and those in production, but did not indicate whether or not an evaluation was performed to assess the impact on the original design function or the existing cards at any licensee facilities; (7) ECH 5711, dated October 16, 1985, Dwg. 549-000-2. This ECN for the Static Switch Logic Assembly required the addition of a resistor and a diode to a circuit board to correct a problem in which the static switch would not stay in "Reverse" when manually selected. The ECN stated that the change was not safety-related even though the referenced drawing is labeled "Nuclear Safety-Related;"

(8) ECN 2075, dated August 11, 1981, Owg. 643-101-42. This ECN for the 25 KVA inverter Logic Board changed the

I

.. g ORGANIZATION: ELGAR CORPORATION SAN DIEGO, CALIFORNIA REPORT INSPECTION N0.: 99900871/88-01 RESULTS: PAGE 10 of 11 resistance values for three resistors. it was not indicated whether or not an evaluation to assess the impact on the original design or other ECH changes was performed; (9) ECN 3116, dated September 27, 1982 Dwg. 642-211-43. This ECN for the Filter Assembly of the 7.5 KVA inverter required the addition of a 2.5 K, 25 watt resistor across the primary of the transformer. No reference to an evaluation of the impact on the original design or other ECN changes was in evidence; and (10) ECN 3726, dated June 1, 1983, Dwg. 642-107-40. This ECN for the Analog logic Board for the 7.5 KVA inverters required several resistive and capacitive device changes.

No reference to an evaluation of the impact on the original design was in evidence.

c. Inadequate Objective Evidence of Satisfactory Performance of Safety-Related Activities - During the review of the ECRs and ECNs it was obvious that in many cases the design change docu-ments did not contain enough detail to show circuit changes, device location changes and other associated details. The applicable drawing revisions were requested that the specific '

ECNs were written against. It was then revealed that Elgar had not retained any of the previous revisions to its current drawings that were requested by the inspectors. Therefore, the ECNs and other change documents could not be compared or correlated with the applicable drawing revisions that would show the "Before" condition and following revision, the "After" condition.

Without the benefit of all the associated documentation to review, it could not be determined if Elgar adequately performed its safety-related design change activities.

6. Quality Assurance Records As discussed in 5.c above, it was revealed that Elgar has not retained previous revisions to several of its 25 and 7.5 KVA inverter drawings, even though its QA manual requires that it retain "obsolete drawing masters." The specific drawings that were requested by the NRC inspectors for their previous revisions are as follows: Drawings 642-269-42, 643-101-42, 643-102-41, 642-107-40, 643-103-42, 642-211-43, 549-000-2, 543-118-1 and 549-000-9. Nonconformance item 88-01-03 was identified in this area.

l GRGANIZATION: ELGAR CORPORATION SAN DIEGO, CALIFORNIA REPORT INSPECTION NO.: 99900871/88-01 RESULTS: PAGE 11 of 11

7. Audits Criterion XVIII, "Audits" of Appendix B to 10 CFR Part 50 requires, in part, that a comprehensive system of pfanned and periodic audits be carried out to verify compliance with all aspects of the quality assurance program and to determine the effectiveness of its QA program. The audits shall be performed by personnel not having direct responsibilities in the areas being audited, and the audit results will be reviewed by the management having responsibility in the audited area.

Contrary to the above, the inspectors revealed that the Elgar 1987-1988 audit schedule, revision 1, indicated numerous examples of QA personnel performing the duties of "auditor" in different QA depart-ment audits. The last two QA department audits, "Inspection" (July 1987) and "Incoming Inspection" (September 1987) did not have personnel from any department other than auditors from the QA depart-ment on the audit team.

The "Inspection" audit contained two auditors who were both assigned to Elgar inspection activities. The "Incoming Inspection" audit included one auditor who was assigned to "Receiving" inspection i

activities. Additionally, on the later audit, the QA manager was the designated audit team leader. Nonconformance item 88-01-04 was identified in this area.

F. PERSONS CONTACTED:

• T. Erickson

• S. Reeves

• R. Parrish l *C. McVicker

• D. Risdon l H. McAlpin S. Sedio  :

• Attended entrance and exit meeting.

. . _ . , - - _ - _ _ - . . . . . . - . . . - .-