ML20127M136
| ML20127M136 | |
| Person / Time | |
|---|---|
| Issue date: | 11/05/1992 |
| From: | Cwalina G, Naidu K Office of Nuclear Reactor Regulation |
| To: | |
| Shared Package | |
| ML20127M116 | List: |
| References | |
| REF-QA-99900366 NUDOCS 9211300013 | |
| Download: ML20127M136 (9) | |
Text
, . - -...-. ._ --- -. . . _ - _-- - - . . - . . .. .
ENCLOSURE 2 U. S. NUCLEAR REGULATORY COMMISSION
} OFFICE OF NUCLEAR REACTOR REGULATION 1- DIVISION OF REACTOR INSPFCTION AND j LICENSEE PERFORMANCE i
. ORGANIZATION: Exide Electronics j Raleigh, North Carolina REPORT NO.: 99900366/92-01
- ORGANIZATIONAL Mr. B. Saylor, Director 3 CONTACT
- Quality and Reliability J (919) 878-2378 l CORRESPONDENCE 3301 Spring Forest Road l ADDRESS: Raleigh, I; orth Carolina 27604 NUCLEAR INDUSTRY Manufactures electrical inverters, j ACTIVITY: uninterruptible po'wer supply (UPS)-units, and rep'acement components.
INSPECTION September 14-17, 1992 4 CONDUCTED:
LEAD INSPECTOR: /- w //
l /r K. R. Naidu, Team Leader Date
/ Reactive Inspection Section No. 2 (RIS-2)
} Vendor Inspection Branch (VIB)
OTHER INSPECTORS: Ronald K. Frahm, Jr. , VIB Jose G. Ibarra, AE0D/R0AB j
APPROVED BY: A* N/5NL G. C. Cwalina, Chief, RIS-2, VIB Da'te i Division of Reactor Inspection and Licensee Performance
.0ffice of Nuclear Reactor Regulation
! INSPECTION BASES: 10 CFR Part 21-and Exide's quality assurance program j- INSPECTION SCOPE: The NRC inspection team evaluated the-circumstances surrounding the loss of five Exide j UPS units following a main transformer failure ,
at the Nine Mile Point Nuclear Station Unit 2.
l PLANT SITE All plants using Exide inverters and i APPLICABILITY: UPS units.
1 9211300013 921123 PDR GA999 EMVEXIDE
!- 99900366 PDR
_ . ~ . . _ . _ _ _ _ . . ~ , . _ _ _ _ _ . _ . _ . _ . - _ _ . _ . _ _ . . _ ~ . . _ . - _ . _ _ _ . - , _ . _ . _ . . . _ _ . _ . _ . . _ _ . . _ _ . -
, 1 INSPECTION
SUMMARY
1.1 yinlations Contrary to the requirements of Part 21 of Title 10 of the Code of Federal Reaulations (10 CFR Part 21), Exide Electronic.s (ExitM frocedure QCP 195, July 1,1985, had not been revised to implement the requirements of the revision of Section 21.21(a) of 10 CFR Part 21 effective October 29, 1991.
[ Violation 99900366/92-01-01, see Section 3.5 of this report).
1.2 Nonconformances None identified during this inspection.
1.3 Unresolved items None identified during this inspection.
2 STATUS OF PREVIOUS INSPECTION FINDINGS 2.1 Deviation 99900366/79-01-01 (Closed) ,
Deviation 79-01-01 stated, in part, that contrary to Criterion VI of Appendix B to 10 CFR Part 50, Exide had not issued a change notice (CN) for changes made to Drawing No. 110603071, Revision A, August 25, 1978. Section 3.3 of the current Quality Control Procedure (QCP) 229, Revision A, of January 25,1991. " Quality Assurance Internal Engineering Change Notice Procedure,"
requires that each drawing that has been marked in accordance with a " signed-off ECN" will be stamped " Red Line Drawing." The Internal ECN clerk or designer will write his or her initials on the stamp mark, and will stamp "NOT VAllD AFTER" on the drawing.
2.2 Deviation 99900366/79-01-02 (Closed)
Deviation 79-01-02 stated, in part, that contrary to Criterion XV of Appendix B to 10 CFR Part 50, rejected material with an attached " Rejected Material" tag was designated as " Accept" (use-as-is) without explicit approval from Engineering. The current QCP 232, " Material Review Board (MRB)
Procedure," requires that a " Rejected Material Tag" (RMT) be completed and attached to nonconforming material. The nonconforming material will be placed in a segregated area pending MRB disposition. Each of the MRB board members, including a representative from Engineering, must be present during the weekly meetings to determine the disposition of such material and complete items 21 through 29 of the RMT. " Accepted" items are so documented on the RMT form indicating approval by the MRB.
?
- 3 INSPECTION FINDINGS AND OTHER COMMENTS 3.1 Entrance and Exit Meetinas During the NRC entrance meeting on September 14, 1992, the inspectors stated the scope of the inspection to the Exide staff. During the exit meeting on September 17, 1992, the team leader summarized to the Exide management and staff the team's findings.
3.2 Principle of Uninterruptible Power Supolv (UPS) Operation The purpose of the UPS is to provide continuous uninterruptible power to important electrical loads should the UPS lose its normal ac input power from the in-plant electrical distribution system. This normal ac power supplies input to the at to de rectifier, which provides regulated de power to the inverter. If the nor':al ac power is unavailable to supply the ac to de rectifier, the alternate de power source to the inverter is supplied by an on-site storage battery. The de to ac inverter provides a reliable, high quality at output power source from the UPS to the critical bus. An alternate source of UPS power output is an off-site maintenance supply, but it is not of equivalent quality to the inverter output.
In earlier vintage nuclear power plants, rectifiers and inverters were installed in separate cubicles to provide uninterruptible ac power.
Subsequent designs combined a rectifier and inverter in a single UPS cubicle.
The switch that transfers ?- power from the UPS to the alternate source is located inside the cabinet and necessitates safety precautions to be taken while performing maintenance to the UPS to prevent injuries to personnel.
Failure reports indicate that the longevity of components inside the UPS units may shorten when operated at abnormal operating conditions. For instance, the condition of power supply batteries and lubricants (grease in circuit breakers) installed inside the UPS rapidly degrade at elevated temperatures and voltages.
3.3 Backaround Information Under various names, Exide Electronics, as the corporation is currently known, designed and manuf actured battery chargers, rectifiers, inverters, and UPS units and supplied them to the nuclear industry. Exide personnel described the manner in which Exide, under the various names of its predecessors, has developed and improved inverter and UPS designs. Exide informed the team that if a customer requests a replacement product for a component that had been manufactured by one of Exide's predecessors, Exide makes a conscientious attempt to locate the necessary drawings to assure the repla m ent is equivalent to the original. For example, when a customer recently requested Exide to supply a replacement printed circuit board (PCB) for a battery charger, Exide conducted a search, located the drawings, and manufactured the PCB even though it had been supplied by its predecessor.
Exide has changed owners several times since producing the battery chargers and the first inverter. Exide personnel stated that the corporation began as the Electric Battery Storage Company and later became International Nickel of
I Canada. The company then became the privately owned Exide Electronics and most recently the publicly owned Exide Electronics.
Records indicate that Exide supplied inverter Models (1) 120/25F1, (2) 120/9.3F1, and (3) 240/30F1, and UPS Models (4) 2730 and (5) 575-60T3-120-208 to various nuclear power plants. Exide personnel stated that items (1),
(2), and (3) were of a common design, which included silicon-centrolled rectifiers (SCRs), a two step inverter (dc-ac), and quasiwave, and did not include an AC-DC rectifier. The corporation, then known as_ Electric Battery Storage Company, first made available the inverter with these components in 1962. Records indicate tiiat these types of inverters were installed at the Oconee Nuclear Station. In the nomenclature of the models, the first number denotes the output ac voltage, the number after the slash denotes the output in kVA, and "Fl" signifies that it is a single phase unit.
In 1982, Exide introduced item (4) as a 2000 series UPS which included transistor-transistor logic (TTL) powered by either rectified dc or 0-cell batteries in the same configuratice as the one installed at the Nine Mile Point Nuclear Station Unit 2 (NMP ~). However, the rectified ac power from the inverter output is preferred. Exide discontinued this single phase product in 1986. In 1972, Exide introduced item (5) as a 3000 series UPS, and continues to produce them with an improved logic design. In 1985, Exide enhanced the design by eliminating 'S "5 relay, which was used to select an alternate power (maintenance power) - 'he load when the inverter output power was lost, adding dual dc logic power supplies and adding an auctioneered configuration to select the source of input power to the logic. Exide personnel stated that they periodically inform customers of design enhancements by conducting presentations when called upon to perform service on their equipment or by distributing sales literature.
3.4 Ouality Assurance (0A) Prooram Exide Electronics used a documented Quality Assurance Procedures Manual (QAPM), Revision A, November 12, 1990, which meets the criteria specified in Appendix B to 10 CFR Part 50 to manufacture and supply safety-related power supplies. However, Exide personnel stated that they discontinued implementing the QA program in 1991 and are revising the manual to meet the requirements of International Organization for Standardization (IS0) 9000 for QA programs.
- Exide expects to issue revised manuals reflecting the requirements for ISO 9000 by the end of October 1992. In a letter of February 24, 1992, Exide informed its nuclear customers that it would no longer accept ar.y purchase orders (P0s) or provide any certificates of conformance (C0Cs) referencing conformance to Appendix B to 10 CFR Part 50. The C0Cs will reference only commercial grade requirements. In the letter, Exide stated that it will continue to give technical and engineering support to all previous orders referencing Appendix B to 10 CFR Part 50. Exide further stated that current
- holders of controlled copies of its QAPM will receive revised manuals reflecting its change in QA policy. The inspection team verified Exide's compliance to selected elements of the existing 0A program as described in the following sections.
- 3.0 Ra u . of 10 CFR Part 24 Procram i
in 1991, Exide had discontinued implementing the QA program v.ch had met the requirements of Appendix B to 10 CFR Part 50. The inspector ceviewed Quality Control Procedure (QCP) 195 "10 CFR Part 21 Compliance," July 1,1985, which addresses the implementation of 10 CFR Part 21. The inspection team found one violation of 10 CFR Part 21 in this area. The QCP does not reflect the requirements of the revision to 10 CFR Part 21 of July 31, 1991 which became i effective October 29, 1991. The QCP also does not include a definition for either a deviation or a defect. The NRC inspection team emphasized to Exide that it still retains 10 CFR Part 21 responsibilities to report defects and noncompliances for those items Exide had supplied in accordance with that regulation. The QA Director stated that he would revise QCP 195 to eflect the current revision of 10 CFR Part 21.
l 3.6 Review of Records for UPS Units Installed at Nuclear Power Plants l Exide supplied a list of the various UPS models supplied to the nuclear power i plants, but was not certain of its accuracy. Records at the NRC reflecting the failures of UPS units indicate that Exide may have supplied safety-related UPS units to at least the Oconee, NMP-2, and Maine Yankee nuclear power plants. Exide personnel stated that '. hey may not have accurate information on the actual users of their UPS units because the purchasers, such as architect engineers and constructors, did not state either where the UPS units were to be used or if they were installed in safety-related applications.
Furthermore, construction personnel at the site frequently installed and started the UPS units. Therefore, Exide could not maintain accurate records on the various types of UPS units installed at the nuclear power plants. If a l customer awarded a contract for Exide to service its UPS units, Exide field personnel dispatched to the site would exchange technical information ard advise the customer of the latest improvements in the UPS design. The inspectors concluded that Exide did not segregate or otherwise track nuclear orders to the extent necessary to trace the different designs that were installed in various nuclear power plants. Exide personnel were, however, able to provide sufficient information to the inspectors to evaluate the specific problem at NMP-2 as discussed below.
3.7 Review of Records Related to the UPS Units That Failed at NMP-2 L On August 13, 1991, an internal failure in the main transformer at NMP-2 caused five, 3-phase, 75 kVA, UPS units to fail. These UPS units were installed in nonsafety-related applications The NRC investigated the incident and documented.its findings in NUREG-1455, " Transformer Failure and Common-Mode loss of Instrument Power at Nine Mile Point Unit 2 on August 13, 1991." The team reviewed the records available at Exide related to the UPS units and determined the following.
The Stone and Webster Company, the architect-engineer for NMP-2, issued purchase order (PO) .NMP2-E035A of January 6,1978, to Exide Power Systems Division to furnish and deliver UPS units in accordance with Specification NMP2-E035A, April 6, 1977. The PO required Exide to ship the complete units i
. - , . . . , . , 4 .-w n.-r.. , ,--,,w,.,. %-w- w ,w+ ,w,v,. ..e. , .y %,&# .,...,wn ,,n r,.y---,.,,,y-.m.,,7v..,-g-- -,,m,w.+.,gg,--wwr.,
I m _,
to NMP-2. Records indicate that Exide shipped the UPS units to NMP-2 on December 5, 1979.
Field Report (FR) N40762, April 2, 1934, stated that Exide personnel performed startup services on January 30 and Fetruary 4, 1984 at NMP-2. During the
- ~ startup, Exide field personnel replaced the control batteries, fuses, and logic boards. Exide stated that all tests had been completed and witnessed and that all UPS units except No. 5 were operational.
In 1985, Exide improved the design of the 3000 series UPS by eliminating the K5 relay, which was used to select an alternate power (maintenance power) to the load when the inverter output power.vas lost, and adding dual de logic power supplies, It also added an auctioneered configuration to select the source of input power to the logic. Exide stated that it had informed all its customers of the design enhancement in a 1985 letter but could not- furnish the team with a copy of the letter. NMP-2 did not purchase this design
- enhancement.
In 1987, NMP-2 requested Exide to furnish replacement technical manuals for UPS units purchased in 1979. Exide stated that since it had not retained copies of the NMP-2 manuals, it had to compile an instruction manual and send it to.NMP-2. In this manual, Exide included a design change that had been incorporated in 1987 to increase the reliability of the UPS units. In the new design, the inverter power (instead of the maintenance power) was the -
preferred power to the logic. Apparently, the NMP-2 site personnel did not recognize that, contrary to the arrangement described in the new manual, the power to the logic in their UPS was connected to the maintenance power supply j
and not the inverter. This discrepancy remained undetected until the post event evaluations and the NRC's Incident Investigation Team (IIT) verified the l configuration after the event of August 13, 1991, as discussed in NUREG-1455.
NRC Information Notice (IN) 91-64 " Site Area Emergency Resulting From a Loss of Non-Class IE Uninterruptible Power Supplies" was issued to alert licensees of potentially generic problems resulting from the common mode failure of the UPS units. Supplement I to IN 91-64 was later issued to provide further -
details on the replacement-intervals for the control logic power supply batteries. The inspection team concluded that the consequences of the August 13, 1991, incident at NMP-2 would have been less severe if there had been better communications between Exide personnel and NMP-2 personnel.
3.8 Review of Desion Documentation Chanae Control Since the IIT identified discrepancies and concerns with changes to the equipment manuals, the team reviewed and verified the implementation of Exide's design documentation change control program. Exide has-an engineering change process (Engineered Change Notice Procedure, RMG007) implemented by an engineering change notice.(ECN) to document changes to controlled engineering documents and drawings. Exide makes engineering changes to its assembly documents and engineering documents to solve design problems and incorporate design enhancements. The Corrective Action Request (CAR, QCP 177) defines the system for continuously correcting the designs.
1 L , . , . - . , . ,.n.. - c. ,.,,,n.,- ,..,_,.,-n,.,,. J. . - . , , . ,...i,,-,,~,-..n _,n,-,._c - ,,rs,.-, ,<n
Exide field engineers can modify the design as they find need. in such instances of a field request, Exide initiates an ECN after receiving the field request to implement the modification. The customer can also submit a special feature request (SFR) to request a design change, which would enter the formal process through the ECN process. The fie?d request is utilized to implement the approved ECN at the site. Another method to change the design is by issuing a field service bulletin (fSB). The field request would reference the FSB when field engineers perform the modification in the field. After installation, the field engineers submit the field request to complete the design change process to assure that the as-built configuration of the UPS units will match the final documentation package. The team concluded that Exide had adequate controls over their design change process.
39 Observation pf Receivino Inspection Activities The team in yectec the area wnere Exide personnel received and inspected incoming material. The team observed that "J1d Drawings" tags had been attached to some material to indicate that the material had been inspected and determined to conform to an earlier revisi..n of the drawings. Exide will continue to use this material until it is depleted. In a separate area, material with "New Drawings" tags had been stored to indicate that the material had been inspected and detM mined in conform with reviced drawings.
Exide stored 5 a segregated era any material that was inspected and determined to be unacceptable. Tne team determined that the identification of the material in the receipt in>pection area conformed with Exide's Procedure, QCP 190, Revision E, of August 24, 1990, " Receipt Inspection."
3.10 Corrective Action Reouest Process The team reviewed the corrective action re oest s (CAR) process used by Exide to find problems and provide solutions. Exide described its CAR system in Quality Control Procedure (QCP) 177, "Correctivs Action Procedure, "
Revision D, February 17, 1992. The team observed that the procedure did not emphasize that corrective actions should be adequate to preclude repetition in the future. Exide uses a computerized, paperless system to control its CAR activities. The originator of a CAR. assigns a priority level defined by QCP 177 as follows: " Top - A problem causing a product line to be shut down; Accelerated - A problem causing inefficiencies, a detriment to the company and the product; Convenience - A problem causing an inconvenience." In paragraph 5.4.1 of QCP 177, Exide stated that " CARS will be monitored aoainst the CAR target date on a weekly basis." In Paragraph 5.4.2, Exide statto that
" monitoring will be done by way of a weekly report which will identify CARS past their carget date and those CARS that have been opened which have not been assigned a target date." In reviewing existing CARS, the NRC inspectors observed numerous CARS that lacked a target completion date and remained open i for several months. Many of these delinquent CARS are coded with a priority l level of " Top" and " Accelerated." Exide informed the inspection team that a previous auditor had identified this same concern during an unrelated audit in August 1992. The NPsC inspectors reviewed the response to the auditor in which Exide committed to review each open CAR by October 12, 1992 and assign it to l an individual with a target completion date. The inspectors found Exide's l actions acceptable. 1 i
\
l l
9 3.11 Maintenance Recommendation for UPS Units Exide performs maintenance on UPS units only upon receiving a service contramt. On such occasions, Exide personnel brief their customers on technical enhancements made on specific models to improve the reliabi'ity of the UPS units. Exide personnel stated that they observed that maintenance cannot be readily performed on the UPS units installed at NMP-2 becuuse the units can not be isolated and removed from service and still provide a means to power the loads. The UPS units were desig..ed and installed with no provision to service the UPS without interrupting power to its load.
Exide recommends that customers perform as a minimum, the following annual maintenance on the UPS units:
Shut down the unit and inspect the physical condition of all components and connections.
- Determine if any electrolytic capacitors leak.
Operate the UPS with low load or no load and check the logic functions.
Verify that no changes were made to the ov,erload settings which jeopardize the reliability of the UPS.
9 Verify the wave shapes wherever applicable.
+
Vary the load to verify all the~ corresponding settings respond appropriately.
Check all the trip functions, fuse failures, and overload conditions.
Verify that the transfer mechanism is operable.
Exide further recommends customers perform minor maintenance on their UPS units at a frequency depending on the environmental conditions in which they operate. Technicians should check and replace the air filters if they are dirty, check for discolored components, such as resistors or capacitors (to indicate that they are overheating), and verify that the D-cell batteries function properly.
i 3.12 Review of Recent Nuclear licensee Audits of Exide The NRC inspectors reviewed recer.t QA audits performed by owners of nuclear power plants and their designees. In March of 1991, the Philadelphia Electric Company (PECO) conducted an audit to determine the implementation of Exide's QA program, which had been written to meet the requirements of Appendix B to 10 CFR Part 50. PECO used a checklist that had been prepared with the guidelines established by the Nuclear Utilities Frocurement Issues Committee (NUPIC). PEC0 did not identify any unacceptable findings and concluded thtt Exide was satisfactorily implementing its QAPM, 10th Edition, November 12,
i.- 1990. The team did not review the NUPic audit checklist because it was not available in the audit folder. Exide stated PEC0 did not send it a copy of this checklist.
In May 1988, the Baltimore Gas and Elactric Company (BG&E) performed a QA audit in which it made five findings. In August 1988, Exide submitted a-1 response to those findings, in which it discussed its proposed corrective a
actions. Exide's audit file did not contain BG&E's acceptance of_the proposed I corrective actions. .In February of 1991, BG&E scheduled an audit but canceled it before the commencement date. The NRC inspectors did not re 'ew the i
adequacy of Exide's proposed corrective actions'to resolve the iindings from i BG&E's 1988 audit. The only concern identified by the team in this area was
- that the audit files did not contain all of the r.ecessary documentation to t
detail the audit findings and their corrective actions.
4 PERSONNEL CONTACTED Exide Electronics A. Schnaidt, Vice President, Manufacturing
- + W. Raddi, Senior Vice President .
K. Agee, Director, Power Systems Group
+ R. Brewer, Manager, Product Support Engineering
- V. Freeman, Technical Support Supervisor
- *+ D. J. Hess, Director, Customer Support
- + M. E. Grady, Manager, Field Service Administration G. D. Jarvis, Manager, Project Engineering
- + J. C. Lovelace, Manager, Quality Control nd Audit
- +
R. A. Machilik, Director, Power Systems Group B. Saylor, Director, Quality Assurance
- + P. Steinman, Lead Auditor l
Attended the entrance meeting on September 14, 1992.
+ Attended the exit meeting on September 17, 1992.
J l
l ,. . . _ - . ,_ ._ -_ . _ _ . _ . _ _ , . _ . _ _ . _ _ _ ~ . _ . _