ML20141E647
| ML20141E647 | |
| Person / Time | |
|---|---|
| Issue date: | 05/15/1997 |
| From: | NRC (Affiliation Not Assigned) |
| To: | |
| Shared Package | |
| ML20141E582 | List: |
| References | |
| REF-QA-99901309 99901309-96-01, 99901309-96-1, NUDOCS 9705210077 | |
| Download: ML20141E647 (21) | |
Text
.
~_
l i
I e
i U.S. NUCLEAR REGULATORY COMMISSION l
OFFICE OF NUCLEAR REACTOR REGULATION oo Revised May 15, 1997 for Public Release oo i
Report No.:
99901309/96-01 l
Organization:
Lucent Technologies, Incorporated Microelectronics Group Systems Business Unit 3000 Skyline Drive Mesquite, TX 75149 1
Contacts:
Mrs. Maureen F. O'Brien, Director Development & Engineering Department (972) 284-2200 Mr. Augustus J. Donnell, Director Quality Assurance (972) 284-3523 Nuclear Industry:
Designed, qualified, and fabricatpd cylindrical lead-acid battery cells, Lineage
- 2000 Round Cell batteries (round cells), that are supplied to nuclear power plant facilities for Class lE electrical safety-related applications.
Date of Inspection:
November 18-20, 1996, and January 6-9, 1997 at the C&D-Leola facility.
Inspectors:
Kamalakar R. Naidu, Team Leader Stephen D. Alexander Joseph J. Petrosino Saba N. Saba Approved by:
Gregory C. Cwalina, Section Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation NOTE: Specific information determined to be proprietary, pursuant to 10 CFR Part 2.790, has been removed from pages 5-10 of this report and replaced with the following symbol:
(oo) g 2$$$99 NO M **
99901309 PDR
~
Registered trademark of Lucent technologies, i
~>
l 1
INSPECTION
SUMMARY
During this inspection, the NRC inspectors reviewed the implementation of selected portions of Lucent Technologies, Incorporated, Microelectronics Group, Power Systems, Dallas Works (Lucent's) quality assurance (QA) program, l
reviewed activities associated with AT&T's (currently known as Lucent Technologies) Lineages 2000 round cells, and subsequently observed round cells l
being manufactured at the C&D* Charter Power Systems, Incorporated (C&D),
facility in Leola, Pennsylvania, on January 6-9, 1997.
The team was aware l
that several nuclear plant round cell batteries experienced premature loss of capacity.
The inspection bases were:
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reculations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" The American National Standard Institute /The American Society of
=
Mechanical Engineers (ANSI /ASME) Standard N45.2-1977, " Quality Assurance Program Requirements for Nuclear Facilities" (ANSI N45.2-1977), as endorsed by U. S. Nuclear Regulatory Commission Regulatory Guide (RegGuide) 1.28, " Quality Assurance Program Requirements (Design and Construction)," Revision 2, February 1979 (Reg Guide 1.28)
The team determined that, during an approximate three year period, Lucent identified weaknesses in the services that C&D were providing during the manufacture and control of its round cells, but Lucent did not appear to take prompt corrective action.
Consequently, two nonconformances to the quality requirements of 10 CFR Part 50, Appendix B were identified as a result of those requirement which had been contractually imposed in 1990 by the Duke Power Company (Duke) and accepted by AT&T/ Bell Laboratories.
These nonconformances are discussed in Section 3.2.
2 STATUS OF PREVIOUS INSPECTION FINDINGS This was the first NRC inspection at the Lucent Technologies Facility.
3 INSPECTION FINDINGS AND OTHER COMMENTS 3.1 Backaround In 1973, Bell Laboratories of the American Telephone and Telegraph Corporation (AT&T), developed AT&T round cells to be used in standby power supply batteries for its telecommunications systems. The round cell design, designated as the AT&T Lineagee 2000 round cell, has been very successful in i
telecommunications applications.
The round cell is a cylindrical, lead-acid j
type battery cell with convex shaped plates that are used in a horizontally l
2
l stacked configuration (see Figure 1).
The round cells were developed using a low specific gravity electrolyte.
Bell Laboratories coordinated a subsequent high specific gravity electrolyte application for nuclear power plants.
In 1988, based upon approximately 15 years of telecommunications field usage, AT&T determined that becau:e of the corrosion product of the pure lead plates (lead-dioxide which is electrochemically active), the available battery cell capacity of the round cells were found to increase over time. AT&T warranted its round cells for 40 years (35 years for their high specific gravity batteries). Consequently, the original design objective found in AT&T's technical information sheets indicated that the round cell would provide:
(1) longer life, (2) higher reliability, (3) low maintenance, and (4) increased safety.
AT&T's 1987-1989 marketing pamphlets stated that the geometrical shape of the round cell assures uniform active material growth rate, while the pure lead grid provides for a slower plate growth rate as compared to lead calcium or lead antimony cells.
Therefore, the combination of uniform growth and a slower plate growth gives the round cell a " life expectancy of up to 70 years with its capacity increasing over its lifetime."
It was estimated by AT&T that the pure lead positive grid will grow about 2%
over 80 years compared to a growth of 4% over just 15 years in lead calcium cells. Therefore, AT&T indicated that "the result is a cell whose capacity will actually increase with age and whose lifcd ce is greater than any other battery available today."
In 1989, Bell Laboratories, in conjunction with Impell Corporation, Melville, New York, developed a round cell seismic testing program to qualify its round cells for nuclear power plant facility Class lE electrical safety-related applications.
In 1991, AT&T contracted with National Technical Systems (NTS),
Acton, Massachusetts to develop a test program and seismically qualify the associated battery racks.
In 1991, after AT&T-Bell Laboratories completed its qualification of the round cells, Low specific gravity round cells were installed and used at:
Oyster Creek - GPU Nuclear Corporatior.
Wolf Creek - Wolf Creek Operating Corporation AT&T's high specific gravity round cells were installed and used at:
McGuire Station - Duke Power Company Palo Verde - Arizona Public Service Company Braidwood - Commonwealth Edison Company, and The team was aware that high specific gravity round cell batteries had lost capacity prematurely at the Palo Verde, McGuire and Braidwood nuclear power i
plants. These batteries were subjected to actual electrical service conditions similar to those of conventional (rectangular) cell batteries.
3 l
However, it appeared that the round cells, particularly those with high specific gravity, may have been more sensitive to these conditions than conventional cells.
The round cells have been manufactured for Lucent Technologies by C&D since i
they were developed in approximately 1973. After manufacture, C&D electrically connects a group of 72 cells together in a series configuration called a string. The strings then go through the an initial charging and conditioning process and then into final capacity testing. After the capacity testing, C&D, Lucent and typically the end user will select 59-64 (for nuclear plants) of the best cells for use in station batteries.
The team noted that each of the round cell strings for the above listed nuclear plants were handled as commercial grade products except for the set supplied to the McGuire station.
Duke imposed 10 CFR Part 50, Appendix B QA and 10 CFR Part 21 requirements on AT&T and they were accepted by AT&T for the McGuire station round cell procurement.
3.2 Control Of Sub-Tier Vendor's Ouality Proaram
- a. Scope During the January 1997, NRC staff inspection at the C&D-Leola facility, the team observed manufacturing activities and reviewed associated documents to assess the adequacy of the manufacture of the round cells.
- b. Observations and Findinas Process Controls The team observed numerous manufacturing process control activities at C&D's facility and reviewed associated documents.
The team noted that many of the C&D manufacturing work stations had key sections of the applicable process control work instructions posted for use of the area personnel.
The team also noted that a Lucent required resident inspector is stationed at the C&D facility.
The resident inspector is typically on the premises several days a week.
The resident inspector witnesses testing activities of round cell battery strings, and performs in-process manufacturing verifications and random inspection of incoming material.
No anomalies were noted by the team in the manufacturing process control activities that were observed.
Charaina & Conditionino The team performed a comparison of the C&D procedures, practices and documentation used during the process of initial charging and conditioning of round cell strings with the charging and conditioning requirements (design bases) of Lucent's design and manufacturing specification KS-20472 and associated documents.
The team reviewed the charging records for string 345-96 for McGuire Station and strings 223-92 and 224-92 for Oyster Creek.
Although AT&T/ Lucent accepted 10 CFR Part 50, Appendix B, as a purchase order requirement j
from Duke for the McGuire string of round cells, C&D conducts all of its manufacturing activities (including charging and conditioning) in the i
{
l 4
l
l same manner.
The team identified several instances in which C&D's procedures, practices and documentation for the charging and conditioning process departed from the requirements of the Lucent design specification. The following are examples of these instances:
KS-20472, Section 5.25, Step (c), prescribed the initial formation charge for the string. The step included a cautionary note that stated that the cell temperature should not be allowed to exceed a i
certain value. Although C&D stated that the temperatures are monitored, they were not recorded, nor was there a provision in the C&D rc:ord forms to do so.
KS-20472, Section 5.25, Step (d), prescribed the first conditioning discharge following the initial formation charge, per Table II for List IS round cells, to be (oo) amperes (amps) for [oo) hours, the
"[oo)-Hour Discharge." This discharge removed [oo) ampere-hours (ah) from each cell of the string. The step included the warning "No individual cell should be allowed to go below (oo) volts."
KS-20472, Section 5.25, Step (e) required that, following the [oo)-
hours removed during the discharge for a minimum of (oo) hours.
This charge then should have rescored a minimum of (oo]ah ((oo]) and a maximum of (oo]ah ((oo)).
However, the overall charging record sheet, called " Procedure Sheet 1," was pre-printed with the annotation, "[oo) amps for
[oo) hours" for this charge.
This charge would put in (oolah, (oo]ah more than allowed by the specification. Although C&D produced a 1986 memorandum from Lucent (then AT&T/ Bell Laboratory) authorizing this departure from KS-20472, the specification had not been updated, in over four years, to reflect this modification to the prescribed charging and conditioning (C&C) process.
The charge records reflected that the string was, in fact, recharged per Procedure Sheet 1 instead of KS-20472.
KS-20472, Section 5.25, Step (f), prescribed a "First Taper Discharge," per Table III for List IS round cells, to be for a certain period at [oo) amperes, a period at [oo), then (oo), for a total of [oo) ampere-hours removed.
The first post-taper recharge per Step (g) was supposed to be [oo) of the.[oolah removed in the previous discharge, i.e., a maximum of [oo]ah ([oo]).
- However, Procedure Sheet I was pre-printed with the annotation, "[oo) amps for [oo) hours" followed by "[oo) amps for [oo) hours," a total of
[oo]ah for this recharge, (oo]ah more than allowed by the specification.
Although C&D produced another 1986 memorandum from i
Lucent (then AT&T/ Bell Laboratory) authorizing this departure from KS-20472, the specification did not reflect this four-year old design basis change to the prescribed charging and conditioning (C&C) process.
The charge records reflected that the string was recharged per Procedure Sheet 1 instead of KS-20472.
i 6
s KS-20472, Section 5.25, Steps (h) and (j), prescribed the [oo) sequential "[oo]-Amp Discharges." The steps stated " Discharge at I
the Table IV rate [oo] amps for List IS round cells] until the first cell reaches [oo] volts." However, C&D 01-20-0900.2, " Charging" (Section 6.3.16, " Water Discharges"), did not provide for normal i
discharge termination upon reaching a prescribed voltage (on the first cell to reach it).
Instead, Step 6.3.16.3 called for j
termination "at the normal discharge end time." The note above step 6.3.16.3 stated:
"If any cell has a voltage of (oo] volts or less, terminate the discharge and notify your supervisor." Therefore, the C&D procedure called for discharging until an end time that is not defined as the termination point for this particular discharge in KS-20472. Also, the C&D procedure prohibited allowing any cell to go below [oo] volts, but did not require discharging until reaching
[oo] volts on the first cell that reaches this voltage as the language of the Lucent specification states.
In view of the explicit language in Step (h) of the specification (as distinguished from language explicitly prohibiting discharging below (oo] volts elsewhere in the specification), the language in the C&D O! was contrary to the requirements of specification KS-20472.
As a result, the charge records showed that both the
[oo]-amp discharges went only to (oo] volts on the lowest cell (s).
Although the practice of discharging until the first cell reached (oo] volts (instead of the prescribed [oo] volts) had been authorized in 1986 on a limited, experimental basis, C&D could not provide documentation authorizing this departure from KS-20472 for general production use.
This resulted in the string not being discharged in this step to the extent required by KS-20472.
KS-20472, Section 5.25, Step (i), prescribed the recharges for the
[oo] sequential "[oo]-Amp Discharges" of Steps (h) and (j) respectively.
The recharges were supposed ia be (oo] of the ampere-hours removed in the previous discharge.
Lince both (oo]-amp discharges continued for about (oo] hours, there were (oo]ah discharged.
Therefore, according to the KS-20472 formula, a minimum of (oo]ah ((oo]) should have been recharged.
However, Procedure Sheet I was pre-printed with the annotation, "(oo] amps for (oo]
hours," a total of only (oo]ah for these recharges or (oo]ah less than required by the KS-20472 formula using the abbreviated discharges.
This was also much less than intended by KS-20472 because the recharge amount, even if calculated and performed in accordance with the KS-20472 formula, would have been based on a discharge to only (oo] volts, less than the amount of discharge specified by KS-20472 in Steps (h) and (j).
KS-20472, Section 5.25, Step (k), prescribed a "Second Taper," or
"[oo]-amp Taper" discharge per Table V for List IS round cells, to be for (oo] amps until the first cell reached [ool volts followed by l
[oo] amps for [oo] hours and finally, (oo] amps for [oo] hours.
Step (1) required discharge to "[oo] volts per cell average" or for a minimum of (oo]ah with the caution that no cell should be allowed 6
l to go below (oo] volts.
However the discharge record (Form RS533) showed that no cell went below [oo] volts during the [oo]-amp phase and none reached [oo] volts during any part of the second taper discharge. Although the minimum total ampere-hours were discharged, no cell achieved the lowest voltage specified by KS-20472.
The final capacity testing specified by KS-20472 following charging and conditioning was modified for Strings 223-92 and 224-92 by a procedure required by Lucent's customer, GPU Nuclear, licensee for Oyster Creek.
GPU Nuclear had contracted National Technical Systems (NTS), Inc., of Acton, Massachusetts, to perform the dedication of the round cell strings for Oyster Creek purchased from Lucent as commercial grade items. NTS Procedure 60145-93N, Revision 2, was used as the basis for dedication of the strings.
The NTS dedication procedure modified the standard capacity test discharge of 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> following the 2.17-volt float period prescribed in KS-20472.
Instead, the NTS procedure called for a high-current (oo)-hour discharge, recharge, and an [oo)-hour discharge, intended to better simulate design basis service conditions at Oyster Creek.
The test record contained a notice of anomaly indicating that only [oo]-
percent capacity had been demonstrat J by the discharge tests, but that this was acceptable.
The explanation of the reduced capacity was the interruption of the standard " conditioning" sequence to perform the NTS dedication tests.
However, this language did not accurately characterize the circumstances in that the conditioning phase of the KS-20472 process, including the final [oo]-day float charge, was, in fact, completed. Only the final [oo)-hour capacity test discharge, not considered part of conditioning, was omitted in favor of the NTS (oo]-hour and [oo)-hour discharges.
In reviewing the test and cell selection records for strings 223-92 and 224-92, the inspector noted that the [oo]-hour test for 223-92 was terminated [oo] minutes early. A notice of anomaly (NOA) also noted the early stop, but had the wrong time.
In addition to the records having a number of other anomalies, such as not all forms being fully or correctly filled out and some erroneous operation ending times recorded.
For example, the (oo]-hour recharge for the first [oo)-amp discharge was listed as staring at (oo] on 11/21/92, but ending at (oo) on 11/22, a period of (oo] hours instead of the
[oo) hours pre-printed on Procedure Sheet 1; yet, the [oo]-amp discharge was recorded as starting at [oo) on 11/22.
Although this entry was likely the correct one, it showed the discharge starting and continuing for (oo] hours while the previous recharge, as recorded, was still in progress.
The charge end time entry of [oo]
on 11/22 was apparently in error, but there was no indication that the record had been reviewed and the error identified by C&D QA, engineering or management or representatives of Lucent or GPU Nuclear.
7
I s
The inspector also noted that the cell in position 43 in string 223-92 (cell serial no. 78471) finished the 8-hour test with a low-out-of-specification voltage of 1.589, yet this cell was selected for shipment by GPU Nuclear according to the selection record.
Conversely, although the cell in position 64 had a low voltage of 1.664 volts, but met the acceptance criterion of greater than 1.65 volts, it was rejected. The record indicated no rationale for the apparently inappropriate cell selection, nor could C&D offer any explanation.
The inspector also reviewed the charging and conditioning records for round cell string 345-96, produced for Lucent's customer, Duke Power for Class IE service at McGuire Station.
The same process was used for.this string as for the Oyster Creek strings discussed above except that the standard final 5-hour capacity test discharge was conducted.
Therefore, this string underwent charging and conditioning with the same deviations from KS-20472 as occurred with the Oyster Creek strings, only some of which, as above, had ever been authorized by Lucent memorandum, and still without formal revision to KS-20472 (Issue 8 still in effect).
Chemical Analysis of Epoxy Mixture The team reviewed the applicable section of KS-20472 regarding preparation and mixture verification of epoxy used to seal the round cell jars around the terminal posts. Also reviewed were the Lucent epoxy titration procedure referenced by KS-20472 that was prescribed on AT&T Drawing L193472, Issue 5, and the C&D epoxy titration procedure, BT-4, Revision 0, dated March 7, 1995.
The procedures specify the quantitative analysis by titration of percent resin in hardener (done on each new mixed batch) and of the equivalent weight of hardener (done on each lot of hardener and each snift), the results of which are used in the percent-resin-in-hardener analysis calculation.
Following this review, the team observed a routine set of epoxy titration analyses, perfor:ned by C&D's usual analyst in the C&D laboratory. The analyses observed were the determination of percent hardener in resin, and the determination of the equivalent weight of hardener.
The team identified two instances in which C&D epoxy titration procedure BT-4 deviated from AT&T Drawing L193472:
L193472 called for titrating into the sample in a 125-ml Erlenmeyer flask; whereas BT-4 prescribed a 250-ml flask, and L193472 required adding a [oo] drops of Brom Creosol green indicator; whereas, BT-4 specified [oo] to [oo] drops.
The size of the flask can affect the efficiency of mixing of the sample (by means of a magnetic stirrer) with the indicator, and more t
l importantly, with the drops of titrant ([oo] sulfuric acid).
The amount j
of indicator added can affect the visibility of the endpoint.
t 8
Other. deficiencies in C&D procedure BT-4 noted were:
The C&D procedure inappropriately defined the titration endpoint as a color change from blue to vellow; however, the analyst correctly -
described the color change as.being from blue to creen.
The acceptance criteria for percent-hardener-in-resin and for hardener equivalent weight given on L193472 were not in the C&D procedure for use by the analyst, although the analyst was familiar with them.
No sample vortex characteristics were specified to ensure uniform mixing in the flask.
The team observed two instances in which the analyst deviated from C&D procedure BT-4:
l-Instead of the 250-ml Erlenmeyer sample flask specified by BT-4,- the l
- analyst used a 125-m1 flask as specified by the Lucent drawing; L
although the analyst was unaware.of the drawing requirement. He i
explained, correctly, that for the specified 50-m1 sample size, the smaller flask provided_ the depth of liquid necessary to achieve an t
adequate vortex (using the specified magnetic stirrer) for proper mixing of the indicator and the titrant in the sample.
The C&D analyst stated that he typically tdds [oo) drops of i.
indicator instead of the (oo) specified in BT-4, and actually added
[oo) drops, nearly twice as much indicator as specified in drawing L193472.
l In addition, the analyst did not employ the standard titration technique when near the expected endpoint of so-called " drop splitting" or " drop transfer" using a glass rod to take each drop allowed to form at the burette nozzle as.the titrant level is lowered by the smallest-burette graduation' and placing the drop in the sample.
Instead, the analyst simply slowed the expected near-endpoint rate of titration to about (oo]
1 l
drop per second, which may not. allow for complete mixing and reaction of the titrant with the' sample and indicator so as to show the endpoint when it first occurs before the next drop falls into _the sample.
This practice, with a less experienced analyst, and also using the 250-m1 flask if BT-4 were followed, could cause the actual endpoint to be missed, thus degrading the sensitivity of the analysis or the accuracy of the results.
Vendor Surveillance _ Criterion VII, " Control of Purchased Material, Equipment, and Services," requires that measures shall be established to j
assure that services conform to the procurement documents, and that the effectiveness of the control of quality by contractors shall be assessed at intervals consistent with the importance and complexity of the product or services. Additionally, Criterion XVI, " Corrective Action,"
l of 10 CFR Part 50, Appendix B requires that conditions adverse to j
quality are promptly corrected.
e A review of the AT&T Quality Management & Engineering Department (QM&E) audits and surveys indicated that AT&T/ Lucent performed on-site audits j
of C&D to observe their capabilities and performance as required by Lucent's quality manual and technical specifications and drawings.
The team noted that the QM&E personnel identified numerous quality problems over a several year period. However, contrary to the Criterion VII requirement, Lucent did not take prompt action to address the effectiveness of its round cell manufacturer's control of product quality, that is, although Lucent identified numerous examples of conditions adverse to quality over a three year period at its round cell sub-tier manufacturer's facility, Lucent did not promptly correct those conditions adverse to quality identified or assure that the sub-tier manufacturer addressed the ineffectiveness of its program implementation consistent with the importance and complexity of the round cell product.
It was also noted that the quality and reliability of the product that was shipped to nuclear power plants did not appear to be addressed in the documents reviewed by the team.
Lucent did not assure that the manufacturing services that it contracted a sub-tier vendor for fabricating its round cells for the McGuire station consistently conformed to the procurement document specifications and requirements during the charging and conditioning manufacturing activities.
For example, a November 2, 1994, AT&T Bell Laboratories letter to the C&D Leola facility quality control (QC) Manager and associated documents discussed the findings of a September 21, 1994, mini quality audit of the round cell manufacturing and quality systems.
AT&T's correspondence indicated that 15 separate areas were investigated during a period when C&D was in the process of increasing production rates from (oo] to [oo]
cells per day.
The correspondence indicated that 11 of the 15 areas audited did not conform to AT&T's specification requirements.
The correspondence also indicated that in light of a recent problem encountered with 1992 series ISH cells sold to Arizona Public Service (Palo Verde), "it would behoove C&D to spend additional time and effort on training new employees and increasing inspection" at those operations where these new employees are located.
The conclusion reached by the AT&T auditors was that the overall round cell quality was not up to usual standards.
The letter to C&D itemized specific areas of concern and requested C&D's corrective action' plan.
A December 19, 1994, AT&T Bell Laboratories Trip report documenting a December 14, 1994, follow-up mini quality audit stated that C&D had not compiled or composed the required corrective action plan, and the overall manufacturing process quality was found to be lower than C&D's usual standards.
The trip report stated that although the observed deficiencies were not critical, they indicated "a general relaxation in standards."
The December 1994 report stated that on December 14, the inspection of round cell manufacturing process quality indicated that the manufacturing quality standards were continuing to trend downward.
The 10
m a
letter indicated that.the observation had been made over a period of time, wasinot specific to any one area, and that C&D's emphasis on quality and the commitment to quality.have diminished over the last six months.
The staff engineer recommended that AT&T and C&D meet to address both Companies' concerns and to " refocus on the critical importance of quality to this product's continued success."
Three months later on March 15, 1995, another mini quality audit was l
conducted and identified findings in 18 areas at-the C&D-Leola facility.
The overall conclusion reached'was that " quality is poor." The AT&T staff performing the audit stated that work instructions need to be reviewed with.both the operators and C&D quality control inspectors and the importance of producing good quality parts needs to be reinforced.
c.
Conclusions l'
No anomalies were noted by the team in the manufacturing process control activities that were observed during the performance of this inspection.
However, in the charging and conditioning process area, the team concluded that:
)
C&D charging and conditioning proced.ces and practices deviated from-KS-20472 and only some of the departures were authorized by Lucent which had not updated its specification in several years to reflect the different practices C&D procedures were not always followed and records were not always fully accurate, nor apparently carefully scrutinized by Lucent, the ultimate user or its representatives, and the 64 cells selected by GPU Nuclear from each of the 72-cell a
strings produced and tested, did not all meet the acceptance criteria following the final test discharge.
In tha chemical analysis of epoxy mixture area, the team concluded that the anomalies from the AT&T/ Lucent drawing as well as the quantitative analysis practices employed could affect the accuracy and repeatability of the epoxy titration analyses for percent-resin-in-hardener and equivalent weight of hardener. Additionally, although there is a wide range in the acceptance criteria for the epoxy titration analyses, the team was given the understanding that neither Lucent nor its customers had" identified any of the anomalies observed by the team in the titration process. Therefore, the effects of these anomalies had not been evaluated by Lucent or their customers to determine their significance.
Contrary to Criterion III, " Design Control," of 10 CFR Part 50, Appendix B,. Lucent had not ensured that all of C1D's procedural and process attributes were appropriately or correctly translated into instructions for C&D manufacturing process controls. Nonconformance 99901309/96-01-01 was identified in this area.
j 11 gr-r-r
'+-7
- -9^
e-t M-+
r---M e--
Tr w
e-w 4+8W+--a--
w---~
4e O
Contrary to Criterion VII, " Control of Purchased Material, Equipment, i
and Services," a review of the AT&T QM&E audits and surveys for 1994-1995 indicated that AT&T/ Lucent did not adequately ensure that its round cell product consistently conformed to Lucent's specifications and drawings, and did not adequately assess the effectiveness of its sub-tier vendor's control of quality of the round cells during portions of 1992-1995.
l Although the team concluded that the'AT&T/ Lucent auditors identified negative quality aspects indicative of an ineffectively established or implemented quality program, appropriate measures to determine the cause of the problem and implement corrective action were not taken.
Neither was the product or process questioned regarding quality and reliability of the product.
Nonconformance 99901309/96-01-02 was identified in this area.
3.3 Environmental and Seismic Oualification of Round Cells a.
Scope The team reviewed the Lucent environmental qualification records, seismic' qualification records and other technical documents prepared in l
support of the use of Lucent's round cells in nuclear power plant Class lE applications.
These documents included:
ABB-Impell Report 02-5012-1493, Revision 1, dated July 1, 1991,
" Qualification Report for Environmental and Seismic-Qualification of AT&T Lineage" 2000 Round Cell Batteries and Battery Stand;"
Lucent Technical Memorandum 52488-880817-12TM, " Accelerated Testing a
of the AT&T Lineage" 2000 Round Cells," dated August 17, 1988; l
Lucent Technical Memorandum 52481-900905-1TM, Accelerated Testing of AT&T Lineage" 2000 Round Cells in 1.300 Specific Gravity Sulfuric Acid," dated September 5, 1990; Lucent. Technical Memorandum 52483-880923-19TM, " Performance of 15-Year-Old Lineage" 2000 Round Cells," dated October 25, 1988; Lucent Memorandum, " Report of Meeting with Nuclear Customers and NRC j
a Regarding Round Cell batteries," dated April 19, 1996; Lucent memorandum, "Model for Failures of APS Round Cells," dated October 24, 1994; and 1
Lucent Technical Memorandum 61419, "Cause and Effect of Premature Capacity Loss in High Specific Gravity Round Cell Batteries During Repetitive Deep Discharge Cycling," Dated August 26, 1996.
l i
a -
a s
u
+w a
w -.-
b.
Observations and Findinas l
l The team determined that the qualification program that was develeped and used did not ensure that the round cells were subjected to expected actual electrical service conditions such as, frequent partial discharges of initially high current, severe load profile discharges, station blackout drills, and more representative load profiles and charging regimens.
Lucent documents indicated that these actual operating conditions, particularly in conjunction with the hiah specific aravity electrolyte was deleterious to the round cells and caused premature degradation in terms of loss of capacity.
For example, the standard capacity discharge test did not accurately predict the effect of routine surveillance testing on the round cells.
This was true in part because The Institute of Electrical and Electronics Engineers, Incorporated (IEEE) Standard 450-1987, l
" Recommended Practice f4r Maintenance, Testing, and Replacement of Large Lead Storage Batteries for Generatina Stations and Substations," was designed for conventional, rectangular-plate, lead-antimony or lead-calcium cells.
As distinguished from the conventional, rectangular-plate battery cell discharging and charging characteristics, the round cells were des! ned 9
for long periods of float with no surveillance discharges.
The long periods of float with no surveillance discharges are the typical service conditions expected in a telephone system central office standby power supply application.
However, safety-related service in nuclear power plant station batteries is characterized by relatively more frequent partial discharges of initially high current during technical specification surveillance and severe load profile deep discharges during full operation tests and station blackout events.
c.
Conclusions The team concluded that the expected service conditions for which the standard qualification program qualified the round cells appeared appropriate in terms of seismic response spectra and ambient environment, with extremes, in a nuclear plant battery room (typically mild thermal and moisture environment with negligible radiation).
Although the team determined that the qualification program complied with the regulations and industry guidance, the program did not subject the round cells to the kind of electrical service conditions in terms of actual load profiles and charging regimens that the batteries that lost capacity prematurely at several nuclear plants were subjected to during plant operation.
These actual electrical service conditions had not led j
to attributable premature loss of capacity in conventional (rectangt.lar) cell batteries.
However, it appeared that the round cells, particularly with high specific gravity, may have been more sensitive to these conditions than a standard qualification prcgram developed for
(
l 13
. ~.
conventional cells would predict.
Therefore, the team concluded that i
although the qualification program met applicable ~ requirements,. it would not have predicted the premature degradation of capacity experienced by
.some of the round cells because it did not envelope actual operating conditions of nuclear power plant applications.
3.4 Customer Communications a.
Scope
.The team. reviewed Lucent's program and practices that had been' developed to communicate with its customers regarding problems identified in the field during the operation of the batteries, b.
Observations and Findinas
- The team conducted discussions and reviewed records regarding problems identified in. field applications. The team identified that Lucent was aware of issues relative to: (1) battery racks, (2) lead sulphate crystals on positive post straps and positive plate lugs of its j
batteries, and (3) premature capacity loss in high specific gravity i
l round cells during repetitive discharging cyc k.
Battery Racks Lucent received two customer reports indicating that some battery racks were exhibiting cracks. One was a general complaint from a commercial customer and the other was from Commonwealth Edison Company's Braidwood nuclear power station.
Lucent examined the root cause for the general complaint and determined that parts molded from a specific mold were susceptible to cracking.
Lucent's supplier and manufacturer of the racks, Diversified Composites Company, (DCC), Rock Falls, Illinois, uses two different molds to L
manufacture compression polyester-glass parts for L5 type battery rack bases.. The first mold was designed to produce thicker parts with higher strength while the second mold was to produce slightly thinner parts at-lower costs.
Lucent instructed DCC to manufacture bases using only the l
first mold - Lucent informed the inspection team that it did not receive.
any further complaints regarding racks exhibiting cracks after the change.
l Commonwealth Edison Company informed Lucent that its Braidwood station i
battery racks exhibited cracks at the bases.
Lucent investigated the l-complaint and determined that Braidwood personnel incorrectly loaded the i
battery rack by placing uneven loads on the racks.
The team's review l
resulted in agreement with this position.
Battery Cell Problems Lucent received complaints from customers that Lineage
- 2000 AT&T type IS round cells exhibited lead sulphate crystals on positive post straps and positive plate lugs.
Lucent investigated this matter and determined that the cause of the lead crystals was i
mostly due to negative polarization.
Lucent took corrective action by 4
4 14
~.
l o
adding platinum to cells with high neaative polarization which eliminated the crystals.
Lucent used its product design information process to implement the change.
l Platinum solution is normally added to round cells as part of manufacturing process to reduce the negative polarization of the cells I
and to avoid lead sulphate crystals which will form on both the positive post straps and plate lugs of round cells when'the cells are placed on l
Lucent recommends that during quarterly maintenance i
checks customers should visually inspect for crystals.
Lucent personnel stated that the presence of crystals is an indication that the positive plates are not fully charged.
(
Caoacity loss ' Lucent stated that it received licensee reports indicating premature loss in high specific gravity round cell capacity when round cells were discharged repetitively during discharging cycles.
Lucent perN'ied a cause and effect analysis of the problem and determined irat when high specific gravity cells were cycled, capacity losses may be observed due to an increase of the internal resistance in the cell.
This information was transmitted to Lucent customers.
Palo Verde One of three round cell batt:,los that were manufactured in the 1992-1993 time period and shipped to the Palo Verde station exhibited capacity degradation.
The battery cells were shipped to the C&D facility in Conshohoken, Pennsylvania. During October 1996, C&D
.i research technical personnel at the Conshohoken facility examined cells from the battery cell and made the following observations:
The strap bond welds of the positive harness were extremely shallow and had a penetration of 60 mils instead of the minimum acceptance criterion of 150 mils Paste was missing on the bottom positive plate in multiple squares (pellets)
Broken pellets were found scattered throughout the element There was no copper contamination, antimony or internal shorts.
Lucent investigated the circumstances at and in conjunction with the C&D-Leola staff.
The following production anomalies were identified for the period when Palo Verde's cells were fabricated:
C&D used temporary workers Production rates increased Manual wiping of positive plates after pasting was inconsistent Turning and facing machines were out of calibration Incomplete formation process of positive plates 15
I l
Number of positive plates per holding rack was increased preventing proper curing / drying, and C0 injection was not increased to compensate for the increase the a
2 number of plates in the dryer However, the team did not find any narrative in the documents that it
- i reviewed, such as audit reports, which indicated that C&D tooK.
appropriate corrective actions to rectify the above discussed adverse conditions that'had prevailed during the manufacture of the PVNGS l
L batteries in 1992.
j The Lucent root cause' investigation concluded that AT&T's round cell battery design is capable of meeting the PVNGS technical specifications, and that C&D-Leola has taken adequate actions to correct identified
.I 4
problems; however, the root cause investigation did not discuss or address correction of manufacturing problems. lionconformance l
99901309/96-02 was identified in this area to address inadduate l-corrective action.
.j l
c.
Conclusions i
Although Lucent indicated that corrective actions were performed, the l
team could not determine the adequacy of the corrective actions taken by 1
Lucent to preclude repetition because of the lack of adequate documented corrective action narrative in the Lucent documents that would verify the adequacy ~of the corrective action performed.
3.5 10 CFR Part 21 Proaram a.
Scope The team reviewed the records and associated documents to determine the adequacy of AT&T/ Lucent. compliance with the Part 21 requirements, b.
0bservations and findinas
[
The inspectors were~ informed by the Lucent staff that itt predecessors AT&T/ Lucent predecessors had not established a procedure to implement the provisions of 10 CFP Part 21'.
Although AT&T accepted the requirements of 10 CFR Part 50, Appendix B requirements from Duke in 1990, it failed to establish a procedure to implement the provisions of 10 CFR Part 21.
During subsequent discussions with the Lucent staff, the Lucent personnel indicated that AT&T/ Lucent had not apparently recognized its responsibility to establish a procedure to implement the
^
requirements of 10 CFR Part 21.
Since AT&T/ Lucent accepted the contractual requirements of 10 CFR Part 50, Appendix B, QA requirements from Duke Power in 1990, the 10 CFR Part 21 requirements would therefore be applicable to Lucent.
Currently, Lucent does not accept any nuclear power plant requirements such as, those delineated in 10 CFR Part 50, l
Appendix B.
16 l
l
l l
During discussions with the Lucent staff, the team determined that although AT&T/ Lucent did not establish a procedure to specifically comply with the requirements of 10 CFR Part 21.
However, Lucent did have two procedures within its QA program that would meet the intent of important elements of Part 21, specifically, Procedure 92ESTFS003,
" Energy Systems Product Change Notice Process," Issue 5, dated August 4, 1994, and Procedure 92ESTFS001, " Power Systems Cars [ customer assistance request] Process," Issue 7, dated November 2, 1995.
c.
Conclusions A review of the two procedures, discussions with Lucent staff, and the review of a nuclear pcwor plant customer notification letter under Lucent Procedure 92ESTFS001 resulted in the conclusion that Lucent's customer oriented procedures should satisfactorily ensure that licensees are informed of deviations and failures to comply, as defined in 621.3 of 10 CFR Part 21.
The team determined that failure to establish a procedure to implement the provisions of 10 CFR Part 21 was a violation of NRC requirements.
However, because Lucent had two procedures regarding customer communication which were established, implemented, and appeared to meet the major intent of S21.21(b) of Part 21, the failure constitutes a violation of minor significance and will be treated as a Non-Cited Violation, consistent with Section IV of the NRC Enforcement Policy.
The Lucent staff committed to establish a procedure in accordance with 10 CFR Part 21.
3.6 Overall Quality Proaram Lucent Technologies Management has committed to an overall Total Quality Management (TQM) concept for its Dallas Power Systems operations.
The key document which delineates this TQM system is the Lucent Technologies, Microelectronics Group, Power Systems, Dallas Works Quality Manual (LQM),
106710882, Revision 12.0, October 1, 1996.
The LQM states that it was established to address the major aspects of a comprehensive quality system as defined by International Organization for Standardization (1S0)-9001, " Quality Systems - Model for Quality Assurance in Design / Development, Production, Installation and Servicing."
3.7 Lucent Ouality Proaram a.
Scope During an NRC inspection conducted at the C&D-leola manufacturing facility in January 1997, the team assessed the establishment and implementation of Lucent's control of the C&D round cell manufacturing activities, and performed a sample comparison of the requirements 17
contained in C&D manufacturing procedures with the requirements contained in Lucent's Kearny Shops (kS)-20472, " Battery," Specification.
Lucent contractually obtains " services" from the C&D-Leola facility The inspector selectively reviewea the LQM, Lucent's KS-20472 Specification and associated documents for the control of the round cell manufacturing parameters.
The Lucent staff stated that the QA and design controls at the round cell manufacturing facility were controlled and verified by uuth the Lucent staff (during audits of C&D), and by C&D's QC staff using its own 150-9001 quality assurance program (see NRC's Inspection Report 99901305/97-01, which documents the January 1997 inspection at the C&D-Leola facility).
b.
Observations and Findinas The in nectors determined that the LQM appeared to ba established to addre s the elements of the 150-9001 Standard that would be important to the control of the round cells such as, Sections 4.0, " Design Control,"
6.0, " Purchasing," and 7.0, " Control of Customer-Supplied Products." A review of Lucent's KS-20472 battery specification determined that it was established to delineate AT&T's round cell design requirements.
As discussed in Section 3.2 above, the NRC inspectors determined that i
although Lucent contractually accepted the requirements of 10 CFR Part 50, Appe. dix B and 10 CFR Part 21 from Duke's McGuire station in n
1990, Lucent aid not establish an Appendix B or Part 21 program.
Instead, Lucent utilized its 150-9001 quality program that was established for the AT&T Microelectronics-Dallas Works facility.
The team determined that both the Lucent and C&D QA program were commercial programs that are not applicable to NRC regulations; consequently, Lucent's round cells are commercial grade batteries that require dedication, as defined in 621.3 of Part 21, prior to use in nuclear safety-related applications.
c.
Conclusions The team concluded that Lucent's KS-20472 specification and associated documents appeared comprehens N and appropriate to control C&D's manufacturing activities of m ruund cells. The team also determined that the AT&T/ Lucent staff performed QA program and technical audits of r
the C&D manufacturing facility, as well as having a resident inspector at C&D's facility.
The NRC inspectors determined that although Lucent contractually accepted the requirements of 10 CFR Part 50, Appendix B and 10 CFR Part 21 from Duke's McGuire station in 1990, AT&T/ Lucent did not establish an Appendix B or Part 21 program.
Instead, Lucent utilized its 150-9001 quality program that was established for the AT&T Microelectronics-r CAD provides contractual services, as discussed in Criterion VII of 10 CFR Part 50, Appendf x s, for Lucent by manufacturing Lucent's Lineage 2000 round cett batteries.
18 l
Dallas Works facility. The team determined that both the Lucent and C&D quality assurance programs are commercial QA programs that are not applicable to NRC regulations.
3.8 Round Cell Battery Racks a.
Scope To assess Lucent's control on the manufacture of the racks for its round cell batteries.
b.
Observations and findinas The Diversified Composites Company, (previously known as Roxite Company) manufactures and supplies battery racks for Lucent's round cells.
Lucent furnishes DCC with the necessary material specifications for the raw material and the dimensional drawings of the round cell battery racks.
DCC purchases the raw plastic material to meet Lucent's specifications and, using a compression molding process, converts the raw plastic material into battery racks to meet Lucent's drawings.
DCC tests the racks at the factory by loadiag them with approximately twice 4
the weight of the batteries for a speci": neriod of time.
Lucent's Supplier Quality Services (SQS), which is headquartered in Princeton, New Jersey, is the designated organ'ization to perform product inspection, quality audits and surveillance at DCC when battery racks are manufactured.
DCC contacts Lucent's SQS representatives and informs them of manufacture and tests scheduled on battery racks to enable them to plan in-process inspections or witness selected tests.
c.
Conclusion The team did not identify any problems in this area.
3.9 Review of Licensee Purchase Orders The Team reviewed each of the five licensee purchase order (P0s) files to determine the scope of the round cell procurement and the imposed contractual requirements. The team determined that Duke was the only licensee to impose 10 CFR Part 50, Appendix B and 10 CFR Part 21 requirements directly on AT&T/ Lucent. As a result of its review, the team did not identify any concerns in this area.
I 3.10 Entrance and Exit Meetinas In the entrance meeting on November 18, 1996, the NRC Team Leader discussed the scope of the inspection, outlined the areas to be inspected, and established interfaces with Lucent management.
In the exit meeting on November 20, 1996, the inspectors discussed.their findings, concerns and plans for the follow-up inspection at the C&D round cell manufacturing facility in Leola, Pennsylvania.
t l
19
_ -. ~
I l
In the subsequent NRC inspection exit meeting at C&D's Leola facility on January 9, 1997, the team discussed its concerns relative to C&D's implementation of its manufacturing controls related to Lucent's round cell requirements.
i 4
PERSONS CONTACTED l
Lucent Technolooies Microelectronics. Dallas. Texas Staff E.B. Kahn, Director, Business Operations A.J. Donnell, Quality Director F.C. Laman, Member, Technical Staff K.A. Murugesamoorthi, Portfolio Manager S.T. Wakim, Manager, Quality Assurance i
i l
Lucent Technolooies-Bell Laboratories. Murray Hill. New Jersey Staff M.C. Weeks, Member, fechnical Staff J.B. Baldasty, Member, Technical Staff i
I t
20 i
-. ~..-_.,
___.~. ~ n - _.-
a e to 9
)
/
Af G ATIVE POST i
g Pott ff VE POSI N
w 4
g j
VENT FUNNEL
,q SOLID PLUG l
1
./.
g m,g.,,.
.s g av 6(
j
$EA AREA
, ;v FLECTROLYTE LEVEL LINES POST SEALS
^;s
_z i
H ARNE S$
[
8
'N j
PLAST 6C CAP t
g f
3
.0 PostTIVE PL ATE t
t
[
TAaSOND i
}
JA R POU1 tv E PL ATE NEGAt LATE fet0ATIVfPLATE GLASS M AT j;
SEPARATOR k'
pos1TtvE PLA TE eAss g
L E AD PLUG MUT CEMIEM POUR COPPER ROO KS-20472 Round Cell (Cutaway View)
Figure 1
!