ML20137A219
| ML20137A219 | |
| Person / Time | |
|---|---|
| Issue date: | 03/31/1997 |
| From: | Office of Nuclear Reactor Regulation |
| To: | |
| References | |
| NUREG-0040, NUREG-0040-V20-N04, NUREG-40, NUREG-40-V20-N4, NUDOCS 9703200250 | |
| Download: ML20137A219 (154) | |
Text
-
NUREG-0040 Vol. 20, No. 4 l Licensee Contractor j and Vendor Inspection l Status Report i
Quarterly Report October - December 1996 i
l U.S. Nuclear Regulatory Commission l
Omce of Nuclear Reactor Regulation i
J gnRigo
{
9
'o,%
I i
%,...../
)
g7 4 llllEEE!E%%ElElIE 97032CO250 970331 O
O PDR
AVAILABILITY NOTICE Availability of Reference Materials Cited in NRC Publications Most documents cited in NRC publications will be available from one of the f ollowing sources:
1.
The NRC Public Document Room, 2120 L Street, NW., Lower Level, Washington, DC 20555-0001 2.
The Superintendent of Documents, U.S. Government Printing Office, P. O. Box 37082, Washington, DC 20402-9328 3.
The National Technical Information Service, Springfield, VA 22161-0002 Although the listing that follows represents the majority of documents cited in NRC publica-tio?s, it is not intended to be exhaustive.
Referenced documents available for inspection and copying for a fee from the NRC Public Document Room include NRC correspondence and internal NRC memoranda: NRC bulletins, circulars, information notices, inspection and investigation notices; licensee event reports; vendor reports and correspondence; Commission papers; and applicant and licensee docu-ments and correspondence.
The following documents in the NUREG series are available for purchase from the Government Printing Office: formal NRC staff and contractor reports, NRC-sponsored conference pro-ceedings, international agreement reports, grantee reports, and NRC booklets and bro-chures. Also available are regulatory guides, NRC regulations in the Code o/ Feceral Regula-tions, and Nuclear Regulatory Commission Issuances.
Documents available from the National Technical information Service include NUREG-series reports and technical reports prepared by other Federal agencies and reports prepared by the Atomic Energy Commission, forerunner agency to the Nucleaf Regulatory Commission.
Documents available from public and special technical libraries include all open literature items, such as books, journal articles, and transactions. Federal Register notices, Federal and State legislation, and congressional reports can usually be obtained from these libraries.
Documents such as theses, dissertations, foreign reports and translations, and non-NRC con-ference proceedings are available for purchase from the organization sponsoring the publica-tion cited.
Single copies of NRC draft reports are available free, to the extent of supply, upon written request to the Office of Administration, Distribution and Mail Services Section, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001.
Copies of industry codes and standards used in a substantive manner in the NRC regulatory process are maintained at the NRC Library, Two White Flint North,11545 Rockville Pike, Rock-ville, MD 20852-2738, for use by the public. Codes and standards are usually copyrighted and may be purchased from the originating organization or, if they are American National Standards, from the American National Standards institute,1430 Broadway, New York, NY 10013-3308.
A year's subscription of this report consists of four quarterly issues.
I t
l 9
NUREG-0040 Vol. 20, No. 4 i
Licensee Contractor and Vendor Inspection Status Report 4
Quarterly Report October - December 1996 Manuscript Completed: March 1997 Date Published: March 1997 1
Division ofInspection and Support Programs Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission
}
Washington, DC 20555-0001 i
i I
t 4
ABSTRACT This periodical covers the results of inspections performed by the NRC's 4
Special Inspection Branch, Vendor Inspection Section, that have been distributed to the inspected organizations during the period from October 1996 through December 1996.
o i
4
}
a i
l 1
i I
111 i
i 4
CONTENTS E8E Abstract................................................................. iii Introduction............................................................. vii Inspection Reports.......................................................
1 l-ABB-Combustion Engineering (99901306/96-01).........
2 Windsor, CT C&D Charter Power Systems Incorporated (99901304/96-01).........
14 Conshohoken, PA Dragon Valves Incorporated (99900264/96-01)......... 40 Norwalk, CA Framatome Cogema Fuels (99900001/96-01).........
53 Lynchburg, VA Panalarm Business Unit (99901303/96-01).........
94 Skokie, IL Westinghouse Electric Corporation (99901307/96-01)........
111 Energy Systems Business Unit Pittsburgh, PA Zetec, Incorporated (99901037/96-01)........
128 Issaquah, WA Select Generic Correspondence on the Adequacy of Vendor.................
145 Audits and the Quality of Vendor Products V
INTRODUCTION A fundamental premise of the U. S. Nuclear Regulatory Commission (NRC) licensing and inspection program is that licensees are responsible for the proper construction and safe and efficient operation of their nuclear power plants.
The Federal government and nuclear industry have established a system for the inspection of commercial nuclear facilities to provide for multiple levels of inspection and verification.
Each licensee, contractor, and ve,ndor participates in a quality verification process in compliance with requirements prescribed by the NRC's rules and regulations (Title 10 of the Code of Federal Regulatfons). The NRC does inspections to oversee the commercial nuclear industry to determine whether its requirements are being met by licensees and their contractors, while the major inspection effort is performed by the industry within the framework of quality verification programs.
The licensee is responsible for developing and maintaining a detailed quality assurance (QA) plan with implementing procedures pursuant to 10 CFR Part 50.
Through a system of planned and periodic audits and inspections, the licensee is responsible for ensuring that suppliers, contractors and vendors also have suitable and appropriate quality programs that meet NRC requirements, guides, codes, and standards.
The Vendor Inspection Section (VIS) of the Special Inspection Branch reviews and inspects nuclear steam system suppliers (NSSSs), architect engineering (AE) firms, suppliers of products and services, independent testing laboratories performing equipment qualification tests, and holders of NRC construction permits and operating licenses in vendor-related areas.
These inspections are done to ensure that the root causes of reported vendor-related problems are determined and appropriate corrective actions are developed. The insr"ctions also review vendors to verify conformance with applicable NRC and industry quality requirements, to verify oversight of their vendors, and coordination between licensees and vendors.
The VIS does inspections to verify the quality and suitability of vendor products, licensee-vendor interface, environmental qualification of equipment, and review of equipment problems found during operation and their corrective action. When nonconformances with NRC requirements and regulations are found, i
the' inspected organization is required to take appropriate corrective action and to institute preventive measures to preclude recurrence. When generic implications are found, NRC ensures that affected licensees are informed through vendor reporting or by NRC generic correspondence such as information notices and bulletins.
vii
l This quarterly report contains copies of all vendor inspection reports issued during the calendar quarter for which it is published.
Each vendor inspection report lists the nuclear facilities inspected.
This information will also alert affected regional offices to any significant problem areas that may require special attention. Appendices list selected bulletins, generic letters, and information notices, and include copies of other pertinent correspondence involving vendor issues.
I viii
INSPECTION REPORTS 1
scoq k
UNITED STATES y-NUCLEAR REGULATORY COMMISSION f
WASHINGTON, D.C. 20555 0001 I
l
's,***** $
December 4, 1996 l
Michael F. Barnoski President, Nuclear Operations l
ABB-Combustion Engineering 2000 Day Hill Road Windsor, CT 06095-0500
SUBJECT:
NRC INSPECTION REPORT 99901306/96-01 4
Dear Mr. Barnoski:
On October 31, 1996, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at your ABB-Combustion Engineering facility.
The enclosed report presents the results of that inspection.
The inspection was conducted to ascertain specific attributes and implementation of your quality assurance (QA) program, and whether licensees effectively monitored your control of quality for safety-related instrumentation and control (!&C) systems and associated spare parts purchased by licensees for nuclear power plant.,.
We assessed your commercial-grade dedication activities, screening of issues for Part 21 applicability, and your monitoring of the control of quality by your subvendors.
During.this inspection, the NRC inspectors determined that the implementation of your quality assurance program did not meet certain NRC requirements imposed on you by your' customers.
Specifically, you failed to implement your documented QA instructions tc consider licensee findings concerning the quality performance of your subvendor, as required by 10 CFR Part 50, Appendix B.
This nonconformance is cited in the enclosed Notice of Nonconformance (NON),
and the circumstances surrounding it are described in detail in the enclosed report.
Please respond to the nonconformance and follow the instructions specified in the enclosed NON when preparing your response.
In addition, the inspectors determined that licensee monitoring of your quality assurance program for I&C systems and components was not always adequate.
Licensees did not monitor your review of subvendor commercial-grade dedication activities and your screening of issues for Part 21 applicability.
2
l M. Barnoski In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its enclosures will be placed in the NRC's Public Document Room.
Sincerely, J
iRobert M.
1
- Gallo, ef Special Inspection Branch Div uion of Inspection and Support Programs Office of Nuclear Reactor Regulation Docket No. 99901306
Enclosures:
1.
Notice of Nonconformance 2.
Inspection Report 99901306/96-01 cc:
See next page 3
cc:
Mr. Ronald Fitzgerald Director, Quality Assurance ABB-Combustion Engineering 2000 Day Hill Road Windsor, CT 06095-0500 Mr. Michael P. Lilley Manager, Quality Assurance Rochester Gas a Electric Corporation 89 East Avenue Rochester, NY I4649 Mr. Richard H. Fassler Supervisor, Quality Assurance Niagara Mohawk Power Corporation Nine Mile Point Nuclear Station P.O. Box 63 Lycoming, NY 13093 Mr. Wallace Woodard Principal Engineer, Florida Power & Light Company P.O. Box 14000 Juno Beach, FL 33408-0420 Mr. Robert Ripley Audit Team Leader Union Electric Company 1901 Chouteau Avenue P.O. Box 149 St. Louis, MO 63166 Mr. Ronald Casavant Technical Specialist Washington Public Power Supply System P.O. Box 968 Richland, WA 99352 Mr. James Young Section Leade Arizona Public Service Company P.O. Box 52034 Phoenix, AZ 85072-2034 Mr. Derek Mercurio Licensing Southern Califo. iia Edison Company 2244 Walnut Grove Avenue Rosemead, CA 91770 Mr. Joel Rachal Procurement Engineer Entergy Operations, Inc.
P.O. Box 31995 Jackson, MS 39286-1995 4
NOTICE OF NONCONFORMANCE ABB-Combustion Engineering (ABB-CE)
Docket No.: 99901306 Windsor, Connecticut On the basis of an inspection by the staff of the U.S. Nuclear Regulatory Commission (NRC) on September 9 through 12, 1996, and October 28 through 31, 1996, it appears that the following activity was not conducted in accordance with NRC requirements:
Criterion V of Appendix B to 10 CFR Part 50, " Instructions, Procedures, and Drawings," requires, in part, that activities affecting quality shall be prescribed by documented instructions and procedures, and accomplished in accordance with these instructions and procedures.
ABB-CE's Quality Assurance Manual 100, System 7.0, Section 2.5.4,
" Control of Purchased Items and Services," Revision 2, requires, in part, that the assessment of the overall quality performance of suppliers of safety-related items shall include consideration of reports of audits from other sources (i.e., customers, NRC, other ABB nuclear groups).
Contrary to the above requirements, ABB-CE failed to consider licensee findings regarding commercial-grade dedication activities of its subvendor ABB-Electro Mechanics (ABB-EM).
Two licensees, Rochester Gas
& Electric Company and Niagara Mohawk Power Corporat' ion, identified problems with ABB-EM's commercial-grade dedication of safety-related instrumentation components. ABB-CE's lack of consideration of licensee findings precluded it from assessing whether safety-related items it purchased from ABB-EM were compromised (99901306/96-01-01).
Please send a t..itten statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555-0001, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date of the letter transmitting this Notice of Nonconformance.
Your reply should be clearly marked as a " Reply to a Notice of Nonconformance" and should contain for the nonconformance (1) a description of steps that have been or will be taken to correct these items, (2) a description of steps that have been or will be taken to prevent recurrence of these items, and (3) the dates your corrective actions and preventive measures were or will be completed.
Dated at Rockville, Maryland this 4th day of December, 1996 5
)
i U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No:
99901306/96-01 Organization:
ABB-Combuttion Engineering
Contact:
Ronald J. Fitzgerald, Director Quality Assurance 860/285-9816 Nuclear' Industry Instrumentation and control systems and Activity:
associated spare and replacement parts i
Dates:
September 9-12 and October 28-31, 1996 Inspectors:
Anil S. Gautam, Senior Engineer Frank Gee, Electrical Engineer Approved by:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs 6
4 1
INSPECTION
SUMMARY
During this inspection, the NRC inspectors reviewed activities associated with implementation of selected portions of ABB-Combustion Engineering's (ABB-CE's) quality assurance (QA) program and licensee monitoring of ABB-CE's control of quality.
The inspection bases were as follows:
Appendix B, quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50).
10 CFR Part 21, " Reporting of Defects and Noncompliance."
NRC Regulatory Guide 1.144, " Auditing of Quality Assurance Programs for Nuclear Power Plants."
ABB-CE's Quality Assurance Manual (QAM) 100, Revision 4, dated May 1, 1996, and associated implementing procedures.
The inspectors noted one instance in which ABB-CE failed to conform to NRC requirements imposed upon it by. NRC licensees.
This nonconformance is discussed in Section 3.1 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS Nonconformance 999005.3_S/95-02-01 (CLOSED 1 Contrary to Criterion V of Appendix B to 10 CFR Part 50, and paragraph D.2 of ABB-CE Nuclear Operations " Administrative Procedure for Reporting Defects and Noncompliance," dated February 28, 1996, ABB-CE did not document its screening of potential Part 21 issues.
ABB-CE responded to the notice of nonconformance (NON) by letter of April 26, 1996, stating that it would make appropriate changes to'its Part 21 implementing procedure.
ABB-CE " Procedure for Evaluating and Reporting of Defects and Noncompliance Pursuant to 10 CFR Part 21," dated. September 30, 1996, addressed the process of screening and reporting of potential Part 21' issues by all Nuclear Systems and Nuclear Operations employees.
The procedure requires each deviation or potential failure to comply to be evaluated, the evaluation documented, and the disposition recorded.
The inspectors found the revised procedure acceptable.
3 INSPECTION FINDINGS AND OTHER COMMENTS 3.1 Quality Assurance Proaram
- a. Insoection Scope The inspectors examined ABB-CE's QA program, policy, implementing procedures, conformance to procurement documents, corrective actions in 2
7 I
l
^-
-^
response to licensee audit findings, commercial-grade item dedication, Part 21 evaluations, and monitoring of subvendors.
- b. Observations and Findinos ABB-CE's nuclear operations comprised five business units, including Total Quality, Engineering, Field Services, Fuel Operations, and Technology Development.
Each business unit was monitored by quality control (QC) inspector::.
The QA staff comprised the QA director and four QA managers.
The QA director reported to the Vice President of Tote Quality who reported to the President of Nuclear Operations.
The i
inspector observed that QAM-100 System 1, Figure 1.2, indicated that the QC inspectors reported to the management of the individual business units rather than to QA.
The QA management stated that irrespective of the i
QAM-100 chart, the QC inspectors were monitored by QA.
The inspector witaessed, in part, testing of a safety-related core cooling monitoring system for Millstone Unit 2 and observed equipment being subjected to various inputs to validate system performance and conformance to licensee procurement specifications.
No concerns were identified.
The inspector evaluated ABB-CE's audit reports EM-9605 and EM-9606, dated August 28, 1996, of its subvendor ABB-EM and determined that the audit reports did not provide adequate objective evidence of quality furnished by the subvendor.
For example, the reports did not address compliance with Appendix B criteria, internal audit findings and corrective actions, negative observations or weaknesses, and any deficiencies identified by other customers.
ABB-CE's QAM-100, System 7.0, Section 2.5.4, " Control of Purchased Items and Services," Revision 2, requires, in part, that the i
assessment of the overall quality performance of its suppliers of safety-related items shall include consideration of reports of audits from other sources (i.e., customers, the U.S. Nuclear regulatory Commission (NRC),
j other ABB nuclear groups). During 1995-1996, Rochester Gas & Electric Company (RG&E) and Niagara Mohawk Power (NMP) Corporation, purchased instrumentation directly from ABB-EM and identified problems with ABB-EM's commercial-grade dedication of safety-related instrumentation components.
Licensee findings included ABB-EH's failure to identify or test critical characteristics of test jacks, diodes, and other printed circuit board (PCB) components.
ABB-CE's lack of consideration of licensee.".adings precluded it from assessing whether ABB-EM provided reasonable assurance that dedicated items will perform their intended safety function.
This practice constitutes Nonconformance 99901306/96-01-01.
The QA manager stated that the ABB-CE audit reports of ABB-EM were not typical and that QA planned to include additional details in its future audit reports.
During 199' 1996, ABB-CE purchased scfety-related instrumentation from ABB-EM for sale to licensees. ABB-EM purchased these items as I
commercial-grade and dedicated them for safety-related applications.
During the NRC inspection on September 9-12, 1996, ABB-CE could not provide the inspector sufficient documentation to demonstrate that 3
8
selected items were qualified to perform their safety functions.
During the-inspection on October 28-31, 1996, two inspectors visited the ABB-CE (Windsor, Connecticut) and ABB-EM (New Britain, Connecticut) facilities to further assess commercial-grade dedication of selected safety-related instrumentation. The inspectors assessed selected purchase orders (P0s),
procedures, licensee surveillance, design drawings and documentation pertinent to commercial-grade dedication activities.
The following ABB-CE P0s to ABB-EM and associated items were examined:
f11 ITEMS 404676 PCB 406610 Relay-406260 Relay and operational amplifier 405516 Switch 404700 Variable set-point card 404489 Test jack 405651 PCB 405842 Card assembly for combinational circuit 9607145 Fiber optic switch 9405001 Repair of bistable trip unit 9503180 Diagnostic testing of setpoint card Upon the basis of commercial-grade dedication documents, the inspectors determined that ABB-CE and ABB-EM reviewed safety functions, identified appropriate critical characteristics, and performed tests and analyses to dedicate the aforementioned items.
No concerns were identified.
The inspector reviewed Part 21 reports issued by ABB-CE for design errors or failures during the past 3 years.
The inspector observed that nonconformance reports (NCRs) did not address Part 21 screening, applicability, and reportability, and there was no mandatory training for the staff on Part 21 programmatic activities.
This concern was also identified in a previous inspection (see Inspection Report 99900538/95-02).
In response to these concerns, ABB-CE revised its Part 21 implementing procedure to address documentation of screening on QA forms (including noncompliance forms), and self-study QA training for employees (see Section 2 of this report).
The inspector examined a summary of 1993-1996 NCRs concerning instrumentation and observed no trending of failures by QA to evaluate root causes in order to prevent recurrence of failures.
The QA director provided evidence that an initiative was in place to improve trending of failures and expected it to be implemented by about January 1997.
The inspector observed that there was no sharing of Part 21 information, j
subvendor audits, or hardware problems among ABB-CE and other ABB companies.
For example, ABB-CE purchased items from ABB-Transmission and Distribution (ABB-T&D) but was not asare of failures of type MG-6 electrically reset latching relay.y (documented in NRC Report 50-280/96-05 and 50-281/96-05, dated July 15, 1996). The QA manager stated that ABB-CE had not purchased the MG-6 relay, but that ABB-CE planned to expand the scope of its questions during subvendor audits.
4 9
3 The inspector observed two unmarked boxes in the holding area for safety-related items and asked whether the boxes contained any non-safety parts 4
that could compromise the ::egregation of safety-related parts. The QC inspector examined the boxes and confirmed that the shipping boxes contained safety-related parts and that the storage area was controlled by receipt inspection and restricted access,
- c. Conclusions The inspectors concluded that, in general, the QA manual, work instructions, and procedures were adequate, except for the nonconformance and screening of potential Part 21 issues described herein.
3.2 Review of Monitorina of ABB-CE by Licensees
- a. Insoection Scone The inspector evaluated licensee monitoring of ABB-CE's control of quality for safety-related items purchased by licensees, including audits and surveillances of ABB-CE's commercial-grade dedication, Part 21 reports, and monitoring of subvendors.
- b. Observations and Findinas The inspector contacted the Nuclear Utilities Procurement Issues Committee (NUPIC) audit team leaJers and selected licensees to discuss scope and findings of the audits and surveillance.
NUPIC audited ABB-CE in October 1993 and October 1995.
Utilities represented on the NUPIC audit teams included Arizona Public Service (APS) Company, Baltimore Gas & Electric Company, Consolidated Edison Company, Consumers Power Company, Entergy Operations (E0), Florida Power and Light (FP&L) Company, Illinois Power Company, Northeast Utilities, j
Pennsylvania Power & Light Company, Rochester Gas & Electric Corporation (RG&E), Union Electric Company, Virginia Electric & Power Company, Washington Public Power Supply System, and Yankee Atomic Electric
- Company, NUPIC's 1993 audit identified a finding concerning ABB-CE's failure to identify critical characteristic: of certain commercial-grade items during ABB-CE's dedication of instrumentation and enntrol items.
In response to this finding, ABB-CE developed procedure No. 00000-ICE-0509,
" Technical Guidelines for Commercial-Grade Dedication," to require identification of critical characteristics and specify form, fit, and function of dedicated items. The NUPIC team also identified a finding concerning ABB-CE's failure to use an approved vendor for calibration of measuring and test equipment (M&TE).
The NUPIC audit team accepted ABB-l CE's corrective actions.
NUPIC's October 1995 audit identified findings concerning deficiencies in ABB-CE's monitoring of subvendors, design calculation packages, and control of M&TE. ABB-CE's monitoring of its subvendor MTS Systems Corporation (supplier of calibration services) was deficient in that ABB-CE did not verify whether MTS implemented its out-of-tolerance-5 10
f notification program, established integrity of traceability documentation, or performed internal audits. NUPIC accepted ABB-CE's corrective actions.
APS audited ABB-CE in March 1995.
The APS audit identified several weaknesses in ABB-CE's implementation of its design control program including documenting design assumptions, identifying design interfaces between ABB-CE and APS engineering, verifying and identifying software, controlling computer output documentation, and maintaining QA records in Chattanooga, Tennessee.
ABB-CE is in the process of resolving the findings.
FP&L audited ABB-CE in June 1995.
The FP&L audit identified findings concerning ABB-CE's failure to monitor the adequacy of the design control activities conducted by its subvendor, and ABB-CE's basing its engineering judgments on unverified design inputs.
FP&L accepted ABB-CE's corrective actions for these findings.
Based on FP&L surveillance reports 08.06.CEPSG.94.1, 95.1, 95.5, and 96.1, FP&L conducted effective surveillances of ABB-CE and ABB-EM.
Activities monitored included commercial-grade dedication activities (review of critical characteristics and verification), nonconformances, and M&TE calibration data.
Discussions with E0 and Southern California Edison procurement staff determined that they relied on NUPIC's audit for monitoring ABB-CE and did not conduct any surveillance of ABB-EM.
The inspector determined that NUPIC and the licensees did not audit the subvendor ABB-EM.
Further, NUPIC and licensee's audits of ABB-CE did not identify that ABB-CE performed inadequate audits of ABB-EM (see section 3.1.b of this report).
The inspectors observed that licenseas, in general, did not identify specific safety functions of spare parts purchased from ABB-CE.
P0s identified spare part numbers and pertinent systems, not pertinent instrumentation circuits or parameters monitored. ABB-CE stated that they did not ask licensees to identify specific safety functions, rather qualified the spare parts to the original design of the system.
Lack of identification of specific safety functior.s by licensees could result in inadvertent misapplication of procured items.
c.
Conclusions In general, licensees audited ABB-CE in accordance with proper criteria, procedures, and checklists.
Licensee monitoring of ABB-CE's QA program for instrumentation components was not always adequate: licensee audits did Mt ensure that ABB-CE was contrciling the quality of their purchased items from ABB-EM.
6 11
i' 1
3.3 Entrance and Exit Meetinas i
In the entrance meeting on September 9, 1996, the NRC inspector discussed the scope of the inspection, outlined the areas to be inspected,_and established interactions with ABB-CE management.
In the exit meetings on September 12 and October 31, 1996, the inspectors discussed their findings and observations.
4 PER5000lEL CONTACTED AB8-CE Don Allen, Vice' President,_ Total Quality Ron Fitzgerald, Director, QA J
Mark Stewart, QA Gary Bloonquist, Manager, Qu.lity Programs and Procedures Richard Bradshaw, Manager, Instrumentation & Controls Engineering (ICEE) j Robert Driscoll, Supplier Audits 1
Edward Stepaneck, QC Inspector, ICEE Assurance Ian Richard, Manager, Operations Licensing Charles Molnar, Licensing Engineer 1
Paul Rohan, Total Quality Mark Stewart, Supervisor, Spare parts QC Kenneth Tomany, ICEE QC Marty Ryan, ICEE Larry Bryan, Procurement Engineer, Nuclear Spare parts ABB-Electro Mechanics i
William Hadovski, Engineering Manager Alex Oja, Contract Manager William Wayland, Supervisor, QA Engineering Edward Rollins, Plant Manager K.Parekh, Senior Engineer M.J. Merlini, Contract Administrator 1
l Licensees (contacted by telephone)
Mike Li. ley, QA Manager, RG&E Company l
Wall' ace Woodard, Principal Engineer, FP&L Company Olga Hanek, Li u nsing Engineer, FP&L Company George Kuhn, Procurement Engineer, FP&L Company Robert Ripley,1993 NUPIC Team Leader, Union Electric Company Ron Casavant, 1995 NUPIC Team Leader, Washington Public Power Supply System Derek Mercurio, Licensing, Southern California Edison Company Joel Rachal, Procurement Engineer, EO 7
12
~.
ITEMS OPENED, CLOSED, AND DISCUSSED Opened 99901306/96-01-01 NON Failure to assess subvendor's control of quality Closed 99900538/95-02-01 NON Inadecuate Part 21 screening process by ABB-CE g..
e e
N I
i l
8 13
1 p3 Rf Cp
[
ft UNITED STATES
]
L
[
j NUCLEAR REGULATORY COMMISSION l
t WASHINGTON, D.C. 20550 0001
% *..+ f l
December 17, 1996 I:
Dr. Leslie S. Holden Vice President Technology C&D Charter Power Systems, Inc.
Washington & Cherry Streets i
Conshohocken, PA 19428 4
~
SUBJECT:
NRC INSPECTION REPORT 99901304/96-01
Dear Dr. Holden:
On October 9,.1996, thy U.S. Nuclear Regulatory Commission (NRC) completed an inspection at the C&D* Charter Power Systems, Inc. (C&D) facilities at Attica, Indiana and Conshohocken, Pennsylvania, regarding your lead-acid battery cells that are manufactured and supplied to nuclear power plant facilities for Class lE electrical u r ty-related applications. The enclosed e
report presents the results of the inspection.
]
During this inspection, the NRC inspectors determined that certain activities appeared to be in violation of NRC requirements. _ Specifically, although C&D l
staff stated that your batteries were dedicated at the end of manufacturing r
and assembly activities, C&D failed to document the dedication methodology and basis. This violation is cited in the enclosed Notice of Violation (NOV), and the circumstances surrounding the violation are described in detail in the enclosed report.
Please note that C&D is required to respond to this letter and should follow the instructions specified in the enclosed NOV when preparing its response.
The NRC will use the C&D response, in part, to determine whether further enforcement action is necessary to ensure compliance with regulatory requirements.
In addition, the NRC team determined that the implementation of the C&D
_ quality assurance program failed to meet certain NRC requirements imposed on C&D by your customers.
The team found certain weaknesses in C&D's quality program applied to the manufacture of batteries intended for installation in nuclear power plant Class IE electrical systems. The team observed that, even though w-itten procedures and instructions had been established to delineate 4
manufacturing process controls at the Attica facility, craftsmen were typically not provided nor made aware of these procedures and instructions.
For example Nonconformance 99901304/96-01-05 revealed that C&D allowed a manufacturing procedural requirement to be performed differently than specified.
i This failure of C&D to comply with its quality system program may have i
contributed to the degraded battery ct.'ls identified at the Hatch nuclear plant. The team also identified that seismic analyses were performed without the benefit of written procedures or instructions.
C&D is a registered trademark of C&D Charter Power Systems, Inc.
14
Dr. L.S. Hold:n These nonconformances are cited in the enclosed Notice of Nonconformance (NON) and C&D should follow the instructions specified in the enclosed NON when 4
preparing your response.
The team concluded that your batteries provided for licensee safety-related applications may not have specifically met the licensee's procurement 4
requirements because C&D did not establish or implement a battery cell dedication program to ensure that each predetermined critical characteristic was verified under a 10 CFR Part 50 Appendix B quality program. Additionally, the team determined that utility company auditors did not adequately evaluate C&D's quality system program implementation because C&D characterized its battery cell manufacturing activities as commercial grade to these auditors.
In accordance with 10 CFR 2.790 of the NRC " Rules of Practice," a copy of this j
letter and enclosures will be placed in the NRC's Public Document Room (PDR).
4
)
Sincerely, 1
Griginal signed by i
Robert M. Gallo, Chief i
Special Inspection Branch i
Division of Inspecti.on and Support Programs Office of Nuclear Reactor Regulation Docket No.:
99901304/96-01 i
Enclosures:
1.
2.
Notice of Nonconformance
{
3.
Inspection Report 99901304/96-01 cc:
Mr. T.J. Kinden, Director Quality Assurance Mr. C. Dale Brown C&D Charter Power Systems, Inc.
Plant Manager j
Washington & Cherry Streets C&D Powercom Conshohocken, PA 19428 200 West Main Street Post Office Box 279 Attica, IN 47918 j
4 1
b i
f 3
r 15 m...
NOTICE OF VIOLATION C&D Charter Power Systems, Incorporated Docket No.: 99901304 Blue Bell, Pennsylvania During an NRC inspection conducted on September 16 through 18 and October 7 through 9,1996, a violation of NRC requirements was identified.
In accordance with the " General Statement of Policy and Procedure for NRC Enforcement Actions," NUREG-1600, the violation is listed below:
l l
Section 21.21, " Notification of failure to comply or existence of a defect and its evaluation," of 10 CFR Part 21 requires, in part, that a dedicating entity is responsible for maintaining auditable records for the dedication process.
Further, Section 21.3, " Definitions," of 10 CFR Part 21 requires, in part, that in all cases, the dedication process must be conducted in accordance with the applicable provisions of 10 CFR Part 50 Appendix B.
Contrary to the above, C&D did not establish and maintain auditable records for the dedication process used to dedicate battery cells destined for Class IE safety-related station battery applications. Adequate objective evidence was not provided to indicate that C&D's battery cell dedication process was being conducted in accordance with the applicable provisions of Appendix B to 10 CFR Part 50.
(99901304/96-01-01)
This'is a Severity Level IV violation (Supplement VII).
Pursuant to the provisions of 10 CFR 2.201, C&D Charter Power Systems, Incorporated, is hereby required to submit a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN:
Document Control Desk, Washington D.C. 20555-0001, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date of the letter transmitting this Notice 4
of Violation. This reply should be clearly. marked as a'" Reply to a Notice of
)
Violation" and should include for each violation:
(1) the reason for the violation, or, if contested, the basis for disputing the violation, (2) the corrective steps that have been taken and the results achieved, (3) the corrective steps that will be taken to avoid further violations, and (4) the date when fuli compliance will be achieved.
Your response may reference or include previous docketed correspondence, if the correspondence adequately addresses the required response. Where good cause is shown, consideration will be given to extending the response time.
Dated at Rockville, Maryland this 17th day of December, 1996 1
16
NOTICE OF NONCONFORMANCE C&D Charter Power Systems, Incorporated Docket No.: 99901304 Blue Bell, Pennsylvania Based on the results of an inspection conducted September 16 through 18 and October 7 through 9,1996, it appears that certain of your activities were not conducted in accordance with the requirements of the U. S. Nuclear Regulatory Commission.
A.
Criterion V, " Instructions, Procedures, and Drawings," of Appendix B to 10 CFR Part 50 states that, activities affecting quality shall be prescribed by documented instructions, procedures, or drawings, of a o
type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings.
Instructions, procedures, or drawings shall include appropriate quantitative or qualitative acceptance criteria for determining that important activities have been satisfactorily accomplished.
Section 3.0, " Responsibilities and Authorities," of C&D Procedure 01-003.6, " Document and Data Control (Corporate)," January 1996, states in part, that department managers and supervisors are responsible to assure that the correct documents are available at the locations where i
needed and that they are being used.
1 A.1 Contrary to the above, assembly line and manufacturing personnel in several areas at the Attica, Indiana facility were not aware of any procedures or instructions to control the work activities which they were performing. The Attica facility supervisors and managers did not ensure that procedures or instructions were being used in areas such as plate pasting, pasted plate curing, cell assembly, post sealing and cell cover gluing.
(99901304/96-01-02) l A.2 Contrary to the above, C&D performed its site specific seismic analyses without the benefit of written procedures.
(99901304/96-01-03)
A.3 Contrary to the above, although C&D stated that it was performing battery cell dedication for customers who imposed 10 CFR Part 50 Appendix B, C&D did not establish a dedication procedure to identify and verify the adequacy of each of the critical characteristics important to ensure that the battery cells would perform satisfactorily in service.
(99901304/96-01-04) l B.
Criterion III, " Design Control," states that, measures shall be established to assure that the design basis for those components that Appendix B applies are correctly translated into specifications, drawings, procedures, and instructions. These measures shall include provisions to assure that appropriate quality standards are specified and included in design documents and that deviations from such standards are controlled.
17
d 3
p i
Criterion V, Instructions, Procedures, and Drawings," requires that activities affecting quality shall be accomplished in accordance with instructions, procedures and drawings.
Section 5.3.4 of C&D Product & Process Specification (PPS)-X4, Revision 3, states: Immediately following cover installation, the cell / unit shall be held in place for a minimum of 30 minutes with a pressure equal to 80-100 psig. The cover shall be held to the jar in a way to eliminate movement and to distribute the pressure uniformly over j
the entire sealing area.
B.1 Contrary to the above, although C&D had translated a design requirement into Section 5.3.4 of Procedure PPS-X4, it did not ensure that the activity was accomplished in accordance with the procedural requirement.
Consequently, deviations from the. design basis contained in the C&D Procedure were not controlled for that work operation.
(99901304/96-01-05)
Please provide a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN: Docuwt Control Desk, Washington, D.C. 20555-i 0001, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date.of the letter transmitting this Notice of Nonconformance.
This reply should be clearly marked as a " Reply to a Notice of Nonconformance" and should include the following for each nonconformance: (1) a description of steps that have been or will be taken to correct these items; (2) a description of steps that have been or will be taken to prevent recurrence of these items; and (3) the dates your corrective actions and preventive measures were or will be completed.
Dated at Rockville, Maryland, this 17th day of December, 1996 18
i 1
U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No.:
99901304/96-01 Organization:
C&D*2 Charter Power Systems, Incorporated
' 1400 Union Meetireg Roao Blue Bell, Pennsylvania 19422-0858 Contact Address:
Terry J. Kinden, Director of Quality Assurance Washington & Cherry Streets Conshohocken, PA 19428 (610) 825-2150 ext. 245 Nuclear Industry Class IE nuclear power plant station batteries, Activity:
battery chargers, and seismically qualified station battery racks.
Dates of Inspection:
September 16-18, 1996 9 Attica, Indiana, and October 7-9, 1996 9 Conshohocken, Pennsylvania Inspectors:
Kamalakar R. Naidu, Senior Reactor Engineer Yueh-Li C. Li, Mechanical Engineer Joseph J. Petrosino, Q.A. Specialist Saba N. Saba, Electrical Engineer Approved by:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation 2
C&D is a registered trademark of C&D Charter Power Systems, Inc.
Ib
1 INSPECTION SUMARY During this inspection, the NRC inspectors reviewed the implementation of selected portions of C&D Charter Power Systems, Incorporated (C&D) quality assurance (QA) program, reviewed activities associated with the manufacture of lead-acid stationary batteries used in Class IE applications, and reviewed pertinent design and engineering documents.
1 The inspection bases were:
Appendix B, " Quality Assurance Criteria for huclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" The American National Standard Institute /The American Society of Mechanical Engineers (ANSI /ASME) Standard N45.2-1977, " Quality Assurance Program Requirements for Nuclear Facilities" (ANSI N45.2-1977), as endorsed by U. S. Nuclear Regulatory Commission Regulatory Guide (RegGuide) 1.28, " Quality Assurance Program Requirements (Design and Construction)," Revision 2, February 1979 (RegGuide 1.28)
The team found that C&D's quality program for its control of nuclear power plant Class lE station batteries, at the Attica, Indiana facility, contained weaknesses'regarding the consistency of assembly and manufacturing process controls as a result of not providing its assembly and machine operators with the required procedures or instructions as required. The team determined that even though procedures were established for C&D's manufacturing craftsmen, and C&D's quality system program required that they be used; the team identified that in many manufacturing areas the craftsman were not aware nor being provided with the required procedures / instructions as discussed in Section 3.6.
A correlation to this process control inconsistency was 'seen when the team reviewed a C&D laboratory inspection report that discussed a phenomena regarding prematurely degraded LCY-35 battery cells in Class IE safety-related systems at the Edwin I. Hatch nuclear plant (Hatch). The results of the C&D investigation that was delineated in the laboratory inspection report prompted C&D to state that the excess sedimentation suggested that the cause was due to
" improper positive plate processing," most likely because of " improper or incomplete steam curing with resultant weak active material structure."
The team was informed that C&D dedicated its battery cells and associated 3
components at the Attica facility after all manufacturing and assembly activities were completed. C&D staff stated that dedication occurred at the time of a successful battery capacity discharge test, performed in accordance with guidance in The Institute of Electrical and Electronics Engineers, Incorporated (IEEE) Standard-450, " Recommended Practice for Maintenance, 3 As defined in Section 21.3 of 10 CFR Part 21.
2 20
Testing, and Replacement of Large Lead Storage Batteries for Generating Stations and Substations"-1987. Although Part 21 requires that the dedication process be conducted in accordance with an Appendix B to 10 CFR Part 50 quality assurance program, the team found that C&D had not documented or established a battery cell assembly dedication process or program as required by NRC regulations.
Additionally, since C&D claimed to dedicate their batteries after manufacture, i
it would be expected that the batteries would be manufactured as comicercial grade items (CGIs), and that all of the tritical characteristics would be verified upon dedication.
However, even though C&D characterized its battery cells as being handled as CGIs to the NRC inspection team and NRC licensee auditors, as discussed in Section 3.3, it appeared that C&D was taking credit i
for in-process inspections and verifications during design and manufacturing i
activities.
The team also noted that C&D's battery cell design for safety-related applications is controlled under C&D's quality system program that was established to meet ANSI N45.2-1977.
As a result of this finding, the team assessed the adequacy of C&D's quality system program manual (QSPM) implementation since it appeared that the adequacy of the Class IE batteries manufactured at Attica received the benefit of being processed in accordance with CCD's documented QSPM controls.
During this inspection, two violations of 10 CFR Part 21 were identified and are discussed in Sections 3.2 and 3.3 of this report, one was a non-cited violation.
Four instances where C&D Charter Power Systems, Incorporated failed to conform to NRC requirements (10 CFR Part 50 Appendix B) imposed upon them by NRC licensees were also identified.
The nonconformances are discussed in Sections 3.6, 3.7 and 3.9 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS This was the first NRC inspection of C&D's facilities at Attica, Indiana, and Conshohocken, Pennsylvania.
3 INSPECTION FINDINGS AND OTHER COMNENTS 3.1 Quality Assurance Prooram a.
Scone The inspection team (team) reviewed the establi_shment and implementation of selected portions of the CAD quality program, which was documented in the C&D Quality System Program Manual.
The QSPM stated that its purpose was to establish the basic operating policies and procedures to be employed by C&D, and to meet applicable requirements of International Organization for Standardization (IS0) Standard 9001-1994, ANSI [N]45.2-1977 and other imposed specifications.
3 21
4 a
i The team was apprised by C&D that it assures that its components conform j
to the specified requirements through implementation of C&D's quality system program that is-delineated in the QSPM. The QSPM's foreword i
explains that C&D's quality system program is implemented through the QSPM, departmental standard operating procedures, work instructions, drawings, bills of materials, materia 1' specifications, process i
specifications, and quality and test procedures.
Section I, " Management Responsibility," of the QSPM indicates that C&D management has the 4
responsibility and authority to ensure that the policy and objectives J
defined in the QSPM, and its supporting procedures and instructions are understood, implemented, and maintained at all levels in the company.
1 i
b.
Observations-and findinas The team noted that the C&D facilities that are under C&O's quality system program are specified in the QSPM, specifically: (1) Corporate Headquarters-Blue Bell, PA-Control and maintenance of design and qualification aspects of all C&D safety-related components; (2) Attica, IN-Class 1E station batteries; (3) Dunlap, TN-C1s IE station battery chargers; (4) Conshohocken, PA-seismically qualisied, Class lE station battery racks; (5) Leola, PA-Class lE station batteries, round cell; (6) Conyers, GA-commercial products; (7) Huguenot, NY-commercial products; and (8) Ratelco, Incorporated, Seattle, WA-electronics, i
including battery charger assemblies used in safety-related applications of C&D battery chargers. The team ascertained that six of the eight facilities manufacture and cortrol components that are destined for j
safett-related applications at nuclear power plants.
L i
Since six of the eight C&D facilities manufacture components that are used in safety-related applications, the team looked at the overall QSPM program establishment to determine whether a quality system that i
addresses 10 CFR Part 50 Appendix B' requirements such as, 150-9001 and ANSI N45.2, was in existence and controlled for the eight C&D facilities l
l that supplied components for safety-related applications.
j The team observed that C&D's Policy Implementation stated that the C&D Quality Program has been designed to assure compliance with 1S0-9001 (1994), ANSI [N]45.2 (1977), and CSA-9001. However, although the team was informed by C&D staff that each facility complied to ANSI N45.2-1977, the team determined that only 2 of the 8 facilities had been cei.ified to 150-9001 and none of the 8 facilities had been certified to G A 2001.
The team was also informed that all of C&D's Attica facility manufacturing, in-process verification and assembly. activities were considered commercial grade, and that C&D dedicated each station battery cell for Class lE safety-related application at the time of successful final testing using guidance contained in IEEE Standard-450-1987.
Based on that position, C&D considered Attica's manufacturing and assembly activities that occurred before the IEEE-450 test to be activities that did not fall under an Appendix B QA program even though i
4 22 i
1 i
i i
its QSPM was established to meet ANSI N45.2-1977requirements and its certificates of compliance for Class lE batteries attested to compliance
[
with licensee purchase orders (which imposed Appendix B) and C&D's QSPM.
c.
Conclusions The team observed that C&D's quality system program manual was generally developed to follow the requirements of ANSI-N45.2-1977, which can, if established and implemented in accordance with NRC RegGuide 1.28
" Quality Assurance Requirements (Design and Construction)," meet the j
j requirements of Appendix B.
The team concluded that C&D's position regarding battery dedication would have been acceptable had C&D complied with appropriate regulatory and industry guidance regarding dedication.
That guidance requires that the battery cell critical characteristics be identified and their acceptability verified by appropriate inspection, tests, or analysis.
In addition, the dedication process must be conducted and controlled in accordance with the applicable provisions of 10 CFR Part 50 Appendix B.
1 i
The team found that C&D had not established nor implemented a dedication program to comply with NRC regulations or industry guidance.
The team i
did not find, nor was it provided with any battery cell dedication i
procedure documents.
Additionally, the team was not provided with any documented basis of C&D's position regarding battery dedication.
i Further, the team determined that the " dedication" test was indicative j
of the battery capacity at the time of the test but did not adequately demonstrate the battery's functionality for its guaranteed life.
Therefore, the team determined that several manufacturing steps were crucial to demonstrate that the battery would meet purchase order requirements.
Those steps need to be verified by the dedication process or controlled in accordance with Appendix 8.
Consequently, the team j
concluded that this quality assurance program area was not well established or implemented in accordance with 10 CFR Part 21 and 10 CFR Part 50 Appendix B, and industry guidance.
)
i The team also found that one area of the QSPM, regarding work instructions and procedures, was not well implemented at the Attica j
facility.
The team's findings in this area are detailed in Section 3.6.
I 3.2 10 CFR Part 11 Proaram t
j a.
Scone 1
The inspectors evaluated the procedure adopted by C&D to implement the requirements of 10 CFR Part 21 by reviewing Standard Policy and Procedure Number A-14-4, " Reporting of Defects and Nonconformances in i
accordance with Federal Regulation 10 CFR 21 (US Nuclear Regulatory Commission)," dated March 18, 1996.
);
a 23 1
d
b.
Observations and findinat After discussing the implementation of the Part 21 responsibility in detail with the C&D.QA Director, the team informed him that Procedure The failure to adequately A-14-4 wes inconsistent with the regulation.
establish the requirements specified in 121.21 constitutes a violation of minor significance and is being treated as a non-cited violation, consistent with Section IV of the NRC Enforcement Manual (NUREG-1600).
Based upon discussions with the C&L staff the team was concerned regarding C&D compliance with other Part 21 requirements. Specifically, the team discussed requirements that are contained in 121.21(b) of 10 CFR Part 21 with C&D staff that require a supplier of basic components to inform purchasers or affected licensees within five working days if the supplier is unable to perform an evaluation of a deviation or failure to comply. The team also informed the C&D QA Director that its procedure did not define or reference 10 CFR Part 21 specific terms such as, discovery and evaluatfon.
As discussed in Section 3.4 of this report, over 40 of the 120 safety-related station battery cells that C&D manufactured and supplied in late The 1993-early 1994 to the Hatch plant exhibited premature degradation.
battery was degraded due to excessive sedimentation but continued to be operable according to the licensee. Therefore, the team reviewed the Hatch event with regard to C&D's responsibility under 10 CFR part 21.
The team noted C&D's June 1996 laboratory inspection report of the cell C&D degradation indicated "an improper or incomplete steam curing."
staff stated that it reviewed its manufacturing and quality control (QC) records during that time period and did not identify other potentially C&D further stated that no other customers affected customers.
(commercial or nuclear) with similar battery cells from that time period The team had no concerns with C&D's have reported similar problems.
efforts regarding the Hatch issue.
c.
Conclusions The team concluded that C&D's procedure adopted to implement Part 21 was not adequately established and lacked clarity. Specifically, the team found that the procedure did not address the preparation of an interim report if an evaluation of a deviation or failure to comply cannot be completed within 60 days of discovery.
Additionally, the team concluded that C&D was not required to inform the NRC or other customers of the manufacturing process control weakness, as discussed in Sections 3.6 and 3.7, since it determined that no other customers were affected during its review of the circumstances of the problem.
6 24
3.3 Dedication of Class IE Station Batteries i
- a.. kgag The team found that the control of the manufacture of C&D's battery cells and associated components could be viewed as being implemented in one of the following two ways: (1) performing dedication activities of its Class 1E battery cells, as conditionally permitted by 10 CFR Part 21, in accordance with the applicable requirements of Appendix B to 10 CFR Part 50, or (2) manufacturing its Class IE battery cells as basic components, under its QSPM.
b.
Observations and findinas C&D staff informed the team that its station batteries were processed in their manufacturing and assembly areas as commercial grade items (CGIs) and were not considered basic components until after each battery cell passed a test (which C&D considers its dedication activity) that C&D performs to meet guidance contained in IEEE-450-1987. Although, due to a lack of adequate documentation and conflicting information, the team was not able to determine how C&D does, in fact, produce its safety-related batteries (whether by dedication or by manufacture in accordance with Appendix B), the team discussed C&D's dedication position to determine its compliance with NRC requirements.
The team also reviewed the manufacturing process for compliance with Appendix B, as discussed elsewhere in this report.
Additionally, the team noted that the C&D safety-related battery cells are typically used at licensee facilities for back-up electrical requirements for instrumentation and control and other power needs to various safety-related circuits, inverter power and emergency and normal Class 1E vital bus applications.
Typical critical characteristics of lead-acid battery cells that would be required to be verified include characteristics such as: ampere-hour capacity; cell voltage; dimensions and configuration; case and componont materials, such as paste mixture for positive and negative plates; and grid, plate, or electrode construction and assembly, such as internal plate assembly-to-post connections.
Section 21.3 of 10 CFR Part 21_ states that dedication is an acceptable process undertaken to provide reasonable assurance that a commercial grade item will perform its intended safety function and, in this respect, is deemed equivalent to an item designed and manufactured under a 10 CFR Part 50, Appendix B, quality assurance program.
This assurance is achieved by identifying the critical characteristics of the item and verifying their acceptability by inspections, tests, or analyses, and that the dedication process must be conducted in accordance with the applicable prov'sions of Appendix B.
Section 21.3 also states that a commercial grade item mean:, a structure, system, or component, or part therefore that affects its safety function, that was not designed and manufactured as a basic component.
25
items [CGIs) do not include items where the design and manufacturing process require in-process inspections and verifications to ensure that defects or failures to comply are identified and corrected (i.e., one or more critical char.acteristics of the item cannot be verified).
- Further, Sect. ion 21.21 of 10 CFR Part 21 requires a dedicating entity to maintain auditable records for the dedication process.
The team determined that C&D: (1) did not establish or have a dedication procedure or program that complied with the applicable provisions of Aprendix B, or 10 CFR Part 21, (2) did not identify or verify critical characteristics that were necessary to perform an adequate dedication process, (3) did not take exceptions to licensee purchase order requirements that imposed Appendix B to 10 CFR Part 50, and (4) indicated on its certificates of compliance that its Class lE batteries are in compliance with the applicable licensee purchase orders and specifications which the team found to typically impose Appendix B to 10 CFR Part 50 and C&D's QSPN.
Further, the team noted that C&D appeared to take credit for the implementation of manufacturing activities at Attica under its QSPM program for compliance to Appendix B to 10 CFR Part 50, as attested to in its CoCs.
Conversely, C&D appeared to characterize the same manufacturing activities to the NRC inspection team and. licensee auditors, that occurred before IEEE-450 testing, as being commercial processes not subject to Appendix B.
The team also noted that C&D's dedication position of its battery cell assembly dedication had been found to be acceptable by recent licensee audit group inspections.
c.
Conclusion The team did not find any objective evidence to indicate that C&D was manufacturing its battery cells destined for Class IE applications as commercial grade items followed by dedication.
The team concluded that even though Section 21.21 of 10 CFR Part 21 requires that a dedicating entity is responsible for maintaining auditable records for the dedication process, and Section 21.3 requires that the dedication process must be conducted in accordance with 10 CFR Part 50 Appendix B, C&D did not have specific records to delineate its battery cell assembly dedication or to substantiate that its activities were carried out in accordance with 10 CFR part 50 Appendix B.
The team identified Vi 'ation 99901304/96-01-01 in this area.
The team concluded that although C&D stated that it was performing battery cell dedication for customers who imposed Appendix B to 10 CFR Part 50, C&D did not establish a dedication procedure to identify and verify the adequacy of each of the critical characteristics important to ensure that the battery cells would perform satisfactorily in service.
Tnerefore, C&D did not ensure that'its battery cell dedication activities affecting quality were prescribed and accomplished in accordance with documented instructions as required by Criterion V, 8
l 26
" Instructions, Procedures, and Drawings," of Appendix B to 10 CFR Part 50. The team identified Nonconformance 99901304/96-01-04 in this area.
Additionally, the lack of a formalized dedication program to address critical characteristics and the lack of an effectively implemented Appendix B to 10 CFR Part 50 type of a quality system program could affect the overall life and functionality of C&D's battery cells that are currently in-use at operating nuclear power plant facilities.
3.4 Batteries for Hatch Nuclear Power Plant a.
Scone The inspectors reviewed the circumstances surrounding premature degradation of over 40 of the 120 C&D LCY-35 type battery cells that it manufactured and supplied to Hatch in 1994-1995.
The scope of the team's review included: (1) C&D documentation to support the LCY-35 i
design; (2) licensee purchase order requirements; (3) C&D's documentation substantiating C&D's certificate of compliance (CoC) for Hatch; and (4) the C&D analysis of the failure of the batteries supplied to Hatch.
b.
Observations and findinas Backaround In April 1996, the'NRC was informed by the Georgia Power Company (GPC) that excessive sedimentation was observed in 21 of 127 LCY-35 type batteries supplied by C&D, and by May 1996, approximately 43 1
battery cells exhibited excessive sedimentation.
The batteries were supplied by C&D in 1994 to the Hatch nuclear power plant (Hatch). C&D committed to the licensee to replace the prematurely degraded battery cells with new battery cells.
Desian Verification The inspectors reviewed the discharge characteristics curves of LCY (35 through 39 plates), and LCUN-33 (33 plates) type batteries at C&D's facility in Conshohocken, PA. These
. batteries are used in switch-gear and control applications at commercial nuclear power plants and are. tested utilizing test procedures meeting the IEEE 450 standards.
The inspectors reviewed the data for a battery with voltage terminal of 125 volts,1.215 electrolyte specific gravity at 77'F (25'C) and found it acceptable.
The team reviewed test data of a typical duty cycle of LCY-35 and LCY-37 cells to verify that the cell voltage at'.the one-minute discharge rate included the effect of the Coup de Fouet The team's review of the test data confirmed that all the one-minute rates did include the Coup de Fouet.
Ap nomena of additional battery voltage drop when the battery is subjected to a discharge after long duration float charges.
9 i
27
i j
The inspectors reviewed the implementation of.the engineering change notices (ECNs) and engineering change requests (ECRs). A typical ECR i
gave detailed information on the reasons for the change and its impact
{
on manufacturing, whereas, a typical ECN referred to the ECR, and was i
sent to different departments, including the manufacturing facilities.
The team did not identify any concerns in this area.
i The inspectors established that one of the principal differences between 4
i Class IE and commercial grade cells is the separator.
In Class IE l
cells, C&D uses a rubber separator, whereas, in the commercial battery cells CAD uses a microporous polyetnylene separator.
The separator i
appeared to be the only component difference between the commercial and Class IE battery cells. The installation of the rubber separator is an example of a manufacturing activity that requires inspection or verification during dedication or control by Appendix B quality program.
i Hatch Purchase Order The inspectors reviewed GPC's PO 60135700000, dated December 1, 1994, to C&D for the supply of LCR-25.and LCY-35 125/250 Vdc, lead calcium,120 cell batteries and accessories for Hatch.
The PO included the following requirements:
Applicable portions of 10 CFR 50, Appendix B,-apply Items shall be supplied new and not used or refurbished in any way 10 CFR 21 applies to safety related nuclear items / services CoC certifying the material / components meet the requirements of Specification NP-93024, Revision 2, Section 3.A.4.
Certificate's of Comoliance (CoC) The inspectors reviewed C&D's CoC dated February 25, 1994, for Hatch and determined that it certified that 127 LCY-35 wet cells with accessories were brand new and had been manufactured in accordance with the requirements of specification NP-93024, Revision 2 and the quality standards specified therein, and that.
it met IEEE-450-1987 capacity acceptance test.
The discharge rate was 459 amperes for a discharge time of 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> and the acceptance volts per cell was 1.75 average. The Capacity Discharge test sheets were attached to the CoC. These test sheets indicated that the average volts per cell j
exceeded 1.75V at the end of the test.
The inspectors noted that C&D issued a CoC certifying that the batteries met all the GPC requirements implying that C&D's QA program complied with 10 CFR Part 50, Appendix B, even though C&D was aware that its instructions and procedures affecting the quality of the components supplied were not comprehensive enough in some areas.
For example, procedures and instructions that did not adequately address all of the Attica facility work cr.tivities, as discussed in Section 3.6.
A review of a sample popul0 tion of :ther NRC licensee purchase order packages at the Attica facility determined that licensees are procuring Class IE battery cells and accessories on P0s which typically impose l
10 l
28
i f
l Appendix B to 10 CFR Part 50, 10 CFR Part 21 and the nuclear related quality program that is delineated in C&D's QSPM (ANSI-N45.2-1977). All of the C&D CoCs reviewed in the riifferent PO packages stated compliance with the applicable licensee P0 and licensee specification.
Additionally, the CoCs contained other compliance statements as required, such as, "all batteries / cells are manufactured in accordance with the manual, me(eting the applicable requirements of 10 CFR 50, Appendix ANSI N45.2-1977, and other applicable industry standards (e.g., IEEE i
450-1987, and IEEE 535-1986)."
In the PO packages reviewed, the inspectors observed that C&D did not take any except. ions to any of the licensee contractual requirements or additional requirements imposed on j
them, nor did any of the CoCs or QSPM discuss or address a dedication 1
basis as discussed in Section 3.3 of this report.
Service Insnection Reoort C&D examined the excessive sedimentation in the Hatch batteries. According to C&D the excessive sediment, which 4
looks like sponge deposits, appeared to be the result of active plate material being shed due to contamination or impurities introduced into the plate material at the factory.
The batteries were considered 4
operable based on their tested' performance to deliver over 116% of rated capacity. On June 25, 1996, after inspecting the batteries, C&D transmitted its findings and included C&D's battery laboratory inspection report.
The inspectors reviewed the report which was based on C&D's analysis of one battery cell from Hatch, stated:
The' individual cell voltage was more than 2.29 V per cell.
The positive and negative grids were in good condition.
The positive plate active surface material was depleted (shed) with approximately 50-100% of the plate surface being affected.
The negative active material was soft.
There was no mossing present either on sides of the frames or the top.
No sulfaration was evident.
The sediment chamber was filled with shed active material that had oxidized after making contact with the feet on the negative plate.
The glass mat was saturated at the bottom.
The plate lug strap burns were good.
Corrosion was normal on the positive post and strap assembly.
The report also stated Hatch's LCY-35 battery suffered from sedimentation, due to improper positive plate processing, and the condition of the positive plate active materi-1 suggested improper or incomplete steam curing with resultant weak active material structure.
11 e
29
During discussions with C&D personnel, the team was informed that C&D is currently in the process of considering additional design enhancements and process control changes to control the curing ovens more effectively for uniform curing of the positive plates.
The team concluded that C&D's laboratory report results and the team's observations of weaknesses in C&D's process control suggests that other Class IE batteries that C&D provided in this same time frame may also l
prematurely degrade.
Therefore, this aspect was discussed with the C&D Quality Assurance Director rega. ding whether C&D should be considering informing its applicable customers of a potential deviation that needs to be monitored and evaluated by certain licensees.
However, as discussed in Section 3.2, C&D investigated this issue and concluded that j
it was not applicable to other licensee battery cells.
c.
Conclusion In general, the inspectors observed that the C&D adequately controlled the design of the batteries through ECNs and ECRs, including technical and engineering notices.
1 The inspectors determined that C&O's CoC implied that C&D's QA program complied with 10 CFR Part 50, Appendix B, even though C&D was aware that its instructions and procedures affecting the quality of the components supplied did not meet the requirements of 10 CFR Part 50 Appendix B or ANSI N45.2-1977.
3Property "ANSI code" (as page type) with input value "ANSI N45.2-1977.</br></br>3" contains invalid characters or is incomplete and therefore can cause unexpected results during a query or annotation process..5 Manaaement Auditi The inspectors reviewed the implementation of the QA program relative to management audits. The plant manager performed audits to satisfy the requirements of the QA Manual.
Paragraph 1.5 of Section 1.0 of the C&D QSPM requires executive management to review the quality system program at defined intervals in accordance with Management Review procedures.
This review is intended to ensure the continuing suitability and effectiveness of the established Quality Management System and the Company's stated quality policies and objectives. As an integral part of the process, confirmation as to preventive actions and results of audits are to be submitted.
Standard Operating Procedure (SOP)01-022.2 outlines the requirements to implement the above.
Paragraph 6.3 reyires the plant manager to canduct a quality system program review on
. warterly basis one month prior to the end of the fiscal quarter or sooner as indicated by necessity or by direction of corporate Quality i
counsel.
The inspectors reviewed the plant manager's audits performed on December 31, 1995, and March 29, 1996, and determined that the audits covered the following: the need to develop additional process instructions and work instructions, the need to revise training for hourly employees, enhancements in process improvement programs and quality improvement programs, and status of corrective actions to a previous external audit.
Copies of the audit were distributed to the " Corporate Quality Counsel" 12 30
It which consists of: the President /CEO, the Director of QA, the VP of Finance, Power Com., Motive, Technology, Power Com Operations, and the Director of Motive Products Services.
The team noted that the audit recognized the necessity to develop procedures to control the manufacturing process as required by C&D's quality system program.
3.6 Manufacturina Process Control a.
Scope The team observed the process controls used by C&D in the manufacture of their batteries and related components.
The team found that C&D's goal was to have all of its facilities certified to 150-9001.
The main document which establishes the basic operating policies and procedures employed to meet the requirements of 150-9001 is the QSPM.
The QSPM states that the C&D quality system program is implemented through the use of operating procedures, work instructions, drawings, quality and test procedures, etc. At the time of the inspection, the primary documents used to control manufacturing activities were Product &
i Process Specifications (PPSs) and Factory Procedures (fps).
b.
Observations and Findinas Adeauacy of orocedure and instructions The team found that in 4
approximately 1993-1994, when the current C&D QA Director' assumed his current position, he recognized that the existing PPSs and fps that had j
been established for many of the C&D facilities woul,d not support ISO-i 9001 certification. The team was told that the PPSs and fps were i
written to encompass multiple facilities and, as a result, were not specific to the battery manufacturing process control.
Therefore, C&D started a program to establish specific instructions and procedures for each particular work activity at each of its facilities i
to replace the existing PPSs and fps.
3 The replacement documents for the PPSs and fps are process instructions (PIs) and operating / work instructions (01s). The team was informed that the Pls and Ols for the Leola facility had been completed but C&D had just begun to develop PIs and Ols for its Attica and Dunlap facilities.
The team reviewed some of C&D's draft Ols and PIs for the Attica i
facility and conducted discussions with C&D process engineering personnel who are responsible for developing these documents.
The team l
noted that the new Ols and PIs appear comprehensive and easy to use, and the team was informed that the new work instruction documents have the banefit of receiving input from the personnel that are performing the work activity.
The QA Director stated that C&D expects to complete the PIs and Ols at Attica and Dunlap in the Fall of 1998.
Use of orocedures and instruction 1 Discussions with manufacturing i
personnel revealed that the personnel were experienced and knowledgeable regarding their respective areas.
The team found that this knowledge was apparently obtained from comprehensive on-the-job training (0JT) 33 31
programs. During the discussions, the team asked wha'. procedures and instructions were used..The team found that craft personnel typically were not aware of specific procedures or work instructions that controlled the activities that they performed, with the exception of those working in the paste mixing area.
Section 3, " Responsibilities and Authorities," of C&D's QA Procedure 01-003.6, " Document and Data Control (Corporate)," states that department managers are responsible for generating, approving, distributing and maintaining work instructions for their department.
Additionally,
" department managers and supervisors are responsible to assure that the correct documents are available at the locations where needed and that j
they are being used"'.
c.
Conclusions 1
The team determined that C&D's quality system program of procedures and instructions was not adequately established to ensure effective and consistent manufacturing processes control. That determination is supported by C&D's effort to replace all of its PPSs and fps. The team concluded that this weakness was not controlled in accordance with corrective action requirements of ANSI-N45.2-1977, 10 CFR Part 50 Appendix B or C&D's QSPM, which has been established to address ANSI f
N45.2-1977. Although this matter could have been cited as a nonconformance being contrary to licensee requirements imposed on C&D, the team did not identify this matter as' a noncompliance to 10 CFR Part 50 Appendix B because the QA Director had identified the problem and had initiated corrective action.
The team found that C&D has been aware of the practice of not providir.g instructions and procedures and has been attempting to correct the problem by establishing specific Ols and Pls for. each of its facilities since approximately 1992-1993.
However, the team noted that until this activity is complete, C&D's Attica activities are contrary to Criterion V, " Instructions, Procedures, and Drawings," of Appendix B to 10 CFR Part 50,-and Section 6, " Instructions, Procedures and Drawings," of ANSI-N45.2-1977, regarding implementation of QSPM.
Nonconformance 99901304/96-01-02 was identified in this area.
~
3.7 Observation of Work Activities a.
Insoection Scope The team inspected several areas at the Attica battery cell and associated product manufacturing and assembly facility including: paste mixing, paste application, curing, hydrosetting and drying, cell assembly, and leak testing. The team observed the manufacturing process control activities that were being performed and conducted discussions with crafts personnel. As discussed in Section 3.1, C&D's quality 14 32
system program has been established to encompass all activities affecting product conformance to specified requirements through its system of operating procedures, work instructions, drawings, bills of material, specifications, and quality and test procedures.
b.
Observations and findinas Paste mixina The inspectors observed the process of mixing paste for application on the gr_ ids.
The paste operator formulated the paste 3
according to two tables observed to be in_ a C&D procedure; one table was for the positive plate and the other table was for the negative plate.
These tables provided information on the components to be used, the materials specification, the quantities and tolerances, and the acceptance criteria for the resulting paste.
The team noted that the i
paste mixing area was the only area at the Attica manufacturing facility where the craftsmen were observed to have a procedure or work instruction available for the craftsmen.
Paste aonlication Paste 'is applied to the grids by gravit'y flow from a hopper to the grid's pasting operations where it is applied to the positive and negative grids.
Finished grids are visually inspected for non-uniform pasting.
Grids which are rejected during the_ initial inspection are run a second time through the process and if the non-uniformity persists, the grid is rejected and scrapped. 'Following pasting, positive plates are sent to the curing area while negative plates go directly to hydrosetting.
i The inspectors observed the pasting of positive plates.
During this^
period, a few plates which C&D personnel observed to be irregular were i
put through the process a second time. The production rate of this specific run appeared to operate smoothly with few slow-downs and. stops during the period observed.
After the pasting was completed, the plates were moved to the curing process area.
Curina In this area, the positive plates were subjected to.a humidity
{
curing process (also known as steam curing).
Curing is performed to obtain mechanical strength, grid paste adhesion, porosity and proper t
lead oxide crystal structure. The plates were cured in an oven required to be maintained at certain humidity and temperature levels specified by i
a chart.
The inspectors observed that the only parameters of the curing that are recorded are the oven temperature and humidity. An alarm has not been installed to announce any failure of the heating or the humidity sources. The inspectors also observed that the access to the oven doors were not mechanically controlled.
The inspectors observed this as a weakness in C&D's curing process because plate curing is one of the critical _ and important stages in the manufacture of the lead acid batteries. The aim _of the curing process is to convert the wet paste to dry, crack-free material with sufficient stre..gth and adhesion to the grid plate.
Improperly cured plates can affect the performance and longevity of the battery cells.
15 33
i Hydrosettina and dryina Depending on the type of the lead oxide used in the manufacture of positive plates, they are either dried immediately or subjected to a hydroset process before drying. All negative plates are hydroset before drying.
Hydroset curing is performed at high humidity and relative low temperature to assure the required grid adhesion, mechanical strength
- and porosity.
During the hydrosetting process, the free lead content of the active material is reduced tn specific levels depending on the application of the battery.
4 All types of plates are subjected to a drying process at 215 i 5'F.
The final moisture content depends on the type of-plates.
The team noted during several times of observing this area that the overhead doors are
- periodically left open without any obvious reconciliation or verification of the humidity and temperature levels.
Cell assembly The inspectors observed craft-persons assemble different battery. cell configurations (such as 4XTL13 and KCR9, that are used in safety-related applications). The batteries were assembled in sequence including: (1) stacking, (2) plate-to-strap bonding, (3) element.
insertion into the container, (4) cover installation, (5) terminal-connector post-to-cover insert bonding, and (6) leak testing.
The team noted that PPS-X4-3, " Standby Cell Assembly," March 28, 1991, described the assembly operation and emphasized proper alignment of plates and separators.
The team determined these aspects were critical to the appearance, performance, and life of the battery cell. The element assembly. consisted of two distinct procedures: stacking and plate to strap bonding. A stacked element consists of alternating negative and positive plates, interleaved with separators.
After the plates were stacked so that the lugs are aligned, the positive and negative burning lugs were on the opposite sides.
The plate was then bonded to the strap and visually inspected for voids, porosity, entrapped dross, lead run downs, and poor bonding.
(At this stage the assembly is cleaned if necessary.) After acceptance, the operator raised the element into the vertical position and inserted support combs and moss shields.
The element fitted snugly into the' battery container.
The cover was then glued on the container.
The team observed that after the glue was inserted onto the mating surface of the cover at the cover-to-jar mating area, the cover was placed on the jar, and tapped into place with a mallet. The craft-person then placed a wooden frame over the cover, and placed a banding type of nylon strap with a ratchet tightening device around the entire jar / cover assembly.
The craft-person hand-tightened the ratchet until the strap appeared to be snug. The cell was allowed to sit at that work station for some undetermined amount of time to allow the glue to dry.
16 34
4 The team observed that the above activity was contrary to the requirements of PPS-X4-3. The procedural requirement that supposedly controlled this area (Section 5.3.4 of PPS-X4-3), stated: "Immediately following cover installation, the cell / unit shall be held in place for a minimum of 30 minutes with a pressure equal to 80-100 psig.
The cover shall be held to the jar in a way to eliminate movement and to 3
distribute the pressure uniformly over the entire sealing area."
As a result of comparing the actual work practice and the procedural requirement, the team noted that: II) although the banding strap appeared to be firmly tightened, tne craftsmen did not have any indication of the actual pressure that was applied each time that the operation was performed, and (2) it was not obvious whether C&D's wooden frame actually " distributed the pressure uniformly."
The inspectors discussed the process with the craft personnel and determined the 4
craftsmen were not aware of any requirement for a specific pressure to be applied nor were they aware of a procedure that they were supposed to be using for that area of the assembly line.
The team asked C&D management about the applied pressure requirement and was informed that the requirement had been modified in practice, but it was left in Procedure PPS-X4-3 because the procedure was scheduled to be replaced with the new Ols and PIs.
The team asked C&D engineering if it had approved the deletion of the requirement because it appeared to the team that it was a design requirement affecting the operability and service life of the battery.
C&D QA staff informed the team that C&D engineering had approved the removal of the specified pressure requirement.
However, the team was subsequently told at the Conshohocken facility that engineering had ng1 approved the discontinuance of the PPS specified activity.
The C&D Chief Engineer in the battery area stated that the work activity step was a C&D engineering requirement and that no engineering approval had permitted the requirement to be discontinued.
The team concluded that C&D management failed to ensure that craft personnel were supplied the procedure that provided the specific design requirement.
That failure resulted in craft personnel implementing a change to the manufacturing process which, in turn, resulted in an unreviewed change to the design requirements.
The team informed C&D that this was contrary to Criterion.III, " Design Control," of Appendix B to 10 CFR Part 50, which requires that design changes be subject to design cont sl measures commensurate with the original design and approved by the organization that performed the original design. The inspectors were unable to determine how long this practice was prevalent.
Nonconformance 99901304/96-01-05 was identified in this area.
Leak testina of cells The team noted that C&D procedures require operators to perform leak tests on all cells using a specific pressure for several seconds to ccafirm that the container will not leak.
The team also noted that final formation of the cells, the electrolyte 17 i
35
filling of cells, charging currents and duration of the different charges are also discussed in C&D procedures. The team found these procedures were in the plant supervisor's office.
Overall, during discussions with the craft personnel, several of the craft personnel admitted that they were not aware of any specific work instructions or procedures for-their specific job function, but relied instead on the OJT programs and co-worker experience and assistance. Generally, the team noted several weaknesses regarding C&D's control of their manufacturing proc.ess.
Primary among those was the unreviewed design change to the battery assembly requirement.
The inspectors were also concerned with the lack of control for the curing i
oven temperature and humidity, including lack of process alarms and lack of control for the oven doors.
In addition, the lack of control of manufacturing area overhead doors during certain manufacturing steps could-also affect required humidity and temperature levels, c.
Conclusions The team concluded that craft personnel were knowledgeable and appeared to be experienced in their job.
Based upon discussions with craft personnel, it was apparent that knowledge and experience was due primarily to OJT.
The failure to use work procedures or instructions is discussed in Section 3.6.
One nonconformance was identified in this area.
3.8 Canacity Discharae Tests 4
The team observed capacity discharge tests in progress on a short duration and long duration battery to confirm that battery cell capacities met the design specifications. A short duration battery is designed to supply relatively high currents for a short period of time.
These type of batteries, used in UPS applications at nuclear plants, are manufactured as off-the-shelf items. A long duration battery is designed to supply relatively small, loads over a much longer period of time, typically eight hours.
Long duration batteries are manufactured for a specific order. The team was informed that C&D monitors individual cell voltages during the discharge test to identify potentially defective cells.
15-minute discharoe rate of a short duration battery The team noted that the test was successfully performed for the full 15 minutes. The team observed that the first low cell voltage alarm occurred at 15 minutes and 40 seconds. The test was terminated at 17.10 minutes from the start indicating a capacity of over-114%.
8-hour discharae rate of a lona duration battery During observation of the test, the team noted that, when the test monitor indicated a low voltage on a cell, the test operatcr was able to identify the faulty cell using a hand-held voltmeter and confirmed that it was a bad wire 4
18 36
connection. The connection was repaired and the voltage on the cell returned to normal.
The test continued thereafter without interruption.
The test was not completed until after the inspection.
The inspectors noted that C&D had adequate instrumentation to monitor the individual battery cell voltages and the discharge currents during the tests.
C&D routinely performed these tests to establish a benchmark rating of the battery before shipment.
The inspectors had no concerns in this area.
3.9 SilSMIC OUALIFICATION REPORT a.
Scone To assess C&D's seismic qualification basis for its LCR-29 and LCY-35 batteries installed at Hatch, the team reviewed the seismic qualification portions of C&D's Report QR-66171-01, " Environmental and Seismic Qualification Report of TYPE LCR-29 and LCY-35 Battery for Hatch Nuclear Plant," dated January 22, 1993.
The team also reviewed Wyle Laboratories, Seismic Simulation Test Reports 43450-1, " Seismic Simulation Test Program on a Battery Rack and Batteries," December 7, 1976, and 44467-1, " Seismic Simulation Test Program on a Battery Rack Containing Two LC-25 Battery Cells and a Battery Rack Containing A 30C0-5 and 3000-7 Battery Cells," March 1979.
b.
Observations and findinas The performance requirement specified in the report for the batteries is that the battery shall be capable of supplying the design loads for the required durations, without the voltage at the terminals falling below 210 volts.
The battery shall be capable of supplying design loads while experiencing any single or combination of the following normal and design service conditions: Temperature-77'F annual average, 65'F minimum, 110*F maximum; Pressure-atmospheric; Radiation-less than 10 rads, 40 year total integrated dose; and Seismic-0BE & DBE Seismic Spectra for turbine building elevation 116'-0."
The seismic qualification for the 125/250 Vdc lead-acid storage battery assembly for the LCR-29 and LCY-35 batteries at Hatch plant was provided in accordance with the requirements of Georgia Power Company Purchase Order 6010129, dated December 16, 1992, as well as the requirements of IEEE Standard 344-1975. The basis for qualification was by combination of test and analysis.
Several seismic qualification tests were previously performed for various "L" type battery cells at Wyle Laboratories.
The tests were performed in accordance with IEEE 344-1975 and the results were found to be well documented. The seismic simulation test included five Operating Base Earthquake (OBE) level tests prior to ore Design Base Earthquake (DBE) test.
In addition to the seismic simulation test, pre-seismic and post-seismic battery discharge capacity tests were also performed.
19 37
The seismic qualification of the Hatch LCR-29 and LCY-35 was demonstrated by similarity to those previously successfully tested l
battery assemblies with the same design and configuration. A detailed description of the similarity evaluation was found and reviewed. The evaluation included comparison of the battery design with emphasis on j
material, dimensions, weights, aging mechanism, battery service j
4 conditions, and service life. The comparison that was reviewed i
indicated that the design of the LCR-29 and LCY-35 battery assembly was j
equal or better than the originally qualified battery assembly.
The evaluation also indicated that the seismic acceleration experienced j
by the specimen assembly enveloped the seismic response spectrum.
Therefore, the loads experienced by the tested specimen enveloped the l_
design loads for the plant Hatch batteries.
The evaluation reviewed by the team was based on the well-designed LCR-29 and LCY-35 battery assemblies and its data base did not include i
battery cells with excess amounts of sedimentation similar to the i
existing LCY-35 battery at Hatch nuclear plant. Therefore, the staff's j
inspection findings and conclusions are not applicable to those existing batteries at Hatch.
t The team observed that C&D did not have a documented procedure for
{
performing the seismic qualification by similarity.
The team informed t
C&D management that Criterion V of Appendix B to 10 CFR Part 50 requires safety-related activities, such as the preparation of the Seismic 4
Report, be performed in accordance with written procedures. The team emphasized that in order to assure consistency and quality of seismic qualification reports, it is necessary to have a documented procedure which includes appropriate quantitative or qualitative acceptance criteria for determining that important technical aspects concerning the j
i similarity have been satisfied.
The team also noted that C&D was not required by Hatch to perform any specific seismic simulation test with C&D's Hatch battery to ensure that the existing Hatch battery will remain functional during and after a seismic event.
c.
Conclusions Tha inspectors concluded that C&D has documented an adequate technical j
basis for the seismic qualification for the LCR-29 and LCY 35 battery assembly, and no concerns in this area were identified.
The team concluded that because C&D did not have a formal dedication program to verify battery cell critical characteristics it was unclear j
as to how C&D had ensured that it has maintained its seismically qualified configurations during its manufacturing process control.
1 1
20 38
The team apprised C&D that failure to have a written procedure for l
performing seismic analysis was contrary to Criterion V of Appendix B to i
10 CFR Part 50. Nonconformance 99901304/96-01-03 was identified in this area.
3.10 PERSONS CONTACTED C&D Charter Power Systems. Inc.. Attica. Indiana
+
C. Brown, Plant Manager J. DeSutter, General Supervisor, Casting
+
B. Donavan, General Supervisor, West Side
+
W. Foster, General Supervisor, Pasting
+
M. Guthrie, Quality Control Manger
+
R. Keller, Process Engineer
+
W. Lucas, Material Control C&D Charter Power Systems. Inc.. Conshohocken. Pennsylvania D. Heimer, Manager, Design & Documentation
- *+
T. Kinden, Director, Quality Assurance E. Urbanski, Manager, Process Engineering F. Wagner, Chief Engineer A. Williamson, Manager, Product Test Laboratories
- +
C. Wood, Coordinator Supplier Quality C&D Charter Power Systems. Inc.. Blue Bell. Pennsv1vania G. Walker, Manager, Applications Engineering
+ - Attendance at entrance meeting on 9/16/96
- - Attendance at exit meeting on 9/18/96
- - Attendance at entrance meeting on 10/7/96
- = Attendance at exit meeting on 10/9/96 21 39
~. - -
~ - -.
.=
9 Kia UNITED STATES NUCLEAR REGULATORY COMMISSION y
g g
WASHINGTON, D.C. 20666-0001 t
4,.....,o October 29, 1996 Mr. Robert E. Bond, President Dragon Valves, Inc.
13457 Excelsior Drive Norwalk, CA 90650
SUBJECT:
NRC INSPECTION REPORT 99900264/96-01 AND NOTICE
Dear Mr. Bond:
the U.S. Nuclear Regulatory Commission (NRC) completed The enclosed On September 26, 1996, an inspection at your Dragon Valves, Inc., (Dragon) facility.
report presents the results of that inspection.
The NRC inspectors found that the implementation of the Specifically, the procedures for upgrading unqualified source material were not controlled and did not comply with the applicable ASME Code its customers.
requirements, Dragon had accepted material certifications without the requ quality system program statements, the procedure for dedicating customer defined safety-related components did not provide a basis for the specified chemical sampling plan, 'and Dragon had not documented any actions for a significant condition adverse to quality, as required by 10 CFR Part 50, Appendix B.
These nonconfo.wances are cited in the enclosed Notice of Nonconform (NON), and the circumstances surrounding them are described in detail in t You are requested to respond to the nonconformances and should follow the instructions specified in the enclosed NON when preparing enclosed report.
l your response.
In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its enclosures will be placed in the NRC's Public Document Room (PDR).
Sincerely, Robert M. Gallo, Chief j
Special Inspection Branch i
Division of Inspection and Support Programs Office of Nuclear Reactor Regulation Docket No. 99900264
Enclosures:
1.
Notice of Nonconformance 2.
Inspection Report 99900264/96-01 4
40
4 i
~
NOTICE OF NONCONFORMANCE Dragon Valves, Inc.
Docket No. 99900264 Norwalk, California I
Based on the results of an inspection conducted on September 23 through 26, 1996, it appears that certain of your activities were not conducted in j
accordance with NRC requirements.
i A.
Criterion V, " Instructions, Procedures, and Drawings," of Appendix B to 4
i 10 CFR Part 50 and Paragraph NCA 3853.2 of Subsection NCA of ASME Code,Section III, requires, in part, that activities affecting quality be i
prescribed by documented instructions or procedures and that these i
procedures include appropriate criteria for determining that these activities are satisfactorily accomplished.
Paragraph NCA 3853.3, " Document Control" of Subsection NCA of ASME Code,Section III, requires, in part, that instructions and procedures be controlled to assure that correct documents are being used.
Paragraph NCA 3855.5, " Utilization of Unqualified Source Material", of Subsection NCA of ASME Code,Section III states, in part, that a Material Organization (MO) may accept certification of the requirements of the material specification which must be performed during melting, heat analysis, and heat treatment from an unqualified supplier and may furnish such material, providing that the M0 performs or subcontracts a product analysis on each piece of such material 10.d performs or subcontracts all other requirements of the material specification on each piece of the unqualified stock material.
Contrary to the above:
(1) The Unqualified Source Material Approved Vendor List, referenced in Proedure 17375, " Procedure for Procurement and Inspection of Unqualified Source Material," Rev. N/C, had not been developed and was not available for use.
(2) Procedures were not controlled, in that two procedures, 17375 and 17053, with the same title, " Procedure for Procurement and Inspection of Unqualified Source Material," but different requirements, were available for use.
(3) Procedure 17375 permitted the acceptance and furnishing of i
unqualified source material based on subcontracting the performance 1
of product analysis but without performing or subcontracting all other requirements of the material specification.
(Nonconformance 99900264/96-01-01)
{
41
1 l
B.
Criterion Vil, " Control of Purchased Material,' Equipment, and Services,"
of Appendix B to 10 CFR Part 50, requires, in part, that measures shall be established to assure that purchased material conforms to procurement documents.
Paragraph NCA 3862.2, " Quality System Program Statement," requires material organizations qualified by parties other than ASME to show the revision and date of the applicable written quality system program on the certified material test report or certificate of compliance.
Contrary to the above, Dragen Valves, Inc., (Dragon) accepted material certifications for 1 inch, type 316 stainless steel-tubing from Sandvik Steel without the appropriate quality system program statement and utilized this material without upgrading.
l (Nonconformance 99900264/96-01-02)
C.
Criterion VII, " Control of Purchased Materials, Equipment, and Services," of Appendix 8 to 10 CFR Part 50, requires, in part, that l
measures shall be' established to assure that purchased material conforms to procurement documents.
i Contrary to the above, Dragon had not established a documented basis to substantiate that the chemical sampling plan for dedicating unqualified material prescribed in Procedure 16562, " Material and-Parts Handling for i
Customer Defined Safety Related Components and Safety Related Items,"
Rev. A, provides reasonable assurance.that the dedicated material conforms to the procurement document requirements.
(Nonconformance 99900264/96-01-03)
D.
Criterion XVI, " Corrective Action,"' of Appendix 8 to 10 CFR Part 50, requires, in part, that for significant conditions adverse to quality measures will be established to assure that the cause of the condition is determined and corrective action taken to preclude repetition.
The identification of the significant condition adverse to quality, the cause of the condition, and the corrective action taken shall be documented and reported to appropriate levels of management.
Section 16'of the Dragon QA manual, " Corrective Action," Fifth Issue, Revision 0, dated March 6, 1996, contains requirements, for significant conditions adverse to quality, to determine the cause, take corrective action to preclude recurrence, document and report these actions to the President and Manager of the departments involved, and to take follow-up action to verify implementation of the corrective action by the QA Manager and document these actions Contrary to the above, Dragon had not documented the cause, corrective actions, reporting to management, or follow-up action to verify correct've action implementation, for a Dragon Instrument Isolation Manifold which had been shipped to Duane Arnold with the "line" and
" instrument" markings, stamped on the manifold body, reversed and where additional mismarked manifolds had been identified in the Duane Arnold and Dragon inventories.
(Nonconformance 99900264/96-01-04) 4 2
42
f i
Please provide a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555, 3
with a copy to the Chief, Special Inspection Branch, Division of Inspection l
and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date of the letter transmitting this Notice of Nonconformance.
This reply i
should be clearly marked as a " Reply to a Notice of Nonconformance" and should include for each Nonconformance:
(1) a description of steps that have been or vill be taken to correct these items; (2) a description of steps that have been or will be takan to prevent recurrence; and (3) the dates your corrective actions and preventative measures were or will be completed.
i i
1 i
1 Dated at Rockville, Maryland this 29th day of October 1996 3
43
i U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No:
99900264/96-01 Organization:
Dragon Valves, Inc.
Norwalk, CA 90650
Contact:
Robert L. Snyder, Quality Assurance Manager (310) 921-6605 Nuclear indestry Manufactures instrumentation, manifold and specialty Activity:
valves Dates:
September 23 - 26, 1996 Inspectors:
Richard McIntyre, Senior Reactor Engineer, Team Leader Uldis Potapovs, Senior Reactor Engineer Bill Rogers, Reactor Engineer Approved by:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs 44
4 1
INSPECTION
SUMMARY
During this inspection, the Nuclear Regulatory Commission (NRC) inspectors reviewed the implementation of selected portions of the Dragon Valves, Inc.,
(Dragon) quality assurance (QA) program, and reviewed activities associated with Dragon's activities in supplying instrumentation, manifold, and specialty valves to NRC licensees in accordance with the NRC licensee purchase order requirements.
The inspection bases were:
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" American Society of Mechanical Engineers Boiler and Pressure Vessel Code,Section III During this inspection, two non-cited violations of NRC requirements were identified.
The violations are discussed in Section 3.1 of this report.
During this inspection, four instances where Dragon failed to conform to NRC requirements imposed on them by NRC licensees were identified.
These nonconformances are discussed in Sections 3.2, 3.3, and 3.4 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS There were no open findings from previous NRC inspections of Dragon.
3 INSPECTION FINDINGS AND OTHER COMMENTS 3.1 10 CFR Part 21 Proaram The inspectors reviewed Dragon Procedure 14501, " Procedure for Compliance to 10 CFR 21," Revision A, dated June 14, 1993, and representative documentation to verify implementation.
Procedure 14501 established the responsibilities and actions for the reporting of i
defects and the informing of deviations in accordance with the requirements of 10 CFR Part 21. The inspectors noted, and discussed with Dragon management, that the definitions of the terms deviation (non-conformance) and defect included in the Procedure 14501 were not in complete agreement with the definitions included in 10 CFR Part 21.
Section 10 CFR 21.21 requires that entities subject to the regulations adopt procedures to evaluate deviations within 60 days of discovery, provide interim reports to the KC within 60 days of discovery, and inform a director or responsible officer within five working days after the completion of an evaluation which determines that a defect or failure to comply exists.
Procedure 14501 stated that, for items which had been shipped to a customer and subsequently identified to contain a 2
45
deviation or defect, Dragon would perform an evaluation within forty-eight hours.
Upon completion of the evaluation, Dragon would notify the customer and the NRC of the deviation or defect and the Dragon disposition of the items affected and impact on equipment.
If the evaluation could not be performed within forty-eight hours, the customer would be notified of the deviation and would be requested to provide Dragon the current status of the delivered items.
The inspectors
)
concluded that although Procedure 14501 included requirements in the areas discussed in 10 CFR 21.21, the procedure's requirements did not agree with, or include all of, the requirements specified in 10 CFR J
21.21. However, the inspectors reviewed Dragon's 10 CFR Part 21 records and concluded that there had not been an occurrence where Dragon's j
actions were not in accordance with the 10 CFR 21.21 requirements.
l The inspectors reviewed Dragons's posting in accordance with the regulations in 10 CFR 21.6.
Dragon had posted 10 CFR Part 21 and Procedure 14501 but had not posted Section 206..The inspectors provided a copy of Section 206 which Dragon subsequently posted.
The failure to adequately proceduralize the requirements specified in -
10 CFR 21.21 and to meet the posting requirements specified in 10 CFR 21.6 constitutes violations of minor significance which are being treated as Non-Cited violations, consistent with Section IV of the NRC.
Enforcement policy (NUREG-1600).
3.2 Procurement and Utilization of Material for Acolications Which Are Reauired to Meet American Society of Mechanical Enaineers (ASME) Boiler i
and Pressure Vessel Code (Code).Section III reauirements.
Although Dragon's ASME Section III, Division 1 Quality Assurance Manual, Fifth Issue, Revision 0, dated March 6,1996, (QA' manual) contains -
provisions for upgrading unqualified source material, most of the material used in the manufacture of N-stamped valves supplied'by Dragon is purchased from ASME-accredited material organizations or suppliers that have been qualified by Dragon.
Since most Dragon-supplied valves i
i are in the 1/2 to 2-inch size range, much of the material used in their construction can be purchased under the ASME Code Section III, Subsection NCA, Subarticle 3800 (NCA 3800) small parts exclusion or the provisions of ASME Code,Section III, Subsection NX, Subarticle 2610 (NX I
2610) which exempt small product categories from most of the NCA 3800 L
quality requirements. The application of NX 2610, when utilized, is indicated on Dragon's certificate of compliance.
It was noted that the purchase orders of several utilities specifically prohibited the use of the NX 2610 exclusions and that most of the material in Dragon's stock J
was purchased to NCA 3800 requirements.
3.2.1 Proaram Controls.
4 The inspectors reviewed the QA manual, Section 7, " Control of Purchased j
Items and Service," Fifth Issue, Revision 0, dated March 6, 1996, which describes their practice for purchasing material for applications 3
46
required to meet ASME Code requirements.
Paragraphs 7.5.1 through 7.5.6 address the upgrading of unqualified source material.
Paragraph 7.5.1 states that material shall be purchased to Section III requirements or upgraded by Dragon as permitted by NCA 3855.5(a)(1) through (a)(4), and outlined in Dragon's Procedure 17375.
Paragraphs 7.5.2 and 7.5.3 require that a certification of the requirements which must be performed during the melting and of the heat i
analysis be obtained from the manufacturer of the source materitl, and that " Dragon Valves, Inc, shall perform or subcontract all other 4
requirements on each piece of the stock material." Paragraph 7.5.4 requirerDrag~on to perform a product analysis on each piece of source material.
In response to inspector questions concerning the above,. Dragon's QA d
manager stated that "all other requirements," as referenced in Paragraph 7.5.3 of the QA manual, pertained to the product analysis.
In summary, no mechanical testing was procedurally required to upgrade unqualified source material.
This position is consistent with Dragon's implementing Procedure 17375, Rev. N/C, " Procedure for Procurement and Inspection of Unqualified Source Material." For material used for valves over 3/4 inch nominal pipe size, Section 4, " Material Verification," requires i
only chemical testing each piece of the source material.
The inspector noted that the text of paragraph NCA 3855.5(a)(3) requires the upgrading organization to perform or subcontract "all other 4
reouirements of the material soecification on each piece of unqualified source material." These requirements typically include tensile testing and may also include hardness testing, bend testing, hydrostatic testing, impact testing and other tests.
Failure to establish source material upgrading requirements in accordance with the provisions of NCA l
3800 was identified as an example of Nonconformance 99900264/96-01-01.
Dragon's quality assurance manager stated that Dragon had not upgraded any unqualified source material exceeding the NCA 3800 small parts exclusion provisions since the implementation of this procedure, therefore the incorrect requirements contained in that procedure should have no _effect on Code compliance of supplied items. A review of a sample of material procurements during this inspection tends to support this conclusion.
The inspector also noted that paragraph 2.1 of Dragon's Procedure 17375, which is referenced in Section 7 of the QA manual for procurement of unqualified source material, states that material may be purchased from
(
any suppliers on the Unqualified Source Material Approved Vendor List.
However, the basis and use of this list was not discussed in the QA manual and the list had not been corgleted and was not available for review.
The QA manager produced another procedure, identified as 17053, Rev. A, " Procedure for Procurement and Inspection of Unqualified Stock Material" which had the same title and appeared similar to Procedure 17375, but was not referenced in the QA manual.
Procedure j
4 47
i l
17053 did not reference the Unqualified Source Material Approved Vendor List (which was referenced l'n Procedure 17375), but stated that material i
may be purchased from any supplier.
Both procedures had been approved on February 29, 1996, and were, apparently, in active status.
Inadequate control of material procurement procedures and failure to prepare the Unqualified Source Material Approved Vendor List were identified as-examples of Nonconformance 99900264\\96-01-01 3.2.2 Procram Imolementation The material certification bases for approximately 15 valve orders from NRC licensees were reviewed to. assess the effectiveness of Dragon's QA program implementation.
The results of this review are discussed in the i
i following paragraphs:
Nebraska Public Power District Purchase Order (P0) 962080, dated September 03, 1996, for one 3/4 inch, 2500 psi needle valve, part number 500F0511SW-875.
The PO requested compliance with ASME Code,Section III, Class 2, 1983 edition including summer 1983 addenda, but waived Data Report and stamping requirements.
A Certificate of Compliance (C0C) was to be provided instead.
3 Review of the supporting documentation showed'that the valve disc material (SA 564, Type 630) was purchased from Carpenter Technology l
in Reading, PA.
Carpenter certification indicated that 492 bars of this material (heat 853543) were shipped to Dragon in 1b85.
Although Carpenter was on Dragon's approved vendor list.at the time of this procurement, their certification did not contain a quality system program statement identifying the revision and date of the applicable written Quality System Program as required by NCA 3800.
The file did contain a generic letter from Carpenter stating that material supplied by them has been manufactured, processed, and distributed in accordance with MIL-I-45208 and Carpenter quality assurance manual revision in effect at the time of manufacture.
Tennessee Valley Authority (TVA) PO 92NNA-76504 B, dated June 29, 1992, for several valves in accordance with TVA Standard Specification MEB-SS-10.19, "ASME Code Valves - 2 Inches and Smaller for TVA Projects."
Review of supporting documentation for a 1/2 inch, 1500 psi globe valve (Fig. 10617-8) showed that the valve body forgings were supplied by Ajax Forge, which was on Dragon's approved vendor list.
Ajax purchased the' starting material, SA 182, Type 316 bar (41 lengths) from ESCO Corp., a distributor, which, in turn, obtained the material from Altech Specialty Steel Corp. Neither ESCO or Altech certifications included a quality system program statement.
From the available documentation, it could not be determined whether the distributor and steel manufacturer had been audited by Ajax to assure heat traceability of this material or whether Ajax had 5
48
i i
upgraded this material.
Review of the supporting documentation for two additional 1/2 inch valves from the same P0, with bodies provided by the same forge shop, showed that Ajax purchased the starting material from Fry Steel Co. of Santa Fe Springs, CA, which, in turn, obtained it from Atlas Specialty Steel. Neither Fry nor Atlas certifications included a quality system program statement.
Review of. supporting documentation for 1 inch, Type 316 stainless steel tubing used in the manufacture of valves covered by the same P0 showed that Dragon had purchased this material in 1988 from a Los Angeles distributor, Tube Sales.
Tube Sales had obtained this material from Sandvik Steel, Scranton, PA. Although Sandvik Steel holds.an ASME quality systems certificate (QSC), and a copy of the certificate was in-the PO file, this material was apparently purchased as commercial grade tubing since Sandvik's certification did not reference their QSC or contain a quality systems program statement.
Sandvik certification identified the material as cold finished ASME SA 213, Type 316 steel (Heat 477898).
In addition to reporting the heat analysis and tensile test results, the certification. indicated. that flattening and flaring tests had been l
performed on this material.
The file also contained a letter from Tube Sales certifying that this material was annealed at 1900 degrees Fahrenheit and water quenched.
Dragon apparently did not j
perform additional upgrading on this material.
The inspector noted that, since the Sandvik certification did not reference their QSC or contain a quality systems program statement and since no upgrading was performed, this material did not fully comply with the requirements of NCA 3800.
The inspector also noted that there was no apparent basis.for the heat treatment statement provided by Tube Sales.
The program implementation review indicated that the majority of material used by Dragon has been purchased from qualified suppliers or certificate holders and, in most cases, sufficient documentation was available to. demonstrate compliance with the applicable requirements.
However, as discussed in the examples above, the inspectors identified several instances where Dragen had not verified supplier qualification.
Inadequate verification of supplier qualification was identified as Nonconformance 99900264/96-01-02 3.3 Procurement and Utilization of Material for Non-Code Acolications Egguirina Comoliance With 10 CFR Part 50. Accendix B Most of the valves supplied for safety-related nuclear applications have been supplied as N-stamped components with procurement and manufacturing activities controlled in accordance with Dragon's ASME QA manual.. In some cases, as discussed in partyaph 3.2.2, licensees have wabed the
' Code stamping requirement.
Under these conditions, the valves are supplied under the same QA program, except that the authorized r.uclear inspector is not involved in the manufacturing process, a Code data report is not provided, and the valve is not Code stamped.
6 49
8 l
JAlthough Dragon's QA manual does not address manufacture of safety-related (10 CFR Part 50, Appendix B) non-Code valves they have i
developed a procedure (No. 16562,;" Material and Parts Handling for Customer Defined Safety Related Components and Safety Related Items,"
Rev. A, Dated 11/03/93).which is apparently intended to control such
- activity, This procedure identifies the safety-related and non-safety-related component parts and defines the procurement, dedication, and processing requirements for each category.
According to this procedure, 4-material for safety-related valve components can be procured in accordance with Dragon's QA manual or purchased to ASME or ASTM
~
specifications and-dedicated.
The prescribed dedication process involves cutting a sample from each size and heat of the material and
)
having chemical analysis performed by a qualified vendor.
Dragon's-QA manager stated that no valves had been supplied to NRC licensees under
[
this procedure.
Although Procedure 16562 referenced Electric Power Research Institute (EPRI) report NP-5652, " Guideline for the Utilization of Commercial Grade Items in Nuclear Safety-Related Applications," the inspector noted that it did not follow the EPRI dedication methodology (identification and-verification of critical characteristics).
The procedure also did not assure material traceability or provide a basis for assurance that the prescribed chemical sampling (one sample from each size and heat of 4
material from a non homogeneous product lot) was adequate to provide-reasonable assurance that the material complied with the specification 1
requirements.
Failure to establish the basis for chemical sampling was 1
identified as Nonconformance 99900264/01-03.
3.4 Corrective Action Proaram and Implementation The inspectors reviewed the QA manual, Section 16, " Corrective Action,"
Fifth Issue, Revision 0, dated March 6,-1996, and representative documentation to verify implementation.. Section h established the responsibilities and actions for addressing conditions adverse to quality.
Section 16 of the QA manual stated that conditions adverse -to quality shall be identified promptly and corrected within a documented time
.be determined and corrective action taken to preclude recurrence.
In
~
period.
For significant conditions adverse to quality, the cause shall addition, identification, cause and corrective actions shall be documented and reported to the President and Manager of the departments involved and follow-up action shall be taken to verify implementation of the corrective action by the QA Manager and documented.
The inspectors discussed-a recent occurrence in which a Dragon Instrument Isolation Manifold had been shipped to the Duane Arnold nuclear power plant with the "line" and " instrument" markings, stamped on the manifold body, reversed.
The situation had been identified in April 1995 by Duane Arnold during the testing of a replacement instrument isolation assembly.
Licensee documentation indicated that if the isolation assembly had been installed with the manifold reversed, 7
50
(
(
the isolating valves would have closed with process pressure below the valve seat instead of above and the equalizing valve would have been located on the process side of the manifold instead of the instrument side. As a result, if the manifold was isolated to calibrate the.
corresponding i'nstrumentation and the equalizing valve opened, this could have caused a reduction in the differential pressure between the two process lines which could have affected other instrumentation fed by the same lines.
Dragon mana o..ent recalled the event and the related discussions with v
Duane Arnold and that, in addition to the mismarked manifold in the i
i assembly, Duane Arnold had located another one in inventory. Dragon had l
instructed Duane Arnold to restamp the manifold and had reviewed the t
Dragon inventory and discovered additional mismarked manifolds.
Dragon determined that the manufed.. ing technician who marked the manifolds had used the correct drawings but had made an error when applying the markings and the inspector who performed the final inspection of the manifold had failed to find the mismarkings.
Further discussion with Dragon and review of the records indicated that Dragon had not documented the condition, had not entered the condition into the'Drgen corrective action program, had not documented any-corrective actions, and had not documented any actions to preclude recurrence.
The inspectors concluded that shipping mismarked manifolds to'a licensee was a significant condition adverse to quality and that Dragon had not met the requirements of Section 16 of the Dragon QA manual to determine the cause, take corrective action taken to preclude recurrence, document and report these actions to the President and Manager of the departments involved, and to take follow-up action to verify implementation of the corrective action-by the QA Manager and document these actions.
Failure to perform and document the required actions for a significant condition adverse to quality was identified as Nonconformance 99900264/01-04.
3.5-Indoctrination and Trainino The inspectors reviewed the QA manual, Section 2, " Program," Fifth Issue, Revision 0, dated March 6, 1996, which described the indoctrination and training requirements for Dragon inspection and test personnel and auditors.
Section 2 addressed job functions such as receiving, in-process, and final inspection; hydrostatic, seat leakage, and functional testing; welding; non-destructive examination; and internal and external vendor audits.
The inspectors reviewed documented training records for various personnel for the job functions described above.
The inspectors also reviewed the training files and qua'ification records for lead auditors, welders, non-destructive examination subcontractors, and the various inspection and test personnel. The inspectors verified that a training plan and schedule was implemented for the job functions examined and that all training that had been planned and scheduled had been completed i
8 51
for the last three years. The inspectors concluded that the training files contained all pertinent training records and qualification documents required by the Dragon QA program.
3.6 Internal Audits The inspectors reviewed the QA manual, Section 18, " Audits," Fif th Issue, Revision 0, dated March 6, 1996, which described the system used for determining the overall effectiveness of the Dragon QA program.
Section 18 requires that all sections of the QA manual be auditeo annually by a lead auditor using approved checklists.
The various QA manual section audits are performed by different lead auditors within Dragon so that no section of the QA manual is audited by an individual who has any responsibility within that area. The inspectors verified i
that the internal audits for all sections of the QA manual had been performed for the last three years.
The inspectors concluded that the audits appeared to be effective in reviewing all aspects of program implementation.
3.6 Entrance and Exit Meetinas In the entrance meeting on September 23, 1996, the NRC inspectors discussed the scope of the meeting, outlined the areas to be inspected, and esiablished interfaces with Dragon management.
In the exit meeting on Sertember 26, 1996, the inspectors discussed their findings and Conce'ns.
1 l
9 52
p ero O
UNITED STATES j
j NUCLEAR REGULATORY COMMISSION e
't WASHINGTON, D.C. 205S5-0001 b.,,,,*
Novanber 25, 1996 Mr. J. R. Bohart Acting President Framatome Cogema Fuels 3315 Old Forest Road P.O. Box 10935 Lynchburg, VA 24506-0935
SUBJECT:
NRC INSPECTION REPORT NO. 99900001/96-01 i
Dear Mr. Bohart:
On April 17, 1996, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection of the Framatome Cogema Fuels (FCF), formerly the Babcock and i
Wilcox Fuel Company, activities at the engineering and fabrication facilities in Lynchburg, Virginia. This letter transmits the report of that inspection.
During this inspection, the NRC inspectors did not identify any instances where activities controlled by quality assurance program 56-1177617-03,
" Quality Assurance Program for B&W Fuel Company," May 30, 1994, failed to meet NRC requirements for the areas inspected.
The enclosed inspection report contains a detailed discussion of the areas examined during the inspection and our conclusions.
The inspectors did, however, observe certain FCF activities that were considered weaknesses that could affect quality.
Of those described in the enclosed report, the inspectors considered the most significant weaknesses to be (a) the lack of detailed design instructions and (b) the lack of an overall multi-disciplinary review of the Three Mile Island Unit 1 Cycle 10 reload core design process by FCF to identify and evaluate the synergistic effect of design changes. To facilitate our closure of this matter, please provide a written explanation of the steps that have been or will be taken to address these weaknesses.
Please send the response to the NRC, ATTN: Document Control Desk, Washington, i
D.C. 20555, and a copy, at the same address, to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation within 30 days of receipt of this letter.
For the nonconformances cited in previous reports of NRC's inspections of the Babcock and Wilcox Fuel Company (NRC reports 99900001/90-01 and 99900001/91-01), the inspectors determined that tr e corrective actions taken were adequate and affectively implemented. The enclosed report documents closure of those nonconformances.
53
l J. Bohart In accordance with Section 2.790(a) of Title 10 of the Code of Federal Reaulation (10 CFR), a copy of this letter and its enclosure will be placed in the NRC Public Document Room and made available to the public unless you notify this office by telephone within 10 days of the date of this letter and submit a written application to withhold the information contained therein.
Such application must be consistent with the requirements of 10 CFR
)
2.790(b)(1). Your response to this letter and its enclosure is not subject to the clearance procedures of the Office of Management and Budget, as required by the Paperwork Reduction Act of 1980, Public Law No.96-511.
Should you have any questions concerning this inspection, we will be pleased to discuss them with you. Thank you for your cooperation during this process.
Sincerely, L
Robert M. Gallo, Chief Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation Docket No.:
99900001
Enclosure:
Inspection Report 99900001/96-01 i
i 54
U.S. NUCLEAR REGULATORY CONNISSION i
0FFICE OF NUCLEAR REACTOR REGULATION Report No.:
99900001/96-01 Organization:
Framatome Technologies, Inc. (FTI)
Framatome Cogema Fuels (FCF)
Lynchburg, Virginia
Contact:
R. L. (Ronnie) Gardner Manager, Quality FCF Nuclear Industry Activity:
FCF provides pressurized-water reactor (PWR) reload core designs, safety analysis, and licensing, fuel assemblies, and fuel-related core components to the U.S. nuclear industry.
Dates:
March 18, 1996 - April 17, 1996 Inspectors:
Steven M. Matthews, DISP /PSIB David H. Brewer, DISP /PSIB Dr. John F. Carew, Brookhaven National Laboratory Carl J. Czajkowski, Brookhaven National Laboratory Geoffrey R. Golub, DSSA/SRXB Rodney L. Grow, Par 4 meter, Inc.
Edward D. Kendrick, DSSA/SRXB Kamalakar R. Naidu, DISP /PSIB Billy H. Rogers, DISP /PSIB Approved by:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs I
55
I I
INSPECTION
SUMMARY
From March 18, 1996, through April 17, 1996, the U.S. Nuclear Regulatory Commission (NRC) conducted an inspection of Framatome Cogema Fuels (FCF)
[formerly the Babcock and Wilcox (B&W) Fuel Company (BWFC)] activities at the engineering and fuel fabrication facilities in Lynchburg, Virginia.
The inspection bases were:
General Design Criterion (GDC) 10, " Reactor Design," and GDC 12,
" Suppression of Reactor Power Oscillations," of Appendix A " General Design Criteria for Nuclear Power Plants," to Part 50, " Licensing of Production and Utilization Facilities," of Title 10 of the.(sig_p/
Federal Reaulations (10 CFR)
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to 10 CFR Part 50 10 CFR Part 21, " Notification of Failure to Comply or Existence of a Defect" Section 4.2, " Fuel System Design," of NRC NUREG-0800, " Standard Review Plan" (SRP), Revision 2, July 1981, and its Appendix A, " Evaluation of Fuel Assembly Structural Response to Externally Applied Forces,"
Revision 0 FCF quality assurance (QA) program 56-1177617-03, " Quality Assurance Program for B&W Fuel Company," May 30, 1994 During this inspection, the team noted weaknesses and observations concerning FCF activities that affect quality. Neither the weaknesses nor the observations described in this report require any specific action or written response by FCF.
A list of acronyms used in this report is provided on page 37.
2 STATUS OF PREVIOUS INSPECTION FINDINGS
)
2.1 Nonconformance 99900001/90-01-01 (CLOSED)
Contrary to Criterion V of Appendix B to 10 CFR Part 50 and Section 5.0 4
of the BWFC, Commercial Nuclear Fuel Plant (CNFP) procedure QC-506,
" Receiving Inspection - Lower End Fitting Mark C," Revision 1, October 27, 1980, CNFP failed to identify raised metal in the form of a burr adjacent to one grid pad bore hole on lower end fitting 15273-04.
FCF personnel stated that the burr had been removed and that prior toAll use, all lower end fittings were cicaned and reinspected for burrs.
receiving inspection personnel were instructed in the performance of visual inspection. 56
The inspectors reviewed FCF corrective actions taken and determined that 4
FCF had implemented actions to ensure that appropriate inspection standards were used to determine the acceptability of the lower end fittings.
2.2 Nonconformance 99900001/90-01-02 (CLOSED) 4 Contrary to Criterion V of Appendix B to 10 CFR Part 50, procedure QC-1414, " Retention and Storage of Quality Assurance Records," Revision 4, September 13, 1990, did not provide requirements for retention of radiographs and did not identify whether the radiographs and the reader sheets were quality records.
The inspectors reviewed FCF corrective actions taken and determined that FCF had revised procedure QC-1414, Revision 5, August 7, 1991, to classify reviewer reader sheets and radiographs as lifetime quality records.
j 2.3 Nonconformance 99900001/91-01-01 (CLOSED) i Contrary to Criterion V of Appendix B to 10 CFR Part 50, and Section 6.3.6 of procedure QC-802, " Fuel Rod Inspection (In process and Final),"
i Revision 12, February 21,1989, a 100% visual inspection was not performed as evidenced by felt cleaning plugs discovered in 22 fuel rods j
4 after visual examination and cleanliness acceptance were performed by quality control (QC).
The QC inspector responsible for performing the visual inspection was a
i counseled on several occasions to ensure that the inspector understood 1
that, by initialing the route card (RC), the QC inspector documented that the inspection had been completed to the requirements specified.
To ensure that instructions were fully understood and followed, the Manager of Inspection and each Inspection Foreman emphasized the importance of and the need for compliance with inspection instructions.
The inspectors reviewed FCF corrective actions taken and determined that FCF had implemented actions to ensure that appropriate inspection standards were used to accept fuel clad tubing. The inspectors noted that subsequent to the implementation of these changes, FCF changed its method for cleaning the inside diameter of the tubing and discontinued the use of felt cleaning plugs.
2.4 Nonconformance 99900001/91-01-02 (CLOSED)
Contrary to Criteria V, XV, and XVI of Appendix B to 10 CFR Part 50, CNFP did not issue a component discrepancy report (CDR) and a contract variation approval report (CVAR) to identify, document, evaluate, resalve, and process the noncon armance ider.tifi;d when felt cleaning plugs were found in 22 fuel rods. 57
A CDR and a CVAR were initiated in August 1991.
The Manager of QA conducted a series of meetings to identify and correct the causes of not i
reporting this nonconformance.
In this case, the true cause for not issuing a CDR was that the responsible personnel viewed the recovery operation as an in-process reject that scraped each fuel rod containing
)
felt cleaning plugs. During that time, the in-process reject practice 1
was allowed without issuing a CDR.
The current practice requires that a CDR be issued if a part or process deviated and the resolution of that J
deviation was not covered by a procedure or RC.
1 The inspectors reviewed FCF corrective actions taken and determined that FCF had implemented appropriate actions to ensure that CDRs and CVARs would be issued as required.
2.5 Nonconformance 99900001/91-01-OLLCLOSED) i Contrary to Criterion V of Appendix B to 10 CFR Part 50, procedure MA-450 did not provide guidance to the operator for the extent of QC inspection required or what notification was to be given to QC to ensure that the visual examination for cleanliness was performed.
Additionally, MA-450, Revision 16, March 8,1991, and QC-802, Revision 13, April 29, 1991, did not provide the extent of the inspections to be performed by the operator or the QC inspector.
The inspectors' review of the procedures referenced determined the following:
Procedure MA-450 was at Revision 23, December 18, 1995, and the applicable paragraphs (4.2 and 4.3) were responsive to the concerns.
Procedure QC-802 was at Revision 20, August 1, 1995, and paragraph
=
6.3.5 was responsive to the concerns.
2.6 Unresolved Item 99900001/91-01-04 (CLOSED)
The inspectors determined that BWFC had initiated actions to identify the root cause for the felt cleaning plugs left in loaded fuel rods. An evaluation in accordance with Section 21.21 of 10 CFR Part 21 may also be required.
Pending receipt by NRC of a written response ;ontaining the evaluations, this issue was considered an unresolved item.
The inspectors reviewed the FCF's root cause evaluation 51-1205252-00 and concluded that FCF had adequately evaluated the event.
3 INSPECTION FINDINGS AND OTHER CONNENTS This inspection included an evaluation of the FCF fuel operations management, engineering, and quality staff, and certain activities for providing reactor licensees PWR neutronic design information, reload core designs, fuel assemblies, fuel related core components, and fuel related field services. 58
i The emphasis of the inspection was on the fuel design and engineering, i
organization, management, training and qualifications, corrective. actions, and compliance with the requirements of 10 CFR part 21.
j 3.1 10 CFR Part 21 Procran p
s.
Inspection Scope The inspectors reviewed FCF compliance with the NRC requirements in 10 i
CFR Part 21.
The FCF procedures implementing the requirements of 10 CFR Part 21 were BWNT-1707-01, " Processing Safety Concerns," Revision 24, April 13, 1994, and BWNT-1707-03, " Reporting of Defects and Failures to Comply (10 CFR 21)," Revision 1, 1994.
The inspectors reviewed selected 3
10 CFR Part 21 evaluations, known as preliminary report of safety i
concern (PSC).
e l
b.
Observations and Findings l
b.1 PSC 5-94 1
1 PSC 5-94 concerned the fuel peak cladding temperature (PCT) during the j
reflood stage of the large break loss-of-coolant accident (LOCA). The j
inspectors discussed the PSC with the LOCA and safet.y analysis task and supervising engineers. The PSC addressed the use of nonconservative i
values for (a) the inlet enthalpy in the core thermal analysis and (b) i the assumed initial inventory in the core flood tank (CFT).
The j
nonconservative inlet enthalpy was due to the use of a coarse time step in the reflood analysis, and an inadequate averaging of the nodal i'
enthalpy following node dryout and rewet.
J The nonconservative initial CFT inventory was first observed on September 30, 1994, during the development of the new FCF RELAP 5/M00 2
-B&WmethodologyforapplicationtotheMARK-BilLOCAanplysis. At this time the licensed LOCA analysis was based on CRAFT 2.
A subsequent CRAFT 2 analysis confirmed the nonconservatism in the assumed initial CFT inventory.
These nonconservatisms in the LOCA analysis resulted in a reduction in the LOCA'kW/ft limits.
FCF evaluated the effect of the reduction and determined that, because of available margin in the operating limits. no change was required in the FCF determined LOCA-based operating limits.
During these discussions, the inspectors noted that four months had elapsed from the time the LOCA analysis initial CFT inventory nonconservatism was first identified (September 1994) and the date NRC
'" CRAFT 2 - Fortran Program for Digital Simulation of Multinode Reactor Plant During Loss of Coolant," J. J. Cudlin, B&W, October 1982 59
2 received notification. Although the inspectors recognized that the initial identification was made using the unlicensed RELAP 5/ MOD 2 - B&W methodology, the inspectors concluded that a final confirmation of this nonconservatism could have been made using existing CRAFT 2 models.
Procedure BWNT 1707-01 required that the NRC be notified within 60 days after the PSC is initiated.
PSC 5-94 was initiated on November 28, 1994, and was reported to the NRC on January 27, 1995; within the 60 day requirement. However,Section VIII, of this procedure also required, in part, that a responsible engineer initiate a PSC for any concern discovered during, and which is related to, the design and analysis that J
has or may have safety implications.
The inspectors determined that approximately 60-days had elapsed between the time the engineer had reasonable indication that the emergency core i
cooling accident (ECCS) LOCA analysis was nonconservative and the time that the PSC was initiated.
j l
b.2 PSC 1-95 The inspectors discussed the identification and status of PSC 1-95 with the FCF responsible engineer.
PSC 1-95 identified the concern regarding a possible return to critical during the recovery stage following a small break LOCA (SBLOCA). During this period the plant may spend some i
time in the boiler / condenser mode and may accumulate deborated water in j
the reactor coolant pump suction piping and downcomer.
If, after this accumulation, an operator performs a pump bump, or under natural circulation, this deborated water enters the core, the core boron concentration may not be sufficient to ensure suberiticality.
(This concern was also the subject of PSC 8-81 which was resolved in 1985.)
i PSC 1-95 was a result of several analyses which indicated that the plant conditions assumed in the 1985 resolution of PSC 8-81 are no longer i
validforB&WdesignedPWRs. The NRC was notified of PSC 1-95 by letter.
FCF recently issued a report' on the initial Phase-I investigation of the issues associated with PSC 1-95. This study was performed for the B&W Owners Group, and was not intended as a detailed evaluation but rather as a path finding activity.
The study proposed a follow-on Phase-II activity including a set of detailed transient analyses and an evaluation of special effects for resolving PSC 1-95.
'B&W letter to NRC, " Report of Preliminary Safety Concern Related to Large Break LOCA ECCS Analysis," January 27, 1995 3B&W letter (0G-1521) to NRC, "Repor; of Potential Core Criticality Concern," June 13, 1995
'FCF 47-1244436-00, " Preliminary Report to the B&W Owners Group on PSC 1-95 Investigation," January 1996 60
4 i
i i
The inspectors concluded that the preliminary Phase-1 evaluation adequately included the major issues associated with PSC 1-95.
1 c.
Conclusions Based on its review of PSC 5-94, the inspectors concluded that FCF could have initiated the PSC before November 28, 1995, and that FCF's failure to evaluate the PCT during LOCA concern when indications of the problem j
were first identified in September 30, 1994, was considered a weakness j
in the implementation of FCF's program.
3.2 Guality Assurance The inspectors reviewed FCF's QA program described in 56-1177617-03, i
" Quality Assurance Program for BW Fuel Company," May 30, 1994. The i
inspectors observed that the_FCF Manager of Quality had the responsibility, authority and organizational freedom to enforce and monitor compliance with the program.
The inspectors concluded that the QA program was generally implemented
)
in an acceptable manner for the areas evaluated during this inspection.
3.3 Reload Core Desion. Safety Analysis and Licensina Process a.
Inspection Scope To evaluate FCF's reload core design, safety analysis, and licensing processes, the inspectors reviewed the performance, interfaces, and documentation of the reload process. The reload core design, analysis, and licensing activities consisted of determining the licensee requirements and the preliminary fuel cycle design; performing the steady-state and transient neutronic, thermal-hydraulic and fuel performance analysis; and updating the cycle-specific reload licensing analysis and the process computer databank.
b.
Observations and Findings b.1 Licensee Requirements The inspectors observed that the. process started with the FCF/ licensee reload fuel contract and the FCF reload project manager (PM) interacting with the licensee to define the requirements for a new reload core.
The i
licensee specified the fuel assembly type and options for the new l
reload.
Preliminary operating requirements and design data were documented in preliminary energy requirements.
The inspectors determined this was an iterative process between FCF and the licensee and found that the i.arrespondence Jc.umented the request for the licensee to approve the requirements and design data. 61
The PM created a contract requirement document (CRD) which outlined the scope of supply and special licensee requirements and used a document release notice (DRN) to release the CRD to FCF fuel engineering.
The inspectors concluded that FCF's licensee requirements process did not use formal checklists or computerized design data.
The inspectors, however, did not observe any instances where the lack of formality led to a design error.
b.2 Reload Core Design Process The inspectors reviewed the reload core design process and observed that the preliminary and scoping fuel cycle designs were based on the preliminary operating requirements specified by the licensee. This initial design effort determined the preliminary reload batch size and enrichment; the analysis utilized the two-dimensional (2D) lattice physics code (CASMO 3) results and the three-dimensional (3D) core simulation calculations from the NEM0 computer program. This initial analysis was documented in the preliminary fuel cycle design (PFCD) report and this information was released via the DRN process.
According to FCF, the reload licensing analysis task engineer (RELATE) was a key individual ir, the reload core design process and was responsible for maintaining the technology for the task.
The inspectors were also told that the reload analysis and licensing services (RALS) inspectors maintained the quality of the service and ensured timely interaction with the licensee.
The cognizant RELATE generated the fuel cycle design requirements (FCDR) following receipt of the CRD. Based on the requirements of the FCDR, the final fuel cycle design (FFCD) was produced and documented in a report to the licensee.
The FFCD analysis consisted of CASMO 3 and NEMO calculations which were documented in calculation packages.
The inspectors reviewed prior cycle FFCDs against the new reload FFCD calculation packages.
From its review, the inspectors determined that the process was controlled by administrative procedures.
FCF had not developed detailed engineering instructions. The inspectors also noted that the calculation packages did not contain documentation identifying what the reviewer checked or verified (e.g., no check marks or checklists).
The inspectors made similar conclusions regarding the reload safety analysis process discussed in Section 3.3.b.3 of this report.
b.3 Reload Safety Analysis Process The reload safety analysis process was performed to ensure that the updated version of the final safety analysis report (FSAR) remained val:J for the reload core design.
ihe accidents analyzed in the FSAR were evaluated to ensure that the reload core thermal performance during i r 62
4 I
Rypothetical transients was not degraded and that site doses were below i
the acceptance criteria of 10 CFR Part 100.
In addition, the reload safety analysis process included the determination of the core operating i
limits and reactor protection system (RPS) setpoints.
The reload safety analysis consisted of both the non-LOCA and LOCA analyses. The safety analysis used the flux / flow limits determined by i
the NEMO}hermal-hydraulicanalysis,thepowerpeakinglimitsdeterminedby in the nuclear analysis, and the RPS offset limits determined in i
the NEMO maneuvering analysis.
In the non-LOCA analysis, the core 3
parameters that impact the FSAR transients were compared to the plant i
licensing basis values (LBVs) to ensure that the licensing basis 3
remained valid for the reload core. The specific parameters evaluated included the moderator and Doppler reactivity coefficients, initial boron concentration and inverse boron worth, maximum ejected and dropped i
rod worths at hot-full-power (HFP) and rod group worth at hot-zero-power j
(HZP).
4 l
The transients considered included both overheating and overcooling i
events. The overheating events included moderator density dilution, rod withdrawal and ejection, loss of flow, loss of power, reactivity change and the startup event. The overcooling transients considered were the j
steam line break, cold water event and the dropped or stuck rod event.
4 3
In the LOCA analysis, the linear heat rate (LHR) effects of fuel l
prepressure, enrichment, and plenum volume, and the cycle-burnup i
dependenceofthefuelrodpressureandtemperaturewereevag'ated.
The i
fuelrpdparameterswerecalculatedwithcomputercodesTAC0 or i
GDTACO in the case of gadolinia (Gd) loaded fuel.
Plant modifications that can affect either the reload LOCA analysis (e.g., ECCS assumptions) or the plant transient analyses were reviewed against the licensing
{
basis analysis.
These changes were eit5er verified to be within the l
LBVs or a re-analysis was performed to demonstrate their acceptability.
The power / imbalance / flow' trip setpoints and safety limits were i
calculated using the cycle-specific offset limits determined in the NEMO i
maneuvering analysis.
i i
5BAW-10180-A, "NEM0-Nodal Expansion Method Optimized," Revision 1, March 1993 4
'BAW-10141P-A, " TACO 2:
Fuel Performance Analysis," Revision 1, June 1983 7BAW-10162P-A, " TACO 3:
Fuel Pin Thermal Analysis Code," Revision 1, November 1989 sBAW-10184P, "GDTACO, Urania-Godolinia Thermal Analysis Code," May 1992 t I 63
=
The reload evaluation also included a NEM0 reactivity analysis to determine the refueling boron concentration required to ensure that the core remains suberitical following a refueling boron dilution event.
1 The NEMO analysis also provided core reactivity requirements for determining (a) the minimum borated water storage tank (BWST) concentration to ensure that the core remains 1% shutdown following a LOCA and (b) the minimum volume of borated water in the BWST and the boric acid storage tank to independently ensure that the core can be held at cold shutdown.
The plant exclusion area and low-population zone boundary doses were evaluated for each reload to insure they were within the 10 CFR Part 100 l
acceptance criteria. The reload fuel and coolant radionuclide inventories were calculated and compared to the LBVs. A reanalysis was performed to demonstrate the acceptability of the boundary doses if the reload values were outside the LBVs.
The inspectors found that the reload safety analyses were documented in formal cycle-specific calculational files. The safety analysis calculations were reported in calculational summary sheets.
From its review of the reload safety analysis process, the inspectors determined that if a core parameter value was not bounded by the LBV for a given transie.nt, then the value was justified by reanalysis of the transient.
This reanalysis may consist of a simple sensitivity calculation or a complete reanalysis of the transient.
The inspectors reviewed prior cycle calculational files against the new reload calculation packages.
From its review, the inspectors determined that the documentation and analysis technique tended to be guided by the i
prior cycle (s) documentation.
The inspectors observed that the process was controlled by administrative procedures and that FCF had not developed detailed engineering instructions. The inspectors also noted that the calculation packages did not contain documentation identifying 4
what the reviewer checked or verified (e.g., no check marks or checklists were used).
Based on these observations and similar observations made by the inspectors during their review of the reload core design process (see Section 3.3.b.2 of this report), the inspectors examined numerous calculation packages for the reload design process and the reload safety analysis process (see Section 3.3.b.4.(1)-(4) of this report).
I b.4 Reload Licensing Process The reload licensing process defined the specific tasks to be performed, licensing schedule, task coordination and interfaces, and required design reviews. The reload PM provided the primary licensee interface for the reload. The reload licensing process consisted of 18 tasks which were identified numerically (e.g., task 4 was the ECCS analysis and task 11 was the nuclear analysis).
A task engineer was assigned to each task and was responsible for the basic technology and methods
-g-64
employed in carrying out the task analysis.
The task analyses were documented in a series of FCF reports.
The FCOR and the site sup document were the primary output of the reload licensing process. port Each of the following reload designs was reviewed by the inspectors with the PM, the RELATE, and the engineers who performed the reload core design and the reload safety analyses.
(1)
Three Mile Island 1 The FCF licensing analysis provided the technical justification supporting Cycle 10 operation of the GPU Nuclear Corporation (GPUN), Three Mile Island Unit 1 (TMI-1). The general reload licensing information consisted of the fuel assemblies, burnable poison rod assemblies (BPRAs), and reactor rod cluster control assemblies (RCCAs) for TMI-l Cycle 10 and was specified in the GPUN/FCF reload fuel contract.
The latest safety analysis and plant systems input for the Cycle 10 reload evaluation were provided in the plant FSAR.
The reload data checklist specified the initial number of fuel assemblies and plant operating conditions (e.g., flows and temperatures).
These requirements were finalized based on the FCF fuels engineering preliminary fuel i
cycle analysis of the cycle-length, number of fuel assemblies, axial blanket, and Gd zoning.
The reload licensing analysis was initiated by the release of the reload licensing schedule, which specified the tasks to be performed, schedule, and deliverables.
The THI-l Cycle 10 licensing analysis consisted of the standard FCF task-labeled analyses.
As part of the evaluation of the FCF reload licensing process, the inspectors reviewed the nuclear analysis with the supervising engineer, the RELATE, and the engineer responsible for the analysis.
The inspectors concluded that the calculations were complete and well documented.
The inspectors found that FCF's QA program for the use of computer codes in the reload licensing process required that approved code versions be used in licen>ing analyses or, when approved versions were not used, a justification must be provided in the calculation files.
During its review of nuclear analysis 32-1218970-01 and l
32-1219600-01, theinspectorsnotedthatgnapprovedversionsof j
NEMO (NEMO 3.lRS and NEMO 4.lRS) and SHUF (SHUF 3.lRS) were used
{
in the nuclear analysis.
However, no justification for the use of these unapproved versions was documented as required by the FCF QA program.
l
'The SHUF program is used to shuffle the fuel isotopic concentrations between core locations.
l l i 65
When the inspectors brought this omission to the attention of the FCF staff, revisions to these calculation files were provided that contained a justification for the use of unapproved computer codes on the basis that the errors in the unapproved versions had no impact on the results of the nuclear analysis.
The TMI-l Cycle 10 reload safety analysis was reviewed by the inspectors with the responsible task engineer and unit manager of Framatome Technologies, Inc. (FTI). The safety analysis compared the Doppler and moderator temperature coefficients (MTC), initial boron concentration, maximum ejected and dropped rod worths, and rod group worths to the bounding values used in the FSAR. The values used in the FSAR were shown to bound the values for TMI-l Cycle 10. The safety analysis demonstrated that the generic set of LOCA analyses bounded the TMl-1 Cycle 10 reload.
f The TMI-l Cycle 10 reload fuel rod analysis was reviewed by the inspectors with the responsible fuel rod analysis engineer.
The fuel rod mechanical design analysis used the power distribution and power history data determined in the fuel cycle design analysis to evaluate the cladding creep collapse and the stress and strain for the Cycle 10 fuel. The design data for each of the Cycle 10 batches was taken from the applicable documents list (ADL).
The axial locations of the fuel stacks, control rod poison stacks, burnable poison stacks and the axial blankets were determined from the core and vessel specifications and drawings.
The stress parameters for each of the four Cycle 10 fuel rod designs (MARK-B8, MARK-B8V, MARK-B9 and MARK-B9-Gd) were bounded by conservative licensing basis analyses. The creep collapse time for both the MARK-88 and MARK-B9 fuel assemblies was bounded by a previous. generic analysis.
As part of the inspectors' review of these generic analyses, the fuel rod growth calculation for the MARK-B9 fuel (32-1202547-00) was reviewed in detail. The growth analysis included a statistical combination of growth tolerances and used a conservative fluence growth model. The documentation was complete and included the necessary verification reviews.
The 28 MARK-B9-Gd fuel assemblies included Gd fuel re. The fuel s
mechanical analysis (32-1222994-00) concluded that the creep collapse analysis for the Gd fuel was bounded by the generic analysis performed for the MARK-89-Gd fuel (32-1219532-00). The generic analysis was reviewed and it was noted that the analysis employed an unapproved version of GOTACO.
However, upon the inspectors' request, the FCF staff provided a subsequent creep collapse analysis (32-1224702-00) which demonstrated that there was no significant difference in the i
results of the calculations performed by the approved versus unapproved versions of GDTACO. 66
The inspectors concluded that the documentation of all analyses reviewed complied with the FCF documentation and verification requirements.
(2) sequoyah 1 The inspectors selected the Tennessee Valley Authority (TVA),
Sequoyah Unit 1 Cycle 9 reload for review because Sequoyah was a non-B&W designed plant (Westinghouse).
Cycle 9 was the first reload provided by FCF for Sequoyah. The inspectors were also concerned about the handling of design inputs for a non-B&W plant and the analysis to support a mixed core reload.
The inspectors were also told that the fuel design for Sequoyah, the Mark-BW, had been used in fuel for Catawba and McGuire.
(FCF had previously supplied fuel and LOCA analysis for McGuire, a sister plant to Sequoyah.)
The inspectors found that this reload was not a typical FCF reload in that a first-of-a-kind (FOAK) analysis was performed prior to initiating the typical cycle specific analysis.
The inspectors found that the F0AK effort addressed changes to the FSAR, technical specifications, core operating limits report (COLR) and also produced the topical report BAW 10220P, " Mark - BW Fuel Assembly Application for Sequoyah Nuclear Units 1 and 2," Revision 0, March 1996.
The inspectort were told that the FOAK analysis addressed the Sequoyah plant specific design and focused on the plant safety analysis.
The inspectors observed that the topical report documented the LOCA and non-LOCA transient analysis, thermal-hydraulic analysis, and mechanical and containment analysis.
The neutronics analysis was largely addressed in prior Mark-BW documents.
The topical report documented the licensing basis for FCF fuel in the Sequoyah reactor and identified the bounding physics parameters for the cycle-specific reload analysis. The inspectors found that the Sequoyah design inputs were documented 4
via correspondence between FCF and the licensee and the documentation showed that the inputs had been verified by the licensee.
The inspectors found that the PFCD had been completed and FCF nad received the final energy requirements from the licensee (TVA).
The inspectors reviewed the list of Sequoyah reload deliverables and contract documents list.
The inspectors concluded that the handling of design data inputs and the documentation of FCF activit'les through the preliminary design were consistent with the FCF administrative manual procedures.
To assess the analysis process the inspectors examined calculation package 32-1228786-00, "Sequoyah Catchup and Accident Analysis."
The inspectors found that this package documented the benchmarking of the CASMO 3/NEMO neutronics model for Cycles 1 to 8. 67
i i
Comparisons to INCORE (computer code used by TVA for technical specifications surveillance) power distributions, measured rod worths, isothermal temperature coefficients, and boron letdown values were documented. The calculation package also documented the main steam line break analysis with comparisons to the FSAR i
results. The inspectors observed that this calculation package was prepared consistent with the FCF administrative manual j
procedure BWNT-0402-01, " Preparing and Processing BWNT Calculations," Revision 29.
l The inspectors concluded that the Sequoyah reload package was adequately documented, the mixed core issues addressed, and the j
design inputs were adequately controlled and verified by the licensee.
j J
(3)
Crystal River 3 4
The inspectors reviewed the reload process for Florida Power l
Corporation, Crystal River Unit 3 (CR3), Cycle 11 with the reload PM and cognizant RELATE.
From its review of the CR3 Cycle 11 i
FCDR, FFCD, and reload report, the inspectors observed the 1
following unique characteristics about the CR3 Cycle 11 fuel i
design:
l (a) the longest cycle length to date (670 effective full power days (EFPD))
(b) the highest enrichment to date (4.96 w/o U235) l (c) first time use of Gd j
The inspectors found that the FFCD was documented in calculation t
package 32-1239620-00 and this package was reviewed by the l
inspectors relative to the Cycle 10 FFCD in calculation package 32-1224533-00.
The inspectors noted that the Cycle 10 and 11 j
analysis packages were prepared by the same individual and the same reviewer.
l Documentation for the MICBURN 3, CASMO 3, and NEMO calculations l
was' reviewed by the inspectors. Based on its review of several specific items including the use of Gd for the first time, the change in axial nodalization in NEMO for the new cycle, the choice of the K-effective bias or critical k-effective for NEMO and the cross section generation process, the inspectors concluded that j
the design process was effective for CR3.
l The CR3 Cycle 11 reload process was examined by the inspectors with the major focus on the engineering analysis which supported reload report BAW-2262, " Crystal River Unit 3 Cycle 11 Reload Report," January 1996. The rsload report was examined for each of the functional areas (fuel system, nuclear design, thermal-hydraulic, transient and technical specification /COLR).
' 68
i 1
?
The use of GDTACO for fuel performance, the application of the j
statistical core design methods for thermal-hydraulics, and the main steam line break analysis were among the topics reviewed by i
the inspectors.
The inspectors observed that the quality of the calculation packages varied significantly in terms of readability and in 2
overall content and in documentation of the verification process.
The inspectors concluded, however, that the reload analysis process for CR3 was adequate.
i l
(4)
Arkansas Nuclear 1
~'
The Entergy Operations, Incorporated, Arkansas Nuclear Unit 1 (ANO-1), Cycle 13 reload core was an FCF full-scope design.
The 8
design included 57 MARK-B8 and 60 MARK-B8ZL reload bundles, and 60 1
MARK-89 fresh ftel assemblies. Unlike TMI-1 Cycle 10, the fresh assemblies were loaded in a symmetric checker board pattern i
throughout the core that reduced the potential for crud deposition on the fuel (see Section 3.4.b.2 of this report).
i The inspectors reviewed the ANO-1 Cycle.13 nuclear analysis with the PM and RELATE. The specific Cycle 13 fuel loading did not require any modifications to the reload methodology and the i
standard FCF reload analysis codes and methods were employed. The i
inspectors reviewed the cesign files documenting the results of j
the Cycle 13 calculations for the on-line computer input (86-i 1232957-00), the physics data supporting the Cycle 13 site-operation (61-1232956-00), and the generation of key safety i
parameters for the evaluation of transients, reactivity effects and control rod worths (32-1232950-00).
While the inspectors review of the calculational files showed that, in several instances, unapproved computer codes were i
employed, the inspectors also noted that each of these applications were justified by an evaluation of the effects on the results of the nuclear analysis. The inspectors determined that the calculations were well-documented and the necessary i
verification was indicated.
c.
Conclusions 1
The inspectors discussed with each individual the technique used to perform certain design functions and learned that no design instructions i
existed for either performing the analysis or for review and t
verification of the analysis.
The inspectors found that the responsible individual follows the analysis process documented for the prior cycle.
i The inspectors also noted that t e PM, RELATE, M the RALS inspectors share the responsibility for the reload analysis. The inspectors determined that FCF performed these analysis processes based on the methodology documented in the prior cycle calculation packages in conformance with the guidance in the administrative manual procedures.
l 69
i
)
The lack of detailed design instructions to perform the reload I
l calculations could result in a nonconservative value that may not be apparent from following the previous cycle methodology as a template for the current reload calculations.
The inspectors, therefore, concluded that the lack of detailed design instructions was a weakness in FCF's reload design process.
However, the inspectors did not identify any instance where the lack of design instructions caused or contributed to an. error.
Specifically, j
the inspectors noted that for the CR3 Cycle 11 reload, the first time use of Gd was handled appropriately.
3.4 Fuel Ass 'ly ".nhanical Desian j
i a.
Inspection Scope The inspectors reviewed recent FCF experience and activities related to fuel assembly bow and the distinctive crud patter (DCP) following TMI-l Cycle 10 and CR3 Cycle 10 operation.
b.
Observations and Findings b.1 Fuel Assembly Bow According to FCF, all fuel assemblies experience some assembly bow.
Fuel assembly bow affects local margin to departure from nucleate boiling (DNBR) as a result of reduced cooling as the assembly-to-J assembly gap was decreased, and reduced the margin to LOCA F -limits as aresultofincreasedpowerpeakingastheassembly-to-assemblygapwas 1
increased.
Because of the stronger fast flux gradient on the core periphery, the differential growth-induced bow was larger for fuel assemblies near the core-edge and, therefore, the core reload pattern typically employs a cross-core shuffle.
FCF post-irradiation examination data taken over three operating cycles indicates that most of the assembly bow occurs during the first cycle and that the increase in later cycles was not large.
Because of recent control rod insertion problems experiencM by NRC PWR licensees, the inspectors reviewed an assembly bow event at the Ringhals 3 and a subsequent related event at Ringhals 4.
Both of these plants included second generation Framatome fuel assemblies.
In the Ringhals 4 incident an RCCA jammed in the dashpot region and did not fully insert following a reactor trip. However, the drop times-(measured to the top of the dashpot) for all other RCCAs was within the value used in-the safety analysis. Only fresh or once-burned fuel assemblies were located in +he RCCA locations.
As part of the investigation of these incidents, UT and Eddy current tests were performed on eight fuel assemblies to evaluate swelling and Fifty-eight fuel assemblies were also inspected for bowing, wear. 70
f i
elongation and the extent of rod-to-nozzle gap closure. These tests
)
indicated larger than usual (both "S" and." banana" shape) assembly bowing.
Based on subsequent measurements and calculations, it was a
concluded that the assembly deformation was the cause of the failure of the RCCAs to insert. A review of the second generation Framatome fuel indicated that, while the fuel rod, guide tube and hold-down system designs were identical, the decreased grid height resulted in reduced lateral stiffness which. increases the susceptibility to assembly bow.
FCF analysis of the assembly bow included the effects of (a) differential growth due to flux gradients and (b) creep amplification of existing deformations. The resulting fuel assembly design modifications resulted in increased guide tube stiffness and a reduced fuel assembly hold-down force.
s The inspectors concluded that FCF was actively evaluatir.; the Ringhals bowing events and their relevance to FCF fuel designs.
b.2 Fuel Cladding Distinctive Crud Pattern Examination of the fuel pins following TMI-l Cycle 10 and CR3 Cycle 10 operation indicated a distinctive crud pattern (DCP), on more than 182 fuel rods at TMI-l and on four fuel rods at CR3, which was significantly different than the." normal" corrosion pattern. Nine fresh (one-cycle burned) fuel rods at TMI-1 with tha DCP had through-wall leaks and many of the other rods with DCP had significant wall-thinning.
FCF provided a detailed description of their investig. tion into the cause of the DCP observed at TMI-1 Cycle 10 and CR3 Cycle 10.
The inspectors reviewed this material in order to identify (a) the factors that could have potentially contributed to the DCP and-(b) the actions that FCF may have taken to. prevent the occurrence of the DCP.
For TMI-l Cycle 10, the DCP tended to occur in the sixth-axial grid-span
-(second from the top of the fuel assembly) and on the outer-row of fuel pins of the assembly. Most of the fuel pins with DCP were located in one of eight symmetrically loaded "T" loading patterns. These "T" loading patterns consisted of four fresh high enrichment (4.0 to 4.75 w/o Uns) fuel assemblies, arranged with three assemblies across the top of the "T" and one assembly centered below. These "T" patterns were located un the core periphery in the low-flow core region with one assembly facing the reflector (see Figure 1 on page 35 of this report).
-Following the TMI-1 Cycle 10 fuel failure presentations by FCF, the inspectors decided to evaluate the DCP event based on the following possibilities:
(a) the presence of crud in the coolant was caused by inadequate water chemistry (b) the deposition of crud on the fuel pins was accelerated due to higher clad surface temperatures in relatively low-flow regions 71
I
)
(c) the corrosion failures'(crud / oxide spalling) were caused by local boiling in or under the crud layer l
The inspectors pursued-investigations into various aspects of the failures and possible mechanisms.
(1)
Reactor Coolant System Chemistry During operation of a 177 fuel assembly B&W plant, a minimum coolant pH level is maintained in order to prevent any significant increase in the deposition of crud on the fuel rods. However, the recent aggressive 24-month high-reactivity core designs, like TMI-1 Cycle 10 and CR3 Cycle 10 have increased boron levels (to ~1500 parts-per-million (ppa) at startup) which has resulted in a reduction in the coolant pH to below the target value of ~6.9.
i This pH reduction is generally compensated for by maintaining an increased lithium (Li) concentration in the coolant which increases the pH.
E In its letter to CPUN, "1991 Revision of BAW-1385 - Water Chemistry Manual 177 FA Plants, Lithium / Boron Control," November 11, 1991, BWFC provided GPUN the recommended Li/ boron control for TMI-l Cycle 10.
This recommendation was to maintain a pH of 6.9 initially and then, if necessary, to increase the pH to a higher value later in the cycle.
Because of the high initial boron concentration in the coolant for this high reactivity core, this would have required a Li concentration higher than the usual BWFC recommended limit of Li <2.2 ppm.
(The upper limit on Li was designed to protect against fuel cladding corrosion.)
In a subsequent letter to GPUN, " Lithium / Boron Control Requirements," December 11, 1991, BWFC recommended a second and different Li/ boron control. This recommendation was to startup at a Li concentration of 2.2 ppm and hold at this value until the pH increases to the desired value abvte 6.9 (as a result of the decreasing boron), and then to decrease the Li in order to maintain the desired pH. Responding ;o a request from GPUN for BWFC to clarify its position, BWFC inlicated that the second recommendation should be followed.
The inspectors concluded that the final BWFC recommendation, to limit the maximum Li to 2.2 ppm, resulted in a pH less than 6.9 and significantly increased the tendency for deposition of crud on the fuel. The low pH was of special concern at TMI-1 Cycle 10 because of the high level of " circulating crud" believed to be present due to (a) the five years of plant shutdown that had occurred earlier and (b) the fact that the reactor coolant system (RCS) makeup flow was less than 50% of nominal. A similar situation existed at CR3 Cycle 10, although the lowest pH was 6.6 and the level of circulating crud was believed to be substantially less. 72
..-.. _ ~
In its letter to GPUN, "TMI Cycle 11 Lithium Levels - Revision to i
Previous Recommendations," August 28, 1995, BWFC relaxed the Li limit to 3.0 ppe for TMI-l Cycle 13 in order to maintain a pH above 6.9 and thus reduce the potential for fuel rod crud deposition.
During the initial months of operation of TMI-1 Cycle 10, GPUN did not add Li to the reactor coolant but rather allowed lithium-7 to
" burn-in" as a result of the B-10 (n, a ) Li reaction.
Because of the high Cycle 10 boron concentration, this resulted in an initial pH of 5.7, which increased to 6.9 after several months of operation as a result of the Li buildup and the decrease ~in boron.
Based on the above evaluations, the inspectors made the following conclusions:
(a)-
The FCF communication to GPUN concerning the TMI-1 Cycle 10 i
Li/ boron. control was not clear and may have contributed to the decision by GPUN to ignore the recommendation and initiate Cycle 10 without Li. The inspectors considered this a weakness in the FCF/ customer interface.
(b)
The FCF Li/ boron control guidance provided to GPUN did not i
ensure a pH >6.9 and provide adequate protection from crud deposition.
The inspectors considered this a result of a weakness in the FT; water chemistry program to establish 1
adequate Li/ boron control procedures.
(2)
. Nuclear and Thermal-Hydraulic Design The nuclear and thermal-hydraulic reload core design is an important consideration in preventing fuel rod crud, since both low-flow velocity and high fuel rod power enhance crud deposition; in part, due to the reduction of solubility with increased temperature.
In addition, since several of the mechanisms suspected of causing the DCP failures involved boiling under the accumulated crud, the relatively small margin of 10-20*F to coolant boiling (in the sixth-axial span where DCP occurred) makes this design relatively sensitive to small increases (decreases) in fuel rod power (flow veluity). This was especially true in TMI-l Cycle 10, where a BOC power-tilt of +15% occurred.at the sixth-axial span in the quadrant of the core where the DCP occurred.
(This tilt is believed to be due to boron " hideout" in the crud.)
In the fuel assembly designs for both the TMI-1.and CR3 core reloads, the highest powered fuel rods were located on the outside row of the fuel assembly, where the lowest assembly flow velocity occurs.
The placement of-the fresh high e richment fuel assemblies in the "T" loading pattern on the core periphery resulted in the unusual situation in which the highest powered rods in the core (they were in the highest eight percent) were located in the peripheral (lowest velocity) locations.
In 73
addition, the relatively low burnable poison loading of both the TM1-1 Cycle 10 and CR3 Cycle 10 cores resulted in higher critical boron concentrations and consequently, lower reactor coolant pH.
It was well known that fuel rod crud deposition is enhanced by high fuel rod power, reduced flow velocity, and low coolant pH.
However, the inspectors determined that several features of the TMI-I Cycle 10 core design resulted in a significant reduction in the margin in these parameters to fuel rod crud deposition. These include (a) the highest-to-date critical boron concentration which lowered the coolant pH and (b) the location of the "T" loading pattern of high powered fuel assemblies (with the highest-to-date enrichment of 4.7 w/o U23s) in a low-flow velocity region on the core periphery.
i The inspectors concluded that this design weakness together with i
the (allowed) operation at low Li, which further lowered the coolant pH, resulted in DCP on the high powered rods in the "T" pattern loaded fuel assemblies. Based on the above, the inspectors concluded that there was a basic weakness in the TMI-l Cycle 10 core design.
The nuclear, thermal-hydraulics and materials analysis of the THI-l Cycle 10 core were considered weak by the inspectors because the reload core could not accommodate the expected lower coolant pH values and protect from deposition of crud on the fuel rods.
(3)
Fuel Assembly Bow Fuel assembly bow, due to guide-thimble / fuel-rod differential growth, occurs preferentially on the core periphery where the flux 1
gradients are strongest. As the fuel assembly-to-assembly gap increases, the power in the fuel rods located on the outer row of the assembly increase (~3% per 50 mil of gap closure).
In the TMI-l Cycle 10 core design, the "T" pattern of fresh high enrichment high-powered fuel assemblies were placed near the core periphery which maximized the flux gradient across these assemblies and the resulting assembly bow.
The inspectors determined that this resulting increase in rod power would increase the potential for crud deposition in exactiv the location where the DCP occurred; the outer row of fuel rods.
For TMI-1 Cycle 10 operation, both the location of the DCP failures in the outer row of the fuel assemblies and on the core periphery was consistent with the assembly bow mechanism which would have resulted in increased fuel rod power and local boiling.
The inspectors concluded that FCF failure to perform an evaluation (including measurements where practical) of assembly bow as a potential mechanism for the deposition of the DCP was a weakness in its DCP true cause analysis. 74
(4)
Neutronic One theory pursued by the inspectors was that the high clad surface temperature and local boiling was caused by the inability to correctly predict the heat flux or pin power distrib9tions in the-fuel assemblies with DCP.
The inspectors evaluated the lik31ihood that the accuracy of the neutronics as applied to the TMI-l Cycle 10 core design was a contributor or the potential cause of the fuel failures.
i The ins)ectors were told by FCF that the majority of the pin failures had occurred in fuel assemblies located in the fresh fuel i
"Ts" (a four fresh fuel assembly group, which has 3 assemblies face-acjacent in a row forming the top of a T, see Figure 1 on page 35 this report). Within these "T" assemblies, the majority i
of the pin failures occurred in fuel pins on the outer row (along the water gap) of the assembly and in the corner pin where the fresh assembly faces meet.
The failures occurred in the top one third of the fuel rod (typically 100- to 130-inches from bottom of the fuel).
The inspectors evaluated FCF's ability to accurately predict the pin power in these locations.
The inspectors reviewed the TMI-l Cycle 10 " measured power distributions" (seven axial values for each of the 57 assemblies containing fixed incore detectors) versus those predicted by NEMO.
Particular attention was given to the comparisons in the sixth and seventh axial location which corresponds to the elevation with the crud deposition and fuel failures. Comparisons from four points in time throughout the cycle were examined.
The inspectors observed that for the assemblies of interest, NEMO consistently over-predicted the power in the seventh level (on the average, 5-10% over-predicted) and more closely predicted the 1
power in the sixth level (on the average 1% under-predicted to 3%
over-predicted).
This would result in the temperatures being cooler in the actual core in these locations as compared to the predicted core.
In addition to the " measured" vs NEM0 predicted power distributions presented in the NEM0 topical report, the inspectors reviewed TMI-l Cycle 9 radial power distributions.
The inspectors observed that the radial power was typically predicted within 2%
(root-mean-squared (RMS) or standard deviations were less than 2%
for various databases).
The inspectors also observed that the power in the face adjacent fresh fuel was predicted as well as other radial locations.
The inspectors reviewed analytical power distribution benchmarks such as comparisons between NEM0 and other nodal codes or between NEMO and PDQ.
Data from TMI-l Cycles 8, 9, and 10 was examined and the inspectors observed that, compared to the other code j
prediction, NEM0 tended to predict the same or higher power in the 75
~
i face adjacent fresh fuel locations.
Based on the above l
evaluations, the inspectors concluded that for the specific "T" assembly locations, there was no evidence to suggest that NEMO under-predicted the assembly power.
The inspectors also observed that there was nothing unusual about the TMI-l Cycle 10 assembly power distributions-relative to the benchmarked cycles (in terms of control rod presence or unusual burnable poison configurations). The inspectors-did not'e that the fresh assembly "T" was placed in a core location which had a steep power gradient (due to presence near the edge of the core). Based on the inspectors conclusion that NEMO was predicting assembly power reasonably well, the inspectors next focused on the NEMO pin-power reconstruction methodology and NEM0's ability to predict the local pin power.
1 The inspectors reviewed single assembly CASMO 3 calculations and NEMO quarter core with pin power reconstruction calculations to determine what the pin power distributions were for current FCF designs.
The inspectors observed that the newer designs, e.g.,
CR3 Cycle 11, have a flatter assembly pin power distribution and the peak / average pin power ratio was typically less than 1.10.
The older designs may have peak / average pin power ratios of up to 1.2.
The inspectors observed'that the NEMO calculations for the pin power ratios in the fresh "T" locations show a very large gradient across the assembly (e.g., _1.4 to 0.6 or-1.5 to 0.8).
The inspectors also noted that the peak pin power often occurs in the corner pin where the fresh assembly faces meet.
The inspectors reviewed multi assembly (2x2 array) comparisons of NEMO pin power distributions vs MCNP and comparisons of NEMO vs l
CASMO 3.
The inspectors observed that the differences were less than 2% and often less than 1%.
The_ inspectors concluded that NEMO was reconstructing the pin power-distribution satisfactorily 1
for these analytical comparison; however, the actual core pin power distributions were more challenging (steep gradients). The inspectors also noted that their concern about-the NEMO accuracy partially arises from the lack of directly applicable benchmarks.
That is, FCF had no benchmarks which duplicate the TMI-1 Cycle 10 core design and the pin power distributions gradients present in these core designs.
Based on the above review of the neutronic aspects of the TMI-1 Cycle 10, the inspectors concluded that the NEM0' methodology can accurately predict the assembly and pin power distributions. Thus there was no evidence to support the hypothesis that the inaccuracy of the neutronic models contributed to the fuel corrosion failures.
4 76
However, the inspectors also concluded that the TMI-1 Cycle 10 design challenged the methodology more than other FCF designs via the "T" placement of fresh assemblies. _ The TMI-1 Cycle 10 design resulted in large pin power peaks in the corner pins and steep power gradients across the assembly and these local conditions can not be found in any of the NEMO benchmarks; thus, the methodology-was stressed to near its maximum capability.
Other FCF core designs have either not used the "T" loading, or used an "L" loading which had lower pin peaking.
The inspectors concluded that these factors resulted in a weakness in the TMI-l Cycle 10 reload core design.
c.
Conclusions Based on the review of the DCP and associated fuel rod failures, the inspectors concluded that the FCF design process was also a contributing factor to the DCP.
The inspectors noted that there were many features of the TMI-1 Cycle 10 core design which by themselves were not significant contributors to DCP, but when taken together, resulted in a reduction in the margin to crud deposition and localized boiling.
These factors included:
(a) increased critical boron and the resulting decrease in reactor coolant pH (b) inadequate Li/ boron control (c) inadequate communication to TMI-1 regarding the establishment of the Li/ boron control (d) low coolant pH was not accommodated by the nuclear, thermal-hydraulic, and material design (e) challenged the NEMO methodology by using "T" loading pattern (f) both the fuel core and lattice designs placed the highest powered rods in the locations of lower-flow velocity and higher assembly bow The inspectors considered the lack of an overall multi-disciplinary review process by FCF to identify and evaluate the synergistic effect of the changes that occurred in the TMI-1 Cycle 10 core design to be a weakness in the FCF design process.
l, The inspectors also considered '.he lack of an evaluation of assembly bow as a potential contributing mechanism as a weakness in FCF's true cause analysis of the DCP, 3.5-Eneineerine Ca== uter Procrams and Database 4
a.
Inspection Scope FCF's use of engineering compute programs (ECPs) was contingent upon a tiered approach to computer program certification.
Programs (codes) can i
be fully certified, conditionally certified, interim certified, and uncertified.
Procedure BWNT 0402-01, " Preparing and Processing BWNT t,
77
4 Calculations," Revision 29, June 1,1994, proscribed the use of l
certified programs in performing calculations, and stated that uncertified programs must have an independent verification of accuracy.
To evaluate FCF's use of ECPs, the inspectors reviewed the certification i
process for new versions of ECPs and the procedures for error reporting, correction, and access control.
i b.
Observations and Findings b.1 Software Certification Process The inspectors reviewed FCF's recent modification to the certification status of NEMO 7.6 from fully certified to conditionally certified. The change was due to development of NEMO 7.7, which included notification messages regarding the use of NEMO. The warning messages were excluded from NEMO 7.6.
FCF chose to downgrade the certification status of NEMO 3
7.6 for this reason.
FCF procedure BWNT 0902-06, " Software Certification," Revision 16, March 31, 1995, proscribed an affectivity statement for the procedure applied to computer programs changing certification status due to errors within the code. The procedure requires changes to the certification status to l
be placed in the certification file.
However, the inspectors determined that the change to NEM0 7.6 was not in the certification file.
i FCF agreed with the inspectors findings and added the change to -the certification file. The inspectors concluded that the procedure was ambiguous because certification changes for any reason, including code errors, should be recorded in the certification file.
FCF also agreed that the procedure governing ECP certification was ambiguous, and comitted to modify the procedure.
T!e inspectors concluded that the ambiguity in the software certification procedure and the resulting incorrect information on.NEMO 7.6 were considered weaknesses in FCF's computer code certification process.
b.2 Engineering Computer Program Error Reporting The inspectors reviewed calculational files for consistency in ECP error reporting procedures.
The Inspectors found that FCF had.no specific procedures to address error reports in calculational files.
Furthermore, the inspectors determined that procedure BWNT 0402-01 did not distinguish between the use of conditionally certified and fully certified ECPs. However, FCF stated that use of a conditionally certified code required justification in the calculation file, particularly if error reports exist for the code. 78
i I
l
.The inspectors examined two cases involving the use of the NEMO.4.lRS code for TMI-l Cycle 10.
NEMO,4.lRS was a conditionally certified code l
which contained an error in the calculation of certain core average values. The first use of this version of NEMO was found in BWNT 32-i i
1219600-00, "TMI-l Cycle 10. Flex Nuclear Data," September 17, 1993.
The calculation file did not include a justification for the use of the code, even though an error record existed for the code.
FCF agreed that this was an oversight and added a reference to the error in the l
calculation file.
4 The second use of_this version of the NEMO code was documented in BWNT 32-1218970-00, "TMI-l Cycle 10 Nuclear data /650 Cycle 9," September 17, j-1983, that also did not contain justification for use of the code.
FCF i
also corrected this oversight during the inspection.
1
[
Based on these findings, the inspectors concluded that inconsistency in addressing ECP error reports was a weakness in the use of ECPs.
b.3 Benchmarking CASM0 3 and NEM0 The applicability of the CASMO 3/NEMO neutronic methodology benchmarking to the.new FCF reload designs was reviewed by the inspectors.
The NEM0 topical report BAW 10180-A, "NEMO - Nodal Expansion Method Optimized,"
Revision 1, March 1993, was used as a reference by the inspectors.
The inspectors focused on the potential concern that the new fuel assemblies and/or core designs may have differed significantly from the range of fuel designs that were benchmarked _in the topical report.
The inspectors reviewed core loading patterns for TMI-l Cycles 8 to 11, CR3 Cycles 10 and 11, ANO-1 Cycle 13, Trojan Cycles 12 and 13, and Sequoyah-1 Cycle 9.
The inspectors noted that operating cycle lengths have increased over time and that some current reload designs were for 650 to 670 EFPD (24 month cycle). The longer cycle designs necessitated the use of higher U enrichments (up.to 4.96 w/o) with corresponding ns
)
higher beginning of cycle (B0C) soluble boron concentrations-(up to 2300 ppe at BOC, HZP), or with higher burnable poison loadings.
The inspectors noted that the core designs, now being loaded or soon to be loaded; used up to 6 w/o Gd as the burnable poison and that some assembly. designs used both Gd and the B C lumped burnable poison.
4 Having determined the past and upcoming fuel designs, the inspectors focused on reviewing the range of designs that have been benchmarked to validate the CASMO 3/NEM0 methodology.
The inspectors's primary concern was the. ability of NEMO to predict core power distributions.
In addition to the benchmark results published in the NEM0 topical report, the inspectors reviewed power distribution comparisons from the following:
)
(a)
NEMO predictions vs measured incore detector data (b)
NEMO predictions vs PDQ predictions (two-dimensional (20) quarter core diffusion theory results)
) 79
(c)
NEMO predictions vs SIMULATE-3 predictions (alternate 3D nodal code results)
(d)
NEMO predictions vs MCNP predictions (2D multi assembly Monte Carlo code results)
Based on this review, the inspectors concluded that NEM0 can accurately predict core power distributions within the accuracy stated in the NEMO topical report. The inspectors also concluded that the current and upcoming core designs were within the range of fuel design configurations and core parameters that were verified in the topical Additional evaluations were performed by the inspectors as part report.
of the TMI-1 fuel corrosion review (see Section 3.4.b.2 of this report).
c.
Conclusions The inspectors concluded that the ambiguity regarding ECP certification changes found in procedure BWNT 0902-06, " Software Certification,"
Revision 16, March 31, 1995, and the resulting incorrect information on NEMO version 7.6 were considered weaknesses in FCF procedures.
The inspectors examined a number of calculation filss to determine adequacy and consistency in ECP error reporting and found examples where Based on ECP error reporting was either inadequate or non-existent.
these findings, the inspectors concluded that FCF's inconsistency in addressing ECP error reports was a weakness in the use of ECPs.
3.6 Eneineerino - Manufacturino Interface a.
Inspection Scope The inspectors reviewed specific aspects of the reload analysis / fuel fabrication process including the role of the RELATE, the engineering / manufacturing interface and the use of the release authorization / applicable documents list (RA/ADL).
b.
Observations and Findings The inspectors found that the first stage of a reload contract involved development of the contract requirements document (CRD), which contained initial licensee requirements for the fuel cycle, including a description of work to be performed, the scope of supply, quantity, and After the special features and customer requirements for the contract.
CRD was issued, the preliminary fuel cycle design (PFCD) process typically developed three fuel designs, with the final decision left to the licensee based on operating requirements. After customer input, the fuel cycle design requirements (FCDR) document was prepared by the RELATE engineer.
The FCDR contained operating parr.eters, batch descriptions, control component descriptions, and other gecial and cycle-specific features.
The inspectors determined that the FLDR can be revised as necessary and as more specific data was received from the licensee, such as exact 80
._. - -. -..~
i 4
l cycle lengths.
Concurrent with development of the FCDR and the final fuel cycle desi control group. gn (FFCD), the RA/ADL was prepared by the fuel design The inspectors reviewed example RA/ ADLS.
The ADL defines a specific base or contract design by listing the applicable design definition documents, part names, and part numbers and was used to release the following:
(a) product documentation as part of a procurement package (b) documents to establish code applicability (c) documentation for equipment that requires professional engineer certification The RA was sent to manufacturing to authorize. fabrication of components specified in the ADL.
Fuel manufacturing began as the final fuel cycle design was nearing completion.
During fuel assembly fabrication, communication between manufacturing and engineering can be in the form of concurrence requests (CR), deviation reports (DR), transmittal records (TR), and design change requests (DCR).
L The inspectors-reviewed a number of these records from current core design contracts.
These transmittals were maintained by the fuel design control group.
Procedure QC-1434, " Submittal of Manufacturing and Quality Documents to Fuel Engineering and Customers," described the methods by which manufacturing, quality, and supplier documents were transmitted to fuel engineering or to customers for approval (CR) or for information (TR) and the means by which such transmittals and approvals were documented.
Design change requests (DCR) were sent from engineering to manufacturing to propose design changes to previously released nuclear fuel and core components based on contract specified designs.
c.
Conclusion The inspectors concluded that the interface between engineering and manufacturing and the use of the RA/ADL were performed in accordance with written instructions and satisfactorily completed for those activities affecting quality.
3.7 Procurement a.
Inspection Scope The inspectors reviewed FCF's activities related to the procurement of material and services for use in safety-related products, the documentation that FCF maintained supporting the purchases of material, equipment, and services used in.he manufacture cT safety-related products, and QA specification 09-1212-05, " Quality Assurance Program Requirements for Suppliers," Revision 5, February 11, 1992. 81
b..
Observations and Findings FCF procedure BWNT-1719-25, " Quality System Surveys &-Quality Audits of BWFC Suppliers,". Revision 4, October 28, 1994, specified the requirements for qualifying vendors to supply safety-related material, l
equipment,.and services.
BWNT-1719-25 required that FCF perform an on-l site quality audit to' verify implementation of the applicable quality program (for initial approval and triennially thereafter) and also to perform an annual evaluation to review the vendor's recent activities with FCF.
The status of vendors was maintained by FCF on the supplier status list (SSL).
The inspectors reviewed several FCF audit reports, supporting placement on the SSL, for Suhm Coil' Spring Works (BWFC 92-36), FCBC Romans Plant (BWFC 94-17), Gray Syracuse Report (BWFC 95-9), and Carpenter Technology (BWFC 95-7). The audit reports documented the basis for the quality audits, results, conclusions, and findings.
In addition, the report files contained documentation between FCF and the vendors which adequately resolved any fir. dings. The inspectors concluded that the quality audits had been adequately performed and that FCF had maintained adequate documentation.
The following purchase orders for materials were reviewed by the inspectors:
(a)
P0 40028, March 25, 1996, to Zircotube, Paimboeuf, France, for fuel clad tubing (b)
P0 21872, January 23, 1995, to Gray Syracuse, Inc., for stainless steel castings to make end fittings (c)
P0 40027, March 25, 1996, to Zircotube for guide tubes (d)
PO 40051, March 25, 1996, to Cezus, Montreuil-Juigne, France, for end plug bar stock (e)
P0 27542, July 5,1995, to Cezus, Rugles, France, for Zircaloy spacer grid strip (f)
PO 36317,. January 19, 1996 to Ulbrich Specialty Strip Mill, i
Wallingford, Connecticut, for Inconel spacer gird strip (g)
PO 4030, March 27, 1996, to Suhm for cruciform holddown spring section manufacturing services for plenum springs (h)
P0 2849, November 16, 1993, to Inco Alloys International, Inc.,
for nickel alloy sheet for plenum springs c.
Conclusions The inspectors determined that specification 09-1212-05 invoked the requirements of 10 CFR Part 50, Appendix B,.and that 10 CFR Part 21 was referenced in each of the P0s reviewed. The inspectors concluded that the procurements wereLperformed-in accordance with written instructions and 3atisfactorily completed for thu.e activities affecting quality.
i l
l 82 4
M 4
w
-m1>+*--s--
.w r--~r e
e
^ v5
"'e
.+-m n
.___m 3.8 Fuel Fabrication l
a.
Inspection Scope; 2
FCF produced PWR fuel bundle and control components for B&W-and Westinghouse-designed reactors.
They also produced fixed incore j
detectors for B&W, Combustion Engineering, and Westinghouse-designed i
reactors. 'The FCF fuel bundle rod matrix design was 15x15 array for Mark-B9,' Mark-B10, and Mark-Bil (the B&W designs), and 17x17 for Mark-BW, the Westinghouse design.
FCF control components consisted of Mark-B burnable poison rod assemblies (BPRAs), Mark-B control rod assemblies (CRAs), Mark-B axial power shaping rod assemblies (APSRAs), Mark-BW BPRAs (15x15 and -17x17) and rod cluster control assemblies (RCCAs).
For the Mark-BW17 fuel bundles,'FCF produced reconstitutable top nozzle assemblies, intermediate spacer grid restraint' system, intermediate spacer grid assemblies, end spacer grid assemblies, debris resistant bottom nozzles, and advanced debris filter bottom nozzles.
During this inspection, FCF was not fabricating fuel rods or fuel bundle assemblies. Therefore, this portion of the inspection was limited to evaluating the fabrication of certain fuel bundle components.
b.
Observations and Findings b.1 Surnable Poison Rod Assembly During this inspection, FCF was fabricating HK-BW type BPRAs for use in Westinghouse-designed PWRs. The fabrication encompassed manufacturing burnable poison rods and thimble plugs,'and fastening them.to an upper structure assembly with appropriate hardware.
The inspectors observed the fabrication'of burnable poison rods and upper structure assemblies, and assembly of BPRAs, as well as inspections performed by product quality (PQ) personnel and reviewed selective records pertaining to the quality of components used to fabricate the BPRAs.
The team identified no adverse observations in this area.
b.2 Rod Cluster Control Assemblies During tnis inspection, FCF was fabricating RCCAs intended for use in Westinghouse-designed reactors.
The main components of an RCCA were the spider, the control rods and the hardware to attach the control rods to the spider.
The team observed the activities related to the fabrication of the_ control rods, and the-final assembly of.the RCCA including inspections by PQ, and reviewed records attesting to the quality of selected components.
The team identified no adverse observations in this area..
83
b.3 End Caps The inspectors reviewed the end cap fabrication process as specified by RC 9110-20, " Fabrication of Fuel Rod End Caps." Receiving inspection of the end plug rod was performed in accordance with QC-595, " Receiving Inspection - Zircaloy Bar Stock and Tubing," Revision 4, October 13,
]
1994. Manufacturing machined the end caps to the dimensions required by the applicable drawing and inspected 100% of the parts. PQ inspected the caps for dimensions and cleanliness. Accepted product was forwarded e
to material control for storage.
PQ reviewed the route card for completeness and forwarded it, along with inspection data, to quality verification for retention.
The inspectors observed that cleaning operations were performed in accordance with MA-549, " Miscellaneous Hardware Cleaning," Revision 4, Paragraph 3.6 stated that within the limitations July 28, 1994.
specified in the product routing or elsewhere in MA-549, the combination and sequence of cleaning methods were left to the discretion of the cleaning technician.
2 However, the inspectors reviewed the training records of the cleaning technicians and determined they had no recorded training regarding the selection and sequence of cleaning methods. The inspectors considered J
this a weakness in the FCF manufacturing system.
(This issue is discussed further in Section 3.12.b.2 of this report.)
~
b.4 Spacer Grids The inspectors reviewed the spacer grid fabrication process as specified by RC 8500-11, "MK-BW Zircaloy Intermediate Spacer Grid Assembly l
Fabrication." Receiving inspection of the zircaloy strip was performed in accordance with QC-586, " Receiving Inspection - Zirconium Alloy Sheet or Strip," Revision 3, August 14, 1995. Manufacturing received released spacer grid strip material, recorded the applicable information for traceability on the route card (RC) and assembled the spacer grid strips. The inspectors considered the manufacturing and inspection activities suitable for the production of grid spacers for safety l
related applications.
c.
Conclusions Other-than the weakness identified in the cleaning of end plugs, the inspectors considered the manufacturing and inspection activities suitable for the production of the safety-related items.
3.9 Calibration a.
Inspection Scope 4
The inspectors reviewed the FCF program for ensuring that equipment was properly controlled and maintained in calibration.
FCF maintained a gage control laboratory staffed by a gage control technician who was 84
responsible for controlling the status and maintaining the records associated with equipment calibration.
FCF calibrated most of the equipment used in the facility with the exception of certain specialty machines, such as the Baldwin tensile tester and the standards used to perform the calibrations, such as gage blocks.
i b.
Observations and Findings FCF maintained the calibration records for the approximately 2,500 items in a hard copy filing system arranged by date of calibration (month or month / day). The gage control. technician was efficient in the use of this system and was able to provide the inspectors with the supporting documentation for all calibrations reviewed.
The inspector's 4
calibration reviews included the gage envelope dial indicator (QC14-17),
water channel gage standard fixture (QC-1949), micrometer inspection gage (QC-1578), guide tube alignment gage (QC-4), lower end fitting casting template (QC-1579), lower end fitting casting inspection gage (QC-1578), and lower end fitting true position functional gage (QC-53).
FCF was able to provide documentation supporting traceability of the calibrations reviewed to the appropriate standards including the National Institute of Standards and Technology (NIST).
FCF procedure QC-1405, " Control of Measuring & Test Equipment," Revision 5, January 31, 1995, paragraph 4.5 stated, in part, that calibration services were classified as commercial grade and the measuring and test equipment were classified as commercial grade items.
However, when the equipment was used to verify the capability of safety-related items, the equipment had a safety-related function to verify the parameters defined for acceptability of the safety-related item.
Accordingly, FCF took actions to qualify the commercial grade service provided by calibration suppliers, to ensure suitability for the intended. safety-related
. application, by performing an audit of the suppliers technical and quality program.
FCF used MIL-STD 45662A, " Calibration System Requirements," as the basis for the audits.
After successful completion of the audit, the calibration suppliers were listed on the SSL.
The inspectors reviewed the FCF audit reports, supporting placement on the SSL, for AA Jansson, Inc., (BWFC 95-1),
American Electronics Laboratory (BWFC 95-18), and Gage Laboratory (BWFC 93-35), which indicated that tM audits were adequately performed and documented.
In addition, the inspectors reviewed P0s to calibration suppliers for AA Jansson, Inc. (P0 36241, January 12,1996), American Electronic Lab (PO 38981, March 5, 1996), and Gage Lab (P0 30301, August 17,1995), and determined that they contained adequate technical and quality requirements to assure performance of the requested service.
The inspectors concluded that FCF had taken adequate actions to verify and document the quality of the calibration services used to support safaty-related work.
- 85
~
During an inspection of production-line equipment, the inspectors noted that~the Baldwin tensile tester, identified as FCF QC-519, exhibited a "Satec" calibration sticker which stated that the machine had been calibrated on March 8, 1995, and was due'for calibration on March 8, 1996. There was no FCF calibration sticker present on the machine. The inspectors noted that the machine was not administrative 1y out of 1
calibration due to procedure QC 800, " Gage Control," paragraph 5.2, which allowed for the next calibration to be performed at any time i
during the calendar month that the previous calibration expired such that the calibration on the Baldwin machine was acceptable for FCF work through March 31, 1996.
Discussion with FCF personnel indicated that the Baldwin machine was intended to be used in its present form for only an additional two months and at that time it was to be modified.
Based on consideration for the short period of intended use, FCF had extended the calibration interval to eighteen months and had documented the extended calibration i
interval in QC 800.
FCF indicated that the basis for the extension from twelve months to eighteen months was ASTM E-4-94, Section 20, " Time Interval Between Verifications" which stated, in part, that it was j
recommended that testing machines be verified annually or more
~
frequently if required and in no case shall the time interval between l
verifications exceed 18 months.
\\
I The inspectors determined that FCF had not established a technical basis for the calibration interval extension. The inspectors also noted that since the previous March 8, 1995, calibration had been performed by Satec, FCF may consider Satec's input when establishing the technical basis.
FCF contacted Satec who provided a letter dated March 27, 1996, that l
l stated that the decision to extend the current verification interval rested with FCF, that the machine could probably be used without a problem, but that FCF should have_the machine's calibration reverified as soon as possible.
However, the inspectors determined that the Satec letter did not provide a technical basis for extension of the calibration interval.
c.
Conclusions The inspectors concluded, based on the review of the FCF documentation and the Satec letter, that FCF had not established a technical basis for using the Baldwin machine past its original calibration date (and its administrative extension of March 31,1996).
This was identified as a
~
weakness in the implementation of the FCF calibration program.
3.10 Field Services Field Services was divided into three functional groups; engineering, outage operations and equipment operations. The Field Services quality assurance activities were governed by the same topical report that governed QA activities in other operations at FCF, document 56-1177617-l 86
I t
)
03, " Quality Assurance Program for B&W Fuel Company," May 30, 1994.
Procedure MA-580, " Specialized Tooling and Equipment," Revision 0, t~
j October 12, 1995, set forth the guidelines and responsibilities for the design, implementation, documentation and control of specialized tooling i
and equipment bearing on the outccme of manufactured product quality.
l On the basis of its review, the inspectors concluded that the QA program
)
for' field service activities was adequate.
}'
3.11 Quality Action Reauests The inspectors reviewed the quality action requests (QARs) program and i
the status of several open QARs and concluded that FCF personnel were i
sufficiently responsive to QARs.
l l
3.12 Trainine i
j a.
Inspection Scope 1
The inspectors evaluated the FCF training program in both the engineering and fabrication areas.
f j
b.
Observation and Findings j
b.1 Engineering h-The inspectors evaluated the adequacy of training in the engineering and l
. services units primarily through interviews of management and staff, including interviews with unit managers in the fuel management and operations analysis, thermal and performance analysis and engineering analysis services units.
The inspectors considered training to be made
[
up of the following two general areas:
l (a) direct, job-related technical training i
(b) non-technical or d inistrative training The inspectors also evaluated training for new employees and training in i
problem solving techniques.
Procedure BWNT-1702-22, " Employee Training," Revision 16, October 10, 1994, provided requirements for j
training. The procedure stated, in part, that all initial job and on-going quality-rr. lated training shall be documented on a personnel i
training report (PTR). On-going quality-related training was described in the procedure as training on new or revised QA program manuals, the records management program manual, working instructions and new or revised quality-related policies, procedures, and forms.
3 In general, the inspectors found that no formal requirements exist for direct technical training, such as the use of engineering computer codes.
Training on administrative procedures was the only training that i
4 l
1 j
s 87
i was officially documented. The inspectors examined available training records for one engineering unit. Although substantial records existed on administrative procedure training and software training, only one i
record was found for training directly applicable to job performance.
b.2 Fuel Fabrication FCF had divided the training of employees into two distinct categories, inspection personnel and non-inspection personnel. The inspectors found that procedures BWNT-1702-22, " Employee Training," Revision 16, October 10, 1994, applied to all FCF employees and that QC-1440, " Qualification of Inspection Personnel," Revision 9, May 31, 1995, applied to personnel performing inspections to verify conformance to specified requirements for the purpose of acceptability.
The inspector reviewed BWNT-1702-22 that stated that employee training was the responsibility of the management of each organization and that each supervisor should evaluate the training needs of the employees within the group on a continuing basis.
All initial job and on-going quality related training was to be documented on a PTR.
The specific training required by BWNT-1702-22 included initial job training to be accomplished prior to beginning work and documented on a PTR. The on-going quality related training was to be provided by the applicable manager.
The inspectors reviewed the training records for a selected group of employees from various areas including QC, PQ, final assembly, gage control, metallurgical laboratory, assembly room, machining, and manufacturing.
FCF provided records for the personnel performing inspection activities which had documentation of the initial and on-i going quality related training.
The inspectors concluded that the inspection personnel training was in accordance with the requirements of QC-1440 and adequately documented.
However, for non-inspection personnel, FCF could provide no records i
documenting either initial training or on-going quality training.
FCF management indicated that this training had occurred but there had previously been no requirement to document the completion of the i
training.
1 c.
Conclusion Through review of FCF procedures and interviews with engineering management and staff, the inspectors found that no formal requirements exist for direct technical training, such as the use of engineering computer codes. Training on administrative procedures is the only training that was officially documented. The inspectors considered this lack of formal requirements for tec,nical training and the limited incorporation of technical training to be a weakness.,
l 88
The inspectors did not identify any examples of production activities having been affected by the lack of documented training.
However, the inspectors concluded that the lack of documentation for the training of non-inspection personnel was a weakness in the implementation of the QA program.
3.13 Continuous Imorovement Procram The inspectors also interviewed the manager of the FCF continuous improvement program (CIP), part of the total quality management program at FCF.
Employees trained in CIP generally lead or were a member on a quality improvement teams (QIT).
Currently, 17 quality improvement teams have been formed.
The CIP program was based on a seven step program which included identifying, defining and analyzing a problem and determining and implementing a solution.
Some topics of recent QITs include eliminating sources of fuel failures and improvements in fuel rod welding.
The final report on QIT B5 on fuel integrity was examined as an example. The Fuel Integrity QIT evaluated approximately 200 ideas submitted as the root cause of the unknown fuel failures.
Formation of the inspectors, its processes for finding the root cause, the results of that effort, and recommendations for corrective action were included in the report.
The program appears to be an effective method for improving product quality.
3.14 Entrance and Exit Meetinas In the entrance meeting on March 18, 1996, the NRC inspectors met with members of FCF management and staff, and discussed the scope of the inspection.
The team also reviewed its responsibilities for handling proprietary information, as well as those of FCF.
In addition the inspectors established contact persons within the management and staff of the applicable FCF organizations.
During the exit meeting with FCF management and staff, on April 17, 1996, the inspectors discussed its findings and concerns, as well as FCF's weaknesses. 89
_. ~..
i i
i i
Figure 1 TMI-1 Cycle-10 Arrangement of the "T" Loading Patterns i
1 2
3 4
5 6
7 8
9 10 11 12 13 14 15 A
l 8
C D
E F
G H
K L
M N
O P
R i 90
PARTIAL LIST OF PERSONS CONTACTED Andrews, B.
Vice President, Fuel Engineering Armentrout, C.
Lead, Quality Audits and Programs Bohart, J.
President Carr, C. W.
Vice President, Manufacturing and Service Coleman, T.
Vice President Cudlin, J.
Unit Manager, Analysis Services Deveney, R.C.
Benchmarking / Methodology (Supervisor Nuclear Technology)
Engelke, W.
Manager, Process Quality Ford, J. T.
Manager, Fuel Manufacturing Gardner, R. L.
Manager, Quality Hanson, G.E.
Benchmarking / Methodology (Manager Fuel Engineering)
Harlow, N.
Technical Specialist Hobson, G.H.
Benchmarking / Methodology (Principal Engineer)
Lamanna, L.
Manager, Chemistry Engineer Matheson, J. E.
Manager, Design and Development Mayberry, J. R.
Manager, Product Quality 2
McPhatter, F.
Principal Engineer i
Meyer, G.
Manager, Thermal and Performance Analysis i
ITEMS OPENED AND CLOSED 4
1 i
Opened none i
Closed 99900001/90-01-01 NON burr on lower end fitting 99900001/90-01-02 NON radiographs as quality records 99900001/91-01-01 NON visual inspection of tubes with felt plugs 99900001/91-01-02 NON issuance of CDR and CVAR i
99900001/91-01-03 NON adequate guidance to operators and inspectors 99900001/91-01-04 URI root cause for the failure to detect felt plugs left in tubes
. 91
ACRONYMS USED ADL Applicable Documents List ANO-1 Arkansas Nuclear Unit 1 APSRA Axial Power Shaping Rod Assembly BOC Beginning Of Cycle BPRAs Burnable Poison Rod Assemblies B&W Babcock and Wilcox Company BWFC B&W Fuel Company BWST Borated Water Storage Tank CDR Component Discrepancy Report CFR Code of Federal Regulation CFT Core Flood Tank CIP Continuous Improvement Process CNFP Commercial Nuclear Fuel Plant COLR Core Operating Limits Report CR Concurrence Request CR3 Crystal River Unit 3 CRA Control Rod Assembly CRD Contract Requirements Document CVAR Contract Variation Approval Report DCP Distinctive Crud Pattern DCR Design Change Request A
Delta (Differential)
DNB Departure From Nucleate Boiling DNBR Departure From Nucleate Boiling Ratio DR Deviation Reports DRN Document Release Notice ECCS Emergency Core Cooling System ECP Engineering Computer Programs EFPD Effective Full Power Days FCDR Fuel Cycle Design Requirements FCF Framatome Cogema Fuels FFCD Final Fuel Cycle Design FOAK First-0f-A-Kind l
FSAR Final Safety Analysis Report FTI Framatome Technologies, Inc.
1 Gd Gadolinium GDC General Design Criterion GPUN GPU Nuclear Corporation HFP Hot-Full-Power HZP Hot-Zero-Power LBVs Licensing Basis Values LHR Linear Heat Rate Li Lithium LOCA Loss-of-Coolant Accident MTC Moderator Temperature Coefficient NIST National Institute of Standards and Technology NRC U.S. Nuclear Regulatory Commission PCT Peak Cladding Temperature PFCD Preliminary Fuel Cycle Design 92
1 5
PM Program Manager P0 Purchase Order ppe Parts Per Million PQ Product Quality PSC Preliminary Report of Safety Concern PTR Personnel Training Report PWR Pressurized-Water Reactor QA Quality Assurance QAR Quality Action Request QC Quality Control QIT Quality Improvement Team RA/ADL Release Authorization / Applicable Documents List RALS Reload Analysis and Licensing Services RC Route Card RCCA Rod Cluster Control Assembly RCS Reactor Coolant System RELATE Reload Licensing Analysis Task Engineer i
RPS Reactor Protection System i
SBLOCA Small Break Loss-of-Coolant Accident SRP Standard Review Plan SSL Supplier Status List 3D Three-Dimensional TMI-l Three Mile Island Unit 1 i
TR-Transmittal Records TVA Tennessee Valley Authority l
2D Two-Dimensional w/o Weight Percent a
i j
1 i
i 4
- 3' -
l 93
k2KfCo t
UNITED STATES NUCLEAR REGULATORY COMMISSION g
g WASHINGTON, D.C. 20&W4001
- \\*
/
November 4, 1996 Mr. Lawrence M. Froman, Manager Panalarm Business Unit Process and Analytical Instruments Division AMETEK, Inc.
7401 North Hamlin Avenue Skokie, IL 60076
SUBJECT:
NRC INSPECTION REPORT 99901303/96-01, NOTICE OF VIOLATION, AND NOTICE OF NONCONFORMANCE
Dear Mr. Froman:
On September 12, 1996, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at your Panalarm Business Unit facility.
The enclosed report presents the results of that inspection.
During.this inspection, the NRC inspector found that one of your activities appeared to be in violation of NRC requirements.
Specifically, your procedure for identifying, evaluating, and reporting deviations from the requirements of safety-related purchase orders confused the terms " deviation" and " defect"; it also failed to address significant changes in 10 CFR Part 21 reporting requirements and procedures since the pracedure was last revised more than six years ago.
'ce of Violation (NOV), and the The violation is cited in the enc' circumstances surrounding the vic 2 described in detail in the enclosed report.
Please note thac.
..e required to respond to this letter and should follow the instructions specified in the enclosed NOV when preparing your response. The NRC will use your response, in part, to determine whether further enforcement action is necessary to ensure compliance with regulatory requirements.
In addition, the NRC inspector found that the implementation of your quality assurance program failed to meet certain NRC requirements imposed on you by your customers.
Specifically, your procedures failed to adequately define the organizational independence and verification functions of your quality assurance group as required by 10 CFR Part 50, Appendix B.
Three nonconformances were identified in this regard.
These nonconformances are cited in the enclosed Notice of Nonconformance (NON), and the circumstances surrounding them are described in detail in the enclosed repurt.
You are requested to respond to the nonconformances and should follow the instructions specified in the enclosed NON when preparing your response.
i 94
Mr. Froman In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its enclosures will be placed in the NRC's Public Document Room (PDR).
1 Sincerely, 1
Robert M. Gallo, Chief Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation Docket No.
99901303 95 I
i h
NOTICE OF VIOLATION Docket No.: 99901303 Panalarm Division, AMETEK, Inc.
4 Skokie, Illinois 12, 1996, a During an NRC inspection conducted on September 9 throughIn c:cordance with the "
violation of NRC requirements was identified.
Statement of Policy and Procedure for NRC Enforcement Actions," NUREG-1600, the violation is listed below:
10 CFR 21.21, " Notification of failure to comply or existence of a defect and its evaluation," requires, in part, that each corporation subject to the regulations adopt appropriate procedures to ensure the evaluation and proper reporting of deviations and failures to comply.
Contrary to the above, Quality Control Procedure QC-90-100, "(10CFR21) htification Procedure," Revision 0, dated March 6,1990, failed to address the identification or evaluation of deviations.
The procedure incorrectly confused the terms " deviation" and " defect", with the result that personnel were not alerted to the need to-identify deviations from safety-related purchase order requirements, and to evaluate the deviations to determine if Procedure 0C-90-100 also failed to reflect they could become defects.
significant changes that were incorporated into 10 CFR Part 21 since 1990 concerning reporting requirements and the content of procedures.
(99901303/96-01-01)
This is a Severity Level IV violation (Supplement VII).
Pursuant to the provisions of 10 CFR 2.201, Panalarm Division of AMETEK, Inc.,
is hereby required to submit a written statement or explanation to the U.S.
Document Control Desk, Washington D.C.
Nuclear Regulatory Commission, ATTN:
20555, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within This 30 days of the date of the letter transmitting this Notice of Violation.
reply should be clearly marked as a " Reply to a Notice of Violatio should include for each violation:
contested, the basis for disputing the violation, (2) the corrective steps that have been taken and the results achieved, (3) the corrective steps that will be taken to avoid further violations, and (4) the date when full compliance will be achieved.
Your response may reference or include previous docketed correspondence, if the correspondence adequately addresses the Where good cause is shown, consideration will be given to required response.
extending the response time.
Dated at Rockville, Maryland this 4th day of November, 1996.
96
i j
NOTICE OF NONCONFORMANCE 2
Panalare Division, AMETEK, Inc.
Docket No.: 99901303 Skokie, Illinois Based on the results of an inspection conducted on September 9 through 12, 1996, it appears that certain of your activities were not conducted in accordance with NRC requirements.
i Criterion I of Appendix B to 10 CFR Part 50, " Organization," requires, in i
part,.that the quality assurance functions shall verify that activities i
affecting safety have been correctly performed.
The authority of persons l
performing activities affecting safety-related functions shall be clearly j
established and delineated in writing.
The persons performing quality assurance functions shall have sufficient authority and organizational freedom 1
to identify quality problems.
j A.
Contrary to the above, Paragraph 12.6 of Section 12, " Test Control," of the "AMETEK, Inc., Panalarm Division Quality Assurance Manual," Document
{
900181, Revision 10, dated December 8, 1993, specifies that documented j
test results are approved by engineering and/or quality assurance, i
Criterion I does not allow substitution of engineering for quality assurance review of final acceptance test results.
(99901303/96-01-02)
B.
Contrary to the above, the organization chart on page 3-2 of the i
"AMETEK, Inc., Panalarm Division Quality Assurance Manual," Document 900181, Revision 10, dated December 8,1993, does not reflect the current organizational status of quality assurance.
Furthermore, procedures such as Quality Control Procedure QC-76-47, " Training of Test and Inspection Personnel," dated April 25, 1986, referred to the QC i
supervisor and QC department, whereas the quality assurance manager j
stated th4t reference to his position was intended.
(99901303/96-01-03) i C.
Contrary to the above, the Panalarm inspector performing the final l
acceptance testing for safety-related temperature switch modules l
reported to a production supervisor, so that there was no independent j
quality assurance verification of the testing.
(99901303/96-01-04)
Please provide a written statement or explanation to the U.S. Nuclear i
Regulatory Commission, ATTN: Document Control Desk, Washington, D.C. 20555, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date of the letter transmitting this Notice of Nonconformance.
This reply should be clearly marked as a " Reply to a Notice of Nonconformance" and should include for each nonconformance: (1) a description of steps that have been or i
i i
3 97
will be taken to correct these items; (2) a description of steps that have been or will be taken to prevent recurrence; and (3) the dates your corrective actions and preventive measures were or will be completed, 1
I l
i Dated at Rockville, Maryland this 4th day of November, 1996. 98
4 U.S. NUCLEAR REGULATORY COMMISSION 0FFICE OF NUCLEAR REACTOR REGULATION Report No:
99901303/96-01 Organization:
Panalarm Business Unit Process and Analytical Instruments Division AMETEK, !ne 7401 North Hamlin Avenue Skokie, Illinois 60076 i
Contact:
A.H. Demerer, Quality Assurance Manager (847) 509-7196 l
Nuclear Industry Annunciator systems including temperature Activity:
monitoring modules l
DATES:
September 9-12, 1996 l
Inspector:
Richard C. Wilson, Senior Reactor Engineer APPROVED BY:
Gregory C. Cwalina, Chief a
Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs i
1 i
t t
i 99
1 INSPECTION SUMARY During this inspection, the NRC inspector reviewed the implementation of selected portions of the AMETEK, Inc., Panalarm Division quality assurance (QA) program, and reviewed activities associated with the supply of safety-grade replacement parts for annunciator systems supplied by Panalarm for nuclear power plants in the 1970s and early 1980s.
In the early 1970s the Panalit company produced Panalarm annunciator sysicias.
The company name was changed to Riley, and subsequent owners were US Filter and Ashland Oil, prior to acquisition by AMETEK.
Panalarm has approximately 100 employees.
Nuclear activity amounts to about 0.1% of sales.
The audit bases were:
Appendix B " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" During this inspection, a violation of NRC requirements was identified and is discussed in Section 3.1 of this report.
During this inspection, three instances where Panalarm failed to conform to NRC requirements imposed upon them by NRC licensees were identified. These j
nonconformances are discussed in Sections 3.2 and 3.3 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS This was the first NRC inspection of Panalarm.
3 INSPECTION FINDINGS AND OTHER COMENTS 3.1 10 CFR Part 21 Procram The inspector reviewed Panalarm's procedure for reporting in accordance with 10 CFR Part 21:
Quality Control procedure QC-90-100, "(10CFR21)
Notification Procedure," Revision 0, dated March 6,1990.
The procedure focused on evaluating a defect to determine if it was a reportable condition; it stated that if the defect was determined to not be a reportable condition, then no further action would be taken.
Procedure QC-90-100 incorrectly confused the terms " deviation" (which is not used in the procedure) and " defect" (which is incorrectly used). As a result, Panalarm personnel were not alerted to the need to identify deviations from purchase order requirements, and to evaluate the deviations to determine if they could become defects.
Further, the evaluation of defects as used in the procedure is not required, nor would it serve any useful purpose.
The procedure also did not address significant changes to Part 21 in the years since the procedure was 2
100
__.________s l
written,.regarding reporting requirements and content of procedures. As a result, the procedure failed to comply with 10 CFR Part 21.
Failure to have procedures that address the reporting and procedural content requirements of 10 CFR Part 21 constitutes Violation 99901303/96-01-01.
3.2 Quality Assurance Proaram a.
Insnection Scone The inspector selectively reviewed the "AMETEK, Inc., Panalarm Division Quality Assurance Manual," Document 900181, Revision 10, dated December 8, 1993, and the "AMETEK Panalarm Division Quality Control Procedures Manual," Document 900004, dated May 2, 1986, which described the Panalara quality assurance (QA) program.
During the inspection the inspector asked about a quality control (QC) procedure that was not included in the Procedures Manual, and the QA manager determined that an obsolete copy of the manual had inadvertently been provided.
The t
inspector then reviewed the table of contents for the January 28, 1994, revision of the QC Procedures Manual, which identified 30 additional QC procedures.
Two of these, covering 10 CFR Part 21 reporting and commercial grade item dedication, had already been identified and reviewed during the inspection.
The inspector selected and reviewed three others.
b.
Observations and Findinas Section 1 of the QA manual stated that the program is based on the American National Standards Institute standard ANSI N45.2-1971, and Section 2 stated that products supplied on purchase orders requiring compliance with Appendix 8 to 10 CFR Part 50 will be processed using the procedures described and referenced in the QA manual.
Revision 0 of the manual was dated May 5, 1977.
Panalarm management advised the inspector that Panalarm was considering changing the QA program to ISO format and was also considering ending their Appendix 8 program.
Neither decision had been made by the conclusion of the inspection.
The. inspector had two concerns with Panalarm's QA program.
The first relates to QA verification of the final acceptance testing of Model 86 modules, which is the principal basis for considering them to be safety-grade, as described in Section 3.3 below.
In Section 12. " Test Control," of the QA manual, paragraph 12.6 stated that documented test results are approved by engineering and/or QA.
Engineering review of the test results is a means of ensuring quality.
However, because there l
was little independent QA review of other activities affecting quality, such a review of the final acceptance testing is necessary to satisfy Criterion I of Appendix 8 to 10 CFR Part 50, " Organization," which requires the independent QA organization to verify that activities affecting safety-related functions have been correctly performed.
3 101 t
The inspector also determined that Paralarm procedures did not reflect the current organization structure. The organization chart dated July 1996 that was provided to the inspector differed from the chart included in Section 3, " Organization", in QA Manual Revision 10 dated December 8, 1993.
The QA manual chart shows the QA function reporting to the operations manager, with no alternate path specified; the July 1996 chart shows the QA manager reporting to the production manager, but with a dashed line representing an administrative link directly to the division vice president and manager.
Several procedures also contained confusing references to QA and QC positions.
For example, QC Procedure QC-76-47, " Training of Test and Inspection Personnel," dated April 25, 1986, referred to the QC Supervisor and the QC department, but the QA manager stated that reference to his position was intended.
This clarification is significant because Panalarm's production department has QC personnel who lack the organizational independence required by Criterion I of Appendix 8 to 10 CFR Part 50.
c.
Conclusions Panalarm's failure to Thermocouple Monitor require QA verification of final acceptance testing const; Lutes Nonconformance No. 99901303/96 02.
Panalarm's failure to provide clear procedural r'eferences to the independent QA function where required constitutes Nonconformance i
No. 99901303/96-01-03, 3.3 Model 86 Thermocouple Monitor a.
Inspection Scoce g
Prior to the inspection, the QA manager identified the Temp-Matic (or TempMatic--Panalarm literature uses both terms) Model 86 thermocouple monitor as the only safety-related component currently supplied.
The inspector reviewed test and dedication procedures and reports, and other pertinent documentation, and observed a final acceptance test demonstration.
b.
Observations and Findinas The Model 86 analog module accepts a thermocouple input, compares it with an adjustable setpoint, and provides status light and contact outputs.
Variations include a differential unit accepting two thermocouple inputs, a " half-gain" version, and a burnout protection feature.
The module consists of two small printed circuit boards with several interconnecting wires, joined by standoffs and bolts to form an assembly about the size of a pack of cigarettes. The module has no case, and is intended for insertion into an annunciator grid structure in the control room.
Licensees commonly refer to the module as a temperature switch. The Model 86 is not part of current Panalarm annunciator systems.
It is supplied in small quantities as a replacement part.
4 102
The inspector reviewed three 1973-4 qualification type test reports and verified that they identified the test specimens clearly enough to permit determining the similarity of current units.
The inspector also reviewed subsequent engineering change notices that documented design t
reviews.
These documents appeared to be adequate for tracing design changes of the Model 86 modules.
Panalarm completed a dedication procedure for the Model 86 temperature modules in 1992 that provided the following:
Document 900396, "Model 86B Temp-Matic EPRI Contormity,"
Revision 0, dated November 5, 1992 (currently in Revision 3 dated January 30,1995), identified each part in the Model 86 versions as critical or non-critical, and tabulated critical characteristics and their verification methods for the critical parts.
Quality Control Procedure QC-92-110,." Commercial Grade Item Dedication for Safety Related Assemblies - Quality Control Procedures for Dedicated Items," Revision 0, dated July 30, 1992, specified batch receipt inspection and controlled handling practices for critical parts; it also required training for personnel performing these activities, and for testing personnel.
Not,es referencing these two new documents were added to the two assembly level drawings for Model 86 modules, 86800-P-2 and 86800-V-1.
The NRC inspector found that the Model 86 parts dedication activities did not address possible heterogeneity of comercial grade parts lots, and no audits of commercial grade parts suppliers were performed.
However, the receipt inspection did verify visual and performance characteristics for many parts, which were then subject to controlled
~ handling.
The principal basis for considering the completed Model 86 modules to be safety grade was.the final acceptance testing, which was relied on for dedication of some parts as well as performance of the assembled module.
The acceptance test procedure written for GE contracts--Document No.
108-8022-900035, " Final and Acceptance Test Procedures.for Model'86 Tempmatic Modules for General Electric (GE] Nuclear Energy Division Jobs," Revision 16, dated December 29, 1989, was more comprehensive than the previous procedure, Document No. 900024, " Calibration and Fir.a1 Test Prxedures for Temp-Matic Model 86," Revision 2, dated July 11, 1975.
The new procedure repeats more of the in-process calibration points; provides more detail for model variations; includes a check for input loop resistance; tests the optional burnout protection; and requires 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> post-test operation. Although the new procedure's title would appear to restrict its use to P0s from GE, this is apparently not the case.
Dedication procedure QC-92-110 requires use of procedure 900396 for dedicating Model 86 switches. This document in turn requires testing in accordance with the revised procedure to verify certain critical 5
103
~.
characteristics at the module level. As discussed below, an IES Utilities audit report stated that they would continue to invoke QC-92-110 and 900396 in P0s.
Although no safety-related work was in progress during the inspection, Panalam demonstrated final and acceptance testing for a Model 86 module using Procedure 108-8022-900035.
The NRC inspector identified a few concerns with the procedure:
steps 6 and 7 of Part A did not distingeish between test values for standard and half-gain modules; Figure 2 specified an incorrect wire type for type J thermocouples; and the procedure did not specify that high and low range, but not setpoint, potentiometers are sealed with glyptol.
In each case the Panalarm inspector was aware of the correct procedure, and explained it.
The NRC inspector did not consider these concerns significant; Panalarm agreed to correct them.
The NRC inspector noted that the Panalarm inspector performing the final acceptance test reported to a production supervisor, and there was no oversight by the QA organization. This practice was not in conformance with Criterion I of Appendix B, " Organization", which requires the QA organization to verify that activities affecting safety-related
~
functions have been correctly performed.
The Panalarm QA manager agreed to immediately notify affected personnel that either the QA manager or the QC technician reporting to him must witness the final acceptance testing of safety-related equipment.
The schedule for a written i
procedural change depends on Panalarm's decision concerning continued supply of safety-related equipment.
The inspector found no purchase orders for non-GE procurement of Model 86 modules.
This was partially because the P0s initially identified by the QA manager did not constitute a complete set of recent safety-related orders.
Subsequently, an additional P0 from a licensee was traced through a file cabinet containing certificates of conformance to a separate set of PO files, but time did not permit pursuit of possible additional safety-related P0s through this path. The additional PO was for the Entergy Operations procurement of isolators described below.
Since the Model 86 dedication program was performed in response to GE requirements, an'd the acceptance test procedure for GE orders was more extensive than the older Panalarm procedure, it would be necessary to examine specific licensee procurement requirements to determine whether direct licensee procurements'of Model 86 modules would be acceptable.
As discussed below, an audit report by IES Utilities stated that their safety-related P0s would continue to invoke Panalarm procedures that require the use of the GE acceptance test procedure, but no related procurement records were available to the NRC inspector within the available time limits.
c.
Conclusions 6
104
The inspector concluded that Penalarm's QA program for Model 86 modules i
supplied to GE was acceptable.
The acceptability of direct licensee procurements would depend on the specific requirements of P0s.
Panalarm's failure ~ to perform incependent verification of final acceptance testing constitutes Nonconformance No. 99901303/96-01-04.
3.4 Purchase Orders l
a.
Egag i
The inspector reviewed three P0s from GE and two from Entergy Operations, Inc.
The P0s covered a variety of equipment and services, l
as described below.
b.
Observations and Findinos E:
GE Nuclear Energy P0 No. 52896020591 dated April 19, 1996, ordered three Model 86-BVTFF-EGCUST temperature switches.
The P0 stated that the switches were safety-related, invoked 10 CFR Part 21, and required the items to be either purchased safety-grade, dedicated, or evaluated as performing no safety-related function.
The switches were delivered
{
to GE San Jose, California, for use at the Grand Gulf power station.
i Panalarm provided acceptance test data sheets which identified the test instruments, and certification of conformance to contract requirements which identified customer drawing no. 164C5687P302, Revision 16.
The data sheets were signed as attested to by the QA manager two days after the production inspector's signoff.
Except for this latter instance of Nonconformance 99901303/96-01-04, the NRC inspector found the
}
documentation for the P0 to be acceptable.
1 i
GE Nuclear Energy P0 No. 205-95L289 dated March 17, 1995, ordered two temperature switches, GE drawing no. 164C5687P320.
The switches were delivered to GE San Jose for use at the Nine Mile Point unit 2 (NMP2) power plant.
In this case the data sheets were attested to by the former quality control manager en April 27, 1995; there is no evidence j
that he actually witnessed the testing, although that was the same day as the production inspector's signoff.
The inspector had no other concerns with the documentation for this P0.
The inspector also briefly reviewed GE PO No. 205-95L600 dated July 13, 1995.
This P0 called for a failure analysis of a temperature switch from NMP2, and required scrapping the switch after submittal of the report.
The PO stated that, when the switch was placed in service on
)
January 7, 1991, a plant operator touched the switch and it spuriously alarmed. The P0 identified the switch as safety-related, and invoked i
Panalarm provided a certification of meeting the PO requirements. A factory service repair report stated that the unit tested satisfactorily, and test. points were verified; spurious alarms were noted only at the edges of the deadband, which is typical of all units. An acceptance test data sheet was also provided, with attestation by the QA manager three days after the production i
inspector's signoff. The NRC. inspector noted that Panalarm's GE test 7
l 105
procedure specified that all setpoints should be set for approximately mid-scale trip before shipping.
The inspector was aware of three LERs, but no 10 CFR Part 21 reports, submitted by NMP2 in recent years involving Panalarm temperature switch modules.
LER 92-001 dated February 3, 1992, addressed a temperature switch that generated a spurious high temperature alarm; it stated that from April 1987 to February 1992, NMP2 had 76 work requests for the repair / replacement of Panalarm temperature switches, with drift in and out of the alarm state as a predominant failure mode. The GE part number for the LER switch was 164C5687P001, whereas the part number for
.the switch returned to Panalarm for evaluation was 164C5687P301; the LER did not give a serial number, but evidently the switch addressed in the
. LER was not the one returned to Panalarm. Another NMP2 LER,92-024, reported a temperature switch with a trip relay coil that failed open, causing a downscale trip output.
Both of these LERs indicated plans for ceplacement of all Panalarm temperature switches with another uanufacturer's products. NMP2 LER 94-04 dated January 17, 1994, referred to replacement of at least 65 Riley temperature switches, but the LER did not identify the replacement equipment.
Enterav:
Entergy Operations, Inc. P0 No. MP519933 dated July 19, 1995, covered two GG81147001 optical isolators for the Grand Gulf nuclear station.
The P0 stated that the parts shall be qualified replacements for original equipment qualified to IEEE 323-74 and 344-75 by Wyle test report 4407411, and referenced the original Riley job no. 607-7002Q; it also required electrical tests "as acceptable by Riley Co."
The P0 stated that the' equipment performs a safety-related function, invoked 10 CFR Part 21, and required the seller to comply with his QA program as approved by the purchaser.
The NRC inspector observed an Entergy letter to Panalarm dated February 23, 1994, which stated that based upon review of Revision 10 of the Panalarm QA manual dated December 8,1993, the Panalarm QA program was considered acceptable to Entergy. The letter also called for Entergy review of any program revisions prior to implementation.
The Panalarm QA manager had no evidence of an Entergy audit, although he was not required to have any.
Panalarm provided a certificate of compliance with P0 requirements. A test report for each isolator showed input and output voltage levels for ten channels measured per procedure QC-77-53 dated November 23, 1981.
Entergy P0 No. WP061473 dated November 23, 1994, covered 20 type LP92268005 printed circuit boards for the Waterford 3 plant.
The P0 stated that the boards were for use on an annunciator panel originally furnished under Ebasco/Riley P0 #NY-403540 dated January 30, 1976, through Supplement #8 dated July 26, 1979..(The NRC inspector found that a 1981 Ebasco audit of Riley found them acceptable for Appendix B listing.) The PO required the item' manufacturer's certification that there were no changes in design, material, manufacturing, or interchangeability between January 30, 1976, and the date of manufacture of the replacement boards. Alternately, a list of changes could be provided for Entergy's approval.
The PC invoked the latest version of 8
106
l Panalara's QA manual, and the QA program which has been approved by Entergy's supplier audit supervisor.
The P0 also stated that the provisions of 10 CFR Part 21 apply to the item included on the order.
The P0 called for certification of conformance to all P0 requirements, 1
and required certification of an engineering equivalency evaluation (but not submittal of the evaluation itself) for any substitute parts, with revised technical manual'pages or product information.
Panalarm ce:'.' Tied that no changes had been made that affected qualification to IEEE 323-1974 and IEEE 344-1975. A list of drawing revisions was provided, with engineering change notices attached that identified 'several parts changes.
Panalarm supplied an inspection y
j report, but there was no evidence of testing.
1 The NRC inspector reviewed an Entergy letter dated February 23, 1994.
The letter stated that based on review of Revision 10 of Panalarm's QA manual, the QA program was considered acceptable for safety-related P0s.
The letter did not mention,any audit, and the Panalarm QA manager had no evidence of an Entergy QA audit, or of Entergy surveillance of any activity on the P0.
The P0 did not specifically invoke Appendix B or state that the equipment was safety-related.
If Entergy did intend to i
perform a safety-related procurement, the NRC inspector did not observe i
documentation at Panalarm that supported that objective.
c.
Conclusions The inspector concluded that the Model 86 modules supplied to GE conformed to Appendix B of 10 CFR Part 50.
The optical isolators supplied to Entergy were not demonstrated to be safety-grade by the Entergy and Panalarm documentation reviewed.
Although Entergy's P0 for printed circuit boards invoked 10 CFR Part 21 and Panalarm's QA manual, it was not clear that the P0 covered safety-related equipment. Absent evidence of an audit or surveillance, there was also.no basis to support Method 2 or 3 dedication of the boards for safety-related use.
3.5 Customer audits a.
Scope The inspector reviewed reports of audits by GE, Wolf Creek Nuclear Operating Corporation, and IES Utilities, and a Commonwealth Edison surveillance report.
b.
Observations and Findinas GE performed an audit of Panalarm on March 7 and 8, 1996.
Upon resolution of the audit findings, GE intended to retain Panalarm as an approved supplier of safety-related. temperature switches and related commercial grade items. The Panalarm QA manager stated that at the time of the NRC inspection, both GE corrective action requests remained open; i
l 9
107
they involved Panalarm's failure to conduct triennial internal and external audits. All five observations were closed.
Wolf Creek scheduled an audit of Panalarm on May 23-24, 1996.
The intent of the audit was to extend the Appendix B scope of Panalarm to include inverters and isolators, and to address Panalarm's failure to notify of a mandatory inspection hold point on a safety-related P0 (this finding was closed by a September 3, 1996, letter).
While at Panalarm, the Wolf Creek representative determined that programmatic refinements would be necessary to maintain an Appendix B program, although he j
considered the procedures adequate to support a commercial grade program.
Panalare did not plan to revise the QA program or j
documentation pending decisions concerning ISO and Appendix B commitments, so the Wolf Creek representative changed his audit to become a surveillance of activities on an existing P0.
Wolf Creek did not extend the safety-related scope of supply.
The NRC inspector considered the' Wolf Creek representative's actions to be prudent.
IES Utilities, for the Duane Arnold Energy Center (DAEC), conducted an audit of Panalarm on June 13-15, 1995, as a triennial followup to a 1992 audit.
The audit had three findings and seven observations, the last of which were closed by an IES letter dated February 7, 1996.
Based en the audit, Panalarm was retained as an approved Appendix B supplier of Model 868 switches, including their repair, for the DAEC.
The NRC inspector noted that the IES audit report and related correspondence were distributed to several other licensees.
Page 1 of the audit report states that it was performed to fulfil DAEC requirements, and there was no indication that it was a joint utility group audit.
The audit report also stated that Panalarm documents 900396 Revision 3 and QC-92-110 Revision 0 would continue to be invoked on DAEC P0s, and specified that subtier supplier documentation would have to be verified by testing, source inspection, and/or audit / survey.
Commonwealth Edison (Comed) conducted a survey of Panalarm on April 6, 1992.
The announcement letter stated that CcmEd intended to perform Method 2 dedication. Comed's May 21, 1992, letter stated that Panalarm was listed as an approved manufacturer of commercial grade items for Series 10, 50, and 70 annunciators and parts.
The Panalarm QA manager had no subsequent audit information regarding Comed.
The NRC inspector noted that some of the audits by Panalarm customers raised concerns with Panalarm's 10 CFR Part 21 program, yet after the concerns were resolved the Panalarm Part 21 procedure was still inadequate as described in Section 3.1.
c.
Conclusions The inspector concluded that the customer audits of Panalarm were effective, apart from the findings of this inspection report.
3.6 Entrance and Exit Meetines 10 108
In the entrance meeting on September 9, 1996, the NRC inspector discussed the scope of the inspection, outlined the areas to be inspected, and established 16terfaces with Panalarm management.
In the exit meeting on September 12, 1996, the inspector discussed his findings and concerns.
LIST OF PERSONS CONTACTED Lawrence M. Froman, Division Vice President and Manager Arlin H. Deserer, Quality Assurance Manager David Minerd, Production Manager Lorenzo Garcia, Production QC Inspector Lou Granato, QA/QC Technician i
I 11 109
4 l
ITENS OPENED, CLOSED, AND DISCUSSED Opened 99901303/96-01-01 VIO inadequate Part 21 procedure 99901303/96-01-02 NON inadequate provision for verifying safety-related activities 99901303/96-01-03 NON inadequate organizational definitions in QA manual 99901303/96-01-04 NON inadequate inspection independence Closed i
Noiia Discussed j
None 12 110
M ao k
UNITED STATES 41j j
NUCLEAR REGULATORY COMMISSION l
2 WASHINoTON, D.C. 206S4001 p
December 31, 1996 Ms. F.J. Harvey, Acting President Energy Systems Business Unit Westinghouse Electric Corporation P.O. Box 355 Pittsburgh, FA 15230
SUBJECT:
NRC INSPECTION REPORT 99901307/96-01
Dear Ms. Harvey:
On October 4,1996, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection of Westinghouse Energy Systems Business Unit (ESBU) facilities in Monroeville and Cheswick, Pennsylvania.
The enclosed inspection report presents the results of that inspection.
The inspection reviewed the implementation of your quality assurance program in supplying safety-related equipment, software, and services to NRC-licensed facilities, and reviewed your program (and its implementation) established pursuant to Part 21, " Reporting Defects and Noncompliance," of Title 10 of the Code of Federal Reaulations (10 CFR Part 21).
i During this inspection, the NRC inspector found that one of your activities appeared to be in violation of NRC requirements.
Specif.ically, your system of procedures adopted pursuant to 10 CFR Part 21 contained minor weaknesses and l
inconsistencies, that, taken in the aggregate, may render them inadequate to meet the requirements of $21.21(a) of 10 CFR Part 21.
In accordance with the i
NRC Enforcement Policy as promulgated in NUREG-1600, this is considered a minor violation and therefore, no Notice of Violation will be issued.
j t
Although the NRC expects that ESBU will take prompt and appropriate corrective action, no response to this item is required.
The NRC inspector found no instances in which ESBU failed to comply with other requirements of 10 CFR Part 21.
However, one unresolved item was identified involving Corrective Action Reports of the ESBU Process Control Division.
The question of adequacy of the justification for closeout of the reports and hence, the potential existence of deviations or failures to comply requiring evaluation in accordance with 10 CFR Part 21 remained unresolved at the end of the inspection.
You are requested to respond to the unresolved item identified in Section 3.1.b.3 of the enclosed report.
In accordance with 10 CFR 2.790 of the NRC's Rules of Practice," a copy of this letter and its enclosures will be placed in the NRC's Public Document Room (POR).
111
2 l
Ms. Harvey,
We appreciate the cooperation of your staff during this inspection.
If you i
have any questions, please contact Mr. Stephen Alexander at (301) 415-2995.
Sincerely, l
Sblif%
F, Hief Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation cc/wencl:
Mr. Nicholas Liparulo, Manager l
Regulatory and Engineering Networks Docket Number:
99901307 l
i 112
U.S. NUCLEAR REGULATORY COMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No.
99901307/96-01 Organization:
Westinghouse Electric Corporation Energy Systems Business Unit i
P.O. Box 355 Pittsburgh, PA 15230 k
Contact:
H.A. Sepp, Manager, Regulatory and Licensing Initiatives l
Nuclear Industry Nuclear Safety-Related Equipment, Components,
~
j Activity:
Replacement Components and Parts, Software, and Services for the Commercial Nuclear Power Industry i
Dates:
September 23 through October 4, 1996 Inspectors:
Stephen D. Alexander, Reactor Engineer Ralph R. Landry, Senior Reactor Engineer Approved By:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs 113
_ _ ~ _ _ _ _ _ _ _ _ _ _ _ _. _ _ _ _.. _ _.. _ _ _ _. _.
1 INSPECTION SUMARY During this inspection, the NRC irispectors reviewed the implementation of selected portions of the quality assurance program of Westinghouse Energy Systems Business Unit (ESBU) for supplying safety-related equipment, software, and services to NRC-licensed facilities. The review focused on the ESBU program (and its implementation) established pursuant to Part 21, " Reporting of Defects and Noncompliance," of Title 10 of the Code of Federal Reaulations (10 CFR Part 21).
The inspectors also reviewed specific technical concerns with regard to certain considerations in the design of some safety analysis computer models that may be deficient and that may lead to nonconservative results, that were not identified in software error vanarts.
Compliance with 10 CFR 50.46 reporting requirements were also within the scope of this inspection.
The inspection bases were:
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to 10 CFR Part 50 10 CFR Part 21, " Reporting of Defects and Noncompliance" 10 CFR Part 5::, Appendix K, "ECCS Evaluation Models" 10 CFR 50.46, " Acceptance Criteria for Emergency Core Cooling Systems for Light Water Nuclear Power Reactors" The NRC inspectors found that the ESBU system of procedures (including those of its major divisions) adopted pursuant to 10 CFR Part 21 contained various weaknesses and inconsistencies that, taken in the aggregate, may render them inadequate to meet the requirements of 621.21(a).
In accordance with the NRC Enforcement Policy as promulgated in NUREG-1600, this was considered a minor violation and no Notice of Violation was issued.
Although the NRC inspector found no conclusive evi9ence of instances in which ESBU failed to comply with other requirements of 10 CFR Part 21, two instances were identified involving Corrective Action Reports of the ESBU Process.
Control Division. The question of adequacy of the justification for closeout of the reports and hence, the potential for the existence of deviations or failures to comply requiring evaluation in accordance with 10 CFR Part 21 remained unresolved at the end of the inspection.
This unresolved item is discussed in Paragraph 3.1.b.3 of this report.
The inspectors found no instances in which ESBU was not in compliance with 10 CFR 50.46 or 10 CFR Part 21 with regard to its system of tracking changes in predicted reactor core parameters (e.g., peak cladding temperature (PCT))
under modelled design basis accident conditions and reporting them to affected licensees.
2 114
2 STATUS 0F PREVIOUS INSPECTION FINDINGS No. previous findings were reviewed during this inspection.
3 INSPECTION FININGS AND OTHER COMMENTS 3.1 10 CFR Part 21 Procram a.
Part 21 Program Review Scope The inspector reviewed the procedures adopted pursuant to 10 CFR Part 21 by ESBU (ESBU-21.0 and its predecessors, ESBU-19.0 and OPR-19.0) and those of selected major ESBU divisions (called Level II procedures, Level I being the ESBU Quality Systems Manual) including the:
Nuclear Services Division (NSD), Monroeville, PA Prc:ess Control Division (PCD), Ohara Township, PA Electro-Mechanical Division (EMD), Cheswick, PA Systems and Major Projects Division (SMPD), Monroeville, PA l
Commercial Nuclear Fuels Division (CNFD), Columbia, SC The inspector noted that the Part 21-related procedures of the Advanced 1
Technology Business Area, Operating Plant Business Area, and the Westinghouse Pensacola, Florida, Division (steam generators and pressurizers) feed into ESBU-21.0 as well.
The inspector also reviewed lower tier (called Leval III) Part 21-related procedures for selected departments within the major divisions including those of Replacement Component Services (RCS; of NSD and Nuclear Safety Analysis (NSA) Department of SMPD.
In addition, the inspector reviewed Level II and Level III procedures used by the various ESBU divisions and departments for implementation of Criterion XV,
" Control of Nonconforming Material," and Criterion XVI, " Corrective Action,"
of 10 CFR Part 50,' Appendix B, to determine how effective those procedures are and how well they are implemented to identify those nonconformances that constitute deviations or failures to comply in basic components that have been supplied or offered for use at NRC-licensed facilities, thus requiring evaluation under Part 21 using the ESBU-21.0 process.
To assess implementation, the inspector reviewed the records of Part 21 evaluations (called Potential Issue (PI) evaluations by ESBU-21.0) performed by the ESBU Safety Review Committee for 1991-1995, with emphasis on those that did not result in NRC notification.
Then the inspector reviewed selected Material Review Reports and Corrective Action Reports (CARS) of EMD; Defective Item Notices (DINS), Design Engineering Orders and Design Engineering Order Notifications (DE0s and DE0Ns), and CARS of PCD: and Software Error Reports of 3
115
the Nuclear Safety Analysis (NSA) Department of SMPD. These records were reviewed to determine which material nonconformances and computer software errors constituted deviations or failures to comply as defined in 521.3, which of these were identified as such and which were evaluated (or should have been evaluated) in accordance with 10 CER Part _21.
b.
Part 21 Program Review Observations b.1 Part 21 Program Procedures ESBU has implemented 10 CFR Part 21 requirements, and the related criteria of 10 CFR Part 50, Appendix B, by a hierarchy of procedures on three levels:
Level 1: The ESBU Quality Systems Manual (QSM)
Level II:
ESBU and major division QA implementing procedures for 10 CFR Part 21 and 10 CFR Part 50, Appendix B, Criteria XV and XVI Level III: Departmental level implementing procedures that would aid in recognition and identification of conditions adverse to safety as defined in ESBU-21.0 The principal Level II procedure for overall ESBU Part 21 compliance is ESBU-21.0, " Identification and Reporting of Conditions Adverse to Safety."
ESBU 21.0 (as well as its predecessors, ESBU-19.0, and OPR-19.0) is applicable to, and is used directly by most ESBU divisions, including PCD, NSD, SMPD, and EMD.
b.2 Part 21 Program Adequacy
)
During the review of the Level II and III procedures mentioned above (except for those of CNFD) of ESBU and selected divisions and departments with emphasis on the various revisions and versions of ESBU-21.0 back to 1993, the inspector determined that Revision 1 of ESBU-21.0, dated September 20, 1996, (as well as previous revisions reviewed), could not be fully relied upon to ensure that all deviations and failures to comply would be evaluated and reported in accordance with 621.21 because of the following weaknesses and inconsistencies:
(1) The Westinghouse-coined combined category of " conditions adverse to safety" (CASs), while useful and complete (including both deviations and failures to comply as well as departures from Westinghouse specifications or requirements) was not used consistently throughout the procedure such that failures to comply and some deviations could be excluded from consideration.
(2) The director or responsible officer designated by the procedure to receive reports of defects or failures to comply associated with substantial safety hazards per $21.21(a)(3) was not consistent'throughout the procedure.
Section V of ESBU-21.0, under "ESBU Safety Review Committee Chairman Responsibilities," required that " recommendations" (the committees findings]
be reported to the "ESBU Vice President and General Manager, or his representative," a position that no longer existed at the time of the 4
116
i i
inspection. However, the procedure designated the ESBU President as the officer responsible for reporting of defects or failures to comply associated with substantial safety hazards to the NRC pursuant to f 21.21(c). The 4
position of ES8U President was not permanently filled at the time of the inspection, although the Chief Operating Officer of the Westinghouse Industries and Technology Group was acting as ESBU President.
The Acting ESBU l
President, in a September 9, 1996, memorandum (WIN: 272-4914), had delegated the responsibility for receiving 521.21(a)(3) notifications to the General Manager of ESBU Operations.
Other observations regarding ESBU-21.0 were as follows:
The description of Part 21 on Page 1 of 13 of ESBU-21.0, in listing those things with which a basic component may fail to comply, omitted the Atomic Energy Act of If,54, as amended, and licenses of the NRC.
]
Among the definitions of Section II, on page 4 of 13, in Paragraph 2 under " Basic Component," the list of things with which a basic component may fail to comply omitted the Atomic Energy Act of 1954, as amended, 4
?
and rules of the NRC.
Discovery was not defined as in 521.3, i.e., the completion of i
documentation of a deviation or failure to comply with the potential for
. creating a substantial safety hazard.
Rather, the procedure simply describes discovery as when a PI is opened. The procedure starts the i
60-day evaluation period when a PI is opened as opposed to clearly requiring the evaluation to be completed within 60 days of Part 21-defined discovery as required by 621.21(a)(1) or an interim report to be submitted to the NRC within 60 days of Part 21-defined discovery as required by 121.21(a)(2).
ESBU-21.0 provided the vague requirement to the ESBU Safety Review Committee (SRC) to " review referred items relative to the requirements of NRC regulations" instead of delineating the specific review requirements of Part 21, i.e., to determine if the deviation (CAS) being evaluated is a defect (that is, could it (a) create a substantial hazard, or (b) lead to exceeding a technical specification safety limit); or if the failure to comply (CAS) being evaluated could be l
associated with a substantial safety hazard.
The requirement for interim reporting on Page 11 of 13,Section VI, Paragraph 2(b) under " Notification Guidelines," cites 10 CFR 21.21(b) which is incorrect.
Interim reports are required by $21.21(a)(2).
Appendix A to this report lists other Level II and III divisional and departmental procedures reviewed that were pertinent to the recognition and identification of deficiencies as deviations or failures to comply (i.e.,
conditions adverse to safety as defined in ESBU-21.0) that should be submitted to the ESBU SRC for evaluation /further evaluation per (21.21(a)(1).
Some specific comments are included.
5 117
b.3 Part 21 Program Implementation / Compliance The inspector reviewed selected records of evaluations of documented deviations and failures to comply, called " potential issues" (PIs) by ESBU-21.0, with emphasis on those that did not result in NRC notification per 621.21(c) because the ESBU SRC had determined that the deviations were not defects or the failures to comply were not associated with substantial safety hazards. -The inspector found only one PI that required additional information (which was provided) beyond that in the file to determine its acceptability.
None were identified that should have been reported to the NRC.
The inspector reviewed selected records of the various divisions and departments within ESBU to assess the effectiveness of their procedures and programs in the recognition and identification (among the various types of deficiencies, error reports, and nonconformances, that are tracked) of deviations and failures to comply as defined by 621.3 that would need to be evaluated per the ESBU-21.0 process in compliance with f21.21(a)(1).
PCD Defective Item Notices (DINS):
The inspector identified no DINS that should have been recognized as deviations or failures to comply as defined in 621.3 (or CASs as defined in ESBU-21.0) in delivered basic components, but were not.
PCD Corrective Action Requests (CARS): The inspector identified two CARS,92-064 and 93-002, with questionable justification for closeout.
The CARS' dealt with the failure of certain purchased components to meet i
all test requirements.
The components had been used in equipment supplied by PCD as basic components.
PCD's disposition of the CARS was that the components were satisfactory as shipped. However, there was insufficient evidence in the records available for review for the inspector to reach that conclusion.
PCD agreed to follow up on the CARS in question and provide additional information.
Should the justification ultimately be deemed inadequate, the potential exists for unrecognized (undiscovered), and hence unevaluated, deviations or failures to comply.
This item will require review of additional information being developed by the vendor. Designated Unresolved item 99901307/96-01-01 PCD Development Engineering Order Notices (DE0Ns):
The inspector identified no DEONs that should have been recognized as deviations or failures to comply in delivered basic components and were not.
EMD Material Review Reports (MRRs):
The inspector identified no MRRs at EMD that should have been recognized as deviations or failures to comply in delivered basic components and were not.
NSA Software Error Reports: The NSA Department of SMPD handles errors in software codes in accordance with NSA's procedure WP-4.19.3, Revision 0, August 31, 1996, formerly DP-3.7.4 (Revision 3, January, 31, 1993, reviewed) and then WP-3.7.4 (Revision 5, August 14, 1995, reviewed),
" Software Error Reporting and Resolution."
In a few cases, some of the safety analysis computer codes are used by certain utilities directly 6
118
i (e.g. the LOCBART and NOTRUMP loss-of-coolant-accident (LOCA) codes are used by Virginia Power (forme.rly VEPCO)).
In these instances, the code itself, and not the analysis, is the delivered basic component.
In i
accordance with ESBU procedures, (and as evidenced in the records notifies these utilities promptly of the error as soon as an error) NSA report is opened because they are direct users of the codes.
Therefore, the user notification meets the requirements of 121.21(b) with regard to identified code errors regardless of the results of further analysis of the imp.ct of those errors (and their correction and reanalysis) on-predicted core accident parameters.
The inspector found that although it had taken Westinghouse as long as 13 months (one instance in 1992) to close out an error report, the determination of the error /cause, correction of the error (writing new code), verification and validation, and rerunning all affected plant analyses using the corrected code to determine the impact of the error (i.e., nonconservatisms) quantitatively (or qualitatively) are
' legitimate discovery activities.
That is, it is part of the discovery process, for which Part-21 has no prescribed time limit, to conduct these activities to determine if the existing analyses (which are the delivered basic components-in most cases, not the codes themselves, fail to comply (e.g., with 10 CFR 50.46, 10 CFR Part 50, Appendix K, or technical specifications) or contain a deviation (departure from a technical procurement specification). Note that the inspector reviewed selected licensee procurement documents (contracts with ESBU for analytic services) and found none in which any of the software errors reviewed by the inspector could be construed as a deviation from the largely non-technically prescriptive language in the contracts.
Either an actual deviation (i.e., an analyzed nogative impact of a software error) or failure to comply, should they have the potential to create a substantial safety hazard or contribute to exceeding a safety
-limit, must be evaluated per 10 CFR 21.21(a)(1) or must be reported to affected licensees or purchasers per 621.21(b).
For example, if an error in the NOTRUMP code, used in conjunction with LOCBART, should be
. identified, ESBU would need to determine what effect or impact such an error, when corrected, would have on, for example, peak cladding temperature (PCT). Only then could ESBU reasonably determine whether any existing plant nuclear safety analyses, that may have been performed using the hypothetically faulty or nonconservative codes or computer models, constitute or contain deviations or failures to comply as defined in 121.3.
In addition, the inspector noted a few instances in which the changes in PCT resulting from a code error (determined after correcting the error and running preliminary analyses to asses the potential error magnitude) were large enough for ESBU to open a PI, and evaluate it under
$21.21(a)(1) per ESBU-21.0 or its predecessors.
The inspector also found that there have been improvements in the interface between the error reporting and resolution procedures and the Part 21 procedure. Also, there was an improving trend in closing out 7
119
. ~
. - - - -.. -. - -. -... - ~
error reports promptly due to increased management attention and followup. The inspector identified no software error reports that should have been. treated as deviations or failures to comply in delivered basic components and were not.
c.
Part 21 Program Review Conclusions The NRC inspector concluded that the ESBU system of procedures (including those of its major divisions) adopted pursuant to 10 CFR Part 21,121.21(a),
contained various weaknesses and inconsistencies that, taken in the aggregate, i'
may render them inadequate to meet the requirements of $21.21(a). That is the procedures could not be relied upon to ensure that (1) all deviations and failures to comply would be adequately evaluated to identify defects and failures to comply associated with substantial safety hazards (121.21(a)(1)],
or that defects or failures to comply associated with substantial safety hazards would in all cases be reported within five working days to a person who qualifies as a director or responsible officer as defined in 121.3 (121.21(a)(3)]. The weaknesses were discussed with the cognizant vendor staff
'during the inspection and summarized at the exit meetings.
In accordance with the NRC Enforcement Policy as promulgated in NUREG-1600, this was considered a minor violation and hence no Notice of Violation was issued. This violation notwithstanding, however, the inspector did not identify any instances in which 10 CFR Part 21 was not otherwise complied with.
The inspector further concluded that the NSA practices regarding handling of software error reports met the requirements of $21.21 whether the error is reported to users immediately or is determined to be (at least potentially) a deviation or failure to comply requiring evaluation per s21.21(a)(1).
]
1 3.2 The LOCA Eneineerine Review Council a.
Scope of LERC Review The inspector, reviewed the function and operation of the LOCA Engineering Review Council (LERC), and the consistency in modelling among analysts to j
assure that model changes are applied uniformly. The function of the LERC was examined through review of the minutes of LERC meetings and the resolution of i
issues brought before the LERC with followup interviews of cognizant staff.
The LOCA code guidelines review was to cover the manner in which the codes are used and applied to insure that a consistent modelling procedure was used from plant-to-plant. The inspector reviewed the vendor-to-licensee reporting procedures for model changes, and resulting PCT changes, to ensure that ESBU procedures and practices supported licensees in meeting the 10 CFR 50.46 annual and 30-day reporting requirements.
b.
LERC Observations In July 1992, NSA formed the LERC to examine perceived errors and proposals for application of new methodologies, design techniques, and analysis procedures to assure that they are reasonable, appropriate and in conformance with NRC regulations. The LERC was made up of senior technical staff members with sufficient background and knowledge of the history of code development 8
120
and applications. The LERC was intended to review the technical aspects of proposed changes and improvements, while also considering licensing, customer relations, and regulatory concerns. The LERC was not intended to be a replacement for the quality assurance technical review requirement of 10 CFR Part 50, Appendix B.
After review of LERC meeting minutes file, the inspector examined the following selected issues in detail:
At the March 27, 1996, meeting, the LERC considered two issues.
The first was a concern regarding handling the integral fuel burnable absorber (IFBA) rod design. A large number of analytical calculations were being performed when IFBA rods were to be used.
Four possible solutions were discussed to either maintain the status quo, or to seek relief from the NRC from the results of certain previously submitted analyses. The methods of lessening the analytic load would require NRC approval. The decision was to continue with the i
current procedures, in spite of the large demand on analysts' time.
The second issue the LERC considered in the March 27, 1996, meeting was a concern regarding the enthalpy transfer methodology used between the SATAN and LOCTA computer codes. At the time of this LERC meeting (and also at the time of the inspection), ESBU used the node-centered method.
The recommendation being considered by the LERC was to use a more modern, so-called " donor-cell" enthalpy transfer methodology.
Coding the donor-cell method and using it with SATAN would result in a reduction in PCT of 50 to 200*F.
SATAN, as then configured, used only four core nodes. While use of the less restrictive enthalpy transfer would lower the PCT, a new core nodalization sensitivity with more core nodes would negate any PCT margin gains. Again, the_ status quo was decided to be maintained.
The donor-cell methodology that was coded for the study was installed in SATAN as an option, with the default being the node-centered enthalpy transfer method.
Related to the concern over the enthalpy transfer was the documentation provided to the NRC.
The pertinent ESBU WCAPs (Westinghouse document designation) refer to the node-centered methodology, while some unofficial presentation materials discuss de sr-cell methods. The conclusion was that the information of record was consistant with the methodology in standard use.
The decision to maintain the status quo with regard to fuel rod analysis procedures does not raise a safety concern and is at the discretion of the vendor. The enthalpy transfer methodology and core nodalization used in the current analyses are consistent with those reviewed and approved by the NRC.
Use of the donor-cell methodology would require review and approval by the NRC. At that time, core nodalization would be reconsidered.
During the August 17, 1996, meeting, the LERC considered the way in which the LOCBART code models the fuel assembly grids; specifically, whether or not the code was correctly solving the heat conduction / mass balance for conditions of insufficient liquid droplets. A new solution technique for the model was 1
prepared, and one computer run indicated a change in PCT of +20*F for a plant in which PCT occurs late in the LOCA sequence.
In that case there was more than 200*F margin to the 2,200*F limit of 10 CFR 50.46. The LERC concluded that.this constituted a model improvement rather than a code error since it reflected a newer approach to problem solution rather than incorrect modelling. The LERC recommended that implementation of this solution 9
'121
technique be discretionary until further model changes, such as axial offset, were implemented as a code update.
The inspector determined that the LERC disposition of this issue was reasonable and justifiable in the case of implementation of solution update techniques.
It was shown during the inspector's discussions with the cognizant ESBU staff that implementation of the updated model would not always result in an increase in PCT.
In the case of early PCT plants, a reduction in PCT could occur.
,, Lhe case of some of the plants in which PCT occurs late in the LOCA sequence, an increase in PCT of as much as 20*F could occur.
No cases were found that violated the limits of 10 CFR 50.46.
In the May 26, 1995, meeting, the LERC considered questions regarding the issue of safety injection versus - m in the broken loop for 4-loop plants.
The concern was that the downcomer delivery was dependent on the break
- location, i.e., bottom, top, or side of the pipe. The discussion resulted in a 4-to-3 LERC vote split and a dissenting opinion.
The opinions of the LERC j
minority were appropriately addressed, and resulted in a revision of the accepted analysis process.
Ultimately, analyses were performed with injection locations at the top, side, bottom, and 45-degree angle.
The bottom location was found to be the limiting case.
The dissenting opinion was withdrawn.
The analysis approach was submitted to the NRC for review and approval as part of the submittal dealing with condensation effects credit during safety injection (the so-called "COSI" submittal). The pertinent WCAPs are:
WCAP-ll767, "COSI SI/ Steam Condensation Experiment Analysis," March 1988.
WCAP-10054-P, Addendum 2, Revision 1, " Addendum to the Westinghouse Small Break ECCS Evaluation Model Using the NOTRUMP Code: Safety Injection into the Broken Loop and COSI Condensation Model," October i
1995. Use of the methodology was reviewed and approved by the NRC.
c.
Conclusions Regarding the LERC The inspector concluded that, for the activities reviewed, the LERC functioned well as a sounding board for issues, concerns and questions regarding code models and methods of application.
In the limited number of issues reviewed by the inspector, concerns were described, discussed, and a satisfactory solution or resolution adopted.
3.3 LOCA Code Use Guidelines Use of the current generation of Westinghouse LOCA codes is defined by the company's cude guidelines.
The analyst uses a preprocessor (SPADES) to access a plant database (IMP) from which a steady state input deck emerges specific to the applicable code, such as NOTRUMP, LOCTA, SATAN, plus a template deck.
Very limited freedom is accorded the analyst in modifying the input stream.
Specific data sets, such as a pumped flow table, can be altered to permit use 10 122
of new data, or to examine possible effects of plant changes.
Changes made in f
the plant input model are reviewed by an independent analyst as part of the quality assurance program. The code guidelines contain notification of code errors and work-arounds until final approved corrections have been made and incorporated.
The inspector determined that the method of ensuring continuity and consistency in code use was reasonable and acceptable.
3.4 10 CFR 50.46 Resortine a.
Scope The inspectors reviewed ESBU loss-of-coolant accident (LOCA) analysis computer code procedures.
Licensee reporting requirements for LOCA analyses must be in accordance with 10 CFR 50.46, and 10 CFR Part 50, Appendix K.
Specifically, changes in an analysis, whether due to code model changes, error corrections, or plant input changes, resulting in a total change in calculated peak cladding temperature (PCT) of 50*F (using the absolute value of the changes) or more must be reported to the NRC within thirty (30) days.
Changes resulting in less than a 50*F total calculated PCT change are to be reported
- annually, b.
Observations 4
ESBU requires the assigned analysts for the various plants for which ESBu performs analyses to keep a running tally of changes in PCT, for example, to know the margin to 2200*F PCT, to know when the absolute value of PCT changes accumulates to >50*F for 30-day reports pursuant to 10 CFR 50.46, and to compile all changes for the 50.46 annual reports.
Since there are approximately forty (40) analysts working on the different Westinghouse NSSS equipped plants, the inspector reviewed the manner in which 1
. changes involving PCT are controlled and applied.
The inspector determined that NSA consistently prepared quality assurance documents and PCT summary sheets for each plant.
Changes made in analyses affecting more than one plant resulted in QA PCT tabulations listing all affected plants and their respective results.
Then NSA prepares reports summarizing PCT, change in PCT, and final PCT for both large-break and small-break LOCAs for each individual customer.
In addition, NSA reports all changes immediately to any customer who uses Westinghouse analysis codes for licensing-basis analyses.
The inspector determined that ESBU transmits code changes resulting in PCT changes of tens of degrees Fahrenheit to customers promptly in case the customer has made changes ESBU is not aware of that would result in a total change in PCT of 50*F or more; in which case, the 30-day reporting requirement would be imposed. Also, NSA reviews the database used to track PCT margin for all affected plants (called the " IMP" database) prior to each use to be sure plant changes have been properly applied and that there are no errors in the database.
Finally, NSA prepares a closeout record for resultant PCT changes providing root cause for the changes, potential nonconformance, corrective action, and conclusions.
11 4
123
c.
Conclusions The procedures used to insure timely reporting of PCT changes resulting from analyses of small and large break LOCAs are sufficient to permit compliance with the reporting requirements of 10 CFR 50.46.
Procedures have been effectively implemented to ensure accuracy and quality assurance in the reporting.
PARTIAL LIST OF PERSONS CONTACTED Jerry Malley, Engineer, AID, SMPD Larry Walker, Manager Aux. Equip Engr., SMPD Larry Kamenicky, Manager, ESBU Quality Systems-I R.0. Kechner, Manager, ESBU Quality Systems-II Rocco A. Asselta, EMD Principal Engr (QA)
Gene R. Strussion, Sr. Engr., NSD, RCS Norm Mueller, SMPD, Mgr. I&C Tech & Analysis Richard B. Miller, NSD, RLI, Fellow Engr.
Steve Tritch, NSD General Mgr.
Philip T. McManus, Process Control Division, T.Q. Engr.
David N. Alsing, ESBU Quality Systems III Curt F. Ciocca, Sr. Engr., SMPD, NSA Sumit Ray, CNFD-CE John S. Galembush, NSD, REN, Senior Engr.
Al Casadei, CNFD-CE Meena Mutyala, Director, ESBU Quality Systems David K. Allison, CNFD, Dev. Prog.
Aldo R. Govi, ESBU/QS, Supervising Engineer Nick Liparulo, Manager, REN, NSD H.A. Sepp, Manager, RLI, REN, NSD William McElroy, Qual & Reliability Engr., PCD Frank Rizzi, QA Eng., RCS, NSD Tom Cornale, Manager of Quality & Safety Engineering, EMD Tim Dunn, Principle Engineer, RCP Engineering Dan Garner, NSD, NA Mark Kachmar, NSD, NA Steve Rupprecht, NSD, NA B.J. Metro, Sr. Engineer, SMPD/SE/ICAT R.J. Sero, General Manager, SMPD ITEMS OPENED, CLOSED, AND DISCUSSED Discussed:
99901307/96-01-01 URI Questionable Justification for Closecut of PCD CARS92-064 and 93-002 12 124
APPENDIX A ESBU Level II QA Procedures of Interest Applicable to NSD, SMPD, ABTA-Implementing Criterion XV:
i ESBU-13.1, " Field Deviation Report (FDR)," Revision 1, August 31, 1996 i
WP-13.2, " Control of Nonconformances," Revision 0, August 31, 1996 WP-13.3. " Deviation Notices," Revision 0, August 31, 1996 Implementing Criterion XVI:
1 ESBU-14.1, "ESBU-Significant Quality Issues," Revision 3, September 20, 1996 ESBU-14.2, " Corrective and Preventive Action," Revision 1, September 20, 1996 1
PCD Level II QA Procedures of Interest:
Implementing Criterion XV:
DP 13-001, " Control of Nonconformances," Revision 0, April 1, 1996, 3
Duplicates, but does not refer to DP 13-003.
Does not refer to 13-002 for a DIN tag.
General: " Reference and follow ESBU-19.0."
In addition, Paragraph 2.d states: " Determine if the nonconformance is a condition adverse to safety per ESBU procedure entitled " Identification and Reporting of Conditions Adverse to Safety."
DP 13-002, " DIN Tags," Revision 0, April 1, 1996 DP 13-003, " Nonconforming Material Review and Disposition," Revision 0, April 1, 1996 DP 13-004, "Veviation Notices," Revision 0, April 1, 1996 2.a., refers deviation notice DN to ESBU SRC per ESBU-19.0 Implementing Criterion XVI:
DP 14-001, " Corrective Action Requests / Reports," under No. 4 Review condition (subject of CAR) directed the reviewer to determine if the issue represents a
" potential substantial safety hazard."
It then referred the reviewer to ESBU-19.0 if the CAR issue is " reportable" instead of referring the reviewer to ESBU-21.0 if the CAR issue could be a condition adverse to safety.
The special PCD QA procedure for implementing Part 21 and Criteria XV and XVI for the type of software (mostly for EPROM) applicable to the type of equipment produced by PCD was DP 04-105, " Software Error Reporting and Resolution." The inspector reviewed the current revision of DP 04-105, Revision 0, April 1, 1996.
1 The PCD procedure that covers how errors and deficiencies are corrected and initiates engineering review and design drawing changes that prompt referral 13 125
__m.___
to.ESBU-21.0 when required is DP 04-004, "DE0N."
The inspector reviewed the current revision of the DEON (Development Engineering Order Notification) procedure, Revision 0, dated April 1, 1996.
EMD Level II QA Procedures of Interest:
Implementing Criterion XV:
IDP-Ql,' Revision 4, March 15, 1996, did'not refer to ESBU-21.0 nor did it provide for determining if any of identified nonconformances ever hav'e been or could be conditions adverse to safety. Although, among the records reviewed at EMD, the inspector found none that should have been treated as conditions adverse to safety.
IDP-Q1 also does not refer to PAI-403 or IDP-Q17 i
Implementing Criterion XVI:
PAI-403, "Significant Quality Problems," Revision 6, March 31,1994.
Items are to be processed per IDP-Q2.
IDP-Q2, revision dated February 2, 1990, out of date, says process per OPR-19.0 IDP-Q17, Revision 0, August 6, 1996, SQP Log NSD Level III QA/Part 21 Procedures of Interest:
The Engineering Technology Department was formerly a part of NSD.
Engineering Technology Instruction Manuals ET-A-1.0, " Glossary" (Revision 1, October 1, 1993), and ET-B-1.0, " Identification, Evaluation and Closeout of Safety Concerns" (Revision 1, October 1, 1993), discuss deviations, but not-failures to comply. The organization (s) to which these two procedures were meant to be applicable have long since changed, and other procedures now perform their functions, but they remained effective at the time of the inspection.
The Level III, departmental procedure in use by the Replacement Component Services Department of NSD for QA/ Criterion XV implementation at-the time of the inspection was RCS-415, " Control of Material Deficiency Reports," Revision 3, June 30, 1994.
RCS-415 did not provide for determination of whether the deficiency described in the material deficiency report (MDR) has ever, or could ever have, constituted a condition adverse to safety, as defined in ESBU-21.0 in a BC delivered to NRC-licensed facility.
RCS-415 used the term
" substantial safety hazard" regarding reporting, but this term effectively excludes those deviations from evaluation which are potential defects by virtue of potentially causing the exceeding of a technical specification safety limit.
RCS uses the ESBU Level II QA procedures applicable to NSD for implementation of Criterion XVI (i.e., ESBU 14.1 and 14.2) 14 126
SMPD Level III QA/Part 21 Procedures of Interest:
NSA's procedure WP-4.19.3, Revision 0, August 31, 1996, formerly DP-3.7.4 (Revision 3, January, 31, 1993, reviewed) and then WP-3.7.4 (Revision 5, August 14, 1995, reviewed), " Software Error Reporting and Resolution." This procedure did not allow error report closeout to be schedulsi more that six months hence, but did allow extensions. Without close management attention, this has the potential for allowing excessive amounts of tira for discovery.
In cases in which the error could have a significant negative impact on PCT, for example, that is, a large PCT penalty relative to exis'.ing licensing-basis analyses, it could delay the timely opening of a PI and commencement of the 521.21(a)(1) evaluation.
4 J
t I
4
}
1 4
15 J
127
Erco y
-t UNITED STATES E
NUCLEAR REGULATORY COMMISSION E
f W ASHINGTON, D.C. 2%%c001
\\**.,+/
October 16, 1996 Mr. Howard E. Houserman General Manager Zetec, Incorporated 1370 NW Mall Street Post Office Box 140 Issaquah, WA 98027-0140
SUBJECT:
NRC INSPECTION REPORT 99901037/96-01
Dear Mr. Houserman:
On July 18, 1996, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at Zetec, Incorporated (Zetec).
The enclosed report presents the results of that inspection.
The NRC inspection team evaluated the measures that you have established for controlling your Company's Eddynet* computer software, which is widely used for nondestructive examination, acquisition, and analysis of nuclear power plant steam generator tubing, and concluded overall, that Zetec has a satisfactorily controlled software development and software validation program. The team observed strengths in Zetec's control associated with its Eddynet software, such as employee involvement, employee communication, proactive solicitation and disposition of potential issues from your software customers, and an overall sense of ownership and pride noted in your employees' demeanor.
However, some weaknesses were identified, as discussed herein, regarding Zetec's control of its Eddynet software relative to commercial industry acceptable guidance. Additionally, the inspectors determined that the implementation of one aspect of your quality assurance program failed to meet certain NRC requirements contractually imposed on you by your customers.
Specifically, one of your activities relative to the quality of safety-related activities was not delineated in written instructions or procedures as required by 10 CFR Part 50, Appendix B.
This nonconformance is cited in the enclosed Notice of Nonconformance (NON), and the circumstances surrounding it are described in detail in the enclosed report.
You are requested to respond to the nonconformance and to follow the instructions specified in the enclosed NON when preparing your response.
Eddynet is a registered trademark of Zetec, Inc.
128
p._m_j
,_A+
s.4 h
+ -* -
H. E. Houserman.
In accordance with 10 CFR 2.790 of the NRC's " Rules of Practice," a copy of this letter and its enclosures will be placed in the NRC's Public Document Room (PDR).
Sincerely, originet signed by Robert M. Gallo, Chief Special Inspection Branch Division of Inspection and Support Programs Office of Nuclear Reactor Regulation Docket No. 99901037
Enclosures:
Inspection Report 99901037/96-01 cc:
Mr. Stephen H. von Fuchs Quality Assurance Manager Zetec, Incorporated Post Office Box 140 Issaquah, Washington 98027-0140 1
129
-. ~
~
NOTICE OF NONCONFORMANCE Zetec, Incorporated Docket No.: 99901037 Issaquah, Washington Based on the results of an inspection conducted July 15 through 18, 1996, it appears that certain of your activities were not conducted in accordance with the requirements of the U. S. Nuclear Regulatory Commission.
Criterion V, " Instructions, Procedures, and Drawings," of Appendix B to 10 CFR Part 50 requires that activities affecting quality be prescribed by documented instructions, procedures, or drawings of a type appropriate to the circumstances and shall include appropriate quantitative or qualitative acceptance criteria for determining that important activities have been satisfactorily accomplished.
Section 2, " Quality Assurance Program," of Zetec's Quality Assurance Program Manual, Revision 14, January 1995, states in part, that "every procedure and all revisions are reviewed and approved by Quality Assurance... This review ensures that the standards of acceptability for all features and requirements i
are clear...."
Contrary to the above, Zetec did not establish, review or. approve procedures to control and ensure repeatable results regarding the development of safety-related eddy current computer data screening techniques used by data analysts for their standard of acceptability.
(99901037/96-01-01)
Please provide a written statement or explanation to the U.S. Nuclear Regulatory Commission, ATTN:
Document' Control Desk, Washington, D.C. 20555-0001, with a copy to the Chief, Special Inspection Branch, Division of Inspection and Support Programs, Office of Nuclear Reactor Regulation, within 30 days of the date of the letter transmitting this Notice of Nonconformance.
This reply should be clearly marked as a " Reply to a Notice of Nonconformance" and should include the following for each nonconformance: (1) a description of steps that have been or will be taken to correct these items: (2) a description of steps that have been or will be taken to prevent recurrence of these items; and (3) the dates your corrective actions and preventive measures were or will be completed.
Dated at Rockville, Maryland, this 16th day of October, 1996 nclosure 1 130
i U.S. NUCLEAR REGULATORY COMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Report No:
99901037/96-01 Organization:
Zetec, Incorporated Issaquah, Washington Contat.t:
Stephen H. von Fuchs, Quality Assurance Manager (206) 392-5316 Nuclear Industry Manufacturer and supplier of steam generator tubing, Activity:
eddy current inspection instruments, probes, software, calibration standards, manipulators, training and services for the commercial nuclear power industry.
Dates:
July 15-18, 1996 Inspectors:
Joseph J. Petrosino, Team Leader John M. Gallagher, Electrical Engineer John K. Ganiere, Electrical Engineer Michael J. Morgan, Reactor Engineer Phillip Rush, Materials Engineer Approved by:
Gregory C. Cwalina, Chief Vendor Inspection Section Special Inspection Branch Division of Inspection and Support Programs 131
_ _. _ __ _. _. _ _ _ -. _. _ _ _ _ _. _.. _.. _. _ __._ _.m._
1 INSPECTION
SUMMARY
During this inspection, the NRC inspectors (team) reviewed the implementation of selected portions of the Zetec, Incorperated (Zetec) quality program, which was expressed in the Zetec Quality Progrr.m Manual (Z-QA, Revision 14, January 1995), and specified controls to encompsss the requirements contained in the criteria of 10 CFR Part 50, Appendix B, American National Standards Institute-American Society of Quality Control (ANSI /ASQC) Q9001 and International Organization for Standardization (IS0) Standard-9001. Through a review of NRC licensee's purchase orders (P0s) and discussions with Zetec staff, the team found that NRC licensees impose Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Federal Reaulations (10 CFR Part 50) and other unique nuclear requirements on Zetec for its nondestructive examination services (NDE), NDE i
analyst activities, EC test calibration standards, and other related components.
Conversely, Zetec's EddynetM computer software and associated testing probes were found classified by Zetec and procured-by NRC licensees and NDE testing customers as commercial grade items (CGI's), as defined in 621.3 of 10 CFR Part 21.
Even though Zetec considers its Eddynet software and associated probes as CGI's, Zetec's quality manual policy stated that its manual defines corporate quality policies and requirements and applied to all Zetec products and services.
The team found that although Zetec classified its Eddynet i
computer software program, probe design, manufacture, and supply as CGI's, it used some elements of its Z-QA quality program for control of these components.
Therefore, in addition to the below inspection bases, the team also considered industry acceptable guidance when it evaluated Zetec's program controls such as, the Institute of Electrical and Electronics Engineers, Incorporated (IEEE)
Standard 7-4.3.2-1993, "IEEE Standard Criteria for Digital Computers in Safety Systems of Nuclear Power Generating Stations," as endorsed by NRC Regulatory Guide 1.152, " Criteria for Digital Computers in Safety Systems of Nuclear Power Plants," and Electric Power Research Institute (EPRI) NP-6201, "PWR Steam Generator Examination Guidelines." The inspection bases were:
Appendix B, " Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants," to Part 50 of Title 10 of the Code of Fedttal Reaulations (10 CFR Part 50) 10 CFR Part 21, " Reporting of Defects and Noncompliance" Overall, the NRC inspection team found that the measures for your Eddynet software development and control were generally sound, with the exception of a few weaknesses that are discussed herein. The team observed strengths in Zetec's program, such as employee involvement, employee communication, and solicitation and disposition of potential issues from your software customers.
EMynet is a registered trohmark of Zetec, Inc.
2 132
_. _ _ _ _ _....--- _ -- ~..-_. _ -
.~..__.......m
._._._..._m.
. -. - -.~
e..,
_m The team also observed that your employees demonstrated an overall sense of ownership and pride in their jobs. The team also believed that the strengths noted contributed to the soundness of Zetec's program and the adequacy of the eddy current-(EC) software.
During this inspection, one instance where Zetec, Incorporated failed to conform to NRC requirements imposed upon them by NRC licensees was identified.
This nonconformance is discussed in Section 3.7 of this report.
2 STATUS OF PREVIOUS INSPECTION FINDINGS i
2.1 Violation 99901037/85-01-01 (CLOSED)
NRC Inspection Report 99901037/85-01 stated that, contrary to Section 21.6 of 10 CFR Part 21, copies of Section 206 of the Energy Reorganization Act and Procedure ZAG-16 were not posted at the Zetec facility.
The team looked at Zetec's current posting and reviewed its content regarding the regulations.
The team noted that the posted documents addressed the requirements of 10 CFR Part 21.
2.2 Violation 99901037/85-01-02 (CLOSED)
NRC Inspection Report 99901037/85-01 stated that, contrary to Section 21.21 of 10 CFR Part 21, Zetec did not communicate the requirements of 10 CFR Part 21 to Westinghouse Specialty Netals Division, which supplied Inconel tubing to Zetec in August 1982.
This issue concerned a single piece of Inconel tubing which had been fabricated into a calibration standard by Zetec for one licensee's steam generator tubing EC inspection. This matter was adequately resolved and corrective action was taken in 1985 when the finding was identified.
2.3 Nonconformances 99901037/85-01-03 throuah 85-01-13 (CLOSED)
Nonconformances 85-01-03 through 85-01-13 are considered closed because adequate corrective action was taken and the applicable licensees were made aware of the issues when the inspection report was issued. Additionally, in a letter dated June 9, 1986, regarding Inspection Report 99901037/85-01, the NRC staff stated that it had reviewed Zetec's replies and found them responsive to the concerns raised in the 1985 notice of nonconformance.
3 INSPECTION FINDINGS AND OTHER CONNENTS 3.1 10 CFR Part 21 Procram The team reviewed Procedure ZAG-16, " Reporting of Safety Hazards," which Zetec adopted pursuant to 10 CFR Part 21. The team noted that although the procedure generally met the intent of eart 21, it iacked some clarity and specificity in certain areas.
For example, the procedure was not very clear about the vendor's responsibilities under 921.21(b) of Part 21 and it did not define some key words, such as evaluation and discovery.
The team discussed 3
133
potential changes to ZAG-16 with the QA Manager, which would appropriately clarify Zetec's procedure. The QA Manager committed to reviewing and modifying the procedure as necessary.within approximately 90 days from the receipt of this report. No other concerns were noted in this area.
3.2 Ouality Assurance Proaram The inspection team selectively reviewed Zetec's Quality Manual Z-QA, Revision 14, dated Januar) 1995, and found that it addressed 10 CFR Part 50, Appendix B, ISO 9001, and ANSI /ASQC Q9001. Z-QA was divided into separate sections which correlated to the different requirements and regulations. The team noted that document Z-QA stated that: "The scope of the quality assurance program plan includes the total operation of Zetec, Incorporated. The measures are to be implemented to the extent necessary to assure that services, equipment and other items supplied to or performed for customers by Zetec shall conform to the specified quality levels of applicable codes, 1
standards, regulatory criteria and purchase order specifications...."
Further, the team noted that the cover page of Zetec's Quality Manual stated, "This manual defines Corporate Quality Policies and Requirements and applies to all Zetec products and services."
The team reviewed a sample of the licensee's purchase orders (P0s) and verified that Appendix B of 10 CFR Part 50 and other unique nuclear requirements had been imposed on Zetec for its nondestructive examination 2
(NDE) services, EC test calibration standards, and related components.
The team also found that Zetec's probe products and Eddynet software were typically procured by NRC licensees and NDE testing customers as commercially available products; consequently, Zetec's quality program would encompass Zetec's control of Eddynet and associated probes under its ISO 9001 aspects.
From its review of Quality Manual Z-QA and observations of Zetec's operation, i
the team concluded that the implementation of Zetec's' program and i
i establishment of its manual appeared to be adequate.
3.3 Eddynet* Software Development. Validation and Confiauration Control J
The team evaluated Zetec's software development validation and configuration control process to determine whether Zetec was achieving and maintaining a i'
consistent level of quality in its Eddynet software products. The Eddynet software is used in the NDE eddy current inspection of safety-related steam generator tubing and plays a critical role in the generation of the eddy current inspection data.
Zetec has a single manager, the Software Development Supervisor, in charge of the entire software development program. This person. is responsible for all aspects'of the development and configuration control of the Eddynet software.
The Software Development Group consists of six programmers. Three of the programmers have had extensive field experience in performing EC acquisition and analysis, and four of the six are considered to be experienced computer language C programmers. An associated group, the Software Support Group, also reports to the Software Development Supervisor and is responsible for
' developing the software validation checklists.
4 134
The Eddynet 95 software product is written in computer language C for application in either of two different Hewlett Packard workstations: Series 300/400 (HP-UX Version 9.03 operating system); or Series 700 (HP-UX Version 9.05 operating system).
Eddynet 95 comprises several independent programs, such as acquisition programs, analysis programs, and inspection management programs. The team noted that the Eddynet series and associated firmware are offered as commercial products. Although Zecec is registered as an ISO 9001 supplier, the team was told by Zetec staff that its software development does not conform to the guidelines of ISO 9000-3, " Guidelines for the Application of ISO 9001 For the Development, Supply and Maintenance of Software."
3.3.1 Software Requirement Specification The Software Development Supervisor told the team that Zetec had not established any formal software requirement specification for the performance expectations of the Eddynet software.
The original Eddynet software was written in an effort to develop a digital acquisition system and consisted of relatively straightforward algorithms. As a result, the original development process did not follow a formal software development life-cycle approach which would encompass a software requirement specification.
From its discussion with the Zetec software staff, the team understood that since it first developed its Eddynet software, Zetec has continued to enhance Eddynet software with strong emphasis on the man-machine interface characteristics of its software program. As a result, Zetec had focused much of its attention on ensuring that the software product is validated before release.
The team discussed with other NRC staff its concern that no documented software requirement specification existed. Upon further consideration of Zetec's utilization of its available resources for the development and validation of the software, the team concluded that the lack of a detailed specification was more a question of development efficiency than quality.
The team observed that the majority of software changes which had been incorporated into new versions of Eddynet 95 were related to man-machine interface enhancements.
These changes wa e derived from Zetec's Software Change Request (SCR) Program which is described in Section 3.5 of this report.
No other concerns were noted in this area.
3.3.2 Software Verification and Validation The team reviewed Zetec's software verification and validation process.
The team reviewed the methodology that Zctec nad established to validate the Eddynet software program after Zetec made changes to the program.
The following three Zetec work instructions were reviewed to evaluate their adequacy and appropriateness:
New Version Product Disk Validation Version Update Disk Validation Prototype Disk Validation 5
135
The team concluded that the major purpose of these work instructions was to ensure that new version releases, version update releases, and prototype releases of the Eddynet software product performed correctly. Additionally, the team found that these procedures appeared to adequately control the applicable portions of the Zetec program for which they were written and were appropriate to the circumstances.
The team found that the Software Support Group had developed a software checkout book that contained Eddynet validation checklists used by the Zetec group responsible for the validation process. The software checkou.t book consists of multiple checkout groups and subgroups which contain specific validation checklists for each of the programs within Eddynet.
The Software Support Group develops and maintains these validation checklists. A version checkout coordinator from the Software Support Group was designated as being The responsible for the overall coordination of the validation process.
validation is performed by employees within Zetec's Product Support Group and, to a limited extent, by the Software Support Group. Zetec indicated that its validator personnel had extensive field experience with Eddynet.
The team concluded that there was sufficient independence between the software developers, the Software Support Group who develop the validation checklists, and the Zetec group that performs the validation process.
The software group stated that the intent of the validation process was to exercise all possible program functions and, therefore, verify the functionality of each of the programs in the software. The team found that functionality problems encountered by Zetec during its validation process were recorded on version validation sheets.
The problems were then compiled by the version checkout coordinator and were reviewed and handled by the Software Development Supervisor. The team observed that after the appropriate changes had been made, Zetec subjected its software to a second complete validation process checkout. That is, Zetec repeats all of the actions that were done for the first validation process.
On the basis of conversations with the Software Support Group, the team concluded that the checkout process is continually revised to account for new features added to the Eddynet program, and the subgroup checklists are updated to ensure that all new features of the software are exercised.
In addition, based on procedural instructions and management instructions, it is the understanding of the individual validators that they are responsible to exercise all functions within the program, even if each function is not specified in the subgroup checklist.
Altho' ugh the team found that the current validators are adequately performing their job functions, Zetec had not established documented controls or release signature requirements for ensuring that the validation checkout subgroup lists are fully updated to include functional tests for all new features or changes associated with a new version or v?rsion update before sending the checkout subgroups to the validation team. As a result, there is a potential that new features of the software may not be functionally tested before they are delivered to customers for safety-related applications.
The inspectors classified this as a weakness in the validation process.
6 136
The team noted that, in addition to the validation of the functional performance of the program, the validation process also served as feedback from the validators about the acceptability of new features. The program i
descriptions associated with software changes are not considered complete until the validation has been finished.
This allows the validation team to evaluate and assets the man-machine interface characteristics and functionality associated with each software change.
Overall, Zetec's method of performing validation appeared to be effective because of the added strength which comes from using personnel who have Eddynet field experience.
Th0 experience aspect also compensates somewhat for Zetec's lack of a documented software requirement specification, as discussed in Section 3.3.1 of this report. No other concerns were noted in this area.
3.3.3 Review of Eddynet* 95, Version 2.0, Validation The team reviewed the checkout group and subgroups completed for the validation of Eddynet 95, Version 2.0, HP-UX Version 9.03 (300/400 Series).
The team observed that all items within the subgroups appeared to be checked.
The team also noted that the employees performing the validation often changed terminology in the subgroup checklists and added comments within the margins of the checklists.
In addition, checklist instructions were modified if the information in the instruction was different from the information regarding the program functionality.
For instance, the team found one checklist instruction that was changed because it incorrectly referred to a " button" rather than to a " pull-down menu."
In most cases, these changes or comments were not recorded on the validation sheets since they did not affect the functionality of the software.
Further review of this area revealed that Zetec had not established any documented controls for ensuring that the written comments or changes made in the validation checklists are reviewed and handled by either a version checkout coordinator or by the Software Development Supervisor. Therefore, during discussions with Zetec staff, the team also identified this as a weakness because of the potentially inconsistent results that could occur during the validation process.
The team also reviewed the master version validation sheet'and verified that all problems encountered during the validation were reviewed and handled by the Software Development Supervisor.
In general, the team concluded that the validation of Eddynet 95, Version 2.0, was performed in accordance with the work instruction "New Version Product Disk Validation," and appeared to be effectively controlled.
No other concerns were identified in this area.
3.3.4 Review of Eddynet* 95, Version 3.0, Validation The team witnessed Zetec's validation activities associated with the inspection planning program of Eddynet 95, Version 3.0.
A data manager / data analyst in the Product Support Group w's performing the validation in accordance with the appropriate subgroup checklist.
The data analyst confirmed to the team that the intent of the validation was to exercise all program functions to ensure that the software performs as designed. To achieve this, validation personnel executed the steps written in the subgroup 7
137
checklist. The team found through its discussion and review that the validators generally exercise more functions than are prescribed in the checkout list in an effort to. verify the functionality of all aspects of the program and to check for the potential impact of incorrect key manipulations.-
The data analyst told the team that all new functions are not always incorporated into the checkout list they receive. As. a result, the analysts may add supplemental checks to test new functions at their discretion. An analyst who encounters a new function that is not on the checklist may contact the Software Support Group for an explanation of the new function.. As described in Section 3.3.2 of this report, the team concluded-that Zetec appeared to lack adequate controls to ensure that all new features of the software were incorporated into the validation subgroup checklist; however, it was viewed as a strength that the analysts may add supplemental checks to test new functions, j
The team determined that new versions of.the Eddynet 95 software are validated on both Series 300/400 and Series 700 Hewlett Packard workstations.
In witnessing the validation of Version 3.0, the team noted that the validation on the workstations was performed essentially in parallel by separate validators. The team believed that this parallel validation process is a i
valuable asset to Zetec's process control and allowed for crosschecks and i
real-time on-line discussions between validators regarding problems associated with the software or with the validation subgroup checklists.
This could also be considered as a diverse implementation of the software programs and, as such, an additional means for finding software errors.
3.3.5 Impact of Source Code Change The data analyst and Software Development Group indicated that changes to the source code that do not affect the high-level functionality of the program or cause a visible impact on the operator's workstation are transparent to the validator. As such, the team concluded that the independent validation process is not structured to assure that software errors which are transparent to the user are detected.
Specifically, there is no formalized independent check of the source code that allows for detection of these types of programming errors.
This practice is not in accordance with acceptable nuclear and software
' development community-accepted quality assurance (QA) procedural guidance and practices that govern the development and validation of software, and would be a considered as a noncompliance if Appendix B to 10 CFR Part 50 were applicable.
The inspectors concluded that Zetec lacked criteria to allow for the determination of source code changes that require an independent check, and a process for performing the. independent check of the source code.
3.3.6 Eddynet* Software Configuration Control The team reviewed Zetec's processes for controlling and reporting cn e p s to software, including a review of procedures for revising Eddynet software programs.
Zetec indicated that software changes were controlled by a software change notice (SCN) system.
Each software change was documented in an SCN 8
l 138 j
form that described the changes, the proposed activities, and the personnel responsibilities for changing the software. The team reviewed QA Procedure QAP-8, " Control of Production Software," Revision 11. This procedure controlled the revision, validation, and release of production software such i
as Eddynet 95, and gave specific details about SCN processing.
The team also reviewed Zetec's SCN log and associated SCNs.
The team concluded that the software revisions appeared to be processed in accordance j
with QAP-8.
Each SCN was uniquely identified and could be traced to a specific Eddynet version. The team also reviewed a sample population of software revisions and noted they were validated before they were released.
Zetec staff stated to the team that the software changes associated with a new version of Eddynet are typically described in a Zetec " announcement bulletin" that is sent to customers when new versions are released.
The team reviewed the release announcement for Eddynet 95, Version 2.0, and confirmed that the information in it pertained to significant software changes or enhancements.
The team concluded that Zetec maintains a good program for tracking and controlling software changes.and for notifying customers of the changes and enhancements associated with new versions of Eddynet.
3.3.7. Development, Validation, and Configuration Control As described above, the team reviewed Zetec's processes for software development, software validation, and software configuration control.
The team concluded overall that Zetec has an effectively implemented and controlled software development and software validation program; however, some weaknesses were apparent.
The team discussed the following two observations in detail with Zetec's staff:
The team noted that high-quality man-machine interface characteristics with respect to effective use of software products in field environment is a principal attribute of the Eddynet software. The development and validation processes support this attribute because they are performed by Software Development and Product Support personnel who have extensive field experience with Eddynet.
However, the team also noted that field experience requirements for programmers and validation personnel-are not controlled by QA procedures.
The team concluded that there are no formal software requirement specifications for software changes.
Instead, there is a heavy reliance on real-time on-line discussions in the performance of both software development and validation job tasks to maintain a high level of agreement during the development and validation activities.
The team believed that this can result in undocumented bases for important decisions regarding the final software product.
3.4 Identifyina and Reportina Software 0.eficiencies The team asked Zetec about how it addressed program errors in software that could_ potentially affect the quality of EC data being manipulated or analyzed.
Zetec indicated that significant software deficiencies attract immediate 9
139
attention as they are processed in accordance with QA Procedure QAP-9,
" Software Change Request," Revision 0, and are reported to customers.
Zetec provided three examples of how it notified customers about software problems or limitations.
The team concluded that Zetec appeared to have addressed the problems promptly and efficiently. Although 10 CFR Part 21 is not applicable 4
to Zetec's Eddynet software, the Zetec process appeared to meet the intent of
$21.21(b) of 10 CFR Part 21.
3.5 Software Chanae Reauests The team reviewed QA Procedure QAP-9, " Software Change Request." This procedure defines the process for documenting, reviewing, and implementing SCRs.
Zetec developed the SCR form for the use of customers and Zetec field personnel to report problems associated with Eddynet or to make recommendations for software enhancements.
The Software Development Supervisor reviews all SCRs and is responsible for deciding if an SCR will be approved (that is, if action will be taken to address the customer's request) and for deciding the schedule for implementing the change or enhancement.
The Software Development Supervisor noted that this decision may require speaking with the originator of the request to get additional information or to modify the request, (to reduce the impact on other parts of the program).
Upon approving an SCR, the Software Development Supervisor assigns the task to a programmer. The Software Development Supervisor discusses the details of the change with the assigned programmer and, to implement the software change, relies on the programmer's experience and judgment.
Preceding the validation phase, the Software Development Supervisor reviews the new or modified code developed by the programmer to ensure that it meets its intended function.
The team noted that the majority of the programmers and validators appear to have strong field experience with Eddynet. Although there is no formal software requirements specification for software changes, formal discussions and reviews take place throughout the change process to ensure that software changes are handled appropriately.
In addition, the software changes are validated on two separate operating systems.
The team reviewed Zetec's process for handling SCRs by tracking several SCRs that had been initiated by Framatome Technologies, Inc.
Specifically, the team reviewed Zetec's SCR log to ensure that the SCRs were reviewed upon' receipt, assigned an SCR number, and appropriately dispositioned.
The team also confirmed that SCNs were implemented as appropriate and that the corresponding software changes were incorporated into new versions of Eddynet.
In general, the' team concluded that Zetec reviewed and addressed the SCRs promptly and in accordance with QAP-9.
Overall, the team concluded that the SCR process appears to be effective in identifying man-machine interface problems and customer requests for software enhancements.
Zetec receives a large number of SCRs.
The team confirmed that most of these SCRs request software enhancements rather than report software 10 140
-+m a ~-
w
l deficiencies. Despite the large numbers of SCRs it receives, Zetec appears to j
have a good program for tracking and addressing them promptly and appropriately.
3.6 Data Analysis Procedures and Acouisition Techniaues The team reviewed the vendor's EC data analysis, " Eddy Current Data Analysis i
Procedure," FSP 301-EVAL, Revision 7, July 7, 1994, and an acquisition procedure, "Eddynet Eddy Current Acquisition Procedure," FSP-301-EN, Revision 3, October 20, 1994.
The team evaluated Zetec's analysis procedures.and site specific procedures to determine to what extent industry data analysis practices discussed in such industry documents as EPRI NP-6201 had been i
incorporated into Zetec's program. The team found that Procedure FSP 301-EVAL addressed general personnel, documentation, equipment, and procedural requirements for the analysis of EC inspection data.
Procedure FSP 301-EN, Revision 3, addressed required actions needed to set up some of the hardware and software for acquiring EC inspection data.
The team found that the guidelines were clearly written.
In addition, the antiysis guidelines containec' mch strengths as specific requirements on lead t.nalyst responsibilities, ana r independence, and noise and rejectable data criteria.
By interviewing NJ echnicians, the team confirmed that Zetec analysts were familiar with the details of the procedure.
The team found that site-specific guidelines for examining steam generator tubing are typically developed and rerised through discussions between the utility and the NDE vendors involved in the steam generator tube inspections.
The team noted that a typical site-specific examination guideline did not contain all of the attributes found in FSP 301-EVAL.
Zetec indicated that the NDE technicians generally focused on revising the finer details of the proposed site-specific guidelines rather than on the broader aspects of the procedure.
The team concluded that Zetec's EC analysis and acquisition procedures contained sufficient detailed instructions for conducting inspection-related activities.
The team noted several strengths in the analysis guidelines; however, these strengths were not always being considered by NDE technicians for incorporation into site-specific examination guidelines.
3.7 Comouter-Assisted Data Analysis The team reviewed the vendor's controls for computer data screening (CDS) techniques used in the analysis of EC inspection data. The review focused on assessing whether Zetec had been adequately managing its activities involving the use of CDS.
The team found that Zetec is often contracted to supply EC data analysis using a CDS algorithm.
These computer algorithms use specific criteria which are inputted by a Zetec data analyst before a particular utility customer S/G tube EC inspectic, to identify expected EC indications.
The individual responsibility is significant for tne data analyst because they must develop these criteria based on expected modes of tube degradation for each different utility S/G tube EC examination.
However, the team noted that Zetec did not have a written or formal type of procedure for controlling or 11 141
establishing the specific process that is to be used by its data analysts for developing Sese criteria.
As a result, the potential exists that screening criteria developed by one analyst may be significantly different from that of another analyst. To establish the parameters for identifying indications in the EC data, a data analyst must independently develop the screening logic.
Before a Zetec analyst starts EC inspection at a particular nuclear power i
plant site, the CDS is typically subjected to a site-specific examination, i
which it must pass in order to be.used in an inspection.
The team's review of this activity identified that when developing CDS criteria, analysts will typically address known degradation modes.. However, because of the different experience levels of the analysts, a variance in the effectiveness of the developed criteria can occur. This can allow types of degradation previously undetected by a particular analyst to be missed by the CDS algorithm.
The team identified to the Zetec staff types of parameters, or ranges, that may need to be delineated within procedures or instructions to i
ensure compliance with the requirements, and to ensure that the widest spectrum of degradation modes will be identified. These aspects were not required by Zetec's program.
3 The. team also recognized that Zetec's data analysts have a broad knowledge of degradation mechanisms and may adapt well to any new degradation mechanisms that appear during inspettions. The team noted that although.Zetec analysts have not been working with written procedures or instructions regarding written guidance and instruction for the development of these criteria, Zetec was using only experienced analysts for developing its CDS criteria.
4 The team determined that the potential existed that the lack of such procedures could lead to differing performance capabilities for CDS techniques under actual inspection conditions and as a result, Zetec's CDS program could fail to identify potentially defective tubes. Therefore, although Section 2.5, " Procedures," of Zetec's Z-QA manual indicated that every procedure and
)
all revisions are reviewed and approved by quality assurance to ensure that j
standards of acceptability for all requirements and features are clear, no requirements for CDS criteria development were found.
The team identified the 1
lack of written guidance to govern the development of CDS criteria as Nonconformance 99901037/96-01-01.
3.8 Eddy Current Probe Fabrication The team observed the EC probe manufacture and interviewed assembly-line personnel involved with the fai>rication of the EC inspection probes. The team reviewed several of Zetec's,orocedures used to manufacture and inspect the EC probes.
For example, Zetec's " Fabrication Procedure for Motorized Rotating 3
~
Coil Probe Head," Revision 0, dated March 13, 1996, specified the sequence for constructing a certain type of EC probe, and the procedure clearly defined the i
steps and essential variables of the fabrication process and required periodic simplo checks to verify coil / probe perforrance.
The team also observed that the assembly-line personnel involved in coil construction were knowledgeable about the use of the various testers used to verify the performance of coils. The team reviewed an acceptance-related 12 142
i l
procedure, QAP-21, "ULC Check-Out Procedure," Revision 2, and noted that the i
procedure contained steps for visual inspections (including physical measurements of probe dimensions), a physical inspection of the overall integrity of the probe, and a functional inspection requirement to verify the i
electrical response of the probe.
The team observed that Zetec's inspection l
personnel used consistent reference standards for verifying probe performance and appeared knowledgeable in their areas of responsibility.
i 3.9 EC Personnel Trainina. Qualifications. and Resoonsibilities The team reviewed Zetec's Procedure QAP-101, " Personnel Qualification &
Certification," Revision 4, and discussed the responsibilities of Zetec's 1
Field Services (FC) Group with Zetec's Supervisor of Product Support and Technical Development.
The team assessed training documentation and reviewed 4
i certification examinations and discussed the specifics of Zetec's 1
certification and qualification program with Zetec's Senior Training Manager.
The team noted that Zetec EC personnel are qualified and certified to Zetec's written Procedure QAP-101 and that Procedure QAP-101 addressed the recommendations contained in ASNT SNT-TC-1A, "Fersonnel Qualification-Certification in Nondestructive Testing," 1984.
The team also reviewed various Level I, II, and III certification examinations and noted that the examinations appeared to be thorough and well-designed, and adequately test knowledge of basic theory and EC application.
The team noted that Zetec's certification and qualification records appeared to generally conform to the requirements of ASME Code Section XI (1989) and met the intent of Criterion IX, of 10 CFR Part 50, Appendix B.
1 The team found that Zetec's analyst training program experience requirements may vary because they are dependent upon the currently available job assignment and the collective amount of specific site-testing and OJT available for the recommended nine-month hands-on period. Therefore, Zetec's current practice could have the potential to limit job variety, exposure to various steam generator designs, and differences in testing practices.
The team also noted that Zetec's program lacked specificity relative to delineating types of experience that would be required for individual qualification.
During the team's review of this area, it also noted that Zetec's supplemental Level I steam generator tube inepection training requirements, that is, training performed after Level I certification for on-site, on-the-job-training (0JT), were typically f:,llowed by routine assignment to Zetec's Field Services Group.
Individual training records that were reviewed in this area showed that some candidates accumulated more than 1,000 hours0 days <br />0 hours <br />0 weeks <br />0 months <br /> of site-specific training, while other records only reflected 400-500 hours of similar training. Consequently, Zetec's program could allow a 500-600 hour disparity between different Level II NDE examiner training hours.
The team noted, in the records which it reviewed, that an individual's particular level of involvement with a specific team and unique job preparation, such as in-house mock-up work, EC equipment setups, and simulated data acquisition run-through, was typically the primary contributor to an individual's specific hours used for their experience and qualification.
13 143
Although a concern regarding analyst experience requirements was noted,.the team concluded that Zetec's current connitment to the ASNT recommendations and its current program meets the intent of ASNT-SNT-TC-1A by providing a methodology that appears to appropriately qualify and certify its eddy current testing personnel.
3.10 Internal Audits Internal auditing takes place within Zetec's Z-QA quality program.
Section XVIII, " Audits" of Z-QA, addresses the requirements contained in j
10 CFR Part 50 Appendix B and ISO 9001. This document calls for a formal in-a house auditing of the EC function, safety-related/ASME Sections III and XI Code activities, on at least a "once per each calendar year" basis.
The team i
verified that internal third-party assessments of Zetec's QA Group is performed by product services personnel at least once per calendar year.
On the basis of its review of documentation and discussions with Zetec's QA Manager, the team concluded that Zetec's internal QA audits were adequate and met both the intent and scope of requirements in Criterion XVIII, " Audits," of 10 CFR Part 50 Appendix B.
The team also reviewed Zetec's Product Service Group's internal audits of QA for 1994 and 1995. On the basis of its review of documentation and discussions with the Product Services Manager, the team concluded that Zetec's audits of QA were adequate and that these audits met both the intent and scope of Criterion XVIII of 10 CFR Part 50 Appendix B requirements.
3.12 Entrance and Exit Meetinas At the entrance meeting on July 15, 1996, the NRC team' leader discussed the scope of the inspection, outlined the areas to be inspected, and established contacts to be used during this inspection with Zetec management.
At the exit meeting on July 18, 1996, the team discussed its findings and concerns.
PERSONNEL CONTACTED Howard E. Houserman General Manager Jimmy Crittenden Product Support Dcug Handy Electronic Supervisor Laura Mcdonald Quality Assurance.
Mike 0'Laughlin-Validation-Data Manager / Data Analyst Bob Vollmer Product Support Manager Stephen H. von Fuchs Quality Assurance Manager Richard Warlick Software Development Supervisor (em Wilson Software Administrative Assistant Troy Woller Software Support Spect011st i
14 j
144
l l
i Selected Generic Correspondence on the Adequacy of Vendor Audits and the Quality of Vendor Products I
i Identifier Title L
Information Notice 96-62 Potential Failure of the Instantaneous Trip j
Function of General Electric RMS-9 Programmers Information Notice 96-68 Incorrect Effective Diaphragm Area Values in Vendor Manual Result in Potential Failure of Pneumatic Diaphragm Actuators Information Notice 96-70 Year 2000 Effect on Computer System Software l
145
Nr.C FORM 335 U.S. NUCLEAR REGULATORY COMMIS$10N
- 1. REPORT NUMBER N
- 1102, Norn En 320t na BIBLIOGRAPHIC DATA SHEET (See irastructoorn on the reverse)
- 2. TITLE AND SUBTITLE Vol. 20, No. 4 Licensee Contractor and Vendor Inspection Status Report 3.
DATE REPORT PUBLISHED Quarterly Report l
ve^a Cetober - December, 1996 March 1997 4 FIN OR GRANT NUMBER
- b. AUTHOR (S)
- 6. TYPE OF REPORT Quarterly
October - December 1996
- 8. PE R F ORMING ORGANIZ ATION - N AME AN D ADDR ESS (tr Nac. prov,de owes on Ortsce or nevion, u.s Nucles n.puurory comm,suon, and m im, address.,s contr cror.,,,,sde nane end one*Isne eddrent Division of Inspection and Support Programs Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Cortmission Washington, D.C.
20555-0001
- 9. SPONSORING ORGANIZATION - NAM E AND AODR ESS IIINnc, tvpe 'some a above";is contreecor. provide Nnc osession, ortwo or nosnon. us Nucker neguratory commisdun.
end mailms adorem)
Same as above.
- 10. SUPPLEMENTARY NOTES
- 11. ABS TR ACT (200.ords er km>
This periodical covers the results of inspections perfonned by the NRC's Special Inspection Branch, Vendor Inspection Section, that have been distributed to the inspected organizations during the period from October - December 1996.
- 12. KE Y WORDS/DESCR!PTOR S ftist nords orparews ener wilt assnt resestr4*rs a iczarms the esport j
- 13. AvAtLAsiLaiy Si AiEMENi Unlimited
- 14. SECUHli V CLAb51F ICAllO*e Vendor Inspection
<ra,s e.,e, Unclassified iThe Neportl Unclassified
- 15. NUMBER OF PAGES
- 16. PRICE w
Nn.; FORM 3361249)
1 I
t 1
I i
I i
i 4
i 4
l i
i i
i I
i Printed
- on recycled paper 1
l Federal Recycling Program
NU C
WL E
P A A EN S
A H R L
I R T
N U
YNG E N F
TGI O
OUT R
LE
T R A ENE 2 YT US 0 C E S S 5
S
,E 5O 5M
-0 30 0M 0
0IS 1
S I
O
~
N W2T AWP SFS HN.y N I-R GEhf N6 T7 AO O N f N UE0 R k EP l
GU B
L I
I C
A I T A I N O I D
N N C
S V I
S I 2
V A 0
C 1 5
S 1 5
8 5
m U
S S
T m
N N R
- o. C m
m