IR 05000275/1998201
| ML16342E186 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 07/16/1998 |
| From: | NRC (Affiliation Not Assigned) |
| To: | |
| Shared Package | |
| ML16342E185 | List: |
| References | |
| 50-275-98-201, 50-323-98-201, NUDOCS 9807270422 | |
| Download: ML16342E186 (48) | |
Text
U.S. NUCLEAR REGULATORYCOMMISSION OFFICE OF NUCLEAR REACTOR REGULATION Docket Nos.:
50-275, 50-323 License Nos.:
50-275/98201, 50-323/98201 Licensee:
Pacific Gas 8 Electric Company Facility:
Diablo Canyon Power Plant, Units 1 and 2 Location:
Avila Beach, California Dates:
April6-9, 1998 Inspectors:
Observers;.
/
Approved by:
Stephen D. Alexander, Reactor Engineer Team Leader, HQMB Gregory C. Cwaiina, Senior Operations Engineer, HQMB BillyH. Rogers, Reactor Engineer, HQMB Julio Crespo, Engineer, CSN Josef Zlathansky, Vice Chairman Nuclear Regulato'ry Authority of the Slovak Republic Jan Bednhi, Quality Assurance Inspector, NRASR Richard P. Correia, Section Chief Reliability.and Maintenance Section Quality Assurance, Vendor Inspection and Maintenance Branch Division of Reactor Controls and Human Factors Office of Nuclear Reactor Regulation Enclosure 9807270422 9807i6
.
PDR ADOCK 05000275 8=
,,
J I
lg (
EXECUTIVE SUMMARY Diablo Canyon Power Plant, Units 1 8 2 NRC Inspection Report 50-275/98201, 50-323/98201 During the period of April 6-9, 1998, representatives of the Quality Assurance, Vendor Inspection and Maintenance Branch, Division of Reactor Controls and Human Factors, NRR, conducted an inspection of Pacific Gas 8 Electric Company's (PG8E's) activities related to the procurement and dedication of commercial grade items and the technical and quality assurance aspects of the installation of an incorrect design voltage regulator circuit board in the 1-3 emergency diesel generator static exciter-voltage regulator in 1994 at the Diablo Canyon Power Plant (DCPP).
~En ineerin
~
Em r c
D'e e enera o a i E c er-Vol e Re ula o Circui Board Re la emen Review of the technical evaluation and quality assurance aspects of this 1994 circuit board replacement (for which Region IVhad cited the licensee for failure to perform a prompt operability determination after the licensee's discovery of the problem in 1996) revealed deficiencies in the original safety classification, procurement process, review for suitability of application and verification that all the board components met the design requirements.
The inspectors determined th'at the licensee had resolved the technical issues satisfactorily and independently verified that the correct configurations of the circuit boards in question were installed in all six emergency diesel generator exciter-regulator cabinets.
~
r cu m
n
'
The program for dedication and upgrading of commercial-grade, warehouse material not originally purchased with intent of dedication for safety-related service was generally technically sound and incorporated adequate quality assurance controls.
However, the inspectors identiTied deficiencies in implementation including:
(1) instances in which the documentation of the technical evaluations or acceptance process did not provide sufficient assurance that all critical characteristics were adequately verified, (2) instances in which it was unclear that replacement part evaluators recognized the difference between like-for-like (identical) replacements and equivalent replacements because they had erroneously deemed certain commercial-grade items as like-for-like replacements, when they should have performed equivalency evaluations, and (3) instances in which changes to replacement part evaluations were processed using the minor change provision in the replacement part evaluation procedure when the changes did not meet the requirements of that provision.
ali ce'n ri A self assessment of replacement part evaluation activities by the Diablo Canyon Independent Safety Engineering Group identified some of the same weaknesses identified by the inspectors.
The self assessment was thorough, insightful and candid.
The inspectors determined that planned corrective actions discussed with the self assessment team and the cognizant licensee staff should adequately address the concerns identifie aj h
E tt fig'
Re o Details En ineeri E1 Conduct of Engineering E1.,1 Emer enc D'
I enera or Exci er-e ula or Circui Bo d
Ins
'on Sco e
fn Aprilof 1996; the licensee identified an incorrect component configuration on the voltage regulator circuit board in the static exciter-voltage regulator cabinet for what PG8E calls No.1-3 Diesel Emergency Generator (DEG 1-3) and took corrective action, but did not promptly perform an operability determination.
Subsequently, NRC Region IVcited PG8 E for failure to perform a prompt operability determination for DEG 1-3 (Inspection Report 50-275,323/96-16, Violation 96-16-06).
Based on RIV's review of the quality assurance (QA) aspects of the procurement of the replacement voltage regulator circuit boards in 1977 and the use of one of them in DEG 1-3 in 1994, it appeared that the licensee had otherwise followed its procedures in effect at the time of the original procurement of the spare boards and of the DEG 1-3 board replacement.
However, the NRC determined that further examination of the circumstances surrounding the procurement of the spares and subsequent activities that might have detected or prevented further problems, including a more detailed review of the technical and QA aspects of this issue was warranted.
Accordingly, the inspectors reviewed current procurement procedures, those in effect at the time of the original procurement of the replacement circuit boards, and those in effect at the time of the replacement in question.
The inspectors reviewed technical information and procurement records for the original procurement of DEG voltage regulators and components and also the procurement records for the spare circuit boards, one of which was used in the 1996 replacement in the DEG 1-3 voltage regulator.
The inspectors reviewed documents associated with the 1994 installation of the incorrectly configured replacement circuit board for DEG 1-3, the 1996 installation of a correctly configured replacement board for DEG 1-3 and three others.
Finally, the inspectors performed a visual examination of all six DEG voltage regulator cabinets to verify that the present configurations of the components of concern was correct on the installed circuit boards.
Subsequent to the onsite portion of the inspection, on several occasions, the licensee provided additional information that enabled certain remaining issues to be resolved.
b.
Observ i
d Findin In reviewing the technical documentation associated with the DEG voltage regulator issue, the inspectors noted that the DEG static exciter-voltage regulator schematic diagram was on PG8 E Drawing DC 663082, Sheet 166, Revision 2, which was a redesignation of Baster Electric Drawing D90 67100 010, Revision "I" (as in "India"),
titled "Schematic and Interconnection Diagram for Series Boost Exciter Regulator and
J r
,1 4 yW8S~"
Ig-4.
r>
Associated Control Functions."
On this drawing, the exciter-voltage regulator was designated "Model No. 90 67100 100." Within the exciter-regulator low-voltage chassis, the block representing the voltage regulator itself was designated "Voltage Regulator Assembly, 3 PH Sensing, Model No. 90 78600 101," and referenced "Schematic Dwg No. D90 78600 910."
The detailed regulator schematic, Basler Drawing D90 78600 910, original date:
November 12, 1970 (revision block blank), titled "Schematic -Voltage Regulator - SVR Modified" showed the regulator components that are mounted on the circuit board in question within a dashed-line block annotated "Etched Circuit Board." This drawing did not give the model number for the regulator or for the printed circuit board itself. The board-mounted components comprise most smaller electronic control components of the regulator; the larger, heavier ones, such as the transformers, being separately mounted on the metal voltage regulator chassis in the exciter-regulator cabinet upon which the circuit board is also mounted.
The circuit board-mounted components in question were resistor R26 and capacitor C5. The regulator schematic showed R26 as two resistors in parallel and originally showed C5 as a separately designated single
, capacitor in parallel with capacitor C4. The Basler detailed regulator schematic had been redesignated PG&E Drawing DC663082, Sheet 60. Revision 4 of this drawing, dated September 25, 1996, showed capacitor C5 removed by a drawing change annotation that cited Field Correction Transmittal 20914, dated July 3, 1996. The operation and service manual for the exciter-regulator', stated that capacitor C5 (with C4) was part of the feedback loop in the voltage regulator stability circuit and that R26, in conjunction with capacitor C8, formed a filter network across. the control rectifier bridge.
It was the requirement for C5 and the corresponding values of associated feedback loop resistors R15 and R16 when C5 is installed that were in question initially in 1996. Subsequently, the configuration of resistor R26 came into question when the licensee discovered that the R26 for the DEG 1-3 and then the DEG 2-3 voltage regulator circuit boards were different from the drawings and from the other R26s.
The inspectors'eview of microfilm prints of archived procurement and maintenance records revealed that the voltage regulator printed circuit board has had a complicated history of changes, errors and confusion involving its part number, model number, and proper configuration for the Diablo Canyon DEGs. The original PG8E purchase order (PO) for the four spare boards, PO No. 558018, dated November 10, 1977, ordered them as part number 90 32101-105.
The associated purchase requisition, No. 7745715, dated July 29, 1977, originally specified Basler Part No. BE-32101-105.
Review of the licensee's history and evaluation of the circuit board problems indicated that this part
'umber was chosen because the circuit boards were silk screened with that number and no part number for the circuit board itself appeared on the Basler drawings.
On the basis of telephone conversations and correspondence with Basler in 1996, the licensee determined that the first error was Basler's marking the original circuit boards with the wrong part or model number.
In addition, Basler had supplied a generic voltage regulator schematic to PG8 E. Although the 1970 drawing did show R26 as a double resistor, it also showed C5 which is not used on all models and without any notes to indicate which circuit board types were supposed to have which components, and more importantly, which one Diablo Canyon was supposed to hav '+
jg
+A~4
T
<<""y~
'~hi
"t A
According to a Basler letter, dated April25, 1996, in response to licensee inquiries, Diablo Canyon's voltage regulators were Model 90 786100 101 in accordance with the Model 90 6700 100 exciter-regulator drawing. However, that type of voltage regulator can have any one of five different configurations of "etched" circuit board. The letter stated that Diablo Canyon's regulators were all built according to configuration codes specified in Basler's "Build Book " (one of the manufacturer's master design control documents).
Various codes from the Build Book form a so-called "style number" which determines how the voltage regulator is to be built, including, for example, major component specifications and interface features.
The style'number of Diablo Canyon's regulators was "SVR01A05B2B1B." The "05" code in the Build Book" specified circuit board assembly Part No. 90 32101 104.
However, even though the boards were reportedly assembled as 104s, the style number code 05 was apparently misinterpreted by the personnel marking the boards and the model number 90 32101 105 was silk-screened onto the fronts of the boards instead of 90 32101 104 as it should have been.
When the spare boards were supplied in 1977 under PO 558018, Basler shipped boards that were marked 90 32101 104 (consistent with the 05 Build Book code) and not with the 105 suffix as ordered.
Basler first explained this substitution in a February 26, 1984, letter as discussed below. Consistent with the 104 design, capacitor C5 was not installed and R15 and R16 were of the correct values for when C5 is not.installed.
However, in May of 1996, the licensee learned that the second error by Basler was that one of the several modifications (component upgrades) and model redesignations prescribed on Basler Drawing 90 78600, Revision 8, dated November 12, 1970, to be performed on all circuit board types (90 32101-100 through 104 became 101 through 109 respectively) used in the 90 78600-series regulators, had not been done on the circuit boards shipped under PO 558018.
Specifically, the boards had not been modified to change R26 from a single 100-ohm, 20-watt resistor to two 200-ohm, 25-watt resistors on a bracket and had not been remarked with modified Model No. 90 32101 109. Further, Basler had not changed the part number of the modified boards used to build Diablo Canyon's original regulators from 90 32101 104 to 90 32101 109 as required by the instructions on the drawing. Therefore, when determining that the Diablo Canyon regulators were built with boards that were originally Model No. 90 32101 104, Basler should have verified that they were configured in accordance with the 1970 modification of the double R26, and remarked them 90 32101 109 before shipping.
Thus, the so-called "new" number described in a Basler memorandum faxed to PGLE on May 7, 1996, i.e., 90 32101 109, "to avoid confusion" was not new at all, but had been established in 1970, and not put on either the original regulator boards or the spare boards supplied in 1977. The spare boards had not been remarked with the 109 suffix possibly because they had not been modified with the double R26 as they were supposed to be; although the original boards supplied with the regulators had double R26s and the other 1970 upgrades.
During further review, the inspectors noted additional discrepancies in the earlier documents.
The 1977 spare board purchase requisition indicated that the voltage regulator was Model No. "SBSR-HV" The inspector noted that this model designation was in the title of the Basler operation and service manual for the exciter-regulator, but not on the equipment name plates.
The inspector noted during the walkdown that the
i
':4 /
n
'
tg
model number on the name plates on all of the original five exciter-regulator cabinets was SBHV 67100-100.
The model number on the exciter-regulator cabinet of the sixth
~ DEG, DEG 2-3, was SBHV 9244900-100.
The inspector determined that the exciter-regulator name plate numbers, also referred to as system part numbers, except for DEG 2-3, corresponded to an exciter-regulator design consistent with the exciter-regulator drawing, but Model No. 90 67100 100 was actually shown on the drawing. The inspectors later determined that the number on the DEG 2-3 exciter-regulator name plate (92 44900 100) was a Basler generic system part number for an exciter-regulator cabinet because this unit had been built about 20 years later than the original five cabinets, had been purchased separately from the other equipment as a commercial-grade item, and had been wired on site. The equipment was also identified on the purchase 1977 requisition as "Ser. 126," which, the inspectors noted, was the serial number for the DEG 1-2 exciter-regulator; although it was not clear why the serial number, ifrelevant, of only one of the five original DEG exciter-regulators was included on the purchase requisition for spare boards for all the regulators.
The parts list sheet attached to the original,1977 purchase requisition and purchase order for the spare boards also indicated that the stock code for the board was 42-1841.
Note that this is the stock code that was later (1996) assigned to the so-called "new" Basler part number for the boards configured for Diablo Canyon's DEGs, 90 32101 109.
The record then contained what appeared to be a photocopy of the original requisition parts list sheet on which the part number was changed (pen and ink) to BE-32101-104 (with the annotation "Model 32101-104") and the stock code was now 37-3770, but the original stock code did not appear.
It was not clear when these changes had been made.
The QA requirements sheets filed with the purchase order and requisition indicated that Basler was not on the qualified suppliers list (QSL), but was exempt because of being the original equipment manufacturer (OEM). Although it was clear that the purchase was for material to be used in safety-related applications, no quality requirements were specified except receipt inspection.
The typed "PG&E Code" on the purchase order was the material class (87) and the material code (4200).
However, on the receiving copy of the purchase order, the 87-4200 was lined out and changed to 37-0770.
On the copy that indicated it would go to the vendor, the 87-4200 was changed to 42-1841, but again, it was not clear when or why these changes were made.
On January 6, 1978, PG&E issued Minor Change 1 (MC01) to its PO 558018 to incorporate a statement of the applicability of 10 CFR Part 21 to the purchase.
Although Basler was not a 10 CFR Part 50, Appendix B, supplier, and was not on the Diablo Canyon Qualified Suppliers List (as mentioned above, it had been exempted because of being the OEM), the inspectors did not identify any Basler correspondence in which the vendor took exception to that added purchase condition.
The receipt inspection report, indicating initial shipping damage inspection on February 8, 1978, and item receiving inspection on February 9, 1978, indicated that all as-received conditions considered applicable by the QC inspector were acceptable (including "Gas Blanket" which was inexplicably checked as "OK" and not marked "N/A" as were some other blocks that could be applicable, e.g., "desiccant").
Also checked as
"OK"was the "I.D. & Markings" block, yet the circuit boards received were marked with
,l-/
(
t,
the model number 90 32101-1 04 instead of BE 32101-1 05 as ordered.
Although the receipt inspection report did not document the discrepancies, it indicated under
"Document Verification" that the "PN Clarification," was acceptable.
The part number clarification referred to was a Basler letter, dated February 6, 1978, in response to PG&E questions and received shortly after the shipment.
The Basler letter referenced PO 558018 and a telephone conversation with a Mr. J.W. Weldy [ofPG&Ej and stated Basler's understanding of the intent of the PO, i.e., that it was for "the correct Printed Circuit Board Assembly used in Basler Senior Voltage Regulator, Model SVR01A05B2B1B"[sic]. However the letter did not explain at the time what this number was (later determined to be a style number, not the model number), nor did PG&E documents.
The fact that it was completely different from the model numbers on the drawings and on the equipment caused considerable confusion during later reviews, but was not questioned at the time. The letter then explained that "Circuit Board Assembly, P/N 90 32101104 is the correct unit, and therefore was supplied on the referenced purchase order." The inspectors concurred that the vendor had supplied incorrectly configured and marked spare circuit boards, but it was at this point that the licensee missed its first opportunity to detect the problem and prevent the eventual installation of the nonconforming material. There was no documented evidence that the apparent discrepancies in circuit board part numbers and in the regulator model numbers prompted the licensee to verify the information against name plates and drawings and also perform a detailed physical comparison between the spare boards received and those already installed in the regulators.
However, the procedural requirements in effect at the time also did not prompt the licensee to perform such verifications in view of the assurance by the original equipment designer and manufacturer that the parts supplied were correct.
On June 21, 1983, the PG&E QA Department issued Audit Report No. 83074P for the audit completed April 29, 1983, on Diablo Canyon's procurement and acceptance process.
As a result of this audit, the PG&E QA department initiated Nonconformance Reports (NCRs) DCO-83-PG-N020, N021 and N022. N020 reported that there was insufficient documented evidence of the attributes checked (and acceptance criteria) in receiving inspections for "off-the-shell" items used in Class 1 applications.
N021 documented that nine 1982 procurements had been identified in which material that was required to be purchased from qualified suppliers, had been purchased from suppliers that had not been qualified by PG&E QA. But this NCR did not list PO 558018 among the nine cited as deficient.
N022 dealt with procurement involving material substitutions without prior approvals.
The prescribed resolution for the NCRs, and in particular, for the one most applicable to PO 558018, N020, was to review receipt inspection records for this type of material and take corrective action on an individual basis.
One of the Purchase Document ChangeNariance forms generated as a result of this review, dated January 6, 1984, addressed the items purchased under PO 558018, although a block on the form was checked indicating that it was associated with the resolution of NCR DCO-83-PG-N021 instead of N020. Nevertheless the evaluation addressed receipt inspection of the "off-the-shelf" items purchased on the referenced PO (the NCR N020 issue) and not whether the supplier was on the QSL (the NCR N021
J
~
IIJ p
U g
Ql'
issue).
This evaluation was inadequate in that it did not directly respond to the NCR issues and simply asserted that the purchased items were "standard, commercial grade, off-the-shelf items," that "There are no special requirements beyond routine assembly line testing," that "qualification for use in safety related applications is through receipt inspection," and that "Proper receipt inspection was performed and documented."
Thus it completely ignored the NCR N020 issue of inadequate inspection attributes and acceptance criteria for "off-the-shelf" material. This evaluation reaccepted the 1977 procurement of the spare DEG voltage regulator circuit boards on the basis of what was deemed a satisfactory receipt inspection despite the NCR N021 issue.
Thus, another opportunity was missed to correct the original error, i.e, that Basler had supplied
.
incorrectly configured boards.
The next opportunity identified by the inspectors to correct the circuit board problem came in 1994 when one of the spare boards was used in the DEG 1-3 voltage regulator.
Work Order C0133243, Work Activity02, indicated that Diablo Canyon technicians replaced the original voltage regulator circuit board in the exciter-regulator of DEG'1-3 on December 15, 1994, because, the regulator had been found to be faulty after erratic voltage control had been experienced during a surveillance run. According to the work record parts documents, including the "PIMS Issue Ticket," (PIMS is the "Plant Information Management System" database) for the spare regulator board, the faulty regulator board was replaced with one of the four spare replacement boards from the onsite warehouse, Stock Code 42-1841, that had been originally procured from Basler under PO 558018 in 1977.
Review of the Integrated Parts Catalog (IPC) data in the PIMS database revealed that Materials Services had changed the stock code from 42-1841 to 37-3770 in 1984. Then in 1994, at the time of the replacement, Stock Code 37-3770 had been changed back to 42-1841 (and the purchase class changed from 7 to 5),
apparently on the basis that the original procurement under PO 558018 was viewed as a safety-related procurement.
The 1984 evaluation (resolution to NCR DCO-83-PG-N020) which changed the stock code to 37-3370, as discussed above, indicated that the boards were acceptable on the basis of satisfactory receipt inspection as stated previously.
On this basis, Materials Services did not perform any additional replacement or equivalency evaluation of the circuit board prior to changing the stock code back to 42-1841 when it was installed in
~ 1994. Although the inspectors determined that the NCR evaluation was inadequate, the use of the spare board from warehouse stock without further evaluation was allowed on this basis by procedures in effect at the time of the DEG 1-3 voltage regulator circuit board replacement on December 15, 1994.
In particular, Revision 2 of Administrative Procedure AD9.ID4, "Establishing Procurement Technical and Quality Requirements,"
Section 5.4.2, "Acceptance," allowed acceptance on the basis of receipt inspection (5.4.2.a.2), alone ifcertain requirements were met.
In addition, Section 5.4.2.a.5 provided for acceptance on the basis of post-installation testing (with a prerequisite of receipt inspection).
The inspectors confirmed that satisfactory post-installation tests and results were documented in the WO C0133243 file for the DEG 1-3 voltage regulator (and governor) troubleshooting and circuit board replacement.
Nevertheless, this was the third missed opportunity to detect the incorrectly configured circuit boards.
The last missed opportunity'to detect and correct the circuit board component configuration
I e
'cj>> p,
Jx
.4 s4 L
~,I
problems was the installation itself. The work order did not require the technicians to document performing a detailed comparison between the installed board and the replacement, apparently because the part from the warehouse was presumed to be correct. The notes section of the work record did not contain any reference to the fact that the replacement circuit did not have capacitor C5 and that R26 was a single resistor instead of double.
The inspectors'eview of documents associated with the licensee's 1996 discovery of the problems with the circuit boards revealed the following: In April 1996, Procurement Design Engineering (PDE) personnel were developing Revision 3 to Replacement Part Evaluation (RPE) E-6802 which was originally developed to accept the voltage regulator supplied as a separate commercial-grade item by Basler in 1990 for DEG 2-3. Revision 3 of RPE E-6802 was to be used to accept regulator circuit board that had been removed from the DEG 1-3 exciter-regulator cabinet in 1994 and repaired by Basler in 1995 and any future replacement boards.
Under PGB E PO D554605 issued February 1, 1995, Basler performed a failure analysis and then repair of the board removed from the DEG 1-3 voltage regulator.
PO 554605 compounded previous errors by requesting that Basler perform a failure analysis on a voltage regulator board "Model 32101-1 0, Part¹ 90-32101-00113"[sic].
The incorrect model number had been written on the associated purchase requisition (Material Requisition 4073677, dated January 26, 1995) and the incorrect part number was the result of misreading the part number written on the requisition 32101-001B.
It was not clear where these numbers came from, or which is correct, ifeither of them is.
In addition, the standard clauses attached to PO 554605 stated (on the same page) that 10 CFR Part 21 was applicable (Standard Clause 5012), but that the purchase was not safety related (Standard Clause 5021). The licensee could offer no satisfactory explanation for these contradictory statements except that on the procurement package cover sheet in PO 554605 record, PSG File No.15523, which referred to Material Requisition 4073677, the "Nonsafety-related" block under "Classification" was erroneously marked.
In the course of examining the records and the equipment in connection with the replacement part evaluation, PDE discovered discrepancies between installed circuit boards and the design drawing referred to above.
These were first documented in Action Request (AR) A0399437 on April 17, 1996. AR A0399437 stated that capacitor C5, which according to Drawing DC663082, Sheet 60, was supposed to be in parallel with capacitor C4 in the voltage regulator feedback loop of the stability circuit, was missing on the regulator boards in the exciter-regulator cabinet for DEG 1-3 the cabinet is designated SED13). The AR stated that a walkdow'n of Unit 2 revealed that the regulator board in SED21 (for DEG 2-1) also did not have C5 installed. This AR was written before PDE had determined that C5 was supposed to be absent, and that the boards with C5 (in SED12, 22 and 23) were incorrect. The AR then listed several items it characterized as documentation and procedure problems: (1) That Basler had repaired a circuit board Model No. 90 32101 100 which was placed on hold in the Diablo Canyon warehouse under Stock Code 42-1841 and listed as Part No. 90 32101 104, (2)
that the board now had C5 installed, which was not like other 90 32 101 104s, (3) that
l
"C
$,f b
PO 554605 in'correctly listed Part No. 90 32101 00113 instead of 001B (as discussed above), (4) that the PO listed Model No. 90 32101 10 (also discussed above) which the AR incorrectly stated should have been 90 32101 100, presumably based on Basler's erroneous failure report, (5) that there were three more boards in stock under 42-1841 that do not match the design drawings, (6) that they were transferred from stock code 37-3770 to "Q class" stock and accepted for safety-related use, and (7) that this was contrary to Section 5.3.5 of Procedure AD9.ID4which the AR stated required an "NES evaluation" to dedicate commercial grade items for safety-related use.
Although the inspectors determined (as discussed below) that dedication was not required at the time of the original purchase (1977), nor at the time of the installation of a spare board (1994)
by the procedures in effect at the time, the AR further stated, quite correctly, that the boards were upgraded from Purchase Class 7 (nonsafety-related) to Class 5 (for safety-related use) and that they were accepted:Based on PO 558018 without any dedication or evaluation of the differences between the original and the replacement.
The AR then went on to relate the history of the issue, generally as also determined by the inspectors, with a few factual er'rors such as misquoting the style number from the 1978
'asler letter explaining its substitution of 90 32101 104 boards for 90 32101 105s.
It was later determined (in part by the Basler failure analysis report for the board) that because PO 554605 had incorrectly specified Model No. 90 32101 10, and also because the board was marked with Model No. 90 32101 105 (which was originally a 90 32101 100 before the 1970 R26 modification) Basler assumed that the board was a Model No. 90 32101 100 (which was supposed to have C5) and treated it as such.
In fact, even though the board was really a 90 32101 109 (a modified 104 with double R26 and no C5), Basler, without verifying the number against its original manufacturing records, believed C5 to be missing and installed one.
It w'as not clear whether Basler checked the values of R15 and R16 to confirm whether C5 should be installed.
Subsequent related ARs A0399596 and A0401207 and Quality Evaluation Q0011867 further addressed the issues first identified in AR A0399437.
One of the corrective actions was to walkdown all six exciter-regulators as the inspectors did during this inspection.
The licensee found that C5 was not present on the boards in exciter-regulators SED11, 13 and 21. C5 was found in SED12, 22, and 23. The licensee determined that SED12 had C5 because the entire voltage regulator had been replaced in 1995 with a spare regulator purchased from Basler that had the incorrect stability circuit feedback loop. However, no discrepancies were identified during this installation either and no evaluations or detailed inspections were performed or required by procedures.
The faulty voltage regulator replaced in 1995 had been examined and repaired by Basler under PGSE PO D554794, issued September 26, 1995. SED23 had C5 in part because of the way it was procured and wired separately in 1990 as part of the addition of the sixth DEG. The licensee was not able to determine why SED22 was originally supplied with C5 on its regulator board.
During one of the walkdowns in response to the ARs, the licensee found that in exciter-regulator cabinets SED13 and 23, R26 was a single, 100-ohm, 20-watt resistor instead of two, 200-ohm, 25-watt resistors in parallel and mounted on a bracket as in the other four units. AR A0401207 was written to address the R26 issue and revised to address
l
~
a P
e'A
.t E
+pc, l~.
w'V
the fact, discovered later, that the R26 in SED13 had overheated, become discolored and had scorched the printed circuit board.
The licensee also found that the replace-men boards from the warehouse had a single R26 of 100 ohms (equivalent to the two 200-ohm resistors in parallel).
However, the single resistor, while having the same resistance as the two in parallel, was only rated for 20 watts and therefore could not dissipate as much heat as the double R26 configuration without becoming too hot.
In addition, not being mounted on a metal bracket which acts as a heat sink (as were the double R26 resistors), and therefore also being closer to the printed circuit board surface, was the reason the single R26 in SED13 had overheated.
In consultation with Basler, the licensee confirmed that the correct configuration of the components in question on the circuit board should be a double R26 on a bracket, no C5, and R15 and R16 of the values required with no C5 (100k ohms and 350 ohms respectively).
As stated previously, Basler had assigned part number 90 32101 109 for this configuration, to which Diablo Canyon Materials Services reassigned Stock Code 42-1841. Accordingly, Revision 04 to RPE E-6802, Minor Change 01, dated July 11,1996, specified Standard Clause 9792 (Revision 2) which, among other inspection acceptance criteria, specified the correct configuration for C5 (and R15 and R16) and R26, along with voltage regulator performance criteria.
During the visual examination of the internals of all six exciter-regulator cabinets, the inspectors noted that the model number for the whole voltage regulator (90 78600 101, shown on the exciter-regulator schematic, drawing DC663082, Sheet 166, Revision 2)
was not visible inside the cabinets.
However, stamped or stenciled on the front of the voltage regulator printed circuit boards in question were several different circuit board model numbers.
The circuit boards for DEGs 1-1, 1-3, and 2-1 bore the model number 90 32101-105, the same number used to order the spare boards in 1977.
However, the number on the board for DEG 1-2 was 90 32101-109 (with the "109" suffix apparently handwritten with a permanent marker). The number on the'DEG 2-2 circuit board was 32101-104 (the number that appeared on the spare boards supplied by Basler in 1977).
The third digit of the suffix of the number on the DEG 2-2 board appeared to have been altered, but was illegible. Nevertheless, the inspectors'isual examination of the circuit boards in all six DEG voltage regulators also confirmed that all the R26 resistors were of the correct double 200-ohm, 25-watt resistor, bracket-mounted configuration, and that all C5 capacitors had been removed.
Co usions On the basis of records associated with the 1994 DEG 1-3 voltage regulator circuit board replacement, the spare board procurement, and subsequent evaluation, the inspectors concluded that there were deficiencies in the original safety classification, procurement process, review for suitability of application and verification that all the board components met the design requirements.
Many of these deficiencies were identified by the licensee in "Action Requests (ARs) written by Procurement Design Engineering in 1996.
However, although those procurements and acceptance processes would not meet the licensee's present standards, the inspectors concluded that the 1977 procurement was done in accordance with the procedures in effect at the
T
I. *
time. The, inspectors further concluded that the procedures applicable to the replacement part installation in effect in 1994 were followed, but that they were weak in that they did not require a reevaluation of the replacement part when it had been originally accepted solely on the basis of (1) having been purchased from the OEM and exempted from having to be on the QSL when the spare boards were purchased for the exciter-voltage regulator, (2) had been reaccepted on the basis of receipt inspection by 1984 NCR evaluation which the inspectors concluded was inadequate.
The fact the board had passed post-installation tests, even though the supplier qualification and technical and quality requirements imposed on Basler originally would not have met the
.licensee's procurement and acceptance requirements in 1994, partially compensated for the lack of reevaluation, but did not preclude installation of the incorrectly configured board.
The inspectors further concluded that the work authorization and control documents used in 1994 when the circuit board was replaced with one of the incorrect part number and with an incorrectly configured R26, were weak in that they did not effectively ensure a close visual comparison to verify that the circuit board component configuration and markings matched that of the original circuit board being replaced and that all discrepancies were resolved.
The inspectors further concluded during the onsite portion of the inspection that the licensee had resolved the technical issues satisfactorily with the exception of the necessity for capacitor C5 and the correct marking of the spare and installed. boards.
Subsequent to the inspection, the licensee provided additional documentation from Basler that satisfactorily addressed the inspectors'oncerns with respect to C5. On the basis of that information as well as on satisfactory voltage regulator operation during tests, the inspectors concluded that the DEG voltage regulators were acceptable as-configured without C5. Confirmation of the licensee's strengthening of procedures as required, verification of the configuration of the boards in the warehouse and of placing the correct markings on all the boards on site is identified as Inspector Followup Item (IFI) 50-275,323/98201-01.
e I
e e
Parts Evalua io ns i
Sco e
The inspectors reviewed Diablo Canyon procedure CF3.ID13, "Replacement or New Part Evaluation (RPE)," Revision 4, dated December 10, 1996, and several RPE packages to determine how the'licensee evaluates replacement items for use in safety-related structures, systems and components.
Although an RPE can cover a wide variety of items, the-inspectors limited their review to commercial grade items, with emphasis on those that were not originally purchased with the intent to dedicate them for safety-related service.
To evaluate implementation of the program, the inspectors reviewed selected RPE packages and also the data in Diablo Canyon's parts database called the Inventory Parts Catalog (IPC).
a >'I I(
g t,4g~dq gk
b~
serv tion ndF'in s
~d
~ Fdd d
333 ID13 t bll h dth F
by td hth D
bl~
Canyon Procurement Design Engineering (PDE) and Materials Services (MS) staff evaluates, specifies and verifies both technical and quality requirements for replacement or new items. According to CF3.ID13, an RPE is used to perform equivalency evaluations'(confirms that a replacement part, which is not like-for-like, willperform its intended function), specify verification activities and implement minor modifications.
CF3.ID13 specified the responsibilities for those involved in the process, including the requestor, preparer, independent technical reviewer and the supervisor.
Attachment 8.1 to CF3.ID13 provided the forms to be used in the RPE, although not all forms are needed for every RPE. CF3.ID13 required, for replacement or new parts, that the preparer identify the safety function of the parent component (component in which the specific item is installed), function of the item, credible failure modes of the item and the effects of the failure modes.
Lastly, the preparer must determine ifthe failure of the item would prevent the parent component from performing its safety function.
The inspectors found that CF3.ID13 was comprehensive and adequately addressed commercial grade dedication requirements with some minor exceptions.
Appendix 7.2, RPE Form Instructions," included a description of Verification Activities as the four acceptance methods presented in Electric Power Research Institute (EPRI) Report NP-5652, "Guideline for the Utilization of Commercial Grade Items in Nuclear Safety Related Applications (NCIG-07)." However, the description of Method 2, Commercial Grade Survey, did not include the NRC restrictions described in NRC Generic Letter 89-02, "Actions to Improve the Detection of Counterfeit and Fraudulently Marketed Products."
In addition, Page 12 of Attachment 8.1 required the preparer to determine if the replacement item met "all three CGI criteria," referring to the previous definition of a commercial grade item in 10 CFR Part 21. The NRC changed that definition for 10 CFR Part 50 licensees in 1995.
Materials Services-PDE personnel committed to change the procedure appropriately.
No further concerns were identified.
b.2 RPE E-1011-Fasteners f r Electrical Conn c o RPE E-1011, used to dedicate commercial grade fasteners (bolts, cap screws, nuts, washers and lock washers) for Class 1E electrical connections, had been prepared'in accordance with CF3.ID13. The RPE identiTied the critical characteristics of dimensions and material, specifying dimensions to be veriTied by measurement (with samples sizes and acceptance criteria given) and material to be verified, also on a sample basis, by an "analysis method consistent with material acceptance requirements,"
The RPE specified the acceptance criterion for material simply as: "Material exhibits the characteristic of silicon bronze (copper-silicon alloy)," and did not specify the verification method or a more specific set of acceptance criteria. The inspectors questioned the material analysis method and acceptance criteria. The licensee explained that test methods and acceptance criteria were, at one time, specified in RPEs.
However, the licensee stated that experience had shown that test personnel were more familiar with test equipment (including new or updated equipment) capabilities and limitations and are, therefore, better qualified to determine the best test method and equipment.
Further discussion identified that test personnel utilize the standard specified in Diablo Canyon procurement documents.
IC P
'
g
,C q sc
The licensee's Inventory Parts Catalog (IPC) data sheet for Part No. 94-5914, a silver-bronze hex head screw, indicated that the part was dedicated in accordance with RPE E-1011. The inspectors noted that the IPC included a limitation (Standard Clause 6177)
regarding the use of the screw stating, "This item has been evaluated by RPE E-1011 as safety related when used in bolted electrical connections only. Any other safety related use must be approved by engineering prior to installation." That clause effectively restricted the use of RPE E-1011 and parts dedicated with it for specified applications.
The inspectors considered the use of such clauses in the IPC, noting a limited dedication, as a good practice which should help prevent part misapplication.
-
Based upon the inspectors'nterviews of materials analysis personnel and a detailed guided tour of the materials testing laboratory, the inspectors were satisfied with the test-personnel's ability to properly verify material critical characteristics.
The inspectors considered that consulting with the materials testing laboratory to establish the applicable standards and the best material verification methods and acceptance criteria to be an appropriate, even necessary practice.
However, the inspectors pointed out that the licensee had established the RPE to be the design output document in which the critical characteristics, verification methods and acceptance criteria are specified.
Although the inspectors did not identify any inadequate dedications attributable to the use by the materials testing laboratory of inappropriate material verification methods or acceptance criteria, even though they were not specified in detail in the RPE, the inspectors considered the practice of not specifying any verification methods, standards or acceptance criteria, once recommended by the materials testing laboratory, to be a weakness.
RPE E-8272-"EDGV Ita e R via or Line R
t r T 1" RPE E-8272isusedfor dedicating what the RPE refers to as "current boost transformers"(CBTs) made by Basler.Electric Company (Basler Part number BE 10771-002) used in the emergency diesel generator static exciter built by Basler (Note that PGB E uses the initialism "DEG" for "diesel emergency generator" elsewhere).. According to the DEG static exciter-voltage regulator schematic diagram, PG8E Drawing DC 663082, Sheet 166, Revision 2, a redesignation of Basler Drawing D90 67100 010, Revision "I"), the three components with Part No. BE 10771-002 are designated T51, T52, and T53. According to Basler Operation and Service Manual 90 67100-990, the components designated T51, T52 and T53 are called saturable transformers.
Apparently used as CBTs, the three saturable transformers are a type of magnetic amplifier that help control the DEG field excitation current in response to voltage regulator output signals on their control windings. The inspectors noted that the title of the RPE was incorrect in that T51 is not the only CBT and it is not a linear reactor.
The Basler manual indicated that the linear reactors are designated L51, L52 and L53. Nevertheless, the RPE identified appropriate dedication activities to be performed, including verifying insulation resistance and no-load voltage ratio of the CBT. RPE E-8272 stated that upon successful dedication of a CBT, its Diablo Canyon part number or stock code should be changed from 27-6697 (CBT commercial grade part number) to 95-4535 (CBT safety-related part number).
ii O.
l~
'
In reviewing the Diablo Canyon Inventory Parts Catalog (IPC) data sheet for the CBT, commercial-grade part number 27-6697, the inspectors noted that Materials Services had changed the part numbers of two CBTs to 27-6697 from 95-0357 which was the safety-related part number for a current transformer (CT), Basler Part No. 10772-001.
Basler BE 10772-001 CTs are used as CT1, CT2 and CT3, as shown on the DEG static exciter-voltage-regulator schematic, to obtain power from the DEG generator output for field excitation. The inspectors questioned why a safety-related part number had been changed to the commercial grade part number of a different part.
The licensee reported that it had determined that the Basler BE 10771-002 CBTs'ad been misidentified upon receipt at Diablo Canyon as Basler BE 10772-001 CTs, most probably due to the very similar Basler part numbers, and therefore had been mistakenly assigned the CT commercial part number 27-6694. According to the licensee, Materials Services then attempted to upgrade the commercial grade CBTs to safety-related status by use of RPE E-6795, the RPE for dedicating the CTs, because the CBTs had been erroneously assigned the CT commercial part number.
During the dedication process, Diablo Canyon technicians identified the incorrect part number and effected a change from the safety-related part number to the correct commercial grade part number. At the completion of the inspection, the dedication of the CBTs using the correct RPE, E-8272, was in progress.
The inspectors were satisfied with the explanation for the changed part numbers.
The inspectors also considered Diablo Canyon's ability to track each component through the system and identify problems associated with each part as a system strength.
b.4 E E-834
- SF6 Br a e h
in Mo r
o in eners Revisions 00 through 02 of RPE E-8346 had been developed to upgrade commercial grade fasteners for use in SF6 safety-related breakers.
Revision 02 provided acceptable specifications for bolts and nuts, but the inspectors noted that Revision 02 identified both a like-for-likeand an equivalency evaluation.
The inspectors questioned why a like-for-like and an equivalency were both identified and therefore, reviewed all earlier revisions to the RPE.
The inspectors found that the previous revisions had identified other acceptable fastener specifications as follows:
ev No ation SAE J429 Grade 8.1 or higher (screw/bolt)
SAE J429 Grade 8 or higher (nut)
DIN 933 Class 8.8 (screw/bolt)
DIN 934 Class 8 (nut)
allo medium carbon alloy medium carbon alloy medium carbon alloy low carbon MC01'SO 898-1(screw/bolt)
ISO 898-2 (nut)
medium carbon or medium carbon alloy low carbon
'inor Change
C S
ll lg
%$
r, e
s ISO 898-1 or ASTM F568 or SAE J1199 Class 8.8 (screw/bolt)
ISO 898-2 or ASTM A563M or SAE J 1199 Class 8 (nut)
medium or low carbon medium or low carbon Based upon their review, the inspectors were concerned regarding the licensee's determination of like-for-like in Revision 2 of the RPE. The inspectors interviewed a procurement engineer involved in the process.
The engineer stated that he believed that the like-for-like block was checked on the RPE form for Revision 2 because MC01 had already identified the ISO standards as acceptable and Revision 2 added the ASTM and SAE standards.
Therefore, like-for-likewas appropriate for the ISO standards while an equivalency evaluation was needed for the others.
Although the explanation may have seemed reasonable from the engineer's standpoint, it raised a concern regarding the RPE process.
Procedure CF3.ID13 prescribed the process for implementing a so-called minor change (MC), i.e., a document used to approve and transmit minor corrections to an issued RPE, but which does not require the same levels of review and approval as a revision to an RPE. The procedure required that for a change to an RPE to be treated as a minor change, it must be within the scope and intent of the RPE and not conflict with any statement, basis, logic or requirement of the effective revision of the RPE. The inspectors determined that the use of the MC process here to incorporate a specification change, as well as to expand the scope of the acceptance criteria (including medium carbon steel) was contrary to the minor change requirements in CF3.ID13 as well as to the intent of its-minor change provision. Further, MC01 did not address the basis for changing the specification from the DIN to the ISO specifications.
The inspectors determined that the tack of a documented basis for changing the material specification constitutes a failure to review material for suitability of application as required by Criterion III of 10 CFR Part 50, Appendix B. However, the changes made to the material specification were not safety significant and Revision 2 provided an acceptable equivalency evaluation.
Another example of what the inspectors considered to be an inappropriate use of the minor change provision involved RPE E-8001 (Revision 0)
which was developed to accept a replacement component cooling water pump motor (early 1995 or late 1994). Subsequent to issuing E-8001, Revision 0, the vendor proposed to substitute a different frame number for the motor ordered.
MC01 to Revision 0 was used to approved the replacement frame number which turned out to be the wrong size.
While the wrong size motor was never shipped, and would not have been able to be installed, the inspectors determined that use of a minor change to approve a change in an attribute as fundamental as the size of the replacement component was inappropriate in that it did not get the same level of review as a full revision presumably would have received.
More complete evaluation and review might have prevented approval of an improperly sized replacement component.
Further discussions with the procurement engineers indicated that Diablo Canyon will, on occasion, use the MC process to expedite needed changes to an RPE. The inspectors were concerned that this misuse of the MC system may result in materials being utilized in the plant that have not had a proper RPE performed.
The inspectors
y (p
+k Pp V+
~ 4
'lC
did not identify any other instances of improper use of the MC provision, however, based upon the discussions with Diablo Canyon employees, the improper use of the MC process is considered a weakness.
However, after discussions with the leader of the licensee's procurement and dedication self assessment team, the inspectors were satisfied that the licensee's planned corrective actions in response to the self assessment (discussed in Section E2 below) should adequately address the concerns identified.
b.5 RPE M-8572-Motor Driven A xiii Fe d Pum P rt The inspectors reviewed RPE M-8572, used to evaluate the acceptance of chrome plating an auxiliary feedwater pump internal rotating element (impellers and balance sleeve).
The chromium buildup was determined necessary because the existing rotating element was found to be undersized.
The manufacturer of the element recommended the chromium buildup as an acceptable repair method and performed the actual repair.
The inspectors found that Revision 00 of the RPE (April 17, 1997) stated that the item was a like-for-like replacement.
The rationale for this statement was that there was no change to the rotating element materials, despite the fact that the RPE identified the addition of chromium.
During an internal review of the RPE process by what the licensee called the':SPIGOT (not an acronym) Action Request (AR) Review Team, Diablo Canyon personnel reviewed Revision 00 of the RPE and determined that the like-for-like determination was in error. The reviewer determined that an equivalency evaluation should be performed and initiated AR A0452626'(Request date: February 6, 1998).
Further, the SPIGOT team determined that the like-for-like determination was a violation of CF3.ID13 and was, therefore, also a quality problem.
'evision 01 of RPE M-8572 was issued on April4, 1998, correctly identifying the RPE as an equivalency evaluation and including the necessary evaluations.
The inspectors had no further technical concerns regarding this RPE.
However, the circumstances surrounding the RPE raised an additional QA concern.
The initial determination that the chromium repair was,a like-for-like replacement raised a question as to whether Diablo Canyon staff consistently recognize what constitutes a like-for-like replacement.
The definition contained in CF3.ID13 stated that a like-for-like replacement is an "item with the same part, make and model number, that exhibits the same technical and physical characteristics as the original." Although this definition seemed clear and restrictive, RPE M-8572 indicated a lack of attention to detail on the'art of the RPE preparer, reviewer and supervisor.
Ifwide spread, this problem would be considered a program implementation weakness.
However, as mentioned above, the licensee's corrective actions in response to the self assessment should adequately address the concerns identified.
b.6 PE-E00025 Fuses Fuse Bio k n
Fu H Id r The inspectors reviewed RPE-E00025 for upgrading commercial grade fuses, fuse blocks, and fuse. holders, from various manufacturers, for safety-related service.
The inspectors noted that the RPE described the dedication activities to be performed.
The
RPE identified the components to be dedicated (fuses, fuse blocks, and fuse holders),
the parent components for, the fuses, fuse blocks, and fuse holders (safety-related electrical panels and equipment), and the item model type and part number.
The RPE identified the safety-related function of the parent components; the safety-related function of the fuses, fuse blocks, and fuse holders; the credible modes of failure such as premature opening of a fuse, failure of the fuse block base material or poor contact between the fuse holder and an installed fuse; and the effects of failure of a fuse, fuse block, or fuse holder on the system where installed (these items were all identified separately for the fuses, fuse blocks, and fuse holders).
The RPE also identified the critical characteristics of the fuses, fuse blocks, and fuse holders which included dimensions, current carrying capacity, and clearing time (fuses);
and dimensions, insulation resistance, and fuse fit (fuse blocks and fuse holders).
Fuse dimensions were verified by measurement, current carrying capacity was verified by testing, and current clearing time was measured by testing. The fuse critical characteristics were verified on a sample of the items received with the sample size specified for each critical characteristic'.
Fuse block and fuse holder dimensions were verified by measurement, insulation resistance verified by testing, and a "pull-out test" verified that the appropriate sized fuse could not be easily pulled out of the fuse holder.
The fuse block and fuse holder critical characteristics were verified on samples of the items received with the sample size specified for each critical characteristic.
In addition, the inspectors reviewed the IPC for Part No. 94-2045; a 30 amp fuse block, dedicated in accordance with. RPE-E00025.
The IPC specified Standard Clause 7071 which provided instruction on the dedication activities to performed.
The inspectors noted that dedication activities specified by Standard Clause 7071 were in agreement with RPE-E00025.
The inspectors also reviewed Work Order No. C015667 which provided instructions on a fuse holder replacement.
The inspectors concluded that the RPE, IPC, and Work Order reviewed had been adequately prepared in accordance with the applicable Diablo Canyon procedures and guidelines.
b.7 RPE E-1013 Tran isto s Generic The inspectors reviewed RPE E-1013 for upgrading commercial grade transistors, from various manufacturers, for safety-related service.
The inspectors noted that the RPE described the dedication activities to be performed.
The RPE identified the component to be dedicated (transistors), the parent components for the transistors (various applications), and the item model type and part number.
The RPE identified the safety-related function of the parent component, the safety-related function of the transistor, the credible modes of failure of a transistor such as short circuit or open circuit, and the effects of the failure of a transistor on the system where installed.
The RPE also identified the critical characteristics of the transistors which included configuration, verification to the transistor characteristics curve, forward-current transfer ratio (HFE), saturation voltage and resistance (VCE), and breakdown voltage emitter to base (VEB). Configuration was verified by inspection.
Verification of performance in accordance with the transistor characteristics curve, HFE, VCE and VEB were verified
"1 t
'<<ie tA
by testing. The transistors'ritical characteristics were verified on a sample of the items received with the sample size specified for each critical characteristic.
In addition, the inspectors reviewed the IPC for Part No. 93-3052, an NPN transistor, dedicated in accordance with RPE E-1013. The IPC specified Standard Clause 9941 which provided instruction on the dedication activities to performed.
The inspectors noted that dedica-tion activities specified by Standard Clause 9941 were in agreement with RPE E-1013.
The inspectors also reviewed Work Order No. C0152725 which provided instructions on a transistor replacement in a boron concentration measurement device.
The inspectors concluded that the RPE, IPC, and Work Order reviewed had been adequately prepared in accordance with the applicable Diablo Canyon procedures and guidelines.
b.8 RPE-E00039 Thermal Ov rl I
nd Ove oad eaters The inspectors reviewed RPE-E00039 for upgrading thermal overload relays and overload heaters, from various manufacturers, for safety-related service.
The inspectors noted that the RPE described the dedication activities to be performed.
The RPE identified the components (overload relays and overload heaters), the parent components for the thermal overload relays and overload heaters (various applications),
and the item model type and part number.
The RPE identified the safety-related
.
function of the parent component, the safety-related function of the thermal overload relays and overload heaters, the credible modes of failure such as open circuit, high resistance due to material failure, and short circuit, and the effects of failure of the thermal overload relays and overload heaters on, the system where installed.
The RPE also identified the critical characteristics of the thermal overload relays and overload heaters which included configuration, dimensions, insulation resistance, contact current rating, contact configuration, and operability (thermal overload relays)
and the applicable overcurrent trip time range and continuity (overload heaters).
Configuration, dimensions, contact current rating, contact configuration, and the applicable overcurrent trip time range were verified by inspection, and insulation resistance, operability and continuity were verified by testing. The thermal overload relays and overload heaters critical characteristics were verified on a sample of the items received with the sample size specified for each critical characteristic.
In addition, the inspectors reviewed the IPC for Part No. 93-1335, an overload heater, dedicated in accordance with RPE-E00039.
The IPC specified Standard Clause 7057 which provided instruction on the dedication activities to performed.
The inspectors noted that dedication activities specified by Standard Clause 7057 were in agreement with RPE-E00039. The inspectors also reviewed Work Order No. C0156917 which provided instructions on an overload heater replacement in an auxiliary feedwater pump motor controller. The inspectors concluded that the RPE, IPC, and Work Order reviewed had been adequately prepared in accordance with the applicable Diablo Canyon procedures and guidelines.
b.9 E-1194 un I
RPE P-1194 documented the dedication of a commercial-grade 3/4-inch, Kunkle relief-valve to be installed in the exhaust end of the auxiliary feed water (AFW) pump turbine casing (DCPP ID Nos. MS-1(2)-RV-57), a Design Class 1 (safety-related) application. The licensee developed this RPE because
C 7'
't
the valve manufacturer, Kunkle Valve Co., was no longer on the Qualified Supplier List (QSL) and did not have an "N-Stamp" under the Boiler and Pressure Vessel Code of the American Society of Mechanical Engineers (ASME Code).
Revision 01 of RPE P-1194, dated January 10, 1996, stated that the functions of RV-57 were overpressure protection for the exhaust end of the AFW pump turbine casing and system pressure retention.
The postulated failure modes given were fracture of the spring or valve body.
,
The RPE indicated that the postulated failure modes could prevent the parent component from performing its safety function. Revision 00 of the RPE had specified that the valve body material (speciflied in both revisions as "typical stainless steel" ) was to be verified by X-ray fluorescence test, but the verification method was deleted in Revision 01. See the previous comments on not specifying verification methods.
To verify performance of the overpressure protection function, both revisions of the RPE specified performance of the DCPP maintenance procedure for safety relief valve set point calibration (25.9 psig for RV-57), but only Revision 1 specified verification of the function of pressure retention (by a 1-minute seat leakage test at 25 psig). However, having stated that fracture of the spring was a credible failure mode and that it could adversely affect the safety function of the turbine, the RPE did not identify spring material as a critical characteristic.
Although a fractured spring would be detected by the performance tests upon acceptance, not specifying the material of the spring would not provide reasonable assurance of acceptable corrosion and erosion resistance.
Also, the RPE did not discuss the effects of the failure modes, e.g., would a failed open RV-57 (due to a fractured spring) impact turbine operation or could the escaping steam potentially damage adjacent safety-related equipment?
However, upon further review in response to the inspectors concerns, the licensee determined that contrary to the RPE, the pressure retention function was not, in this case, a relevant function for the system since a,complete loss of RV-57 was considered to have a negligible effect on AFW turbine operation.
In addition, although this was not clear from the RPE, the inspectors learned that the particular model of Kunkle relief valve used for RV-57 was a sentinel valve whose main purpose was to serve as an audible alarm in case the exhaust of the turbine should become blocked. Therefore the RPE had erroneously identified pressure retention as a safety function of the valve.
With regard to the effect of escaping steam (from a fractured valve body as opposed to normal valve discharge-same as failed open valve) on adjacent equipment, the licensee had not evaluated that scenario, but committed to doing so and to include a more complete discussion of the various failure effects in the RPE.
Qgncfu~si n
The inspectors found that CF3.ID13 was comprehensive and, ifproperly implemented, should assure that dedicated commercial grade items will perform their intended safety function. However, the inspectors considered not specifying standards, verification methods and acceptance criteria in the RPE to be a weakness.
The inspectors also considered misuses of the RPE minor changes a weakness.
In addition, based upon an RPE review, the inspectors were concerned as to whether Diablo Canyon procurement engineers consistently recognize what constitutes a like-for-like replacement versus
C r v'
Q I
.sg
= 4 t
1 s ~re
what needs an equivalency evaluation. After discussions=with the leader of the licensee's procurement and dedication self assessment team, the inspectors were satisfied that the licensee's planned corrective actions in response to the self assessment should adequately address the concerns identified. With respect to the Kunkle relief valve dedication, the inspectors concluded that RPE P-1194 had failed to identify the appropriate safety related functions of the relief valve and did not adequately address spring material. Taken in the aggregate, the deficiencies identified above could be considered violations of the Criteria III, V, and Vllof 10 CFR Part 50, Appendix B.
However, in view of their relatively low safety significance, the deficiencies are being treated as a minor violations consistent with Section IVof NUREG-1600, Revision 1, May 1998, Gene al Statement of Polic and Procedures for'NRC Enforcement ctions, and no notice of violation will be issued.
E2 eviewofDi bloC n onR I
e t Ev I ation Self A es n
Ins ection Sco e
'I The inspectors reviewed the recently issued Diablo Canyon Independent Safety Engineering Group (ISEG) Report No. 980360053.
The ISEG report documented an assessment performed to review the RPEs issued since January 1995 and the process used to upgrade items initiallypurchased as non-safety related for safety-related use.
The inspectors also reviewed the ARs which had been generated as a result of the ISEG RPE assessment.
The inspectors reviewed the corrective actions which had been documented in the ARs and the weaknesses identified.
ervations and indin s The assessment team's report indicated that it had reviewed approximately 50 RPEs covering a variety of equipment, components, and material including Velan valves, Crosby relief valves, steam generator primary manway gaskets, Gould pump
'eplacement parts, emergency diesel generator saturable transformer, Sorrento power supplies, and 4-kV circuit breaker auxiliary switches.
The results of the assessment identified seven quality problems which had been documented in Action Requests (ARs)
and had identified nine weaknesses.
The assessment team had concluded that "the RPE process is in a state of decline" and that "the rate of decline, and the reason for it are unknown at this time, but merits follow-up in the future."
The ISEG RPE ARs documented problems such as an inadequate technical evaluation for two stock codes, misclassification of identical versus equivalent, inadequate ASME Code reconciliation during procurement, misclassification of an RPE, an RPE evaluated as generic as opposed to specific, an inadequate equivalency evaluation, and lack of a design change evaluation.
The inspectors noted that all of the ARs had documented corrective actions, most of which had been completed.
The inspectors discussed the assessment with both Nuclear Quality Services (NQS) and procurement personnel to obtain further insight on the significance of the conclusion drawn by the ISEG assessment team concerning the state of the RPE process.
NQS
e A-
indicated that the assessment was a component of a continuous audit cycle and was therefore most accurately viewed in that broader context. The audits were typically performed on a quarterly basis and any additional information received (including corrective actions) after the assessment was distributed, but prior to the issue of the Quarterly Audit Report, would be considered.
The assessment would receive further review when discussed in the Quarterly Audit exit meeting and the results would be incorporated into the Quarterly Audit Report.
NQS indicated that the "state of decline" was relative to the last assessment performed in the RPE area and did not indicate a breakdown of the RPE process.
In addition, NQS stated that the assessment results indicated the possible need for increased monitoring and oversight of the process and the need for another assessment of the area after an appropriate period of time, but that the results of the assessment were not significant enough to require the generation of a nonconforming condition report (NCR) to address the issue.
Procurement Services indicated that they had reviewed the assessment and were developing an approach to address the general concerns documented in the ARs and in the identified weaknesses.
Con lus'o s On the basis of the inspectors'wn review of weaknesses identified by the ISEG, the inspectors concluded that the ISEG had adequately documented areas where attention could be focused by personnel using the assessment to provide input or develop actions to further improve the RPE process.
The inspectors further concluded with respect to the self assessment that the results of the assessment and the general conclusions given by the ISEG assessment team should be viewed in the broader context of the Quarterly Audit process and that the summary of the current state was relative to earlier-assessments and not necessarily indicative of an absolute status.
The inspectors concluded, based on the review of RPEs, IPC, Standard Clauses, and Work Orders, performed during the inspection, that in general the RPE program was adequate, but suffered from some implementation problems.
The inspectors were concerned that sufficient resources may not always be devoted to staffing and training in the materials services and procurement design engineering areas to ensure the consistent quality and suitability of purchased material and services associated with safety-related equipment applications and to avoid further problems with proper review and authorization of document changes, material equivalency evaluations and problems such as those cited in the procurement self assessment.
E3 rocuremen e
r'3.1 Warehoejse ns ecio Sco As a part of the evaluation of the effectivenes's of the Diablo Canyon QA Commercial Grade Dedication Program, the inspectors reviewed the material control processes for handling and storage of items in the main warehouse.
The inspectors also performed a walkdown of the warehouse.
C
~
hE f~
Obs rvatio nd Fin in s The warehouse was divided into two main areas designated Level B and Level A in accordance with requirements of ANSI N45.2.2. The storage level was specified in the Inventory Parts Catalog (IPC) by the plant procurement group personnel in accordance with plant procedure CF5.ID2, "Control of Material in Storage."
Plant Material Services personnel store the materials in the designated warehouse locations.
The inspectors noted that the Level "A"area was a seg'regated zone with controlled access and appropriate means for the control of temperature and humidity, although it was mainly occupied by items permitted to be Level "B" storage.
Allitems stored in the warehouse were uniquely identified in order to permit traceability with procurement documentation.
The identification of items consisted of labels directly affixed to the item or on bags holding small spare parts.
The identification code consisted of a six-digit number, called a stock code, which correlated to the PG&E purchase order for the item. This stock code was also used in the IPC computer data base allowing warehouse personnel to find the physical location of each individual item.
During the warehouse walkdown, the inspectors randomly selected samples of stocked items to verify that they had been stored in the appropriate. location. One of these items (caps 3/8" SS Hex Tubes - valve spare parts), physically located in bin AB-58-07-5 was labeled with stock code No. 93.9919, but with warehouse location identification No.
AB-58-06-6. The inspectors also observed that bin AB-58-06-6 was actually occupied by paramedic medical scissors with stock code number 76.6355.
The licensee explained this apparent error by stating that the designation of all physical locations for items, in the warehouse was controlled in the IPC data base via the stock codes.
The inspectors verified that the IPC data base actually listed stock code No.
93.9919 as having warehouse location identification No. AB-58-07-5 as its physical location. Upon further scrutiny, the inspectors learned that material under stock No.
93.9919 had been relocated to bin AB-58-07-5 but that, at least one of the individual items remained identified with the previous bin designation (i.e., AB-58-06-6).
Apparently, warehouse personnel had relied on the bin location identified for the item in the IPC data base and had not updated the package's original bin location marking. The inspectors'eview of procedure CF5.ID2 "Control of material storage", and the warehouse notebook instructions, confirmed that this was an acceptable practice.
Conclusi ns The inspectors concluded that the facilities and associated controls in the main warehouse were adequate.
However, absent specific instructions requiring that warehouse personnel verify the physical location/relocation based only on the IPC data, there is the possibility that the printed location (in the label affixed to the items) could become a source of errors in-the storage of individual items. The licensee agreed to evaluate this possibility and take appropriate action ifwarranted.
I
C
~
z kp
'J I
0'I
el Mana ement Meetin s A formal entrance meeting was held with Diablo Canyon management and cognizant staff on Tuesday, April 7, 1998. The inspectors explained the scope and objectives of the inspection, and discussed report issuance and how any enforcement or followup items would be handled.
Logistics and communications arrangements during the during the inspection were agreed upon including special arrangements for the concurrent visit of, and inspection observation by, two representatives of the nuclear regulatory agency of the Slovak Republic.
The inspectors conducted the exit meeting with Diablo Canyon management and cognizant staff on Thursday, April 9, 1998. The inspectors briefed the licensee on the major inspection findings and their possible and probable disposition, subject to NRC management review.
PARTIALLIST OF PERSONS CONTACTED PG8 E Lic n ee for Diablo Can on R.P. Powers, Vice President, Diablo Canyon Power Plant Stan Ketelsten, Supervisor, NSAL-Regulatory Services Dwight Christensen, Engineer, NSAL-Regulatory Services Charlie Nichols, Director, Procurement Services Michael Jacobson, Sr. Engineer, Nuclear Quality Services Aaron Silva, Intern, NSAL-RS Dave Taggart, Director, Nuclear Quality Services Tom Fetterman, Director, IBC/Electrical Design Engineering David Oatley, Manager, Maintenance Services Terry Grebel, Director, Regulatory Services Jim Love, PQA, Procurement Services Paul Milne, Engineer, PPE Jeff Nubbe, Procurement Specialist, Materials Services Bill Crocket, Manager, Nuclear Quality Services John J. Griffin, Supervisor, Nuclear Quality Services Stefan Bednarz, System Engineer, Diesel Emergency Generators Usama Elbakahish, Engineer, Procurement Design Engineering Mike Williamson, Meg Supervisor, Materials Services Jay Young, Director, Nuclear Quality Services
~USNR Ken Perkins, Acting Director, Walnut Creek Field Office David Proulx, Senior Resident Inspector, Diablo Canyon Power Plant Brad Olson, Region IV, Walnut Creek Field Office, Project Engineer Don Allen, Region IV, Diablo Canyon Resident Inspector
A
~
'I 0