ML13092A118
| ML13092A118 | |
| Person / Time | |
|---|---|
| Site: | Hatch, Davis Besse, Cooper, Farley, Robinson  |
| Issue date: | 03/29/2013 |
| From: | Damerell P MPR Associates |
| To: | Document Control Desk, Office of Nuclear Reactor Regulation |
| References | |
| 48660 | |
| Download: ML13092A118 (12) | |
Text
0312912013 U.S. Nuclear Regulatory Commission Operations Center Event Report Paze I Part 21 (PAR) Event# 48660 Rep Org: MPR ASSOCIATES, INC ENGINEERS Notification Date I Time: 01/09/2013 18:09 (EDT)
Supplier: BASLER ELECTRIC Event Date / Time: 10/27/2012 (EST)
Last Modification: 03/29/2013 Region: 1 Docket #:
City: ALEXANDRIA Agreement State: Yes County: License #:
State: VA NRC Notified by: PAUL DAMERELL Notifications: CHRISTOPHER NEWPORT R1DO HQ Ops Officer: BILL HUFFMAN MARK FRANKE R2DO Emergency Class: NON EMERGENCY JOHN GIESSNER R3DO 10 CFR Section: VINCENT GADDY R4DO 21.21 (a)(2) INTERIM EVAL OF DEVIATION PART 21 REACTORS GRP E-MAIL PART 21 INTERIM REPORT ON THE FAILURE OF AN EMERGENCY DIESEL GENERATOR EXCITATION SYSTEM The following report was received from MPR Associates via facsimile:
"MPR Associates (MPR) is investigating the failure of a replacement emergency diesel generator excitation system that MPR supplied to Cooper Nuclear Station. The root cause investigation is still in-process and will not be completed within 60 days of discovery as defined by 10 CFR Part 21.
"The 10 CFR Part 21 Interim Report [below] provides the information known at this time. An updated report will be provided once the root cause investigation is completed.
IDENTIFICATION OF THE BASIC COMPONENT THAT FAILED "The basic component is a Basler Electric SBSR emergency diesel generator (EDG) excitation system that was supplied as a replacement system to Cooper Nuclear Station (CNS). The replacement system included design changes relative to the original CNS excitation system, which is also a Basler Electric SBSR design. The design changes included larger magnetic components, which were intended to allow for continuous operation of the new exciter at the.EDG overload rating.
IDENTIFICATION OF THE SUPPLIER "The excitation system was supplied by MPR Associates (headquarters in Alexandria, VA).
"Basler Electric (headquarters in Highland, IL) designed and fabricated the system under a commercial grade cr~i 9
03/29/2013 U.S. Nuclear Regulatory Commission OperationsCenter Event Report Paae__2 Part 21 (PAR) Event# 48660 program certified to ISO 9001:2008. MPR Associates dedicated the commercial grade item for nuclear use under the MPR Nuclear QA Program, which complies with 10 CFR 50 Appendix B and ASME NQA-1.
NATURE OF THE FAILURE "CNS installed the replacement excitation system in the Division 1 EDG system during refueling outage RE27.
Prior to declaring the EDG operable, CNS manually terminated an EDG maintenance run due to erratic EDG reactive power indication, which was followed by a sudden drop of EDG reactive load and an indication of negative reactive power.
"Several rounds of troubleshooting and surveillance testing were performed unsuccessfully. The surveillance testing resulted in faults to ground and overheating and failure of components in the excitation system. Some of the failed equipment included the automatic voltage regulator (AVR), manual voltage control autotransformer (T60),
rectifier power diode failure indication resistors and light-emitting diodes (LEDs), insulation on the control windings of two saturable transformers, and elements of the data acquisition equipment used to record data during the testing.
"The root cause of the failure has not yet been determined. However, on-site troubleshooting efforts at CNS by MPR and Basler Electric identified unexpected high voltages across the direct current (DC) control winding of the saturable transformers. These voltages are likely the cause of the failures experienced in-situ at CNS.
"Follow-up tests at Basler Electric on a similar replacement system designed for the Hatch Plant (but not yet installed in the plant) also identified higher than expected voltages across the DC control windings of the saturable transformers. MPR and Basler Electric recommended postponement of the Hatch Plant installation until the impact of this condition (i.e., the higher than expected voltages) is evaluated. Note that the testing on the Hatch replacement system to date did not result in failure of the system or abnormal function of components external to the saturable transformers.
NATURE OF THE FAILURE "Testing of the replacement excitation system began on October 27, 2012. The maintenance run resulting in erratic EDG reactive power indication was performed on October 30, 2012. CNS, MPR, and Basler Electric discontinued troubleshooting efforts for the replacement excitation system on November 9, 2012.
"The replacement system was removed, and the original Basler Electric SBSR excitation system was re-installed.
CNS declared the re-installed system operable on November 14, 2012.
"MPR formally documented the issue in the MPR corrective action program on November 13, 2012.
NUMBER AND LOCATION OF THE AFFECTED BASIC COMPONENTS "Based on the information known to date, this 10 CFR Part 21 Interim Report affects the following SBSR type excitation systems that were dedicated and supplied by MPR.
Nuclear Plant Date Equipment Provided Items Supplied Cooper Nuclear 2012 1 Systems (failed during installation)
Hatch 2012 5 Systems (not yet installed)
CORRECTIVE ACTION PLAN "MPR is performing a failure analysis and root cause investigation to determine the extent of the condition, corrective actions, and actions to prevent recurrence. The root cause investigation is scheduled for completion by March 29, 2013.
03/29/2013 U.S. Nuclear Regulatory Commission OperationsCenter Event Report Paoe 3 Part 21 (PAR) Event# 48660 ADVICE GIVEN TO PURCHASERS OR LICENSEES "There are numerous Basler Electric SBSR type excitation systems in service at multiple plants throughout the nuclear industry. In addition to CNS and Hatch, MPR has dedicated and supplied SBSR excitation systems to the Beaver Valley, Davis-Besse, and Robinson plants. Each system supplied is custom designed for the generator that it is slated to control. Basler Electric SBSR excitation systems have demonstrated reliable service for many years.
"The replacement SBSR excitation system supplied to CNS was not identical to the original system. Specifically, there were design differences in some of the components, including larger transformers, which were intended to allow for continuous operation of the new exciter at the EDG overload rating. Although it was not foreseen (and not revealed by factory acceptance testing), it seems that the design changes in the replacement system led to its maloperation and failure when it was initially installed at CNS. This faulty operation and failure were readily observed as part of normal EDG surveillance testing. Upon re-installation, the original SBSR excitation system functioned properly. Hence, it appears that:
- Differences between the replacement system and original system lead to the problem, and The problem is readily detectable in normal surveillance testing.
"For these reasons, SBSR excitation systems installed at plants that have shown reliable operation during surveillance testing are in a satisfactory state and condition. MPR has no evidence that the mechanism or conditions that led to the failure at CNS will lead to failures at other installations. Therefore, plants with SBSR excitation systems installed should continue to use them and conduct normal surveillance testing."
- *
- UPDATE FROM PAUL DAMERELL (VIA FAX) TO HOWIE CROUCH AT 1830 EDT ON 3/29/13 **
- Reporting Individual: Paul Damerell, Principal Officer, MPR Associates, Inc., 320 King Street Alexandria, VA 22314.
As indicated in the interim report, the EDG excitation system supplied to Cooper Nuclear Station failed. The root cause analysis identified a manufacturing defect induced design error in the saturable transformers supplied with the excitation system. The manufacturing error allowed the transformer control winding coils to shift which weakened the dielectric strength of the transformer control winding insulation system such that it could not withstand the voltage induced in the control windings. Testing and analysis estimated induced voltages exceeded 10 kV peak at motor start.
MPR Associates also determined that H.B. Robinson has 6 saturable transformers installed with 4 spares and Farley has 3 spares with potential manufacturing defects. They noted that all installed transformers that have passed installation and surveillance testing to date have no immediate concerns but could have long-term reliability issues, if the manufacturing deficiency is present.
MPR and Basler are coordinating with Cooper and Hatch to replace saturable transformers supplied by MPR.
Cooper has 3 and Hatch has 15 of the devices. Additionally, they will coordinate with Robinson and Farley to test their transformers to determine if the manufactured defects exist.
All affected licensees have been notified by MPR Associates.
Notified R1 DO (Krohn), R2DO (Seymour), R3DO (Lara), R4DO (Pick) and Part 21 Group (via email).
UMPR Date: March 29, 2013 ASSOCIATES INC.,
E N G 11N E F R S From: Michael K. Dunkelberger To: Document Control Desk Company: Nuclear Regulatory Commission Fax: 301-816-5151 Phone: N/A
Subject:
10 CFR Part 21 Report Submittal Total Pages: 9 pages including cover sheet Message:
Thu purpose of this fax is to submit a Report in accordance with the requirements of 10 CFR Part 21.
The details regarding this submittal are provided in the pages attached.
320 KING STREET ALEXANDRIA, VA 22314-3230 703-510-0200 FAX: 703-619-0224 http:l/www.mpr.com
- MPR ASSOCIATES INC ENGINE ERS March 29, 2013 RPT-03 15-0062-2110 Revision 0 Document Control Desk U.S. Nuclear Regulatory Commission Washington, DC 20555-0001
Subject:
Report in Accordance with 10 CFR Part 21
Dear Sir or Madam:
MPR Associates (MPR) investigated the failure of a replacement emergency diesel generator excitation system that MPR supplied to Cooper Nuclear Station. An interim 10 CFR Part 21 Report related to this failure was submitted on January 9, 2013.
The investigation determined that this failure is reportable per the requirements of 10 CFR Part 21. The attached 10 CFR Part 21 Report provides the required information.
If you have any questions, please contact the undersigned.
Sincerely, Paul S. Damerell Principal Officer Attachment 320 KING STREET ALEXANDRIA, VA 22314-3230 703-519-0200 FAX: 703-51ýM224 http:/Auww.mpr.com I
UMPR A8QCIATES iNQ ENGINEERS Attachment to MPR Letter dated March 29, 2013 RPT-0315-0062-2110 Revision 0 10 CFR Part 21 Report on Replacement EDG Excitation System Failure at Cooper Nuclear Station This attachment provides the information required for reporting a defect in accordance with 10 CFR Part 21.
- 1. Reporting Individual Paul Damerell, Principal Officer MPR Associates, Inc.
320 King Street Alexandria, VA 22314
- 2. Identification of the Basic Components that Failed or have the Potential to Contain a Defect The basic component that failed is a Basler Electric Company SBSR emergency diesel generator (EDG) excitation system (part number 9457800100) that was supplied as a replacement system to Cooper Nuclear Station (CNS).
The root cause investigation and extent of condition evaluation identified a manufacturing deficiency on a Basler Electric Company SBSR excitation system (part number 9451700100) saturable transformer (part number BE34093-001) that was supplied to Hatch Nuclear Plant (HNP) for training purposes (i.e., a non-safety related item). The manufacturing deficiency would be classified as a defect if found in the saturable transformers that were supplied with the safety related systems. Therefore, the saturable transformers supplied with the safety related systems may have a defect and will be replaced. The safety related systems were not installed at HNP. See Section 6.
The extent of condition evaluation identified saturable transformers that require inspections and/or tests to ensure they are suitable for the application. See Section 6.
- 3. Identification of the Supplier MPR Associates (headquarters in Alexandria, VA) supplied the CNS excitation system that failed and the saturable transformers listed in Table 6-1 of this report.
Basler Electric Company (headquarters in Highland, IL) designed and f~bricated the equipment under a commercial quality program certified to ISO 9001:2008. MPR Associates dedicated the
commercial grade items for nuclear use under the MPR Nuclear QA Program, which complies with 10 CFR 50 Appendix B and ASME NQA-1.
- 4. Nature of the Failure I Defects and the Safety Hazard Created 4.1. Failureat CooperNuclearStation CNS installed the replacement excitation system in the Division 1 EDG system during refueling outage RE27. Prior to declaring the EDG operable, CNS manually terminated an EDG maintenance run due to erratic EDG reactive power indication, which was followed by a sudden drop.in EDG reactive power.
Several rounds of troubleshooting and surveillance testing were performed unsuccessfully. The surveillance testing resulted in ground faults, overheating, and failure of components in the excitation system. Some of the failed equipment included the automatic voltage regulator (AVR), manual voltage control autotransformer (T60), rectifier power diode failure indication resistors and light-emitting diodes (LEDs), insulation on the control windings of two saturable transformers, and elements of the data acquisition equipment used to record data during the testing.
4.2. Nature of the Defects The defects in the replacement EDO excitation system that specifically led to the failure at CNS were:
- The saturable transformer control winding insulation system was not designed to withstand the AC voltages that are induced in the windings during motor starts.
" The control winding coils in the failed saturable transformers were shifted due to a manufacturing deficiency, which weakened the dielectric strength of the saturable transformer control winding insulation system such that it could not withstand the voltage induced in the control windings.
MPR and Basler Electric Company personnel believed that the magnetic flux generated by the primary and secondary windings of the saturable transformer cancelled in the center leg (i.e., the control winding) and did not provide a significant contribution to the voltage experienced by the control winding during operation. Tests and analyses performed as part of the root cause investigation estimated that AC voltages of over 10 kV peak were induced in the control windings of the replacement system during actual motor starts at Cooper Nuclear Station.
Furthermore, teardown inspections found that the innermost layer of the control winding in each saturable transformer (part number BE34009-001) had shifted from the intended location such that the end turn was at or beyond the edge of the insulating layer. Indications of arcing were found at the locations of this manufacturing deficiency on the failed saturable transformers in the CNS excitation system.
The induced AC voltages caused ground faults in two saturable transformer control windings.
These ground faults led to other failures in the excitation system and to the inability of the system to control reactive power.
4.3. The Safety Hazard Createdby the Defects The safety hazard created by failure of the EDG excitation system is the inability of the EDO to provide emergency AC power during a loss of off-site power scenario.
- 5. Date of the Failure Testing of the replacement excitation system began on October 27, 2012. The maintenance run resulting in erratic EDG reactive power indication was performed on October 30, 2012. CNS, MPR, and Basler Electric Company discontinued troubleshooting efforts for the replacement excitation system on November 9, 2012.
The replacement system was removed, and the original Basler Electric Company SBSR excitation system was re-installed. CNS declared the re-installed system operable on November 14,2012.
MPR formally documented the issue in the MPR corrective action program on November 13, 2012. An interim 10 CFR Part 21 Report was submitted by MPR on January 9, 2013.
- 6. Number and Location of the Basic Components Supplied by MPR that Failed or have the Potential for a Defect The Basler Electric Company SBSR EDG excitation system (part number 9457800100) that was supplied as a replacement system to CNS is a modified version of the system originally installed at CNS (i.e., the legacy system) and different from any other excitation systems supplied by MPR. The extent of condition evaluation considered whether the identified defects (i.e., the control winding insulation system design and manufacturing deficiency) could cause failures of similar systems or components supplied by MPR.
Saturable transformers supplied in Basler Electric Company SBSR EDG excitation systems have either a series-series or series-parallel wiring configuration. Each system is custom designed for the specific application. Series-series type saturable transformers (part number BE34009-001) were supplied with the CNS replacement EDG excitation system. Both series-series and series-parallel saturable transformer designs supplied by MPR were considered in evaluating the extent of condition, Magnitude of AC Voltages Induced Tests on SBSR systems quantified the AC voltages induced in the DC control windings of the saturable transformers under a range of operating conditions. The magnitude of the AC voltage induced in the control winding is dependent on the system design parameters and is significantly higher for series-series designs than for series-parallel designs.
The design changes incorporated into the CNS replacement system significantly increased the magnitude of the AC voltages induced on the control windings relative to the legacy system that has operated reliably for over thirty years.
Investigation of the Manufacturing Deficiency As part of the root cause investigation for the CNS SBSR excitation system failure, the HNP excitation system replacement test article (part number 9451700100), which had been provided to HNP as a non-safety related item for training purposes, was tested at Basler Electric Company. During testing, a saturable transformer in the HNP test article shorted to ground and failed.
Disassembly of the failed saturable transformer uncovered shifting of the control winding coils in a manner similar to that seen in the CNS transformer teardowns but to a lesser extent. The innermost layer of coils had uniformly shifted but not to the extent that the entire layer of coils was shifted to the edge of the insulation. Instead, the end turn (a single wire) had separated from the rest of the coils in the layer, to a location near the edge of the insulation. Evidence of arcing was present on the 'stray' wire.
The failure of the HNP test article and the resulting teardown confirmed that the manufacturing deficiency seen on the CNS transformers is not isolated to the CNS replacement design. The
'stray' wire was not a result of coil movement, but resulted from the method of connecting this wire to the lead out wire for customer connection.
Tests and inspections conducted by Basler Electric Company indicate that windings wrapped around a rectangular tube that is somewhat flexible are more susceptible ýo shifting than are windings wrapped around a more rigid tube. The shift occurs during manufacturing, not during operation of the transformer. Basler Electric Company determined that the location of the coil edge in a winding can be identified by conducting non-destructive examinations.
Impact on Equipment Previously Supplied by MPR The impact on equipment previously supplied by MPR was evaluated to determine whether: a) the design is not capable of withstanding the induced voltages or b) there is the potential for a manufacturing deficiency, which raises concerns on the ability of the equipment to perform its safety fimction.
Table 6-1 identifies the saturable transformers supplied by MPR that are confirmed to have a defect, suspected of having a defect, or have the potential for a defect. The table also indicates whether the defects (if present) are a design and/or manufacturing issue.
Table 6-1. Saturable Transformers (STs) Supplied by MPR that have Confirmed, Suspected, or the Potential for Defects Confirmed, Plant Configuration Basler Part Provided as I Number Suspected, or Number Status (Note 1) of STe Potential Defect CNS Confirmed (2012) Series-Series BE34009-001 System / Returned 3 Design and Mfg. Deficiency HNP Suspected (2012) Series-Series 8E34093-001 System I QA Hold 15 Mfg. Deficiency HNP Potential (2011) Series-Series BE34093-001 Spares I OA Hold 3 Mfg. Deficiency CNS - Spares / QA Hold 2 Potential Legacy Series-Series BE34154-001 Mfg. Deficiency (2011) Spare / Installed 1 Note 2 Systems / Installed 6 Note 2 Robinson Series-Parallel BE33821-001 Spares / Advised Potential (2010) by MPR to place on 4 Mfg. Deficiency QA Hold Farley Spares ( Advised Potential (2010) Series-Parallel BE10618-002 by MPR 3 OAtoHold place on Mfg. Deficiency (2010)
Note 1: The status shown reflects MPRrs understanding based on information received from or provided to the plant.
Note 2: installed items that have passed installation and surveillance testing to date have no immediate concerns but could have long-term reliability issues, if the manufacturing deficiency is present.
Impact on Equipment that was Not Supplied by MPR MPR does not have the information needed to evaluate the impact on equipment that was not supplied by MPR. See Section 8.
- 7. Corrective Action Plan Transformers to be Replaced Table 7-1 identifies saturable transformers previously supplied by MPR that will be replaced by MPR and Basler Electric Company. MPR will coordinate with the plant and Basler Electric Company to determine the schedule for replacing the saturable transformers.
Table 7-1. Saturable Transformers (STs) Supplied by MPR that will be Replaced Plant Configuration Basler Part Number Number of STS CNS (2012) Series-Series BE34009-001 3 HNP (2012) Series-Series BE34093-001 15 Transformers Requiring Verification Table 7-2 identifies saturable transformers previously supplied by MPR that require inspections and/or tests to ensure they are suitable for the application. MPR recommends that the transformers be sample inspected or tested to determine whether the manufacturing deficiency is present. MPR will coordinate with affected plants to determine the schedule for completing the inspections or tests.
Table 7-2. Saturable Transformers (STs) Supplied by MPR that Require Verification Number of Plant Configuration Basler Part Number STe Robinson (2010) Series-Parallel BE3382-001 4 Farley (2010) Series-Parallel BE1 0618-002 3 HNP (2011) Series-Series BE34093-001 3 CNS - Legacy (2011) Series-Series BE34154-001 2 Industry Notifications The plants identified in the tables above were notified of MPR's recommended actions. MPR has attended industry EDG owners' groups to inform attendees of the failure at CNS. MPR issued an Interim Part 21 Report regarding the CNS failure on January 9, 2013.
Actions to Prevent Recurrence MPR and Basler Electric Company fully understand the induced voltages in the saturable transformer control windings and have developed methods to predict the voltage and confirm them by test. MPR and Basler Electric Company have implemented actions to evaluate control winding voltage, which will ensure the insulation system for future saturable transformers is designed to withstand the maximum expected voltage.
Basler Electric Company is implementing a plan to determine the cause f the manufacturing deficiency and prevent recurrence. The Basler Electric Company planned actions are scheduled I
for completion by April 30, 2013. Compensatory measures are in place to ensure the quality of work in-process.
- 8. Advice Given to Purchasers or Licensees AC voltages are induced in the DC control windings of saturable transformers supplied in Basler Electric Company SBSR EDG excitation systems under normal operating conditions. The magnitude of the induced voltages is dependent on the system design parameters and is significantly higher for series-series designs than for series-parallel designs.
There are numerous Basler Electric Company SBSR type excitation systems in service at multiple plants throughout the nuclear industry. Each system supplied is custom designed for the generator that it is slated to control. Basler Electric Company SBSR excitation systems have demonstrated reliable service for many years.
SBSR excitation systems that are installed, have provided many years of reliable service, and pass routine surveillance tests are not susceptible to the premature (i.e., short lifetime) failures described in this report.
As with many items supplied to a nuclear power plant, saturable transformers have a finite life.
The AC voltages induced in the control windings may impact the life expectancy of the saturable transformer insulation. This factor should be considered in the aging management plan for the plant. However, there are factors other than dielectric stress (e.g., thermal aging) that should also be considered and may be greater contributors to aging than the induced AC voltage.
The operating experience (OE) review conducted as part of this investigation identified a few failures of saturable transformers that occurred during start-up or early in plant operation. These failures occurred in designs with series-series connections. No failures of series-parallel connections were identified. Other than the early failures, only one failure of a series-series saturable transformer was identified. This unit had operated reliably for over thirty years. The OE affirms the advice given above.
- iL. IV Date:' 3/z__/_ __
7 %K. Z/
QA Approval: Michael Dnebrget*, QA Manager Final Approval: (Pan A SŽ Date: 5 (9 /Iao 1.3 Paul S. Damerell, Principal Officer