IR 05000482/2002006
| ML020960002 | |
| Person / Time | |
|---|---|
| Site: | Wolf Creek  |
| Issue date: | 04/05/2002 |
| From: | Graves D NRC/RGN-IV/DRP/RPB-B |
| To: | Maynard O Wolf Creek |
| References | |
| GL-89-013 IR-02-006 | |
| Download: ML020960002 (29) | |
Text
ril 5, 2002
SUBJECT:
NRC SPECIAL INSPECTION REPORT 50-482/02-06
Dear Mr. Maynard:
On March 8, 2002, the NRC completed an inspection at your Wolf Creek Generating Station.
The enclosed report documents the inspection findings which were discussed with Ms. D. Jacobs and other members of your staff on March 8, 2002.
The inspection examined activities associated with the identification of Emergency Diesel Generator A lube oil, intercooler, and jacket water heat exchanger tube degradation on January 4 and 5, 2002, and subsequent identification of tube degradation on Emergency Diesel Generator B on January 7. The inspection focused on the root cause and extent of condition of the event and the corrective actions taken following identification of the degraded heat exchanger tubes. The inspectors reviewed selected procedures, records, and evaluation activities and interviewed plant personnel. As a result of this inspection, the NRC developed a sequence of events, attempted to assess the risk significance of the overall event, and assessed the quality of response of your plant staff and managers. The long-term actions to prevent recurrence will be evaluated separately.
Four unresolved items were identified in this inspection report: (1) Failure to take corrective action for previous examples of emergency diesel generator heat exchanger tube degradation; (2) the work order to perform eddy current testing on the emergency diesel generator heat exchanger tubes did not provide adequate acceptance criterion; (3) personnel failed to promptly identify significantly degraded tubes in the Emergency Diesel Generator A heat exchanger; and (4) Emergency Diesel Generator B may have been inoperable for a significant period of time because of the degraded heat exchanger tubes. As of the end of this inspection period, the risk significance of these issues has yet to be determined. Several degraded diesel generator heat exchanger tubes will be destructively examined for structural integrity and to determine a definite root cause. Pending examination of these tubes, these four issues remain unresolved.
In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Wolf Creek Nuclear Operating Corporation -2-Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/
David N. Graves, Chief Project Branch B Division of Reactor Projects Docket: 50-482 License: NPF-42
Enclosure:
NRC Inspection Report 50-482/02-06
REGION IV==
Docket: 50-482 License: NPF-42 Report: 50-482/02-06 Licensee: Wolf Creek Nuclear Operating Corporation Facility: Wolf Creek Generating Station Location: 1550 Oxen Lane, NE Burlington, Kansas Dates: February 11 through March 8, 2002 Inspectors: D. L. Proulx, Team Leader M. S. Peck, Resident Inspector, Columbia Generating Station Approved By: D. N. Graves, Chief, Project Branch B ATTACHMENTS:
1. Supplemental Information 2. Chronological Sequence of Events, Emergency Diesel Generator Heat Exchanger Tube Degradation 3. Wolf Creek Special Inspection Charter
SUMMARY OF FINDINGS Wolf Creek Generating Station NRC Inspection Report 50-482/02-06 IR 50-482/02-06; on 2/11 - 3/8/2002; Wolf Creek Nuclear Operating Corporation; Wolf Creek Generating Station. Special Inspection Report. Event Followup.
The report covers a special inspection conducted by Region IV inspectors concerning emergency diesel generator heat exchanger tube degradation. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for which the significance determination process does not apply are indicated by No Color or by the severity level of the applicable violation. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described at its Reactor Oversight Process website at http://www.nrc.gov/NRR/OVERSIGHT/index.html.
A. Inspector Identified Findings Cornerstone: Mitigating Systems C TBD. The inspectors identified a violation of 10 CFR Part 50, Appendix B, Criterion XVI, for failure to implement corrective action to prevent recurrence for a significant condition adverse to quality. In 1990 and 1991, the licensee identified that the emergency diesel generator heat exchangers exhibited severe wall thinning and pitting because of de-alloying of the base metals and flow accelerated corrosion. As corrective action, the licensee planned to perform periodic eddy current examination of the heat exchanger tubes to provide early indication of tube degradation. However, this corrective action was not implemented until December 13, 2001, despite several missed opportunities to implement this action. This item is in the corrective action system as Performance Improvement Request 2002-0048. This item is being treated as an unresolved item pending licensee metallurgical and structural analysis of several degraded tubes after which the risk significance of this finding will be determined (Section 4OA3.2).
C TBD. The inspectors identified a violation of 10 CFR Part 50, Appendix B, Criterion V, for failure to include appropriate acceptance criteria in an activity affecting quality. As of December 13, 2001, Work Order WO 01-229167-001, which directed the licensee to perform eddy current testing on Emergency Diesel Generator A heat exchanger tubes, did not include acceptance criteria for maximum allowable heat exchanger tube wall thinning. As a result, the licensee did not document the significant tube degradation identified or take corrective action to plug the degraded tubes until January 4, 2002.
This item is in the corrective action system as Performance Improvement Request 2002-0048. This item is being treated as an unresolved item pending licensee metallurgical and structural analysis of several degraded tubes following which the risk significance of this finding will be determined (Section 4OA3.4).
C TBD. The inspectors identified a violation of 10 CFR Part 50, Appendix B, Criterion XVI, for failure to promptly document and report to the appropriate management a significant condition adverse to quality. Specifically, on December 13, 2001, the licensee performed eddy current examination of the Emergency Diesel Generator A heat
-2-exchanger tubes, and identified significant tube degradation, but failed to document the condition or report it to management until January 4, 2002. The licensee had no acceptance criteria for performance of the testing, and personnel believed that the condition could be corrected during a future outage. This item is in the corrective action system as Performance Improvement Request 2002-0048. This item is being treated as an unresolved item pending licensee metallurgical and structural analysis of several degraded tubes following which the risk significance of this finding will be determined (Section 4OA3.4).
C TBD. Emergency Diesel Generator B may have been inoperable for a significant period of time. During eddy current testing, the licensee identified nine intercooler heat exchanger tubes that exhibited suspected de-alloying, indicating that the structural integrity of the tubes was indeterminate. The licensee bounding calculation determined that the intercooler heat exchanger would be operable with up to three intercooler heat exchanger tubes failing. The licensee subsequently plugged these heat exchanger tubes on January 7, 2002. This item is in the corrective action system as Performance Improvement Request 2002-0048. This item is being treated as an unresolved item pending licensee metallurgical and structural analysis of several degraded tubes following which the risk significance of this finding will be determined (Section 4OA3.5).
B. Licensee Identified Findings None.
TABLE OF CONTENTS PAGE OTHER ACTIVITIES 4OA3 Event Followup 4OA3.1 Sequence of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 4OA3.2 Event Precursors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 4OA3.3 Root Cause Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 4OA3.4 Licensee Response to Tube Degradation . . . . . . . . . . . . . . . . . . . . . . 8 4OA3.5 Common Mode Failure Potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4OA3.6 Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4OA3.7 Generic Implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 4OA6 Exit Meeting Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 ATTACHMENTS Attachment 1 - Supplemental Information Attachment 2 - Chronological Sequence of Events, Emergency Diesel Generator Heat Exchanger Tube Degradation Attachment 3 - Wolf Creek Special Inspection Charter
Report Details Summary of Plant Status The plant operated at essentially 100 percent power for the report period.
4. OTHER ACTIVITIES Cornerstones: Mitigating Systems 4OA3 Event Followup (93812)
.1 Sequence of Events a. Inspection Scope The inspectors developed a complete sequence of events related to the identification and timeliness of actions taken in response to indications of diesel generator heat exchanger degradation. In compiling the sequence of events, the inspectors examined licensee records and interviewed personnel.
Background Wolf Creek on-site standby power was supplied by two emergency diesel generators.
Each emergency diesel generator incorporated three heat exchangers: the intercooler (EKJ03A/B), water jacket cooler (EKJ06A/B) and lube oil cooler (EKJ04A/B). The emergency diesel generator heat exchanger tubes were constructed of admiralty brass (ASME SB111, C70600, commonly know as 90-10 copper-nickel). Station service water continually supplied the tube side of each heat exchanger. Service water supplied each of the three heat exchangers in series with a design flow rate of 1,200 gallons per minute (gpm). 1,200 gpm corresponded to an intercooler flow velocity of 5.9 feet per second (fps) and lube oil cooler/jacket water heat exchanger velocity of 4.5 fps.
Emergency service water was also available to supply each emergency diesel generator heat exchanger following a loss of service water.
Problem Identification On December 13, 2001, the licensee removed Emergency Diesel Generator A from service and performed intercooler, jacket water, and lube oil heat exchanger visual inspection and eddy current nondestructive examination (NDE) testing. The licensee closed the heat exchangers and restored the emergency diesel generator to operable status the following day. The NDE Work Order (WO 01-229167-001) did not include acceptance criteria for maximum allowable heat exchanger tube wall thinning. The eddy current testing technician applied a generic value of 30 percent remaining wall thickness as an acceptance criteria based on previous experience. The eddy current testing technician identified five intercooler tubes with indications of less than 30 percent remaining wall and three tubes with absolute drift indications (indeterminate flaw indications). The eddy current testing technician recommended to the system engineer that these eight intercooler tubes be plugged. The system engineer initially determined that tube plugging could be delayed until the next refueling outage, scheduled to begin March 26, 2002, based on the indication that no tubes had completely failed.
-2-The system engineer subsequently requested assistance from design engineering personnel for further evaluation of the Emergency Diesel Generator A heat exchanger NDE results. On January 3, 2002, design engineering provided the system engineer with Calculation KJ-MW-008, Revision 0, Diesel Generator Intercooler Heat Exchanger and Lube Oil Exchanger Minimum Tube Wall Thickness. Calculation KJ-MW-008 established a maximum 55 percent through-wall tube thinning for the intercooler heat exchangers based on structural integrity requirements established by ASME Section III, 1977, Class 3. The system engineer reviewed the NDE data and identified 12 intercooler tubes, one water jacket tube, and two lube oil cooler tubes degraded based on the 55 percent through-wall criteria and uncertainty associated with the identified absolute drift indications. On January 4, 2002, the licensee declared Emergency Diesel Generator A inoperable and maintenance personnel plugged the affected heat exchanger tubes. The licensee restored the heat exchangers and declared Emergency Diesel Generator A operable on January 6, 2002.
Following restoration of Emergency Diesel Generator A, the licensee removed Emergency Diesel Generator B from service and completed visual inspection and eddy current testing on all three heat exchangers. NDE technicians identified 21 degraded tubes on the Emergency Diesel Generator B intercooler, with wall pit depths ranging from 21 percent to 98 percent through-wall and nine tubes with suspected de-alloying.
On January 6, 2002, maintenance personnel plugged the 21 degraded tubes and the licensee restored Emergency Diesel Generator B to operable.
On January 6, 2002, the licensee determined that the heat exchanger tube degradation resulted in the potential common mode failure of both emergency diesel generators following a design basis earthquake. The licensee postulated that the degraded heat exchanger tubes no longer met seismic qualifications and that tube failure would prevent the fulfillment of the emergency diesel generator safety function. The licensee also initiated an Incident Investigation Team (IIT) to investigate the condition. The licensee IIT focused on three areas: (1) why the nonconforming condition (tube degradation)
existed from December 13, 2001, to January 2, 2002, before actions were taken to correct the condition; (2) why the heat exchanger preventive maintenance program did not identify the degraded condition; and (3) the root cause of the material degradation.
A more detailed sequence of events is contained in Attachment 2 of this inspection report.
b. Findings No findings of significance were identified.
.2 Event Precursors a. Inspection Scope The inspector evaluated event precursors to the degraded emergency diesel generator heat exchanger tubes identified on December 13, 2001. The following information was
-3-assessed for its contribution to the event. The inspectors assessed the effectiveness of the licensees corrective action with respect to emergency diesel generator heat exchanger tube degradation.
Background Emergency diesel generator heat exchanger tube degradation occurred previously at Wolf Creek. Tube wall thinning began early in plant life due to a combination of dezincification corrosion (de-alloying) and erosion/corrosion caused by high flow velocities. All of the emergency diesel generator heat exchanger tubes were previously replaced due to corrosion between 1987 and 1991 (Table 1).
Table 1 Previous Emergency Diesel Generator Tube Replacement EKJ03A (Intercooler) Refueling Outage (RF) -4 (1990)
EKJ04A (Lube Oil Cooler) RF-5 (1991)
EKJ06A (Water Jacket Cooler) RF-2 (1987)
RF-5 (1991)
EKJ03B (Intercooler) RF-4 (1990)
EKJ04B (Lube Oil Cooler) RF-5 (1991)
EKJ06B (Water Jacket Cooler) RF-2 (1987)
RF-5 (1991)
The licensee procured a metallurgical analysis of Emergency Diesel Generator A Intercooler tubes following replacement in 1990. The analysis (Metlab Testing Services, MLTS 90-2213, April 20, 1990) concluded that tube failure was the result of a combination of de-alloying and severe erosion/flow assisted pitting.
De-Alloying The 1990 metallurgical analysis concluded that de-alloying or dezincification of the tubes occurred during the long periods of stagnant flow conditions prior to plant startup.
Brasses containing more than 15 percent zinc are susceptible to dezincification. During de-alloying, the brass was selectively leached out, leaving a relatively porous and weak layer of copper and copper oxide. Conditions that favor dezincification included stagnant water flow.
Flow Induced Erosion The intercooler tubes were designed for a 5.9 fps flow velocity. The metallurgical analyses concluded tube wall thinning was the result of high linear tube flow velocities aggravated by small suspended (abrasion) particles. Plant personnel reported
-4-intercooler flows of 5 - 8 fps with seasonal flow rates of up to 12 fps. High flow velocities contributed to removal of protective metal oxides on the tube surfaces, which lead to continued corrosion/erosion, pitting, and wall thinning.
Past Corrective Actions Prior to 1990, the licensee throttled back service water flow to the component cooling water heat exchangers in the winter months. This action increased service water flow to the emergency diesel generator heat exchangers, resulting in flow velocities up to 12 fps. The licensee stopped the practice of throttling back service water to the component cooling water heat exchangers and planned to continue with heat exchanger eddy current testing to monitor the corrective action effectiveness.
Degraded emergency diesel generator heat exchanger tubes resulted from less than adequate implementation of the licensees preventive maintenance program. The licensee previously identified severe heat exchanger tube degradation leading to re-tubing of all six emergency diesel generator heat exchangers. In 1991, the licensee incorporated periodic eddy current testing into the preventive maintenance program to ensure continued structural integrity of the heat exchanger tubing. The licensee tied this action to interim actions in response to NRC Generic Letter (GL) 89-13, Service Water System Problems Affecting Safety-Related Equipment. The licensee again directed that eddy current testing be included in the preventive maintenance program in 1993 (TS 93-0143, Maintenance and Testing Frequencies on Emergency Service Water Heat Exchanger Tubes - GL 89-13, March 18, 1993). However, the last periodic eddy current testing of emergency diesel generator heat exchangers was performed in 1991. The cause of the program deficiencies was identified as organizational interface ineffectiveness.
b. Findings On April 20, 1990, the licensee identified severe wall thinning of the emergency diesel generator heat exchanger tubes requiring replacement, but did not implement corrective actions that recommended periodic eddy current evaluation of the emergency diesel generator heat exchanger tubes to ensure continued structural integrity of the tubes.
The licensee missed opportunities in 1993 and in 1997 to initiate periodic eddy current testing as well. As a result, as of December 13, 2001, the licensee did not perform eddy current examination of the emergency diesel generator heat exchanger tubes until several tubes exhibited severe degradation. The inspectors concluded that the failure to implement previously identified corrective actions (preventive maintenance tasks for eddy current testing on the emergency diesel generator heat exchanger tubes) was a violation of 10 CFR Part 50, Appendix B, Criterion XVI. However, until the licensee performs destructive testing of the degraded emergency diesel generator heat exchanger tubes, the risk significance of this issue cannot be determined. Therefore, the failure to implement corrective action to prevent recurrence of degraded emergency diesel generator heat exchanger tubing is being treated as an unresolved item (URI)
(50-482/2002006-01).
-5-
.3 Root Cause Evaluation a. Inspection Scope The inspectors reviewed the licensees root cause determination for independence, completeness, and accuracy. The inspectors performed independent review of the apparent cause using the IIT 02-001 as a reference.
Background Tube Degradation Previous instances of degraded emergency diesel generator heat exchanger tubes resulted from less than adequate implementation of the licensees preventive maintenance program. The licensee previously identified severe heat exchanger tube degradation leading to the degradation and re-tubing of all six emergency diesel generator heat exchangers. In 1991, the licensee incorporated periodic eddy current testing into the Visual Inspection Program to ensure continued structural integrity of the heat exchanger tubing. The licensee tied this visual inspection to interim actions in response to NRC GL 89-13, Service Water System Problems Affecting Safety-Related Equipment. The licensee again included eddy current testing in the preventive maintenance program in 1993 (Technical Specification 93-0143, Maintenance and Testing Frequencies on Emergency Service Water Heat Exchanger Tubes - GL 89-13, March 18, 1993). However, the last periodic eddy current testing of emergency diesel generator heat exchangers was performed in 1991. The cause of the inspection deficiencies was organizational interface ineffectiveness. The licensee identified that, in 1990, the degradation of the emergency diesel generator heat exchanger tubes was caused by de-alloying -- a materials degradation mechanism in which constituent metals of an alloy separate and flake off. In the case of admiralty brass, as found in the licensees emergency diesel generator heat exchanger tubes, de-alloying resulted in separation of the zinc and copper and significant weakening of the structural integrity of the emergency diesel generator heat exchanger tubes in April 1990.
Because of the past history of the emergency diesel generator heat exchangers, the licensee suspected that de-alloying of the tubes was the apparent cause of the emergency diesel generator heat exchanger tube degradation identified December 13, 2001.
High Heat Exchanger Flow Rates The emergency diesel generator heat exchangers were constructed of 3/4-inch diameter (0.049-inch wall thickness) admiralty brass tubing (ASME SB111, C44300). The vendor recommended a maximum 6.0 fps water flow velocity for admiralty brass tubes to avoid corrosion/erosion (Uhligs Corrosion Handbook, Second Edition, 2000). The licensee recorded Emergency Diesel Generator Heat Exchanger A service water flows ranging from 1,120 gpm to 1,930 gpm several times a year between 1990 and 2001. These volumetric flow rates corresponded to intercooler tube velocities from 5.8 fps to 10 fps
-6-(Table 2). The licensee generally operated the intercooler with flow velocities greater 7.8 fps between 1995 and 2001. The flow velocities for the Heat Exchanger B intercooler were similar.
Table 2 Emergency Diesel Generator A Intercooler Service Water Flow Date Measured Flow (gpm) Intercooler Vel (fps) Percent >
Design Vel 8/90 1,510 7.8 131%
3/90 1,134 5.9 98%
12/91 1,300 6.8 113%
2/92 1,200 6.2 104%
5/92 1,450 7.5 126%
11/92 1,195 6.2 103%
2/93 1,265 6.6 110%
5/93 1.420 7.4 123%
7/93 1,450 7.5 126%
10/93 1,420 7.4 123%
1/95 1,930 10.0 167%
4/95 1,735 9.0 150%
3/96 1,528 7.9 132%
8/96 1,555 8.1 135%
12/96 1,465 7.6 127%
5/97 1,640 8.5 142%
1/98 1,585 8.2 137%
4/98 1,575 8.2 136%
8/98 1,575 8.2 136%
11/98 1,662 8.6 144%
2/99 1,565 8.1 136%
6/99 1,435 7.5 124%
-7-10/99 1,447 7.5 125%
11/99 1,434 7.5 124%
2/00 1,530 8.0 133%
7/00 1,530 8.0 133%
5/01 1,503 7.8 130%
9/01 1,508 7.8 131%
The corrosion resistance of the admiralty brass tubing depended both on the inherent nobility of copper and protection by a film of corrosion product. The effect of high flow velocities on the tube corrosion layer depended on the amount of entrained air and suspended abrasive materials. These conditions can result in erosion and impingement leading to film breakdown and accelerated corrosion. The licensee concluded that the high flow conditions could have contributed to the erosion of the tubes.
The licensees eddy current testing identified absolute drift indications. Absolute drift indication was an indication of increased electrical impedance of the tube wall that extended along the large axial distance. A possible cause of the impedance change was general tube thinning caused by a loss of material from the heat exchanger wall.
Absolute drift indication was an indication of the remaining wall rather than the configuration of the tube. Determination of the wall thinning magnitude required knowledge of the nature of the defect, which caused the absolute drift indication and a calibrated standard suitable for the defect type. The licensee did not have a standard available to determine wall thickness at the site of absolute drift indications. This along with other unknown factors resulted in difficulty in determining if enough tube wall thickness was left to meet design requirements.
b. Findings The inspectors noted that Hardware Failure Analysis Report 01624-90, Emergency Diesel Generator Heat Exchanger Tube Degradation, of April 1990, determined that a significant contributing cause of the de-alloying exhibited in 1990 was the long period of time in which the heat exchanger spent in a stagnant condition prior to plant operation.
The inspectors noted that the emergency diesel generator heat exchangers had significant flow through them with very little time under no flow conditions during the time period of 1990 to 2001. Therefore, the inspectors concluded that erosion-corrosion, due to high flow across the heat exchanger tubes was also a likely cause of the emergency diesel generator heat exchanger tube degradation. Following discussions with the inspectors, the licensee received consultation with a materials engineer, who stated that either de-alloying or erosion-corrosion was the likely cause of the heat exchanger degradation. As of the end of this inspection, the licensee planned to remove the suspect tubes from the emergency diesel generator heat exchangers and subject them to laboratory analysis for final root cause determination. The inspectors will review the final root cause in a future inspection (see URI 50-482/2002006-04).
-8-
.4 Licensee Response to Tube Degradation a. Inspection Scope The inspectors reviewed IIT 02-001 and Performance Improvement Request 2002-0048 to evaluate the adequacy of licensee response to indications of emergency diesel generator heat exchanger tube degradation. The inspectors examined licensee records and interviewed applicable personnel.
b. Findings On December 13, 2001, the licensee performed intercooler, jacket water, and lube oil heat exchanger visual inspection and eddy current testing on Emergency Diesel Generator A. WO 01-229167-001 did not initially include acceptance criteria for maximum allowable heat exchanger tube wall thinning. The eddy current testing technician applied a generic value of 30 percent remaining wall thickness acceptance criteria based on previous experience. This acceptance criteria was not based on an engineering calculation or code requirement. The inspectors noted that the failure to have definitive acceptance criterion in the WO as part of the planning process led to a significant delay in entering the issue of the degraded tubes into the corrective action system. The failure to have acceptance criteria in instructions or procedures for an activity affecting quality is a violation of 10 CFR Part 50, Appendix B, Criterion V.
However, until the licensee performs destructive testing of the degraded emergency diesel generator heat exchanger tubes, the risk significance of this finding cannot be determined. Therefore, the failure to have acceptance criterion for eddy current testing of the emergency diesel generator heat exchanger tubes in Work Order WO 01-229167-001 is being treated as a URI (50-482/2002006-02).
The eddy current testing technician identified five intercooler tubes with indications of less than 30 percent remaining wall and three tubes with absolute drift indications. The eddy current testing technician recommended to the system engineer that these eight intercooler tubes be plugged. The system engineer initially determined that tube plugging could be delayed until the next refueling outage, scheduled to begin March 26, 2002, based on the indication that no tubes had completely failed. However, the system engineer did not solicit input from design engineering or other senior plant staff as to the structural integrity or operability of Emergency Diesel Generator A, nor did the system engineer or nondestructive examination technicians inform the operators (shift manager). Therefore, the licensee failed to enter the degraded heat exchanger tubes of Emergency Diesel Generator A into the corrective action system and did not take action to address the potential for the condition to exist on Emergency Diesel Generator B.
The failure to promptly identify a significant condition adverse to quality on December 13, 2001, is a violation of 10 CFR Part 50, Appendix B, Criterion XVI. This condition existed from December 13 until January 4, 2001, when Performance Improvement Request 2002-0048 was initiated. However, until the licensee performs destructive testing of the degraded emergency diesel generator heat exchanger tubes, the risk significance of this issue cannot be determined. Therefore, the failure to document the condition and enter it into the corrective action system is being treated as a URI (50-482/2002006-03).
-9-The system engineer continued to evaluate the data internally until January 4, 2002, when he discussed the results of the December 13, 2001, eddy current testing with design engineering. On January 3, 2002, Design Engineering provided the system engineer with Calculation KJ-MW-008, Revision 0, Diesel Generator Intercooler heat Exchanger and Lube Oil Exchanger Minimum Tube Wall Thickness. Calculation KJ-MW-008 established a maximum 55 percent through-wall tube thinning criteria for the intercooler heat exchangers based on structural integrity requirements established by ASME Section III, 1977, Class 3. The system engineer reviewed the NDE data and identified 12 intercooler tubes, one water jacket tube, and two lube oil cooler tubes that were degraded based on the 55 percent through-wall criteria and uncertainty associated with the identified absolute drift indications. On January 4, 2002, the licensee declared Emergency Diesel Generator A inoperable and maintenance personnel plugged the affected heat exchanger tubes. The licensee restored the heat exchangers and declared Emergency Diesel Generator A operable on January 6, 2002.
Following restoration of Emergency Diesel Generator A, the licensee removed Emergency Diesel Generator B from service and completed visual inspection and eddy current testing on all three heat exchangers. NDE technicians identified 21 degraded tubes on the Emergency Diesel Generator B intercooler, with wall pit depths ranging from 21 percent to 98 percent through-wall and 9 tubes with suspected de-alloying. On January 6, 2002, maintenance personnel plugged the 21 degraded tubes and the licensee restored Emergency Diesel Generator B to operable status.
One January 6, 2002, the licensee determined that the heat exchanger tube degradation resulted in the potential common mode failure of both emergency diesel generators following a design basis earthquake. The licensee postulated that the degraded heat exchanger tubes no longer met seismic qualifications and tube failure would prevent the fulfillment of the emergency diesel generator safety function. The licensee also initiated an IIT to investigate the condition. The licensee IIT focused on three areas: (1) why the nonconforming condition (tube degradation) existed from December 13, 2001, to January 2, 2002, before actions were taken to correct the condition; (2) why the heat exchanger preventative maintenance program did not identify the degraded condition; and (3) the root cause of the material degradation.
.5 Common Mode Failure Potential a. Inspection Scope The inspectors reviewed the licensees evaluation of the condition with respect to past operability of the emergency diesel generators and the potential for common mode failure.
b. Findings The licensees initial evaluation of the degraded emergency diesel generator heat exchanger tubes was that heat exchanger structural integrity could not be guaranteed for both emergency diesel generators following a seismic event given the uncertainty of the absolute drift indications, the wall thinning, and the pitting observed. Thus, on
-10-January 6, 2002, the licensee reported the condition to the NRC Operations Center based on both emergency diesel generators being declared inoperable due to a common mode failure.
As part of the charter of IIT 02-001, the licensee performed an additional analysis concerning the past operability of the emergency diesel generators. The licensee analysis concluded that both emergency diesel generators were fully available to perform their safety function and no potential for common mode failure existed prior to plugging the degraded heat exchanger tubes. The licensees conclusion was based on the following:
Postulated Intercooler Tube Failure Calculation KJ-M-109 (Revision 0) demonstrated continued emergency diesel generator function with the failure of up to three intercooler tubes. The inspectors noted that this calculation did not bound the potential significance of the condition in that nine heat exchanger tubes were listed as suspected de-alloying.
Postulated Seismic Event The licensees operability evaluation demonstrated that seismic induced stress (up to 4g acceleration) on the degraded heat exchanger tubes was bounded by flow dynamic stress and vibrations. Calculation KJ-MW-004, Revision 0, Diesel Jacket Heat Exchanger Tube Pressure, Revision 0, March 17, 1990, demonstrated heat exchanger tube integrity with up to 22 mils remaining wall thickness when exposed to a 200 psig dynamic pressure applied to the outside of the tube. The 200 psig case represented the most limiting tube dynamic loading following a loss of off-site power and emergency service water pump start. The heat exchangers were highly damped against seismic affects due to the heat exchanger shell being filled with water, immersing the tubes and providing a fluid coupling along the shell length. The tubes were supported at frequent intervals by loose fitting plates, which provided nonlinear support, limiting resonance condition. Also, the bundles were designed to prevent flow induced vibrations.
Suspected De-Alloying The licensee identified nine tubes that were characterized as suspected de-alloying on the Emergency Diesel Generator B intercooler. Suspected de-alloying was indicative of exclusions in the alloy from the depletion or removal of the zinc from the tube material leaving a porous layer of copper inside the tube. The suspected de-alloying condition was indicated on the eddy current testing as poor signal-to-noise ratio along the length of the tube. The eddy current testing technician was unable to determine wall thickness for the tubes with suspected de-alloying. Failure of the nine tubes with suspected de-alloying indications on the Emergency Diesel Generator B intercooler was outside of the bounding conditions established in the licensees operability evaluation. The inspectors concluded that Emergency Diesel Generator B past operability was indeterminate pending a more detailed evaluation of the material condition of the nine tubes with the suspected de-alloying condition.
-11-Failure of Lube Oil Cooler The failure of Heat Exchanger EKJ04 (lube oil) tubes would result in the rapid loss of emergency diesel generator safety function. The licensee postulated that emergency diesel generator failure from increased engine crankcase hydrodynamic effects in 10 minutes following a lube oil cooler tube rupture. However, the licensee only identified one degraded lube oil cooler (EKJ04A) tube. This tube was degraded due to identification of an absolute drift indication. The eddy current testing technician later determined the absolute drift indication was indicative of a 17 percent through-wall pit.
The 17 percent through-wall condition was within acceptable tube thinning per ASME Section III 74, Summer 76 Addenda.
The inspectors concluded that the licensee adequately demonstrated current operability of both emergency diesel generators with the degraded heat exchanger tubes plugged and the past operability of Emergency Diesel Generator A. However, the inspectors noted the licensees past operability evaluation of Emergency Diesel Generator B was inconclusive due to the high number of intercooler tubes designated as potentially degraded by suspected de-alloying. The licensee plans to remove the tubes indicating suspected de-alloying during Refueling Outage 12 and perform a metallurgical analysis to determine the extent of degradation and corrosion mechanism. Until the licensee performs this analysis, the inspectors could not assess the potential risk significance of this event. The potential for a significant period of time in which Emergency Diesel Generator B was inoperable is being treated as a URI (50-482/2002006-04).
.6 Corrective Actions a. Inspection Scope The inspectors reviewed the licensees proposed corrective actions as listed in IIT Report 02-001 (listed below), to determine if the licensee adequately addressed the hardware and performance issues revealed in the event.
Immediate actions that were taken as a result of declaring Emergency Diesel Generator A inoperable at 1:47 p.m. on January 4, 2002, included:
C Emergency Diesel Generator A heat exchangers were removed from service at 1:47 p.m. on January 4, 2002, and specific degraded tubes were plugged.
Emergency Diesel Generator A was declared operable at 11:34 a.m. on January 5, 2002.
C A management focus meeting was held on January 4, 2002, to discuss the potential of a common mode failure and it was decided that the Emergency Diesel Generator B heat exchangers would be examined after Emergency Diesel Generator A was restored to an operable condition.
C Engineering commenced an evaluation of operability of Emergency Diesel Generator B heat exchangers on January 4, 2002.
-12-C Emergency Diesel Generator B heat exchangers were removed from service on January 5, 2002, eddy current tested, and specific degraded tubes were plugged. Emergency Diesel Generator B was declared operable on January 6, 2002.
C Notification of potential common mode failure of the emergency diesel generators was made to the NRC Operations Center in EN#38610 on January 6, 2002.
C The IIT was chartered and briefed on January 9, 2002.
C On January 10, 2002 a Site Bulletin was issued based on the event of January 4, 2002, to reinforce Wolf Creeks corrective action components, particularly in regard to hardware nonconformances. The input of this communication was to:
- Review the requirements of the Corrective Action program.
- Identify these requirements.
- Note that the responsibilities of licensee employees was to ensure these requirements were met.
C Scheduling was tasked with conducting an independent review of the upcoming work week schedules.
C All groups were tasked with reviewing their scheduled work activities based on the allowed outage time and plan-of-the-day to reassure that contingencies were planned where necessary and that they were adequate.
C A site-wide standdown was performed on January 11, 2002. The purpose of the stand down was to conduct facilitated group discussions of the definition of nonconforming conditions and expectations for reporting a nonconformance using the corrective action process, which included performance improvement requests and work requests.
To address the root cause of the untimely identification and resolution of nonconforming conditions, the licensee proposed that:
C Responsible managers implement the accountability process for the appropriate system engineering personnel.
C Human Performance initiate a lessons-learned bulletin on this event to station personnel, to address human performance shortcomings in error prevention tool use, decision-making, group-think, and error-likely behaviors.
C Training incorporate lessons learned from this event into the appropriate programs.
-13-C Maintenance revise the Work Order Processing Guide.
C Support Engineering correct work instructions to not allow heat exchanger restoration until eddy test results are analyzed and tube nonconformances are dispositioned.
The licensee also proposed to revise several procedures associated with the corrective action process.
The licensee also recommended:
C Operations enhance procedures to instruct operators to check the lube oil expansion tank overflow to the drain hub on the west wall of the room for early indication of an emergency diesel generator lube oil heat exchanger leak.
C To facilitate more rapid detection of large leaks in lube oil, intercooler, or jacket water heat exchangers, the licensee recommended that procedures be revised to prompt the operators to check jacket water expansion tank and lube oil tank overflows to room drains.
C Training provide operator contingencies in emergency diesel generator lesson plans.
C Engineering consider formation of a structural integrity hit team which could be rapidly mobilized when structural questions arise.
C Engineering define responsibilities for tube integrity condition monitoring of emergency diesel generator heat exchangers.
b. Findings No findings of significance were identified.
.7 Generic Implications a. Inspection Scope The inspectors assessed the generic implications of this event. The inspectors reviewed previous operational event notifications and licensee contacts with other nuclear units. The inspectors reviewed the licensee responses and commitments related to NRC GL 89-13.
The licensee contacted several other nuclear facilities to determined the usage of eddy current testing for emergency diesel generator heat exchangers. Of the 24 sites contacted, 12 had performed eddy current inspection of emergency diesel generator heat exchanger tubes as part of their preventive maintenance on the units. No NRC or
-14-industry operational events discussed eddy current inspection of emergency diesel generator heat exchanger tubes or other nondestructive examination to maintain structural integrity of the tubes.
b. Findings No findings of significance were identified.
4OAB Meetings Exit Meeting Summary The inspectors presented the inspection results to D. Jacobs and other members of licensee management on March 8, 2002.
The inspectors asked the licensee whether or not any materials examined during the inspection should be considered proprietary. No proprietary material was identified.
ATTACHMENT 1 SUPPLEMENTAL INFORMATION PARTIAL LIST OF PERSONS CONTACTED Licensee K. A. Harris, Manager, Regulatory Affairs M. W. Hicks, Manager, Operations D. Jacobs, Plant Manager J. W. Johnson, Manager, Resource Protection O. L. Maynard, President and Chief Executive Officer B. T. McKinney, Vice President Operations R. Muench, Vice President Technical Services NRC D. Graves, Chief, Project Branch B F. Brush, Senior Resident Inspector ITEMS OPENED, CLOSED, AND DISCUSSED Opened 50-482/2002006-01 URI Failure to implement corrective action for past indications of emergency diesel generator heat exchanger tube degradation (Section 4OA3.2)
50-482/2002006-02 URI Failure to provide acceptance criteria for eddy current testing (Section 4OA3.4)
50-482/2002006-03 URI Failure to promptly identify significantly degraded emergency diesel generator heat exchanger tubes (Section 4OA3.4)
50-482/2002006-04 URI Evaluate past operability of Emergency Diesel Generator B following analysis of heat exchanger tubes (Section 4OA3.5)
Previous Items Closed None
-2-LIST OF ACRONYMS USED CFR Code of Federal Regulations fps feet per second GL generic letter gpm gallons per minute IIT Incident Investigation Team NDE nondestructive examination NRC Nuclear Regulatory Commission RF Refueling Outage URI unresolved item WO work order
ATTACHMENT 2 CHRONOLOGICAL SEQUENCE OF EVENTS Date Event 7/85 Visual Inspection of EKJ03A (Intercooler heat exchanger)
11/85 Visual inspection of EKJ03A, EKJ04A/B (lube oil heat exchanger), &
EKJ06A/B (jacket water heat exchanger).
2/86 Implemented reduced service water flow to component cooling water heat exchangers during winter months (EER-85-EG-11). Modification resulted in flows up to 2,400 gpm to emergency diesel generator heat exchangers.
RF1 Eddy current testing of Heat Exchangers EKJ03A, EKJ04A & EKJ06A 10/86 - 12/86 concluded 40 percent, 20 percent & 30 percent (respectively) tube wall losses RF 2 Eddy current testing performed. Heat Exchanger EKJ06A/B tube bundles 9/87- 1/88 replaced due to erosion.
RF 3 Visual inspection of Heat Exchangers EKJ03A,EKJ 04A/B, & EKJ06A/B.
10/88 - 1/89 1/90 GL 89-13, Licensee committed to perform periodic testing to verify emergency diesel generator heat exchanger capabilities starting with RF 5 RF 4 Licensee established 55 percent as maximum heat exchanger tube wall 3/90 - 5/90 thinning criteria (EER 90-KJ-04) based on ASME Section III.
Eddy current testing of all emergency diesel generator heat exchangers.
Retubed Heat Exchanger EKJ03A/B due to severe erosion and though-wall pitting.
Outside material testing laboratory analyzed failed tubes from Heat Exchanger EKJ03A. Laboratory concluded two failure mechanisms:
(1) Dezincification - Occurred during the long periods of stagnant conditions prior to plant startup.
(2) Accelerated Erosion - Due to significantly high service water flow rates during the winter months (high flows contributed to removal of the protective metal oxide surfaces).
2/91 Licensee determined emergency diesel generator heat exchanger performance testing was not practical. Licensee revised GL 89-13 to perform periodic visual inspections.
-2-10/91 Licensee lowered maximum number of heat exchanger plugged tube limit from 26 to 13 based on an increase in bounding ultimate heat sink temperature from 95E to 96EF.
RF 5 Licensee identified four blocked tubes (foreign material) on Heat 9/91 - 1/92 Exchanger EKJ03A. Licensee plugged three tubes.
Tubes replaced on Heat Exchangers EKJ04A/B and EKJ06A/B.
1/93 Eddy current testing dropped from emergency diesel generator heat exchanger inspection/preventive maintenance program.
3/93 Plant engineering recommended that eddy current testing on emergency diesel generator heat exchangers be continued (Letter TS 93-0143, referenced GL 89-13 commitment).
RF 6 Visual inspection performed on Heat Exchangers EKJ03A/B, EKJ04A/B &
3/93 - 5/93 EKJ06A/B. Minor tube thinning - no eddy current testing performed.
2/94 Licensee submitted updated response to GL 89-13.
RF 7 Visual inspection performed on Heat Exchangers EKJ03A/B, EKJ04A/B &
9/94 - 11/94 EKJ06A/B. Minor tube thinning, few clams and mineral deposits - no eddy current testing performed.
RF 8 Visual inspection performed on Heat Exchangers EKJ03A/B, EKJ04A/B &
2/96 - 4/96 EKJ06A/B. Minor tube thinning and pitting - no eddy current testing performed.
RF 9 Visual inspection and boroscope performed on Heat Exchangers EKJ03A/B, 10/97 - 12/97 EKJ04A/B & EKJ06A/B. Minor tube thinning - eddy current testing performed only on Heat Exchanger EKJ06B (no supporting documentation why eddy current testing performed).
3/6/98 Quality control personnel asked heat exchanger engineer (e-mail) why eddy current testing not performed per recommendations in TS 93-0143. Heat exchanger engineer did not respond.
12/98 - 2/99 Visual inspection and boroscope performed on Heat Exchangers EKJ03A/B, EKJ04A/B & EKJ06A/B. Tube thinning and pitting identified - licensee repaired EKJ03B.
11/17/99 Letter ET 99-0042 to NRC updating GL 89-13 commitment - heat exchanger inspection and cleaning would be performed as warranted by trending.
3/00 Licensee developed Heat exchanger Program Procedure (AI 23L-003)
implementing GL 89-13 commitments.
-3-6/00 - 8/00 Visual inspection and boroscope performed on Heat Exchangers EKJ03A/B, EKJ04A/B & EKJ06A/B. Pitting and erosion identified in 11 Heat Exchanger EKJ03A tubes and tube thinning in Heat Exchangers EKJ03B & EKJ04B.
9/12/00 The licensee initiated work orders to perform eddy current testing of emergency diesel generator heat exchanger tubes during the next scheduled maintenance window. Eddy current testing was to collect benchmark data.
12/7/01 Eddy current testing technician requested tube thinning acceptance criteria from the system engineer. Maintenance testing acceptance criteria not provided.
12/10/01 Eddy current testing technician again requested tube thinning acceptance criteria from the system engineer. Maintenance testing acceptance criteria was not provided.
12/12/01 Eddy current testing technician requested tube thinning acceptance criteria from the system engineer. Maintenance testing acceptance criteria was not provided.
12/13/01 Emergency Diesel Generator A declared inoperable for eddy current testing and heat exchanger visual inspection.
Inspection identified large pitting/flaws within the tubing.
12/13/01 Eddy current testing technician requested tube thinning acceptance criteria from the system engineer. Maintenance testing acceptance criteria was not provided.
Eddy current testing technician recommended tube plugging to the system engineer based on a generic value of 70 percent through-wall acceptance criteria. Analysis indicated five Heat Exchanger EKJ03A tubes measured 70 percent or greater through-wall and three tubes contained absolute drift indications.
12/14/01 The heat exchangers were restored and Emergency Diesel Generator A was declared operable.
Analysis for Heat Exchanger EKJ04A indicated one absolute drift indication.
Analysis for Heat Exchanger EKJ06A indicated one tube with greater than 70 percent through-wall. Degraded heat exchanger tubes not recognized as a nonconforming condition by plant staff.
12/24/01 - System engineer was on vacation.
1/01/02
-4-1/03/02 System and design engineering personnel agreed 55 percent through-wall acceptance criteria should be applied to the heat exchanger eddy current testing results.
1/04/02 Operators notified of degraded heat exchanger tubes on Emergency Diesel Generator A. Emergency Diesel Generator A declared inoperable at 1:47 p.m.
Licensee identified the degraded heat exchanger condition into the corrective action program at 4:30 p.m.
1/05/02 Emergency Diesel Generator A heat exchanger tubes plugged (12 tubes in Heat Exchanger EJK03A, 1 tube in Heat Exchanger EKJ04A, and 2 tubes in Heat Exchanger EKJ 06A) and the heat exchangers were restored.
Emergency Diesel Generator A declared was declared operable at 11:34 a.m.
Emergency Diesel Generator B was declared inoperable at 1 p.m. for eddy current testing.
1/06/02 Several Emergency Diesel Generator B heat exchanger tubes were plugged (21 tubes in Heat Exchanger EJK03B) and Emergency Diesel Generator B was restored to operable at 11:34 a.m.
The licensee reported to the NRC Operations center per 10 CFR 50.72 the potential for emergency diesel generator common mode failure associated with degraded heat exchanger tubes during a seismic event.
Licensee established an Incident Investigation Team to review the events associated with emergency diesel generator heat exchanger issues.
1/10/02 The licensee reinforced the corrective action program reporting expectations by issuing a Site Bulletin.
1/11/02 The licensee held a Site Wide Stand Down to facilitate group discussions of the definition and reporting of nonconforming conditions.
ATTACHMENT 3 WOLF CREEK SPECIAL INSPECTION CHARTER MEMORANDUM FOR: David L Proulx, Senior Resident Inspector, Diablo Canyon Power Plant FROM: Ken E. Brockman, Director, Division of Reactor Projects SUBJECT: SPECIAL INSPECTION TEAM AT WOLF CREEK In response to our initial evaluation of the impact of the Emergency Diesel Generators A and B lube oil, intercooler, and jacket water heat exchanger tube degradation identified by eddy current testing on January 4 and 5 2002, and subsequent identification of tube degradation on the Train B diesel generator on January 7, a Special Inspection Team is being chartered. You are hereby designated as the Special Inspection Team leader.
A. Basis On December 13, 2001, the licensee conducted eddy current testing of the Emergency Diesel Generator A lube oil, intercooler, and jacket water heat exchanger tubes. The licensee completed analysis of these data on January 4, 2002 and determined that the Train A diesel generator may have been inoperable during a seismic event. To determine extent of condition, on January 5, 2002, the licensee performed eddy current testing of the train B diesel generator heat exchanger tubes and found similar degradation. Because of the potential for common mode failure of both safety-related diesel generators due to a seismic event, the licensee reported this condition to the NRC Operations Center as required by 10 CFR 50.72. The initial conditional core damage probability for this event is between 3 E-5 and 3 E-6.
A Special Inspection Team will be dispatched to better understand the cause of the subject diesel generator heat exchanger degradation and licensee actions leading up to and including the event. The team is also tasked with gaining a better understanding of the licensees common mode failure analysis as related to their root cause(s). The team is expected to perform data gathering and fact-finding in order to address the following items.
B. Inspection Scope 1. Develop a complete sequence of events related to the identification and timeliness of actions taken in response to indications of diesel generator heat exchanger degradation.
2. Evaluate potential precursors to the event, including Performance Improvement Requests and completed surveillances to evaluate previous indications of a problem.
3. Review the licensee's root cause determination for independence, completeness, and accuracy including risk analysis of the event.
-2-4. Review the overall adequacy of the licensees response to indications of diesel generator heat exchanger tube degradation, including immediate and long-term corrective actions. Evaluate timeliness of eddy current testing analysis for Emergency Diesel Generator A and performance of eddy current testing for Emergency Diesel Generator B.
5. Evaluate and determine the potential for common mode failure.
6. Identify the licensees corrective actions to address the root cause of the condition.
7. Review and assess the corrective actions proposed by the licensee.
8. Review the event for generic implications.
C. Guidance Inspection Procedure 93812, "Special Inspection," provides additional guidance to be used by the Special Inspection Team.
This memorandum designates you as the Special Inspection Team leader. Your duties will be as described in Inspection Procedure 93812. The team composition will be discussed with you directly. During performance of the Special Inspection, designated team members are separated from their normal duties and report directly to you. The team is to emphasize fact-finding in its review of the circumstances surrounding the event, and it is not the responsibility of the team to examine the regulatory process.
Safety concerns identified that are not directly related to the event should be reported to the Region IV office for appropriate action.
The Team will travel to the site on Monday, February 11, 2002. The team will report to the site, conduct an entrance, and begin inspection on Tuesday, January 12.
Tentatively, the inspection should be completed by close of business February 14. A formal exit will be scheduled following completion of the on site inspection. A report documenting the results of the inspection will issued within 45 days of the completion of the inspection. While the team is on site, you will provide daily status briefings to Region IV management.
This Charter may be modified should the team develop significant new information that warrants review. Should you have any questions concerning this Charter, contact Ken Brockman, Director, Division of Reactor Projects at (817) 860-8248.