ML031040196

Letter to Jim Dyer from Howard Bergendahl Re Confirmatory Action Letter Response - Root Cause Analysis Report with Handwritten Notes

ML031040196
Person / Time
Site:	Davis Besse
Issue date:	04/18/2002
From:	Bergendahl H FirstEnergy Corp
To:	Dyer J NRC/RGN-III
References
1-1270, CAL, FOIA/PA-2003-0018
Download: ML031040196 (181)

Text

FENOC FENO :5501

- - -. ~,'-.Dav~is-BesseNuclear Power Station North State Route 2 Oak Harbor. Ohio 43449-9760 fnstfnerayxearOpeiths Cmany Howard W.Bergendahl 419-321-8337 Vice President- Nuclear Fax 419321 8337 Docket Number 50-346 License Number NPF-3 Serial Number 1-1270 April 18, 2002 Mr. J. E. Dyer, Administrator United States Nuclear Regulatory Commission Region III 801 Warrenville Road Lisle, IL 60532-4351

Subject:

Confirmatory Action Letter Response - Root Cause Analysis Report Ladies and Gentlemen:

On March 13, 2002, the Nuclear Regulatory Commission (NRC) issued a Confirmatory Action Letter (CAL) regarding the Reactor Pressure Vessel (RPV) head degradation at the Davis-Besse Nuclear Power Station, Unit I (DBNPS). The CAL requires FirstEnergy Nuclear Operating Company (FENOC) to determine the root causes of the degradation and meet with the NRC to discuss that information. The purpose of this letter is to provide the NRC with our Root Cause Analysis Report in preparation for the meeting with the NRC to discuss this topic.

The Root Cause Analysis Report for the RPV head degradation is enclosed. This Report is consistent with our preliminary Probable Cause Summary Report dated March 22, 2002 (which we previously provided to the NRC Augmented Inspection Team).

However, the enclosed Report provides substantial additional detail and supercedes the preliminary report.

In summary, we have determined that the cause of the degradation was boric acid corrosion resulting from leakage through a crack in a RPV penetration nozzle attributable to primary water stress corrosion cracking. We have further determined that the corrosion had likely occurred over a period of several years, but was not recognized until its discovery by FENOC last month.

The enclosed report discusses management and programmatic issues that led to this situation. Additionally, the report contains recommendations by the root cause

-.... t'.z . .r vls do'cted . ';

w ,-,co~r of InformatiOrl ii;o hot, xe~iapIuionls - l i 2GO3~ al .

EOlk

Docket Number 50-346 license Number NPF-3 Serial Number 1-1270 Page 2 investigation team for corrective actions, preventive actions, and enhancements to address these issues. We are continuing to refine our analysis of these issues, and we are developing our plans for corrective and preventive actions. We will be finalizing these matters in the near term, and we will be providing the NRC with an integrated report that describes these matters and the other matters in the CAL. This integrated report will be submitted to NRC to support restart approval in accordance with the CAL.

If you have any questions or require additional information, please contact Mr. David H.

Lockwood, Manager- Regulatory Affairs, at (419) 321-8450.

Very truly yours, Enclosure and Attachment cc: USNRC Document Control Desk SY Sands, DB-1 NRC/NRR Project Manager D.V. Picket, DB-1 NRC/NRR Backup Project Manager C.S. Thomas, DB-1 Senior Resident Inspector Utility Radiological Safety Board

Docket Number 50-346 License Number NPF-3 Serial Number 1-1270 Attachment Page 1 of 1 COMMITMENT LIST Nuclear The following list identifies those actions committed to by the Davis-Besse submittal in the Power Station (DBNPS) in this document. Any other actions discussedonly for represent intended or planned actions the DBNPS. They are described

- Regulatory information and are not regulatory commitments. Please notify the Manager document or Affairs (419-321-8450) at the DBNPS of any questions regarding this associated regulatory commitments.

COMMIT DUE DATE None

PlrstEnargy, DAVIS4BESSE NUCLEA POWER STATION Root Cause Analysis Report Signiffcant Degradation of the Recrtor Pressure Vessel Head CR 20024891 Dated 34.2002 DATE! 4*16.2002 Prepared by: , 4'o C.A. LD £emLi -Cf=m Reiewed b Wfl A V.WC D.

pfru T4w Davfs-Rcsre SponsorV LiG- t L Hak- D.ct

. b . -4

PURPOSE AND SCOPE OF THE ROOT CAUSE ANALYSIS REPORT Purpose Determine the root and contributing causes for Reactor Pressure Vessel closure head (RPV head) damage experienced at nozzle 3 and minor corrosion at nozzle 2, to support the operability determination for the station's as-found condition and the future repair plan.

Scope Very early in the development of the response to this condition, it became clear that the technical causes behind the cracking of the Control Rod Drive Mechanism (CRDM) nozzles and the ensuing corrosion of the head material would draw much attention and comparison to the previously developed body of knowledge on related topics and conditions. In fact, the possibility of nozzle cracks existing at Davis-Besse was well recognized prior to the condition, but the identified significant damage to the RPV head had not been anticipated.

The unexpected finding of the significant damage at Davis-Besse immediately became a concern, both for the possible extent of condition implications at Davis-Besse and the potential impact to the industry. Therefore, the objective of the Initial Investigative Team was a prompt investigation into the primary cause(s) of the damage. This Root Cause Analysis Report supports this specific objective. These initial findings are expected to invite input from industry experts and scientists resulting in additional study of the evidence, and further research into the topics of CRDM nozzle cracking and boric acid corrosion.

Owing to the urgency in developing useful insight to the plant and the industry, revision 0 of this report has been prepared with the full knowledge that a number of activities are still continuing.

These have been captured as action items in the report, and the results of these could lead to a future revision to the report. It is expected that these activities will provide additional understanding, but should not affect the fundamental conclusions of this report Investigative Team Membership Steve Loehlein, FENOC (Beaver Valley), Team Lead Chuck Ackerman, FENOC (Davis-Besse)

Ted Lang, FENOC (Davis-Besse)

Todd Pleune, FENOC (Davis-Besse)

Neil Morrison, FENOC (Beaver Valley)

William Mugge, FENOC (Davis-Besse)

Joseph Rogers, FENOC (Davis-Besse)

Techiical expertise provided bv:

Dr. Mark Bridavsky, FirstEnergy, Beta Labs - Failure Analysis Expert Stephen Hunt, Dominion Engineering - Corrosion Expert Steve Fyfitch, Frarnatome ANP, Metallurgical Expert Christine King, EPRI, Material Reliability Program Manager Assessment of management aspects/decision making John B. Martin, Corporate Nuclear Review Board E. J.-Galbreath, Senior Representative, Assistance, Institute of Nuclear Power Operations

Table of Contents Title Paee No.

I 1.0 Problem Statement 1.1 Reason for Investigation 1 1.2 Consequences of Event/Condition Investigated 1 1

1.3 Immediate Actions Taken 2

2.0 Event Narrative 2

2.1 Background

3 2.2 Sequence of Events 4

3.0 Data Analysis 3.1 Non-Destructive Examinations of RPV Head and Nozzles 4 3.1.1 Potential EvidentiaryRequest List 5 3.1.2 Locations of Cracks and Corrosionon RPV Head 5 3.1.3 NDE Examinationsof CRDM Nozzles 5 3.1.A Visual Examinationsof RPV Top Head andPenetrations 7 3.2 Cracks, Leaks and Corrosion 8 3.2.1 CRDM Nozzle Cracks and Propagationto Leakage 8 3.2.2 Leakage Rate From CRDM Nozzle Cracks 15 3.2.3 Source of Boric Acid Depositson RPV Head 19 3.2.4 Corrosion ofRPV Top HeadSurface 20 26 3.3 Investigation of Lead Indicators 3.3.1 Timeline 26 3.3.2 Sequence ofRelevant Events 26 3.3.3 CRDM Flangeand RPV HeadInspections during Refueling Outages 26 3.3.4 ContainmentAir CoolerCleaning 31 3.3.5 ContainmentRadiation MonitorRE4597 Observations& FilterPlugging 33 3.3.6 Containment Recirculation Fan/FanFailures 37 3.4 Programs Important to Preventing Problems 37 3.4.1 B&W Owners Group andlndustry CRDM Nozzle Related Initiatives 37 3.4.2 Davis-Besse Boric Acid CorrosionControl Program 43 3.4.3 Davis-Besse InserviceInspection Program 45 3.4A Evaluation of Condition Report Responses 46 47 3.5 Management Issues 3.5.1 PotentialRisk to the RPVHeadSurface 47 3.5.2 Deferralof Service Structure Modificationsto Allow Improved Access 47 3.5.3 Management Involvement/Awareness of Plant Conditionsduring 12RFO 48 3.5.4 Recognizing the Needfor Collective SignificanceReviews 48 3.5.5RPVHeadInspections 48 3.5.6 RPV Head Cleanings.. 48 3.5.7 Restart Readiness 49 3.6 Causal Factors/Conclusions 49

• uut la^f -nat ucalerWIwl"W,.s; - ,

Root Cause Analysis Report

4.0 Experience Review 52 4.1 Davis-Besse Experience 52 4.2 Nuclear Industry Experience 52 4.3 Conclusions 52 5.0 Root Cause Determination 53 5.1 Probable/Root Causes 53 5.2 Contributing Causes 53 6.0 Extent of Condition 56 6.1 Degradation Mechanism Issues 56 6.2 Management Issues 57 7.0 Recommended Corrective Actions 59 7.1 Probable/Root Causes Corrective Actions 59 7.2 Contributing Causes Corrective Actions 60 7.3 Additional Actions Requiring Investigation 62 8.0 References 63 8.1 Davis-Besse References 63 8.2 Vendor References 65 8.3 NRC References 65 8.4 INPO References 66 8.5 Industry References 67 8.6 Other References 67 9.0 Personnel Interviews 69 9.1 Personnel Interviewed 69 9.2 Personnel Consulted 70 10.0 Methodologies Employed 71 TCDIe 0? Lonenls . II Table of Contents a 11 Root Cause Analysis Report

Tables Title Paee No.

1. Nozzle 1 NDE Examination Results 72

2. Nozzle 2 NDE Examination Results 73

3. Nozzle 3 NDE Examination Results 74

4. Nozzle 5 NDE Examination Results 75

5. Nozzle 47 NDE Examination Results 76

6. Comparison of Davis-Besse to Other B&W Design Plants 77

7. Nuclear Industry Experience Review Results 77 Tables s Ill Report Root Cause Analysis Report Tables

• Dit

Figures Title Pane No.

1. Davis-Besse RPV Top of Head Section View 90

2. Davis-Besse RPV Top of Head Plan View 91

3. Davis-Besse CRDM Nozzle General Arrangement 92

4. Boric Acid and Iron Oxide on Vessel Flange at 12RFO 93

5. Corrosion at Nozzle 2 Drawing Side View 94 Sa. Nozzle 2 Corrosion Area Location, Size, and Profile 95

6. Cavity in Reactor Vessel Head between Nozzle 3 and 11 96

7. Locations of Cracks and Corrosion on Davis-Besse RPV Head at 13RFO 97

8. Nozzle 1 Crack Locations and Sizing 98

9. Nozzle 2 Crack Locations and Sizing 99

10. Nozzle 3 Crack Locations and Sizing 100

11. Nozzle 5 Crack Locations and Sizing 101

12. Nozzle 47 Crack Locations and Sizing 102

13. Nozzle 3 Degraded Areas 103

14. Nozzle 3 Clad Thickness Measurements 104

15. Hoop Stresses and Operating Condition Deflections in CRDM Nozzles 2-5 105

16. Location of Leaking Nozzles in B&W Design Plants 106

17. Distribution of Leaking Nozzles in B&W Design Plant 107

18. CRDM Nozzle Leakage Observed at Oconee 3 108

19. Unidentified Leak Rate at Davis-Besse (Cycle 13) 109

20. As Found Locations of Boric Acid Deposits on Davis-Besse Vessel Head 110 (lORFO to 13RFO)

21. Nozzle Crack Leakage Rat_ Calculation Results 111

22. Finite Element Model Boundary Conditions to Simulate Axial Crack 112

23. Crack Opening Displacement with the Crack Surface Nodes Released 113

24. Boric Acid Deposits on Top of Head at Start of 13 RFO 114

25. Corrosion Rate for EPRI Experiments (Proprietary) 115

26. Timnline of Key Events Related to Reactor Vessel Head Boric Acid Wastage 116

27. Events & Casual Factor Chart 117a-e iigures . Iv Analysis Report Root Cause Analysls Figures 6 he

Title Page No.

28. Leaking Flanges Found and Repaired During Each Outage 118

29. Flange Leakage with Stalactite Formation from Insulation and Stalagmite 119 Formation on top of Reactor Vessel Head (8RFO)

30. Flange Leakage Crusted On Side of Nozzles and Stalactites from Gaps in 120 Insulation (8RFO)

31. Reddish Brown Boron Deposits Crusted on Side of Nozzle (8RFO) 121

32. Boron Deposits - Source Unclear (8RFO) 122

33. North Side of Reactor Vessel Head (IORFO) 123

34. Boron Deposits Near Top of Reactor Vessel Head (10RFO) 124

35. Typical Deposits for Periphery (1ORFO) 125

36. Red Rusty Boric Acid Deposits on Vessel Flange (12RFO) 126

37. Boron Piled Under the Insulation (1IRFO) 127

38. Boric Acid Deposits with Heavy Iron Concentration on Underside of Nozzle 3 128 (13RFO)

39. 2000 Interferences with CRDM Flange Inspection 129

40. RE4597 Sample Location 130

41. CTMT Radiation Monitors RE4597AAIBA (Combined Iodine Channels) 131

42. CITMT Radiation Monitors RE4597AA & BA (Both Noble Gas Channels) 132

43. Potential Effects of Boric Acid Deposits on Vessel Top Head Surface 133 3

s..

.1 ..

vlvwjd .M W Root Cause Analysis Report

Attachments Title Pane No.

1. Potential Evidentiary Request List (Rev. 4) 134

2. Sequence of Relevant Events 137 Attachments a vi Analysis Report Cause Analysis Root Cause Report Attachments

• vi

1.0 Problem Statement 1.1 Reason for Investigation Significant degradation of the reactor pressure vessel top head base metal was discovered at nozzle 3 (toward nozzle 11) and minor corrosion at nozzle 2 during the thirteenth refueling outage (13RFO) in March, 2002.

This root cause report addresses the cause of the loss of RPV head base metal in the region of nozzle 3 and 2. This issue of the root cause report addresses conditions and information available through April 6, 2002.

1.2 Consequences of EventlConditon Investigated The RPV head is an integral part of the reactor coolant pressure boundary, and its integrity is vital to the safe operation of the plant. Degradation of the RPV head or other portions of the reactor coolant pressure boundary can pose a significant safety risk if permitted to progress to the point where there is risk of a loss of coolant accident. Analysis indicates that the as-found condition of the affected nozzles would not have been expected to result in failure of the pressure integrity of the reactor coolant system. However, the degraded condition had been progressing over a period of time, without knowledge of the condition.

1.3 Immediate Actions Taken

1. At the time of discovery, the plant was already in a safe, shutdown condition. Ongoing outage activities related to the repair of the CRDM nozzle on the RPV head were suspended.

2. A root cause evaluation team was convened to perform the initial investigation.

3. A plan was created to preserve and collect, evidence necessary for the investigation.

Report 1.0 Problem Statement

• 1 Analysis Report Root Cause Analysis 1.0 Problem Statement

• 1

2.0 Event Narrative

2.1 Background

Davis-Besse is a raised loop pressurized water reactor (PWR) manufactured by Babcock and Wilcox (B&W). The reactor licensed thermal power output is 2772 megawatts. The plant achieved initial criticality on August 12, 1977. The RPV has an operating pressure of 2155 psig and a design pressure of 2500 psig. Davis-Besse has accumulated 15.78 effective full power years (EFPY) of operation when the plant was shut down for 13RFO.

The RPV head has 69 CRDM nozzles welded to the RPV head of which 61 are used for CRDMs, seven are spare, and one is used for the RPV head vent piping. Each CRDM nozzle is constructed of Alloy 600 and is attached to the RPV head by an Alloy 182 J-groove weld. The RPV head is constructed of low-alloy steel and is internally clad with stainless steel. Figures 1, 2 and 3 show the arrangement of the Davis-Besse RPV head. Figure 1 is a section view through the RPV centerline, Figure 2 is a plan view from the top of the RPV closure head, and Figure 3 shows how the CRDM nozzles are welded into the RPV head.

Throughout this report the CRDM nozzles will be addressed as nozzles 1, 2, ...69 and not the associated nozzle core grid location. Given that many of the sources referenced during the root cause analysis utilized the nozzle core grid location, the list below provides a correlation between the CRDM nozzle and core grid location.

CRDM CORE GRID CRDM CORE GRID CRDM CORE GRID NOZZLE# LOCxATION NOZZLE# LOCA77ON NOZ:ZLE LC1O I HS 24 N8 47 D12 2 G7 25 H4 48 N12 3 09 26 ES 49 N4 4 K9 27 Ell 50 CS S K7 28 Mul Sl Cll 6 FE 29 MO 52 E13 7 HIO 30 D6 53 M13 a LE 31 DIO 54 Ol 9 H6 32 F12 55 05 10 F6 33 L12 56 M3 II Fi0 34 NIO 57 E3 12 L1O 35 N6 58 Es 13 L6 36 LA 59 H14 14 E7** 37 F4 60 Ps 15 E9* 38 C? 61 H2 16 Gl* 39 C9 62 B6 17 KIP 40 013 63 BIO 1859q 41 K13 64 F14 19 M7* 42 09 65 L14 20 KS. 43 07 66 Plo 21 OP 44 K3 67 P6 22 DS 45 03 68 L2 23 H12 46' D4 69 F2

• Spare nozzles

• • Head vent connection Report Analysis Report 2.0 Event Narrative e 2 Cause Analysis Root Cause 2.0 Event Narrative a 2

On August 12, 2001, Davis-Besse received Nuclear Regulatory Commission (NRC)Bulletin 2001-01 Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles (reference 3.5). In discussion held with the NRC on November 28,2001, in response to this bulletin, Davis-Besse committed to a 100% qualified visual inspection, non-destructive examination (NDE) of 100% of the CRDM nozzles and characterization of flaws through destructive examination should cracks be detected. During performance of these inspections during 13RFO significant degradation of the RPV top head base metal was discovered between nozzles 3 and 11 and some minor corrosion at nozzle 2 in March 2002.

2.2 Sequence of Events Because the sequence of events is in part developed based upon inferred information rather than conclusive validated facts, a sequence of events will not be discussed here. Instead the data analysis section will develop the bases for the sequence of events in determining the associated causes. Attachment 2 provides a sequence of relevant events from source documents reviewed during the root cause analysis process. Figures 26 and 27 provide a timeline of key events related to RPV head boric acid corrosion and the event and causal factors chart that provide a summary level sequence of events information developed as a result of the data analysis.

LU VCflI arraiive U 2.0 Event Narrative a a Root Cause Analysis Report

3.0 Data Analysis The data analysis section provides a summary of the data gathered during the root cause investigation. The data gathered for this analysis is from related potential condition adverse to quality reports (PCAQRs) and condition reports (CRs), the System Engineer's System Performance Book, pictures taken during inspections, personnel interviews, procedures, and other documents identified in the references section.

3.1 Non-Destructive Examination of RPV Head and Nozzles The following is a summary of the non-destructive examination effort on the cracked nozzles and degraded RPV head. After removal of insulation from the RPV flange early in 13RFO, boric acid crystal deposits and iron oxide were found to have flowed out from several of the openings (mouse holes) in the lower service structure support skirt (Figure 20). Figure 4 shows deposits on the flange during the inspection in the twelfth refueling outage (12RFO).

Blade probe ultrasonic (UTI) examination of the CRDM nozzles from below the RPV head for circumferential cracks and large axial cracks, identified axial cracks in nozzles 1, 2, 3, 5, 47, and

58. Supplemental top-down UT examination of these nozzles confirmed through-wall axial cracks extending above the i-groove weld elevation in nozzles 1, 2 and 3. Axial cracks were confirmed in nozzles 5 and 47, but they did not extend above the top of the J-groove weld and would not have caused a leak. Axial cracking was not confirmed in nozzle 58. A small (340 arc length and 0.34' deep) circumferential crack was discovered above the J-groove weld on the outside of nozzle 2 on the downhill side. It was determined that all five nozzles would be repaired by boring out the lower part of the nozzle containing the cracks, and rewelding the end of the nozzle to the opening in the RPV head using the Framatome ANP repair method.

After removing the lower part of nozzle 3, a cavity was discovered in the low-alloy steel RPV head material above the J-groove weld on the downhill side. Additionally, after removing the lower part of nozzle 2, a smaller area of corrosion of the low-alloy steel RPV head material was discovered between the bottom of the machined nozzle and the top of the J-groove weld (Figure 5). This area of corrosion was found to extend under the portion of the nozzle left in place after machining.

After pulling nozzle 3 and cleaning by hydrolasing, the top of the RPV head was inspected using a video camera on a long pole through the vacated nozzle 3 penetration. This inspection showed a large cavity in the low-alloy steel RPV head material between nozzles 3 and 11 (Figure 6). The area with missing material was reported as being about 6.6" long and approximately 4-5" at the widest point. Ultrasonic thickness measurements from the underside of the RPV head showed the thickness of the remaining material (cladding) to be an average of approximately 0.3%,which is greater than the 3/16" nominal or 1/8" minimum specified clad thickness. The videotape inspections also showed a small area of corrosion where nozzle 2 penetrates the RPV top head surface. The small area of corrosion at the top of nozzle 2 was found to lie directly over the area of corrosion at the bottom of nozzle 2 as seen in Figure 5. The videotape inspection also showed evidence.of a small leak path where nozzle 1 penetrates the RPV top head surface.

Data Afl8liS s4 3.0 Data Analysis

• 4 Report Root Cause Analysis Report 3.0

3.1.1 Potential Evidentlary Request List One of the first priorities of the root cause evaluation team was to collect and preserve the evidence necessary to facilitate the root cause evaluation. A Potential Evidentiary Request List was created-specifying the evidence to be collected and preserved and the reason for collecting the evidence. The list was revised several times throughout the evaluation. The latest revision is included as Attachment 1. This list was used to create integrated examination and inspection plans for field implementation.

3.1.2 Locations of Cracks and Corrosionon RPV Head Figure 7 shows locations of cracks and corrosion on the Davis-Besse RPV top head surface. This figure is included to serve as a reference for the following descriptions of the degraded areas.

3.1.3 NDE Examinationsof CRDM Nozzles Automated ultrasonic examinations of all 69 CRDM nozzles were performed from beneath the RPV head using the ARAIMS inspection tool and a "Circ." blade probe. The techniques utilized for this examination are intended for the detection and through-wall (depth) sizing of circumferential inside diameter (ID) and outside diameter (OD) initiating flaws in the nozzle base metal only. Forward scatter time of flight detector (TOFD), longitudinal-wave techniques are used. The examinations were conducted from the bore of the CRDM nozzles in the J-groove weld region of the nozzle.

The examinations performed with the blade probe consisted of scanning for circumferential and significant axial flaws within the nozzle wall. The tooling consisted of a blade containing a nominal 5 MHz, 50 degree TOFD transducer set. The circ. blade probe provides flaw detection (axial and circumferential flaws) and sizing (non-axial flaws) information. For the forward scatter transducers, flaw detection is identified by loss of signal response either from the lateral wave or backwall responses as well as the presence of crack tip diffracted responses.

Prior to the examinations, demonstrations were performed using the Electric Power Research Institute (EPRI)/Materials Reliability Program (MRP) samples removed from Oconee. This demonstration showed that circumferential and axial flaws can be detected with the circ. blade probe.

During the initial examination, some nozzles had areas where the gap between the nozzle and the leadscrew mechanism was too narrow to insert the probe. All of these areas were rescanned after the initial inspection by moving the leadscrew support tube to open the gap for examination.

The examination with the blade probe identified potential flaw indications in nozzles 1, 2, 3, 5, 47 and 58. Because almost all of the flaws detected on these nozzles were characterized as axial, only limited information was available with the circ. blade probe. These axial flaws could have been characterized using axial blade probes. However, because there was a high probability that these nozzles would require repair, the CRDMs for these nozzles were removed to perform additional UT examination using the top-down tool.

The top-down tool contains 10 transducers and provides the ability to detect and characterize axial and circumferential flaws and also provides additional information required for the repair activity. Images of the nozzles identified for repair are included in the reference 1.1 report and show the various features required to implement the repair. These generally include the location of the RPV head OD, the elevation of the proposed cut line, and the location of the top of the J-groove weld.

AnlssRpr u__-_ Analysi_

s e Root Caus Roo Cause Analysis Report 3.0 Daza Analysis a S

Automated ultrasonic examinations of CRDM nozzles 1,2, 3, 5, 47 and 58 were performed using the top-down inspection tool. The techniques utilized for the examination are intended for the detection and through-wall (depth) sizing of axial and circumferential ID and OD initiating flaws in the nozzle base metal only. Forward scatter, longitudinal-wave and backward scatter shear wave techniques were used. The examinations were conducted from the bore of the CRDM nozzles in the l-groove weld region. The potential flaw indication on nozzle 58 was determined to be a false indication using the top-down tool, potentially due to nozzle ovality.

The inspections consisted of scanning for axial and circumferential flaws within the nozzle. The tooling consisted of a transducer head that holds 10 individual search units. These search units were divided into two sets, one for the axial beam direction and one for the circumferential beam direction. The axial beam search units consisted of 5.0 MHz, longitudinal wave forward scatter time of flight search units with angles of 300 and 450; backward scatter pulse echo, 2.25 MHz 60° shear wave search units; and a 5.0 MHz 0° search unit. The circumferential beam search units consisted of 5.0 MHz, longitudinal wave forward scatter time of flight search units with angles of 450, 550, and 65°; backward scatter pulse echo, 2.25 MHz 600 shear wave search units; and a 5.0 MHz 0° search unit.

The detection of flaw indications is based upon the expected responses for each search unit and technique. The 0° transducer provides weld position information and also provides positional information regarding any lack of backwall response in the region of the flaw. The forward scatter time of flight techniques provide flaw detection and sizing information. For the forward scatter transducers, flaw detection is identified by loss of signal response either from the lateral wave or backwall responses as well as detection of crack tip diffracted responses. The 600 shear wave transducer provides detection by means of corner trap responses between the flaw and nozzle surface and sizing with tip diffracted signals.

Reference 1.1 contains the data sheets from ultrasonic examination of the six CRDM nozzles that were identified as having flaws with the blade probe. Included in this report are the data sheets for the blade UT and the rotating UT using the top-down tool. Images of the UT data are also included to show the features identifying detected flaws.

The data was also reviewed for evidence of a leak path in the penetration bore with the blade and rotating UT techniques. Leak paths were detected in nozzles 1,2, and 3 with blade and rotating UT. Images of the leak paths are included in the reference 1.1 report. Subsequent review by EPRI of all UT results indicated that although nozzle 46 had no detected cracks, some evidence of a leakage flow path was identified, due to the characteristics of the backwall reflection.

The examination results are summarized in the following table:

Nozzle # Summary of NDE Results 1 9 Axial Flaws, 2 through-wall (PIW) with a leak path 2 8 Axial Flaws, 1 Circ. Flaw, 6 TW with a leak path 3 4 Axial Flaws, 2 TW with a leak path 5 I Axial Flaw 46 No Flaw Indication, potential leak path 47 I Axial Faw 58 No Recordable Indications Root Cause Analysis Report 3.0 Data Analysis

• 6

A pictorial layout of the identified flaws in nozzles 1,2, 3, 5 and 47 is provided in Figures 7-12.

Detailed NDE results are provided in Tables 1-5.

3.1.4 Visual Examinations of RPV Top Head and Penetrations Visual examinations were made of the RPV top head surface both before and after removing a section of insulation over the nozzles of concern. The RPV head penetrations were also examined following removal of nozzles 2 and 3. Results of these examinations were as follows:

Degradation at Nozzle 3 Degradation observed at nozzle 3 is pictorially shown in Figure 13. The 180° (uphill toward nozzle 1) location is essentially intact, with little to no degradation. The 00 (downhill toward nozzle 11) location exhibits the worst degradation, with the low-alloy steel material corroded away, down to the stainless steel cladding, for approximately 6.6 inches in length and 4 to 5 inches at the widest part. Figure 14 shows cladding thickness measurements made by UT from the underside of the RPV head. (Note: There are several low readings outside the designated area of damage. These are attributed to inclusions or bad readings. These readings do not correspond with visual observations, and will be further verified following excavation of the damaged area.)

From the 2700 location to the 00 location (counterclockwise looking down from the top of the RPV head), there is a large undercut area. From the 00 location to the 90° location (counterclockwise looking down from the top of the RPV head), the corrosion is less. Additional data is forthcoming from profilometry and the failure analysis efforts.

Deeradation at Nozzle 2 Degradation was observed at nozzle 2 following the initiation of repair efforts. Figure 5 and 5a shows the observed area of corrosion. The overall corroded area, based on the video examination and approximate measurements from the impression, is 3-1/2 to 4 inches in length starting from the top of the RPV head, about 3/8 inch deep (at the deepest location approximately 1-3/4 inches from the top of the RPV head), and between 1-1/4 to 2 inches at it's widest location. The depth of corrosion decreased as the annulus opening was approached. This type of corrosion profile is similar to testing that has been performed by EPRI (reference 5.3) regarding location of the deepest corrosion and the fact that it could have been identified on the top of the RPV head.

Deeradation at Nozzle 1 The observed deg aon at the nozzle I location is minimal. A small, crevice (d116 inch wide LaQlJL3L14inch circumnferentially) lctdath oaton (looking down from top of RPV head clockwise with 00 in the West direction), was identified at the surface of the RPV head: The observed degradation at nozzle I was within the boundary of the pre-established repair plan.

dB A E%-& A..S..1-4.W LIULU PUMMIYO&O Root Cause Analysis Report W

3.2 Cracks, Leaks and Corrosion This data analysis section is a technical review of the causes of cracks and leaks throughout the industry, and the resultant corrosion of the RPV top head surface. This information provides key inputs to the probable cause determination.

3.2.1 CRDM Nozzle Cracks and Propagation to Leakage The following is a review of cracking experience in Alloy 600 RPV head CRDM nozzles, and identification of the possible causes of cracks in Davis-Besse nozzles 1, 2, 3, 5 and 47.

Primarv Water Stress Corrosion Crackin! of Alloy 600 and Alloy 82/182 Materials There have been numerous incidents of cracked Alloy 600 nozzles, and Alloy 82/182 welds, in domestic non-steam generator related PWR plant primary system applications since a pressurizer instrument nozzle leak at San Onofre 3 in 1986. These applications include pressurizer instrument nozzles, pressurizer heater sleeves, hot leg piping instrument nozzles, CRDM nozzles, steam generator drain nozzles, RPV outlet nozzle butt welds, and a pressurizer spray line safe end. In all cases, the leakage has been discovered before failure of the components.

In all but a few cases, cracking in nozzle applications has been attributed to primary water stress corrosion cracking (PWSCC). The mechanism of PWSCC is not completely understood, and prediction of crack initiation time has proven to be difficult, if not impossible. It is known, however, that PWSCC of Alloy 600 occurs as a result of the following three factors:

• A susceptible material

• A high tensile stress (including both operating and residual stress) at J-groove welds, roll expansions, and expansion transitions

• An aggressive environment (PWR primary water at high temperature)

The few exceptions are related to weld defects (Ringhals J-groove welds) and resin intrusions (Zorita). These incidents are documented in EPRI TR-103696, PWSCC of Alloy 600 Materials in PWR Primary System Penetrations (reference 5.4).

The susceptibility of Alloy 600 material depends on several factors including the chemical composition, heat treatment during metal production, heat treatment during fabrication of the component, and operating parameters. Alloy 600 is known to be susceptible to PWSCC with some heats of material being more susceptible than others principally due to a poorer microstructure.

High stresses are induced into the nozzle by the J-groove weld. Since the RPV head is stiff relative to the nozzle wall, shrinkage of the J-groove weld during cooling pulls the nozzle wall radially outward causing high tensile hoop stresses as shown in Figure 15 (the deflections in Figure 15 are exaggerated to illustrate the welding induced distortion). If the nozzle is machined prior to welding, higher stresses can be induced in the cold worked machined surface.

PWSCC has been the subject of much research and analysis in recent years as a result of the many leaks that have been attributed to PWSCC. However, an accurate crack initiation model has yet to be developed.

Effect of Alloy 600 Heat Treatments Chemical composition and heat-treatment are interrelated in several ways. For example, orle.

reason for annealing Alloy 600 is to solutionize the carbon in the alloy. As the material cools, the available carbon and chromium will precipitate (in the form of chromium carbides) from

~~ Cas AnlssRpr . aaAayi Root Cause AnalysisRoot~~~~

~

Report 3.0 Data Analysis a 8

solution at both intragranular and intergranular locations. If the cooldown from the anneal is sufficiently slow, a greater number of carbides will precipitate at the grain boundaries (i.e.,

intergranularly), and the resistance to PWSCC will be improved.

Well decorated grain boundaries are an indication that an Alloy 600 material has received a proper heat treatment and that sufficient carbon was available in solution to combine with chromium. If adequate amounts of carbon and chromium exist, but the anneal was not at a high enough temperature or sufficient time was not allowed to solutionize the carbon, an adequate amount of carbon will not be available to precipitate intergranularly as chromium carbides, leading to minimal grain boundary decoration. For example, a temperature of 1850"F is necessary to solutionize material with a carbon content of 0.04%.

The actual annealing temperatures for the Davis-Besse CRDM nozzle Alloy 600 materials could not be located. However, the minimum range of annealing temperatures used by the 0

manufacturer (B&W Tubular Products) at the time were 1600-1700 F. Therefore, it can be assumed that the microstructure of the heats of material utilized on the Davis-Besse RPV head is likely to be less than optimum relative to resistance to PWSCC. Additional information on the microstructure will be obtained during destructive examinations of nozzles 3 and 2.

Recent RPV Head Nozzle and Weld Leakage Experience The first leak from an RPV head CRDM nozzle occurred at the EdF Bugey 3 plant in France in 1991. A small amount of leakage [<1 liter/hr (0.004 gpm)] was discovered on the outside surface of the RPV head during a primary system hydrostatic test. Investigation showed the leak was from a through-wall crack in an outer row CRDM nozzle that had initiated from the inside surface. Failure analysis confirmed that the crack was PWSCC and that contributing factors included susceptible material microstructure, stress concentration at a counterbore on the nozzle inside surface, high hardness of the cold worked machined surface, and high residual tensile stresses induced in the nozzle during welding.

Subsequent to the Bugey 3 experience, PWSCC of Alloy 600 base metal and welds has been discovered in other PWR RPV heads worldwide. In 1994 a partial through-wall crack was found at DC Cook 2. like the Bugey 3 crack, the crack at DC Cook 2 initiated on the ID surface of the nozzle at the elevation of the J-groove weld. CRDM nozzle inspections have also found shallow craze cracking on the nozzle ID near the weld in a few nozzles at Oconee 2 and Millstone 2.

As of February 2000, about 6.5% of all EdF nozzles inspected had been found to contain cracks and about 1.25% of inspected nozzles in other plants worldwide had been found to contain cracks greater than the minimum measurable depth of about 2 mm (0.08 inches). Further details regarding the extent of condition are provided in MRP-44, Part 2, PWR Materials Reliability Program - Interim Alloy 600 Safety Assessments for US PWR Plants, Part 2: Reactor Vessel Top Head Penetrations (reference 5.5).

In November 2000 a through-weld leak at a CRDM nozzle weld at Oconee 1 was attributed to PWSCC. Laboratory analysis of a boat sample removed from this weldment confirmed that the crack was PWSCC.

In February 2001, PWSCC was detected in nine nozzles at Oconee 3 that were from one material heat (M3935) that had a yield strength of 48.5 ksi. Most of these cracks were axial and initiated on the nozzle OD surface. However, some axial.pocks had initiated on the ID and propagated partially through the wall. In addition, most of the cracks that were found in Oconee 3 nozzles 0 A~ nt.#.. A-c - a Root Cause Analysis Report Q.W V04UX aby1164 - W

initiated below the weld, similar to those found at Davis-Besse. However, three Oconee 3 nozzles contained OD circumferential cracks above the weld.

In April 2001, Oconee 2 performed a visual inspection of the RPV head during a refueling outage at the end-of-cycle 18 (approximately 21 EFPY). Boric acid crystals were observed at four CRDM nozzles. The inspections performed at Oconee-2 in 2001 identified OD crack-like axial indications below the weld on all four nozzles. Ultrasonic examinations showed that these indications were OD-initiated and that none of the indications were through-wall. An OD-initiated circumferential indication, 0.1 inch (2.5 mm) in depth and 1.26 inch (32 mmn) in length, was noted above the weld on one of the nozzles. Eddy current examinations of the ID of the nozzles revealed shallow craze-type flaw clusters in all four nozzles that were distributed around the entire ID circumference (3600 and above the weld). Based on these results, the leak path was through the interface between the nozzle and the J-groove weld.

In November 2001, a visual inspection of the top surface of the Oconee 3 reactor RPV head showed evidence of primary water leakage on the RPV head surface. This inspection was performed in accordance with Duke Energy's response to NRC Bulletin 2001-01 as a "Qualified Visual" inspection. Boric acid deposits with a wet appearance were identified around four CRDM nozzles and determined to be probable leak locations. Three additional CRDM nozzles were identified as being masked by boric acid crystal deposits from an indeterminate leakage flow path and were therefore classified as possible leaking nozzles. This is the same visual inspection performed during the previous outages except that a VT-2 qualified inspector participated. UT examinations showed that five nozzles had indications that extended from below the weld to above the weld indicating a leak path in addition to various other ID and OD indications. One nozzle had a circumferential indication in the nozzle above the weld. Seven CRDM Nozzles were repaired during this outage using the automated Framatome-ANP ID Ambient Temper Bead Repair" technique that is being employed at Davis-Besse.

At least five other PWRs have identified similar PWSCC in the last year.

In summary, since Noveiv 2Q Iaks have been discovered from at least 30 CRDM nozzles at PWRs in the United States. Most of the leaks have been through axial cracks in the nozzle base material, but some have also been through axial/radial oriented cracks in the J-groove welds. Investigation of these leaks has led to the discovery of circumferential cracks above the J-groove weld at some plants, including 165' through-wall circumferential cracks in two nozzles at Oconee 3. None of these plants reported loss of matenal due to general corrosion that was similar to Davis-Besse nozzle 3.

PWSCC Cracks in B&W Deslen PWR Plants The most directly related experience has been cracking and leakage of CRDM nozzles at the other B&W design plants: Arkansas Nuclear One Unit 1 (ANO 1), Crystal River 3, Oconee 1-3, and Three Mile Island Unit I (CI 1). AU of these units have experienced cracks and leaks from the nozzles near the J-groove weld elevation. The cracks have been predominately axial and have tended to initiate on the outside surface of the nozzle at the weld toe or in the weld. In some cases, there have been circumferential cracks in the nozzle wall above and below the J-groove weld.

Laboratory examination of specimens removed from Oconee 1 and Oconee 3 confirmed that the axial cracIs (Oconee 1) and circumferential cracks above the J-groove weld (Oconee 3) were PWSCC.

3.0 Data Afl!Y5I5 S iLl Root Cause Analysis Report Analysis Report 3.0 Data Analysls a* 1U

Figure 16 shows the locations of the leaking nozzles in the B&W design plants. It should be noted that the leaking nozzles are distributed across the RPV head. Figure 17 provides further information regarding the distribution of leaking nozzles. This figure shows the fraction of all of the nozzles at each row that have leaked and whether the cracks are purely axial or axial with circumferential cracks above the J-groove weld elevation.

Lack of Fusion In l-Groove Welds During a CRDM nozzle inspection at Ringhals Unit 2 in 1992, an indication was detected in the J-groove weld at one of the penetrations. The indication was not indicative of PWSCC; rather, the indication was attributed to a weld defect that occurred during fabrication of the CRDM nozzle to the RPV head. The B&WOG took action to address this concern by acquiring additional data from several sources. First, the data from Ringhals Units 2 and 4 and data from a cancelled Westinghouse reactor, Shearon Harris, were acquired from the Westinghouse Owners Group (WOG). Second, the B&W Owners Group (B&WOG) performed an inspection of the RPV head from Midland Unit 1, which was a cancelled nuclear station fabricated by B&W.

An addendum to the B&WOG safety evaluation was prepared to analyze these data (reference 2.9). This evaluation included a statistical review and analysis of the J-groove weld inspection data and a stress analysis of the CRDM J-groove weld to determine the minimum weld area that is required to meet the American Society of Mechanical Engineers (ASME) Boiler and Pressure Vessel (B&PV) Code primary shear stress limits. It was shown in this report that the maximum areas of weld lack of fusion detected for the Midland Unit 1, Shearon Harris, and Ringhals Unit 2 RPV heads are well below the ASME Code allowable limits for weld structural integrity. It was concluded that a large margin exists between the statistical bound of the total lack of weld fusion areas in the Midland Unit 1 RPV head and the ASME Code allowable limits. Therefore, although some areas of lack of fusion are expected to be observed, they do not give rise to a safety concern.

Comparison of Davis-Besse Nozzles to Other B&W DesiEn Plants Table 6 is a comparison of key features of the Davis-Besse RPV head to the other B&W design plants from the standpoint of cracks and leaks. This table shows several potentially significant design and fabrication differences.

Operating Temnerature The abi¶BPe operating condition RPV head temperature is reported to be 60O°F relative to the 601-602 0F for the other six B&W dsidgnesJaIts. This small temperature difference has some effect on the predicted time to leakage, and this fact is reflected in the row which reports the EFPYs adjusted to a common 6000F operating RPV head temperature. The effect of the higher Davis-'Besse RPV head temperature is offset by the shorter operating time, leading to a temperature adjusted time that is less than Oconee 1,2 and 3, and ANO 1.

Counterbores at th To and Bottom of CRDM Nozzles All of the B&W design plantsxE1tforDavis-Be were designed with a counterbore machined into the pen o in th.R head before instllit CRDM nozzles.

Elastic-plastic finite element stress analyses show little difference in welding residual in-a operating stresses for the two designs, especially for nozzles near the center of the RPV head where the counterbore is only a short distance above the top of the J-groove weld. Therefore, the lack of a counterbore in tfig Davis-Besse's design is not considered to be a factorln the condition.

Report 3.0 Data Analysis

• 11 3.0 Data Analysis e 11 Root Cause Analysis Report

RPV Head to Hot Leg Vent Line at CRDM Nozzle 14 Davis-Besse has a vent line that runs from nozzle 14 to the steam generator 2 upper primary hand hole. This line is unique to Davis-Besse. The purpose of the line is to vent non-condensable gases from the head during a loss of coolant accident. This vent line could have a minor effect on head temperature. However, since this nozzle is displaced from the cracked nozzles, its effect on other nozzles is considered to be very small. There is no evidence of thermal fatigue on this penetration.

Susceptibility of Davis-Besse CRDM Nozzles to PWSCC The 69 CRDM nozzles in the Davis-Besse RPV head were manufactured from four different heats of material as shown in the following table (reference 2.1). Three of the heats (C2649-1,M3935, and M4437) of material were manufactured by B&W Tubular Products (B&W-TPD), and the fourth heat was manufactured by the International Nickel Corporation (INCO).

According to an assessment performed by Framatome ANP, these CRDM nozzles have a relatively high susceptibility to PWSCC, mainly because of the residual stress distribution

/ calculated in the vicinity of the J-groove weld and the Davis-Besse RPV head operating temperature of 605F (reference 2.2).

---I CRDM Nozzle Heats at Davis-Besse Heat YS _m Can Anneal Number No. ofNozzles (kii) (I) M Temp (F) Nczzlk Nos.

7,12.16.20.

C2649-1 32 44.9 92.6 0042 1600.1700 22-25.27-29.

38.44, 47.55*

57,64.65.68.

69

/

i3 1.1935 48.S 1 3.6 0.025 1600-1700 1.5 1-10, 13-15.

23 92.2 0.059 1600.1700 17-19,21.26.

M4437 359 30-37.61-63, 67

_ I. . I NX5940 9 39.0 U3.0 0.030 1600 win. 6. 11.45.46.

56.580.66 I -. l Experience to date has shown that:

• There are more leaks (15) from nozzles fabricated from heat M3935 than any other single heat (4 max) in B&W design plants, and

• A larger fraction of Fizzles (20.3%) from heat M3935 have developed leaks than any other single heat (13.3% max) in B&W design plants.

Nozzles Ithrough 5at Davis-Besse are from heat M3935. Nozzles 1, 2 and 3 have through-wall axial cracks and leaks with nozzle 2 having notable RPV head corrosion and nozzle 3 having extensive corrosion. In addition, nozzle 5 had an axial crack requiring repair. Nozzle 47 also had an axial crack requiring repair, but the nozzle was from heat C2649-1. Ini summary, the leaks at Davis-Besse are all from the heat of material that has previously resulted in more leaks than any other heat in the industry.

a n nttn A -.

Q.V Gus~rWIQj4R&

• teeIt

- Ad Root Cause Analysis Report

Range of Interference Fits Davis-Besse is similar in design to Oconee, Crystal River-3, TM] 1 and ANO 1, which have demonstrated an ability to identify leaking CRDM nozzles through an interference fit by visual inspection for boric acid crystal deposits. During fabrication, CRDM bores were inspected for final top and bottom bore diameter and verticality. After custom grinding individual CRDM nozzle shaft to approximately 0.001" greater in diameter than the final CRDM bore diameter, the shafts were measured at both the top and the bottom of the custom ground length. CRDM nozzle shafts are longer than CRDM bores are deep. Thus, CRDM nozzle shaft diameter measurements do not directly line up with CRDM bore diameter measurements, although in the case of Davis-Besse these locations should be fairly close because of the lack of counterbores. Therefore, the resulting top and bottom dimensional fits are considered approximate. The values for the Davis-Besse RPV head are calculated to range from a gap of 0.0010" to a maximum interference fit of 0.0021".

High PWSCC Susceptibility In Heat M3935 The reason for the higher susceptibility of heat M3935 has not been determined although it may be related to a lower than 2ptimum annealin temperm or through-thickness hardness gradient created in the material by a Additional data will be acquired during examinations of nozzle 3. However, this data is not needed to support the basis of this root cause document.

While heat M3935 appears to have higher PWSCC susceptibility than some other heats, several other heats of material have also experienced multiple leaks in B&W design plants. Heats of material that have not experienced leaks to date may experience cracks and leaks in the future.

The same is true for J-groove welds.

Other Possible Causes of Cracks Several other potential mechanisms for crack initiation and propagation were considered for the observed flaws in the CRDM nozzles. These are:

Fabrication and In tecion Anomalies All the CRDM nozzle Alloy 600 materials used by B&W during the Davis-Besse RPV head manufacturing were either supplied by the B&W-TPD or by INCO. The materials were ordered to ASME Boiler and Pressure Vessel (B&PV)Section II Specification SB-167 and Section m requirements. nyfabrication or inspection anomalies would have been identified since dye penetrate testing (FT and UT was performed).

Thermal Fatigue CRDM nozzles may be subject to thermal fatigue induced by thermal fluctuations, which result from particular operating transienth. Several permutations of stratified fluid conditions have been observed to result in fatigue cracking and component failures in PWRs. However, no CRDM nozzles have experienced this type of failure, and there is no historical evidence to support thermal fatigue cracking. Given the past experience, it seems unlikely that thermal fatigue degradation would result in the formation of discrete axial cracks located at high stress locations (uphill and downhill sides) within the bore of the CRDM nozzles.

lntergranular Stress Corrosion Cracking aGSCCQ Cracking can possibly occur due to additional factors not normally associated with PWSCC.

Contaminant species such as sulfur, chloride, or fluoride compounds could result in IGSCC.

Oxygen at levels found in boiling water reactors also can cause IGSCC. The presence of

_1 I *1 .... _ 3a..U Root Cause Analysis Report ax LJUIU #Maly,51F, a la

these contaminants in combination with high stress and less than ideal material microstructure could lead to IGSCC. However, there is no evidence that this occurred at Davis-Besse (see following section). In PWRs, there is insufficient oxygen to cause IGSCC.

Davis-Besse has not experienced any incidents of resin ingress, which is the most common source of sulfur (Reference 2.1). Also, chlorides and fluorides are controlled in the primary water. Thus IGSCC is discounted as a failure mechanism since oxygen, chlorides, fluorides or sulfur were not present in sufficient quantities in the reactor coolant system (RCS).

RCS Chemistry Control Chemical transients in the primary water were considered to determine if nozzle cracking was influenced by conditions other than those causing PWSCC. In response to Generic Letter 97-01, Framatome report BAW-2301 (reference 2.1) summarized abnormal chemistry time periods at each of the B&WOG plants. The primary water chemistry analysis results at each of the B&WOG plants were reviewed for excursions during power operation, hot shutdowns, and cold shutdowns. At the time of this report, no events have occurred at Davis-Besse since December 10, 1983 when a resin specification problem led to a short transient in chlorides (up to 0.26 ppm) and lithium. This event is not considered significant.

The amount of hydrogen in the primary coolant during the last three cycles was analyzed to confirm that excess oxygen was not available to promote corrosion within the primary side.

Boric acid quality was researched as a possible issue, and potential for impurities to contribute to nozzle cracking. It was determined that the boric acid used at Davis-Besse is common to the industry and that the quality control program in place for the boric acid is appropriate. Additionally, pure boric acid with no impurities has been shown in the Boric Acid Corrosion Guidebook (reference 5.2 or 5.3) to be capable of the corrosion rates seen in this condition.

Other Failure Mechanisms Other failure mechanisms were considered briefly and discarded since either they would already be encompassed by the environmentally assisted mechanisms noted above or the review of the evidence did not support them. These include environmentally assisted fatigue, mechanically induced fatigue, and hydrogen damage.

Conclusions Reeardine Source of Cracks The similarity of the flaws in the Davis-Besse CRDM nozzles 1, 2,3,5, and 47 to the PWSCC cracks found at the other B&W designed nuclear power plants supports the evidence for concluding that the flaws are PWSCC. The flaws are similar in length and orientation to confirmed PWSCC at other plants and there is no other credible mechanism for these types of flaws. Four of the five cracked nozzles at Davis-Besse, and all three of the leaking nozzles at Davis-Besse, are from heat M3935 that has exhibited the highest percentage of leakage of any heat of material in domestic PWR plants. Therefore, the probable cause of cracks in the Davis-Besse nozzles is PWSCC.

Crack Propaeation to Leak PWSCC of Alloy 600 components in RCS can lead to through-wall cracking, and, thus, leakage of primary water. Based on the visual inspections of the Davis-Besse RPV head, containment air cooler cleaning frequency, interviews, etc., a reasonable time-frame for the appearance ojpLek on the RPV head at Davis-Besse is approximatel 1994-1996. Uitizing an averae PWSCC bftcligrwth rate of approximately 4 mm/year (reference 5.9) iiough the 16 mm (0.62 inch) 3.0 Data Analysis

• 14 Root Cause Analysis Report 3.0 Data Analysis e 14

thick CRDM nozzle material, the time-frame at which crack initiation occurred would correspond to approximately 1990 i3 years. This is a reasonable approximation to the more detailed type of calculations performed by the B&WOG in the safety assessment (reference 5.5),

which assumes approximately 4-6 years for a through-wall flaw to develop in the area near the J-groove weld.

3.2.2 Leakage Rate From CRDM Nozzle Cracks Nozzles with through-wall PWSCC cracks in either the nozzle wall or 3-groove weld can develop leaks into the annulus between the nozzle and hole in the RPV head. The following is a discussion of Davis-Besse and industry experience regarding leak rates.

Industry Experience Prior to Davis-Besse, industry experience had been that PWSCC cracks at RPV head nozzle penetrations only result in a small ring of boric acid crystal deposits as shown in Figure 18.

Estimates from Oconee are that the volume of deposits from these leaks is less than I in3. Using Figure 6-3 of the Boric Acid Corrosion Guidebook, Revision 1 (reference 5.3), and an assumed average boron concentration of 750 ppm over an operating cycle, one cubic inch of boric acid corresponds to leakage of about 1gallon of water. This corresponds to an average leak rate of about 1 gpm over an operating cycle (two year period).

Similar low leak rates have been reported for most other nozzles attached by J-groove welds including pressurizer instrument nozzles, pressurizer heater sleeves, and hot leg piping instrument nozzles. However, there have been some cases where larger leakage has been reported. These cases include a pressurizer heater sleeve containing a failed heater (ANO 2), and several piping instrument nozzles (ANO 1 & Palo Verde 2). In summary, while most through-wall cracks at Alloy 600 nozzle attachment welds result in very small leaks, there are exceptions where greater leakage has occurred.

There are several main theories that explain why leak rates are typically low.

• The cracks only extend a short length in the high tensile residual stress zone above the J-groove weld.

• PWSCC cracks are tight and may become plugged by small amounts of impurities in the primary coolant

• The leaking fluid flashes within the crack, leaving boric acid deposits that block further flow through the crack.

The most likely explanation is that low leakage results from tight PWSCC cracks that extend a short distance above the J-groove weld. The basis is that low leakage is also observed from most smaller diameter instrument nozzles that are also installed in pressure boundary parts by J-groove welds without an interference fit.

While the exact cause of the previously observed low leak rates has not been conclusively established, it has been determined that the leak rates are typically low, and qualified visual inspection programs are required to identify the leaks. A 'qualified visual inspection" requires 1) a clean enough RPV head to identify small rings of boric acid crystal deposits, 2) visual access to locations where nozzles penetrate the RPV head, and 3) confirmation that there will be a flow path through the annulus between the RPV head and nozzle under operating conditions. Because an interference fit was predicted, Davis-Besse could not "qualify" nozzles 1-5 for a visual exam, whether or not they were clean. However, because or the very small percentage oractual metal-Cas . Dt nlss* _nlssRpr Root Cause AnalysisRoot~~~~

Report 3.0 Data Analysis e 16

to-metal interference (i.e., high points only) leakage is anticipated despite the predicted interference.

Unidentified Primary System Leakage Rates at Davis-Besse Normal operational leakage in the RCS is recorded and analyzed in MODES 1-4, looking for adverse trends and to verify compliance with Technical Specification 4.4.6.2. This Technical Specification requires that there is no pressure boundary leakage, and that unidentified leakage is maintained at less than 1 gpm. Although the surveillance is required once per 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br />, RCS leakage is genera d once per da. DB-SP-03357, RCS Water Inventory Balance, provides the methodology to determine the RCS leakage rate. The test calculates total RCS leakage by resolving changes in initial and final values of Pressurizer and RCS Makeup Tank levels over a 1 to 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> period, and providing corrections based on RCS temperatures and pes-surcs- Identified sources of leakage that are appaent through changes in Pressurizer Quench Tank level, normal (measured) Reactor Coolant Pump seal leakage, or quantified primary to secondary tube leakage, are subtracted from the calculated total RCS leakage to obtain the unidentified leakage value. This method of determining unidentified leakage has a significant daily variation (in the range of 0.05 to o0.1Apm)jkhat depends on accuracy of identified leakage measurements, stability of the plant during data collection, and duration of data collection.

Therefore, monthly or running averages are most useful to determine leakage trends, similar to that presented in the figure below.

Monthly Average Unldentifled Leakage, Cycles 10 3 Upm) e(gpm) 0.9 0.8 0.6 0.5 0.4 10 3 _ _ _ _ _ _ _ _ _ _

0.1

.11 J. .2,1111111111211 111111MMM H W ru. . . . . . . . .. . . .

. .onthIYear During operating cycles 10 through 13 (November 1994 through February 2002), measured unidentified leakage has ranged from slightly negative to as much as approximately 0.8 gpm in April 1999. From review of data over that time span, it would appear that average unidentified leakage tends to be in the range of 0.0 to 0.03 gpm.when the plant is"tight! with no pending maintenance concerns. If unidentified leakage begins to trend upward, efforts are expended to determine the cause, and if necessary, effort is made to repair the source of the leakage. For Reo_ 3.0aaAay . _

Cas _nlyi

_oo Root Cause Analysis Report 3.0 Data Analysis e -it

example, the high leakage in April 1999 was identified as from the Pressurizer Code Safety Valves and the plant was shutdown to service the valves. Following that shutdown, unidentified leakage remained in the range of 0.15 to 0.25 gpm, some of which was attributable to CRDM flange leakage and some of which may have been attributed to CRDM nozzle leakage.

When a CRDM nozzle begins to leak, experience has shown that one of the first signs is the appearance of a small boric acid deposit crust at the base of the CRDM flanges where it emerges from the RPV head. To put nozzle leakage in perspective, a leak rate of 0.00001 gpm could deposit 10 cubic inches of boric acid over the course of a fuel cycle, which would be quite visible on a clean RPV head. Compared with unidentified leakage of a "tight" RCS at 0.03 gpm, it is apparent that unidentified leakage measurements cannot be used to detect early through-wall leakage at a CRDM nozzle. Figure 19 shows the unidentified leakage rate over cycle 13.JIts possible that the approximately 0.10-0.15 gpm increase in unidentified leakage starting in October 2001 is related to changing conditions at the-im-Win-no-z3.-u-bbratg~ a base leakage rate of 0.05 gpm from the total unmdEthf1 ik rate in higure MYthe maximum leakage rate from the CRDM nozzles did not exceed 0.15- 0.20 gpm at any point in time.

In sumnary, RCS leakage is at best a late-term indicator of leaking CRDM nozzles. Pybthtime.

RCS leakage can be used directly, there is a potential for advanced corrosion of the low-alloy steel RPV head.~

Predicted Leak Rates from PWSCC Cracks Dominion Engineering, Inc. Calculation No. C-5509-00-6 (reference 2.4) provides predicted leak rates from nozzles with PWSCC cracks. Key results are plotted in Figure 21. The basis for these leak rates are as follows:

Length of Cracks in Davis-Besse Nozzles 2 and 3 The longest crack lengths above the top of the J-groove weld determined by UT measurements are 1.1"Lfr nozzle .2and1.2" for nozzle 3. The longest cracks above the J-groove weld previously discovered in other plants with low observed leakage are <.0 inch.

Since the Davis-Besse cracks are longer than in other B&W design plants, higher leak rates would be expected.

Leakage From an Axial Crack in aPipe Cracks above the i-groove weld can be modeled as through-wall axial cracks in a straight length of pipe subjected to internal pressure. The model used for the results plotted in Figure 21 is based on crack opening areas from the EPRI Ductile Fracture Handbook (reference 5.6).

i2ak rates are computed from the crack opening area using models developed by EPRI

[Steam Generator Integrity Assessment Guidelines: Revision 1 (reference 5.7), and PWR Steam Generator Tube Repair limits: Technical Support Document for Expansion Zone PWSCC in Roll Transitions (reference 5.8)].

The prite-leak rates arew gpm for the .2"crack at Davis-Besse nozzle 3 for the 1.1" cr at nozzle 2, and about 0.012 gpm for the longest previously encountered crack. These results show higher predicted leak rates for the longer Davis-Besse cracks and that rate is significantly lessthanthe-01-0.2 gminferredfromthe ltetotalcteak measured unidentified leak rate.

f

-. UUIft At ~ -y EUF am IJCLLM Analyula W I ff Root Cause Analysis Report

• X A -

Effect of J-Groove Weld on Crack Qpening Area The finite clement model in Figure 15 shows the hoop stresses in the nozzle including the effects of welding residual stresses and operating pressure and temperature. The tensile hoop stress shown in the area of the J-groove weld will tend to open a crack at this location.

The finite element model shown in Figure 15 was modified by releasing the nodes on the plane of symmetry in the area of the axial crack as shown in Figure 22 and described in Dominion Engineering, Inc. Calculation No. C-5509-00-7 (reference 2.5). Teresultant (3 crack opening displacements shown in Figure 23 result in a predicted leak rate of over 1 gpm for the 1.2" long crackc at nozzle 3.

An additional load step was added to the model to simulate the loss of low-alloy steel from behind the J-groove we i emov o e constraint provided by this material resulted in the crac ccosing up somewhat and a predicted leak rate of about 0.8 gpm for the 1.2" long crack

! nozl 3._5 The above analyses illustrate the potential for leakage rates ranging from 0.025 gpm for the case of a 1.2" long axial crack in a straight run of nozzle material remote from the weld to 1 gpm or more for the case where weld shrinkage forces act on a long crack that extends 1.2" above the top of the J-groove weld.

Estimated Leak Rate Based on Boric Acid Deposits on RPV Head at 13RFO An alternate means to estimate leak rates for this condition would be from boric acid accumulations. Because of uncertainties in how much boric acid left the head region, this

~M ethod is only useful as a comparison. Figure 20 shows boric acid deposits on the RPV head

p~~irior to cleaning at 13RFO. The volume and weight of these deposits are estimated to he I1 5 ft3 and 900 lb at an assumed density of 1.25 g3cc( n.13), which is about midway between the density of boric acid crystals anid powderedfboric acid. Assuming that the average boron concentration in the primary coolant is 750 ppm, Figure 6-3 of Boric Acid Corrosion Guidebook, Revision I (reference 5.3) shows that 900 lb of boric acid deposits are the result of about 20,000 gallons of water. Assuming a linearly increasing leak rate over ao-aperif tfme. the maximum leak rate at the end of three years would be about 0.04 gpm. However, a substantial (but unquantified) amount of boric acid appears to have ben drawn out of the CRDM service structure by the CRDM ventilation system and deposited in containment (CTMT). Tbus, the leakage estimate may be low. More specifically, 10 ftO of wet boric acid were removed from the containment air cooler Rlenumri4ng 13RFO. While this is almost as much material as was found on the RPV head, it is important to reialize that the total unidentified primary leakage during cycle 13 was more than 0.1 gpm. It is not likely that all of this leakage was from nozzle cracks on the RPV head. Therefore, the estimate, 0.04 gpm, based on 900 lbs. of boric acid is a minirmum predicted leakage rate from the nozzle cracks.

Conclusions Reeardine Leak Rate From PWSCC Cracks The unidentified leakage during late 2001 attributed to CRDM nozzle leaks (0.1-0.2 gpm) is bracketed by the predictions based on leakage from an axial crack in a pipe (0.025 gpm) and the finite element analysis of crack opening area at the J-groove weld elevation after corrosion of the low-alloy steel material (0.8 gpm). Further refinement of the predicted leak rate is not possible due to the significant uncertainty regarding the exact shape of the crack in the nozzle wall and in tie 3-groove weld. Uovever, theanalyses clearly demonstrate the potential for significant increases in flow rate as the crack grows in length.

Report 3.0 Data Analysis

• 18 Root Cause Analysis Report 3.0 Data AnalysIs 16

3.2.3 Source of Boric Acid Deposits on RPV Head As shown in Figure 24 there were extensive boric acid deposits on the RPV top head surface at the start of 13RFO. These deposits are considered to have comne from two main sources, leakage from CRDM nozzle flange joints (uncleaned from previous cycles) and leakage from PWSCC cracks at nozzles 2 and 3.

Leakaze From CRDM Nozzle Flange Gaskets Figure 3 shows a typical CRDM flanged joint in a B&W-design plant. The joint consists of an Alloy 600 nozzle welded to the underside of the RPV head by a J-groove weld, a stainless steel flange welded to the Alloy 600 nozzle, a flange on the CRDM, two spiral wound gaskets, two 1800 split nut ring segments below the flange and eight bolts.

Leakage from the CRDM flange gaskets was experienced early in life at B&W designed plants.

Leakage from the flanged joints sometimes resulted in formation of concentrated boric acid on the flange with resultant corrosion of the originally installed low-alloy steel nut ring segments.

One such condition at ANO 1 in 1989 is described in the Boric Acid Corrosion Guidebook (reference 5.2). During the 1980's and 1990's, the gaskets were changed to graphite/stainless steel (SST) spiral wound gaskets and the split nut ring was changed to a corrosion resistant SST material.

Leakage from Davis-Besse CRDM Nozzle Flange Gaskets Prior to 13RFO It is reported that graphite/SST gaskets and corrosion resistant nut rings were installed at Davis-Besse over several outages.

• 6RFO Replaced 23 gaskets

• 7RFO Replaced 15 gaskets

• 8RFO Replaced 14 gaskets  ; k

• 9RFO Replaced 8 gaskets

• IORFO Replaced 9 gaskets/

It has been reported by Framatome that Davis-Besse is the only plant to have experienced leaks with the new gaskets and bolting materials. Specifically,

• 8RFO Replaced gasket on nozzle 66 (a minor leaker)

• lIRFO Small leak detected at nozzle 31 (was not repaired)

• 12RFO Nozzle 31 identified as leaker and repaired. Nozzles 3, 6, 11, and 51 identified as possible leakers and gaskets replaced

• FC3No flange leaks identified]

The largest of these leaks was from nozzle 31 at 12RFO. It is reported that steam cutting had occurred and that flange repairs were required in addition to just replacing the gasket.

Sorce of Boric Acid Delposits on Davis-Besse RPV Head It is considered that most of the boric acid deposits found on the Davis-Besse RPV head at 13RFO have come from leaking nozzle 3 with potential contributions from nozzle 2. The basis is that the vessel head was reported to be clean at 9RFO, significant boric acid deposits had appeared on the vessel head by 1IRFO, there were no significant gasket leaks prior to 1IRFO, experience in the industry does not suggest that leakage from the nozzle 31 flange gasket would have resulted in extensive deposits on the vessel head at 12RFO, and additional deposits appeared during cycle 13 when there were no reported flange leaks.

Root~~~~~

Reot30Dt

~ Report ~ ~ Cas Anl:i nlss*1 Root Cause Analysis 3.0 Data Analysis

• 19

Volume of Boric Acid Deposits on Davis-Besse RPV head at i3RFO The volume of boric acid deposits on the RPV head at 13RFO is estimated in a Dominion Engineering, Inc. calculation (reference 2.13). The approach used was to divide the RPV head into sixteen areas, estimate the depth of deposits in each area by reviewing inspection videotapes, and then calculate the weight of deposits in each area using the area, depth of coverage for each sector, plus an assumed density midway between that of solid boric acid and loose boric acid crystals. The worksheet calculations show an estimated volume of 11.5 ft3 and a weight of 900 pounds.

In summary, while the -case is not conclusive, it is probable that the approximately 900 pounds of boric acid deposits that accumulated on the RPV head are the result of leakage from the PWSCC crack at nozzles 2 and 3.

3.2.4 Corrosion of RPV Top Head Surface As shown in Figure 6, the RPV top head surface was corroded. During this investigation, attention was focused on boric acid corrosion as the source of the large volume of material loss downhill from nozzle 3. The potential for boric acid corrosion of low-alloy steel RPV heads has been known since the mid-1980's and there is no other plausible explanation for loss of this much material.

Historical Perspective on Boric Acid Corrosion of PWR Primary System Components The potential for boric acid corrosion of PWR primary loop components has been recognized since the plants were designed. Several incidents between the late 1970's and the mid 1980's led to the NRC issuing Generic Letter 88-05, Boric Acid Corrosion of Carbon Steel Reactor Pressurizer Boundary Components in PWR Plants (reference 3.1). EPRI issued the original Boric Acid Corrosion Guidebook in 1995 (reference 5.2), and the guidebook was revised in 2001 (reference 5.3). The later document includes summary descriptions of more than 100 incidents including corrosion of RPV heads, high pressure injection nozzles, reactor coolant pump studs, etc.

Previous Boric Acid Corrosion of RPV Top Head Surfaces Prior to the current condition at Davis-Besse, the greatest reported quantity of boric acid deposits on a RPV head was over 500 pounds at Turkey Point 4. These deposits were kept wet from a leak rate of less than 0.45 gpm (from a Conoseal leak). Corrosion on the RPV head was relatively minor, (approximately 0.25 inches depth).

There was corrosion of the low-alloy steel bottom head of a Combustion Engineering pressurizer at ANO 2 in 1987. In this case, a leak of about 0.002 gpm over less than six months time resulted in a corroded area about 1.5 inches in diameter and 0.75 inches deep. This leak resulted from a crack in an Alloy 600 sleeve associated with swelling of a failed Alloy 600 heater inside the sleeve.

Estimated Corrosion Rates at Davis-Besse Nozzle 3 The volume of material lost at the cavity between nozzles 3 and 11 was estimated to be about 125in' giving a weight loss of approximately 35 pounds.

Review of the sequence of relevant events in Attachment 2 suggests that the corrosion rate beRan to increasesignificantly starting at about 1IRFO and acted for a f6ur Mar period of time. With iherinaxiimumn corrosion lendoT ibbETA inches between nozzles 3 and 11 taverage corrosion rate would be about 2.0 inches/year. As a bounding assumption, if the rate increased linearly Root Cause Analysis Report 3.0 Data Analysis

• 20

about 4.0 inches/year.

with time, the maximum corrosion rate near the end of Cycle 13 would be The rates growing laterally from the main t t woulbe about half of the rates growing axially, or 1.0 to 2.0 inches/year.

5.3) summarizes the Figure 25 from the Boric Acid Corrosion Guidebook, Revision 1 (reference most of the data points available test data regarding boric acid corrosion. These data show that surfaces, or leaking for borated water dripping onto hot metal surfaces, impinging onto hot metal with the into a heated annulus, are in the range of 1.0 to 5.0 inches/year. This is consistent observed conditions.

final measured size of Further effort is ongoing to better define the corrosion rates based on the the cavity and thermal-hydraulic modeling being performed by the MRP.

Progression from Initial Small Leak to High Corrosion Rates nozzle 3) at Davis-An important issue is why some of the leaking CRDM nozzles (especially other B&W design Besse progressed to a high leak rate and corrosion while leaks at the six plants have remained small. Several possibilities were explored.

Crack Grows Loneer With Time time and this One possibility is that the axial PWSCC crack simply grows longer with above the J-increases the leakage rate. Prior to Davis-Besse, the greatest crack extension 1.1" above the groove weld was just under 1 inch. The longest cracks at Davis-Besse extend weld at nozzle 3.

top of the J-groove weld at nozzle 2 and 1.2" above the top of the J-groove Corrosion Begins Deen in Annulus and Increases With Time to the surface as It is likely that corrosion initiates deep within the crevice and progresses conducted by indicated by the corrosion at nozzle 2. This would be consistent with a test Guidebook, Southwest Research Institute for EPRI and described in the Boric Acid Corrosion below corrosion Revision 1 (reference 5.3). However, for there to be significant boric acid crystal deposits at the annulus the surface, there would have to be evidence of boric acid deposits around the outlet. Since other plants have not reported significant boric acid crystal annulus this model does not explain the difference.

Boric Acid on Top of RPV Head Acts as Incubator or Insulator rates can 4LaraWMt.tc experience with bolted flanges has demonstrated that corrosion by increase for conditions where leakg b traed water is retained ina bolted flag sources, head from other injsulafobnjbr aloose fitting band. Boric acid deposits on the RPV possibly provide the same type of iuch as leakage frotmCRDM flange joints, could to be a short term "incubator' as insulation on flanged joint. However, this is only expected create its "head start" since leakage of borated water from a PWSCC crack will eventually from boric acid own boric acid deposits at the annulus which would behave the same as flanged joints.

Morphology of the Affected Area as Damaee Proresses Based on the investigations of the root cause team, it is clear that leakage from PWSCC cracks Or to the material loss adjacent to nozzles 2 and 3. These leaks led to was Xnecessay much nozzle 2 and local environmental conditions that produced modest material loss around of the leakage was to provide a more extensive degradation around nozzle.3. The main effect leak path. Provided boric acid solution that concentrated through boiling heat transfer along the through a crevice that sufficient levels of oxygen are available-either directly or remotely relatively high corrosion mechanism-the concentrated liquid boric acid solution may cause 3.0 Data Analysis

• 21 Root Cause Analysis Report

secondary effect of the corrosion rates up to on the order of four inches per year. A possible flow-related mechanisms.

leakage is to enhance the material loss of the low alloy steel through particle impingement and These mechanisms are flow accelerated corrosion (FAC), droplet erosion, and potentially steam cutting.

degradation and the lack Given the current limited experimental data applicable to the observed thermochemical and of existing detailed analytical calculations of the thermal-hydraulic state the exact progression environment along the nozzle leak path, it is not possible to definitely along the leak path-of mechanisms that led to the observed material loss. The environment the axial PWSCC crack from the primary system pressure inside the CRDM nozzle, through or cavity on the periphery extending above the top of the J-groove weld, up through the annulus surface-is the result of head of the nozzle, and then out to the ambient pressure above the top and acceleration pressure frictional complex processes such as critical two-phase flow, two-phase acid solution concentration, drops, boiling heat transfer, boiling point elevation due to boric convective heat transfer on oxygen and hydrogen transport, various electrochemical processes, head materials. Therefore, a the surfaces of the head, and conduction heat transfer within the physical mechanisms with a detailed description of the damage progression including the precise would be speculative.

quantitative breakdown of the relative importance of each mechanism progression are known with However, the degradation modes on the two extremes of the overall the possible modes of reasonable confidence, and some conclusions can be made regarding is associated with the lack of degradation in between these two extremes. The first extreme of the leaking CRDM nozzles in material losg arzt extreey.mall leak rates observed for most teFor these extreely low leakage rates (on the order o toeleaking of the principal flashing location, flow completely vaporizes to steam immediately downstream the gap between the nozzle and most likely at the exit of the PWSCC crack. The result is to keep In addition, the small velocities head dry, precluding high rates of low alloy steel material loss. from being active.

associated with the extremely small leakage preclude the flow mechanisms with the large cavity located The other extreme of the degradation progression is associated proceeded by the classic adjacent to nozzle 3. For this cavity, it is dear that the degradationsolution concentrated through f boric acid corrosion mechanism associated with liquid boric acid the ambient atmosphere. The magnitude boiling and oxygen directly available for corrosion from ohig leak rate of nozzle 3 is sufficient to of the boiling heat transfer associated with the relatively walls of the cavity.

cool the head enough to allow liquid solution to coverthe PWSCC crack above the top of In between these two extremes, increase in the extent of the axial

3. It is likelythate the J-groove weld resulted in increasing rates of leakage for nozzle provided by the leak increased to degradation Eoceeded at relatively low rates until the cooling the annulus between te wiere a concentrated liquid solution could exist high enough in mechanism enhanced to some the nozzle and the head in order to support a crevice corrosion dissimilar metal couple 0! the degree be a galvanic corrosion mechanism associated with the nozzle and low alloy steel a identify a progression of While it is not possible within current knowledge to definitively is apparent. Linking the corrosion mechanisms, the overall effect and cumulative timeframe understanding), it is corrosion mechanisms that were described above (with some supplemental possible to construct a "viable" progression of events.

ROOt ususe analysis Repan 3.0 Data Analysis

• 22

Staee I - Crack initiation and Progression to through wall First, based on the body of knowledge available, a crack initiated in nozzle 3 at around 1990

(+/-3years) due to PWSCC. The crack grew at a rate consistent with industry data, progressing to a through-wall crack that penetrated above the J-groove weld in the 1994 to 1996 time frame. At this point the RCS leakage would have been miniscule, and in no way detectable by any currently installed leakage monitoring system.

Stage 2 - Minor Weevage / Latency Period Leakage would have entered the annular region between the Alloy 600 nozzle and the low-alloy steel base material of the RPV head. However, the interference fit that was initially expected in nozzle 3 is composed of only about 5 percent of actual metal-to-metal high point contact. At its tightest, the rest of the interface is essentially an annular "capillary" flow path.

Even if the flow path could not actively leak, it would still be permeated with moisture from the newly developed crack. With addition of moist boric acid in this bi-metallic annulus, several forms of boric acid corrosion are possible, in addition to galvanic attack. These biN corrosion mechanisms wouid opena nnular In g it did not previously exist, and allow leakage flow to the surface. If the Bob head had been initially clean, and if a timely 100%

bare head visual inspection had been completed, the leakage would most probably have expressed itself within a short time as the classical "popcorn" crust of boric acid deposits.

This would have been apparent within one or two fuel cycles from the time the crack progressed through the nozzle wall and would not have been accompanied by large scale corrosion of the low-alloy steel. However, at Davis-Besse, the "popcom" manifestation was not yet observed, and its detection could have been obscured by previous flange leakage deposits.

Given time, the crack continued to grow, leakage increased,and the annulargap increased in width.Withan ever iap, ox ntire ly dfromentcriw4g~I arijijlus, thus accelerating boric acid in the gap and diminishing the relative iiporftfe ofgMvanic corrosion. However, due torestrion oxygen and moisture, corroion mechanisms havenot been fully accelerated. As observed at other facilities (and also in Davis-Besse nozzle 1,the least advanced of the leaking nozzles), there was widening of the annular gap and development of flow "channels" in the annulus leads to the near certainty that the principle flow resistance would have been due to the dimensions of the N,- crack, and not due to any restriction offered by the annulus. This is also supported by the reIatively low crack growth rate (i.e., <0.2 inches/year with m icrosc c oping width) compared to documented borc acid corrosion rates on the order of 0.02 to 0.08 inches per year for similar geometry as cited in the Boric Acid Corrosion Guidebook (reference 53).

Since the growth in annulus width tends to occur over a broad length around the annulus, the anulus flow area increases faster than therack flow area. Thus, the crack dimensions dominate the flow resistance, and the majority of the pressure drop occurs as effluent traverses the crack. Based on the reactor coolant enthalpy at the RPV head, approximately 45% of the reactor coolant that espcas from the crack flashesupon dischrg the rest is immediately vaporized by heat transfer from the metal surfaces at this stage. Thus, boric acid is both atomized with the steam and deposited as molten boric acid on the surrounding surfaces, with moisture escaping as steam.

Stage

-D 3 Dep Annulus Corrosive Attack Toward the end of the latency period, the gap has widened and crack leakage has increased.

Oxygen penetration is ever more pervasive since the flow area in the annulus will very likely RoCase Anlyi Report 3. DaaAlIei Root Cause Analysis Report 3.0 Date AnaPPIS a X4

more than offset increases in leakage due to crack growth. This would cause annulus velocity and differential pressure to decrease, allowing greater inward penetration of oxygen. With increasing oxygen levels deep in the annulus, it is probable that a small amount of material would be preferentially corroded in the vicinity of the crack, as evidenced by test EPRI-6, modeling of leakage into annular gaps original Boric Acid Corrosion Guidebook (reference 5.2). This test was characterized as having excessive oxygen in the supply water, which would be similar in effect to having oxygen suppliedbyiliernativermeans (ie.,m the top downward). The net effect is that the corrosion rate can be substantially greater in areas of greater velocity. The velocity increase does not need to be sufficient to cause scrubbing of beneficial oxide las (i.e., erosion-corrosion) rather, t nain a f supply of new reactive oxidizing ions in the boundary layer near the corroding metallic surface. The expected pattern was found at nozzle 2.

Stage 4 - General Boric Acid Corrosion Progression to this stage is dependent on crack leakage rate. With high leakage rates, the annulus is flooded with an ever increasing amount of moist steam, partially flashing as it exits. Due to the fact that the annulus still exists, basically in the same geometry, any effluent is directed vertically upward. A large amount of discharged boric acid has already accumulated in the area around the nozzle. With increased leakage, heat transfer from the surrounding metal is no longer sufficient to immediately vaporize the portion of leakage that does not flash (due to its own initial enthalpy and pressure reduction) as it exits the crack. In effect, the metal surface temperature is being suppressed by the cooling effect of the large heat flux required to vaporize the now lingering coolant. The principle characteristic of this stage is that the annulus begins to overflow or expel unflashed.liquid. This has the effect of allowing a greater area to be wetted underneath the accumulations of boric acid.

Rapid general boric acid corrosion on the wet, oxygenated surface of the low-alloy steel RPV head is now in progress. Even reductions in reactor coolant system boric acid concentration toward the end of the operating cycle would have little effect because the concentration at the metal surface is continuously re-supplied by the boric acid that was previously stored. The wetted surface area is dictated by the leakage rate as determined by crack size and system pressure, the ability of the RPV head to vaporize the liquid via conduction of heat from the interior of the RPV (i.e., it would be vaporized as it runs), and is also affected by the character of surrounding deposits. However, s ple ilatioIsatfull temperaturend pressure indicate thattaffected area would be consistent WI he amount of leakage that appears to have occurred. Further, since the wetted area would be the result of liquid overflow, it would be expected to be predominantly downhill from the nozzle, leaving the uphill side much less affected, and is the pattern observed at affe-This fizzle 3.

As general corrosion progresses, it would tend to carve out a "bowl" of corroded (or, oxidized) material. Initially, this bowl would gradually increase in surface area as the leak rate from the crack increases. The area in the middle, having been wetted longer, would be slightly deeper. With a sufficient flow rate, the bowl could begit fill with a saturated boj acid solution. Thelsfii ton tetr and consistency in the bowl could be anywhere between that of dilute boric acid (-watery at 212F), to that of moist, molten orthoboric acid (H3B03 ) (viscous at <365SF). As the bowl defpens, thermalcefeits would limit the widening of the bowl, even as leakage incrementaly increases. As the bowl deee thare w b lesser need for as much projected surface area to transfer the heat. This is because the thermal resistance to heat transfer would continually decrease as the corrosion depth Root Cause Analysis Report 3.0 Data Analysis

• 24

approaches the stainless steel cladding. Further, as the bowl attains a liquid level, lateral heat transfer from the sides would increase the steaming rate, and tend to govern level. A third self-governing effect would be that if leakage increased, decreasing the boric acid concentration in the bowl, the boiling temperature would decrease. This would increase heat transfer (and vaporization rate) by increasing the temperature difference. Heat transfer would also increase due to decreases in viscosity. Thus, with a relatively constant level, the corrosion surface slope might well be expected to be very steep. Finally, when the liquid at the corrosion front reaches the of the stainless steel, downward progression ends. At this point the wetted surface would stop its vertica teId tautg.

The height icaid wou d teto tendiquidincrease with further increases in leakage, unless the increases in diameter due to outward corrosion were sufficient to offset the increases in leakage. This represents the as-found condition of nozzle 3, with steep walls and an undercut nose on the downhill ide.

Throughout the majority of this process, being predominantly top-down, the annulus could remain somewhat intact until the approaching general corrosion front overcomes it. This is because the annular region would remain somewhat protected by the upward flow of de-oxygenated water and steam. Thus, flashing effluent from the crack would be directed upward and out of the annulus while the annulus is in place. However, as soon as the low-alloy steel corrosion front is below the elevation of the crack, the effluent would be directed laterally. This would undoubtedlv chantthedgree of atomization of boric acid and affect the particle size of the boric acid ca late in the roess. If this sequence is accurate, the point at which the corrosion depth reached the crack location mi hav beleoan

__a00 May 2001 . A.th t that tm,fe. ,I time the cleaning .,rp frequency of containment air coolers (CACs) due to boric acid foulin~ eased.

Although the above progression is, at best, a likely construction, it provides an example of a viable path that could explain the evidence.

Boric Acid Formations on the RPV Head The following is a general description of phase changes that boric acid is known to undergo as temperature is increased. This information is being used as part of the MRP modeling effort to develop a consistent model, including boric acid morphology as the corrosion progresses.

When boric acid is left behind by boiling water, it is first deposited as orthoboric acid (H3B0 3).

Although solubility of this material is limited at cooler temperatures, near the melting point of 3650F, the solubility in water is infinite. Thus, as boric acid is deposited on the RPV head, it would tend to increase in temperature from that of saturated water (2120 F) to 3650F, at which point it is a viscous liquid. In this form it will tend to flow, causing the formations that have been noted. However, even before all the free water is driven out, at around 3400 F the H3BO3 begins to dehydrate to metaboric acid (HB0 2 ). This is a white, cubic crystalline solid, and is only slightly soluble in cool water. Metaboric acid has a melting point of 4570 F, and may tend to form a "crust" on the deposits and formations of orthoboric acid. With further application of heat, the HBB 2 will further dehydrate at approximately 5720 F to tetraboric acid (H 4 B4 0 7 ).

Tetraboric acid is a vitreous solid or white powder, and is water soluble. At the temperatures encountered on the RPV head, all of the above forms can be found, depending on age, contact with-he RPV head, and local temperature.

When boric acid accumulates at a leaking nozzle, some flowing of the orthoboric acid would be expected. Boric acid in the cavity formed at nozzle 3 is most likely highly hydrated H3 B03 ,

Root~~CasnlssRpr 30Dt nss*2 Root Cause Analysis Report 3.0 Data Analysis a 25

since moisture is continually supplied. As it was expelled or extruded from the cavity, it would flow, and undergo the above transformations. These transformations would drive off some moisture that could conceivably contribute to corrosion, but this is expected to be a trivial effect.

However, experimental data to predict the extent of motion or the degree of corrosion has not been located.

When nozzle 3 was removed, it was reported from the field that the column of boric acid surrounding the nozzle was porous, with winding tube-like channels (at the time still believed to be carbon steel due to the rust color). A small cavity was below the material, where liquid boric acid of lesser concentration presumably drained or washed out during machining for the original aborted repair attempt. The boric acid remaining would have solidified during cooldown, but would be expected to be full of voids and steam tubes to allow venting of the flashing leakage.

The appearance of other formations is consistent with expectations of the transformations and crusty appearance.

3.3 Investigation of Lead Indicators This data analysis section provides a discussion of plant operational and equipment issues that provide possible lead indicators for the subject condition.

3.3.1 rimelIne An early step in the root cause evaluation was to establish a timeline of key events. The timeline was revised as the data analysis proceeded and the current evolution is shown in Figure 26.

While the timeline was created based on the information that follows, it is presented first so that it can serve as a useful guide to help focus subsequent discussions.

The timeline summarizes the following information:

• Years from 1995 to present

• Operation from Cycle 10 to Cycle 13

• Refueling outages lORFO through 13RFO including mid-cycle outage during Cycle 12

• Condition of the CRDM flanges, RPV head flange and RPV top head surface at each outage

• RCS unidentified leakage (discussed previously)

• Containment air cooler cleaning operations

• Containment radiation monitor performance and filter cleaning

• The estimated weight of boric acid deposits on RPV head

• Dates of key industry findings and initiatives relative to RPV head condition 3.3.2 Sequence of Relevant Events Attacbment 2 is a table of events relevant to the subject condition. This table was used as input to creating the timeline, the logic chart of key decision points and the other potential lead indicators.

Figure 27 is an events and causal factors chart outlining key decision points and other potential lead indicators.

3.3.3 CRDM Flange and RPV Head Inspections during Refueling Outages In the early 1990's, several B&W design plants began cutting openings in the service structure surrounding the RPV head to afford better access to the center top of tbe RPV head for inspection and cleaning. Fiamnatome ANP (Framatome Technologies, Inc. at the time) provided proposals to Davis-Besse over a period of several years to perform this work. However, Davis-Besse has not R.or Root~~~~

~ ~ ~ Cas A.lyi .. _at _nlss*

Root Cause Analysis Report 3.0 Data Analysis a 26

through the mouse installed these openings. Without these openings, the head visual inspection finite distance. The holes is hampered in that the pole-mounted camera can only be inserted a this inspection curvature of the RPV head impedes seeing the top of the RPV head with of the inspector arrangement. Based on review of video by the root cause team in the presence the camera, the curvature during 1IRFO, the optical illusion created by the short focal length of the top of the RPV of the RPV head and the close proximity of the insulation (nominally 2") at the RPV head had been head appears to have potentially led inspectors to believe that the top of ayfrmthe inspected; however, the mp inmay have been approxiimatiyW-2 nozzles ciirofte RPV head.

cameras that is Framatome provides a tool to inspect CRDM flanges for leakage with two look under the lowered down between adjacent flanges. The lower camera is angled up to view of the flange interface.

flanges for boric acid deposits. The upper camera is a straight ahead this height, the lower The housing for the cameras is designed to rest on top of the insulation. At and upper cameras are properly positioned relative to the flange.

Prior to 1996 flange gasket During this time frame, B&W had recommended replacing the original CRDM all the B&W that with an improved graphitedSST spiral wound gasket to fix leakage problems gaskets by 1996.

design plants had experienced. The plant replaced all of the CRDM flange leaking flange gaskets Davis-Besse developed a priority ranking system and replaced a number of system was each outage based on outage duration rather than 100% repair. The ranking developed by the RCS engineer and is as follows:

Ranking System Developed by the RCS System Engineer Category Descripon No.

- ~Weepage visible above nozzle at motor tube (hM) 1 interface and/or below the nozzle at the nut ring (N.R.) joint Minimal leakage at M.T. and/or N.R. to nozzle 2interfaces (with one or more runs)

Moderate leakage at M.T. and/or N.R. to nozzle 3 interfaces (with appreciable boron deposits adherent to the flange)

Heavy leakage with boron bridging adjacent flange surfaces

- Excessive boron accumulations on the insulation below the nozzle In 1990 (6RFO), gaskets were replaced in 23 CRDM flanges. Figure 28 shows the leaking head.

flanges. There are no specific records indicating an inspection of the RPV on the RPV hel. The JnJ99 (7RFO), the RCS engineer reported an excessiyg. amount of boron by the closure bolts.

boron flowed through the mouse oles and stoppe ion the RPV head flange 15 were repaired.

The CPRDM flanges were inspected and 21 were identified as leaking and

__ .,nata Analysls

• 27 Root Cause Analysis Report --- --- - -,--a - --

Figure 28 shows all the leaking flanges identified in 1991 and the flanges that were justified for use-as-is.

In 1993 (8RFO), an inspection of the RPV head was performed, shown in Figures 29-32. In Figure 29, the boron deposits are dripping through the gaps in the insulation forming stalactites.

The boron deposits started forming stalagmites on the RPV head. Figures 30 and 31 show more boron deposits coming through gaps in the insulation and clinging to the side of the CRDM nozzles. The boron deposits in Figure 31 were reddish brown in color. The boron deposits on the RPV head in Figure 32 do not exhibit a clear picture of the source ofleoae (CCRDM fgei Based on the results of the head inspection, the RPV head and flange was cleaned with deionized water. The effectiveness of the cleaning could not be verified in that the RPV head had already been returned to the RPV. A cleaning effectiveness inspection was recommended as a follow-up activity for the next outage. The CRDM flange inspection revealed 15 leaking flanges as shown in Figure 28. Framatome generated a non-conformance report (NCR) that noted degradation to the flange sealing surface found during the repair of CRDM nozzle 31. The corrective action taken was to perform flange surface polishing and gasket replacement. The 1993 NCR also recommended that the flange surface be machined if further leakage occurs.

In 1994 (9RFO), the CRDM flanges were inspected; however, no records have been identified indicating a visual inspection of the RPV head was completed. Performing a video inspection of weep holes was an activity in the outage schedule. There were no boric acid deposits interference problems with inspection equipment reported. Eight CRDM flanges were identified as leaking and repaired during this outage (Figure 28) 1ORFO (1996)

Figure 20 provides an overview of the boric acid deposits on the RPV head from lORFO to 13RFO.

In IORFO, the remaining ten flanges without the new gasket material were upgraded. The 1ORFO's head visual inspection under the insulation, the majority of the RPV head was inspected except for the top center. A couple of nozzles are shown in a couple of background frames (Figures 33 and 34). These frames are approximately two to three nozzles away from the top center of the RPV head. The most consorative assumption that can be made from these figures is that boric acid extended from behind nozzles 2, 3, 4, and 5 to the bottom of the insulation, e assumed footprint of the boic acid is shown in Figure 20. Comparing Figure 33 and Figures 29 and 30, the underside of the insulation in 1996 does not show crusted boron deposits or stalactites hanging.

The boric acid was Iwdeand white. Bc:ic acid seemed to be flowing toward the mouse holes. The boric acid was very thin at the front edge with powder and small clumps of boric acid on top. Because the mouse hole locations were not periodically noted during the visual inspection, the location of this flow path is uncertain. However, based on future evidence, it is assumed to be the southeast quadrant of the RPV head. The remaining area of the RPV head was clean with speckles of white boric acid deposit. Figure 35 show a typical photo of the condition of the RPV head during this inspection.

lIRFO (1998)

'--zle 31 was identified as having a minor flange leak using the following criteria: 1)there were italactites hanging from the flange, 2) there was no boric acid bridging to adjacent flanges,

_.u .~t _nlyi . . ^

at Cause Analysis Report ax Dam Analys is a za

and 3) there was no rust present on either the flange or the split nut rings. Initial and follow-up review of the leaking flange by Davis-Besse Plant Engineering indicated that no immediate repair was required, and that this drive should be inspected during 12RFO and repairs made as required.

Frarnatome reiterated the recommendation from 8RFO to machine the surface of the nozzle 31 flange if further leakage occurred. Unidentified leakage data was reviewed for the past several cycles. With the numerous flange leaks present in both 7RFO and 8RFO, the highest unidentified leakage was approximately 0.3 gpm in cycle 7 and 0.4 gpm in cycle 8. The unidentified leakage in cycle 11 averaged 0.05 gpm. No Technical Specifications were exceeded even when the highest flange leakage was present. During the visual inspection of the control rod drive flanges, no interferences from boric acid accumulation on top of the insulation were identified.

During lIRFO, boric acid deposits were identified fiiigmu~t of the mouse holes in the i67s 'tat -The boric acid was a reddish rusty color. The RPV of tlheeRPVhbad 'fla-nge.

flange was decon itn6p1itor iiimspetion of the RPV head.

The Service Water System Engineer conducted the RPV head visual inspection during 1 IRFO.

The engineer worked with a Framatome crew using a pole-mounted camera to inspect the RP.

head for "crackcs in nozzles and degradation adjacent to~eiEoiziel During the head visual inspection, the center nozzles were again very difficult to inspect through the mouse holes using available techniques. The engineer noted white streaks on the nozzles; however, there was no boron hanging from the insulation. The engineer noted in a recent itd travel of the droplets as oppto what would be expected (downward traveTFe w el oThe droplets was noted on several nozzles and attributed to ventilation flow. Boric acid was present in fist-sized clumps behind nozzles 9 and 137orc acid was collecting on the RPV head as shown in Figure 37.

Boric acid seemed to be falling from the top of the RPV head and collecting behind peripheral nozzles especially in the northwest and southeast quadrants. During this outage inspection, the boric acid was noted to bea mix of white and red deposits. Upon identfication of red, rusty boric acid mixed in with white boric acid on the RPV head; the engineer worked with the Framatome crew to vacuum the RPV head and remove as much boron as possible. The equipment available to do the work and the limited access to the very top of the RPV head limited the removal process. During the removal of boric acid from the RPV head, the boric acid was noted to be brittle and porous. Other than these areas of accumulated boric acid, the RPV head was basically clean. Due to the limited inspection capability, the video evidence suggests that the most conservative estimate of the boric acid present would be to assume that behind nozzles 6, 7, 8, and 9 the boric acid extends to the bottom of the insulation and tapers off to the back of the next nozzle location. The approximate footprint of boric acid on the RPV head is shown in Figure 20.

12RFO (2000)

During the CRDM flange inspection, the upper camera was not positioned properly at four locations. The interference was attributed to a pile of boron on top of the insulation. The boron was a red, rusty color and hard. Normally, boriound on top of the insulation is a loose powder and in ieioior range from whitane to y.nvdpg onvideo and interviews).

The boron pile encountered during this inspection was hard an could not easily be pushed to the side with-the Framatome inspection tool. The underside of nozzle 3 was caked with red boric acid deposits. The inspection of the flange interface was accomplished by lifting the lower

^l ..

A

- - e-.- a-_2-ax UaTA #knelysir, a "

Root Cause Analysis Report

camera to see the upper flange interface. The interference locations, as shown in Figure 39, were identified in the center of the following nozzle blocks:

• Nozzles6, 15, 11,and3

• Nozzles 11, 27,32, and 16

• Nozzles 15, 31, 27, and 11

• Nozzles 1, 3, 7, and 4 Based on the CRDM flange inspection, nozzlesJ3I6, 11, 31 and 51 flange leaks were repaired.

The CRDM flange on nozzle 31 was machined to remove a steam cut from the seating surface.

The Service Water System Engineer that conducted the RPV head visual inspection in the previous refueling outage requested to inspect the RPV head during this outage. To prepare for the inspection, he interviewed design and mechanical engineers familiar with this component and the industry issues associated with it. Another contributing factor for the Service Water engineer to request to assist in the inspection of the RPV head was the fact that the RCS engineer was new to the Davis-Bess p.Qywer plant. By assisting in the inspection, any changing conditions of the borc acid on the RPV head could be easily identified based on his experience in the 1998 inspection.

As shown in Figure 36, boric acid had accumulated on the RPV headflan ehind the studs flowing out of the mouse holes in the southeasjguat. The boric acidillad a red, rusty appearanci Co~use holes in this quadrant were significantly blocked with boric acid deposits. With the studs in place on the RPV head flange and the accumulation of boric acid, the inspection through the mouse holes was significantly hampered. The engineer requested that the RPV flange be decontaminated and the studs removed to afford a better inspection. This work was completed. Boric acid on the RPV head was identified as an outage issue.

The RCS engineer supervised the cleaning effort, which entailed the following.

0

• Pressurized (approximately 200 psi), dernineralized water heated to 175 F.

• Water was sprayed on the boron deposits through the mouse holes and ventilation duct openings.

• Estimated volume of water 100 to 600 gallons.

'

• An inspection video was required post cleaning.

. If the video revealed boric acid remaigqng on the RPV head, the cleaning steps were expected l_ betorpepated-The RCS en eer ackflQ edgesft th;-eleaoWg was not 100% successful and some boric acid e wee efeh onhe hea. The engineer ste he was running out of time to continue cleaning the YPV head (the RPV head was scheduled to return toie1PV duriiingte net shi). ELI haeflI~nt concurreclthat no additional time and dose should be spent because further osuccessl results and the results were believed to be acceptable. Radiation Work Permit (RP)2000-5132 package was written as a tool to control radiological exposure for cleaning boric acid from the RPV head on April 6, 2000. The RWP identified 30 man-hours and a 100 mRem dose was estimated for the work. There were 28231 man-hours and 1611 mRem expended for cleaning the RPV head.

No written evaluation was a-pe r _oalowjKhe boric acid to remain on the RPV head. At this in time, the modification tocut the openings in the service.structure was scheduled for the point next outage. With these openings and a more aggressive cleaning technique, the RPV head could

-art no& Aualupt-. -an Root Cause Analysis Report 1W.V WGaud R.W844173

• Gu

be completely cleaned of the boric acid deposits and inspected. The amount of boric acid deposits left on the RPV head can not be estimated.

13RFO (2002)

During the CRDM flange inspection, the camera again encountered a boron pile in the vicinity of nozzle 3 making the inspection of the underside of the flange difficult. No flange leakage was identified during thi outage indicating that previous repairs were successful.

The engineers responsible for inspecting the CRDM flanges reported boric acid deposits flowing out of the mouse holes and piled up to 4 inches high in the sothVea fl and exten ad flange. The boric acid depo quadrant were hard-baked whereas the deposits around the remainder of the RPV head flange were loose. During the inspection of the RPOhead under the insulation, signfficant boric acid was eountered in the southeast quadrant. In the remaining quadrants, significant piles of boric acid were encountered two to three nozzles in towards the center of the RPV head as shown in Figure 24. The deposits were hard, rus desits and were a mnixtueofrddi sbrown and white deposits. The deposits were removed by hydrolasing, which opera 2,000 psi.

Documentation Available for Review RPV Head Flange RPV Head Accumulation Under Insulation Above Insulation Prior to 1996 PCAQR 91.0353 Video lORFO (1996) PCAQR 96-0551 Video I IRFO (1998) Pictures PCAQR 98V767 12RFO(2000 CR 200040782 CR 2000-1037 CRDM Flange

) Pictures Video Inspection Video CR 2002-00685 CR 2002-00846 CRDM Flange l3RFO (2002) Video Video Inspection Video PCAQR: Potential Condition Adverse to Quality Report CR: Condition Report 3.3.4 Containment Air Cooler Cleaning The CAC system is an engineered safety feature and is provided, in conjunction with the-Containment Spray System, to meet the requirements of 10 CFR 50, Appendix A, General Design Criteria (GDC) 38, Containment Heat Removal. It consists of three separate tube/fin fan-coolers, one associated with each of two trains. Two of the three coolers are associated with each of two safety related trains. The third cooler is a swing cooler (spare) and can be aligned mechanically and electrically to take the place of either of the other two coolers. Service water, ultimately supplied from Lake Erie, is supplied directly through the cooling coils to remove heat from Containment under both normal operating or accident conditions. The system has safety functions to cool CTMT during postulated accident conditions such as Loss of Coolant Accidents and Steam Line Breaks. During postulated accidents, operating in slow speed, each CAC is Report 3.D Data Analysis

• 31 3.0 Data Analysis a 31 Root Cause Analysis Report

designed to move 58,000 cfmn. During normal operation, the CACs are operated in high-speed and are available to remove normal process heat in CTMT, maintaining a maximum air temperature of 1200F at the inlet of the CACs. The three CACs are located in a row, next to each other, on the 585' elevation in CTMT. All CAC inlet air is drawn in at this location through the sides of the tube banks by the fan. The cooled discharge air supplies a distribution network inside the secondary shield structures, the reactor incore instrument tank, and RPV regions. The outside surfaces of the tube banks are readily visible from outside the coolers.

During operation, the service water supplied to the CACs is typically between approximately 400 F and 75F. Being substantially cooler than CTMT, depending on CTMT humidity, the CACs remove water from the CTMT by condensation on the fin surfaces. This action would be expected to vary throughout the year. Both the dampness of the fin surfaces and high volumetric air flow rate cause the CACs to readily acquire a loading of boric acid particulate material, if it is present. In addition to collecting on the CAC cooling fins, boric acid accumulations have been observed and removed from CAC ductwork. Foreampl..apiml7gallens (lfcubic feet) of wet boric acid were removed from the C Fouling of the ACcanbe trend remotely by indication of plenum pressure. At Davis-Besse a fouling condition oc n 922 (P*AQl 2?Q .ldue toa leakin flange on the primary side of a steam generator. Inspection of the CACs at that time revealed that the CACs were evenly fouled white onc acid, which was readily cleaned with either steam or hot water sprays. After repair of te the cflanoeuleiing of the CA .ss~d.and no further cleanings (for rapid boric acid fouling) were needed for5Y~l l TS In October of 1998, there was a concern over the configuration of the pressurizer code safety valve discharge piping configuration. In brief, the safety valves discharged to a piping tee, with a rupture disc on each branch. Any weeping from the safety valves would be contained by the rupture discs and conducted to the pressurizer quench tank through a small drain line, quantified, and not counted as unidentified leakage. The tee was used with the design assumption that both rupture discs would simultaneously relieve if the code safety valves actuated. This would produce equal and opposite piping reactions, canceling each other to produce a zero net bending moment. After it was postulated that one or the other, but not both discs might relieve, it was realized that the design could result in a very large moment. Short term remedial action to resolve that concern involved deliberately failing the rupture disks. In November of 1998, PCAQR'98-1980 identified that fouling of the CACsappeared tobeS mng, based on plenum pressure trends, coincidingwth increased JeakgeR the pressurizer safety vaves. Cleaning of the CACs continued, with 17 cleanings ein&needed between Novmber 1 a May nd 1999.2 During the MAyJ 999 mid-cyWcle outae, a.piessurzer codsfet valye Ripin&gmodification resolved that isiHowever, two subsyeue t,CACS k.niv e stil,1,mqnee 9i99 an another in Jul 1999. Although the boric acid was generally reported to be white, a PJ written post-job critique was located from the July 1999 cleaning that indicated a "rust color" P was noticed "on and in the bo ,WWn~g go*,_CAC ngpa, L

\Afer 12RFO, in June 2000, CAC plenum pressure again began to decrease (CR 2000-1547),

iequiring resumption of cleaning. This was followed by five total cleanings in June, August 9ctober and December of 20D(,Cleanings continued in 2001, with four mbr8() in Jainur Feebruary, March, and ologA!y~n clea &er.bz balance of the otiratln ,

During 12RFO, some CRDM flange leakage was repaired. Following 12RFO, but before 13RFO, it was not known whether those repairs had been fully successful. Therefore, the CAC AI VEtS flUI5!5 S Analysis Report Root Cause Analysis Report U. Data Analysis

• Liz

cleaning could potentially have been attributed to CRDM flange leakage. However, 13RFO inspections revealed that the CRDM flange repairs in 12RFO were apparently successful.

Further, earlier experience with leaking flanges (pre-1992, and 1993 - 1998) did not result in the need to clean CACs. Therefore, CRDM flange leakage can now reasonably be ruled out as the cause of the cleaning of the CACs after 12RF0.

Attributing the need for CAC cleaning to leaking CRDM nozzles is plausible. but has sexal 1 oijionsistncies th-at W-oul e .bpl he most prominent is that if nozzle leakage Tn ti g td frm= li ntilFebruary 200Z, why did the need to clean CACs end in May 2001? The answer to this question can only be postulated and will not be known u a different source of leakage is later identified. However, there are several potential explanations. These are related to CTIMT humidity vs. SW temperature over the period, to reduction in RCS boron concentration at the end of the fuel cycle, but also possibly to chans in the morphology of the nozzle leak. For example, if the corrosion cavity at CRDM nozzle 3 enlarged suliuially during the last half of the exitveloity) or the boric acid dip contained the leakage differently, the nature and amount of particulate matter might have clinged. Larger panricles might settle andinotbe subject to ingestion by the CACs. (The laieiiheory has some anecdotal support based on observations that the boric acid dust on horizontal CTMT surfaces was more granularirT 13RFQ.Las 9pp ofnto fipe wwnapQ Eie e outages). However, this conjecture is subject to the similar pitfalls of the earlier (disproven) hypothesis that CRDM flange leakage (circa 1999) was different from CRDM flange leakage (pre-1992) and was therefore able to cause CAC fouling.

In summary, there was circumstantial evidence that CAC fouling was related to nozzle leakage prior to 13RF0. Because of variations in plant conditions, CAC fouling, by itself, could not be directly correlated with CRDM nozzle leakage.

3.3.5 ContainmentRadiationMonitor RE4597 Observations &FilterPlugging Radiation monitors RE 4597AA and RE 4597BA are two identical CTMIT air sample monitoring systems, each with three detection channels and two sample locations. The monitors provide two of the three RCS leakage detection methods described by Reg. Guide 1.4 and required by TS 3.3.3.1 and 3A.6.1, namely CTMT particulate and noble gas activity detection. These parameters are monitored because of their sensitivity and rapid response to leaks in the Reactor Coolant Pressure Boundary. Detection of radioactive iodine is also provided. A continuous sample drawn from CTMT passes through a fixed particulate filter, an iodine cartridge, and a pump. The sample then passes through a noble gas chamber and is discharged back to containment atmosphere. The containment radioactive gas monitor is less sensitive than the containment air particulate monitor and would function in the event that significant RC gaseous activity existed from fuel cladding defects. The normal sample location of RE4597AA is approximately 4 feet above the top elevation of the South wall of the West secondary shield structure in CTIMT (see Figure 40). The alternate sample location is below the polar crane, at approximately 270 degrees azimuth (due West) in CTMT. The normal sample location of RE4597BA is approximately 4 feet above the top elevation of the East secondary shield structure in CTMT, but against the CTMT wall at approximately 90 degrees azimuth (due East). The alternate sample location is by the stairway to the incore instrument tank platform on the 603' elevation (i.e., near the personnel lock).

The aieas of interest pertaining to the Containmeni Radiation Monitors revolves around two' issues: 1) their capability to detect a leaking CRDM nozzle directly by their output indication, or

2) other incidental maintenance observations. For the case in point, the maintenance Root Cause _ew .at_ .ys . _

3. _nlyi U. Data Analysis a 33 Root Cause Analysis Report

observations centered on unusual collection of boric acid and iron deposits on the filter elements of the monitors, necessitating frequent replacement. These points are discussed in the following paragraphs.

Particulate Monitor (Channel 2)

The containment airborne particulate monitor measures the buildup of particulates on a fixed filter and compares this to the integrated sample flow that produced the particulate buildup. In five minutes the airborne particulate radioactivity monitor can detect the increase in particulate radioactivity concentrations from a 0.1 gpm reactor coolant leak into the containment vessel postulated to occur when reactor coolant fission product activity concentrations result from 0.1%

failed fuel at the beginning of core life (4 EFPD). Once in the equilibrium cycle with 0.19% failed fuel, a 1 gpm leak can also be detected in five minutes. The particulate monitor consists of a fixed particulate filter in a 3 inch 4 pi lead shield. A beta detector is inserted into the lead shield to detect the activity deposited on the filter paper. The filter paper is 99 percent efficient for 0.3 micron and larger particles. The output from the detector is fed to the microprocessor where the counts per minute are converted to puCi/cc. Although this detector is effective in identification of a rapid change in leakage, some of the predominant isotopes that provide the indication include long lived Cesiuml37 and Cobalt 60. These isotopes have a half life in the range of 5 to 30 years. Based on this, even with a constant RCS leak rate and coolant activity, they could tend to constantly accumulate in containment over the course of a fuel cycle, giving a continuously increasing detector response that could be difficult to distinguish from subtle changes in leakage.

The output would also be expected to fluctuate with filter changes. Therefore, the particulate detector was not further considered for possible long term detection of CRDM nozzle leakage.

Iodine Monitor (Channel 3)

After passing through the particulate filter, the sample is drawn through an iodine collector. The iodine monitor is a 3 inch 4 pi lead shied containing the iodine c~ollecton cartridge and a ---

scintillation detector. The iodine collector efficiency is greater than 95 percent. The output from the detector is fed to a microprocessor. The microprocessor looks at two windows from this V r detector. The upper window is a background (5 percent above the iodine peak) and the lower window is centered on the iodine peak. The upper value is subtracted from the lower value giving a true iodine reading with output converted to pCi/cc. The iodine detector is capable of The detecting iodine radioactivity on concentrations as low as 7xle pCi/cc of containment air.

7 and 133 with half predominant Iodine isotopes released from the reactor coolant are Iodine 131 activity, lives of 8 days and 21 hours2.430556e-4 days <br />0.00583 hours <br />3.472222e-5 weeks <br />7.9905e-6 months <br /> respectively. For a constant RCS leakage rate and coolant these isotopes will reach a stable equilibrium value in containment and would thus theoretically isotopes provide a direct and valid indication of a slowly evolving RCS leakage trend. These tracking of fuel have the added advantage of being actively trended in the reactor coolant for that output defects, so that changes in coolant activity could theoretically be accounted for in so could be used to determine RCS leakage rates.

through.

Output data from the detectors was manually recorded on a monthl basis from late 1992 the present. However, due to high cycle 13 coolant activi and no the detectors frquently saturated dunn te of 2001. This resulted in a loss of alarm function for the remaining channels. Therefore, the carbon filters were removed from the detectors in November 2001, effectively removing the Iodine Channels from service. Data prior trend, to thiimi e is Ptesd on) ute 4 Athough re. pgut indicates a clearly increasing the output readings from this monitor suffer from a significant amount of scatter. I ne cause of the scatter is not definitively known, however, it might be related to readings being taken near the i.e usia naiyuis e 3.0 08M An8lySIS 0 44 Root Cause Analysis Report

time of change-out of the carbon elements (response not at equilibrium) or it might be related to actual changes in CTMT atmosphere conditions (e.g. scrubbing of the iodine by condensate on the containment air coolers, or retention by condensate in the sample lines.) Further, a large portion of the trend is undoubtedly due to increasing RCS activity due to fuel defects. An attempt was made to separate the effects of the coolat activity by taking a ratio of detor oput with cont activity. This also result in an inreasin d, but it suffered doubly from the scatter in both the monitor data and RCS activity data. In the end, although increased leakage was clearly detectable, there is no means to distinguish CRDM nozzle leakage from any other RCS leakage, and so this indication was not particularly valuable.

Noble Gas Monitor (Channel I)

After passing through the particulate and Iodine monitors, the gas sample is finally drawn through the noble gas monitor. The monitor housing is a 4 inch 4 pi lead container. The detector is a beta detector with an internal light emitting diode (LED) check source. The output of the detector is fed to a microprocessor where the counts per minute are converted to LCi/cc. Some of the predominant isotopes that remain to be counted by this detector are Xenon 133 and 135, with half lives of 9.2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> to 5.2 days. For a constant RCS leakage rate and coolant activity, these isotopes will also reach a stable equilibrium value in containment and would thus theoretically provide a direct and valid indication of a slowly evolving RCS leakage trend. Trend data from these monitors is presented in Figure 42.

e output from these detectors is much more consistent than the output of the iodine channels.

These detectors are sensitive and reflective of RCS leakage trends and changes in RCS activity.

Detector output is particularly sensitive when RCS activity is high, as during cycle 13. Under this condition, noble gas activity could.provide indication of very small RCS Ieakage prior to identification through the RCS inventory balance. To determine a leak rate, a rpeettvcornbinatioiopes in the RCSiwouldneed to be found to achieve an appropriate scaling factor to screen out the effects of RCS activity. Assuming this was accomplished, other RCS leakages could still mask the relatively small leakage expected from a CRDM nozzle leak.

Therefore, these detectors are also of limited value for diagnosis of CRDM nozzle leakage.

RE4597 BA and RE4S97AA Filter ChanMes RE4597AA and RE4597BA have been the subject of numerous Condition Reports due to moisture in the lines, and clogging of the filter elements with boric acid. Moisture in the lines has been associated with restarting from outages and is attributable to high CTMT humidity. The humidity arises from the initial humidity when CTMT is closed and long term accumulation from a variety of primary and secondary leak sources. Temperature changes along the sample piping as the sample is continuously drawn from CTMT to the monitors can cause condensation of water in the piping before the monitors and can interfere with monitor operation. Due to.the variety of sources of moisture, and the fact that this condition has occurred for a prolonged period of time (reference PCAQR 92-0346), it is not particularly associated with nozzle leakage.

Relatively large RCS leakage sources have the demonstrated potential to produce an aerosol mist due to flashing and evaporization of the jet of liquid as it exits from the leak. An example of this type of leakage occurred when the 'head to hot leg vent" line developed a flange leak in 1992.

When the leakage source contains borated water, the boric acid is dispersed with the aerosol as a fine particulate material. This material remains suspended in the CIMT atmosphere for a prolonged period, before eventually settling out on CTMT surfaces and appearing as a fine powder. When ingested by the CTMT radiation monitors, the boric acid will prematurely clog 3.0 Data Analysis

• 35 Analysis Report Root Cause Analysis Report 3.0 Data Analysis o 35

RPV the monitor filters and require frequent filter changes. This condition occurred during the head to hot leg vent line flange leak, but returned to normal following that repair.

30 Normally, the change frequency for the RE4597AA and RE4597BA filters is approximately of the days, and is dictated by schedule rather than low flow. However, in March of 1999 fouling was monitor filters began to recur (CR1999-0372, CR1999-0861, CR1999-0882). Initially, this ittiibuted to the disabling of the pressurizer code safety valve rupture discs in late 19W9 (discussed in the CAC section). it was noLad that ihe service had decreased, filters particularly for RE4597BA. However, by May 19, 1999, the boric acid deposits on the sent had developed a "ellow" or "brown" appearance. Under CR1999-1300, samSe filters were totitute (SRIf The SRI report (Project 18-2321-190) indicated that the samples contained predominantly ferric oxide from corrosion of iron

,. componts. An adjunct report from Sar n wjeciJQ294A033) indicated that the fineness of the particles suggested that it was attributable to a steam leak. From May of 1999 until April 2001, filter changes on 597BA were required on an irregular 1 to 3 week interval.

point It was noted in CR01- 110 that the filter life had reduced to around 3 days. The sample life improved slightly.

was changed to the alternate location near the personnel lock, and service every However, by November of 2001, filter replacements were again required approximately in the iodine channels of other day. On November 2, a blank (no carbon) cartridge was installed both monitors to eliminate a frequent alarm condition. Throughout the period of 1999 through 2001, RE4597AA exhibited similar, but slightly less severe symptoms.

The reactor service structure, which encloses the CRDMs, CRDM flanges, and CRDM nozzles, is ventilated by one of two fans that take suction on the area immediately surrounding the CRDMs and CRDM flanges. It takes an indirect suction on the area surrounding the CRDM nozzles, drawing through the mirror insulation that separates the CRDM nozzles from the CRDMs and CRDM flanges. The fans discharge on the 603 elevation, in the North-East quadrant of the reactor building. Airbome flange leakage or nozzle leakage would be exhausted by the ventilation fans to this area. The fan exhaust is closer to the normal suction of RE4597BA than to the normal suction of RE4597AA. This would tend to explain why boric acid fouling was more severe for RE4597BA, and why the symptoms were reduced when switching to the esainpklecaW~iiowhich is Wbidiii opposie tileentilatio7 M f-of Accumulation of boric acid on the radiation monitor filters was recognized to be symptomatic 12 an RCS leak as soon as it occurred. Significant efforts wer made, especial durigjhe cycle mid-cycle outage in 1999 and 12RF.nO in 2000tolocate the source.of leakage. Durng that acid outage, the only significant leakage potentially capable of producing the amount of boric necessary to exhibit the necessary s- s found tbe leaking CRDM Tlaiiges, waymptom paiFclarly at the nozzle 31 location. Howev..;L theboric iniIcdonthei filter elements was not explained.

on In August 1999, four high efficiency particulate filters were placed in CTMT near the elevator the 603' elevation. These 500 cfm filtration units were intended to help clean up the CTMT outage.

atmosphere on the theory that airborne material was left over from the cycle 12 mid-cycle The filters were removed in October 1999. The filters had no notable effect on the C'I'MT atmosphere.

exhaust Based on the observations that theje was a high boric acid accumulation near the CRDM acid fans and no leaking CRDM flanges found in 13 RFO, it can now be inferred that the borc found in the RE4597 filters (and in the CACs) originated at the CRDM nozzles and was dispersed by the CRDM exhaust fan.

0rats~ A...heEh:~ *- - sIM yOu UGLG FUIIQIJUI Q.W Lfa&G VW4083r040 - WW Root Cause Analysis Report

3.3.6 Containment Recirculation Fan/Fan Failures The Containment Recirculation System (CRS) is composed of two relatedfansd associated ductwork. The CRS circulates the air in the Containfii tome during all plant modes of operation to eliminate the temperature stratification. The CRS is normally operated continuously. However, in February of 1999, CRS fan 1 failed and remained out of service. In March of 2001,LRStZn iailed andremained out of service. The failure mode involved failure of the motor bearings, and significant destructive rubbing of the fan blades on the housings.

Failure of fan 1 significantly preceded discovery of brown deposits on RE4597 filters in May of 1999. Failure of fan 2 did not occur until well after iron appeared on the RE4597 filters. Iron continued to appear on the filters well after both fans were out of service. The out of service dates also do not coincide with other events. The particulate iron that would be expected from the fan blades is not similar in particle size that was found on the RE4597 filters. Therefore, the failure of the CRS fans does not appear to be the cause of the iron deposits on the RE4597 filters.

3.4 Programs Important to Preventing Problems This section of the data analysis provides a discussion of programs the root cause team viewed as important to preventing this type problem. Industry progrars are intended to provide advance warning and to recommend approaches to avoiding significant problems. The Boric Acid Corroion Control and the Inservice Inspection (11S) programsare intended to provide a level of defense by ensuring the integrity of the RCS and supporting systems used to mitigate plant transients. The review included interviews with the program owners. Both programs were reviewed as part of the root cause investigation.

3.4.1 B&W Owners Group and Industry CRDM Nozzle Related Initiatives In November 1990, the B&WOG Materials Committee issued Report 51-1201160-00, Alloy 600 SCC Susceptibility Scoping Study of Components at Crystal River 3 (reference 2.10). Very little attention had been given to inspection for PWSCC in Alloy 600 applications other than that associated with the steam generator tubing. As a result of the reported instances of PWSCC in the pressurizer heater sleeves and instrument nozzles in several domestic and foreign PWRs, the NRC felt that it may be prudent for licensees of all PWRs to review their Alloy 600 applications in the primary coolant pressure boundary, and, when necessary, implement an augmented inspection program (reference IN 90-10). The Materials Committee initiated a scoping study to investigate potential problems associated with PWSCC of Alloy 600 material used in B&W designed RCS components. The report summarized the completed research regarding Alloy 600 components used at a target B&WOG plant Crystal River 3. Based on this information, a susceptibility rating was given, along with recommendations for ensuring RCS integrity through inspections of appropriate components. The applications of Alloy 600 at other B&W operating plants were identified and the applicability of the target plant evaluation to these other operating plants was confirmed. This summary was to be used by the B&WOG Materials Committee in assessing the potential for future PWSCC occurrences with Alloy 600 components at B&W operating plants. The report notes that it is expected that the locations having the highest temperatures in the RCS would be the most susceptible to PWSCC. The RPV upper head is identified as one area where attention should be given. The report recommends the control rod Uising bodies be inspected, if possible, at an opportune time. The report includes a table of Alloy 600 locations at Davis-Besse, which includes the 69 CRDM nozzles. The report also NiOclues a summary of PWSCC occurrences of in-service RCS Alloy 600 components.

Report 3.0 Data Analysis e 37 Root Cause Analysis Report 3.0 Data Analysis a 37

In December 1990, EPRI issued EPRI NP-7094, Literature Survey of Cracking of Alloy 600 Penetrations (EPRI Project 2006-18) (reference 5.10) to document the problem of stress corrosion cracking of Alloy 600 penetrations in PWR pressurizers and to identify corrective actions that utilities can take to address this issue. The document lists the CRDM nozzles as an Alloy 600 component.

~- In October 1991, the first EPRI workshop on PWSCC of non-steam generator Alloy 600 J 1 materials in PWR plants was held, with representatives from the U.S. and French nuclear facilities, all U.S. Owners Groups (Westinghouse, Combustion Engineering, and B&W), EPRI, the U.S. Navy, and various vendors/consultants. This workshop provided extensive coverage of PWSCC in pressurizer instrument nozzles, pressurizer heater sleeves, steam generator drain lines, and hot leg instrument nozzles. The B&WOG provided an update on B&W activities, including the Materials Committee scoping study of Crystal River 3 and the areas of concern, including the Control Rod Housing Bodies. Later, it was learned that during a 10-vear hydrostatic test in Sep b .the French Bugey 3 plant discovered a leak in a CRDM noziaaugh-wall crack. The crack was cased by WS C in an area of high residual iresses caiisiby the J-.grome weld joining the nozzle to the RPV head. Additional cracks were subsequentlyfound in other plants in France, Sweden, and Belgium.

On May 12,1992, the B&WOG Materials Committee met with the NRC staff and provided a i presentation on "Work on PWSCC of Alloy 600 Nozzles and Components" which included information on the Bugey 3 CRD nozzle leakage. NRC concurred with the B&WOG that, based on the available infonnation on the French C azzl i n ere im a

-afety concern due to the fact that the identified cracks pa-ialij.nttuM. The NRC suggested another meeting duIng 1st quarter 1993 to cover the following on the CRDM nozzle cracking vis-a-vis B&WOG plants:

1. 50.59 Safety Evaluation to provide sufficient assurance that the issue is not a safety concern

2. CRDM nozzle inspection strategy/criteria

3. Evaluation of leak detection/monitoring system.

On 8/10/92 - 8/11/92, there was an EPRI Alloy 600 Coordinating Group Meeting Concerning CRDM Nozzle Cracking attended by representatives from each of the NSS vendors, several utilities, and Dominion Engineering. Work on CRDM nozzle cracking in the Owners Groups was presented and discussed. One item discussed was that no one was expected to inspect CRDM nozzles during the 1992 fall outage schedule unless required by the NRC. The NRC position was expected to be finalized at a Westinghouse Owners Group (WOG) meeting on 8/18/92.

On August 18, 1992, the NRC met with members of WOO to discuss the safety significance of 1.RDM penetration cracking and update the status of WOO's Alloy 600 program. The meeting was attended by representatives from each of the Nuclear Steam Supply (NSS) vendors, each of.

the owners groups, several utilities, and consultants. The NRC provided an overview of Alloy 600 PWSCC and their view on CRDM nozzle inspections. The staff viewed the CRDM nozzle cracking as a minimal safety impact, but that prudence suggested an orderly inspection program.

The NRC was concerned that the potential for cracking exists in a large number of nozzles and that thiere incern with boric acid corrosion of the RPV ha. The staff presentation slides indicated the following inspection, evaluation, and repair guidance: (1) For PWR plants refueling before spring 1993, visual inspection during leakage test, with speciai attention to CRDM penetrations at periphery locations and visual inspections (VT-2 quality) remote or direct to inspect the inside surface of the spare CRDM penetrations; (2) For PWR plants refueling after u gaa . nalysi Root Cause AnalysIs Report ax maw fmalybirb a 40

Spring 1993, PT and eddy current (EC) inspections of the inside surface of all spare CRDM penetrations; (3) EC inspection of CRD sleeved penetrations if cracks are found; (4) Provide flaw acceptance criteria; and (5) Develop corrective actions for CRDM penetrations. Recent work on CRDM nozzle cracking in the WOG was then presented and discussed. It was stated that inspection of CRDM nozzles during the 1992 fall outage schedule was not planned by any of the owners groups unless ongoing safety evaluations indicate that there is a safety concern. The NRC appeared to agree with this, but wanted to review the WOG safety evaluation (scheduled for completion 10131/92) and requested another meeting with the WOG in November. The NRC also stated that they would entertain a submittal without an NDE (ECT or UT) inspection plan but the basis for this decision must be very convincing. Coordination of the activities of the Owners Groups on Alloy 600 CRDM penetration cracking was planned to be done by Nuclear Utility Management and Resource Council (NUMARC). The NRC staff believed the reported cracking in CRDM penetrations was not an immediate safety issue requiring regulatory action.

There was time for a thorough, disciplined analysis of the safety significance, the approach to RPV head inspection, criteria for taking repair actions, and possible regulatory guidance.

On October 2,1992, the B&WOG issued a proprietary Alloy 600 PWSCC Time-To-Failure Models report (reference 2.11), presenting a PWSCC susceptibility ranking model and six susceptibility models that had been proposed within the nuclear industry to model time-to-failure of Alloy 600 components as a result of PWSCC. The PWSCC susceptibility ranking model for Alloy 600 RCS components was based on cabon content of he materiaL anneal temperature and

-cfifraion, operating temperature, and operating and residual stresses. A ranking of 4, 4-5, or 5 ndicates a high (50%)probability of failure within 20 yearsanlng of 3 or 34 indicates a medium (%) probability of failure within years anda ranng f 2-3 or bow incates low probability of failure within 40years. All failures at the lime had ben ranked between74ind 5 with this ranking model The report provided the susceptibility ranking of the Alloy 600 components on B&W designed plants. The Davis-Besse CRDM nozzles were of four different heat numbers: heat M3935 was ranked as 2-3; heat NX5940 was ranked 3-4; heat C2649 was ranked 5; and heat M4437 was ranked 4-5. Based on one of the models, the time-to-failure calculation for the worst case (heat C2649) predicted 123 EFPY for 50% of the population to initiate cracks. The report concluded that, although none of the models addressed in this document accurately predicts any of the existing industry failures of Alloy 600 components, it contained a good base of ideas to improve the time-to-failure model.

In December 1992, the second EPRI workshop on PWSCC of Alloy 600 in PWRs was held, with representatives from U.S., French, Swedish, and Japanese nuclear facilities, all U.S. Owners Groups, the U.S. Navy, and various vendors. Workshop sessions focused on concerns about PWSCC of alloy 600 penetrations in the RPV head (CRDM nozzles) in several plants, including the Bugey 3 pl=int in France. A stress analysis summary concluded the stresses are highest in the outermost nozzles for Westinghouse plants, while the stresses are essentially the same for central and outelrow nozzles for B&W I ts. Another report indicated field experience to date shows that cracks have occurred predominantly in peipheral row nozzles, Eonsisten with the results of finite element stress anfalyses Later that month, B&W issued a proprietary CRDM Nozzle Characterization report (reference 2.12), regarding PWSCC of CRDM nozzles. The fabrication and manufacturing processes for B&W-design CRDM nozzles and French-design CRDM nozzles were discussed. A comparison of this information was made, and the similarities and differences were noted: It was deterained that B&W-design nozzles are not significantly different than the French-design nozzles, and, thus, are not immune to PWSCC. In the report, Davis-Besse is noted as having all 24 of its Root Cause Analysis Report 3.0 Data Analysis e 39

peripheral nozzles rated as "very high susceptibility" for PWSCC, as are 40 of its 45 non.

peripheral nozzles. This report differs from the previous report (10/2/92) in that heat NX5940

• Was now ranked as 5 (instead of the previous 3-4). The report also lists the heat number for each

/ CRDM nozzles and notes that nozzles 1-5 are all of heat number M3935, the lowest

! !. susceptibility ranking (2-3) for Davis-Besse nozzles.

An Ad Hoc Advisory Committee (AHAC) headed by NUMARC with members from all three Owners Groups and EPRI was formed to formulate the CRDM nozzle inspection criteria and coordinate the relevant industry activities. On March 3, 1993, the AHAC met with the NRC and discussed the WOG Safety Evaluation. The B&WOG committed to perform an evaluation of the safety significance of potential nozzle cracking.

On May 26,1993, the B&WOG issued BAW-10190P, Safety Evaluation For B&W Design Reactor Vessel Head Control Rod Drive Mechanism Nozzle Cracking (reference 2.7) summarizing the stress analysis, crack growth analysis, leakage assessment, and wastage assessment for flaws initiating on the inner surface of the B&W designed CRDM nozzles. The overall conclusion reached in this evaluation was that the potential for cracking in the CRDM nozzles does not present a near-term safety concern. Crack growth analysis predicted that once a crack initiates, it will take a minimum of six years for the flow to propagate through-wall. If a crac propagates through-wall above the nozzle-to-head weld, leakage was anticipated and a large amount of boric acid deposition was expected. Once boric acid deposition occurs from leakage, wastage of the RPV head can initiate. It was predicted that wastage of the RPV head can continue for six years before ASME code limits are exceeded. The B&WOG utilities developed plans to visually inspect the CRDM nozzle area to determine if through-wall cracking had occurred and if boric acid deposition was occurring as result of a through-wall crack. The report identifies that at each of the B&WOG utilities' plants, a walkdown inspection of the RPV 4s head was implemented as part of the response to NRC Generic Letter 88-05. Enhanced visual tiV l inspection of the CRDM nozzle areas was also incorporated. If any leaks or boric acid crMstal deposits are located during the inspection of the RPV head area, an evaluation of the source of Lthe lekan eex ent of any wastae was reguired to be compet~ed. Aconservative wastage volume of 1.07 cubic inches per year was believed to be possible from a leaking CRDM nozzle.

A pe postulate corrosion wastage within and in the vicinit ithtRPVthzat-penetration from a leaking CRDM nozzle would not affect safe operation of the plant for at least six years. The boric acid deposition was expected to be detectable by the current GL 88-05 inspections. Since

/ inspections of the RPV head area (for leakage and boric acid deposits) are performed during each outage, it was thought to be unlikely that a leak wouliod s years gofc The evaluation concludes excessive wastage of the RPV head will not occur before leakage is detected either by visual observations in accordance with utility responses to GL 88-05 or the plant leakage detect'on system. The B&WOG also stated it was evaluating the potential for crack initiation and propagation on the nozzle outer surface, although preliminary evaluation of the through-wall stress distribution indicates that, even if a circumferential crack initiates on the outer surface, the crack will be self ni ing xnd willLnsra -eLp tration of the nozzle. The BREWO was -continuingits involvement in the NUMARC-sponsored AHAC for PWSCC of CRDM nozzles, including the industry-sponsored crack growth testing of CRDM penetration materials. Duke Power Company scheduled an inspection of one B&W designed reactor in the fourth quarter of 1994.

On November 19.e1993, thd N its.SfetydiEls ll for Pntential Rettor Veel Head Adaptor Tube Cracking to NUMARC (reference 3.20). The staff concluded that there was no immediate safety concern for cracking of the CRDM penetrations. The bases for this conclusic

^^.D .at .nlyi a Root Cause Analysis Report 3.0 Uata Analysis a -

(reference 3.2) were that if PWSCC occurred at RPV head closure penetrations: the cracks would be predominately axial in orientation, the cracks would result in detectable leakage before catastrophic failure, and the leakage would be detected during visual examinations performed as part of surveillance walkdown inspections before significant damage to the RPV head would occur. This finding was predicated on the performance of the visual inspection activities requested in Geieneric Leter (GL) t85. -also, special nondestructive examinations were scheduled to commence in te spnng 011994 to confirm the safety analyses for each PWR owners group.

On December 14,1993, the B&WOG Materials Committee issued BAW-10190P Addendum 1, External Circumferential Crack Growth Analysis for B&W Design Reactor Vessel Head CRDM Nozzles (reference 2.8) providing an evaluation of external circumferential crack growth, gross leak-before-break mechanism, and the stress affects of CRDM nozzle straightening. The report concludes that there wasi Fop-oss-ibilityfra etra] circumferential flaw indication to grow circu ferenti-Oy to the point of becoming a safety concern. Te overall conclusions presented in B&W-10190P remained unchagigedwith this addendum. It was concluded the GL 88-05 walkdown visual inspections of the RPV head areas provide adequate leak detection capability.

In March 13, 1994, the RCS System Engineer initiated PCAQR 94-0295 regarding a commitment in the commitment management system that was closed (complete) and not converted to an ongoing commitment. The commitment required a visual inspection of the RVP head every refueling to determine the potential for CRDM nozzle cracking in support of a B&W safety evaluation to the NRC. The PCAQR evaluation identifies the inspection is covered in the existing program as outlined in NG-EN-00324, Boric Acid Corrosion Control. The commitment was closed based on the NG-EN-00324 inspections and the fact that the NRC saw enhanced inspections as being "prudent" but not necessary were to be put in the next outage contract.

In November 1994, the 1994 EPRI Workshop on PWSCC of Alloy 600 in PWRs was held. The workshop summarized the field experience associated with PWSCC of Alloy 600 CRDM nozzles, reviewed the current status of inspection, repair, and remedial methods as well as strategic planning models, and discussed stress analysis results as well as PWSCC initiation and growth in Alloy 600. The workshop was attended by domestic and overseas utilities, PWR vendors, research laboratories, and consulting organizations. Three U.S. plants had inspected CRDM nozzles; no cracks were found in one plant and only minor cracking was observed on one nozzle in each of the other two plants. Results of inspections in France, Sweden, Spain, fBlgigumJ apan, and Brazil revealed a trend toward earlier axial cracking in plants with forged nzzles asopposed to those made from rolled ba orii erioiins.Ttwas alEso hougt that other factors such as surface finishing a role (see reference 5.4).

On April , 1997, Davis-Besse received GL 97-01 Degradation of CRDMICEDM Nozzle and other Vessel Closure Head Penetrations (reference 3.2). The letter requested plants describe their program for ensuring the timely inspection of PWR CRDM and other RPV head penetrations (VHP). In July1997, the B&WOG Materials Committee issued BAW-2301, B&WOG Integrated Response to Generic Letter 97-01: "Degradation of Control Rod Drive Mechanism Nozzle and other Vessel Closure Head Penetrations" (reference 2.1). On July 28, 1997, Davis-Besse responded to the GL 97-01 endorsi BAW-2301. The BAW topical report provides the justification and scheae foranintegrated VHP inspection program.

The BAWY2301 introduction reiterates conclusions discussed in references 2.7 and 3.20. The introduction furthermore states PWSCC for CRDM nozzles and other VHPs will not become a long-term safety issue provided the enhanced boric acid visual inspections, performed in dB A-1-.

llel.vvazaIan QVA U&

.0.VW0LU#V1Q1FQ40 lb *-

A4 A

no Root Cause Analysis Report

accordance with GL 88-05, are continued. An axial crack would lead to a leak on one or more nozzles and result in a significant deposition of boron crystals. It is very unlikely that this type of accumulation would continue undetected with regular walkdown inspections of the RPV head area. If the crystals remain hidden by the RPV insulation, the insulation would begin to bulge as a result of this accumulation o This deposition would easily be detected prior to significant damage to the RPV head. Therefore, the RPV head's structural integrity would not be jeopardized, thereby eliminating any safety concerns with PWSCC of these nozzles. In order to assure the assumptions of the original safety evaluation remain valid, an integrated inspection program had been developed to address this issue for the B&WOG plants.

The BAW-2301 report presents the integrated B&WOG inspection program. Oconee 2 and Crystal River 3 are identified as two of the B&WOG plants most susceptible to PWSCC, as currently ranked. These two plants either have or will perform inspections of the RPV head nozzles from beneath the RPV head. Oconee 1 and 3, Davis-Besse, ANO 1, and T`M 1 do not have CRDM nozzle inspections planned in the near term (1998-2000).

In May 1998, the Davis-Besse Materials Committee representative initiated a procedure change request to NG-EN-00324, Boric Acid Corrosion Control. The change requested the B&WOG Materials Committee Report 51-1229638 Boric Acid Corrosion Data Summary and Evaluation be add to a note that identifies material that contain helpful reference material for determining boric acid corrosion rates. The information was incorporated into the procedure as requested in April 1999.

On April 30,2001, the NRC issued Information Notice (IN) 2001-05 to alert plants to the recent detection of through-wall circumferential cracks in two CRDMs nozzles and weldments at Oconee 3. On May 2,2001, CR 01-1191 initiated identifying the need for a project plan with team members developed to prepare Davis-Besse for a cracked CRDM J-groove weld. The CR identifies all three units at Oconee and one unit at ANO have inspected for and found cracked 3-groove welds around their CRDM nozzles.

On August 3,2001, NRC issues NRC Bulletin 2001-01 Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles. The discussion section identifies recent identification of circumferential cracking in CRDM nozzles and axial cracking in the J-groove weld has resulted in the NRC staff reassessing its conclusion in GL 97-01 that cracking of VHP nozzles is not an immediate safety concern. Circumferential cracking in CRDM nozzles were identified at Oconee 2 and 3, and axial cracking in the J-groove weld in CRDM nozzles were identified at Oconee 1 and ANO 1 (i.e., B&W plants). The findings at Oconee 2 and 3 highlight the possible existence of a more aggressive environment in the CRDM housing annulus following through-wall leakage; potentially highly concentrated borated primary water could become oxygenated in this annulus and possibly cause increased propensity for the initiation of cracking and higher crack growth rates. Regulatory Affairs initiated CR 01-2012 in response to the bulletin.

Between September 4 and November 30, 2001, Davis-Besse met with and docketed responses to the NRC regarding NRC Bulletin 2001-01. In discussion held with the NRC on November 28, 2001, Davis-Besse committed to a 100% qualified visual inspection, non-destructive examination (NDE) of 100% of the CRDM nozzles and characterization of flaws through destructive examination should cracks be detected. Several other commitments were also made at that time including moving forward the start of the scheduled refueling outage from April 1 to no later than February i6,2001. -

3.0 Data AflIYSI5 e 4Z Root Cause Analysis Report 3.0 Data A1181Y813 0 4Z

3.4.2 Davis-Besse Boric Acid CorrosionControl Program As discussed above, and as part of the root cause evaluation of the Davis-Besse RPV head degradation, the Boric Acid Corrosion Control program was reviewed. The intent of this review was to compare various aspects of the program to GL 88-05 and how the program is currently being implemented as it relates to the RPV head.

Generic Letter 88-05 was issued March 17, 1988 to address the effects of boric acid leaks on carbon steel components. All license holders for PWRs were required to address the issues identified in the generic letter. GL 88-05 identifies four areas that must be addressed in the plant specific boric acid program. The areas include:

• determination of principal leak locations where leaks may occur that are smaller than technical specifications allow

• procedures for locating small leaks

• methods of conducting examinations and performing engineering evaluations to address the impact of the leak

• corrective actions to prevent recurrence of this type of corrosion The Boric Acid Corrosion Control program procedure (NG-EN-00324) was reviewed against these same points.

Princidpal Leak Locations The procedure identifies the following areas as principal locations for possible leaks:

• Steam Generator and Pressurizer manways and handholes

• Seal Welds

• Thermowells

• Reactor Coolant Pump and other pump seals and casing flanges

• Control Rod Drive Flanges

• Piping Flanges and Bolted Connects

• Valve Bonnets and Packing Glands

• Reactor Vessel Head O-rings The procedure does not address industry known leakage areas such as the CRDM nozzle issue or potential leakage areas such as the lower RPV head area. Currently the procedure is limited to inspections of systems and components inside Containment.

As an example, PCAQR 94-0295 discussed the need for enhanced inspections of the RPV head in addressing the CRDM nozzle cracking issue. The later text in PCAQR 94-0295 states that B&W amended its safety evaluation following feed back from the NRC that stated enhanced inspections were not required. B&W's amended safety evaluation took credit for the GL 88-05 inspections. Discussion with Framatome indicate the safety evaluation dated May 1993 was never changed to eliminate the need for enhanced inspections. See PCAQR 94-0295 discussion in the condition report section for additional details concerning this specific issue.

Procedures for Locatine Small Leaks Step 2.1.1 of the Boric Acid Corrosion Control procedure identifies a number of station procedures that support inspection and identification of leakage. Several of the procedures were reviewed to generally determine the kinds of inspections that are required. DB-OP-06900 Plant Heatup requires an inspection at operating temperature and pressure. DB-OP-01200 Reactor Coolant System Leakage Management provides guidance and trigger points with regard to dBePt not6 AlI-rt- -

Root Cause Analysis Report ax Wow miallaso W "'a

Technical Specification leakage values. The DB-OP-01200 includes trigger points for "buildup of boric acid on equipment requires frequent containment entries to clean and/or inspect". Both procedures prompt action to identify and characterize leakage however, the values of unidentified RCS leakage causing this type of a condition are relatively small.

DB-PF-03065 Pressure and Augmented Leakage Test performs a leakage inspection at temperature and pressure in support of the ISI program. This procedure will be addressed in the ISI section.

Conductine Examinations and Enelneerin2 Evaluations The procedure outlines various activities that "should" be performed to ensure the component is serviceable and meets code requirements. These activities include assessing for corrosion, wastage, and performing engineering evaluations. The procedure provides reasonable guidance in this area; however, there are many places in the procedure that use the word "should" instead of "shall." The use of "should" allows a choice to be made in an area that involves technical insight. The use of "should" may be used if the technical staff involved with the decision making is highly experienced or reviewed by a highly experienced peer or supervisor.

The RCS System Engineer was interviewed concerning activities related to the spring 2000 refueling outage (12RFO). The Boric Acid Corrosion Control Inspection Checklist (BACCIC) provides a method to both characterize and disposition a Boric Acid leak. The RCS System Engineer was questioned concerning the dispositioning of the BACCIC that was issued to document the Boric Acid on the RPV head. The BACCIC closure process is not well defined.

The RCS System Engineer could not recall how the BACCIC was closed out in 12RFO.

Additionally, there was no evaluation to address the Boric Acid remaining on the RPV head during cycle 13. See CR 2000-0782 in the Condition Report section for additional details concerning this specific issue.

Corrective Actions to Prevent Recurrence This portion of the procedure describes various types of modifications to prevent leaks and/or mitigate the outcome of the leak. The section does not discuss, however, reviewing the maintenance history of the leaking component, reviewing maintenance procedures and work practices, reviewing industry documents such as "EPRI Good Bolting Practices," and reviewing industry issues (Operating Experience) for possible improvements to recurring issues.

The program owner was interviewed on the subject of the Boric Acid Corrosion Control program. The program owner characterized his role in the Boric Acid Corrosion Control program as a caretaker. The program owner coordinates the walk down of Containment and then provides the System Engineer with a copy of the Boric Acid Corrosion Control Inspection Checklist resolution. The program owner also is responsible for maintaining and updating the administrative procedure that controls the program.

Step 6.7.4 of the Boric Acid Corrosion Control procedure (NG-EN-00324) identifies increased responsibilities for the program owner during outages. The procedure describes the program owner functions as: coordinates decontamination and insulation removal for detailed inspection of components, develop plans to resolve leaks (identify and prioritize), coordinate repairs with the pressure test engineer, and provide a status of the repair activities to Outage Management.

The Boric Acid Corrosion Control proam does not require the retention of any Boric Acid -

Corrosion Control Inspection Checklist. The BACCIC contains both the initial assessment of the leak (including corrosion) and the results of any subsequent evaluations. The BACCIC contains Root Cause Analysis Report 3.0 Data Analysis

• 44

signatures for the resolution of the leak, but does not require review or supervisory acceptance.

Neither the Boric Acid Corrosion Control program owner or the RCS System Engineer could produce copies of the completed BACCIC sheets from the 12RFO, therefore an evaluation of the effectiveness of the actual reviews could not be performed.

3.4.3 Davis-Besse Inservice Inspection Program The focus in the ISI program is related to the pressure test performed at the end of a refueling outage (Mode 3 walk down) and performing certain cold (Mode 5) inspections.

In support of this review, the following documents were reviewed:

• DB-PF-03065 Pressure Test dated 5/20198 (I1RFO)

• WO 99-000320-000 (RX VESSEL) Reactor Vessel Bolting VT-2 Examination at the start of 12 RFO

• DB-PF-03065 Pressure Test dated 5/13/00 (12RFO)

• DB-PF-03010 RCS Leak and Hrydrostatic Test dated 612/00 (12RFO)

• Inservice Test Plan (1ST Plan) Volume II Second Ten Year Interval Pressure Test Program dated 10127/99 DB-PF-03065 Pressure Test dated 5120198 (1IRFO) was reviewed for program compliance and understanding of the objects being inspected. The person involved with the 1IRFO Pressure Test that was performed at the conclusion of the outage was interviewed. The person was level 3 certified in NDE. The person described his entering the Reactor Cavity and walking around the RPV head looking for evidence of leakage from the CRDM nozzles. There was no requirement for a hold time at pressure. The test was performed per the requirements of the ISI program and the person demonstrated cognizance of the task.

Work order (WO) 99-000320-000 (RX VESSEL) for 12RFO included performing a VT-2 examination of the installed RPV studs, nuts, and washers. The VT-2 examination could not be performed because of the presence of Boric Acid on the RPV head flange. Condition Report 2000-0781 was written to document the condition of the RPV fasteners. See the Condition Report section for the additional details concerning this specific issue. Final inspection of the fasteners occurred after removal for refueling activities; however, no evidence was presented to document a follow up examination of the RPV head following boric acid removal.

DB-PF-03065 Pressure Test dated 5/13/00 and DB-PF-03010 RCS Leak and Hrydrostatic Test dated 6/2/00 (both from 12RFO) were reviewed. The two inspection reports include CRDM nozzle inspections with the plant at operating temperature and pressure (Mode 3). The examination report dated 5/13/00 indicates that the CRDM nozzles were included in the examination by inspection on top of the service structure looking downward; however, the CRDM nozzle to CRDM flange weld view is obstructed by the CRDM mechanisms and the CRDM flange. It is not clear what is being inspected by this line item. The 6/2/00 dated examination report identifies the CRDM nozzle to RPV head welds were inspected for leakage looking for indications of leakage under the RPV. The report identifies that the inspection for leakage below vessel meets code requirements. No leakage was identified during these inspections.

The IS! Pressure Test Engineer was asked, how post boric acid removal inspections are handled specifically related to the RPV head. The person processing the.BACCIC weuld contact the. ISI Pressure Test Engineer to have the affected area inspected. If he is not contacted, there is no follow up. The RPV head has not been specifically included in the Boric Acid Corrosion Control Report Analysis Report 3.0 Data Analysis

• 45 Cause Analysis Root Cause 3.0 Data Analysis e 45

program. In addition, the ISI program inspection techniques did not identify CRDM nozzle weld leakage when leakage is now believed to be present.

3.4.4 Evaluation of Condition Report Responses The Root Cause team as part of the investigation, reviewed the completion of a number of Condition Reports. The results of this review are discussed with each Condition Report and then at the end of this section.

PCAQR 94-0295 - Addresses the need to convert commitment A16892 to an Ongoing Commitment. The PCAQR was closed by Regulatory Affairs after they determined that B&W had changed the safety evaluation related to the CRDM nozzle issue to eliminate the need for enhanced inspections. The statement goes on to take credit for the GL 88-05 program to address leaking CRDM nozzles. Discussions with Framatome (purchased B&W) indicate the safety evaluation dated May 1993 was never changed to eliminate the need for enhanced inspections.

PCAQR 96-0551 - Addresses boric acid in several areas on the RPV head and that all of the steps for the Boric Acid Corrosion Control program procedure in effect at that time (NO-EN-00324 rev 1) may not have been followed. The text within the document response takes credit for the GL 88-05 walk downs (GL 88-05 walk downs related to the RPV head at DB only address CRDM flange leakage). The response to the PCAQR endorses implementing MOD 94-0025 (inspection holes for Service Structure). The modification is outstanding.

CR 1999-1300 - Addressed the iron oxide deposits on RE4597 AAIBA. The response discusses the results of the radiation monitor action plan. The lab analysis report evaluating the iron oxide states that the iron oxide was not from magna flux powder but appears to be from an iron based component in containment. There does not appear to have been an aggressive effort to locate the actual source of iron oxide after the original proposed source was disproven. The source of the iron oxide was not conclusively determined under this CR.

CR 2000-0782 - (Categorized as "Routine") This CR was written to address the buildup of boric acid on the RPV head. The CR describes the areas affected by the boric acid. A BACCIC sheet I was attached to the CR. The BACCIC characterized the leak as "heavy",

red/brown deposits, new leakage not seen during 1IRFO, and recommended a detailed inspection. The CR response does not address the concerns discussed in the CR text or the BACCIC. The responder to the CR believed the leakage was from CRDM flange leakage. It discusses whether there is a need for issuing OE. There is no inspection or evaluation for Boric Acid corrosion and no discussion about other possible sources of boric acid.

CR 2000-1037 - (Categorized as "Routine") This CR was written to document the boric acid on the RPV head and on top of the mirror insulation. Operations Review b'.ck contained the following note This CR should be sent to SYME for resolution. This CR will address the effects of the boron on the head. CR 2000-782 will address the hardware issue of leaking flanges." The response to the CR states that "Accumulated boron deposits between the RPV head and the thermal insulation was removed during cleaning process performed under WO 00-001846-000. No boric acid induced damage to the head surface was noted during the subsequent inspection." The boric acid being left on the RPV head for cycle 13 was not discussed in the CR and not evaluated.

Condition Reports 2000-0782 and 1037 were characterized as "Routine". Both Condition Reports discuss accumulated boric acid on the RPV head. The Condition Reports were not Report 3.0 Data Analysis a 46 3.0 Data Analysis

• 46 Root Cause Analysis Analysis Report

elevated to the appropriate significance level. In addition reference section 4.1, Davis-Besse Experience Review regarding a discussion of the RC-2, Pressurizer Spray Valve incident in 1998.

3.5 Management Issues The development of the contributing causes for the management and organizational issues was accomplished using methodologies as described herein. The physical factors that caused corrosion of the RPV head are CRDM nozzle leakage associated with through-wall cracking followed by boric acid corrosion of the RPV low-alloy steel. Management issues have been determined to play a role in the condition in that the nozzle leakage went undetected until after significant damage had occurred.

To ensure complete independence, professionals and specialists were brought to Davis-Besse.

They led the technical root cause investigation and provided this report to the site. Additional experts, with extensive industry experience were brought in to specifically analyze management issues. This team comprehensively reviewed the data, independently interviewed many individuals and performed a thorough analysis of management practices. The results have formed the basis for the corrective actions in this report and facilitate a step change in technical standards in addition to other areas.

The following sources of data were used to conduct the analysis of the management issues: the technical portion of this root cause report, interviews conducted by the investigation team, and results of analysis conducted by an independent consultant.

Analysis identified that several opportunities for identification of and scoping the significance of field conditions were missed when reviewed collectively. The relating of independent but common-causal data was missed at all levels of the organization.

To ensure continued follow-up, Davis-Besse management was added to the team only after root cause determination was complete. After the independent team leaves site, these managers will have the responsibility to ensure follow-through actions are completed. Independent oversight will still be in place to ensure the maintenance of improved technical standards.

In performing this investigation, the team gathered information and conducted data analysis that is grouped below.

3.5.1 PotentialRisk to the RPV Head Surface The potential threat for damage to the RPV head was not recognized because it was believed significant leakage was not likely to occur. Station personnel also believed that even if there was any that leakage because of the high head temperature, leakage would instantly flash to steam and the boric acid on the head would be dry and of no corrosive risk. For these reasons, the boric acid accumulation was not addressed aggressively. In addition, at times very minor CRDM flange leaks were not always fixed when they were found. This was considered acceptable for the same reasons described above. A more rigorous adherence to requirements outlined in GL 88-5 may have allowed earlier detection of nozzle leakage.

3.5.2 Deferral of Service Structure Modifications to Allow Improved Access The modification to the service structure was considered not necessary in 1993 because we were not susceptible to CRDM nozzle cracking, and was reviewed periodically to determine the appropriate time for its installation. It was scheduled to be implemented in 13RFO, but then was voided as a modification since Davis-Besse was planning on replacing the head and service Report 3.0 Data Analysis

• 47 Analysis Report Root Cause Analysis 3.0 Data Analysis

• 47

structure in 14RFO which would include the same types of access specified in the modification.

Deferral was considered acceptable since it was incorrectly believed that Davis-Besse was not yet susceptible to the types of nozzle cracking described in industry operating experience at the current stage of plant operating life. The modification was intended to be installed later in operating life when susceptibility to cracking would be increased. In addition, it was thought that other similar plants with a longer operating history would be susceptible to cracking prior to Davis-Besse. The results of their inspections were monitored very closely. The earlier installation of this modification would have allowed the head to be both cleaned and inspected more thoroughly. This would have provided an opportunity for earlier detection of nozzle leakage.

3.5.3 Management InvolvementlAwareness of PlantConditions during 12RFO There was insufficient management involvement to assess the conditions in the Containment and at the RPV head. This resulted from management being assigned to various collateral roles within the outage organization and from a lack of rigor in the review of formal documentation of the head inspection. There was no specific oversight assigned to the inspection. This resulted in depending on senior engineers to provide oversight for less experienced engineers. During l2RFiO, eReactor Coolant System engineer was relatively new and was left to manage the problem of boron on the head. He was successful in getting the head partially cleaned, but management did not recognize that the cleaning performed was inadequate to perform a comprehensive inspection. The formal reviews performed by the engineer were documented using the corrective action process. The condition report was reviewed by supervision and signed off even though it did not satisfactorily address the issue. Closer supervision of the head inspection and rigorous adherence to the corrective action process would have provided an opportunity to address head leakage at an earlier time.

3.5.4 Recognzizng the Need for Collective Significance Review Potentially related issues of boric acid accumulations on the Containment Air Coolers, RCS unidentified leakage rates, iron oxide and boric acid accumulations on radiation monitor filters, and boric acid deposits on the RPV flange, and horizontal surfaces in the Containment were not recognized. The System Health Report for the fourth quarter of 2001 reflected the inconsistent conclusions between indications. Management was aware of the high probability of an RCS leak in the Containment Building and plans were in place to attempt to locate the leak while the plant was shutting down in a still pressurized condition. A rigorous collective significance review might have made management aware of an increased probability of head leakage.

3.5.5 RPV Head Inspections Prior.to 13RFO, head inspections were not a scheduled activity. The Framatome inspection contract only generated a video tape for retention and use by First Energy. The procedure for conducting the analysis by Davis-Besse personnel was not followed. The lack of analysis was a missed barrier to identification of the leak evaluating the structural integrity of the RPV head.

3.5.6 RPV Head Cleanings Prior to 13RFO, head cleanings were not a scheduled activity. The RPV head cleaning work was an emergent issue, controlled by a work order and RWP during l2RFO. No specific acceptance criteria was identified in the work order or RWP, however included in both the work order and RWP is text identifying the needt have a cleanhead in support of OL 97-01 The RPV head 3.0 Data Analysis

• 48 Root Cause Analysis Report 3.0 Data Analysis e 48

The potential for significant corrosion of the RPV head as a result of accumulating boric acid and local leakage was not recognized as a safety significant issue by the staff and management of the plant. The lack of understanding of this was a CAUSAL FACTOR.

Containment building related conditions like iron oxide, boric acid and moisture found in radiation monitor filters, boric acid accumulations on the air coolers and boric acid accumulations on the RPV flange were all recognized, but no collective significance was recognized. However, it is not clear if these could have led to the discovery of the problem on the RPV head in time to prevent significant damage.

All three CRDM nozzles that were found to have leaks were located in the center top region of the RPV head. The team was not able to determine how important this location would be to the potential for development of corrosion as a result of an unattended leak, compared to that of a leak that might exist on the steeper sloped regions of the RPV head. It can be stated that no other significant wastage has been reported at other sites, regardless of the locations on the RPV head where nozzle leakage was found. It is probable that the close proximity of the RPV head to the overhead insulation layer allowed for boric acid to concentrate and remain in this region. This in turn could have provided a means for accelerated corrosion rates earlier in the process, in that large accumulations of boric acid may have been available to mix with a continuous moisture supply, once it developed from below.

Evaluations are continuing to determine why the corrosion at Davis-Besse was more severe than at other B&W design plants such as Oconee 1-3, ANO 1, TM 1 and Crystal River 3. The team has identified two possible reasons for this:

• First, the cracks in nozzles 2 and 3 at Davis-Besse extend farther above the top of the J-groove weld (1.1" - 1.2") than cracks measured at other B&W design plants (<1.0").

Analyses in Section 5 demonstrate that the leak rate is sensitive to the length that the crack extends above the J-groove weld. However, the analyses also show that changes in support provided by the low-alloy steel RPV head material can affect the crack opening displacement and area

• Second, presence of pre-existing boric acid deposits on top of the RPV head may have increased the initial corrosion rates at the exit of the annulus. This theory is supported by test data, which shows that placing insulation around a bolted flange tends to capture the escaping steam and increase the corrosion rate on the heaviest corroded stud, and increase the corrosion rate at other studs around the flange.

In any event, the large scale corrosion occurred as a result of not detecting and arresting the leakage until advanced symptoms had occurrd Report 3.0 Data Analysis

• 51 Analysis Report Root Cause Analysis 3.0 Data Analysis a 51

4.0 Experience Review An experience review was performed and the Davis-Besse and nuclear industry searches identified the following related issues.

4.1 Davls-Besse Experience In 1998, two body-to-bonnet flange nuts on RC-2, Pressurizer Spray Valve, were identified as missing. The CR 1998-0020 root cause analysis report identifies the nuts were missing as a result of boric acid corrosion. Boric acid corrosion resulted due to a packing leak and the nuts being carbon steel versus stainless steel. The root and contributing causes are similar to the conditions described in this root cause report.

4.2 Nuclear Industry Experience Reference discussions provided throughout the Data Analysis section and Table 7 Nuclear Industry Experience Review Results for a summary of Davis-Besse response to NRC and Institute Of Nuclear Operations (NPO) related documents.

4.3 Conclusions Previous Davis-Besse and nuclear industry experience were not effectively used to prevent the current condition and therefore is considered a casual factor.

4-4.0 ExperIence Review

• 52 Root Cause Analysis Report 4.0 Experience Review e 52

5.0 Root Cause Determination This summary presents the collective judgment of the Root Cause Investigative Team based on the data and evidence that has been characterized at this time in the investigation (current to 4/5/02). Additional management insight will be provided by a high level industry review team to ensure the management issues are fully developed and addressed. This will occur prior to startup, and is addressed by corrective action 10.

5.1 ProbablelRoot Causes

1. Probable Cause - PWSCC cracking in the CRDM nozzle interface at the J-groove weld due to material susceptibility in the presence of a suitable environment resulted in:

• CRDM nozzle crack initiated

• CRDM nozzle crack propagation to through wall leak

• Boric acid corrosion of the low-alloy steel RPV head material

2. Root Cause - Boric Acid Corrosion Control and ISI programs and program implementation regarding the RPV head resulted in:

• Plant not identifying the through wall crack/leak during outages

• Plant returned to power with boron on the RPV head after outages

• Plant not identifying degradation of RPV head base metal during 12RFO

• Boric acid corrosion of the low-alloy steel RPV head material 5.2 Contributing Causes

1. Environmental conditions, cramped conditions due to the design and high radiation at the RPV head, resulted in:

• Plant not identifying the through wall crack/leak during outages

• Plant returned to power with boron on the RPV head after outages

• Plant not identifying degradation of RPV head base metal during 12RFO

• Boric acid corrosion of the low-alloy steel RPV head material

2. Equipment condition due to uncorrected CRDM flange leakage (especially at nozzle 31 due to its close proximity to nozzle 3):

• Plant not identifying the through wall crack/leak during outages

• Plant not identifying degradation of RPV head base metal during 12RFO

• . Boric acid corrosion of the low-alloy steel RPV head material

3. Management monitoring of field activities did not identify problems.

• The management team was not aware of the significance of the boric acid found on the reactor head during 12RFO.

• It took until 13RFO to thoroughly clean the reactor head and conduct inspections.

Generic letter 97-01 has leakage detection by visual inspection as an assumption for the safety evaluation.

• Management was not aware the Boric Acid Corrosion Control program was not being

- implemented in accordance with the priveaure administrative requiremnents. -

4. Management monitoring of activities did not identify changes in conditions.

Report 5.0 Hoot cause 5.0 Root Determination ** 63 cause Determination 53 Root Cause Analysis Report

• The boric acid deposits on the head changed from white to red. The expected color is white, with red indicating metal oxide.

• The radiation monitor filter paper analysis contained metal oxide. There would have been no carbon steel in the leakage path from a postulated CRDM flange leak.

• Reactor coolant system leakage would normally flash to steam, resulting in snowy boric acid deposits. The 1996 head inspection the deposits were solid flow, not loose powder.

• Deposits of boric acid repeatedly formed on the CACs, even after the proposed source had been repaired in mid-1999.

• There was an increasing rate of boric acid accumulation on the head without a known corresponding increase in CRDM flange leakage.

• After 12 RFO the CAC plenum pressure decrease was attributed to boric acid fouling on the cooling coils. The fouling was stated to be coming from CRDM flange leakage.

Earlier experience with leaking flanges did not result in the need to clean CAC coils.

• The possibility that nozzle leakage could be contributing to the boric acid accumulation on the top of the head was not considered.

• The staff did not do a complete re-review of the information available on the reactor head in a rigorous and questioning manner after NRC Bulletin 2001-01was issued. No documented review with checking and independent verification was made.

5. Technical standards.

• Assumptions made in supporting technical decisions were not verified by direct inspection. In fact, the head could not be inspected to verify its condition due to boric acid deposits. B&WOG report (1993) assumes that the head would be inspected periodically for evidence of nozzle leakage.

• Technical problem solving shall be based on an assessment of all reasonable, potential causes, and they shall be systematically proven or disproved until the cause is identified.

Examples where a systematic assessment was not performed are the precipitation on the air monitoring filters and the investigation of RCS unidentified leakage.

• In making technical evaluations, the limitations and uncertainties of data shall be made clear. The staff did not fully understand the limitations and uncertainties involved with the head inspections and the data supporting the safety evaluation.

• The modification to open the inspection holes was deferred without a technical evaluation.

• During 12RFO, the reactor head was reinstalled on the vessel without a complete cleaning.

• During refueling outages personnel, such as engineering supervisors, are assigned outage positions. This results in a reduction in supervisor oversight of the technical staff.

6. Oversight

• The site does not have independent internal oversight in engineering. As such, the barrier provided by such a group does not exist.

• Corporate Nuclear Review Board met only infrequently and the majority of their time was spent on reviews of safety evaluations and Licensing Amendment Requests.

7. Previous industry and in-house experience were not effectively used to prevent problems.

• The lessons learned and experience gained with boric acid corrosion on valve RC 2 were not used in assessing the condition of the reactor head.

Report 5.0 Root Cause 5.0 Root Determination

• 54 Cause Determination

• 54 Analysis Report Root Cause Analysis

• The RPV head is not a specific item in the Boric Acid Corrosion Control Program even though IN 86-108, Supplement 1 (April 20, 1987) documents severe corrosion of various components on the RPV head resulting from boric acid corrosion.

8. Execution of the Condition Report Program

• Appropriate categorization of CR

• Over sight of CR responses for technical accuracy and license impacts

• Discussions with persons indicate that standards for initiating CRs may not be aligned with program expectations DetermInatIon

• 55 Root Cause Analysis Report 5.0 Root Cause 5.0 Root Cause Determination

• Us

6.0 Extent of Condition 6.1 Degradation Mechanism Issues There are two specific degradation mechanisms observed on the RPV head that will be addressed in this extent of condition evaluation. The mechanisms are PWSCC and boric acid corrosion.

The 69 CRDM RPV bead penetrations will be evaluated by the RPV head repair team and Engineering ensuring all necessary inspections and examinations are performed on the CRDM nozzles to address extent of condition.

The extent of condition within the containment will be evaluated via walkdowns of structures, systems and components (SCC) within containment. In defining the scope of the walkdowns, three separate criteria were developed to ensure that a bounding evaluation is performed. These three separate criteria are:

(1) Sources: As used in this evaluation, sources are components containing borated water that are considered likely leak locations. The sources are further divided into three groups:

Valves, Threaded/Bolted joints (e.g. thermowells, manways, handholes, reactor coolant pumps), and Alloy 600 components/welds. The Alloy 600 components/welds are susceptible to PWSCC. The intent is to (1) verify there is no additional RCS pressure boundary leakage at Davis-Besse (from Alloy 600 components/welds) and (2) verify that evidence of RCS leakage from any source is properly evaluated (including the potential impact on susceptible materials of the RCS pressure boundary).

(2) Targets: As used in this evaluation, targets are components within the RCS pressure boundary that utilize materials susceptible to boric acid corrosion (carbon and low-alloy steels) as part of the pressure boundary. The targets include the following RCS components: RPV, steam generators, pressurizer, RCPs and individual piping sections.

The intent is to verify that boric acid corrosion has not degraded the RCS pressure boundary. Additionally, although technically not within the RCS pressure boundary, the core flood tanks will be evaluated as targets. It should be noted that certain valves within the RCS pressure boundary may contain susceptible materials but for convenience the valves are listed as sources.

(3) Safety-related (non RCS pressure boundary) SSCs: This criteria refers to safety related SSCs that utilize materials susceptible to boric acid corrosion but are not part of the RCS pressure boundary. The intent is to verify that boric acid corrosion has not adversely impacted the function of safety related SSCs.

Methodology:

(1) Plant Engineering will develop a list of inspection points to address the sources and targets. A table of valves and threaded/bolted connections previously developed for the boric acid corrosion control program mode 5 walkdowns will be used to identify these sources (most of these walkdowns are complete at this date). A list of Alloy 600 components/welds within the RCS pressure boundary has been provided by Design Basis Engineering. A series of inspection points will be needed to adequately address each-target. Each target will receive a visual inspection of the external surfaces of installed insulation for evidence of leakage (boric acid residue or bulging of the insulation).

Root Cause Analysis Report 6.0 Extent of Condition a 66

Additionally, each connection point between a target and non-susceptible piping will be inspected to verify that no boric acid has migrated undetected under the insulation to reach a susceptible component. This will require removal of insulation to permit a visual inspection. These inspections (external inspection of the insulation and visual inspection of connection points) will provide adequate assurance that there is no undetected degradation of the RCS pressure boundary. It should be noted that many of the "connection points" are Alloy 600 components/welds that also require inspection as potential sources. These inspections will be performed by VT-2 qualified personnel.

Representative photographs will be made to document the "as found" condition of each inspection point.

(2) The use of visual inspection of Alloy 600 components/welds to detect evidence of throughwall PWSCC requires adequate access to perform a visual inspection.

Additionally the design of component/weld must provide assurance that leakage will be detectable at the surface. This may require additional evaluation of certain nozzles (such as incore nozzles) to verify that a visual inspection is adequate.

(3) In any case where evidence of boric acid deposits exists, the source of the deposits and the leak path must be traced to ensure that there is no wastage of the RCS pressure boundary. It is known that there are boric acid deposits on the insulation on the bottom of the RPV. There are boric acid deposits on the seam between pieces of insulation suggesting that the boric acid came from inside the insulation. It is therefore necessary to perform an inspection under the insulation to determine whether or not there is wastage on the RPV and to determine the source of the boric acid. Due to the difficulty of this task and ALARA considerations, a specific plan is being developed to perform this inspection.

(4) The third category, safety-related (non RCS pressure boundary) SSCs, will be addressed by general area walkdowns of the containment building. These walkdowns will be primarily conducted by Design Engineering MechanicallStructural (DEMS) and Design Engineering Electrical/Controls (DEEC). The DEMS personnel will focus on safety related SSCs such as structural steel, concrete, pipe supports, control rod guide tube supports, susceptible non RCS piping and coatings. DEECS will focus on cabling, conduit, junction boxes, etc. Plant Engineering will perform inspections of ventilation systems within containment (such as CACs and ductwork). Photographs will be made to document any boric acid deposits/corrosion discovered during these walkdowns.

(5) It is expected that (after proper documentation) existing boric acid deposits will be cleaned up. This will prevent future degradation of susceptible materials due to re-wetting of dry boric acid deposits. It will additionally ensure a proper baseline condition for future inspections.

(6) It is also expected that any SSC that has experienced degradation due to boric acid corrosion will be evaluated then reworked or preserved as needed to ensure high standards of material condition and housekeeping.

6.2 Management Issues Relative to the management issues, *theFirstEnergy Nuclear Operating Company (FENOC) had made recent organizational changes to promote an Eanphasis on engineering and engineering decision making. A separate engineering organization under the direction of an independent vice president was put in place after the first of the year to ensure consistent standards in performance Alyi Reo_ ~. *xen ... conmo _ _

Root Roo Cas Analysis Cause Report Lo Extent at Condition

• V

and decision making would be promulgated throughout the FirstEnergy nuclear organization.

This structure promotes the sharing of methodologies and expertise between sites. The team that was put together to provide the leadership and oversight to deal with engineering issues demonstrates the ability to effectively pull expertise and decision making skills from the organization at large. In addition, further management changes at Davis-Besse have been recently made including a new plant manager.

FENOC has the infrastructure to make a step change in technical standards and expertise. In the 1999 to early 2000 timeframe, FENOC and the CNRB were aware that technical activities at the plant had deficiencies and were in need of improvement. FENOC addressed this by reassigning several Managers of demonstrated ability to key positions as "change agents."

Several new programs have been put in place over the last year, many as a result of common process teams, composed from nuclear workers from all First Energy nuclear sites. These processes incorporate best processes from each site and incorporate benchmarking of best work practices from the industry. Many of these processes help to support the going forward infrastructure and technical standards changes that Davis-Besse is making. Examples include an improved corrective action process, and safety evaluation program. Further changes that have been made recently include development a new ownership model, which improves accountability and ensures clear expectations and a new internal self-evaluation program has been implemented.

Finally responsibilities for modification decision making will be elevated to a higher level, requiring site directors to be involved in review of significant capital modifications.

The plant will not restart until a board made up of members of FENOC senior management and independent industry experts review the effectiveness of actions taken. This also includes a review of management issues and their resolution. In addition, the extent of condition review for boric acid damage is extensive and detailed and will ensure that there are no latent unidentified issues related to boric acid corrosion.

6.0 Extent of Condition

• 58 Analysis Report Root Cause Analysis Report 6.0 Extent of Condftlon

• 58

7.0 Recommended Corrective Actions 7.1 Probable/Root Causes Corrective Actions PWSCC cracking In the CRDM nozzle Interface at the J-groove weld due to material susceptibility In the presence of a suitable environment.

1. Develop a plan to monitor for CRDM nozzle leakage. The plan must include steps to repair once leakage is detected. (Plant Engineering)

2. Review Davis-Besse results for CRDM nozzle crack initiation/propagation against the susceptibility model. (Design Basis Engineering Completion prior to restart )

Boric Acid Corrosion Control and ISI programs and program Implementation regarding the RPV head.

3. An extent of condition review for boric acid damage will be performed to ensure that there are no latent unidentified issues related to boric acid corrosion. The results will be reviewed by the senior management team prior to startup. (Plant Engineering Completion prior to restart )

4. The self evaluation program will be revised and ties completed to the Ownership Model.

Bench marking and FENOC common process methods will be used to produce a best-in-industry program. (Learning Organization )

5. Perform Self-Assessments of the boric acid corrosion control and ISI programs. (Plant Engineering Completion prior to restart) The purpose of these Self-Assessments is to evaluate the deficiencies documented in this report. Items to be considered should include:

Boric Acid Corrosion Control Program

• Incorporating as areas for inspection, industry issues such as CRDM nozzle leakage

• Incorporating into the inspection plan systems that carry borated water and provide mitigating type functions that help to preserve the Reactor Coolant Pressure Boundary during plant transients and/or accidents

• Incorporate Boric Acid Corrosion Control Inspection Checklist document retention requirements (retention should be at least several fuel cycles)

• Incorporating a signature block for the Boric Acid Corrosion Control Program Owner to document his review and concurrence with the disposition activities

• Review the use of "should" versus "shall" throughout the procedure.

• Incorporating requirement that boric acid "shall" be removed from affected areas and the affected area inspected to identify any signs of potential corrosion.

a Incorporating a signature block for the System Engineers supervisor to document his review and concurrence with the disposition activities

• Review station commitments to determine if other areas or equipment must be included in the Boric Acid Corrosion Control Program

• Establish a hard link between the Boric Acid Corrosion Control Program and the ISI Program t*at requires both groups to approve the close out of a Boric Acid Corrosion Control Inspection Checklist.

Report Analysis Report 7.0 Recommended corrective Actions

• 59 Recommended Corrective Actions

• 69 cause Analysis Root Cause .7.0

IS Program

• Improve the text descriptions of the areas to be inspected, include sketches of the area and provide a pre-job brief prior to inspecting for bolted connections and Mode 3 leakage during plant heat up

• Eliminate the conflicting text descriptions that are contained in some of the inspection plans

• Evaluate the techniques employed for monitoring CRDM nozzle welds for leakage.

• Reinforce the obligation the ISI program has to protect and preserve the RCS pressure boundary including addressing Boric Acid deposits on the RCS pressure boundary when that specific area was not included in the original inspection plan

• Establish a hard link between the ISI Program and the Boric Acid Corrosion Control Program that requires both groups to approve the close out of a Boric Acid Corrosion Control Inspection Checklist 7.2 Contributing Causes Corrective Actions Environmental conditions, cramped conditions due to the design and high radiation at the RPV head.

1. Provide improved access for inspection and cleaning of the RPV head. (Design Basis Engineering Completion prior to restart)

Equipment condition due to uncorrected CRDM flange leakage.

2. No corrective action is required. No CRDM flange leakage was noted during 13RFO. This contributing cause has been resolved. The monitoring for leakage will continue in the Boric Acid Corrosion Control Program.

Management monitoring of field activities did not Identify problems.

3. Develop a plan for increased presence of management in the field both during outages and during normal operations. Formalization of this program is intended to look for degraded conditions, open opportunities for coaching, and enforcement of management expectations.

(Plant Manager)

Management monitoring of activities did not Identify changes In conditions.

4. Standards and expectations will be immediately adjusted. Pre-startup training will be conducted in small groups to all site personnel ensuring internalization of the missed opportunities associated with the degradation on the reactor head. A case study based on this condition, the missed opportunities, and lessons leamed will be created and provided to all site personnel. (Training Completion prior to restart)

5. Follow-up training will be held over the next 12 months to reinforce technical standards and problem solving skills. This will be required of appropriate management and technical staff.

(Director Technical Services)

6. An operational/decision-making model will be developed and presented to the management team. (Plant Manager)

7. Review/revise charter and membership for the Project Review Committee and Corrective Action Review Board. (Directors Work Management/Support Services)

Report Analysis Report Cause Analysis 7.0 Recommended Corrective Actions e 60 Root Cause 7.0 Recommended Corrective Actions

• 60

Technical standards

8. Augment engineering staff to shore up technical capability and improve engineering rigor and standards. (VP Nuclear Engineering Services)

9. Clarify technical staff expectations to ensure that degraded conditions on systems are promptly identified, corrected, and prevented from recurring. (Director Technical Services Completion prior to restart)

Also reference contributing causes corrective actions 3 and 4.

Oversight

10. A restart review board will be put in place made up of independent industry experts to verify effectiveness of actions taken, and to ensure the management issues are fully developed and addressed prior to startup. (VP Davis-Besse Completion prior to restart)

11. A operation confidence review will be performed prior to startup. The following items should be considered for review: outage issues, condition reports, modifications, work orders, etc. and interviews with the technical staff and program owners. The aggregate system health must be discussed including challenges to reliable operation that may self reveal during operating cycle. (Plant Manager Completion prior to restart)

12. Develop a formal restart readiness review process to be used whenever the plant is to be restarted following plant outages. (Outage Management)

13. Quality Assurance will increase oversight of engineering activities. (Manager QA Due)

14. The CNRB safety focus will be improved by less emphasis on status and LARs and more review of key technical and safety issues. The interval between CNRB oversight visits will be evaluated. (Director OPID)

Previous industry and in-house experience were not effectively used to prevent problems.

15. Improve Operating Experience and benchmarking programs to verify lessons from in-house and industry experience are brought to the Davis-Besse team, meeting programmatic requirements and management expectations. (Director Support Services)

Also reference contributing causes corrective actions 3 and 4.

Execution of the Condition Report Program

16. Review the PCAQR 94-0295 disposition, and initiate commitments and associated document changes as appropriate for performing RPV head visual inspections. (Learning Organization Completion prior to restart)

17. Perform an effectiveness assessment of the Corrective Action program. The purpose of the Self-Assessment is to ensure the categorization of issues, thoroughness of investigation, and that initiation of Condition Reports occurs in accordance with progranmatic requirements and management expectations. (Director Support Services) 7.3 Additional Actions Requiring Investigation

1. Complete the historical Alloy 600 review associated with the CRDM nozzles and summarize the results in the B&W Owners Group and Industry CRDM Nozzle Related Initiatives section of the Root Cause Analysis Report. Items for consideration include:

• The 1994 EPRI Workshop report Root Cause Analysis Report 7.0 Recommended Corrective Actions

• 61

• The EPRI TR-103696 report referenced in the 1994 Workshop Report.

• EPRI NP-6719-M-SD (Feb 8-10, 1989)

• March 5, 1996 NEI white paper entitled Alloy 600 RPV Head Penetration PWSCC

• 1997 EPRI Workshop on PWSCC of Alloy 600 in PWRs Parts 1 & 2 (TR-109138-P2).

• EPRI Workshop on PWSCC Alloy 600 in PWRs, 2/14-16/2000, St. Pete Beach.

• EPRI MRP Alloy 600 Industry Workshop, 6/13-6/1412001, Atlanta, Report 1006278.

• B&WOG Materials Committee Report 51-1229638

• Automated Ultrasonic Inside Surface Examinations of Reactor Coolant System Alloy 82/182 Nozzle Welds Performed in Spring 2001: PWR Materials Reliability Project -

Alloy 600 Issue Task Group, 82/182 Weld Integrity Inspection Committee, EPRI Report 1006225

2. Complete the CRDM flange identification of leak and repair historical review and summarize the results in the Root Cause Analysis Report.

3. Track the analysis of samples collected from the RPV head wastage root cause investigation.

The following samples have already been collected:

• Four samples of rusty boric acid from initial head investigation following insulation removal

• Nozzle 3

• Four samples of deposits including corrosion products from Nozzle Two Removal

• Nozzle 2 Additionally, the wastage area adjacent to nozzle three will be removed from the head and investigated destructively. Currently the samples of boric acid and corrosion products are expected to undergo elemental analysis by ICP and X-ray diffraction to determine crystalline constituents. The nozzles will undergo visual-stereo inspection & measurement, metallography, SEM-EDS, and possibly further analysis to facilitate a better industry-wide understanding of these corrosion phenomena.

4. Extensive effort is currently in progress by the MRP to develop a model for how small leaks from PWSCC cracks progress to modest amounts of corrosion such as seen at nozzle 2 and much larger amounts of corrosion as seen at nozzle 3. While the corrosion is obviously due to the boric acid, the exact stages of progression are being assessed. Mechanisms being evaluated include boric acid corrosion, crevice corrosion, impingement, flow accelerated corrosion, low oxygen corrosion, steam cutting, molton boric acid corrosion, etc. This work includes finite element thermal-hydraulic modeling to determine the effect of steam leakage on locally suppressing the metal temperature in the annulus. It is anticipated that preliminary conclusions from this work will be completed by the end of April 2002. Follow-up on these activities to determine if there is any effect on the root cause.

5. Review the stresses of the CRDM nozzles at both operating conditions and cold conditions.

Determine based upon the stress review if extended time periods at mode 5 conditions increase the likelihood of PWSCC crack initiation.

Root Cause Analysis Report 7.0 Recommended Corrective Actions

• 62

8.0 References 8.1 Davis-Besse References

1. Davis-Besse 13RFO CRDM Nozzle Examination Report, Revision 1,Frarnatome ANP UT Report, March 11, 2002.

2. Potential Condition Adverse to Quality Reports 90-0120 Boron Leakage and CRDM Stator Cooling 90-0221 CRDM Flange F-2 Slight Erosion of Outer Gasket Groove 91-0353 Boron on Reactor Vessel Head from Leaking CRDM Flanges 92-0072 CAC Cooler Degraded Below Acceptable Performance 92-0248 Boron Found in Filter RE4597AA 93-0098 Reactor Head Vent Flange Leakage 93-0132 Reactor Coolant Found Leaking from CRD Flanges 93-0175 Service Water Piping to CAC's Have Accumulated Boric Acid 94-0295 TERMS A16892 Requires Visual Exam of Reactor Vessel Head each Outage 94-0912 Documents CRDM Leakage 94-0974 Documents Scratches and Gouges on Seating Surface Location G-5 94-0975 Document 'AMoon Gouge CRDM Flange M-3 94-1338 Westinghouse CRDM part 21 96-0551 Video of CRDM Flanges Shows Evidence of Leakage 96-0650 VT-2 Exam of RCP Stud Shows Evidence of Boric Acid Leakage 96-1018 Info Notice 96-032 Received Concerning Augmented Inspection of Rx Vessel 1998-0649 Inspection Results of Reactor Vessel Head 1998-0650 Video Inspection Results CRDM Nozzle/Head Interface 1998-0824 CAC's 2 and 3 Have Accumulated Boric Acid 1998-1164 Water Collecting in Sample Line for RE4597AA 1998-1885 RC-2 Carbon Steel Nuts 1998-1895 Containment Normal Sump Leakage > IGPM 1998-1980 Containment Cooler Plenum Pressure Decreasing

3. Condition Reports 1998-0020 Multiple Problems with RC-2 1999-0372 Containment Rad RE4S97AA/AB High 1999-0510 RE4597AA OOS Low Flow Root Cause Analysis Report 8.0 References

• 63

1999-0845 Boric Acid Clumps Room 181 1999-0861 RE4597AA Sample Line Full of Water 1999-0928 Document Increased RE Filter Change Frequency 1999-1300 RE Filter Analysis Results from Southwest Research Institute 1999-1614 LER 1998-009 1999-1098 Issues with DB-OP-01200 RCS Leakage Management 2000-0781 Boric Acid on RV Studs 2000-0782 RV Flange Boric Acid from Weep Holes 2000-0903 Two CRDM Flange Fasteners Fail Preservice Exam.

2000-0994 CRDM Flange F-10 Pitted 2000-0995 CRDM Flange D-10 Pitted 2000-1037 Reactor Head Inspection Indicates Boric Acid Accumulation 2000-1210 CRDM D-10 Out of Plum 2000-1547 Containment Cooler Plenum Pressure Dropped 00-4138 Increased Frequency of Containment Air Cooler Cleaning 01-0039 Step Drop in Containment Air Cooler Plenum Pressure 01-0487 Higher Containment Temperatures 01-0890 RCS Leakage Calculation Data Scatter 01-1110 RE4597BA Filter Change Occurring More Frequently 01-1822 Increasing Frequency of RE4597BA Filter Changeout 01-1857 RCS Leakage Anomalies 01-2012 Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles 01-2769 Containment Wide Range Radiation Element (RE2387) Spiking 01-2795 RE4597BA Alarm 01-2862 Potential Adverse Trend in Unidentified RCS Leakage 01-2936 Unable to Perform RE4S97BAJBB Functional by the Technical Specification 01-3025 RCS Leakage 01-3411 Equipment Failure on Detector Saturation During RE4597BA Testing 02-00685 Boron Build Up on Reactor Vessel Head 02-00846 More Boron on Head Than Expected 02-00891 Control Rod Drive Nozzle Crack Indication 02-00932 CRDM Nozzle Crack Indications 02-01053 Unexpected Tool Movement

4. Procedures 8.0 References

• 64 Analysis1 Report Root Cause Analysis Report 8.0 References

• 64

DB-OP-01200 Reactor Coolant System Leakage Management (rev 0 thru 3)

DB-OP-06900 Plant Heatup DB-PF-00204 ASME XI Pressure Testing DB-PF-03010 RCS Leakage and RCS Hydrostatic Test DB-PF-03065 Pressure and Augmented Leakage Test NG-EN-00324 Boric Acid Corrosion Control (rev 2)

5. Other Station Documents Davis Besse System Health Report, 4"' Quarter 2001 Request For Modification 94-0025 Install Service Structure Inspection Opening Inservice Inspection Plan (1SI Plan) Volume I Third Ten-Year Interval Pressure Test Program Inservice Inspection Plan (SI Plan) Volume II Second Ten-Year Interval Pressure Test Program Relief Request RR-A3 Insulated ASME Class 1 and 2 Pressure Retaining Bolted Connections Relief Request RR-A10 ASME Class I and 2 Pressure Retaining Bolted Connections System

Description:

• SD-022B Containment Air Cooling System and Recirculation System

• SD-39A Reactor Coolant System Technical Specifications:

• 314A.6.1 Reactor Coolant Leakage Detection Systems

• 3/4A.6.2 Reactor Coolant System Operational Leakage

• 3/4A.10 Structural Integrity ASME Code Class 1,2, and 3 Components Updated Safety Analysis Report Sections:

• 5.1 Reactor Coolant System summary Description

• 5.2 Integrity of Reactor Coolant Pressure Boundary (RCPB)

• 11.4A.4.5 Containment Vessel Monitor

• Fig. 5.1-2 Functional Drawing Reactor Coolant System

• Fig. 5.1-3 Reactor Coolant System and Supporting Structures - Plan

• Fig. 5.1-4 Reactor Coolant System and Supporting Structures - Plan RWP 2000-5132 Clean Boric Acid from Rx Head 8.2 Vendor References

1. B&WOG Integrated Response to NRC Generic Letter 97-01 Degradation of Control Rod Drive Mechanism Nozzle and Other Vessel Closure Head Penetrations, BAW-2301, Framatome ANP Report, Julys1997 *

2. Frarnatome ANP Report 51-5001951-01, Alloy 600 PWSCC Susceptibility Model, December 9, 1998 (Proprietary) 8.0 References

• 65 Root Cause Analysis Report Cause Analysts Report 8.0 References

• 65

3. Oconee 1 RPV Head Nozzle Leaks presented by Dave Whitaker at EPRI Alloy 1TG meeting January 19, 2001

4. Dominion Engineering, Inc. Calculation No. C-5509-00-6 Davis Besse CRDM Leak Rates using ANSYS Crack Opening Area (non-safety related), Revision 0 3119/2002 (Proprietary)

5. Dominion Engineering, Inc. Calculation No. C-5509-00-7 Davis Besse CRDM Nozzle Crack Opening Displacement Analysis, Revision 0 3/19/2002 (Proprietary)

6. Dominion Engineering, Inc. Calculation No. C-5509-00-5 Leak Rate through Axial Crack in Davis Besse CRDMs (non-safety related), Revision 1 3119/2002 (Proprietary)

7. BAW-10190P Safety Evaluation for B&W-Design Reactor Vessel Head Control Rod Drive Mechanism Nozzle Cracking (Proprietary)

8. BAW-1019P Addendum 1 External Circumference Crack Growth Analysis for B&W Design Reactor Vessel head CRDM Nozzles (Proprietary)

9. BAW-1019P Addendum 2 Safety Evaluation for Control Rod Drive Mechanism Nozzle J-Groove Weld (Proprietary)

10. B&WOG Materials Committee Report 51-1201160-00 Alloy 600 SCC Susceptibility Scoping Study of Components at Crystal River 3

11. B&W Report 51-1218440-00 Alloy PWSCC Time-To-Failure Models (Proprietary)

12. B&W Report 51-1219143-00 CRDM Nozzle Characterization (Proprietary)

13. Dominion Engineering, Inc. Calculation No. C-5509-00-7 Volume and Weight of Boric Acid Deposits on Vessel Head.

8.3 NRC References

1. GL 88-05 Boric Acid Corrosion of Carbon Steel Reactor Pressure Boundary Components in PWR Plants

2. GL 97-01 Degradation of CRDM/CEDM Nozzle and Other Vessel Closure Head Penetrations

3. Regulatory Guide 1.A5 Reactor Coolant Pressure Boundary Leakage Detection Systems

4. Bulletin 82-2 Degradation of Threaded Fasteners in the Reactor Coolant Pressure Boundary of PWR Plants

5.Bulletin 2001-01 Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles

6. Bulleting 2002-01 Reactor pressure Vessel head Degradation and Reactor Coolant Pressure Boundary Integrity

7. IN 80-27 Degradation of Reactor Coolant Pump Studs

8. IN 82-6 Failure of Steam Generator Primary Side Manway Closure Studs

9. IN 86-108 Degradation of RCS Pressure Boundary Resulting From Boric Acid Corrosion

10. IN 86-108 Supplements 1 & 2 Degradation of RCS Pressure Boundary Resulting From Boric Acid Corrosion 8.D eierencGs ab Analysis Report Root Cause Analysls 8.0 References a us

11. IN 86-108 Supplement 3 Degradation of RCS Pressure Boundary Resulting From Boric Acid Corrosion

12. IN 90-10 Primary Water Stress Corrosion Cracking (PWSCC) of Inconel 600

13. IN 94-63 Boric Acid Corrosion of Charging Pump Casing Caused by Cladding Cracks

14. IN 96-11 Ingress of Demineralizer Resins Increases Potential for Stress Corrosion Cracking of Control Rod Drive Mechanism Penetrations

15. IN 2001-5 Through-Wall Circumferential Cracking of Reactor Pressure Vessel Head Control Rod Drive Mechanism Penetration Nozzles at Oconee Nuclear Station, Unit 3

16. IN 2000-17 Crack in Weld Area of Reactor Coolant System Hot Leg Piping at V.C. Summer

17. IN 2000-17 Supplement 1 Crack in Weld Area of Reactor Coolant System Hot Leg Piping at V.C. Summer

18. IN 2000-17 Supplement 2 Crack in Weld Area of Reactor Coolant System Hot Leg Piping at V.C. Summer

19. IN 2002-11 Recent Experience with Degradation of Reactor Pressure Vessel Head

20. Safety Evaluation for Potential Reactor Vessel Head Adaptor Tube Cracking, November 19, 1993 8.4 INPO References

1. SOER 81-12 Reactor Coolant Pump Closure Stud Corrosion

2. SOER 84-5 Bolt Degradation or Failure in Nuclear Power Plants

3. SER 46-80 Reactor Coolant Pump Closure Stud Corrosion

4. SER 35-81 Corrosion of Reactor Coolant System Piping

5. SER 11-82 Reactor Coolant Pump Closure Flange Stud Corrosion

6. SER 57-83 Cracking in Stagnant Boric Acid Piping

7. SER 72-83 Damage to Carbon Steel Bolts and Studs on Valves in Small Diameter Piping Caused by Leakage of Borated Water

8. SER 32-84 Contamination of Reactor Coolant System by Magnetite and Sulfates

9. SER 41-85 Containment Spraying Events

10. SER 13-87 Reactor Vessel Stud Corrosion from Primary Coolant Leak

11. SrR 31-87 Pressurizer Vessel Corrosion due to Pressurizer Heater Rupture

12. SER 35-87 Non-Isolable Reactor Coolant System Leak

13. SER 10-89 Reactor Coolant Pump Flange Leak from Loss of Bolt Preload. Bolts should be checked for preload

14. SER 90-2 Pressurizer Heater Sleeve Cracking

15. SER 20-93 Intergranular Stress Corrosion Cracking in Control Rod Drive Mechanism Penetrations

16. SER 4-01 Recent Events Involving Reactor Coolant System Leakage at Pressurized Water Reactors Root Cause Analysis Report 8.0 References

• 67

17. SEN 6 Boric Acid Corrosion

18. SEN 18 Reactor Vessel Head Corrosion

19. SEN 190 Pressurizer Spray Valve Bonnet Nuts Dissolved by Boric Acid

20. SEN 216 Leakage from Reactor Vessel Nozzle-to-Hot Leg Weld

21. SEN 220 Pressure Boundary Leakage at Palisades. Palisades had a through-wall crack in a CRDM housing

22. O&MR 348 Failure of a Limitorque Operator Stem Nut 8.5 Industry References

1. PWSCC of Alloy 600 Materials in PWR Primary System Penetrations, EPRI TR-103696.

(Proprietary)

2. EPRI Technical Report -104748 Boric Acid Corrosion Guidebook (Proprietary)

3. EPRI Technical Report -1000975 Boric Acid Corrosion Guidebook, Revision 1 (Proprietary)

4. EPRI Technical Report -103696 PWSCC of Alloy 600 Materials in PWR Primary System Penetrations (Proprietary)

5. MRP-44, Part 2, PWR Materials Reliability Program - Interim Alloy 600 Safety Assessments for US PWR Plants, Part 2: Reactor Vessel Top Head Penetrations (Proprietary)

6. EPRI NP-6301-D, Ductile Fracture Handbook

7. EPRI Technical Report -107621-Ri, Steam Generator Integrity Assessment Guidelines:

Revision 1 (Proprietary)

8. EPRI draft report NP-6864-L, PWR Steam Generator Tube Repair Limits: Technical Support Document for Expansion Zone PWSCC in Roll Transitions

9. MRP crack growth rate report (Proprietary)

10. EPRI NP-7094, Literature Survey of Cracking of Alloy 600 Penetrations 8.6 Other References V.C. Summer Nuclear Station Root Cause Investigation "A" Hot Let Nozzle Weld Cracks 8.0 Reterences

• 68 Analysis Report Root Cause Analysis Report 8.0 References a 68

9.0 Personnel Interviews 9.1 Personnel Interviewed Andrew Siernaszko, current Davis-Besse RCS System Engineer Ed Chimahusky, former Davis-Besse RCS System Engineer Dan Haley, former Davis-Besse RCS System Engineer George Chung, current Davis-Besse Radiation Monitor System Engineer Bob Hovland, former Davis-Besse Radiation Monitor System Engineer Walt Molpus, current Davis-Besse Boric Acid Corrosion Control Program owner Peter Mainhardt, performed Davis-Besse Reactor Vessel Head inpections Jerry Lee, Davis-Besse Leak Program owner Glenn McIntrye, former Davis-Besse Mechanical Systems Supervisor Jim Marley, Davis-Besse System Engineering Pete Seniuk, Davis-Besse ISI Pressure Test Engineer Chuck Daft, Davis-Besse ISI Engineer Mike Shepherd, Davis-Besse ISI Engineer Prasoon Goyal, Davis-Besse B&WOG Material Committee representative Ken Byrd, Davis-Besse Nuclear Engineering (PSA Engineer) Supervisor Rich Edwards, Davis-Besse Chemistry Technologist Bruce Geddes, Davis-Besse Containment Deconing Mark Mclaughlin, Davis-Besse CRDM Project Manager Charles (Steve) Steagall, Davis-Besse VT-2 Inspector Richard Cockrell, Davis-Besse VT-2 Inspector Chuck Ackerman, FENOC Quality Assurance Engineering Supervisor Henry Stevens, FENOC Manager Quality Assurance Dave Lockwood, Davis-Besse Manager Learning Organization and Regulatory Programs Dave Geisen, Davis-Besse Design Basis Engineering Manager Dave Eshelman, former Davis-Besse Plant Engineering Manager Joe Rogers, Davis-Besse Outage Director Scott Coakley, Davis-Besse Outage Director Steve Moffitt, Davis-Besse Director Technical Services John Messina, Davis-Besse Director Work Management John Wood, FENOC Vice President Engineering Services Jim Harris, Framatome 13R Reactor Services Lead Fred Currence, Framatome 13R Reactor Services Lead Mike Hacker, Framatome UT expert Rich Garrison, Frarnatome CRDM Nozzle InspectionlRepair Manager Ron Pillow, Framatome CRDM Component Engineer Steve Fyfitch, Framatome Metallurgist Cary Bowles. Framatome Rod Emery, Oconee CRDM Engineer w- i'ersonnei ineviw Root Cause Analysis Report 9.0 Personnel Interviews au

9.2 Personnel Consulted Steve Moffitt, Davis-Besse Director Technical Services John Hickling, EPRI Materials expert Kim Kietzman, EPRI UT expert Chuck Welty, EPRI Director Jeff Gorman, Dominion Engineering PhD Materials expert Chuck Marks, Dominion Engineering PhD Chemistry expert Matt Brown, Radiation Protection Servicemen uuerviews . iu Repofl Root Cause Analysis Report G-0 Personnel

.u Personnel interviews

• 70

10.0 Methodologies Employed Event & Causal Factors Charting Procedure Review/Analysis Difference Analysis Barrier Analysis Possible Cause Analysis Structured Interviewing 10.0 MethodologIes Employed

• 71 Root Analysis Report Cause Analysis Root Cause Report 10.0 Methodologies Emloyed

• 71

IFigures RV HEAD INSULATION SERVICE STRUCTURE CRDM NOZZLES' SUPPORT STEEL -

18 ACCESS OPENINGS 2" MIN GAP BETWEEN

'MOUSE-HOLES' AT INSULATION AND TOP DAVIS BESSE OF RV HEAD I I

I Figure 1.Davis-Besse RPV Top of Head Section View "gft Root Cause Analysis Report Figures *wow

Somwce: EPRJ/DE N

Location of e/xSisig ai 0 0 ° °0 h/oles used for CRDM cle cleaning nn Nozzles and /0in)Rspection cr CrcrcrO

/0~ d, d ( d 6 \ 0

/0 ~ c0 C dd d d' d 4 W EE FW 2. °BI re o o Hd VE O 6 o r Cot (5 o r o r o

\O\ \ d C (T C ( //0

\01. Cr 0 C d //0 Closuree' VW&0° Fig=r 2. Davis-Besse RPV Top offHcad Plan View Caus AnlssReoti-aue Root Cause Analysis Root~~~~~~~2 Report FlIgures

• V-1

__~ ;._

i ***..-

-e _ ~ .O Convol oI Drive Mcthanihin

_ }alrxWlliccTypcGuLres L', SrlitNutRios CoverPlaie AUNcy <<m Forge Low-Aloy Strel RencVnrcsscl IL-ad Stainless Steel Cadding Wdd e , 3e I 5 Figur 3. Daiis-Besse CRDM-Nozzle Genieal Arrangement figures .uz Root Cause Analysis Report IFigures a Uz

Figure 4. Boric Acid and Iron Oxide on Vessel Flange at 12RFQ rigures 6 fl Root Cause Report Analysis Report Cause Analysis Flgures a V3

-1

/~ X

- AIDS DEEp CORXbDEf

• 4Wjr occur D-rBCVEL.V.

b-caA+CO *_D CatC I.J CA A

Figure 5. Corrosion at Nozzle 2 Drawing Side View iigures 0 4 Root Cause Analysis Report Figures 0 W4

'st

- ',/ .',,'- g.- ' Z .- '- ,',' *L......

- AM.r 0 - - 10 D., , .

I ,g . ;..

2t0

....-----........... -,. ,,. k;-----'-- ------ I-

...;j... ...........

Z30

............. .. . ..... . .. -... t-225 230 .................. -A-.. .... ... ...

2(5 .................

Xss ......... ......... .. .. .1. . . . .. 1

~~~~. r-t-

,,......~ ~..!.~ ..... .. ..

bss

. ~.....-..

ii__'K'1 Wzso ..... ...... .... . .. . . . . ... . 1 .S,.._

'. l*l J-.....--- .

7I5

.......... .. ... i

.......... .... ....--. 1 230 Z5.

S25 ao.

-Mo.. _ _ _ l _

F 1-2-~-3-4-5-67-7--9-10-11-12-13 14 - 10Nxf I _

1/0 t

_ , _t

-Jk- - -

0= , ...- I X .. o I

& 3'1 I= I d4 Figure 5a. Nozzle 2 Corrosion Area Location, Size, and Profile.

Figures .95 Root Cause Analysis Analysis Report Report Figures a 95

I L

Figure 6. Cavity in Reactor Vessel Head Between Nozzle 3 and II El-, OdIm - a ffuskIWV. -WB Root Cause Analysis Report

Source: EPRI/DEI Thru-Wall Axial Crack Circ Crack Corrosion of Low-Alloy Steel (extends down along annulus wall)

Wastage I (extends to dad)

Figure 7. Locations of Cracks and Corrosion on Davis-Besse RPV Head at 13RFO r uw, Root Cause Analysis Report rigurab awe

Ne1edge TqaDana 0 9m tflA 2L5 - - - - - - - - -

I 225 ----------------

20 ---- - - -- -- - .-.

25 ---....--------.

1----------------

---

I

. . - - - - I--

20 25 - -- -- -- -- -- 1-- - -

-------- 1---- -- -- T F--'--------------

- - ------- -F-1---;1 --

--t r-l-------r-- ---1----

2M0*

1----- 1--

BO Z;1

-- ~__

t -Il

, , l j-i--- I----I1-----I-------1---

Sr

--- --- 1 t

_ l so - .---. - --.- -------- -----I wi5-- -------- - - - - - - -- - ----- J mofl -------

Ms U------------------ --- --------------- -- - - - - ---

1 r-1 3-4-5-6-7 1 12-13-14-15-16-Nz Ed Figure 8. Nozzle I Crack Locations and Sizing er"..gua n oU riulfarS

• VU Root Cause Analysis Report

N=Je2 Pf~die 183mit 0

91fl4 i A 2L5 220 0 -----------------

725 ----------------- ----------------- ----------------- 7----------------

---

----------------- ----------------- L----------------

2W0 ----------------- ----------------- ---------------- ----------------

2g5 ---------------- ---------------- ----------------- ----------------

M:

Xso 25 -----------------

20. ----------------- ------------------ ----------------

20 ---------- - - - - - - - - - - - - - - ---------------

25 20 MBO 25 M5 ------------------ ----------- -

i I _ _ _ _ _

_______________ I _______________

nnn ECU

_-1 _2-3-4_5_6-7-8-9-V_11 13 1-15-- -Nx Bf Figure 9. Nozzle 2 Crack Locations and Sizing U Aft Root Cause Analysis Report riuUSue ova

INe3Rdle 0 OD 2Z O" AlI __ _ _ _ _ _ _ _ _ _ _ _ 1.

But M5. ----------------

.1----------- -- ---- ---------------- ---------------

X5. ---------------- ----------------

240* ---------------- ---------------- --------------- ---------------

Its. ---------------- ---------------- ---------------- ----------------

EX0- ---------------- ----------- 1-1I EXz5. ---------------- - - - - - - - - - - - - - - - ---------------

E 25. -------------- - - - - - - - - - - - -

fr.

25 -------------- ----------------- - - - - - - - - - - - - - - - ---------------

---

l 325 A2W -I- -I - II -

I1 Fiu 3-C2-1k Lc-ai-n 1N2-3-4-5-6-7-8-9-ozzl an- S Figure 10. Nozzle 3 Crack Locations and Sizing Root Cause Analysis Report Figures

• 1W

• I REtS ile 1TD~bnrma 0 a) zOJ von 4

,M5. -----------------

---

----------------- I---------------- -----------------

---

I-----------------

2W. ----------------- ----------------- ----------------- -----------------

X~5' ----------------- ----------------- ----------------- -----------------

zo. ----------------- ----------------- ----------------- ----------------

Z5- ----------------- ----------------- --- ------------- -----------------

&Mo. ----------------- ----------------- .. .............. -----------------

ro.------------- --- ----------------- ----- ---------- -----------------

W~5. ----------------- ----------------- -----------------

3EW. -----------------

• p

• - -- - - I-- - - - - - - - ---- - - - - -

£ ___________________ a a 2- 3-4-5-6-7-8-9-1-- 11-12-3 14 16--N Figure 11. Nozzle 5 Crack Locations and Sizing a

RepotCaus Ana ysis Fi ures @10 Root Cause Analysis Root Report Figures

• 101

NuieQ7R~ce 0 IED Ma . 2 a I MS. ----------------- --------------- -----------------

J75. ----------------- --------------- ----------------- - - - - - - - - - - - - - - - -

Sao. ----------------- --------------- ----------------- - - - - - - - - - - - - - - - -

ao------------------ -------------

&4W-425- ---------------- ----------------

610 ---------------- ----------------

65 ----------------- ---------------- - - - - - - - - - - - - - - - -

I _ _ _ _ _ _ _ _ _

1-1 3-4-5-6-7--9-0-11-12-1 4-5--16-NBl Figure 12. Nozzle 47 Crack Locations and Sizing

- IUftf Root Cause Analysis Report rlwUreS a fun

Source: EPRIIDEI Areas of Overflow During Parts of Cycle

- Remaining Portion of J-Groove Weld Note: This sketch is bated an preliminay inspection I infonnationpriorto receiving the cavity casting impression . . -

Figure 13. Characterization of Corrosion and Impingement at Nozzle N-3 r.;u.ro - q Un r-1vurao

• swj Root Cause Analysis Report

DAVIS-BESSE RFO-13 1O

- p - p - U- F p - p-A B C b I JT Nozzle 3 u I 2770 1 . .1 I ._

F4R I 4 9 62 6.2 t.or24 K s/

1 *n H

n N

6.- .4 -00 -Q4.4- - -.-

- - - , -. 49- -040-.255 0427J -0420 - 04352 0.299 50- 00 6  :.94 -00 ,0240 -04:00

-040fE- 2Ool -040- -0400N- -4i3A0- a -1F0 0.310 --6.- .3a 5 7 a.S

-- <^F-0.01 C. 4htH>--4hUX# -4ffiB- .aooa AO% 2-4406-6.

9 . ah_ 0.279 -4.2UKS 0.402KS -4.3UK0 O.J0 0 270 0.370 0 303

-- NJ-- .070-- -

10 -NR-11 2 9 AA __

A B C , D t r a n a 4 "

Nozzle I 1 Approximate Location

• Figure 14, Nozzle 3 Clad Thickness Measurements

- dOA Root Cause AnaWls Report Figures a 10W

ANSYS 5.7 ii MAR 19 2002

'~ '~14:46:34 PLOT NO. I ilil NODAL SOLUTION SY (AVG)

RSYS=11 DMX -.442069 SMX 060

-10000

_ _ 0 10000 20000

__30000 40000 50000 100000 ODBCRDM(8d,48.5k,4/2.765,0,A) -Operating Figure 15. Hoop Stresses and Operating Condition Deflections in CRDM Nozzles 2-5

- 40%r rlSwufreb* Mlaw-Root Cause Analysis Report

Source: EPRI/DEI

• Leaking CRDM Nozzle (with circ cracks)

G Leaking CRDM Nozzle (no circ cracks)

Leaking TIC Nozzle Oconee 3 Uphill 165-100% Thru Oconee 3 Uphill 165' Also location of ANO- I leaking nozzle without Doo 100% Thru circ crack above wed 0t X00

\X\ \ O\ Downhill 48-O V\ 5 \ 29% Thru Davis-Besse 0" /Do Downhill 33'

, 55% Tnru Oconee 3 Downhill 66-35%Thru Figure 16. Location of Leaking Nozzles in B&W Design Plants

- 4" Root Cause Analysis Report FljUMes

• sub

I0 Leaks Without Circ Cracks Above Weld D Leaks With Circ Cracks Above Weld 0.25 0.20 Fraction of 0.15 Nozzles With Lcaks 0. 10 0.05 0.00 0.0 8.0 11.4 16.2 18.1 23.2 24.7 26.1 30.1 33.9 35.0 36.2 38.5 Angle of Head Slope (deg)

Figure 17. Distribution of Leaking Nozzles in B&W Design Plants

, -  : . . . . . a Figures .107 Root Cause Analysis Report Analysis Report Figures a 107

Figure Observed at Oconee 3 Clt~Itme a lflR 5 fj - -- - . --

Root Cause Analysis Report

0.4 0.3 E-

.S 0.2

• 5s 0.1 to/IJOO [VIMO 131JOI 4n1 6/WI U20I 1a1V 12no1 21D fls VI=o Date Figure 19. Unidentified Leak Rate at Davis-Besse (Cycle 13)

-_,nISeteP ItlO ra Uwvw -; aw lO Root Cause Analysis Report

Source: EPRI/DEI N Leaking Nozzles

/N-I, N-2. N-3 Possible Leaking Flangeas N-3, N-6. N-Il, N-5I 0A w IE S

l I Affected Area at RF-IO Additional Affected Area at RF- I I l l Additional Affected Area at RF-12 Additional Affected Area at RF-13 Figure 20. As Found Locations of Boric Acid Deposits on Davis-Besse Vessel Head (lORFO to 13RFO)

.- 4 I

- 44qft Root Cause Analysis Report rgIulwow I Nuv

Figure 21. Nozzle Crack Leakage Rate Calculation Results Flnures

. .B .11I-Root Cause Analysis Report

Head Material Removed 90 Around Nozzle From Symmetry Plane

. ' Nodes Spaced Axially at 0.125' g ~~Weld Topv6 E Axial Cracking Region 101 \ 05 Downhill Plane Nodes are O's Serics Uphill Plane Nodes am 30,000's Series Tube Node Scries I's a Nozzle ID. 5's at Nozzle OD Shell Node Series: 5's at Shell ID (merged w/Iube OD) in weld region 6's at Shell ID above weld region 15's at edge of shell section Node Numbers Incmese by 100 up the length or the tube and shell Node Numbers Increase by I along the tube and shell tadius Figure 22. Finite Element Model Boundary Conditions to Simulate Axial Crack r-ue U lIt Root Cause Analysis Report Fliquir-s a 12.

DB CRDM(Sd,46.Sk,4/2.765,)- Ax. Crack to 1.25 in. Above Weld I Figure 23. Crack Opening Displacement with the Crack Surface Nodes Released

.  :. . a;

_ a do Root Cause Analysis Report F19UreS a* 114

8 \\ 09 Co.

1 0 D .I C m ISsr (i) F)X s*+e(D 0- e w___8__ e_, 'D e FT io Figure 24. Boric Acid Deposits on Top of Head at Start of 13 RFO FIgures .114 Root Cause Analysis Report Figures a 114

Source: EPRI/PEI Rev. 3 (4117)011 4.

WIrC ACId Crygalo I II.) 3. 4) to am id air % 7037 IRI30122 - I 4 III 0.nt~d Not..

-I-

• .50 if) :4)

-2.51 100.7 la **j ii

• 2.50 . 22 140?

11 3 .000 F. 2 01

• 2.000 . . 5008, S

II4)

.1

-, I ,,

-)

!I. b900 Ii

?) I)

- 72pp. s.20) at

• . - -- mm- -

pp..0 420) 4D31.43)

-- NiI 4.;

-- , , I.'-

~

• 2.00 070;?IA)

• -+-

'FHi-w 2 .000 pp. £ 0 M0  :  ; L . - "4

• 2.000L. to0 4001-1' S_ _' 1 el r ___C.

. . t  : &I

• 2.00 5 100)?

.i -

AT23 .404 0400 10) II  !:.I ....... I

• 22.048 no. 20 Sel (I 1

-10 I

4Jliii" 00 I

I .. 4, i .m

$ 5-,. .. I. - ..w. I 4.300 PP.0 2 22,7 10.2  ; i ,, _ 4 i i  !

. 1. ! jjj:

-22.000 pp 22070 gut II ... ,F.,.j. . r

-20.000 PM a 220,7' 40) Ii i : '.

44.4000 376 20410, 801) . -.i.k1k .-

70.000 pp.0 220.710 1*1 II Ii I i. I ta-'

A0TO1*d NOW0 -

• 4.000 pp0 1501?

S202) IF)

+:.......

7 i

ji l

, i I.

_f

,:-)_ _ _s. _

,'4.

j_ I I _ h__ _

- ...- ,, - '1: -I

-"!0 I : ;!;

I-4ft-33

..  : -- -- --- -t!

- 4.0040 pp 0 410100r 47)

I I.

U -

;ij D.1.01. MteOW0.S0 glrfa.

l..-.m

- 24.,1- .W. , 2_k ?*.. ,, -_- -.

-- , i i ..

• 14. sea pp. 0 200)? I I 3000~Md T000OS OI-

__,l -44 -- H44V-1' 4- JI

-1

• 11.00 375 200) Wl *I -. i

- 12.000 *p 07l I 2.00 2200pp. a

- "I

00) t1.r(2 t

4.3 (3002.0) t0) 1. I !i TI I

Ii :!1 I

I ,T

-'r

~I I

.1.

I:

, . I I 3 *I ,-I0 --

-  ! );1 . I ,: -'!":- `'--1 --7 J 1 ,0 .

• 1t1) 41) I..

_ iI .'.

-ea s.@ 'o: .Lo i v saig S. V

• 1.60

• 200)? 102 I )

4 .:

-.W ; I I.

1.000 pp.

2.00

• 0008 w 000,? 83001.7)7200 tl.)

0n mp 0mm - 3  : . I .;

i . ;

. . 7 11 0 I-i ,

.*Q0 t. O,-, i - 7.

-.~4 ',:i{l1 .

inoA 0 I 13, I . .

.~~I aS_ F l-61 . .:- I .;- - :: i I - . .: :

cw' Ocm1 -c1 .1 I 10 Corsion Rate (ichesar)

Figure 25. Corrosion Rate for EPRI Experiments W

Root Cause Analysis Report Vigures *- 14)

0 0

3WrCC EPIWDE1 l 1* I ov Cs I so& AcHub" -

I III9i I bm I I I lI M aq 0 Cr, t4 0 0

Ii:: 9 til S-o

$I m 0

9 0

0 20 90 BA G1 V.rff Rqacewne AA 00 0,

0 (at 0

~o VI 0

-A 2 w

0 0

0e<

PA. lim. LI s~ikuicinom PoiuetoIORF CRtDM flmg 0 B.

3OWUmdabbbalbpsWd. bppw bh RPViftU I)Imsp~mtW tDA~~mbust Fidi .d.mere ilomkwil~po S~AWtbwesbi~b~fubwM t fb~uWbmbisma "a*hbhu-i Jim~i &pukhsm I 1P"W*.v k IASbS.q.db Cg FN" 1.d QAUI 43*puofmudMoeb mmo COMMHe

_ _ _ _ _ __ __ _ __ __ __ __ _ _ _ _ I.- ?

In MANhof KreyEweaftRetam to rqmw HA. .

Reacte VieW lied Dark AddCUTUsiI

Figure 27. Events and Causal Factors Chart On following 5 pages:

Figures .117 Root Cause Analysis Report Cause Analysis Report Figures

• 117

AwMl~E causes;

@Primary Water Stress Corrosion Cracking Fabrication and inspecdon anomolles Thernal atigue

'RCS chemistry cons  :

Resin hitsions Ovter failure mechanisms

,- -h

.4 Jloy 600 nozzle a susceptible to Pdrary Water

% St=ssComaoslon Cracking ,

Modifications

Credited.

Is ton 50% mmesibility to RPV hwa.

less esolves PCAQ 96-055. MOD addresses rMOD plant life extension. Committee concurred hat MOD should be approved but discussed vadrous Issues related to schdulng InI2RFO.

Radiation Monitors and Containment Air Coolers I

_ig fz vokbe ~ vfi-RFQ noa bfolble 40hedule, RCS leakage source known and notB head, we have Inspected any bodc add sting on the head. dry condion and corcslon attark not an Issue, delay to 13RFO does not add risk. however factor and the MOD shotid be addressed.

.4 .

0 0 0 a 0 ES4 0 0 0 0 0 0 0 0h AeN Q 0

~~0 0 A06 A AM a 0 90k .. A "9&7 CIM>

Mrv 0

A A 0 0 0

0 0v Q 0 0 0 1990 Repaired CRDM Flanges (23) 0 1994 Repaired CRDM Flanges (8) 1991 Repaired CRDM Flanges (15) 0 1996 Repaired CRDM Flanges (10) 0 1991 Leaking CRDM Flanges (7) 0 2000 Repaired CRDM Flanges (5)

G 1993 Repaired CRDM Flanges (15)

Figure 28. Leaking Flanges Found and Repaired During Each Outage Report FIgures .118 Root Cause Analysis Analysis Report Figures a 118

mm I

- at. - - - i; Fiur 9.Flne eaae it tacteFo ainfrmIslin and tpo eatrVeslHad 4tlgitjomtino RO 4 I

rigures .1-lu Analysis Report Root Cause Analysis Report . _ .e _ . 4v

Figure 30. Flange Leakage Crusted On Side of Nozzles and Stalactites from Gaps in Insulation (8RFO)

Root Cause Analysis Report Figures e iZU

Figure 31. Reddish Brown Boron Deposits Crusted on Side of Nozzle (8RFO)

- 4054 rapture: - Ise Root Cause Analysis Report

Figure 32. Boron Deposits - Source Unclear (SRFO)

- 4_On Root Cause Analysis Report rlwu[sw . .&

Figure 33. North Side of Reactor Vessel Head (lORFO)

, &14 w .'

uI---- 4gMe5 Root Cause Analysis Report Flguumb

• lul

- ?t' -. - r, *'

Ir A:;~~ ~ ~ ~ t -A ,...-J Figure 34. Boron Deposits Near Top of Reactor Vessel Head (lORFO)

Root Cause Analysis Report Figures a 1Z4

liO Figure 35: Typical Deposits for Periphery (ORJFO)

Root Cause Analysis Report Figures

• 125

Figure 36. Red Rusty Boric Acid Deposits on Vessel Flange (12RFO)

. ,e ..

1-agures . azo Root Cause Analysis Report Report FIgures

• l2b

Figure 37. Boron Piled Under the Insulation (1IRFO)

Ftnuras a 127

• *t, Root Cause Analysis Report

Figure 38. Boric Acid Deposits with Heavy Iron Concentration on Underside of Nozzle 3 (13RFO)

, ,J i_ a 198

,Mrivimc _

Root Cause Analysis Report

I a a 0 a a 0 0

0 0D a 0 0 0D 0D 0 0D O so 00 a 0 0@0 a

F - I Figure 39. 2000 Interferenc'8 with CRDM Flange Inspection Figures

-

• 129 Root Cause Analysis Report

CTMT Air Coolers:

Lower level - 585' 00 CRDM Vent. Fans (2)

Refueling floor -608' N (not to scale)

Figure 40. RE4597 Sample Location Figures

• 130 Root Cause Analysis Report

CTMT Radiation Monitors RE4597AA/BA (Combined Iodine Channels)

I.E-07 I.E.07 II

I.E-07 8.E-0B 6.E-08 4.E-08 2.E-08 O.EOO ff ws lz rr s. *,-....... . . . .. . . . . . . . . . . .. . . . I . . . - . . I . . . . .. . . ...

Month/Year Figure 41. CTMT Radiation Monitors RE4597AAABA (Combined Iodine Channels)

- ala's Root Cause Analysis Report nlurutu

• 10

CTMT Radiation Monitors RE4597AA & BA (Noble Gas Channels) z IIit HI11MII I H I1itI HIIIH Month/ear HUMP I-I Figure 42. CTMT Radiation Monitors RE4597AA & BA (Both Noble Gas Channels)

Figures

• 132 Root Cause Analysis Report

Sourc: FYIRVUFI Al H1iph Loak kRe. lkxic; Add My Chunnl I1k tip Thwuph lnumti tw03oCV1cL3 Ian'iep 4.

II 01/iell It 111111"

,% / tll%.'Z-', i lii, If TkmWer Nom PM-aidnl? 00r Acid Depm t on tim Ledwe Fo0 PW.C Crack Wbocd Soo Cast

/DIll /antth

Ilod a the Annugusil

? I ladc Acid anbid AcIab Rea MoInuaauAiEI ud PoubLMY I R1r litnM1Ie Uls MUDl PmvideS Thnml lmnIliti EWeCt D/I/l dalit Surface.

Figure 43. Potential Effects of Boric Acid Deposits on Vessel Top Head Figures a 133 Root Cause Analysis Report

Attachment 1. Potential Evidentiarv Recuest List

1. Metallurgical Samples From Nozzle 3 It is desirable to obtain the remaining section of Nozzle 3 from the elevation of the cut to the bimetallic input from the Root weld to the CRDM nozzle flange. Note: The specimen should not be cleaned priorto Cause Team. This specimen may be used for the following examinations and tests:

a. Examination of external surface of nozzle looling for evidence of flow up through the annulus.

Include high quality photographs

b. Metallurgical examinations including chemistry, microstructure, etc.

c. Hardness traverse through the wall thickness (similar to Oconee 3)

d. Tensile properties at three locations through thickness (similar to Oconee 3)

e. Others as identified

2. Non-destructive InsRections of Top Head Surface at Nozzle 3 Location It is desirable to perform several non-destructive inspections of the top head surface:

of the

a. Priority I: High quality photographs of the corroded areas adjacent to Nozzle 3. The purpose photographs is to show:

- General extent of corrosion

- Evidence of flow across clad and base metal surface

- Evidence of possible impingement of steam jet on surfaces in

b. PriojiM2y: Casting impression of cavity. The purpose of the impression would be to further aid identifying the boric acid corrosion mechanisms such as

- Volume loss

- Location of volume loss relative to leak

- Undercutting of low-alloy steel at cladding interface (potential)

3. Specimens From Remaining Material at Nozzle 3 Location of the It is desirable to remove specimens of the unsupported J-groove weld and adjacent areas unsupported clad. The priorities for these examinations are as follows:

This

a. Priority 1: The section of the J-groove containing the downhill (ft0) crack should be removed.

specimen would be used to:

- Determine the crack geometry (single crack, branches, etc.)

- Determine the crack width

- Assess flow induced erosion on the crack faces

- Assess the potential for the crack to have sared at the J-grooye weld surface

- Assess the potentialfor weld defects

- Assess the clad thickness and integrity Attachments *134 Root Cause Analysis Report

b. Priority 2: The exposed surface at the location of the uphill crack located by UT examination (ml 80°) should be evaluated.

The first step should be to perform a PT examination of the surface to determine if the crack remains at the machined surface of the weld metal and weld buttering, and if there is any wastage of the low-alloy steel that may have occurred as a result of the leakage.

If there is evidence of the crack, or of wastage that extends deeper than the machined surface, a casting impression should be made of the surface to record the crack and wastage.

c. Priority 3: A section from the J-groove weld and small amounts of adjacent low-alloy steel base metal and cladding at the triple point between the weld, low-alloy steel and unsupported cladding.

This specimen will be used to assess the surface of the corroded low-alloy steel and the potential for galvanic corrosion between the Alloy 182 weld/clad and low-alloy steel material.

d. Priority 4: If the unsupported section of clad and J-groove weld are to be removed as part of the repair, it is desirable to remove this entire piece intact including a small amount of the low-alloy steel base material at the ends of the unsupported section of the J-groove weld. This larger specimen would be used for.

- Further assessment of flow and impingement on the clad surface

- Thickness and structural integrity of the complete unsupported clad

- Corrosion of the low-alloy steel material adjacent to the cladding

4. Examinations and Potential Specimens From Nozzle 2 The wastage uncovered when the lower part of the nozzle was removed needs to be further characterized since it maybe a lead indicator of the type of wastage discovered at Nozzle 3:

a A casting impression should be taken of the wastage below the remaining section of the nozzle.

b. Specimens of boric acid deposits from the cavity behind the remaining nozzle wall should be removed and should be removed and collected in a clean specimen container.

c. The cavity behind the remaining nozzle should be further probed to establish height above the bottom edge of the remaining nozzle, width, and depth. This information will supplement the already performed boroscope examination.

d. Afer access is provided to the top surface of the vessel head the location where the nozzle penetrates the vessel head should be photographed 360° around the nozzle in its current condition. The surface should then be cleaned of any remaining boric acid deposits and the area photographed again.

Finally, any crevice between the nozzle and penetration should be characterized by feeler gauge measurements to establish the width and depth of the cavity.

If the above examinations show that the areas of wastage on the top and bottom of the vessel head are not vertically aligned the Root Cause Evaluation Team-should be notified imimnediately to deterimine if further' examination is required.

ML~~IIIUIbD- O Root Cause Analysis Report asinmun 40ai

If the nozzle is removed as part of the repair, a casting impression should be made of the inside surface of the bore in the vessel head that contains the wastage.

Rini"34mveo

3. Weld Idoilt 2 PT of weld and base metal ,7 3iwd N~h Wan rfae Casting siO if there are clc or evidence of wastage Uphill

,J 270 deg 3-Groove weld, cladding and edge of cornoded low-Pdrtv I alloysteel J-Groove weld (includingcack) and small pan of adjacent cladding

. 1 Note: Angular positions am the same as for theI UT examinations and art as Specimens for Root Csuse Failure Anslvsis viewed firnm the top of the head Figure I Rev. 4 Attachment 1 Athes~-oo 449; Root Cause Analysis Report SAcAM61IUMDn v aW