ML13084A155: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
 
(Created page by program invented by StriderTol)
 
(7 intermediate revisions by the same user not shown)
Line 15: Line 15:
| page count = 64
| page count = 64
}}
}}
=Text=
{{#Wiki_filter:(b)(4)I UES-20120I~Rev.O (1/64) INon-Proprielary, San Onofre Nuclear Generating Station, Unit 2 & 3 REPLACEMENT STEAM GENERATORS Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station QUALITY AS SURANCE DEPARTMENT (b)(4),(b)(6)
Information in this record was deleted        MITSUBISHI HEAVY INDUSTRIES, LTD.
R in accordance wit h teeom Act, exemptions FOIA- '702 00 9141 bL%)
of Information Ii
(b)(4)
UES-20120254Rev.O (2/64) INon-Proprietar Revision History Date    Approved  Checked    Prepared Rev. Summary of Changes  issued      By          by          by Original Issue          See        See        See        See Cover Page Cover Page  Cover Page Cover Page
(b)(4)
UES-20120254 Rev.0 (3/64) jNon-Proprietar*
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station Root CauseAnalysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
UES-20120254 Rev.0 (4/64) [Non-Proprietary.
Disclosure Statement The following organization and programmatic Root Cause Analysis has been prepared in accordance with the Mitsubishi Heavy Industries (MHI) corrective action program, which uses an after-the-fact hindsight-based analysis. The information identified in this evaluation was discovered and analyzed using all information and results available at the time it was written. These results and much of the information considered in this evaluation were not available to the organizations, management, or individuals during the period that relevant actions were taken and decisions were made.
This evaluation does not attempt to make a determination whether any of the actions or decisions taken by management, internal organizations, or individual personnel at the time of the event was reasonable or prudent based on the information that was known or available at the time they took such actions or made such decisions. Any individual statements or conclusions included in the evaluation as to whether incorrect actions may have been taken or improvements are warranted are based upon all of the information considered, including information and results learned after-the-fact and evaluation in hindsight after the results of actions or decisions are known, and do not reflect any conclusion or determination as to the prudence or reasonableness of actions or decisions at the time they were made.
Root Cause Analysis Reportfor tube wear identifiedin the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (5/64) INon-ProprietaryV Table of contents 1.0 Executive Summary ....................................................................................          6 2.0 Background of the Incident .......................................................................              7 2.1P roject Background ..............................................................................            7 2.2 Technical Specification requirements potentially involved in the P ro b lem ............................................................................................... 9 3.0 Statement of Problem ............................................................................              10 4.0 Extent of Condition Evaluation ................................................................                10 5.0 Analysis, Results, and Conclusions ...............................                                            11 5.1 Evaluation Team Formation ..............................................................                  11 5.2 Evaluation Methodology ...................................................................                11 5.3 Technical Investigation of the Incident .............................................                      12 5.4 Description of Main Wear Mechanisms .............................................                          14 5.5Discussion of Tube to Tube Wear .......................................................                    20 5.6 Discussion of Tube to AVB Wear .......................................................                    23 5.7 Discussion of Retainer Bar to Tube Wear ...........................................                        23 5.8 Root Causes ......................................................................................        24 5.9 Contributing Causes ..........................................................................            25 6.0 Corrective Action Matrix ..........................................................................            27 7.0 Extent of Cause Evaluation .....................................................................              32 8.0 Safety Culture Review ............................................................................            33 Attachment-1        Cause-effect analysis .......................................................              41 Attachment-2        Barrier analysis ................................................................          44 Attachment-3        Change analysis ..............................................................              47 Attachment-4        RCA charter .....................................................................          54 Attachment-5        Time line ..........................................................................        55 Attachment-6 Interview list ...................................                                                60 Attachment-7 Reference documents .......................................................                        62 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (6/64) Non-Proprieta 1.0    Executive Summary On January 31, 2012, after the replacement steam generators (RSGs) supplied by MHl had been operating for approximately 11 months, SONGS Unit 3 was brought into an unplanned shutdown due to primary to secondary leakage of approximately 82 gallons/day in one RSG. The direct cause of the leakage was determined to be tube to tube wear in the free span section of the U-bend region of the RSG, leading to a leak from one of the tubes in that region.
SONGS Unit 2 was in a refueling outage when the event occurred in Unit 3. During the normally scheduled outage inspections of the Unit 2 RSGs, tube wear was discovered in the vicinity of the retainer bars in the U-bend region of both RSGs. This wear was determined to have been caused by random vibration of the retainer bars.
It was determined that all four RSGs experienced higher than expected tube wear. This wear is comprised of: (i) tube to tube wear in the tube free-span sections between the Anti-Vibration-Bars (AVBs) located in the U-bend region observed almost exclusively in Unit 3; (ii) tube to AVB wear, observed at discrete tube to AVB intersections, with no wear indications in the tube free-span sections (the tube to AVB wear indications are short in length, and are associated with small tube motions); (iii) tube to Tube Support Plate (TSP) wear; and (iv) retainer bar to tube wear. One RSG experienced minor tube wear from a foreign object, which has since been removed.
MHI, working in conjunction with SCE personnel and other industry experts, determined the mechanistic causes of the tube wear. MHI formed a team composed of personnel from MHI and its U.S. subsidiary, plus outside consultants, to perform the Root Cause Analysis (RCA) of the tube wear identified in the SONGS Unit 2 and Unit 3 RSGs. The two wear mechanisms that produced the deepest wear are evaluated in this report. They include:
: 1. Tube to tube wear in the in-plane direction due to fluid-elastic instability (FEI)
: 2. Retainer bar to tube wear due to turbulence induced vibration (also referred to as random vibration) and the low natural frequency of the retainer bar Additionally, because many tubes exhibit it, this report also addresses a third wear mechanism:
: 3. Tube-to-AVB wear caused by turbulence induced vibration (also referred to as random vibration).
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (7/64) Non-Proprietaryj The RCA team used Cause-effect analysis, Barrier analysis and Change analysis to arrive at two Root Causes and three Contributing Causes. The Root Causes are:
: 1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
: 2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.
The corrective actions to preclude repetition include:
: 1. Revise Procedure SBBB60-NO1 "Procedurefor Controlling of the Design Activities" to require that the need for effective tube to AVB contact force under high localized thermal-hydraulic conditions (steam quality (void fraction),
flow velocity and hydro-dynamic pressure) be addressed in all MHI SG designs.
l.a Further revise Procedure 5BBB60-NO1 "ProcedureforControlling of the Design Activities" to require that sufficient contact force is assured under high localized thermal-hydraulic conditions (steam quality (void fraction) flow velocity and hydro-dynamic pressure), e.g., compare to the design parameters of previous successful MHI steam generator designs.
: 2. Revise procedure 5BBB60-NOI "Procedurefor Controlling of the Design Activities" to require that retainer bars and other steam generator parts subject to flow induced vibration be evaluated to determine the different analyses and the level of analysis that need to be performed to support the steam generator design.
2.0      Backeround of the Incident 2.1    Proiect Background In September 2004, MHI was awarded a contract to replace Southern California Edison's (SCE) original steam generators (OSGs) at Units 2 and 3 of the San Onofre Nuclear Generating Station (SONGS). The MHI-supplied replacement SGs (RSGs) had a number of differences from the OSGs provided by Combustion Engineering. One of the main differences was the substitution of Inconel 690 for Inconel 600 as the tube material. Inconel 690 is more resistant to corrosion than Inconel 600. However, Inconel Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (8/64) Non-Proprieta*
690 has a thermal conductivity approximately 10% less than that of Inconel 600. The requirement that the SG's thermal performance be maintained, in conjunction with maintaining a specified tube plugging margin, necessitated increasing the tube bundle heat transfer surface area from 105,000 ft2 to 116,100 ft 2 (an 11% increase).The Certified Design Specification 5023-617-01, Rev. 3 stated that SCE intended to use the provisions of 10 C.F.R. §50.59 as the justification for the RSG design, which imposed physical and other constraints on the characteristics of the RSG design in order to assure compliance with that regulation. The RSGs were also required to fit within the same space occupied by the OSGs.
The Certified Design Specification issued by SCE also required that MHI incorporate many design changes to minimize degradation and maximize reliability. The following are the design requirements specified for the U-bend supports:
          "3.10.3.5 ... The Supplier shall develop and submit for Edison's approvalan Engineering and FabricationGap Control Methodology describing control of an effective "zero" tube-to-flat bargap, gap uniformity and parallelismof the tube bundle in the out-of-plane directionprior to tube fabrication. The gap statisticalsize (mean value +3sigma) shall not exceed 0.003"' and shall be validated by empiricaldata."
The Unit 2 RSGs were delivered to SONGS in February 2009 and installed during a refueling outage between September 2009 and April 2010. The Unit 3 RSGs were delivered to SONGS in October 2010 and installed during a refueling outage between October 2010 and February 2011.
On January 31, 2012, after the Unit 3 RSGs had been operating for approximately 11 months, the unit was brought into an unplanned shutdown due to maximum primary to secondary leakage of approximately 82 gallons/day in one RSG. The direct cause of the leakage was determined to be tube to tube wear in the free span section of the U-bend region of the RSG, leading to a leak from one of the tubes in that region.
Inspections of the Unit 2 RSGs(which was offline undergoing a refueling outage) revealed significant tube wear in the vicinity of the retainer bars in the U-bend region.
In addition to these two forms of tube wear, all four RSGs were found to have experienced higher than expected tube to Anti-Vibration-Bar (AVB) and tube to Tube Support Plate (TSP) wear. One RSG had experienced minor tube wear due to a foreign object.
Root CauseAnalysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (9/64) 1Non-ProprietaryI 2.2      Technical Specification requirements Potentially involved in the Problem Technical Specification (TS) 3.4.17 requires that SG tube integrity be maintained and that all SG tubes meeting the tube repair criteria be plugged in accordance with the Steam Generator Program.
TS 5.5.2.11 requires a Steam Generator Program to be established and implemented to ensure that SG tube integrity is maintained.
TS 5.5.2.11.b specifies three performance criteria that must be met for SG tube integrity:
: 1. "Structural integrity performance criterion: All in-service steam generator tubes shall retain structural integrity over the full range of normal operating conditions (including startup, operation in the power range, hot standby, and cool down and all anticipated transients included in the design specification) and Design Basis Accidents (DBAs). This includes retaining a safety factor of 3.0 against burst under normal steady state full power operation primary-to-secondary pressure differential and a safety factor of 1.4 against burst applied to the design basis accident primary-to-secondary pressure differentials. Apart from the above requirements, additional loading conditions associated with the design basis accidents, or combination of accidents in accordance with the design and licensing basis, shall also be evaluated to determine if the associated loads contribute significantly to burst or rupture. In the assessment of tube integrity, those loads that do significantly affect burst or rupture shall be determined and assessed in combination with the loads due to pressure with a safety factor of 1.2 on the combined primary loads and 1.0 on axial secondary loads."
: 2. "Accident induced leakage performance criterion: The primary to secondary accident induced leakage rate for any DBA, other than a SG tube rupture, shall not exceed the leakage rate assumed in the accident analysis in terms of total leakage rate for all SGs and leakage rate for an individual SG. Leakage is not to exceed 0.5 gpm per SG and 1 gpm through both SGs."
: 3. "The operational leakage performance criterion is specified in LCO 3.4.13, "RCS Operational Leakage." [This LCO is applicable in Modes 1-4 and states RCS operational leakage shall be limited to: (a) no pressure boundary leakage; (b) 1 gpm unidentified leakage; (c) 10 gpm identified leakage; and (d) 150 gallons per day (gpd) primary to secondary leakage through any one SG."]
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (10/64) INon-Proprietar 3.0    Statement of Problem This Root Cause Analysis (RCA) was performed based on the following problem statement, which was adopted as part of the Root Cause Analysis Team Charter:
(1) Requirement No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for the duration of the Warranty Period. (per 17.2.3 of General T&C with EMS)
(2) Deviation Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure of eight tubes during in-situ pressure testing. (Both due to Defects)
(3) Consequences (For MHI)
( 10CFR21 Reoort required (b)(4) 4.0    Extent of Condition Evaluation To determine the extent of condition, other MHI SGs with similar design and construction were analyzed to see if the same tube wear conditions identified at the SONGS RSGs were present.
The replacement steam generators for OPPD's Fort Calhoun Nuclear Generating Station are the only other steam generators designed by MHI operating in the United States. The OPPD RSGs replaced Combustion Engineering OSGs and are of a similar design and construction as the SONGS RSGs with certain differences, including:
* Identical tube diameter (3/4") and wall thickness (0.043")
    "    Identical tube pitch (1.0" equilateral triangle)
    "    Identical pitch-to-diameter ratio (P/D = 1.33)
    "    OPPD has greater average tube to AVB gap
* OPPD RSGs are smaller than SONGS RSGs
    "    Fewer AVBs than SONGS
    "    Fewer tubes than SONGS
    " Smaller U-bend radius than SONGS
    "    Lower maximum steam quality (void fraction) than SONGS Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (11/64) Non-Proprietary The Fort Calhoun RSGs have operated more than three fuel cycles with no evidence of U-bend tube degradation (no tube-to-AVB wear, no tube-to-tube wear, and no retainer bar-to-tube wear).Other steam generators designed by MHI (operating outside of the United States)are of a different design and have a variety of tube sizes, tube pitches and operating conditions. These steam generators have years of operation without significant tube wear. Therefore, it is concluded that the MHI SGs in operation today are not part of extent of condition. However, these other MHI SGs will be evaluated for susceptibility based on extent of cause.
5.0 Analysis, Results, and Conclusions 5.1    Evaluation Team Formation On March 23, 2012 MHI formed a team composed of personnel from MHI and its U.S.
subsidiary, plus outside consultants, to perform the Root Cause Analysis of the tube wear identified in the SONGS Unit 2 and Unit 3 RSGs. The team was given the task of investigating the organizational and programmatic Root Causes of the tube wear. SCE also performed separate technical and Root Cause evaluations.
The Root Cause Analysis commenced on March 26, 2012, and was conducted concurrently with the development of MHI's technical evaluation reports.
5.2    Evaluation Methodology The evaluation team used the results of the technical investigations (identified below) as the basis for its analysis of the organizational and programmatic Root Causes for the tube to tube wear, retainer bar to tube wear, and tube to AVB wear seen in the RSGs.
The extent of cause was evaluated based on organizational and programmatic causes.
The team closely consulted with the MHI engineering team performing the technical evaluations, and with SCE representatives, in order to understand fully the technical causes of the tube wear. Additionally, the evaluation team gathered evidence through interviews, examination of procedures and plans and previous audits and surveillances, review of design and technical review meeting documents, and analysis of technical work products.
To determine the organizational and programmatic Root and Contributing Causes of the three wear mechanisms evaluated in this report, the evaluation team used three Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (12/64) Non-Proprietar cause analysis tools: Cause-effect analysis, Barrier analysis, and Change analysis. The Root and Contributing Causes were determined primarily through the Cause-effect analysis. The results of the Barrier analysis and the Change analysis support the findings of the Cause-effect analysis. In addition to supporting the Cause-effect analysis, the Change analysis identified an additional Contributing Cause.
In performing these analyses, the evaluation team closely looked at and took into account the technical evaluations prepared by MHI and SCE to understand fully the mechanistic causes of the tube to tube wear, the retainer bar to tube wear, and the tube to AVB wear, in order to better assess the underlying organizational and programmatic Root and Contributing Causes. The team then reviewed and evaluated, with the benefit of what is now known in hindsight, the design process for the RSGs to identify what could have been done differently that would have prevented the tube wear from occurring. Based on its reviews, the evaluation team identified the programmatic Root Causes of the RSG tube wear.
5.3      Technical Investigation of the Incident MHI performed technical evaluations to identify the mechanistic causes of the tube wear, which identified fluid elastic instability as the mechanistic cause of the tube to tube wear, turbulence induced vibration (often referred to as "random vibration" because the excitation modes over time are unpredictable) as the mechanistic cause of the tube to AVB wear, and turbulence induced vibration of the retainer bar as the mechanistic cause of the retainer bar to tube wear. These evaluations are reflected in the MHI reports Tube Wear of Unit-3 RSG Technical Evaluation Report, LS-04GA564 Rev.9; RetainerBar Tube Wear Report, L5-04GA561 Rev.4; Validity of Use of the FIT-Ill Results During Design, L5-04GA591 Rev. 3;and Supplemental Technical Evaluation Report, LS-O4GA588 draft. SCE also performed Root Cause evaluations.SCE reports Root Cause Evaluation NN201843216 Steam GeneratorTube Wear San Onofre Nuclear GeneratingStation, Unit 2datedApril 2, 2012, and Root Cause Evaluation: Unit 3 GeneratorTube Leak and Tube-to-Tube Wear Condition Report: 201836127, Rev.Ocontain the SCE Root Cause evaluations.
The MHI and SCE mechanistic cause analysis reports used Fault Tree Analysis and Kepnor-Tregeo (respectively) as the primary analysis tools. Each of these analyses considered a broad range of potential causes. The following causes were evaluated in detail:
Root CauseAnalysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (13/64) INon-Proprietary Manufacturing/fabrication                      Shipping Primary side flow induced vibration            Divider plate weld failure and repair Additional rotations following divider        TSP distortion plate repair Tube bundle distortion during operation        T/H conditions/modeling (flowering)
Each of these causes is evaluated in the MHI and SCE technical evaluation reports.
These technical evaluations identified five different wear categories for the tubewear observed in the SONGS RSGs. Two of these wear categories are responsible for the most significant instances of tube degradation(in terms of the depth of wear and potential for failing to meet the technical specification requirements) and are being evaluated in this report to determine their organizational and programmatic causes.
The two significant wear categories that are evaluated in this RCA are:
: 1. Tube to Tube Wear due to in-plane FEI: Tube to tube wear was found in the U-bend region, located between AVBs, in the free span. Many of the tubes exhibiting tube to tube wear also exhibited wear at the AVBs and TSPs, in particular at the top tube support plate. For tubes with wear at the top tube support plate, it is considered that the entire tube, including its straight region, is vibrating. Tube to tube wear occurs when there is tube in-plane motion (vibration) with a displacement (amplitude) greater than the distance between 1
the tubes in the adjacent rows, resulting in tube-to-tube contact.
: 2. Retainer Bar to Tube Wear due to Flow Induced Vibration: Tube wear occurred on tubes at the periphery of the U-bend, adjacent to the retainer bars. These tubes have no wear indications at any other location along their length, which 1 Some of the tubes with tube-tube wear did not experience large amplitude vibration but were impacted by tubes that did experience large amplitude vibration. Also the two tubes in Unit 2 with tube-to-tube wear had different wear characteristics than the Unit 3 tube-to-tube wear.
Root Cause Analysis Reportfor tube wearidentified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
D (b)(4)
UES-20120254 Rev.O (14/64) Non-Proprietar indicates that they are stationary, and that the wear is caused by the movement (vibration) of the retainer bars.
Additionally, because many tubes have smaller-depth wear indications at the AVB intersections, this report also addresses another wear category:
: 3. Tube to AVB Wear (for tubes without free span wear) due to random vibration:
Tube wear occurred at discrete tube-to-AVB intersections, with no wear indications in the tube free-span sections. These wear indications are short in length and are associated with small tube motions.
The other two categories of wear identified were: (i) wear at the TSPs (small bend radius tubes and tubes at the tube bundle periphery), and (ii) wear due to a foreign object. These two categories are not considered in this report because the degree of wear due to them is relatively small.
The conclusions of the MHI and SCE technical evaluations have been accepted as the basis of this analysis. To the extent these evaluations are revised or amended to reflect additional information or new understandings, this evaluation may be affected.
5.4      Description of Main Wear Mechanisms Fluid Elastic Instability In a tube array, a momentary displacement of one tube from its equilibrium position will alter the flow field and change the forces to which the neighboring tubes are subjected, causing them to change their positions in a vibratory manner. When the energy extracted from the flow by the tubes exceeds the energy dissipated by damping it produces fluid elastic vibration.
Fluid Elastic Instability (FEI) is a term used to describe a range of tube vibrations that starts at a point on a curve of vibration amplitude versus flow velocity. As depicted in Figure 1, one axis (Y)of that curve is vibration amplitude and the other (X)is flow velocity. The graph shows that as flow velocity increases vibration amplitude increases at a small linear rate until it reaches a point where the slope of the curve increases abruptly. The point in the curve where the slope changes is termed "critical velocity".
The critical velocity is a function of several variables. These include tube natural frequency, which is dependent on the tube geometry and support conditions, damping, Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
UES-20120254 Rev.O (15/64) *Ion-Proprietar*
which is a function of the steam-to-water ratio, flow velocity, which is dependent of the tube spacing.
  .0 Fluid Elastic Instability.
Random Vibration Flow Velocity "Critical Velocity" Figure 1 As discussed below and in the technical reports referenced above (See Supplemental Technical Evaluation Report), MHI has determined that, due to ineffective support for the tubes in the in-plane direction resulting from the very small and uniform tube-to AVB gaps, some of the tubes exceeded the fluid elastic critical velocity resulting in in-plane FEI, which in turn produced the large amplitude tube-to-tube wear. This mechanism is influenced by the local thermal hydraulic conditions around the tube.
Regions of high void fraction have lower tube damping, which reduces the fluid elastic critical velocity threshold. High void fraction regions also have higher cross flow velocities. Therefore, tubes with low or no contact force in the region of highest void fraction are most susceptible to this mechanism.
Random Vibration Random vibration is the vibration mechanism caused by flow turbulence that changes proportionately to changes in the fluid flow forces(dynamic pressure) and is present at all flow velocities. Turbulent flow forces are random in nature, so this form of vibration is referred to as random vibration. As discussed below and in the technical reports referenced above, MHI has determined that the tube wear at the AVB intersections Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
Jl (b)(4)
UES-20120254 Rev.0 (16/64) INon-ProprietarEJ with no wear indications in the tube free span sections is due to turbulence induced vibration caused by insufficient contact force between the tube and the AVBs due to very small, uniform tube-to-AVB gaps. Since dynamic pressure and damping is proportional to the void fraction, tubes in the region of highest void fraction are most susceptible to this mechanism.
Tube to Tube Wear Tube-to-tube wear was caused by large displacements of tubes in the in-plane direction. Tubes are known to have moved in-plane because of the locations and magnitudes of their wear scars. The wear scars indicate that the tubes were generally vibrating in their first fundamental in-plane mode, which implies that none of the twelve (12) AVB supports were restraining the tube motion. Yet, it also indicates that the tube-to-AVB gaps are very small and uniform, because none of the tubes exhibited out-of-plane FEI, which is the tube's preferential fluid elastic vibration mode?. It can therefore be concluded that the tube-to-AVB contact forces were negligible and the tube-to-AVB gaps (on both sides of each tube at each of the 12 AVB intersections) were very small. Both of these conclusions are consistent with the original design intent discussed below.
In-plane FEI is a phenomenon that had not been experienced in nuclear U-tube steam generators prior to its being identified in the SONGS RSGs. The practice in the nuclear industry at the time the SONGS RSGs were designed was to provide measures to preclude out-of-plane FEI in the U-bend region, which was based on the understanding set forth above. Reflecting this industry practice, the Japan Society of Mechanical Engineers' "Guideline for Fluid-elastic Vibration Evaluation of U-bend Tubes in Steam Generators" states that in-plane FEI does not need to be considered if out-of-plane FEI is controlled. The design of the SONGS RSGs is consistent with the contemporary industry practice and guidance. The RSGs were designed to provide effective tube support (by means of AVBs) to avoid out-of-plane FEI. MHI sought to maximize the 2
In U-bend SGs, because the tubes are curved, for the same support conditions the critical velocity for out-of-plane FEI will be lower than that for in-plane FEI because the natural frequency of tubes in the in-plane direction is higher, due to the tubes greater stiffness in-plane, than the natural frequency of the tubes in the out-of-plane direction.
Root Cause Analysis Reportfor tube wearidentified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
UES-20120254 Rev.0 (17/64) INon-Proprietar1 adequacy of the supports against out-of-plane FEI by increasing the number of AVBs to a number, 12, that exceeds that in other U-tube SGs designed by MHI or by other major U-tube SG manufacturers.
Minimizing tube vibration wear in the U-bend region was given high priority in the SONGS RSG Design Specification, the RSG design program, and in the manufacturing processes. Early in the project, SCE and MHI formed an AVB Design Team with the goal of minimizing U-bend tube vibration and wear. The AVB Design Team conducted numerous technical and design review meetings. The agreed-upon tube bundle U-bend support design and fabrication were as follows:
    "  Six (6) V-shaped AVBs (three sets of two) were to be provided between each tube column (12 AVB intersections total around the U-bend).
    " Tube and AVB dimensional control, including increasing the AVB thickness was to achieve an effective "zero" tube-to-AVB gap under operating (hot) conditions with gap uniformity and parallelism being maintained throughout the tube bundle. Effective "zero" gap was desirable as an industry practice in order to maximize the effectiveness of the supports. The tube and AVB tolerances were to be tighter than that of any prior MHI SG.
    "  Excessive preload contact force was to be avoided in order to minimize ding/dent indications, and to maintain mechanical damping and thus minimize tube vibration.
MHI investigated field experience with U-bend tube degradation using INPO, NRC and NPE data bases, and concluded that the SONGS RSGs were designed to minimize the potential for tube wear by providing extra support points with shorter spans in the U-bend region along with effective zero tube-to-AVB gaps.
In the fabrication process, MHI manufacturing focused on achieving very small, uniform tube-to-AVB gaps during assembly.
The AVB Design Team included consultants with knowledge and experience in the design and construction large U-bend SGs. One consultant had experience with the design of a plant whose SGs were similar to the proposed RSGs (the "comparison" or "reference" plant). Together, the AVB Design Team concluded that the SONGS RSGs had more tube vibration margin than the comparison plant, which had experienced only a small number of tube wear occurrences. This conclusion was due to the following considerations:(i) SONGS RSG tubes are larger, have thicker walls, and are Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (18/64) INon-Proprietarv stiffer than those of the comparison plant; (ii) the SONGS distances between AVB tube supports are shorter than those at the comparison plant; (iii) SONGS has 12 AVB tube supports where the comparison plant only has 10; (iv) SONGS's tube-to-AVB gap requirement was more stringent than that of the comparison plant.
The Certified Design Specification S023-617-01, Rev. 3, issued by SCE required an effective zero gap and gap uniformity and parallelism of the tube bundle in the out-of-plane direction. Establishing the goal to reduce tube-AVB gaps to an effective zero gap was in accordance with well accepted industry practice and understanding that minimizing gaps was highly desirable in preventing tube vibration wear. MHI had sought to minimize tube-AVB gaps in its previous SG designs. However, MHI took additional steps to minimize the tube-AVB gaps for the SONGs RSGs and to provide for gap uniformity throughout the U-bend region of the tube bundle.
These steps included increasing the nominal thickness of the AVB compared to previous MHI SGs and reducing the manufacturing tolerance of AVB thickness and twist in order to achieve effective zero gaps and provide gap uniformity. Steps were taken as well to minimize tube ovality and to minimize variations from the design value.
Also, numerous additional steps were taken in fabricating the tube bundle to assure gap uniformity throughout the U-bend region. Additionally, in the fabrication of the Unit 2 RSGs MHI identified other enhancements that were implemented in the fabrication of the Unit 3 RSGs. These included, for example, taking steps to minimize AVB twist by applying a larger(from          ons toF(j)tjons) pressing force in the Unit 3 fabrication and thus providing for more uniform AVBs in the Unit 3 RSGs.
The adequacy of the design against out-of-plane FEI was confirmed through test data and analyses that conservatively assumed that one of the AVBs provided in the design was inactive (that is, ineffective against out-of-plane FEI).Analyses using this criterion showed that an adequate margin against out-of-plane FEI exists in the SONGS RSGs. An additional AVB had been added to the design to provide further margin against out-of-plane FEI.
The MHI technical evaluations performed after the January 2012 incident determined that, despite the robustness of the MHI design, in-plane FEI had occurred. This occurrence was due to a combination of a lack of effective contact forces between the tube and AVB in the in-plane direction and localized thermal-hydraulic (T/H) conditions (high steam quality (void fraction) and high fluid velocity).The evaluations found that the average contact force in the Unit 3 RSGs was smaller than the average contact force in the Unit 2 RSGs. Therefore, the contact forces of the Unit 3 RSGs were more Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (19/64) lNon-Proprietary likely to be ineffective in preventing in-plane motion of tubes so that the Unit 3 RSGs were more susceptible to in-plane tube vibration than those in Unit 2. The difference in the contact forces between the Unit 2 and Unit3 RSGs is caused by the reduction in dimensional variations during the manufacture of the Unit 3 RSGs, mainly due to improvement of the control over tube and AVB dimensions in the manufacture of the Unit 3 RSGs. The reduced contact forces resulted in far more tubes in the Unit 3 RSGs experiencing tube-to-tube wear than those in the Unit 2 RSGs. For those tubes, given these support conditions, the vibratory energy in high localized thermal-hydraulic (T/H) environment produced in-plane FEI that led to large amplitude displacement of the tubes in the in-plane direction, which caused wear from contact between adjacent tubes.
Tube Wear at AVBs Tube-to-AVB wear is a function of the amplitude of the random tube vibration and the tube-to-AVB gap. Where there is a gap between the AVB and the tube and the vibration amplitude is less than the gap, there will be minimal or no wear. If the AVB is in contact with the tube but there is insufficient contact force to lock the two together, there will be relative motion between the two and wear will occur. In the case where there is sufficient contact force to lock the two together, there will be minimal or no relative motion and only minimal wear will occur. In the SONGS RSGs, the zero gap design philosophy resulted in the AVBs being in contact with the tubes or very close to the tubes, but there was insufficient contact force to lock the two together, thus allowing tube wear at the AVBs.
The degree of wear is also affected by the amount of damping provided by the water film between the tubes and AVBs. In the SONGS RSGs, damping was reduced in areas of high steam quality (void fraction)because there is less two-phase damping and little or no water film in the gaps between the tubes, resulting in more pronounced wear.
Tube Wear at Retainer Bars The tubes exhibiting retainer bar wear have no indications of tube-to-tube or tube-to-AVB wear, which indicates that the wear is caused solely by retainer bar vibration. The SONGS RSGs have two types of retainer bars: [b)(4)
I(b)(4)I diameter and                      diameter. Tube wear was only found on tues adjacent to the smaller iame er re ainer bars. The retainer bars with the smaller diameter have a relatively long span as compared with those for other SGs fabricated by MHI, which means that the natural frequency of these retainer bars is lower, making them more Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (20/64) [Non-Proprietar likely to vibrate. This type of wear is caused by random flow-induced vibration of the retainer bars caused by the secondary fluid exiting the tube bundle.
5.5      Discussion of Tube to Tube Wear Tube Contact Force During the fabrication of the AVBs and the tubing and assembly of the tube bundle, MHI's manufacturing practices achieved dimensional control that resulted in smaller tube-to-AVB gaps and smaller tube-to-AVB contact forces. It was not recognized at the time that a certain amount of tube-to-AVB contact force was required to prevent in-plane FEI under high steam quality (void fraction) conditions, because the contact force serves to increase the in-plane natural frequency of the tube.
The technical investigations after the tube leak incident determined that the amount of contact force necessary to prevent in-plane FEI depends on the localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).As the steam quality (void fraction) increases, the amount of contact force necessary to prevent vibration increases. This increase in required contact force occurs because as the steam quality (void fraction) becomes higher, the damping provided by the liquid phase in the form of a liquid flim decreases.
The reduced in-plane contact force due to the SONGS "effective zero gap" design and the avoidance of "excessive preload" resulted in lowering the tubes' natural frequency in the in-plane direction. The combination of the localized high steam quality (void fraction) and reduced tube to AVB contact force resulted in exceeding the in-plane critical velocity, which created a condition that led to tube to tube contact.
The dominant role played by the low contact force is reflected by the differences in the tube-to-tube wear that was observed in the Unit 2 and the Unit 3 RSGs. Each of the Unit 3 RSGs had approximately 160 tubes that experienced tube-to-tube wear whereas only one of the Unit 2 RSGs experienced tube-to-tube wear in just two tubes, even though the Unit 2 RSGs have operated twice as long as the Unit3 RSGs. MHI did a comprehensive statistical evaluation of the contact forces between the tubes and the AVBs of the two units and concluded, based on the manufacturing data , that the contact force between the tubes and the AVBs in the Unit 2 RSGs is approximately double the contact force in the Unit 3 RSGs. Thus, the lower contact forces in Unit 3 are consistent with the conditions determined necessary to permit in-plane FEI to occur and with the fact that tube-to-tube wear occurred almost exclusively in Unit 3.
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (21/64) LNon-Proprietar1 Thermal-hydraulic Conditions Many analyses are performed during the steam generator design process. One of these is MHI's FIT-Ill tube bundle flow analysis, which calculates tube bundle thermal /
hydraulic parameters, including U-bend flow velocity and steam quality (void fraction).An after-the-fact comparison between the T/H parameters that FIT-Ill predicted and those predicted by ATHOS, another T/H code, determined that FIT-III's calculated values are lower than those obtained using ATHOS. Part of the difference was because the pressure loss coefficients for the tube bundle and the two-phase mixture density utilized in the two codes were different.
Also, during the computation of the flow velocity, MHI used an inappropriate definition of the gap between tubes, with the result that the flow velocities were underestimated.
These differences between MHI's use of the FIT-Ill model and the ATHOS model resulted in a higher margin to out-of-plane FEI than the margin that would have been determined using the appropriate the definition of the gap and an ATHOS-calculated steam quality (void fraction). The margin calculated using ATHOS, nonetheless, would still have resulted in adequate margin against out-of-plane FEI. Using the ATHOS outputs, with all AVBs assumed active, the stability ratio was less than 1.0 for out-of-plane FEI, even for those case studies assuming reduced damping that could occur under high void fraction conditions.3 Thus, the use of ATHOS as opposed to FIT-Ill would not have identified an inadequate design margin against FEI.
Moreover, because industry practice was focused on out-of-plane FEI, use of ATHOS would not have identified the potential for in-plane vibration. Both the academic literature and subsequently conducted tests show that the thermal-hydraulic environment under which In-plane FEI arises is different from those that result in out-of-plane FEI. (See Supplemental Technical Evaluation Report). If the steam quality (void fraction) predicted by FIT-Ill had been the same as the ATHOS calculated value, 3
The maximum stability ratio based on ATHOS outputs for all supports are active isEEZl, which    is less than 0.75, which is the conservative industry practice for judging acceptability of stability ratios (which in turn is less than the ASME Section III Appendix N-1330 recommended stability ratio criterion of 1.0). Assuming reduced damping, the maximum stability ratio calculated using ATHOS i Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (22/64) JNon-Proprieta and if the appropriate tube to tube gap value had been utilized to compute the flow velocity, MHI would have identified a decreased margin against out-of-plane FEI. In that case, MHI might have incorporated an additional AVB to increase the design margin against out-of-plane FEI, but would not have taken measures to protect against in-plane FEI, for it was assumed (as was the practice and guidance in the industry) that the controlling effect of a well-designed AVB system was adequate to preclude it.
Thus, not using ATHOS, which predicts higher void fractions than FIT-Ill at the time of design represented, at most, a missed opportunity to take further design steps, not directed at in-plane FEI, that might have resulted in a different design that might have avoided in-plane FEI. However, the AVB Design Team recognized that the design for the SONGS RSGs resulted in higher steam quality (void fraction) than previous designs and had considered making changes to the design to reduce the void fraction (e.g.,
using a larger downcomer, using larger flow slot design for the tube support plates, and even removing a TSP). But each of the considered changes had unacceptable consequences and the AVB Design Team agreed not to implement them. Among the difficulties associated with the potential changes was the possibility that making them could impede the ability to justify the RSG design under the provisions of 10 C.F.R.
§50.59. Thus, one cannot say that use of a different code than FIT-Ill would have prevented the occurrence of the in-plane FEI observed in the SONGs RSGs or that any feasible design changes arising from the use of a different code would have reduced the void fraction sufficiently to avoid tube-to-tube wear.
For the same reason, an analysis of the cumulative effects of the design changes including the departures from the OSG's design and MHI's previously successful designs would not have resulted in a design change that directly addressed in-plane FEl.
Summary Thus, the organizational and programmatic .Root Cause for the In-plane FEI as set forth in this RCA is the insufficient programmatic requirement to assure effective AVB contact force to control in-plane FEI under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydrodynamic pressure). The underlying reason for this insufficiency is that the MHI SONGS RSG design did not consider the phenomenon of in-plane FEI because contemporary knowledge and industry U-tubeSG operation experience did not indicate a need to consider in-plane FEI.
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
UES-20120254 Rev.O (23/64) INon-Proprietary 5.6      Discussion of Tube to AVB Wear Tube-to-AVB wear in the SONGS RSG occurs at the tube-to-AVB intersections and is produced by turbulence induced (random) vibration. This population only includes tubes with wear at the tube-to-AVB intersections with no wear indications in the tube free-span sections.
Tube wear at the AVB intersections (in the absence of tube-to-tube free span wear) occurs when the tube movement causes it to impact or slide along the supporting AVBs. The most common cause of this condition is out-of-plane FEI. In the SONGS RSG design, the large number of AVB supports and the superior gap control prevent out-of-plane FEI. However, because of the low contact forces between tubes and AVBs, the very small and uniform tube-to-AVB gaps, and the localized T/H conditions (high steam quality (void fraction) and high flow velocity), turbulent flow conditions are sufficient to produce tube wear at the AVB intersections. Again the effect of the different contact forces between Unit 3 and Unit 2 can be seen in the observed tube-to-AVB wear populations of the two units. Unit 2 had about two-thirds as many tube-to-AVB indications than Unit-3 and Unit 2 operated longer than Unit 3, indicated that the wear rate is greater at Unit 3. This is attributable to the lower contact forces.
(See Supplemental Technical Evaluation Report).
As was the case with tube-to-tube wear, it was not recognized at the time of the RSG design that a certain amount tube to AVB contact force is required to prevent random vibration under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).The combination of the reduced tube to AVB contact force and the localized T/H conditions (high steam quality (void fraction) and high flow velocity) resulted in tube to AVB wear.
5.7      Discussion of Retainer Bar to Tube Wear The design function of the retainer bar is to support the AVB assembly during manufacturing and prevent excessive AVB assembly movement during operational transients. The retainer bar must be strong enough to support the AVB assembly and fit within the physical constraints of the U-bend.
The tubesheet drilling pattern is one of the first design decisions made for a new steam generator and it is at that time that each tube location along the periphery of the tube bundle is established. The tube bundle design thus determines the retainer bar's length and thickness. At SONGS, in order to accommodate the increased number of Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (24/64) jNon-Proprietar tubes, the retainer bars are relatively long and thin as compared to the retainer bars in other SGs designed by MHI, resulting in their having low natural frequencies.
The engineer responsible for the retainer bar design did not recognize the need to analyze the retainer bar for flow induced vibration because no such analysis had been performed on previous MHI SG designs. The design control procedure for this design activity did not identify this issue, nor was it recognized during the design review process.
During operation, the secondary flow velocity and steam quality (void fraction) created turbulent flow conditions capable of causing high amplitude vibration if the retainer bar natural frequency was low enough, which turned out to be the case. The high amplitude vibration resulted in the retainer bar contacting some tubes and causing tube wear.
5.8      Root Causes As used in this evaluation, "Root Causes" are defined as the basic reasons (e.g.,
hardware, process, or human performance) for a problem, which if corrected, will prevent recurrence of that problem.
The programmatic Root Causes of the RSG tube wear are:
: 1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
Basis: The evaluation team concluded that the fundamental Root Cause for the in-plane FEI and the resulting tube-to-tube wear was the fact that in-plane FEI was not considered in the design of the SONGS RSGs. The fundamental reason for this lack of consideration was that industry practice and guidance, supported by the operating experience up to that time of U-bend type steam generators, indicated that the control out-of-plane FEI would prevent the occurrence of in-plane FEI.
Likewise, the evaluation team concluded that the tube to AVB wear was caused by insufficient contact force under high localized thermal-hydraulic conditions, which was not recognized at the time of the design of the SONGS RSGs, and that the fundamental reasons for the ineffectiveness of the contact force were the established industry practice of minimizing the tube support gaps and Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation
I(b)(4)
UES-20120254 Rev.O (25/64) INon-Proprietar'*
avoiding an excessive preload as well as other steps to control gap uniformity and parallelism.
: 2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.
Basis: The evaluation team concluded that the fundamental reason for the retainer bar FIV was the lack of clear direction in the MHI design procedures to require an evaluation to determine the different analyses and the level of analysis that were required for the RSG design in light of changes in the SONGS RSG design from previous MHI steam generator designs.
5.9    Contributing Causes As used in this evaluation, "Contributing Causes" are defined as causes that by themselves would not create the problem but are important enough to be recognized as needing corrective action. Contributing causes are sometimes referred to as causal factors. Causal factors are those actions, conditions, or events that directly or indirectly influence the outcome of a situation or problem. The evaluation team closely evaluated the mechanistic causes and the design process for the potential existence of Contributing Causes.
The programmatic Contributing Causes of the RSG tube wear are:
(b(4)
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (26/64) Non-Proprietary (b)(4)
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (27/64) Non-Proprietar 6.0    Corrective Action Matrix
      *,  *    ~...
                  -          = ;..  '  :6CreeAction
                                          ,r;***. ;,,*.    -    .*: J, "Due.. Da6i; Root Cause                  CAPR 1:Revise Procedure                  Completed
!.'Insufficient            5BBB60-NO1 "Procedure for programmatic              Controlling of the Design Activities" requirement to assure      to require that the need for effective effective AVB contact      tube to AVB contact force under high force to prevent            localized thermal-hydraulic in-plane fluid elastic      conditions(steam quality (void instability and random      fraction), flow velocity and vibration and              hydro-dynamic pressure) be subsequent wear            addressed in all MHI SG designs, under high localized CAPR 1.a:Further revise Procedure        11/15/2012 thermal-hydraulic 5BBB60-NO1 "Procedure for conditions (steam Controlling of the Design Activities" quality (void fraction),
to require that sufficient contact flow velocity and force is assured under high localized hydro-dynamic thermal-hydraulic conditions (steam pressure).
quality (void fraction) flow velocity and hydro-dynamic pressure), e.g.,
compare to the design parameters of previous successful MHI steam generator designs.
CA 1:Provide training for all Steam      Completed Generator Engineers (included new hires and continuing training) covering this event and the details concerning in-plane FEI and tube-AVB wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (28/64) Non-Proprieta Cause        -'            Corrective Action                <Due ?a~e Root Cause 2:The          CAPR 2:Revise procedure                10/31/2012 design control process    5BBB60-NO1 "Procedurefor did not provide            Controllingof the Design Activities" sufficient direction to    to require that retainer bars and assure that an            other steam generator parts subject evaluation of the need    to flow induced vibration be for an analysis of flow    evaluated to determine the different induced vibration of      analyses and the level of analysis that the retainer bar was      need to be performed to support the performed and              steam generator design.
verified.
CA 2:Revise Engineer Training          10/31/2012 program (included new hires and continuing training) to include the necessary assessment for required analyses of each Steam Generator part subject to flow induced vibration.
(b)(4)
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
I(b)(4)I UES-20120254 Rev.O (29/64) [Non-ProprietaN~
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.0 (30/b4) iNon-Proprietaryj
    ..:, Cause Cause                  ,*...      o r c .*  cln*-*
Action:      . ...        Due Date Extent of Cause          CA 6: Conduct a program design                  3/31/2013 review for other SG design procedures and primary pressure boundary components (Reactor vessel, Core internals, Pressurizer, Reactor coolant piping, CRDMs) using senior engineers to determine if other design features have assumptions that are not programmatically captured and evaluated.
CA 7: Reconfirm MHI steam                      11/30/2012 generator designs using the                    for SONGS SG design procedure developed for Root Cause              3/31/2013
: 2.                                              for OTHER SG designs CA 8: Reconfirm that the appropriate            Completed for analyses were performed and that                SONGS SG design correct values were used as inputs              10/31/2012 for each thermal hydraulic analysis,            for OTHER SG designs vibration analysis, and wear analysis (FIT-Ill, FIVATS, IVHET) in the design and fabrication processes of MHI steam generators.
CA 9: Reconfirm that the computer              Completed validation was performed adequately for each thermal hydraulic analysis, vibration analysis, and wear analysis (FIT-Ill, FIVATS, IVHET).
                        *If necessary, additional comparison to other validation methods shall be performed.
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre NuclearGeneratingStation
(b)(4)                          I UES-20120254 Rev.O (31/64) N*on-Proprietarj Cause.:  ,                CorrectvieA&tion    ,A      .          Date Effectiveness Review    In accordance with MHI's QA program, "Corrective action reports" will be issued for all CAPRs and CAs and the confirmation of effectiveness of completed corrective actions will be performed by the Nuclear Plant Quality Assurance Section.
Effectiveness reviews will be completed in six (6) months by verifying corrective actions for the addressed problems.
In addition, review the results of the initial Unit 2 & 3 mid-cycle outage and SG inspections to determine the effectiveness of corrective actions.
There is no evidence of :
                          , Additional tube to tube wear (in-plane FEI)
                          -Additional tube to retainer bar wear (turbulence induced vibration (random vibration))
                          - Additional tube to AVB wear (turbulence induced vibration (random vibration)).
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (32/64) INon-Proprietar7, 7.0  Extent of Cause Evaluation The Root Causes were evaluated for the extent to which they would be applicable and present elsewhere in the MHI steam generator design process.
The two Root Causes are:
: 1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
: 2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.
Root Cause 1 is associated with the design program and procedures not capturing necessary design elements affecting the primary pressure boundary. MHI has different nuclear engineering sections responsible for different aspects of the primary pressure boundary design, and each section has its own controlling design programs and procedures. Therefore, the extent of cause applies to the SG design program and areas of design outside the SG design program that could impact the primary pressure boundary. Sections outside the SG program with design responsibility related to the primary pressure boundary include:
: a. Reactor Vessel
: b. Core internals
: c. Pressurizer
: d. Reactor coolant piping*
: e. Control Rod Drive Mechanisms To address this extent of cause evaluation, each MHI engineering section will conduct a program and procedures review, based on what was learned from this event, to determine if there are other SG program elements or other primary components that rely on design assumptions that are not captured in the design program or procedures.
For Root Cause 2, an analysis that should have been performed was not. Therefore, this extent of cause applies to other SG design analyses that should have been performed but were not. Because there is no controlling document that identifies what analyses should be performed for each component, CAPR 2 must be developed and Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (33/64) INon-ProprietarM then a complete review of the different MHI SG project needs to be performed to confirm that all required analyses have been completed.
8.0  Safety Culture Review A safety culture review was performed using the NRC's Inspection Manual Section IMC0310 COMPONENTS WITHIN THE CROSS-CUTTING AREAS and applying the guidance in that section to the Root and Contributing Causes identified in this report.
The review examined all four safety culture areas, the thirteen cross-cutting and other area components, and the thirty-seven aspects comprised in those components. A summary table 1 that compares the identified Root and Contributing Causes with the requirements of each of the safety culture areas, components and aspects is provided below.
As the table I shows, both Root Causes and all Contributing Causes are associated with aspect 6 (H.2(c)) of the "resources" component in the Human Performance Area. One Root Cause and all Contributing Causes are associated with aspect 2 (H.1(b)), of the "decision-making" component in the Human Performance Area. One Root Cause and all Contributing Causes are associated with aspect 4 (H.2(a)), of the "resources" component in the Human Performance area. Finally, one Root Cause and two of the Contributing Causes are associated with aspect 12 (H.4(c)) of the "work practices" component in the Human Performance Area.
The component from the Human Performance Area applicable to the second Root Cause and the three Contributing Causes is aspect 6 (H.2(c)) of the "resources" component, which calls for complete, accurate and up-to-date design documentation, procedures, and work packages, and correct labeling of components. This aspect of the resources component was not satisfied because, while the decision making and the designs were properly documented, they were inaccurate in that they did not require analyses to evaluate the potential FIV of the retainer bars (Root Cause 2);
(b)(4)
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
UES-20120254 Rev.O (34/64)INon-Proprietary I(b)(4)
  'This component from the Human Performance Area is also associated with Root Cause" 1, in that the design procedures did not contain any requirement to assure effective AVB contact force. However, there is no safety culture related deficiency with respect to Root Cause 1 in that MHI was following accepted industry practices to design AVB and in fact sought to make its design more conservative than previous AVB designs.
An aspect of a component from the Human Performance Area applicable to one of the Root Causes and the three Contributing Causes is aspect 2 (H.1(b)) of the "decision-making" component, which requires that conservative assumptions be used in the design. The design did not require analyses to evaluate the potential FIV of the retainer bars (Root Cause 21:
(b)(4)
The discrepancies between the design and aspect 2 (H.1(b)) of the "decision-making" component also apply to aspect 4 (H.2(a)) of "resources" component.
Finally, an aspect of a component from the Human Performance Area applicable to one Root Cause and two of the Contributing Causes is aspect 12 (H.4(c)) of component 4 ("work practices"), which requires that appropriate supervision and management oversight be applied to the design. While design activities were reviewed and confirmed by the design section the design supervision and review process failed to recognize that FIV analysis of the retainer bars was needed (Root Cause 2);
&#xfd;b)(4)
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
UES-20120254 Rev.O (35/64)INon-Proprietary (b)(4)
MHI has identified a number of corrective actions, which are being taken or will be completed in the near future, to address the safety culture discrepancies identified in this review. These corrective actions are described in Section 6.0 above.. The predominant safety culture aspect was determined to be H.2.(c) Work Documents because the decision making and work practices were not influenced by programmatic requirements. The H.2.(c) safety culture aspect has the associated corrective action to establish the programmatic requirements for both Root Causes and the Contributing Causes, Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
UES-20120254 Rev.O (36/64) INon-ProprietarN Table 1 Safety Culture Review - Cross Cutting Components and Aspect X: Not sufficient SafetyCulture Area,                      Root Cause 1                                  Root Cause 2                      Contributing Cause 1                            Contributing Cause 2              Contributing Cause 3 Component, Aspect Insufficient programmatic requirement to assure effective AVBcontact force to prevent in-plane fluid elastic instability  The sfiin design control ieto    process oasr  didhta not provide sufficient direction to assure that an and random vibration and subsequent          evaluation of the need for an analysis of wear under high locailied flow induced vibration of the retainer bar thermal-hydraulic conditions (steam quality (void fraction), flowvelocity and hydro-dynamic pressure).
Area 1. Human Performance (H)
Component 1. Decision-Making Aspect 1.
Risksignificant decisions Sufficient - fMHI's AVBand tube bundle designs were reviewed and confirmed followed a decision-waking process to evaluate and review the technical aspects of the design.
H.i(a)
Aspect 2.                                                                                                                                                                                                                                a Conservative                                                              i" assumCtionservativ-      Sfficient - The AVBdesign decision was          i assumptions 11.1(b)                                                          Not sufficient -The engineer based on a FIT-Illanalysis which had a          s responsible for the retainer bar design built in safety margin and assumed one            I Xdid not recognize the need to analyze inactive support as an additional measure othe retainer bar for potential flow of conservatism additionally MHI's design        Iinduced vibration had more AVBsthan previous designs.
Aspect 3.
Timely communication      Sufficient - The decisions of the AVBand SGteam were documented and distributed to the team members in a timely manner.
H.I(c)
Component 2. Resources Aspect4.4."
Managing maintenance                                                                                                I Sufficient - The FIT-luanalysis had a built H.2(a)                    in safety margin and assumed that one              Not sufficient - The engineer did not inactive support as an additional measure  Xt recognize the need to analyze the I retainer bars for potential FIV.
of conservatismeUn Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Oncifre Nuclear GeneratingStation
UES-20120254 Rev.O (37164) fNon-Proprietar I
Safety Culture Area, Component, Aspect Root Cause 1                                    Root Cause 2                          Contributing Cause 1                            Contributing Cause 2                        Contributing Cause 3 AspectS.
Training and qualification personnel XI                                                                                                                                                                                                eswr ndqae Not sufficient - While the design section included euperts in SGdesign and manufacture, however procedure for training program was not sufficient because the training materials and procedures were inadequate.
H.2(b}
Aspect 6.
Work documents H.2(c)
Not deficient The decision making and design were documented, but the design                Not sufficient - The decision making and procedures did not include a requirement              design were documented, but the to prevent ir-plane FEfand random                    design procedures did not include a vibration related wear under high                    requirement to evaluate the retainer localized thermal-hVdraulic conditions.              bars for potential FIV. The predominant There was no programmatic requirement                safety culture aspect was determined to prevent in-plane FEIand random                  X to be H.2,(c) Work Documents because vibration, but MMIsought to make the                there was no programmatic AVBdesign more conservative than                      requirement to influence the engineer.
previous designs so no safety culture                The H.2.(c) safety culture aspect has the defioency is found. A corrective action is          associated corrective action to establish nevertheless provided to address this new            the programmatic requirement to understanding based on the tube wear                evaluate for the need for an FIV analysis observed at SONGS.
___________          I.
Aspect 7.
Facilities and.Equipment  Sufficient - The SGdesign section was provided with adequate facilities and other resources to conduct design review meetings and decision-making.
H.2(d)
Component 3. Work Control Aspect 8.
Work planning H.3(a)
Not applicable  - Aspects 8 and 9 are not applicable because they address work in the plant and coordination of removal of safety systems during plant maintenance.
Aspect 9.
Work coordinationH.3(b)
Component 4. Work Practices Aspect 10.
Error prevention techniques H.4(a)        Sufficient - Design activities were established in compliance with QAprograms to prevent error and personnel followed appropriate procedures.
Aspect 11.
Procedure compliance    Sufficient - MHI's corrective action program governed the design process. Additionally the design section decisions were made pursuant to decision making procedures.
H.4(b)
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
UES-20120254 Rev.0 (38/64) INon-Proprietani Safety Culture Area, Component, Aspect Root Cause 1                                    Root Cause 2                        Contributing Cause 1                        Contributing Cause 2                          Contributing Cause 3
                          -I.                                              I-                                            1; Aspect 12.
Supervision and management oversight          Sufficient -MHrs SGdesign activities were INot sufficient - The need for a FIV H.4(c)                        reviewed and confirmed by the design X2 analysis of retainer bar was not section at design review and technical I detected In the design review process.
review meetings.
Area 2. Problem Identification and Resolution (P)
Component S.Corrective Action Program Aspect 13.
Risk-based identification    Sufficient - MHI's corrective action program governed the design process.
threshold P.1(a)
Aspect 14.
trending program P.1(b)      Sufficient - MHI's corrective action program indcludes trend based assessments.
Aspect 15.                    Sufficient - MHI's corrective action program includes Root Cause and apparent cause assessments.
Cause evaluations P.1(c)
Aspect 16.                    Sufficient - No unresolved corrective actions were at issue.
Corrective actions P.1(d)
Aspect 17.
Alternative Processes          Sufficient - MHIl has alternative programs In addition to its regular reporting program.
P-1(e)
Component 6. Operating Experience Aspect 18.
Systematic process            Sufficient - MHIl investigated operating experience with U-bend tube degradation using INPO, NRC and NPE data bases, and communicated internally in a timely manner.
P.2(a)
Aspect 19.
Process changes P.2(b)
Sufficient - MHI conducted benchmarklng and concluded that the SONGS RSG was designed to minimize the potential'for tube wear by providing more support points with shorter spans in the U-bend region along with effective zero tube-to-AVG gaps during SG operation.
Component 7. Self- and Independent Assessments spect 20.            " --    ...........................
Nature of assessments        Sufficient -MHI periodically and appropriately conducted self-assessments.
___.3(a)____________II Root Cause Analvsis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
                                                                                                                                                                                                                    ,                            A*I UES-20120254 Rev.0 (39/64) [Non-Proprietarl Safety Culture Area,                  Root Cause 1                                  Root Cause 2                          Contributing Cause 1                        Contribut ing Cause 2 Component, Aspect Contributing Cause 3 Aspect 21.
Tracking and trending    Sufficient - MHI periodically and appropriately conducted self-assessment.
P.3(b)
Aspect 22.
Coordination and        Sufficient - MHI coordinated and communicated result from self-assessment to affect personnel and took appropriate corrective actions.
communication P.3(c)
Area 3. Safety Conscious Work Environment (S)
Component 8. Environment for Raising Concerns Aspect 23.
Free and open            Sufficient - The SG design team and AVB design team encouraged discussions of safety issues and openly exchanged information on design alternatives information exchange 5.I(a)
Aspect 24.
Alternate processes      Sufficient - MHI has alternative programs for raising safety concerns in confidence.
5.1(bi Component 9- Preventing, Detecting, and Mitigating Perceptions of Retaliation Aspec*t 25.1 Sufficient - There were no claims of harassment bySG team members.
Training S.2(a)
Aspect 26.              Sufficient - There were no claims of retaliation by SG team members.
inv..i....:__,_ i urk Aspect 27.
Chsillingeffect 5.2(c)  Sufficient - MIII appropriately considers chilling effect. Nokdisciplinary actions were taken.
Area 4. Other Safety Culture Components (0)
Component 10 .Accountability Aspect 28.
Alignment of safety and Sufficient - Accountability for SGdesign decisions was clearly understood within MHI.
rewards O.1(a)
Aspect 29.
Reinforcement O.1(b)    Sufficient - Management reinforced safety standards.
Aspect 30.              Sufficient - MIII demonstrated safety focus, review of neeting minutes indicates focus of SGdesign team was to come up with design with appropriate margin which demonstrated focus on safety.
Safety focus 0,1{c)
Root Cause Analysis Report far tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
a-I ICC 5111SASCa  O~.. fl IAflaCAl,,..O.~.~.,,,a*snA 01401641  a P o rieta SafetynCutureArea.                        Root Cause 1                                  Root Cause 2                        Contributing Cause I                            Contributing Cause 2                          Contributing Cause 3 Componsent. Aspect JCnrbtn                            as                          otiuigCue.3                        I Component 11. Continuous learning environment Aspect 31.
Training and knowledge      Sufficient - SG design team was trained on SGdesign continuously.
O.2(a)
Aspect 32.
Internal and external learning 0.2(b)            Sufficient - SGdesign team was trained on SGdesign continuously and transfer of knowledge was provided by internal experts, however the issue of effective AVBto contact force was not known within the industry so training could not be effective.
Component12. Organizational change management Aspect 33.
Organizational change      Sufficent - MHI management used systematic process and evaluated of impacts of decisions when organization was changed.
management 0.3 Componentl3. Safety policies Aspect 34.
Sufficient - Mill has appropriate policies which required reinforce to raise safety concern.
Raising concernsO.4(a)
Aspect 35.
Safety policy training    Sufficient - MHI has appropriate policy training to raise individual safety concern.
0.4(b)
                          +
Aspect 36.
Decisions consistent      Sufficient - Decisions related to SG design were consistent with MHI policies.
with safetypriorityO.4(c)
Aspect 37.
Top management              Sufficient - Top management communicated need for safe SG design as issue of effective AVB to contact force was not known within the industry so management communication could not be effective.
commitment    0.4(d)
Root CauseAnalysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
Attachment-I UES-20120254 Rev.0 (41/64)lNon-Proprietaryi (Legend)
Cause-effect analysis for the tube to tube wear                                                                          Phenomenon
                                                                                                                                                                                            ~:]~                "'B"isunderlying cause of  A F        :Cause n          Contriuting Cause RootCause
                                                                                                                                                                                                ] "
Design element to Insufficient action to decrease                                  avoid excessove steam quality (void fraction)                                    between AVB and tube Till condition was judged acceptable byFIV  Ontherpotential design changes arely, (SR- I and no excessive wear) arid  considered to decrease steam dry out evaluation                          quality (void fraction) but they had unacceptable outcomes
                                                                                                                                                                                                  -Note 1: Each SG on unit 3 had about 160 tubes with TTW. Only 2 tubes in one unit 2 SG had TTiN. This difference was caused by manufacturing and fabrication improvements implemented for unit 3. These improvements resulted in lower tube to AVB contact force in Root Cause Ill Insuffticient progremmatic                                        unit 3 SGs when compared to unit 2 SGs.
requirement to assure effective                                  'Note 2. At the time of design a narrow gap AVB contact force to prevent                                      between the AVB and tube was believed to Root Causes are defined as the basic reasons (e.g. hardware,                                                                    in-olane fluid elastic                                            achieve effective support condition-process, or human performance) for a problem, which If                                                                          Instability and random corrected, will prevent recurrence of that problem.                                                                                                                                                *Note 3: To consider identifying specific vibration and subsequent wear Contributing Causes are defined as causes that by themselves                                                                                                                                      customer specification but also note that MHI under high localized would not create the problem but are Important enough to be                                                                                                                                        evaluated and found such thermal-ftydmulic conditions recognized as needing corrective action. Conbibuting Causes                                                                                                                                        speciftcation/requirement acceptable based are sometimes referred to as causal factors. Causal factors are                                                                  (steam quality (void fraction),
on experiences.
those actions, conditions, or events that directly or Indirectly                                                                flow velocity and influence the outcome of a situation or problem.                                                                                hydro-clynamic oressurel.
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
I Attachment-i UES-20120254 Rev.0 (42/64) Non-ProprtetaryJ Cause-effect analysis for the tube to AVB wear (Legend)
                                                                                                                                                                                                                                                                =    - Phenomenon Random          aln FA[-          ]      -  is underlying cause of "A High localized TH conditin                                                                                                                    Inadequate support (insufficient tube to AVB contact force)
EL 0
I :Cause
:Contributing Cause F
High Vefocty                                                                                                                                                                                                                              : Root Cause i                                    -                                                                                  -
Localized high steam quality (void tractiofn)
I~
Insuftfcient action to decrease steam quality (void fraction)
T/H condition wasjudged I+
Other potential design changes1 Deineeett (cotat ore) 1 7
eteen vi  xesv rla Vantube III 0'                    acceptable by FIV analysis (SR<I                  considered to decrease steam and no excessive wea) and dryout                  quality (void traction) butthey evaluation                                        had unacceptable outcomes I-.      .      .                                    TomiOi                  id niotrecognize          Satisfy design mechanical damping          need for contact            specification not to by the sliclng of tube      force to control            exceed 7% ding along AVB                    random vibration            indications with ECT
                                                                                                                                                                                                              *Nlte I                      *Note 2 (b)(4) t I1                                        I
                                                                                                          ;-IMHI relied on the          IN Root Cause 41l                                                                I MHISONGS RSG design did not Insufficient programmatic based on the past                                                          consider sufficient AVS contact force requirement to assure effective successful experiences                                                      to prevent the random vibration AVB contact force to prevent in triangular tube                                                          under high steam quality (void in-plane fluid elastic instability                                                      *NOte 1- Al the time of design a narrow configuration SGS                                                          fraction) condition because and random vibration and                                                                gap between the AVB and tube and low Root Causes are defined as the basic reasons (e.g, hardware,                                                                                                                            contemporary knowledge and subsequent wear under high                                                              contact force was believed to achieve process, or human performance) for a problem, which If                                                                                                                                  industry U-bend SG operation localized thermal-nydraultc                                                              effective support concition.
corrected, wIN prevent recurrence of that problem.                                                                                                                                      experience dil not indicate a need conditions (steam quality (void C~ ntftbutlneC u e are defined as causes that by themselves                                                                                                                            to consider such phenomenon would not create the problem but are impofant enough to be                                                                                    fraction), flow velocity and              'Note I                                        'Note 2. To consider identifying specific recognized as needing corrective action. Contfrbuting Causes                                                                                  hydro-dynamic pressure).                                                                customer specification but also note that are sometimes referred to as causal factors. Causal factors are                                                                                                                                                                        MHI evaluated and found such thoe actions, conditions, or events that directly or indirectly specficationflequirement acceptable Influence the outcome of a situation or problem.
based on excenences.
Root Ca,.    .711uy07  n-Palu, jr    Zuur wmuz identir)ed in the Unit 2 and Unit 3 Steam Generators of San Onafre NuclearGeneratingStation
Attachment-1 UES-20120254 Rev.0 (43/64)an-Proprietar Cause-effect analysis for the retainer bar to tube wear (Legend)
Retainer bar to tube wear
                                                                                                                                                                                        -      : Phenomenon il    : 'Er is underlying cause of A Random vibration F-1      Cause M1  ---  Root Cause IRoot Cause (2)
The design cont      process did not provide suffiient direction to assure that an evaluation of the need for an analysis of flow induced vibration Root Causes are defined as the basic    .asons(e.g., hardware.
process, or human performance) for a problem, which If corrected,              I_of the retainer bar was performed and verified- Jjjfl will prevent recurrence of that problem.
Root Cause Analysis Reportfor tube wear identified in the Unit2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-2 UES-20120254 Rev.0 (44/64) Non-Proprietar Barrier analysis A barrier analysis for the SONGS RSGs was performed with respect to the items listed below.
Design Fabrication Post Installation testing/monitoring Post Operational Inspection The barrier analysis was developed to assess the barriers at each of the major stages of the steam generator replacement program. The two primary barriers assessed at each stage included training/ qualification of personnel and procedures. As shown in the Barrier analysis table, procedures and training / qualification were lacking for the three wear mechanisms evaluated (tube to tube, retainer bar to tube, and tube to AVB).
There were no issues identified with the fabrication process so there were no failed barriers. The results of the barrier analysis support the cause-effect analysis.
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
Attachment-2 UES-20120254 Rev.0 (45/64) INon-Proprietar ConsequI Tube to Tube
_ I Barrier Design Training /
Qualificati In-plane FEI t
Outcome Not Effective Evaluation The design section procedure did not contain guidance on in-plane FEI. As a result, training programs did not cover the phenomenon of in-plane FEI. This will be addressed with CA 1 in the Corrective action matrix.
wea r              on T-1model                                                                                                                                  0~
FIV analysis Not Effective Procedures  In-plane FEI    Not        Analyses were not performed because there was no consideration of this phenomenon mentioned in the Effective    procedure. This will be addressed with CAPR 1 in the Corrective action matrix.
TH model        Not        FIT-Ill predicted a lower velocity due to use of inappropriate gap value. This will be addressed with CA 3 in FIV analysis  Effective    the Corrective action matrix.
Supervisio  In-plane FEI                Supervisors used same procedures and received the same training as design engineers.
n                                      Every 3 months, an Executive Oversight Meeting was held. Deputy Head of MHI and Department Managers Not participated. However, there were no questions related to in-plane FEI because it was not considered under Effective MHI procedure or industry practice. This will be addressed with CAPR 1 and CA 1 in the Corrective action matrix.
TH model                    FIT-Ill output indicated higher steam quality (void fraction) than previous SG designs. However, the senior FIV analysis                engineer did not consider the potential adverse effects of the higher steam quality (void fraction).
Potentially    This will be addressed with CA lin the Corrective action matrix.
Not Effective Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
Attachment-2 UES-20120254 Rev.O (46/64) Non-Proprietary Retainer  Design  Training      Required          Not      Necessary analyses for each component were selected based on engineering judgment and past success.
Analyses    I  Effective  Training was insufficient.
to tube                                                    This will be addressed with CAPR2in the Corrective action matrix.
bar wear          Procedures    Required                  There was no requirement to confirm the consideration of a FIV analysis for changes made to a component Not Analyses                  in the flow stream.
Effective This will be addressed with CA 2in the Corrective action matrix.
I            i Supervision                              Based on past successful experience, engineering did not recognize the need to perform additional analysis Not      for the retainer bars. The senior engineer did not identify the need for FIV analysis during the design Effective  verification review.
This will be addressed with CA 2in the Corrective action matrix.
Tube to    Design Training    Contact force      Not      SG design training does not discuss contact force as a control mechanism to address vibration related wear under high      Effective  under high steam quality (void fraction) condition.
AVB wear                      steam quality                This will be addressed with CA 1in the Corrective action matrix.
Procedur    Contact force      Not      SG design procedures do not mention AVB contact force as a control mechanism to address vibration es          under high      Effective  related wear under high steam quality (void fraction) condition.
steam quality                This will be addressed with CAPRlin the Corrective action matrix.
Consequence all  Fabrication                              The SGs were fabricated as intended. For unit 2 it was done using the normal fabrication process. For unit 3 Not      it required divider plate failure repair. There were no causes identified associated with fabrication deviation applicable  from the design.
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-3 UES-20120254 Rev.0 (47/64) Non-Proprietar Chanee analysis For the SONGS RSGs, a change analysis was performed in two stages. The first stage compared the SONGS SG design to previous MHI SG designs for the triangular tube configuration. MHI had previously performed three steam generator designs using a triangular tube configuration. The second stage compared the SONGS RSGs to the previous SONGS SG design (Combustion Engineering type design). Only the most significant changes are included in this analysis.
The change analysis results are set out below.
(1) Differences between SONGS RSGs and previous MHI SG triangular design.--
(b)(4)
The SONGS RSGs have:
(b)(4)      circulation ratio
* maximum flow velocity
* average flow velocity P/D ratio
* out-of-plane FEI stability ratio Largest U bundle radius
* Specified AVB twist (b)(4)
(bo)(4)
[(b)(4)_      'ange o; G-va C    ue (tube diameter, out-of-plane)
Highest steam quality (void fraction)
* Thinnest and longest retainer bar
          * (b)(4)    ]  nominal tube-to-AVB gap (0.002" cold      / 0.000" hot) variation in tube-to-AVB gap (3 sigma (b)(4)
(2) Differences between SONGS RSGs and the previous SONGS OSG design.            --
(b)(4)
* Increase in tube bundle heat transfer surface area (11%)
Increase in number of tubes (5%)
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-3 UES-20120254 Rev.0 (48/64) INon-Proprietaryj Removal of stay cylinder Change from lattice bars to trefoil broached tube support plates
* Change in tube support configuration in U region
* Change from CE to MHI moisture separators
* Power level / operating temperature / tube plugging margin (3) Identification of the changes from previous SG designs led to the recognition that the RSG design deserved close scrutiny. MHI considered the changes in the SONGS design from previous steam generator designs and compared the basic design parameters of the SONGs RSGs (e.g., heat transfer area, circulation ratio, steam pressure, etc.) with other steam generator designs. Further, as part of the development of the SONGS RSG design, MHI conducted a detailed comparison between its proposed AVB support for the tubes in the U-bend region and that of a comparison plant of similar design. A special AVB team was formed and included industry experts to conduct an extensive design review process in 2005 / 2006 to optimize the U-bend design and address the technical issues. The team concluded that the SONGS design was significantly more conservative than previous designs in addressing U-bend tube vibration and wear.
Also MHI and SCE recognized that the SONGS RSG steam quality (void fraction) was high and MHI performed feasibility studies of different methods to decrease it.
Several design adjustments were made to reduce the steam quality (void fraction) but the effects were small. Design measures to reduce the steam quality (void fraction) by a greater amount were considered, but these changes had unacceptable consequences and MHI and SCE agreed not to implement them. It was concluded that the final design was optimal based on the overall RSG design requirements and constraints. These included physical and other constraints on the RSG design in order to assure compliance with the provisions of 10 C.F.R. &sect;50.59.
Thus, MHI did compare the SONGs RSG design with previous steam generator designs, and in particular did a detailed evaluation of different options of the AVB design taking into account other large steam generator designs.
(b)(4)
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
Attachment-3 UES-20120254 Rev.0 (49/64) Non-Proprietary
:b)(4)
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation
Attachment-3 UES-20120254 Rev.0 (50/64) INon-Proprieta Comparison between SONGS RSG Design and Previous MHI Designs Operating          SCE RSG U2      SCE RSG      Comparison        Potential    Evaluation Conditions                          U3          to other          Ca use MHIl design
                                              -I, Pressure (ata)      58.9
*1 Steam Flow          3.44E+06 (kg/h)
FW                  228 Temperature (Mc)
S/G Level          1612 (mm)*2
            -- 1    - -          1                                        ~*11 Circulation        3.3                                                    U                                                      &#xfd;&#xfd;&the high steam quality (void fraction) with r.J*    ...........................
Ratiol-) *5                                                                44.      JUWVCILULJCU lII$JIII            WII11.I II% u UIlUl Id lIUI I WILt OLIIer IdLtUr ca..n IedU LU LULU.
vibration.
Maximum                          "*<--                                            The high flow velocity provides the large dynamic pressure to the tube, which in Flow Velocity combination with other factors can lead to tube vibration.
(mls) *S Average Flow                        <-                                          The high flow velocity provides the large dynamic pressure to the tube, which in velocity (m/s) combination with other factors can lead to tube vibration.
-*5-              1 P/D Ratio -)        1.33
* Root Cause Ar_., .      . T j___
ear identified in the Unit 2 and Unit 3 Steam I
Generatorsof San Onofre Nuclear GeneratingStation
Attachment-3 UES-20120254 Rev.O (51/64) INon-Proprieta Operating          SCE RSG U2      SCE RSG    Comparison      Potential        Evaluation Conditions                        U3          to other        Cause MHI design 1~                                          -I-Stability Ratio                a-
                                                                          -~ I            Astability ratio of out- of- plane FEI(
(highest) (-)
(Where) *5 U-bend Radius                      4-r Largest 1  I The large bending radius gives(
(mm)                                                                                                                    I AVB Thickness (mm) 4 AVB width (mrm)
AVB twist (mm) j G-Values (mm)                      4 a
i -
Nominal span between AVBs (mm)
I Root Cause Analysis Report for tube wear identified in the Unit2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
Attachment-3 UES-20120254 Rev.O (52/64) INon-Proprietarv Operating          SCE RSG U2              SCE RSG    Comparison  Potential  Evaluation Conditions                                  U3        to other    Cause MHI design Natural i .
C-Frequency (Hz)
(tubes of Concern) *4 Steam Quality      0.9                                Highest      V          The high steam quality (void fraction) gives the low tube damping, which in (Void Fraction)    (0.996)                                                      combination with other factors can lead to tube vibration.
(H*5
_______________j                                                                    I AVB design *3      Solid type                                                                                                                                    0~
4                    -I            4-'          p Retainer bar                        0'                Thinnest    Y          The thinnest and longest retainer bar gives a low frequency, so FIV of retainer dimension                                                                      bar may result.
Longest
*1 This parameter shows secondary pressure.
*2 The distance between the U-bend top to water level.
*3 This parameter shows RSG AVB.
*4 The lowest natural frequency of tube is provided
*5 Circulation ratio is obtained from SSPC code, and Max. flow vel., Avg. flow vel., stability ratio, and steam quality (void fraction) are obtained ATHOS code.
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear Generating Station
Attachment-3 UES-20120254 Rev.0 (53/64) INon-Proprietary 2
Comparison between SONGS RSG and Previous SONGS (CEI SG (OSG) Design*",
Design Element        OSG Specification                      RSG Specification              Potenti  Evaluation al Cause Number of            9350                                  9727                                                                                                a-Tubes Channel Head          Stay cylinder to support tubesheet,    Thick welded structural and Tubesheet        floating divider plate                divider plate Configuration Tube Support          Batwing assembly, diagonal and        Floating structure consisting Configuration        vertical strips with interlocking      of 6 V-shaped anti-vibration JU-Bend Section)      horizontal strips between tubes,      bars (AVBs) with 12 support lattice bars attached to structural    points, retaining bars, members (shroud) external to the      bridges, and retainer bars tube bundle Tube Support          Lattice bars (egg crates) positioned  7 trefoil broached tube Configuration        by tie rods and wedge-welded to the    support plates (TSPs)
(Straight Section)    shroud after alignment with            positioned by stay rods, tubesheet, shroud is active part of    radial load path at all TSPs radial load path, 2-inch linecontact on 2 sides, 1-inch line contact on 2 sides
*1 This analysis focused on mechanical differences because T/H conditions were expected to be similar.
*2 Five design elements listed above were obtained from MPR report 'Original Steam Generator and Replacement Steam Generator Design Feature/Change Evaluation'.
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-4 UES-20120254 Rev.0 (54/64) INon-Proprietar RCA charter
==Title:==
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station (b)(4)
Management Sponsor:
(b)(4)
Team:
Problem Statement:
(1) Requirement No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for the duration of the Warranty Period. (per 17.2.3 of General T&C with EMS)
(2) Deviation Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure of eight tubes during in-situ pressure testing. (Both due to Defects)
(3) Consequences (For MHI) 10CFR21 Report required (b)(4)
Timeline and Deliverables:
RCA Team Assigned : March 23, 2012 Problem Statement committed : March 23, 2012 Prepare begun: March 26, 2012 DRAFT Cause-effect analysis : April13, 2012 DRAFT RCA Summary : JulyS, 2012 Review RCA Summary : July7, 2012 DRAFT RCA Report : July20, 2012 Review Revised RCA Summary : August30, 2012 Review RCA report: September 6 and 11, 2012 RCA Due Date: October12, 2012 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-5 UES-20120254 Rev.O (55/64) INon-Proprietary1 Time line 046/90P-dlw.
(EIKA64O.NIJI
  /I-o L!,.
_2012 R- O0.51/3 (Roplaced wrth588OO-2010 R-18 . 11/24 Rev.7. 3/12 Rev.6. 1/15 R.0.1/. 2/12 Reo16 2/5 2009
,~  65512)2 R~  22                                                                      Rev. 9. 8/29 Rev.5. 7/3  IRev.4. 713                                                5ey.
7/14 Rev.5. 6/11  R/v1.4.6/12
                                                      'Rev.3. 4/4                Rev 8. 4/7 Rev.4. 3/21                                            ev.3. 9 IRev.3. 3114 Rev.2. 2/27 R.o /4. 10/23 R8. 53. 1/07 R061l1 ,4/26 R-11/ 4//2              Rev.2. 416 Rev. 7. 2/27 Rev.2. 1/19  Rev.6. 1/23 Revi. 8/22  Rev.1. 9/20
[Rev.O 4/26
_2006                      Rev.3. 2)17              Rev.O, 2/17                              Rev 2. 2/3 R-r10..1/2                              1Rev 2, 1/19 Rev.l, 12119 R4.8,12/1/                                Rev.1. 11/25                                                      1Rev.1. 11/14 Rev,?, 7/1
      ?/1                              IRev.O. 9/30                                                      ,REv.O.9/5 R..?.                                                                            Rev.5. 10/21 Rev.4. 9/29 RevS. /17                                                                        Rev 3. 6/24 Rev.2. 6&~  Rev.2. 4/29 Rl 5,.2/7                                                                        Rev 1    4*29 Rev.0. 2/28                                              Rev ,    319 2005                                                                          Rev.1. 1/7
:ev.0. 11/30 R, 4.1022 R.v 3 8/23 2004 Ihe'"ee I,,      090/go 01      Analyt"o. W.I.,/ Leve/ DOtaf0m      Des/gn Avib-Vibboenv  Revod 0/    CoobodablOty LK-o./g              /he Repot for    Re/oll of 9.1            AVO        Anly/sis    Support      Tube Shele    Tube 11.504GA5002      LS-04CA428    A-onbly    L5-O4GASI  9 6/0/158      Region        L4r.GA418
                      $upo..ftd by L"AG4A61    0 LS-D4GA419              1.5-15GASI I  L"-04GA401 Re.A Root Cause Analysis Reportfortube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-5 UES-20120254 Rev.0 (56/64) Non-Proprietairy Design Procedure (BKA60,-NO)
Time Line 2012 Rev.0, 5131 (Reptaced w"th5BBB60-NO1) 2011 Rev.18, 11124 2010                                                            Rev.4. 1/25 Rev.17, 2/12 Re6. 2/5 2009 Rev.9. 9/25 Rev.3. 8/4    Rev.8. 8/20 Rev.7. 6/4 Rev.6. 4/4 Rev.7, 3/19 Rev.5. 2/29  Rev.2. 2/28 2008 Rev.15, 12/21 Rev.14. 10/23-Rev.13. 10119 Rev.12. 4/26 Re.11. 4/12                                                                                  Rev.6. 4/2 Rev.5. 2/13 2007 Rev.4. 12/12 Rev.3. 11/17 Rev.4. 5/12 Rev.3. 4/19 Rev.2. 3/24 Re.10, 1/26      2006 Rev.2. 1/19                                                                          Rev.1. 1/20 Re .9 10/17          Rev.1. 11/11                                Rev.4. 1017                            Rev.0. 11/18 Rev.2. 10/5 Re.08, 919            Rev.0, 9/30                                  Rev.3. 9/7 jev.1. 9/9 IRev.0.6/9                Rev.l, 8/5    Rev.1. 8/12 Rev.7. 7/13                            Rev.3. 7/29 Rev.6. 5/17                            Rev.2. 5/17
                                                      ~Rev.0. 6/27 Rev.2. 6/14                Rev.0. 6/27 Rev. 1 3/9                  Rev.1, 3/23 Rev.5, 2/7                                                                      Rev.0. 2/24 2005                IRev.0. 1/21                Rev.0. 1/19 Rv.4, 10122 Rev.3, 8/23 2004 Material        Material      Material    Design      Basic Sizing Design of    Qualification Selection        Selection      Selection    Repor of    Calculation  Tube Support  Report of Report for      Report tot    Report lot  Tube Support Report      Plate        AVB End Cap Anti-Vibration  Heat Transfer  Tube Suppot  Plate and    L5-04GA421  L5-04GA425    L5.04GA429 Bar              Tubing        Plate        Stay ROd LS-04GA224      L5-04GA208    L5-04GA223  L5-04GA411 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-5 UES-20120254 Rev.0 (57/64) [Non-Proprietary Design Procedure (A      01)
Time bOe 2012 Re-0. /S31 (Reptaced
.ath 5BOB60-N01) 2011 Rev.18. 11124 2010 Revo. 2/12 Re. 16.2/5 2009 Rev.5. 11112 Rev.4. 10/28 Rev.3. 8/29 Rev.2. 3/31 2008 Rev.l. 1/31 ReO.15. 12/21 Rev. 4.10/23 Rev.13. 10/19 Rev.0. 8/3 Rev.12. 4/26 Rev. 1.4/12 2007 IRev.2. 10/18                                                  Rev.2. 10/18  ~Rev.2. 10/18
                                                    ~Rev. 1. 8/22 Rev.1. 6/13                                                  Rev.i. 6/13    ~Rev. 1. 6/13 Rev.10. 1/26    2006 Rev.9. 10117                        Rev.0. 11/11  Rev.0. 11/11  IRev.0. 11111  IRev.0. 11/11  RevO.011/11    Rev.0. 11/11 Rev.8, 9/9 Re-. 7113 Rev.6. 517 Rev5. 2/7 2005 R,.4. 10/22 Rev.3. 8/23
                &#xfd;  I Thermal and    Anti-Vibration Anti-Vibration  Anti-Vibration  Anti-Vibration Anti-Vibration Anti-Vibration Hydraulic      Bar Assembly  Bar Assembly    Bar Assembly    Bar Assembly  Bar Assembly  Bar Assembly Parametric    1/9            2/9            3/9            4/9            5/9            6/9 Calculations  L5-04FUI 11    LS-04FU1 12    LS-04FUl 13    L5-04FUl 14    LS-04FU 115    L5-04FU1 16 L5-04GA510 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
Attachment-5 UES-20120254 II                      (b)(4)
Rev.0 (58/64) Non-Proprietary Design Procedure (BKA60-N01)
Time Line 2012 Rev.0, 5/31 (Replaced with58BB60- -
N01) 2011 Rev.18, 11/24:
2010 Rev.17, 2t12 Rev.16. 2/5 2009 Rev.15 12./2T Red.14 10123-Rev.13. 10/19 -
Rov.12. 4126                                Retainer Bar Rev.1. 4/12              Rev.9. 3/9 IrDimesion
                                        ,Stress Engineer sign off 0cange of 2007  Rev.8. 12/8 Rev.7. 12/5 Rev.6. 11/9 Rev.5. 10/18 Rev.4. 9/26                                      Rev.3. 9/4 Rev.3. 8/24 Rev.2. 6/13 Rev.3. 4/6 Rev.1. 2/28      Rev.2. 2/17 Rev 10. 1/26      2006                    Rev.1, 1/19      ORev.11/19 Rev.2. 12/2 Rev.9. 10/17            Rev.0. 11/11    Rev.0. 11/11      Rev.0. 11/11 Rev.8, 9/9 Rev.l. 8/12 Rev7. 7/13 Rev.6, 5/17                                                                Rev.0. 5/27 Rev.5, 2/7 20O5 Rv.d. 10/22 Rev.3, 8r23                                                                            Rev.3.8/10  IRev.3. 8/10 2O0                                                                  Rev.2. 7/24  IRev.2. 7/24 Rev. 1. 6/8  IRev.l. 6/8 Rev.0.2/15  IRev.0. 2/15  IRevO. 2/15 Anti-Vibration    Anti-Vibration    Anti-Vibration Tube Suppod  General      General        General Bar Assembly      Bar Assembly      Bar Assembly  Plate        Assembly 1/3  Assembly 2/3  Assembly 3/3 7/9              8/9              9/9            Assembly 313 KAZ-04-0021  KAZ-04-0022    KAZ-04-0023 LS-04FU1 17      L5-04FU1 18      LS-04FU 119    L-OP4FUI1O8 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-5 UES-20120254 Rev.O (59/64)"N-on-Proprietar Design Procedure    -                                                                Rev 97          Rev.3. 9/27    10/12 (eKA60-Nl)    -                                                                Rev.8, 9/14    Rev,2,    8131 TimeLine                                                                      Rev.7, 8/30    Rev.l,    7/27 Rev.5. 6/18        IRev.4.6/20            Rev.6. 8/10    Rev,0. 7/4 Rev.4, 5/22        IRev3    5111        Rev.5. 7/17 Rev.2. 415          Rev.4. 7/13 IRev:1    3/16        Rev.3. 6/27                        3/23 IRev.O. 2/13        Rev.2. 5/21
_2012                                                          Revi1. 4/27 Rev.0 3/30 Rev.O.5W1 (Replacedwith 50000-)1l) 2011 Rev.18,11/24 2010 Rav.17, 2/12=
Rev.1, 2152 2009 2006 Rev.1,12/2-                          Rev.3.12/11 Rev,4.,10,23 Rev.13,10/19 Ray.12. 4/26                              RViolette. M.J.Pettigrew. and N.W.Mureithi Rev.11,4112 Fluidelastic Instability of an Array of Tubes Preferentiallyl Flexible in the Flow Direction Subjected to Two-Phasel 2007                    Cross Flow" ASME Journal of Pressure Vessel Technology Vo1.1 28l Feb.2006                                                        I11 10/2 9/11 Rev.9. 8/4 Rev.8. 6/16                                                                                                  6/2 Rev.7. 4/19 3/13 Rev.6. 2/13 Rev10 1/26        2006                                                    Ojiscussionabouttube to AVB gap control Rev.5, 12/21  Rev.2. 12/7                        '12/5 Rev9.10/17 Rev.4. 10/21                                                                                                  10/17 Rev.8,919              Rev.3. 9/26                    V.P.Janzen. EG.Hagberg, M.J.Pettigrew, Rev.2. 8/22                      nd C.F.Taylor                            l98/17 Rev. 7/13                                              Fluidelastic instability and Work- Rate                                      7/11 Rev. 1. 6/8      Measurements of Steam-Generator Rev.6,5/17            Rev.l. 5/17                    U- Tubes in Air-Water Cross-Flow"                                            5/23 IASME Journal of Pressure Vessel Rev.5. 27                                            Technology Vo1l.27                                                            3/28 Rev.. 2/25        Feb. 2005                                                                    12/14 2605 Re4. 10722            Rev.0, 10/15                                                                    JSME Guideline "Fluid-elastic Vibration Rev.3,823                                                          M.K IM. K. AEvaluation Au-Yang,        l          Mar. 2002 of U-bend Tube in SGs" "Flow-Induced Vibration              Mr 2002 of Power and Process Plant Components" Jul. 2001 Replacement  Evluabaonof        Retaner Bar          Tubewear of      vahiity of Uhe    ACWmyperiod        H-/SCEDiesgn Steam        Tue Vibraton        TubeWear            Una3 RSG -      of the FIT-til    of the RCA eaem  Revion eeing at Goeeators    L5-04GA504          Report              Technical        Results during                      fo. Ave Oesign Pro~ect                          LS-04GA561          Evaluaton        Ceslgn                              and Fabricaton Eiecuuon Ran                                          Report          L5-04C.591 L5-04GB001                                            L5-04*A*64 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation
(b)(4)
Attachment-6 UES-20120254 Rev.0 (60/64) INon-Proprietar Interview list (b)(4) / (b)(6)
Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-6 UES-20120254 Rev.0 (61/64) INon-Proprietar (b)(4) / (b)(6),
Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-7 UES-20120254 Rev.0 (62/64) INon-Proprietar Reference documents
: 1. 5BB60-NO6Rev.1 "ASME Code Job Procedure for Preparation of Design Reports", 5/30/2012
: 2. 5BBB60-NOlRev.0 "Procedure for Controlling of the Design Activities", 5/31/2011
: 3. KAS-20050202 Rev.1 "FIT-Ill Code Description Note (Code User's Manual)", 11/18/2005
: 4. L5-04GA504 Rev.5 "Evaluation of Tube Vibration", 6/18/2012
: 5. L5-04GA588 draft "Summary of Technical Evaluation"
: 6. L5-04GA561 Rev.4 "Retainer Bar Tube Wear Report", 6/20/2012
: 7. L5-04GA564 Rev.9 "Tube wear of Unit-3 RSG - Technical Evaluation Report", 9/27/2012
: 8. L5-04GA591Rev.3 "Validity of Use of the FIT-Ill Results during Design", 9/27/2012
: 9. L5-04GA428 Rev.5 "Design of Anti-Vibration Bar", 7/3/2008
: 10. L5-04GA419 Rev.4 "Analytical Report of AVB Assembly", 7/3/2008
: 11. L5-04GA519Rev.2 "Water Level Controllability Analysis", 1/19/2007
: 12. L5-04GA511 Rev.9 "Data for Licensing Support Analysis", 8/29/2008
: 13. L5-04GA401 Rev.7 "Design Report for the Tube Sheet Region", 3/12/2010
: 14. L5-04GA418 Rev.5 "Design Report of Tube", 7/14/2008
: 15. L5-04GA224 Rev.2 "Material Selection Report for Anti-Vibration Bar", 1/19 2006
: 16. L5-04GA208 Rev.3 "Material Selection Report for Heat Transfer Tubing",7/29/2005
: 17. L5-04GA223 Rev.1 "Material Selection Report for Tube Support Plate", 9/9/2005
: 18. L5-04GA411 Rev.7 "Design Report of Tube Support Plate and Stay Rod", 6/4/2008
: 19. L5-04GA421 Rev.4 "Basic Sizing Calculation Report", 1/25/2010
: 20. L5-04GA425 Rev.9 "Design of Tube Support Plate", 9/25/2008
: 21. L5-04GA429 Rev.4 "Qualification Report of AVB End Cap", 5/12/2006 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-7 UES-20120254 Rev.0 (63/64) INon-Proprietar
: 22. L5-04GA510 Rev.5 "Thermal and Hydraulic Parametric Calculations", 11/12/2008
: 23. L5-04FU111 Rev.2 "Anti-Vibration Bar Assembly 1/9",10/18/2006
: 24. L5-04FU112 Rev.1 "Anti-Vibration Bar Assembly 2/9", 8/22/2006
: 25. L5-04FU113 Rev.0 "Anti-Vibration Bar Assembly 3/9", 11/11/2005
: 26. L5-04FU114 Rev.0 "Anti-Vibration Bar Assembly 4/9", 11/11/2005
: 27. L5-04FU115 Rev.2 "Anti-Vibration Bar Assembly 5/9", 10/18/2006
: 28. L5-04FU116 Rev.2 "Anti-Vibration Bar Assembly 6/9", 10/18/2006
: 29. L5-04FU117 Rev.9 "Anti-Vibration Bar Assembly 7/9", 3/9/2007
: 30. L5-04FU118 Rev.3 "Anti-Vibration Bar Assembly 8/9", 4/6/2006
: 31. L5-04FU119 Rev.1 "Anti-Vibration Bar Assembly 9/9", 1/19/2006
: 32. L5-04FU108 Rev.3 "Tube Support Plate Assembly 3/3", 9/4/2006
: 33. KAZ-04-0021 Rev.3 "General Assembly 1/3",8/10/2004
: 34. KAZ-04-0022 Rev.3 "General Assembly 2/3", 8/10/2004
: 35. KAZ-04-0023 Rev.0 "General Assembly 3/3", 2/15/2004
: 36. L5-04GBOO1 Rev.9 "Replacement Steam Generators Project Execution Plan", 8/4/2006
: 37. SB-SO-FB-0003 Rev.7 "Tubing and AVB Installation Procedure", 12/18/2007
: 38. SB-SO-FB-0004 Rev.10 "AVB Structure Assembly Procedure", 7/30/2007
: 39. UGS-L5-050044 Rev.7 "Anti-Vibration Bar Inspection Procedure (after assembling)", 5/21/2007
: 40. UGS-L5-050045 Rev.10 "Inspection Procedure for Tube and Anti-Vibration Bar Insertion", 11/2/2007
: 41. UGNR-SON2-RSG-067 Rev.7 "Nonconformance Report (Unacceptable gaps between Tubes and AVBs for Unit-2A)",9/21/2007
: 42. UGNR-SON2-RSG-075 Rev.1 "Nonconformance Report (Unacceptable gaps between Tubes and AVBs for Unit-2B)", 12/3/2007 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
(b)(4)
Attachment-7 UES-20120254 Rev.0 (64/64) [Non-Proprietar&#xfd;
: 43. UGNR-SON3-RSG-024 Rev.1 "Nonconformance Report for some Gaps between Tubes and AVBs are larger than the criterion", 4/24/2008
: 44. UGNR-SON3-RSG-030 Rev.0 "Nonconformance Report for:some Gaps between Tubes and AVBs are larger than the criterion",3/20/2008 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation}}

Latest revision as of 07:08, 6 February 2020

Root Cause Analysis Report for Tube Wear Identified in the Unit 2 and 3 Steam Generators of San Onofre Nuclear Generating Station
ML13084A155
Person / Time
Site: San Onofre  Southern California Edison icon.png
Issue date: 10/12/2012
From:
NRC/OIS/IRSD/RFSB
To:
References
FOIA/PA-2013-00114 UES-20120254, Rev 0
Download: ML13084A155 (64)
v • d • e


Text

(b)(4)I UES-20120I~Rev.O (1/64) INon-Proprielary, San Onofre Nuclear Generating Station, Unit 2 & 3 REPLACEMENT STEAM GENERATORS Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station QUALITY AS SURANCE DEPARTMENT (b)(4),(b)(6)

Information in this record was deleted MITSUBISHI HEAVY INDUSTRIES, LTD.

R in accordance wit h teeom Act, exemptions FOIA- '702 00 9141 bL%)

of Information Ii

(b)(4)

UES-20120254Rev.O (2/64) INon-Proprietar Revision History Date Approved Checked Prepared Rev. Summary of Changes issued By by by Original Issue See See See See Cover Page Cover Page Cover Page Cover Page

(b)(4)

UES-20120254 Rev.0 (3/64) jNon-Proprietar*

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station Root CauseAnalysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

UES-20120254 Rev.0 (4/64) [Non-Proprietary.

Disclosure Statement The following organization and programmatic Root Cause Analysis has been prepared in accordance with the Mitsubishi Heavy Industries (MHI) corrective action program, which uses an after-the-fact hindsight-based analysis. The information identified in this evaluation was discovered and analyzed using all information and results available at the time it was written. These results and much of the information considered in this evaluation were not available to the organizations, management, or individuals during the period that relevant actions were taken and decisions were made.

This evaluation does not attempt to make a determination whether any of the actions or decisions taken by management, internal organizations, or individual personnel at the time of the event was reasonable or prudent based on the information that was known or available at the time they took such actions or made such decisions. Any individual statements or conclusions included in the evaluation as to whether incorrect actions may have been taken or improvements are warranted are based upon all of the information considered, including information and results learned after-the-fact and evaluation in hindsight after the results of actions or decisions are known, and do not reflect any conclusion or determination as to the prudence or reasonableness of actions or decisions at the time they were made.

Root Cause Analysis Reportfor tube wear identifiedin the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (5/64) INon-ProprietaryV Table of contents 1.0 Executive Summary .................................................................................... 6 2.0 Background of the Incident ....................................................................... 7 2.1P roject Background .............................................................................. 7 2.2 Technical Specification requirements potentially involved in the P ro b lem ............................................................................................... 9 3.0 Statement of Problem ............................................................................ 10 4.0 Extent of Condition Evaluation ................................................................ 10 5.0 Analysis, Results, and Conclusions ............................... 11 5.1 Evaluation Team Formation .............................................................. 11 5.2 Evaluation Methodology ................................................................... 11 5.3 Technical Investigation of the Incident ............................................. 12 5.4 Description of Main Wear Mechanisms ............................................. 14 5.5Discussion of Tube to Tube Wear ....................................................... 20 5.6 Discussion of Tube to AVB Wear ....................................................... 23 5.7 Discussion of Retainer Bar to Tube Wear ........................................... 23 5.8 Root Causes ...................................................................................... 24 5.9 Contributing Causes .......................................................................... 25 6.0 Corrective Action Matrix .......................................................................... 27 7.0 Extent of Cause Evaluation ..................................................................... 32 8.0 Safety Culture Review ............................................................................ 33 Attachment-1 Cause-effect analysis ....................................................... 41 Attachment-2 Barrier analysis ................................................................ 44 Attachment-3 Change analysis .............................................................. 47 Attachment-4 RCA charter ..................................................................... 54 Attachment-5 Time line .......................................................................... 55 Attachment-6 Interview list ................................... 60 Attachment-7 Reference documents ....................................................... 62 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (6/64) Non-Proprieta 1.0 Executive Summary On January 31, 2012, after the replacement steam generators (RSGs) supplied by MHl had been operating for approximately 11 months, SONGS Unit 3 was brought into an unplanned shutdown due to primary to secondary leakage of approximately 82 gallons/day in one RSG. The direct cause of the leakage was determined to be tube to tube wear in the free span section of the U-bend region of the RSG, leading to a leak from one of the tubes in that region.

SONGS Unit 2 was in a refueling outage when the event occurred in Unit 3. During the normally scheduled outage inspections of the Unit 2 RSGs, tube wear was discovered in the vicinity of the retainer bars in the U-bend region of both RSGs. This wear was determined to have been caused by random vibration of the retainer bars.

It was determined that all four RSGs experienced higher than expected tube wear. This wear is comprised of: (i) tube to tube wear in the tube free-span sections between the Anti-Vibration-Bars (AVBs) located in the U-bend region observed almost exclusively in Unit 3; (ii) tube to AVB wear, observed at discrete tube to AVB intersections, with no wear indications in the tube free-span sections (the tube to AVB wear indications are short in length, and are associated with small tube motions); (iii) tube to Tube Support Plate (TSP) wear; and (iv) retainer bar to tube wear. One RSG experienced minor tube wear from a foreign object, which has since been removed.

MHI, working in conjunction with SCE personnel and other industry experts, determined the mechanistic causes of the tube wear. MHI formed a team composed of personnel from MHI and its U.S. subsidiary, plus outside consultants, to perform the Root Cause Analysis (RCA) of the tube wear identified in the SONGS Unit 2 and Unit 3 RSGs. The two wear mechanisms that produced the deepest wear are evaluated in this report. They include:

1. Tube to tube wear in the in-plane direction due to fluid-elastic instability (FEI)
2. Retainer bar to tube wear due to turbulence induced vibration (also referred to as random vibration) and the low natural frequency of the retainer bar Additionally, because many tubes exhibit it, this report also addresses a third wear mechanism:
3. Tube-to-AVB wear caused by turbulence induced vibration (also referred to as random vibration).

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (7/64) Non-Proprietaryj The RCA team used Cause-effect analysis, Barrier analysis and Change analysis to arrive at two Root Causes and three Contributing Causes. The Root Causes are:

1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.

The corrective actions to preclude repetition include:

1. Revise Procedure SBBB60-NO1 "Procedurefor Controlling of the Design Activities" to require that the need for effective tube to AVB contact force under high localized thermal-hydraulic conditions (steam quality (void fraction),

flow velocity and hydro-dynamic pressure) be addressed in all MHI SG designs.

l.a Further revise Procedure 5BBB60-NO1 "ProcedureforControlling of the Design Activities" to require that sufficient contact force is assured under high localized thermal-hydraulic conditions (steam quality (void fraction) flow velocity and hydro-dynamic pressure), e.g., compare to the design parameters of previous successful MHI steam generator designs.

2. Revise procedure 5BBB60-NOI "Procedurefor Controlling of the Design Activities" to require that retainer bars and other steam generator parts subject to flow induced vibration be evaluated to determine the different analyses and the level of analysis that need to be performed to support the steam generator design.

2.0 Backeround of the Incident 2.1 Proiect Background In September 2004, MHI was awarded a contract to replace Southern California Edison's (SCE) original steam generators (OSGs) at Units 2 and 3 of the San Onofre Nuclear Generating Station (SONGS). The MHI-supplied replacement SGs (RSGs) had a number of differences from the OSGs provided by Combustion Engineering. One of the main differences was the substitution of Inconel 690 for Inconel 600 as the tube material. Inconel 690 is more resistant to corrosion than Inconel 600. However, Inconel Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (8/64) Non-Proprieta*

690 has a thermal conductivity approximately 10% less than that of Inconel 600. The requirement that the SG's thermal performance be maintained, in conjunction with maintaining a specified tube plugging margin, necessitated increasing the tube bundle heat transfer surface area from 105,000 ft2 to 116,100 ft 2 (an 11% increase).The Certified Design Specification 5023-617-01, Rev. 3 stated that SCE intended to use the provisions of 10 C.F.R. §50.59 as the justification for the RSG design, which imposed physical and other constraints on the characteristics of the RSG design in order to assure compliance with that regulation. The RSGs were also required to fit within the same space occupied by the OSGs.

The Certified Design Specification issued by SCE also required that MHI incorporate many design changes to minimize degradation and maximize reliability. The following are the design requirements specified for the U-bend supports:

"3.10.3.5 ... The Supplier shall develop and submit for Edison's approvalan Engineering and FabricationGap Control Methodology describing control of an effective "zero" tube-to-flat bargap, gap uniformity and parallelismof the tube bundle in the out-of-plane directionprior to tube fabrication. The gap statisticalsize (mean value +3sigma) shall not exceed 0.003"' and shall be validated by empiricaldata."

The Unit 2 RSGs were delivered to SONGS in February 2009 and installed during a refueling outage between September 2009 and April 2010. The Unit 3 RSGs were delivered to SONGS in October 2010 and installed during a refueling outage between October 2010 and February 2011.

On January 31, 2012, after the Unit 3 RSGs had been operating for approximately 11 months, the unit was brought into an unplanned shutdown due to maximum primary to secondary leakage of approximately 82 gallons/day in one RSG. The direct cause of the leakage was determined to be tube to tube wear in the free span section of the U-bend region of the RSG, leading to a leak from one of the tubes in that region.

Inspections of the Unit 2 RSGs(which was offline undergoing a refueling outage) revealed significant tube wear in the vicinity of the retainer bars in the U-bend region.

In addition to these two forms of tube wear, all four RSGs were found to have experienced higher than expected tube to Anti-Vibration-Bar (AVB) and tube to Tube Support Plate (TSP) wear. One RSG had experienced minor tube wear due to a foreign object.

Root CauseAnalysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (9/64) 1Non-ProprietaryI 2.2 Technical Specification requirements Potentially involved in the Problem Technical Specification (TS) 3.4.17 requires that SG tube integrity be maintained and that all SG tubes meeting the tube repair criteria be plugged in accordance with the Steam Generator Program.

TS 5.5.2.11 requires a Steam Generator Program to be established and implemented to ensure that SG tube integrity is maintained.

TS 5.5.2.11.b specifies three performance criteria that must be met for SG tube integrity:

1. "Structural integrity performance criterion: All in-service steam generator tubes shall retain structural integrity over the full range of normal operating conditions (including startup, operation in the power range, hot standby, and cool down and all anticipated transients included in the design specification) and Design Basis Accidents (DBAs). This includes retaining a safety factor of 3.0 against burst under normal steady state full power operation primary-to-secondary pressure differential and a safety factor of 1.4 against burst applied to the design basis accident primary-to-secondary pressure differentials. Apart from the above requirements, additional loading conditions associated with the design basis accidents, or combination of accidents in accordance with the design and licensing basis, shall also be evaluated to determine if the associated loads contribute significantly to burst or rupture. In the assessment of tube integrity, those loads that do significantly affect burst or rupture shall be determined and assessed in combination with the loads due to pressure with a safety factor of 1.2 on the combined primary loads and 1.0 on axial secondary loads."
2. "Accident induced leakage performance criterion: The primary to secondary accident induced leakage rate for any DBA, other than a SG tube rupture, shall not exceed the leakage rate assumed in the accident analysis in terms of total leakage rate for all SGs and leakage rate for an individual SG. Leakage is not to exceed 0.5 gpm per SG and 1 gpm through both SGs."
3. "The operational leakage performance criterion is specified in LCO 3.4.13, "RCS Operational Leakage." [This LCO is applicable in Modes 1-4 and states RCS operational leakage shall be limited to: (a) no pressure boundary leakage; (b) 1 gpm unidentified leakage; (c) 10 gpm identified leakage; and (d) 150 gallons per day (gpd) primary to secondary leakage through any one SG."]

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (10/64) INon-Proprietar 3.0 Statement of Problem This Root Cause Analysis (RCA) was performed based on the following problem statement, which was adopted as part of the Root Cause Analysis Team Charter:

(1) Requirement No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for the duration of the Warranty Period. (per 17.2.3 of General T&C with EMS)

(2) Deviation Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure of eight tubes during in-situ pressure testing. (Both due to Defects)

(3) Consequences (For MHI)

( 10CFR21 Reoort required (b)(4) 4.0 Extent of Condition Evaluation To determine the extent of condition, other MHI SGs with similar design and construction were analyzed to see if the same tube wear conditions identified at the SONGS RSGs were present.

The replacement steam generators for OPPD's Fort Calhoun Nuclear Generating Station are the only other steam generators designed by MHI operating in the United States. The OPPD RSGs replaced Combustion Engineering OSGs and are of a similar design and construction as the SONGS RSGs with certain differences, including:

  • Identical tube diameter (3/4") and wall thickness (0.043")

" Identical tube pitch (1.0" equilateral triangle)

" Identical pitch-to-diameter ratio (P/D = 1.33)

" OPPD has greater average tube to AVB gap

" Fewer AVBs than SONGS

" Fewer tubes than SONGS

" Smaller U-bend radius than SONGS

" Lower maximum steam quality (void fraction) than SONGS Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (11/64) Non-Proprietary The Fort Calhoun RSGs have operated more than three fuel cycles with no evidence of U-bend tube degradation (no tube-to-AVB wear, no tube-to-tube wear, and no retainer bar-to-tube wear).Other steam generators designed by MHI (operating outside of the United States)are of a different design and have a variety of tube sizes, tube pitches and operating conditions. These steam generators have years of operation without significant tube wear. Therefore, it is concluded that the MHI SGs in operation today are not part of extent of condition. However, these other MHI SGs will be evaluated for susceptibility based on extent of cause.

5.0 Analysis, Results, and Conclusions 5.1 Evaluation Team Formation On March 23, 2012 MHI formed a team composed of personnel from MHI and its U.S.

subsidiary, plus outside consultants, to perform the Root Cause Analysis of the tube wear identified in the SONGS Unit 2 and Unit 3 RSGs. The team was given the task of investigating the organizational and programmatic Root Causes of the tube wear. SCE also performed separate technical and Root Cause evaluations.

The Root Cause Analysis commenced on March 26, 2012, and was conducted concurrently with the development of MHI's technical evaluation reports.

5.2 Evaluation Methodology The evaluation team used the results of the technical investigations (identified below) as the basis for its analysis of the organizational and programmatic Root Causes for the tube to tube wear, retainer bar to tube wear, and tube to AVB wear seen in the RSGs.

The extent of cause was evaluated based on organizational and programmatic causes.

The team closely consulted with the MHI engineering team performing the technical evaluations, and with SCE representatives, in order to understand fully the technical causes of the tube wear. Additionally, the evaluation team gathered evidence through interviews, examination of procedures and plans and previous audits and surveillances, review of design and technical review meeting documents, and analysis of technical work products.

To determine the organizational and programmatic Root and Contributing Causes of the three wear mechanisms evaluated in this report, the evaluation team used three Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (12/64) Non-Proprietar cause analysis tools: Cause-effect analysis, Barrier analysis, and Change analysis. The Root and Contributing Causes were determined primarily through the Cause-effect analysis. The results of the Barrier analysis and the Change analysis support the findings of the Cause-effect analysis. In addition to supporting the Cause-effect analysis, the Change analysis identified an additional Contributing Cause.

In performing these analyses, the evaluation team closely looked at and took into account the technical evaluations prepared by MHI and SCE to understand fully the mechanistic causes of the tube to tube wear, the retainer bar to tube wear, and the tube to AVB wear, in order to better assess the underlying organizational and programmatic Root and Contributing Causes. The team then reviewed and evaluated, with the benefit of what is now known in hindsight, the design process for the RSGs to identify what could have been done differently that would have prevented the tube wear from occurring. Based on its reviews, the evaluation team identified the programmatic Root Causes of the RSG tube wear.

5.3 Technical Investigation of the Incident MHI performed technical evaluations to identify the mechanistic causes of the tube wear, which identified fluid elastic instability as the mechanistic cause of the tube to tube wear, turbulence induced vibration (often referred to as "random vibration" because the excitation modes over time are unpredictable) as the mechanistic cause of the tube to AVB wear, and turbulence induced vibration of the retainer bar as the mechanistic cause of the retainer bar to tube wear. These evaluations are reflected in the MHI reports Tube Wear of Unit-3 RSG Technical Evaluation Report, LS-04GA564 Rev.9; RetainerBar Tube Wear Report, L5-04GA561 Rev.4; Validity of Use of the FIT-Ill Results During Design, L5-04GA591 Rev. 3;and Supplemental Technical Evaluation Report, LS-O4GA588 draft. SCE also performed Root Cause evaluations.SCE reports Root Cause Evaluation NN201843216 Steam GeneratorTube Wear San Onofre Nuclear GeneratingStation, Unit 2datedApril 2, 2012, and Root Cause Evaluation: Unit 3 GeneratorTube Leak and Tube-to-Tube Wear Condition Report: 201836127, Rev.Ocontain the SCE Root Cause evaluations.

The MHI and SCE mechanistic cause analysis reports used Fault Tree Analysis and Kepnor-Tregeo (respectively) as the primary analysis tools. Each of these analyses considered a broad range of potential causes. The following causes were evaluated in detail:

Root CauseAnalysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (13/64) INon-Proprietary Manufacturing/fabrication Shipping Primary side flow induced vibration Divider plate weld failure and repair Additional rotations following divider TSP distortion plate repair Tube bundle distortion during operation T/H conditions/modeling (flowering)

Each of these causes is evaluated in the MHI and SCE technical evaluation reports.

These technical evaluations identified five different wear categories for the tubewear observed in the SONGS RSGs. Two of these wear categories are responsible for the most significant instances of tube degradation(in terms of the depth of wear and potential for failing to meet the technical specification requirements) and are being evaluated in this report to determine their organizational and programmatic causes.

The two significant wear categories that are evaluated in this RCA are:

1. Tube to Tube Wear due to in-plane FEI: Tube to tube wear was found in the U-bend region, located between AVBs, in the free span. Many of the tubes exhibiting tube to tube wear also exhibited wear at the AVBs and TSPs, in particular at the top tube support plate. For tubes with wear at the top tube support plate, it is considered that the entire tube, including its straight region, is vibrating. Tube to tube wear occurs when there is tube in-plane motion (vibration) with a displacement (amplitude) greater than the distance between 1

the tubes in the adjacent rows, resulting in tube-to-tube contact.

2. Retainer Bar to Tube Wear due to Flow Induced Vibration: Tube wear occurred on tubes at the periphery of the U-bend, adjacent to the retainer bars. These tubes have no wear indications at any other location along their length, which 1 Some of the tubes with tube-tube wear did not experience large amplitude vibration but were impacted by tubes that did experience large amplitude vibration. Also the two tubes in Unit 2 with tube-to-tube wear had different wear characteristics than the Unit 3 tube-to-tube wear.

Root Cause Analysis Reportfor tube wearidentified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

D (b)(4)

UES-20120254 Rev.O (14/64) Non-Proprietar indicates that they are stationary, and that the wear is caused by the movement (vibration) of the retainer bars.

Additionally, because many tubes have smaller-depth wear indications at the AVB intersections, this report also addresses another wear category:

3. Tube to AVB Wear (for tubes without free span wear) due to random vibration:

Tube wear occurred at discrete tube-to-AVB intersections, with no wear indications in the tube free-span sections. These wear indications are short in length and are associated with small tube motions.

The other two categories of wear identified were: (i) wear at the TSPs (small bend radius tubes and tubes at the tube bundle periphery), and (ii) wear due to a foreign object. These two categories are not considered in this report because the degree of wear due to them is relatively small.

The conclusions of the MHI and SCE technical evaluations have been accepted as the basis of this analysis. To the extent these evaluations are revised or amended to reflect additional information or new understandings, this evaluation may be affected.

5.4 Description of Main Wear Mechanisms Fluid Elastic Instability In a tube array, a momentary displacement of one tube from its equilibrium position will alter the flow field and change the forces to which the neighboring tubes are subjected, causing them to change their positions in a vibratory manner. When the energy extracted from the flow by the tubes exceeds the energy dissipated by damping it produces fluid elastic vibration.

Fluid Elastic Instability (FEI) is a term used to describe a range of tube vibrations that starts at a point on a curve of vibration amplitude versus flow velocity. As depicted in Figure 1, one axis (Y)of that curve is vibration amplitude and the other (X)is flow velocity. The graph shows that as flow velocity increases vibration amplitude increases at a small linear rate until it reaches a point where the slope of the curve increases abruptly. The point in the curve where the slope changes is termed "critical velocity".

The critical velocity is a function of several variables. These include tube natural frequency, which is dependent on the tube geometry and support conditions, damping, Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

UES-20120254 Rev.O (15/64) *Ion-Proprietar*

which is a function of the steam-to-water ratio, flow velocity, which is dependent of the tube spacing.

.0 Fluid Elastic Instability.

Random Vibration Flow Velocity "Critical Velocity" Figure 1 As discussed below and in the technical reports referenced above (See Supplemental Technical Evaluation Report), MHI has determined that, due to ineffective support for the tubes in the in-plane direction resulting from the very small and uniform tube-to AVB gaps, some of the tubes exceeded the fluid elastic critical velocity resulting in in-plane FEI, which in turn produced the large amplitude tube-to-tube wear. This mechanism is influenced by the local thermal hydraulic conditions around the tube.

Regions of high void fraction have lower tube damping, which reduces the fluid elastic critical velocity threshold. High void fraction regions also have higher cross flow velocities. Therefore, tubes with low or no contact force in the region of highest void fraction are most susceptible to this mechanism.

Random Vibration Random vibration is the vibration mechanism caused by flow turbulence that changes proportionately to changes in the fluid flow forces(dynamic pressure) and is present at all flow velocities. Turbulent flow forces are random in nature, so this form of vibration is referred to as random vibration. As discussed below and in the technical reports referenced above, MHI has determined that the tube wear at the AVB intersections Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

Jl (b)(4)

UES-20120254 Rev.0 (16/64) INon-ProprietarEJ with no wear indications in the tube free span sections is due to turbulence induced vibration caused by insufficient contact force between the tube and the AVBs due to very small, uniform tube-to-AVB gaps. Since dynamic pressure and damping is proportional to the void fraction, tubes in the region of highest void fraction are most susceptible to this mechanism.

Tube to Tube Wear Tube-to-tube wear was caused by large displacements of tubes in the in-plane direction. Tubes are known to have moved in-plane because of the locations and magnitudes of their wear scars. The wear scars indicate that the tubes were generally vibrating in their first fundamental in-plane mode, which implies that none of the twelve (12) AVB supports were restraining the tube motion. Yet, it also indicates that the tube-to-AVB gaps are very small and uniform, because none of the tubes exhibited out-of-plane FEI, which is the tube's preferential fluid elastic vibration mode?. It can therefore be concluded that the tube-to-AVB contact forces were negligible and the tube-to-AVB gaps (on both sides of each tube at each of the 12 AVB intersections) were very small. Both of these conclusions are consistent with the original design intent discussed below.

In-plane FEI is a phenomenon that had not been experienced in nuclear U-tube steam generators prior to its being identified in the SONGS RSGs. The practice in the nuclear industry at the time the SONGS RSGs were designed was to provide measures to preclude out-of-plane FEI in the U-bend region, which was based on the understanding set forth above. Reflecting this industry practice, the Japan Society of Mechanical Engineers' "Guideline for Fluid-elastic Vibration Evaluation of U-bend Tubes in Steam Generators" states that in-plane FEI does not need to be considered if out-of-plane FEI is controlled. The design of the SONGS RSGs is consistent with the contemporary industry practice and guidance. The RSGs were designed to provide effective tube support (by means of AVBs) to avoid out-of-plane FEI. MHI sought to maximize the 2

In U-bend SGs, because the tubes are curved, for the same support conditions the critical velocity for out-of-plane FEI will be lower than that for in-plane FEI because the natural frequency of tubes in the in-plane direction is higher, due to the tubes greater stiffness in-plane, than the natural frequency of the tubes in the out-of-plane direction.

Root Cause Analysis Reportfor tube wearidentified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

UES-20120254 Rev.0 (17/64) INon-Proprietar1 adequacy of the supports against out-of-plane FEI by increasing the number of AVBs to a number, 12, that exceeds that in other U-tube SGs designed by MHI or by other major U-tube SG manufacturers.

Minimizing tube vibration wear in the U-bend region was given high priority in the SONGS RSG Design Specification, the RSG design program, and in the manufacturing processes. Early in the project, SCE and MHI formed an AVB Design Team with the goal of minimizing U-bend tube vibration and wear. The AVB Design Team conducted numerous technical and design review meetings. The agreed-upon tube bundle U-bend support design and fabrication were as follows:

" Six (6) V-shaped AVBs (three sets of two) were to be provided between each tube column (12 AVB intersections total around the U-bend).

" Tube and AVB dimensional control, including increasing the AVB thickness was to achieve an effective "zero" tube-to-AVB gap under operating (hot) conditions with gap uniformity and parallelism being maintained throughout the tube bundle. Effective "zero" gap was desirable as an industry practice in order to maximize the effectiveness of the supports. The tube and AVB tolerances were to be tighter than that of any prior MHI SG.

" Excessive preload contact force was to be avoided in order to minimize ding/dent indications, and to maintain mechanical damping and thus minimize tube vibration.

MHI investigated field experience with U-bend tube degradation using INPO, NRC and NPE data bases, and concluded that the SONGS RSGs were designed to minimize the potential for tube wear by providing extra support points with shorter spans in the U-bend region along with effective zero tube-to-AVB gaps.

In the fabrication process, MHI manufacturing focused on achieving very small, uniform tube-to-AVB gaps during assembly.

The AVB Design Team included consultants with knowledge and experience in the design and construction large U-bend SGs. One consultant had experience with the design of a plant whose SGs were similar to the proposed RSGs (the "comparison" or "reference" plant). Together, the AVB Design Team concluded that the SONGS RSGs had more tube vibration margin than the comparison plant, which had experienced only a small number of tube wear occurrences. This conclusion was due to the following considerations:(i) SONGS RSG tubes are larger, have thicker walls, and are Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (18/64) INon-Proprietarv stiffer than those of the comparison plant; (ii) the SONGS distances between AVB tube supports are shorter than those at the comparison plant; (iii) SONGS has 12 AVB tube supports where the comparison plant only has 10; (iv) SONGS's tube-to-AVB gap requirement was more stringent than that of the comparison plant.

The Certified Design Specification S023-617-01, Rev. 3, issued by SCE required an effective zero gap and gap uniformity and parallelism of the tube bundle in the out-of-plane direction. Establishing the goal to reduce tube-AVB gaps to an effective zero gap was in accordance with well accepted industry practice and understanding that minimizing gaps was highly desirable in preventing tube vibration wear. MHI had sought to minimize tube-AVB gaps in its previous SG designs. However, MHI took additional steps to minimize the tube-AVB gaps for the SONGs RSGs and to provide for gap uniformity throughout the U-bend region of the tube bundle.

These steps included increasing the nominal thickness of the AVB compared to previous MHI SGs and reducing the manufacturing tolerance of AVB thickness and twist in order to achieve effective zero gaps and provide gap uniformity. Steps were taken as well to minimize tube ovality and to minimize variations from the design value.

Also, numerous additional steps were taken in fabricating the tube bundle to assure gap uniformity throughout the U-bend region. Additionally, in the fabrication of the Unit 2 RSGs MHI identified other enhancements that were implemented in the fabrication of the Unit 3 RSGs. These included, for example, taking steps to minimize AVB twist by applying a larger(from ons toF(j)tjons) pressing force in the Unit 3 fabrication and thus providing for more uniform AVBs in the Unit 3 RSGs.

The adequacy of the design against out-of-plane FEI was confirmed through test data and analyses that conservatively assumed that one of the AVBs provided in the design was inactive (that is, ineffective against out-of-plane FEI).Analyses using this criterion showed that an adequate margin against out-of-plane FEI exists in the SONGS RSGs. An additional AVB had been added to the design to provide further margin against out-of-plane FEI.

The MHI technical evaluations performed after the January 2012 incident determined that, despite the robustness of the MHI design, in-plane FEI had occurred. This occurrence was due to a combination of a lack of effective contact forces between the tube and AVB in the in-plane direction and localized thermal-hydraulic (T/H) conditions (high steam quality (void fraction) and high fluid velocity).The evaluations found that the average contact force in the Unit 3 RSGs was smaller than the average contact force in the Unit 2 RSGs. Therefore, the contact forces of the Unit 3 RSGs were more Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (19/64) lNon-Proprietary likely to be ineffective in preventing in-plane motion of tubes so that the Unit 3 RSGs were more susceptible to in-plane tube vibration than those in Unit 2. The difference in the contact forces between the Unit 2 and Unit3 RSGs is caused by the reduction in dimensional variations during the manufacture of the Unit 3 RSGs, mainly due to improvement of the control over tube and AVB dimensions in the manufacture of the Unit 3 RSGs. The reduced contact forces resulted in far more tubes in the Unit 3 RSGs experiencing tube-to-tube wear than those in the Unit 2 RSGs. For those tubes, given these support conditions, the vibratory energy in high localized thermal-hydraulic (T/H) environment produced in-plane FEI that led to large amplitude displacement of the tubes in the in-plane direction, which caused wear from contact between adjacent tubes.

Tube Wear at AVBs Tube-to-AVB wear is a function of the amplitude of the random tube vibration and the tube-to-AVB gap. Where there is a gap between the AVB and the tube and the vibration amplitude is less than the gap, there will be minimal or no wear. If the AVB is in contact with the tube but there is insufficient contact force to lock the two together, there will be relative motion between the two and wear will occur. In the case where there is sufficient contact force to lock the two together, there will be minimal or no relative motion and only minimal wear will occur. In the SONGS RSGs, the zero gap design philosophy resulted in the AVBs being in contact with the tubes or very close to the tubes, but there was insufficient contact force to lock the two together, thus allowing tube wear at the AVBs.

The degree of wear is also affected by the amount of damping provided by the water film between the tubes and AVBs. In the SONGS RSGs, damping was reduced in areas of high steam quality (void fraction)because there is less two-phase damping and little or no water film in the gaps between the tubes, resulting in more pronounced wear.

Tube Wear at Retainer Bars The tubes exhibiting retainer bar wear have no indications of tube-to-tube or tube-to-AVB wear, which indicates that the wear is caused solely by retainer bar vibration. The SONGS RSGs have two types of retainer bars: [b)(4)

I(b)(4)I diameter and diameter. Tube wear was only found on tues adjacent to the smaller iame er re ainer bars. The retainer bars with the smaller diameter have a relatively long span as compared with those for other SGs fabricated by MHI, which means that the natural frequency of these retainer bars is lower, making them more Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (20/64) [Non-Proprietar likely to vibrate. This type of wear is caused by random flow-induced vibration of the retainer bars caused by the secondary fluid exiting the tube bundle.

5.5 Discussion of Tube to Tube Wear Tube Contact Force During the fabrication of the AVBs and the tubing and assembly of the tube bundle, MHI's manufacturing practices achieved dimensional control that resulted in smaller tube-to-AVB gaps and smaller tube-to-AVB contact forces. It was not recognized at the time that a certain amount of tube-to-AVB contact force was required to prevent in-plane FEI under high steam quality (void fraction) conditions, because the contact force serves to increase the in-plane natural frequency of the tube.

The technical investigations after the tube leak incident determined that the amount of contact force necessary to prevent in-plane FEI depends on the localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).As the steam quality (void fraction) increases, the amount of contact force necessary to prevent vibration increases. This increase in required contact force occurs because as the steam quality (void fraction) becomes higher, the damping provided by the liquid phase in the form of a liquid flim decreases.

The reduced in-plane contact force due to the SONGS "effective zero gap" design and the avoidance of "excessive preload" resulted in lowering the tubes' natural frequency in the in-plane direction. The combination of the localized high steam quality (void fraction) and reduced tube to AVB contact force resulted in exceeding the in-plane critical velocity, which created a condition that led to tube to tube contact.

The dominant role played by the low contact force is reflected by the differences in the tube-to-tube wear that was observed in the Unit 2 and the Unit 3 RSGs. Each of the Unit 3 RSGs had approximately 160 tubes that experienced tube-to-tube wear whereas only one of the Unit 2 RSGs experienced tube-to-tube wear in just two tubes, even though the Unit 2 RSGs have operated twice as long as the Unit3 RSGs. MHI did a comprehensive statistical evaluation of the contact forces between the tubes and the AVBs of the two units and concluded, based on the manufacturing data , that the contact force between the tubes and the AVBs in the Unit 2 RSGs is approximately double the contact force in the Unit 3 RSGs. Thus, the lower contact forces in Unit 3 are consistent with the conditions determined necessary to permit in-plane FEI to occur and with the fact that tube-to-tube wear occurred almost exclusively in Unit 3.

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (21/64) LNon-Proprietar1 Thermal-hydraulic Conditions Many analyses are performed during the steam generator design process. One of these is MHI's FIT-Ill tube bundle flow analysis, which calculates tube bundle thermal /

hydraulic parameters, including U-bend flow velocity and steam quality (void fraction).An after-the-fact comparison between the T/H parameters that FIT-Ill predicted and those predicted by ATHOS, another T/H code, determined that FIT-III's calculated values are lower than those obtained using ATHOS. Part of the difference was because the pressure loss coefficients for the tube bundle and the two-phase mixture density utilized in the two codes were different.

Also, during the computation of the flow velocity, MHI used an inappropriate definition of the gap between tubes, with the result that the flow velocities were underestimated.

These differences between MHI's use of the FIT-Ill model and the ATHOS model resulted in a higher margin to out-of-plane FEI than the margin that would have been determined using the appropriate the definition of the gap and an ATHOS-calculated steam quality (void fraction). The margin calculated using ATHOS, nonetheless, would still have resulted in adequate margin against out-of-plane FEI. Using the ATHOS outputs, with all AVBs assumed active, the stability ratio was less than 1.0 for out-of-plane FEI, even for those case studies assuming reduced damping that could occur under high void fraction conditions.3 Thus, the use of ATHOS as opposed to FIT-Ill would not have identified an inadequate design margin against FEI.

Moreover, because industry practice was focused on out-of-plane FEI, use of ATHOS would not have identified the potential for in-plane vibration. Both the academic literature and subsequently conducted tests show that the thermal-hydraulic environment under which In-plane FEI arises is different from those that result in out-of-plane FEI. (See Supplemental Technical Evaluation Report). If the steam quality (void fraction) predicted by FIT-Ill had been the same as the ATHOS calculated value, 3

The maximum stability ratio based on ATHOS outputs for all supports are active isEEZl, which is less than 0.75, which is the conservative industry practice for judging acceptability of stability ratios (which in turn is less than the ASME Section III Appendix N-1330 recommended stability ratio criterion of 1.0). Assuming reduced damping, the maximum stability ratio calculated using ATHOS i Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (22/64) JNon-Proprieta and if the appropriate tube to tube gap value had been utilized to compute the flow velocity, MHI would have identified a decreased margin against out-of-plane FEI. In that case, MHI might have incorporated an additional AVB to increase the design margin against out-of-plane FEI, but would not have taken measures to protect against in-plane FEI, for it was assumed (as was the practice and guidance in the industry) that the controlling effect of a well-designed AVB system was adequate to preclude it.

Thus, not using ATHOS, which predicts higher void fractions than FIT-Ill at the time of design represented, at most, a missed opportunity to take further design steps, not directed at in-plane FEI, that might have resulted in a different design that might have avoided in-plane FEI. However, the AVB Design Team recognized that the design for the SONGS RSGs resulted in higher steam quality (void fraction) than previous designs and had considered making changes to the design to reduce the void fraction (e.g.,

using a larger downcomer, using larger flow slot design for the tube support plates, and even removing a TSP). But each of the considered changes had unacceptable consequences and the AVB Design Team agreed not to implement them. Among the difficulties associated with the potential changes was the possibility that making them could impede the ability to justify the RSG design under the provisions of 10 C.F.R.

§50.59. Thus, one cannot say that use of a different code than FIT-Ill would have prevented the occurrence of the in-plane FEI observed in the SONGs RSGs or that any feasible design changes arising from the use of a different code would have reduced the void fraction sufficiently to avoid tube-to-tube wear.

For the same reason, an analysis of the cumulative effects of the design changes including the departures from the OSG's design and MHI's previously successful designs would not have resulted in a design change that directly addressed in-plane FEl.

Summary Thus, the organizational and programmatic .Root Cause for the In-plane FEI as set forth in this RCA is the insufficient programmatic requirement to assure effective AVB contact force to control in-plane FEI under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydrodynamic pressure). The underlying reason for this insufficiency is that the MHI SONGS RSG design did not consider the phenomenon of in-plane FEI because contemporary knowledge and industry U-tubeSG operation experience did not indicate a need to consider in-plane FEI.

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

UES-20120254 Rev.O (23/64) INon-Proprietary 5.6 Discussion of Tube to AVB Wear Tube-to-AVB wear in the SONGS RSG occurs at the tube-to-AVB intersections and is produced by turbulence induced (random) vibration. This population only includes tubes with wear at the tube-to-AVB intersections with no wear indications in the tube free-span sections.

Tube wear at the AVB intersections (in the absence of tube-to-tube free span wear) occurs when the tube movement causes it to impact or slide along the supporting AVBs. The most common cause of this condition is out-of-plane FEI. In the SONGS RSG design, the large number of AVB supports and the superior gap control prevent out-of-plane FEI. However, because of the low contact forces between tubes and AVBs, the very small and uniform tube-to-AVB gaps, and the localized T/H conditions (high steam quality (void fraction) and high flow velocity), turbulent flow conditions are sufficient to produce tube wear at the AVB intersections. Again the effect of the different contact forces between Unit 3 and Unit 2 can be seen in the observed tube-to-AVB wear populations of the two units. Unit 2 had about two-thirds as many tube-to-AVB indications than Unit-3 and Unit 2 operated longer than Unit 3, indicated that the wear rate is greater at Unit 3. This is attributable to the lower contact forces.

(See Supplemental Technical Evaluation Report).

As was the case with tube-to-tube wear, it was not recognized at the time of the RSG design that a certain amount tube to AVB contact force is required to prevent random vibration under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).The combination of the reduced tube to AVB contact force and the localized T/H conditions (high steam quality (void fraction) and high flow velocity) resulted in tube to AVB wear.

5.7 Discussion of Retainer Bar to Tube Wear The design function of the retainer bar is to support the AVB assembly during manufacturing and prevent excessive AVB assembly movement during operational transients. The retainer bar must be strong enough to support the AVB assembly and fit within the physical constraints of the U-bend.

The tubesheet drilling pattern is one of the first design decisions made for a new steam generator and it is at that time that each tube location along the periphery of the tube bundle is established. The tube bundle design thus determines the retainer bar's length and thickness. At SONGS, in order to accommodate the increased number of Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (24/64) jNon-Proprietar tubes, the retainer bars are relatively long and thin as compared to the retainer bars in other SGs designed by MHI, resulting in their having low natural frequencies.

The engineer responsible for the retainer bar design did not recognize the need to analyze the retainer bar for flow induced vibration because no such analysis had been performed on previous MHI SG designs. The design control procedure for this design activity did not identify this issue, nor was it recognized during the design review process.

During operation, the secondary flow velocity and steam quality (void fraction) created turbulent flow conditions capable of causing high amplitude vibration if the retainer bar natural frequency was low enough, which turned out to be the case. The high amplitude vibration resulted in the retainer bar contacting some tubes and causing tube wear.

5.8 Root Causes As used in this evaluation, "Root Causes" are defined as the basic reasons (e.g.,

hardware, process, or human performance) for a problem, which if corrected, will prevent recurrence of that problem.

The programmatic Root Causes of the RSG tube wear are:

1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).

Basis: The evaluation team concluded that the fundamental Root Cause for the in-plane FEI and the resulting tube-to-tube wear was the fact that in-plane FEI was not considered in the design of the SONGS RSGs. The fundamental reason for this lack of consideration was that industry practice and guidance, supported by the operating experience up to that time of U-bend type steam generators, indicated that the control out-of-plane FEI would prevent the occurrence of in-plane FEI.

Likewise, the evaluation team concluded that the tube to AVB wear was caused by insufficient contact force under high localized thermal-hydraulic conditions, which was not recognized at the time of the design of the SONGS RSGs, and that the fundamental reasons for the ineffectiveness of the contact force were the established industry practice of minimizing the tube support gaps and Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation

I(b)(4)

UES-20120254 Rev.O (25/64) INon-Proprietar'*

avoiding an excessive preload as well as other steps to control gap uniformity and parallelism.

2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.

Basis: The evaluation team concluded that the fundamental reason for the retainer bar FIV was the lack of clear direction in the MHI design procedures to require an evaluation to determine the different analyses and the level of analysis that were required for the RSG design in light of changes in the SONGS RSG design from previous MHI steam generator designs.

5.9 Contributing Causes As used in this evaluation, "Contributing Causes" are defined as causes that by themselves would not create the problem but are important enough to be recognized as needing corrective action. Contributing causes are sometimes referred to as causal factors. Causal factors are those actions, conditions, or events that directly or indirectly influence the outcome of a situation or problem. The evaluation team closely evaluated the mechanistic causes and the design process for the potential existence of Contributing Causes.

The programmatic Contributing Causes of the RSG tube wear are:

(b(4)

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (26/64) Non-Proprietary (b)(4)

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (27/64) Non-Proprietar 6.0 Corrective Action Matrix

  • , * ~...

- = ;.. ' :6CreeAction

,r;***. ;,,*. - .*: J, "Due.. Da6i; Root Cause CAPR 1:Revise Procedure Completed

!.'Insufficient 5BBB60-NO1 "Procedure for programmatic Controlling of the Design Activities" requirement to assure to require that the need for effective effective AVB contact tube to AVB contact force under high force to prevent localized thermal-hydraulic in-plane fluid elastic conditions(steam quality (void instability and random fraction), flow velocity and vibration and hydro-dynamic pressure) be subsequent wear addressed in all MHI SG designs, under high localized CAPR 1.a:Further revise Procedure 11/15/2012 thermal-hydraulic 5BBB60-NO1 "Procedure for conditions (steam Controlling of the Design Activities" quality (void fraction),

to require that sufficient contact flow velocity and force is assured under high localized hydro-dynamic thermal-hydraulic conditions (steam pressure).

quality (void fraction) flow velocity and hydro-dynamic pressure), e.g.,

compare to the design parameters of previous successful MHI steam generator designs.

CA 1:Provide training for all Steam Completed Generator Engineers (included new hires and continuing training) covering this event and the details concerning in-plane FEI and tube-AVB wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (28/64) Non-Proprieta Cause -' Corrective Action <Due ?a~e Root Cause 2:The CAPR 2:Revise procedure 10/31/2012 design control process 5BBB60-NO1 "Procedurefor did not provide Controllingof the Design Activities" sufficient direction to to require that retainer bars and assure that an other steam generator parts subject evaluation of the need to flow induced vibration be for an analysis of flow evaluated to determine the different induced vibration of analyses and the level of analysis that the retainer bar was need to be performed to support the performed and steam generator design.

verified.

CA 2:Revise Engineer Training 10/31/2012 program (included new hires and continuing training) to include the necessary assessment for required analyses of each Steam Generator part subject to flow induced vibration.

(b)(4)

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

I(b)(4)I UES-20120254 Rev.O (29/64) [Non-ProprietaN~

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.0 (30/b4) iNon-Proprietaryj

..:, Cause Cause ,*... o r c .* cln*-*

Action: . ... Due Date Extent of Cause CA 6: Conduct a program design 3/31/2013 review for other SG design procedures and primary pressure boundary components (Reactor vessel, Core internals, Pressurizer, Reactor coolant piping, CRDMs) using senior engineers to determine if other design features have assumptions that are not programmatically captured and evaluated.

CA 7: Reconfirm MHI steam 11/30/2012 generator designs using the for SONGS SG design procedure developed for Root Cause 3/31/2013

2. for OTHER SG designs CA 8: Reconfirm that the appropriate Completed for analyses were performed and that SONGS SG design correct values were used as inputs 10/31/2012 for each thermal hydraulic analysis, for OTHER SG designs vibration analysis, and wear analysis (FIT-Ill, FIVATS, IVHET) in the design and fabrication processes of MHI steam generators.

CA 9: Reconfirm that the computer Completed validation was performed adequately for each thermal hydraulic analysis, vibration analysis, and wear analysis (FIT-Ill, FIVATS, IVHET).

  • If necessary, additional comparison to other validation methods shall be performed.

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Onofre NuclearGeneratingStation

(b)(4) I UES-20120254 Rev.O (31/64) N*on-Proprietarj Cause.: , CorrectvieA&tion ,A . Date Effectiveness Review In accordance with MHI's QA program, "Corrective action reports" will be issued for all CAPRs and CAs and the confirmation of effectiveness of completed corrective actions will be performed by the Nuclear Plant Quality Assurance Section.

Effectiveness reviews will be completed in six (6) months by verifying corrective actions for the addressed problems.

In addition, review the results of the initial Unit 2 & 3 mid-cycle outage and SG inspections to determine the effectiveness of corrective actions.

There is no evidence of :

, Additional tube to tube wear (in-plane FEI)

-Additional tube to retainer bar wear (turbulence induced vibration (random vibration))

- Additional tube to AVB wear (turbulence induced vibration (random vibration)).

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (32/64) INon-Proprietar7, 7.0 Extent of Cause Evaluation The Root Causes were evaluated for the extent to which they would be applicable and present elsewhere in the MHI steam generator design process.

The two Root Causes are:

1. Insufficient programmatic requirement to assure effective AVB contact force to prevent in-plane fluid elastic instability and random vibration and subsequent wear under high localized thermal-hydraulic conditions (steam quality (void fraction), flow velocity and hydro-dynamic pressure).
2. The design control process did not provide sufficient direction to assure that an evaluation of the need for an analysis of flow induced vibration of the retainer bar was performed and verified.

Root Cause 1 is associated with the design program and procedures not capturing necessary design elements affecting the primary pressure boundary. MHI has different nuclear engineering sections responsible for different aspects of the primary pressure boundary design, and each section has its own controlling design programs and procedures. Therefore, the extent of cause applies to the SG design program and areas of design outside the SG design program that could impact the primary pressure boundary. Sections outside the SG program with design responsibility related to the primary pressure boundary include:

a. Reactor Vessel
b. Core internals
c. Pressurizer
d. Reactor coolant piping*
e. Control Rod Drive Mechanisms To address this extent of cause evaluation, each MHI engineering section will conduct a program and procedures review, based on what was learned from this event, to determine if there are other SG program elements or other primary components that rely on design assumptions that are not captured in the design program or procedures.

For Root Cause 2, an analysis that should have been performed was not. Therefore, this extent of cause applies to other SG design analyses that should have been performed but were not. Because there is no controlling document that identifies what analyses should be performed for each component, CAPR 2 must be developed and Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (33/64) INon-ProprietarM then a complete review of the different MHI SG project needs to be performed to confirm that all required analyses have been completed.

8.0 Safety Culture Review A safety culture review was performed using the NRC's Inspection Manual Section IMC0310 COMPONENTS WITHIN THE CROSS-CUTTING AREAS and applying the guidance in that section to the Root and Contributing Causes identified in this report.

The review examined all four safety culture areas, the thirteen cross-cutting and other area components, and the thirty-seven aspects comprised in those components. A summary table 1 that compares the identified Root and Contributing Causes with the requirements of each of the safety culture areas, components and aspects is provided below.

As the table I shows, both Root Causes and all Contributing Causes are associated with aspect 6 (H.2(c)) of the "resources" component in the Human Performance Area. One Root Cause and all Contributing Causes are associated with aspect 2 (H.1(b)), of the "decision-making" component in the Human Performance Area. One Root Cause and all Contributing Causes are associated with aspect 4 (H.2(a)), of the "resources" component in the Human Performance area. Finally, one Root Cause and two of the Contributing Causes are associated with aspect 12 (H.4(c)) of the "work practices" component in the Human Performance Area.

The component from the Human Performance Area applicable to the second Root Cause and the three Contributing Causes is aspect 6 (H.2(c)) of the "resources" component, which calls for complete, accurate and up-to-date design documentation, procedures, and work packages, and correct labeling of components. This aspect of the resources component was not satisfied because, while the decision making and the designs were properly documented, they were inaccurate in that they did not require analyses to evaluate the potential FIV of the retainer bars (Root Cause 2);

(b)(4)

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

UES-20120254 Rev.O (34/64)INon-Proprietary I(b)(4)

'This component from the Human Performance Area is also associated with Root Cause" 1, in that the design procedures did not contain any requirement to assure effective AVB contact force. However, there is no safety culture related deficiency with respect to Root Cause 1 in that MHI was following accepted industry practices to design AVB and in fact sought to make its design more conservative than previous AVB designs.

An aspect of a component from the Human Performance Area applicable to one of the Root Causes and the three Contributing Causes is aspect 2 (H.1(b)) of the "decision-making" component, which requires that conservative assumptions be used in the design. The design did not require analyses to evaluate the potential FIV of the retainer bars (Root Cause 21:

(b)(4)

The discrepancies between the design and aspect 2 (H.1(b)) of the "decision-making" component also apply to aspect 4 (H.2(a)) of "resources" component.

Finally, an aspect of a component from the Human Performance Area applicable to one Root Cause and two of the Contributing Causes is aspect 12 (H.4(c)) of component 4 ("work practices"), which requires that appropriate supervision and management oversight be applied to the design. While design activities were reviewed and confirmed by the design section the design supervision and review process failed to recognize that FIV analysis of the retainer bars was needed (Root Cause 2);

ýb)(4)

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

UES-20120254 Rev.O (35/64)INon-Proprietary (b)(4)

MHI has identified a number of corrective actions, which are being taken or will be completed in the near future, to address the safety culture discrepancies identified in this review. These corrective actions are described in Section 6.0 above.. The predominant safety culture aspect was determined to be H.2.(c) Work Documents because the decision making and work practices were not influenced by programmatic requirements. The H.2.(c) safety culture aspect has the associated corrective action to establish the programmatic requirements for both Root Causes and the Contributing Causes, Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

UES-20120254 Rev.O (36/64) INon-ProprietarN Table 1 Safety Culture Review - Cross Cutting Components and Aspect X: Not sufficient SafetyCulture Area, Root Cause 1 Root Cause 2 Contributing Cause 1 Contributing Cause 2 Contributing Cause 3 Component, Aspect Insufficient programmatic requirement to assure effective AVBcontact force to prevent in-plane fluid elastic instability The sfiin design control ieto process oasr didhta not provide sufficient direction to assure that an and random vibration and subsequent evaluation of the need for an analysis of wear under high locailied flow induced vibration of the retainer bar thermal-hydraulic conditions (steam quality (void fraction), flowvelocity and hydro-dynamic pressure).

Area 1. Human Performance (H)

Component 1. Decision-Making Aspect 1.

Risksignificant decisions Sufficient - fMHI's AVBand tube bundle designs were reviewed and confirmed followed a decision-waking process to evaluate and review the technical aspects of the design.

H.i(a)

Aspect 2. a Conservative i" assumCtionservativ- Sfficient - The AVBdesign decision was i assumptions 11.1(b) Not sufficient -The engineer based on a FIT-Illanalysis which had a s responsible for the retainer bar design built in safety margin and assumed one I Xdid not recognize the need to analyze inactive support as an additional measure othe retainer bar for potential flow of conservatism additionally MHI's design Iinduced vibration had more AVBsthan previous designs.

Aspect 3.

Timely communication Sufficient - The decisions of the AVBand SGteam were documented and distributed to the team members in a timely manner.

H.I(c)

Component 2. Resources Aspect4.4."

Managing maintenance I Sufficient - The FIT-luanalysis had a built H.2(a) in safety margin and assumed that one Not sufficient - The engineer did not inactive support as an additional measure Xt recognize the need to analyze the I retainer bars for potential FIV.

of conservatismeUn Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof Son Oncifre Nuclear GeneratingStation

UES-20120254 Rev.O (37164) fNon-Proprietar I

Safety Culture Area, Component, Aspect Root Cause 1 Root Cause 2 Contributing Cause 1 Contributing Cause 2 Contributing Cause 3 AspectS.

Training and qualification personnel XI eswr ndqae Not sufficient - While the design section included euperts in SGdesign and manufacture, however procedure for training program was not sufficient because the training materials and procedures were inadequate.

H.2(b}

Aspect 6.

Work documents H.2(c)

Not deficient The decision making and design were documented, but the design Not sufficient - The decision making and procedures did not include a requirement design were documented, but the to prevent ir-plane FEfand random design procedures did not include a vibration related wear under high requirement to evaluate the retainer localized thermal-hVdraulic conditions. bars for potential FIV. The predominant There was no programmatic requirement safety culture aspect was determined to prevent in-plane FEIand random X to be H.2,(c) Work Documents because vibration, but MMIsought to make the there was no programmatic AVBdesign more conservative than requirement to influence the engineer.

previous designs so no safety culture The H.2.(c) safety culture aspect has the defioency is found. A corrective action is associated corrective action to establish nevertheless provided to address this new the programmatic requirement to understanding based on the tube wear evaluate for the need for an FIV analysis observed at SONGS.

___________ I.

Aspect 7.

Facilities and.Equipment Sufficient - The SGdesign section was provided with adequate facilities and other resources to conduct design review meetings and decision-making.

H.2(d)

Component 3. Work Control Aspect 8.

Work planning H.3(a)

Not applicable - Aspects 8 and 9 are not applicable because they address work in the plant and coordination of removal of safety systems during plant maintenance.

Aspect 9.

Work coordinationH.3(b)

Component 4. Work Practices Aspect 10.

Error prevention techniques H.4(a) Sufficient - Design activities were established in compliance with QAprograms to prevent error and personnel followed appropriate procedures.

Aspect 11.

Procedure compliance Sufficient - MHI's corrective action program governed the design process. Additionally the design section decisions were made pursuant to decision making procedures.

H.4(b)

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

UES-20120254 Rev.0 (38/64) INon-Proprietani Safety Culture Area, Component, Aspect Root Cause 1 Root Cause 2 Contributing Cause 1 Contributing Cause 2 Contributing Cause 3

-I. I- 1; Aspect 12.

Supervision and management oversight Sufficient -MHrs SGdesign activities were INot sufficient - The need for a FIV H.4(c) reviewed and confirmed by the design X2 analysis of retainer bar was not section at design review and technical I detected In the design review process.

review meetings.

Area 2. Problem Identification and Resolution (P)

Component S.Corrective Action Program Aspect 13.

Risk-based identification Sufficient - MHI's corrective action program governed the design process.

threshold P.1(a)

Aspect 14.

trending program P.1(b) Sufficient - MHI's corrective action program indcludes trend based assessments.

Aspect 15. Sufficient - MHI's corrective action program includes Root Cause and apparent cause assessments.

Cause evaluations P.1(c)

Aspect 16. Sufficient - No unresolved corrective actions were at issue.

Corrective actions P.1(d)

Aspect 17.

Alternative Processes Sufficient - MHIl has alternative programs In addition to its regular reporting program.

P-1(e)

Component 6. Operating Experience Aspect 18.

Systematic process Sufficient - MHIl investigated operating experience with U-bend tube degradation using INPO, NRC and NPE data bases, and communicated internally in a timely manner.

P.2(a)

Aspect 19.

Process changes P.2(b)

Sufficient - MHI conducted benchmarklng and concluded that the SONGS RSG was designed to minimize the potential'for tube wear by providing more support points with shorter spans in the U-bend region along with effective zero tube-to-AVG gaps during SG operation.

Component 7. Self- and Independent Assessments spect 20. " -- ...........................

Nature of assessments Sufficient -MHI periodically and appropriately conducted self-assessments.

___.3(a)____________II Root Cause Analvsis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

, A*I UES-20120254 Rev.0 (39/64) [Non-Proprietarl Safety Culture Area, Root Cause 1 Root Cause 2 Contributing Cause 1 Contribut ing Cause 2 Component, Aspect Contributing Cause 3 Aspect 21.

Tracking and trending Sufficient - MHI periodically and appropriately conducted self-assessment.

P.3(b)

Aspect 22.

Coordination and Sufficient - MHI coordinated and communicated result from self-assessment to affect personnel and took appropriate corrective actions.

communication P.3(c)

Area 3. Safety Conscious Work Environment (S)

Component 8. Environment for Raising Concerns Aspect 23.

Free and open Sufficient - The SG design team and AVB design team encouraged discussions of safety issues and openly exchanged information on design alternatives information exchange 5.I(a)

Aspect 24.

Alternate processes Sufficient - MHI has alternative programs for raising safety concerns in confidence.

5.1(bi Component 9- Preventing, Detecting, and Mitigating Perceptions of Retaliation Aspec*t 25.1 Sufficient - There were no claims of harassment bySG team members.

Training S.2(a)

Aspect 26. Sufficient - There were no claims of retaliation by SG team members.

inv..i....:__,_ i urk Aspect 27.

Chsillingeffect 5.2(c) Sufficient - MIII appropriately considers chilling effect. Nokdisciplinary actions were taken.

Area 4. Other Safety Culture Components (0)

Component 10 .Accountability Aspect 28.

Alignment of safety and Sufficient - Accountability for SGdesign decisions was clearly understood within MHI.

rewards O.1(a)

Aspect 29.

Reinforcement O.1(b) Sufficient - Management reinforced safety standards.

Aspect 30. Sufficient - MIII demonstrated safety focus, review of neeting minutes indicates focus of SGdesign team was to come up with design with appropriate margin which demonstrated focus on safety.

Safety focus 0,1{c)

Root Cause Analysis Report far tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

a-I ICC 5111SASCa O~.. fl IAflaCAl,,..O.~.~.,,,a*snA 01401641 a P o rieta SafetynCutureArea. Root Cause 1 Root Cause 2 Contributing Cause I Contributing Cause 2 Contributing Cause 3 Componsent. Aspect JCnrbtn as otiuigCue.3 I Component 11. Continuous learning environment Aspect 31.

Training and knowledge Sufficient - SG design team was trained on SGdesign continuously.

O.2(a)

Aspect 32.

Internal and external learning 0.2(b) Sufficient - SGdesign team was trained on SGdesign continuously and transfer of knowledge was provided by internal experts, however the issue of effective AVBto contact force was not known within the industry so training could not be effective.

Component12. Organizational change management Aspect 33.

Organizational change Sufficent - MHI management used systematic process and evaluated of impacts of decisions when organization was changed.

management 0.3 Componentl3. Safety policies Aspect 34.

Sufficient - Mill has appropriate policies which required reinforce to raise safety concern.

Raising concernsO.4(a)

Aspect 35.

Safety policy training Sufficient - MHI has appropriate policy training to raise individual safety concern.

0.4(b)

+

Aspect 36.

Decisions consistent Sufficient - Decisions related to SG design were consistent with MHI policies.

with safetypriorityO.4(c)

Aspect 37.

Top management Sufficient - Top management communicated need for safe SG design as issue of effective AVB to contact force was not known within the industry so management communication could not be effective.

commitment 0.4(d)

Root CauseAnalysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

Attachment-I UES-20120254 Rev.0 (41/64)lNon-Proprietaryi (Legend)

Cause-effect analysis for the tube to tube wear Phenomenon

~:]~ "'B"isunderlying cause of A F :Cause n Contriuting Cause RootCause

] "

Design element to Insufficient action to decrease avoid excessove steam quality (void fraction) between AVB and tube Till condition was judged acceptable byFIV Ontherpotential design changes arely, (SR- I and no excessive wear) arid considered to decrease steam dry out evaluation quality (void fraction) but they had unacceptable outcomes

-Note 1: Each SG on unit 3 had about 160 tubes with TTW. Only 2 tubes in one unit 2 SG had TTiN. This difference was caused by manufacturing and fabrication improvements implemented for unit 3. These improvements resulted in lower tube to AVB contact force in Root Cause Ill Insuffticient progremmatic unit 3 SGs when compared to unit 2 SGs.

requirement to assure effective 'Note 2. At the time of design a narrow gap AVB contact force to prevent between the AVB and tube was believed to Root Causes are defined as the basic reasons (e.g. hardware, in-olane fluid elastic achieve effective support condition-process, or human performance) for a problem, which If Instability and random corrected, will prevent recurrence of that problem. *Note 3: To consider identifying specific vibration and subsequent wear Contributing Causes are defined as causes that by themselves customer specification but also note that MHI under high localized would not create the problem but are Important enough to be evaluated and found such thermal-ftydmulic conditions recognized as needing corrective action. Conbibuting Causes speciftcation/requirement acceptable based are sometimes referred to as causal factors. Causal factors are (steam quality (void fraction),

on experiences.

those actions, conditions, or events that directly or Indirectly flow velocity and influence the outcome of a situation or problem. hydro-clynamic oressurel.

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

I Attachment-i UES-20120254 Rev.0 (42/64) Non-ProprtetaryJ Cause-effect analysis for the tube to AVB wear (Legend)

= - Phenomenon Random aln FA[- ] - is underlying cause of "A High localized TH conditin Inadequate support (insufficient tube to AVB contact force)

EL 0

I :Cause

Contributing Cause F

High Vefocty  : Root Cause i - -

Localized high steam quality (void tractiofn)

I~

Insuftfcient action to decrease steam quality (void fraction)

T/H condition wasjudged I+

Other potential design changes1 Deineeett (cotat ore) 1 7

eteen vi xesv rla Vantube III 0' acceptable by FIV analysis (SR<I considered to decrease steam and no excessive wea) and dryout quality (void traction) butthey evaluation had unacceptable outcomes I-. . . TomiOi id niotrecognize Satisfy design mechanical damping need for contact specification not to by the sliclng of tube force to control exceed 7% ding along AVB random vibration indications with ECT

  • Nlte I *Note 2 (b)(4) t I1 I
-IMHI relied on the IN Root Cause 41l I MHISONGS RSG design did not Insufficient programmatic based on the past consider sufficient AVS contact force requirement to assure effective successful experiences to prevent the random vibration AVB contact force to prevent in triangular tube under high steam quality (void in-plane fluid elastic instability *NOte 1- Al the time of design a narrow configuration SGS fraction) condition because and random vibration and gap between the AVB and tube and low Root Causes are defined as the basic reasons (e.g, hardware, contemporary knowledge and subsequent wear under high contact force was believed to achieve process, or human performance) for a problem, which If industry U-bend SG operation localized thermal-nydraultc effective support concition.

corrected, wIN prevent recurrence of that problem. experience dil not indicate a need conditions (steam quality (void C~ ntftbutlneC u e are defined as causes that by themselves to consider such phenomenon would not create the problem but are impofant enough to be fraction), flow velocity and 'Note I 'Note 2. To consider identifying specific recognized as needing corrective action. Contfrbuting Causes hydro-dynamic pressure). customer specification but also note that are sometimes referred to as causal factors. Causal factors are MHI evaluated and found such thoe actions, conditions, or events that directly or indirectly specficationflequirement acceptable Influence the outcome of a situation or problem.

based on excenences.

Root Ca,. .711uy07 n-Palu, jr Zuur wmuz identir)ed in the Unit 2 and Unit 3 Steam Generators of San Onafre NuclearGeneratingStation

Attachment-1 UES-20120254 Rev.0 (43/64)an-Proprietar Cause-effect analysis for the retainer bar to tube wear (Legend)

Retainer bar to tube wear

-  : Phenomenon il  : 'Er is underlying cause of A Random vibration F-1 Cause M1 --- Root Cause IRoot Cause (2)

The design cont process did not provide suffiient direction to assure that an evaluation of the need for an analysis of flow induced vibration Root Causes are defined as the basic .asons(e.g., hardware.

process, or human performance) for a problem, which If corrected, I_of the retainer bar was performed and verified- Jjjfl will prevent recurrence of that problem.

Root Cause Analysis Reportfor tube wear identified in the Unit2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-2 UES-20120254 Rev.0 (44/64) Non-Proprietar Barrier analysis A barrier analysis for the SONGS RSGs was performed with respect to the items listed below.

Design Fabrication Post Installation testing/monitoring Post Operational Inspection The barrier analysis was developed to assess the barriers at each of the major stages of the steam generator replacement program. The two primary barriers assessed at each stage included training/ qualification of personnel and procedures. As shown in the Barrier analysis table, procedures and training / qualification were lacking for the three wear mechanisms evaluated (tube to tube, retainer bar to tube, and tube to AVB).

There were no issues identified with the fabrication process so there were no failed barriers. The results of the barrier analysis support the cause-effect analysis.

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

Attachment-2 UES-20120254 Rev.0 (45/64) INon-Proprietar ConsequI Tube to Tube

_ I Barrier Design Training /

Qualificati In-plane FEI t

Outcome Not Effective Evaluation The design section procedure did not contain guidance on in-plane FEI. As a result, training programs did not cover the phenomenon of in-plane FEI. This will be addressed with CA 1 in the Corrective action matrix.

wea r on T-1model 0~

FIV analysis Not Effective Procedures In-plane FEI Not Analyses were not performed because there was no consideration of this phenomenon mentioned in the Effective procedure. This will be addressed with CAPR 1 in the Corrective action matrix.

TH model Not FIT-Ill predicted a lower velocity due to use of inappropriate gap value. This will be addressed with CA 3 in FIV analysis Effective the Corrective action matrix.

Supervisio In-plane FEI Supervisors used same procedures and received the same training as design engineers.

n Every 3 months, an Executive Oversight Meeting was held. Deputy Head of MHI and Department Managers Not participated. However, there were no questions related to in-plane FEI because it was not considered under Effective MHI procedure or industry practice. This will be addressed with CAPR 1 and CA 1 in the Corrective action matrix.

TH model FIT-Ill output indicated higher steam quality (void fraction) than previous SG designs. However, the senior FIV analysis engineer did not consider the potential adverse effects of the higher steam quality (void fraction).

Potentially This will be addressed with CA lin the Corrective action matrix.

Not Effective Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

Attachment-2 UES-20120254 Rev.O (46/64) Non-Proprietary Retainer Design Training Required Not Necessary analyses for each component were selected based on engineering judgment and past success.

Analyses I Effective Training was insufficient.

to tube This will be addressed with CAPR2in the Corrective action matrix.

bar wear Procedures Required There was no requirement to confirm the consideration of a FIV analysis for changes made to a component Not Analyses in the flow stream.

Effective This will be addressed with CA 2in the Corrective action matrix.

I i Supervision Based on past successful experience, engineering did not recognize the need to perform additional analysis Not for the retainer bars. The senior engineer did not identify the need for FIV analysis during the design Effective verification review.

This will be addressed with CA 2in the Corrective action matrix.

Tube to Design Training Contact force Not SG design training does not discuss contact force as a control mechanism to address vibration related wear under high Effective under high steam quality (void fraction) condition.

AVB wear steam quality This will be addressed with CA 1in the Corrective action matrix.

Procedur Contact force Not SG design procedures do not mention AVB contact force as a control mechanism to address vibration es under high Effective related wear under high steam quality (void fraction) condition.

steam quality This will be addressed with CAPRlin the Corrective action matrix.

Consequence all Fabrication The SGs were fabricated as intended. For unit 2 it was done using the normal fabrication process. For unit 3 Not it required divider plate failure repair. There were no causes identified associated with fabrication deviation applicable from the design.

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-3 UES-20120254 Rev.0 (47/64) Non-Proprietar Chanee analysis For the SONGS RSGs, a change analysis was performed in two stages. The first stage compared the SONGS SG design to previous MHI SG designs for the triangular tube configuration. MHI had previously performed three steam generator designs using a triangular tube configuration. The second stage compared the SONGS RSGs to the previous SONGS SG design (Combustion Engineering type design). Only the most significant changes are included in this analysis.

The change analysis results are set out below.

(1) Differences between SONGS RSGs and previous MHI SG triangular design.--

(b)(4)

The SONGS RSGs have:

(b)(4) circulation ratio

  • maximum flow velocity
  • average flow velocity P/D ratio
  • out-of-plane FEI stability ratio Largest U bundle radius
  • Specified AVB twist (b)(4)

(bo)(4)

[(b)(4)_ 'ange o; G-va C ue (tube diameter, out-of-plane)

Highest steam quality (void fraction)

  • Thinnest and longest retainer bar
  • (b)(4) ] nominal tube-to-AVB gap (0.002" cold / 0.000" hot) variation in tube-to-AVB gap (3 sigma (b)(4)

(2) Differences between SONGS RSGs and the previous SONGS OSG design. --

(b)(4)

  • Increase in tube bundle heat transfer surface area (11%)

Increase in number of tubes (5%)

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-3 UES-20120254 Rev.0 (48/64) INon-Proprietaryj Removal of stay cylinder Change from lattice bars to trefoil broached tube support plates

  • Change in tube support configuration in U region
  • Change from CE to MHI moisture separators
  • Power level / operating temperature / tube plugging margin (3) Identification of the changes from previous SG designs led to the recognition that the RSG design deserved close scrutiny. MHI considered the changes in the SONGS design from previous steam generator designs and compared the basic design parameters of the SONGs RSGs (e.g., heat transfer area, circulation ratio, steam pressure, etc.) with other steam generator designs. Further, as part of the development of the SONGS RSG design, MHI conducted a detailed comparison between its proposed AVB support for the tubes in the U-bend region and that of a comparison plant of similar design. A special AVB team was formed and included industry experts to conduct an extensive design review process in 2005 / 2006 to optimize the U-bend design and address the technical issues. The team concluded that the SONGS design was significantly more conservative than previous designs in addressing U-bend tube vibration and wear.

Also MHI and SCE recognized that the SONGS RSG steam quality (void fraction) was high and MHI performed feasibility studies of different methods to decrease it.

Several design adjustments were made to reduce the steam quality (void fraction) but the effects were small. Design measures to reduce the steam quality (void fraction) by a greater amount were considered, but these changes had unacceptable consequences and MHI and SCE agreed not to implement them. It was concluded that the final design was optimal based on the overall RSG design requirements and constraints. These included physical and other constraints on the RSG design in order to assure compliance with the provisions of 10 C.F.R. §50.59.

Thus, MHI did compare the SONGs RSG design with previous steam generator designs, and in particular did a detailed evaluation of different options of the AVB design taking into account other large steam generator designs.

(b)(4)

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

Attachment-3 UES-20120254 Rev.0 (49/64) Non-Proprietary

b)(4)

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear GeneratingStation

Attachment-3 UES-20120254 Rev.0 (50/64) INon-Proprieta Comparison between SONGS RSG Design and Previous MHI Designs Operating SCE RSG U2 SCE RSG Comparison Potential Evaluation Conditions U3 to other Ca use MHIl design

-I, Pressure (ata) 58.9

  • 1 Steam Flow 3.44E+06 (kg/h)

FW 228 Temperature (Mc)

S/G Level 1612 (mm)*2

-- 1 - - 1 ~*11 Circulation 3.3 U ýý&the high steam quality (void fraction) with r.J* ...........................

Ratiol-) *5 44. JUWVCILULJCU lII$JIII WII11.I II% u UIlUl Id lIUI I WILt OLIIer IdLtUr ca..n IedU LU LULU.

vibration.

Maximum "*<-- The high flow velocity provides the large dynamic pressure to the tube, which in Flow Velocity combination with other factors can lead to tube vibration.

(mls) *S Average Flow <- The high flow velocity provides the large dynamic pressure to the tube, which in velocity (m/s) combination with other factors can lead to tube vibration.

-*5- 1 P/D Ratio -) 1.33

  • Root Cause Ar_., . . T j___

ear identified in the Unit 2 and Unit 3 Steam I

Generatorsof San Onofre Nuclear GeneratingStation

Attachment-3 UES-20120254 Rev.O (51/64) INon-Proprieta Operating SCE RSG U2 SCE RSG Comparison Potential Evaluation Conditions U3 to other Cause MHI design 1~ -I-Stability Ratio a-

-~ I Astability ratio of out- of- plane FEI(

(highest) (-)

(Where) *5 U-bend Radius 4-r Largest 1 I The large bending radius gives(

(mm) I AVB Thickness (mm) 4 AVB width (mrm)

AVB twist (mm) j G-Values (mm) 4 a

i -

Nominal span between AVBs (mm)

I Root Cause Analysis Report for tube wear identified in the Unit2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

Attachment-3 UES-20120254 Rev.O (52/64) INon-Proprietarv Operating SCE RSG U2 SCE RSG Comparison Potential Evaluation Conditions U3 to other Cause MHI design Natural i .

C-Frequency (Hz)

(tubes of Concern) *4 Steam Quality 0.9 Highest V The high steam quality (void fraction) gives the low tube damping, which in (Void Fraction) (0.996) combination with other factors can lead to tube vibration.

(H*5

_______________j I AVB design *3 Solid type 0~

4 -I 4-' p Retainer bar 0' Thinnest Y The thinnest and longest retainer bar gives a low frequency, so FIV of retainer dimension bar may result.

Longest

  • 1 This parameter shows secondary pressure.
  • 2 The distance between the U-bend top to water level.
  • 3 This parameter shows RSG AVB.
  • 4 The lowest natural frequency of tube is provided
  • 5 Circulation ratio is obtained from SSPC code, and Max. flow vel., Avg. flow vel., stability ratio, and steam quality (void fraction) are obtained ATHOS code.

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear Generating Station

Attachment-3 UES-20120254 Rev.0 (53/64) INon-Proprietary 2

Comparison between SONGS RSG and Previous SONGS (CEI SG (OSG) Design*",

Design Element OSG Specification RSG Specification Potenti Evaluation al Cause Number of 9350 9727 a-Tubes Channel Head Stay cylinder to support tubesheet, Thick welded structural and Tubesheet floating divider plate divider plate Configuration Tube Support Batwing assembly, diagonal and Floating structure consisting Configuration vertical strips with interlocking of 6 V-shaped anti-vibration JU-Bend Section) horizontal strips between tubes, bars (AVBs) with 12 support lattice bars attached to structural points, retaining bars, members (shroud) external to the bridges, and retainer bars tube bundle Tube Support Lattice bars (egg crates) positioned 7 trefoil broached tube Configuration by tie rods and wedge-welded to the support plates (TSPs)

(Straight Section) shroud after alignment with positioned by stay rods, tubesheet, shroud is active part of radial load path at all TSPs radial load path, 2-inch linecontact on 2 sides, 1-inch line contact on 2 sides

  • 1 This analysis focused on mechanical differences because T/H conditions were expected to be similar.

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-4 UES-20120254 Rev.0 (54/64) INon-Proprietar RCA charter

Title:

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generators of San Onofre Nuclear Generating Station (b)(4)

Management Sponsor:

(b)(4)

Team:

Problem Statement:

(1) Requirement No Primary-to-Secondary Leakage due to Defects in any of the RSG Units for the duration of the Warranty Period. (per 17.2.3 of General T&C with EMS)

(2) Deviation Unit 3 SG-B (SCE SG088) experienced tube leakage during operation and failure of eight tubes during in-situ pressure testing. (Both due to Defects)

(3) Consequences (For MHI) 10CFR21 Report required (b)(4)

Timeline and Deliverables:

RCA Team Assigned : March 23, 2012 Problem Statement committed : March 23, 2012 Prepare begun: March 26, 2012 DRAFT Cause-effect analysis : April13, 2012 DRAFT RCA Summary : JulyS, 2012 Review RCA Summary : July7, 2012 DRAFT RCA Report : July20, 2012 Review Revised RCA Summary : August30, 2012 Review RCA report: September 6 and 11, 2012 RCA Due Date: October12, 2012 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-5 UES-20120254 Rev.O (55/64) INon-Proprietary1 Time line 046/90P-dlw.

(EIKA64O.NIJI

/I-o L!,.

_2012 R- O0.51/3 (Roplaced wrth588OO-2010 R-18 . 11/24 Rev.7. 3/12 Rev.6. 1/15 R.0.1/. 2/12 Reo16 2/5 2009

,~ 65512)2 R~ 22 Rev. 9. 8/29 Rev.5. 7/3 IRev.4. 713 5ey.

7/14 Rev.5. 6/11 R/v1.4.6/12

'Rev.3. 4/4 Rev 8. 4/7 Rev.4. 3/21 ev.3. 9 IRev.3. 3114 Rev.2. 2/27 R.o /4. 10/23 R8. 53. 1/07 R061l1 ,4/26 R-11/ 4//2 Rev.2. 416 Rev. 7. 2/27 Rev.2. 1/19 Rev.6. 1/23 Revi. 8/22 Rev.1. 9/20

[Rev.O 4/26

_2006 Rev.3. 2)17 Rev.O, 2/17 Rev 2. 2/3 R-r10..1/2 1Rev 2, 1/19 Rev.l, 12119 R4.8,12/1/ Rev.1. 11/25 1Rev.1. 11/14 Rev,?, 7/1

?/1 IRev.O. 9/30 ,REv.O.9/5 R..?. Rev.5. 10/21 Rev.4. 9/29 RevS. /17 Rev 3. 6/24 Rev.2. 6&~ Rev.2. 4/29 Rl 5,.2/7 Rev 1 4*29 Rev.0. 2/28 Rev , 319 2005 Rev.1. 1/7

ev.0. 11/30 R, 4.1022 R.v 3 8/23 2004 Ihe'"ee I,, 090/go 01 Analyt"o. W.I.,/ Leve/ DOtaf0m Des/gn Avib-Vibboenv Revod 0/ CoobodablOty LK-o./g /he Repot for Re/oll of 9.1 AVO Anly/sis Support Tube Shele Tube 11.504GA5002 LS-04CA428 A-onbly L5-O4GASI 9 6/0/158 Region L4r.GA418

$upo..ftd by L"AG4A61 0 LS-D4GA419 1.5-15GASI I L"-04GA401 Re.A Root Cause Analysis Reportfortube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-5 UES-20120254 Rev.0 (56/64) Non-Proprietairy Design Procedure (BKA60,-NO)

Time Line 2012 Rev.0, 5131 (Reptaced w"th5BBB60-NO1) 2011 Rev.18, 11124 2010 Rev.4. 1/25 Rev.17, 2/12 Re6. 2/5 2009 Rev.9. 9/25 Rev.3. 8/4 Rev.8. 8/20 Rev.7. 6/4 Rev.6. 4/4 Rev.7, 3/19 Rev.5. 2/29 Rev.2. 2/28 2008 Rev.15, 12/21 Rev.14. 10/23-Rev.13. 10119 Rev.12. 4/26 Re.11. 4/12 Rev.6. 4/2 Rev.5. 2/13 2007 Rev.4. 12/12 Rev.3. 11/17 Rev.4. 5/12 Rev.3. 4/19 Rev.2. 3/24 Re.10, 1/26 2006 Rev.2. 1/19 Rev.1. 1/20 Re .9 10/17 Rev.1. 11/11 Rev.4. 1017 Rev.0. 11/18 Rev.2. 10/5 Re.08, 919 Rev.0, 9/30 Rev.3. 9/7 jev.1. 9/9 IRev.0.6/9 Rev.l, 8/5 Rev.1. 8/12 Rev.7. 7/13 Rev.3. 7/29 Rev.6. 5/17 Rev.2. 5/17

~Rev.0. 6/27 Rev.2. 6/14 Rev.0. 6/27 Rev. 1 3/9 Rev.1, 3/23 Rev.5, 2/7 Rev.0. 2/24 2005 IRev.0. 1/21 Rev.0. 1/19 Rv.4, 10122 Rev.3, 8/23 2004 Material Material Material Design Basic Sizing Design of Qualification Selection Selection Selection Repor of Calculation Tube Support Report of Report for Report tot Report lot Tube Support Report Plate AVB End Cap Anti-Vibration Heat Transfer Tube Suppot Plate and L5-04GA421 L5-04GA425 L5.04GA429 Bar Tubing Plate Stay ROd LS-04GA224 L5-04GA208 L5-04GA223 L5-04GA411 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-5 UES-20120254 Rev.0 (57/64) [Non-Proprietary Design Procedure (A 01)

Time bOe 2012 Re-0. /S31 (Reptaced

.ath 5BOB60-N01) 2011 Rev.18. 11124 2010 Revo. 2/12 Re. 16.2/5 2009 Rev.5. 11112 Rev.4. 10/28 Rev.3. 8/29 Rev.2. 3/31 2008 Rev.l. 1/31 ReO.15. 12/21 Rev. 4.10/23 Rev.13. 10/19 Rev.0. 8/3 Rev.12. 4/26 Rev. 1.4/12 2007 IRev.2. 10/18 Rev.2. 10/18 ~Rev.2. 10/18

~Rev. 1. 8/22 Rev.1. 6/13 Rev.i. 6/13 ~Rev. 1. 6/13 Rev.10. 1/26 2006 Rev.9. 10117 Rev.0. 11/11 Rev.0. 11/11 IRev.0. 11111 IRev.0. 11/11 RevO.011/11 Rev.0. 11/11 Rev.8, 9/9 Re-. 7113 Rev.6. 517 Rev5. 2/7 2005 R,.4. 10/22 Rev.3. 8/23

ý I Thermal and Anti-Vibration Anti-Vibration Anti-Vibration Anti-Vibration Anti-Vibration Anti-Vibration Hydraulic Bar Assembly Bar Assembly Bar Assembly Bar Assembly Bar Assembly Bar Assembly Parametric 1/9 2/9 3/9 4/9 5/9 6/9 Calculations L5-04FUI 11 LS-04FU1 12 LS-04FUl 13 L5-04FUl 14 LS-04FU 115 L5-04FU1 16 L5-04GA510 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

Attachment-5 UES-20120254 II (b)(4)

Rev.0 (58/64) Non-Proprietary Design Procedure (BKA60-N01)

Time Line 2012 Rev.0, 5/31 (Replaced with58BB60- -

N01) 2011 Rev.18, 11/24:

2010 Rev.17, 2t12 Rev.16. 2/5 2009 Rev.15 12./2T Red.14 10123-Rev.13. 10/19 -

Rov.12. 4126 Retainer Bar Rev.1. 4/12 Rev.9. 3/9 IrDimesion

,Stress Engineer sign off 0cange of 2007 Rev.8. 12/8 Rev.7. 12/5 Rev.6. 11/9 Rev.5. 10/18 Rev.4. 9/26 Rev.3. 9/4 Rev.3. 8/24 Rev.2. 6/13 Rev.3. 4/6 Rev.1. 2/28 Rev.2. 2/17 Rev 10. 1/26 2006 Rev.1, 1/19 ORev.11/19 Rev.2. 12/2 Rev.9. 10/17 Rev.0. 11/11 Rev.0. 11/11 Rev.0. 11/11 Rev.8, 9/9 Rev.l. 8/12 Rev7. 7/13 Rev.6, 5/17 Rev.0. 5/27 Rev.5, 2/7 20O5 Rv.d. 10/22 Rev.3, 8r23 Rev.3.8/10 IRev.3. 8/10 2O0 Rev.2. 7/24 IRev.2. 7/24 Rev. 1. 6/8 IRev.l. 6/8 Rev.0.2/15 IRev.0. 2/15 IRevO. 2/15 Anti-Vibration Anti-Vibration Anti-Vibration Tube Suppod General General General Bar Assembly Bar Assembly Bar Assembly Plate Assembly 1/3 Assembly 2/3 Assembly 3/3 7/9 8/9 9/9 Assembly 313 KAZ-04-0021 KAZ-04-0022 KAZ-04-0023 LS-04FU1 17 L5-04FU1 18 LS-04FU 119 L-OP4FUI1O8 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-5 UES-20120254 Rev.O (59/64)"N-on-Proprietar Design Procedure - Rev 97 Rev.3. 9/27 10/12 (eKA60-Nl) - Rev.8, 9/14 Rev,2, 8131 TimeLine Rev.7, 8/30 Rev.l, 7/27 Rev.5. 6/18 IRev.4.6/20 Rev.6. 8/10 Rev,0. 7/4 Rev.4, 5/22 IRev3 5111 Rev.5. 7/17 Rev.2. 415 Rev.4. 7/13 IRev:1 3/16 Rev.3. 6/27 3/23 IRev.O. 2/13 Rev.2. 5/21

_2012 Revi1. 4/27 Rev.0 3/30 Rev.O.5W1 (Replacedwith 50000-)1l) 2011 Rev.18,11/24 2010 Rav.17, 2/12=

Rev.1, 2152 2009 2006 Rev.1,12/2- Rev.3.12/11 Rev,4.,10,23 Rev.13,10/19 Ray.12. 4/26 RViolette. M.J.Pettigrew. and N.W.Mureithi Rev.11,4112 Fluidelastic Instability of an Array of Tubes Preferentiallyl Flexible in the Flow Direction Subjected to Two-Phasel 2007 Cross Flow" ASME Journal of Pressure Vessel Technology Vo1.1 28l Feb.2006 I11 10/2 9/11 Rev.9. 8/4 Rev.8. 6/16 6/2 Rev.7. 4/19 3/13 Rev.6. 2/13 Rev10 1/26 2006 Ojiscussionabouttube to AVB gap control Rev.5, 12/21 Rev.2. 12/7 '12/5 Rev9.10/17 Rev.4. 10/21 10/17 Rev.8,919 Rev.3. 9/26 V.P.Janzen. EG.Hagberg, M.J.Pettigrew, Rev.2. 8/22 nd C.F.Taylor l98/17 Rev. 7/13 Fluidelastic instability and Work- Rate 7/11 Rev. 1. 6/8 Measurements of Steam-Generator Rev.6,5/17 Rev.l. 5/17 U- Tubes in Air-Water Cross-Flow" 5/23 IASME Journal of Pressure Vessel Rev.5. 27 Technology Vo1l.27 3/28 Rev.. 2/25 Feb. 2005 12/14 2605 Re4. 10722 Rev.0, 10/15 JSME Guideline "Fluid-elastic Vibration Rev.3,823 M.K IM. K. AEvaluation Au-Yang, l Mar. 2002 of U-bend Tube in SGs" "Flow-Induced Vibration Mr 2002 of Power and Process Plant Components" Jul. 2001 Replacement Evluabaonof Retaner Bar Tubewear of vahiity of Uhe ACWmyperiod H-/SCEDiesgn Steam Tue Vibraton TubeWear Una3 RSG - of the FIT-til of the RCA eaem Revion eeing at Goeeators L5-04GA504 Report Technical Results during fo. Ave Oesign Pro~ect LS-04GA561 Evaluaton Ceslgn and Fabricaton Eiecuuon Ran Report L5-04C.591 L5-04GB001 L5-04*A*64 Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre NuclearGeneratingStation

(b)(4)

Attachment-6 UES-20120254 Rev.0 (60/64) INon-Proprietar Interview list (b)(4) / (b)(6)

Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-6 UES-20120254 Rev.0 (61/64) INon-Proprietar (b)(4) / (b)(6),

Root Cause Analysis Reportfor tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-7 UES-20120254 Rev.0 (62/64) INon-Proprietar Reference documents

1. 5BB60-NO6Rev.1 "ASME Code Job Procedure for Preparation of Design Reports", 5/30/2012
2. 5BBB60-NOlRev.0 "Procedure for Controlling of the Design Activities", 5/31/2011
3. KAS-20050202 Rev.1 "FIT-Ill Code Description Note (Code User's Manual)", 11/18/2005
4. L5-04GA504 Rev.5 "Evaluation of Tube Vibration", 6/18/2012
5. L5-04GA588 draft "Summary of Technical Evaluation"
6. L5-04GA561 Rev.4 "Retainer Bar Tube Wear Report", 6/20/2012
7. L5-04GA564 Rev.9 "Tube wear of Unit-3 RSG - Technical Evaluation Report", 9/27/2012
8. L5-04GA591Rev.3 "Validity of Use of the FIT-Ill Results during Design", 9/27/2012
9. L5-04GA428 Rev.5 "Design of Anti-Vibration Bar", 7/3/2008
10. L5-04GA419 Rev.4 "Analytical Report of AVB Assembly", 7/3/2008
11. L5-04GA519Rev.2 "Water Level Controllability Analysis", 1/19/2007
12. L5-04GA511 Rev.9 "Data for Licensing Support Analysis", 8/29/2008
13. L5-04GA401 Rev.7 "Design Report for the Tube Sheet Region", 3/12/2010
14. L5-04GA418 Rev.5 "Design Report of Tube", 7/14/2008
15. L5-04GA224 Rev.2 "Material Selection Report for Anti-Vibration Bar", 1/19 2006
16. L5-04GA208 Rev.3 "Material Selection Report for Heat Transfer Tubing",7/29/2005
17. L5-04GA223 Rev.1 "Material Selection Report for Tube Support Plate", 9/9/2005
18. L5-04GA411 Rev.7 "Design Report of Tube Support Plate and Stay Rod", 6/4/2008
19. L5-04GA421 Rev.4 "Basic Sizing Calculation Report", 1/25/2010
20. L5-04GA425 Rev.9 "Design of Tube Support Plate", 9/25/2008
21. L5-04GA429 Rev.4 "Qualification Report of AVB End Cap", 5/12/2006 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-7 UES-20120254 Rev.0 (63/64) INon-Proprietar

22. L5-04GA510 Rev.5 "Thermal and Hydraulic Parametric Calculations", 11/12/2008
23. L5-04FU111 Rev.2 "Anti-Vibration Bar Assembly 1/9",10/18/2006
24. L5-04FU112 Rev.1 "Anti-Vibration Bar Assembly 2/9", 8/22/2006
25. L5-04FU113 Rev.0 "Anti-Vibration Bar Assembly 3/9", 11/11/2005
26. L5-04FU114 Rev.0 "Anti-Vibration Bar Assembly 4/9", 11/11/2005
27. L5-04FU115 Rev.2 "Anti-Vibration Bar Assembly 5/9", 10/18/2006
28. L5-04FU116 Rev.2 "Anti-Vibration Bar Assembly 6/9", 10/18/2006
29. L5-04FU117 Rev.9 "Anti-Vibration Bar Assembly 7/9", 3/9/2007
30. L5-04FU118 Rev.3 "Anti-Vibration Bar Assembly 8/9", 4/6/2006
31. L5-04FU119 Rev.1 "Anti-Vibration Bar Assembly 9/9", 1/19/2006
32. L5-04FU108 Rev.3 "Tube Support Plate Assembly 3/3", 9/4/2006
33. KAZ-04-0021 Rev.3 "General Assembly 1/3",8/10/2004
34. KAZ-04-0022 Rev.3 "General Assembly 2/3", 8/10/2004
35. KAZ-04-0023 Rev.0 "General Assembly 3/3", 2/15/2004
36. L5-04GBOO1 Rev.9 "Replacement Steam Generators Project Execution Plan", 8/4/2006
37. SB-SO-FB-0003 Rev.7 "Tubing and AVB Installation Procedure", 12/18/2007
38. SB-SO-FB-0004 Rev.10 "AVB Structure Assembly Procedure", 7/30/2007
39. UGS-L5-050044 Rev.7 "Anti-Vibration Bar Inspection Procedure (after assembling)", 5/21/2007
40. UGS-L5-050045 Rev.10 "Inspection Procedure for Tube and Anti-Vibration Bar Insertion", 11/2/2007
41. UGNR-SON2-RSG-067 Rev.7 "Nonconformance Report (Unacceptable gaps between Tubes and AVBs for Unit-2A)",9/21/2007
42. UGNR-SON2-RSG-075 Rev.1 "Nonconformance Report (Unacceptable gaps between Tubes and AVBs for Unit-2B)", 12/3/2007 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation

(b)(4)

Attachment-7 UES-20120254 Rev.0 (64/64) [Non-Proprietarý

43. UGNR-SON3-RSG-024 Rev.1 "Nonconformance Report for some Gaps between Tubes and AVBs are larger than the criterion", 4/24/2008
44. UGNR-SON3-RSG-030 Rev.0 "Nonconformance Report for:some Gaps between Tubes and AVBs are larger than the criterion",3/20/2008 Root Cause Analysis Report for tube wear identified in the Unit 2 and Unit 3 Steam Generatorsof San Onofre Nuclear GeneratingStation
Retrieved from "https://kanterella.com/w/index.php?title=ML13084A155&oldid=2457661"