ML042390208

E-mail from Lefrancois to Pelton Root Cause Analysis Report for MSIV Stem Galling

ML042390208
Person / Time
Site:	Vermont Yankee
Issue date:	05/10/2004
From:	Lefrancois M Entergy Nuclear Vermont Yankee
To:	David Pelton NRC Region 1
References
FOIA/PA-2004-0267
Download: ML042390208 (25)

Text

Pelton, David From: Lefrancois, Mark Sent: Monday, May 10, 2004 1:35 PM To: Pelton, DavidjeZ Cc: Annett, Michael; Rusin, Richard; Wierzbowski, George; Dreyfuss, John

Subject:

Root Cause Analysis Report for MSIV Stem Galling Please find attached the subject report. At your convenience, we are available to discuss any aspect of the MSIV stem galling issue. Note that the file is 1Mb in size, so it should probably be saved outside of the email system.

U MSIVRootCause.pd f (I Me)

MarkLe9Francois, System Engineering Supervisor, Entergy Northeast Vermont Yankee Tel 802-451-3058 This e-mail and any attachments thereto are intended only for the use by the addressee(s) named herein and contain proprietary and confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachments thereto is strictly prohibited. If you have received this e-mail in error, please notify me and permanently delete the original and any copy of any e-mail and any printout thereof.

A/

ROOT CAUSE ANALYSIS REPORT

.CRNumber: CR-VTY-2004-00955 Page 1 of 24 ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY- 2004.00955 Category: 0A MB Assigned Department: System Enineering Investigation Team:

Name Department Function Rich Rusin Components - Sponsor Michael Annetf System Engineering Team Leader lany Pfill ENNE-JAF- Valve Valve Expert Englneer George Short Training Team member with MSIV Maintenance Experience and Root Cause expertise Larry Lukens Programs Team Member Jim Taylor Technical Support Mentor Date Report Completed:

491004/4162004 0M The CR was caused by or Idenified an equipment/component failure.

(If YES, complete VYAPF 0009.06(EqipmentFailureEvaluation Chiecklist) in accordancewithAppendix.M.)

03 The CR Involved one or more HU Inappropriate acts.

(f YES, complete VYAPF 0009.05 (ENVY Human PerfornumceEvaluationForm) in accordncewith Appendix H.)

Problem Statement:

Ilvestigation fiolowing a failed As-found leak rate test during RF024 found MSIV V2-8OB to have a galled stem.

Executive Summary:

During investigation following a failed as-found LRT for MSIV V2-80B, the valve stem was found galled and bound into pacldng follower. Stem material was raised up due to scoring at the packing spacer and took up clearances between the stem and the pacldng follower, resulting in binding. There had been metal-to-metal contact between thc packing spacer and the valve stem at the "12:00 position" or "top df stem. The as-found diametrical clearances of V2-80B packing spacer/stuffing box allowed for metal-to-metal

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 2004-00955 Page 2 of 24 contact at the top of stem. The stuffing box ID is oversized by (0.009 to 0.015) [2.299 as-found; 2.4-2.290 design]. This oversized condition resulted from a maintenance activity pedormed prior to 1999 when the stuffing box bore was "cleaned p", "honed",

or "machined" without adequate control of the work activity. Prior to 1999, the significance of ttuffing box clearances was not explicitly documented and may not have been recognized. The failure to recognize the significance of the clearance tolerances within the stuffing box msulted in the valve being reassembled with inadequate clearances. With the excessive clearaice between the packing spacer and the stuffing box ID, the clearance between the stem and pacdking spacer ID is compromised, allowing metal-to-metal contact.

Immediate corrective actions in RF024 included modification of the staffing box configuration to utilize a custom fit carbon packing spacer In lieu of the cadmium plated carbon steel standard packing spacer and increased clearances in the backseat and packing follower areas.

Long term corrective actions include development of a MSV Maintenance Procedure to improve control of the maintenance activity on the MSMs and ensure configuration control.

Report Narrative:

Description of the Condition.

Duing investigtion following a filed as-foundLRT forMSIV V2-BOB, the valve stem was foumd galled and bound into packig follower. Stem material was raised up due to scoring at the packing spacer and took up clearances between the stem and the packing follower, resulting in binding. There had been metal-to-metal contact between the packing spacer and the valve stem at the "12:00 position" or "top" of stem The as-found diametrical clearances of V2-80B pacing spaceristufflng box allowed for metal-to-metal contact at tIhe top of stem. The staffing box ID is oversized by (0.009 to0.O15) [2.299 as-found; 2.284-2290 design]. This oversized condition resed from a maintenance activity perfonned prior to 1999 when the stuffing box bore was "cleaned up", "honed",

or "machined" without adequate control of the work activity. Prior to 1999, the significance of stuffing box clearances was not explicitly documented and may not have been recognized. The failure to recognize the significance of the clearance tolerances within the stuffing box resulted in the valve being reassembled with inadequate clearances. With the excessive clearance between the packing spacer and the stuffing box ID, the clearance between the stem and pacldng spacer ID is compromised, allowing metal-to-metai contact.

Barrier Analysis:

A barrier analysis (see Attachment A) was performed. The following barriers which should have prevented this condition failed:.

ROOT CAUSE ANALYSIS REPORT

'CRNmber: CR-VTY-204-00955 Page 3 of 24 Existing procedmrlized Packing Muideline (OP 5281) is too generic to address specific req ts for the MSIVs. For example, stufing box dimensional checks are m ured in inches as opposed to the nearest 0.001" and necessary stem centeringlalignment techniques during valve packing are not addressedc The work control process failed to control the work activity that resulted in the oversized stuffing box bore. Ihis oversized condition resulted from maintenance activity performed prior to and daring RFO19 (1998). Stuffing box wear and dimensional information was not obtained from the Vendor until the 1999 RFO when the 80AD and 86 AD valves were refurbished.

The corrctive action program faled as the gafling which occurred in 1998 was identified as due to excessive clearance between the suffng box bore and the packIng junk rng (at that time, it was because the junk ring was under sized). Ihe coretive actions in 1998 did not address th stem galling, but were instead simply focused on restoring the valve to pass the ILRT. The setm was replaced, but no actions were developed to prevent recurrence.

'Te design change process utilized in 1989 when the pacldng configuration was changed failed to incorporate guidance from the vendor that the tuffing box dimensional clearances should be verified.

A timeline of the events is provided as Attachment B.

IntervIews:

Interviews were conducted with the RF024 Day and Night leads and the mechanic responsible for disassembly of the MSIs. These individuals have been involved with maintenance of the MSIVs for the full span of the timeline since before 1998 to the present and have significant expertise on the subject valves. The interviewees were forthcoming and showed a very positive effort to determine the cause of the condition.

The most significant result of the interviews was a confidence that the as-found condition of the valve was not the result of actua orlvae misalignment. The descriptions of the medthds to disassemble, the results of feeler gauge inspections of actuator feet, and the lack of any cold spring in the stems upon decoupling provided this confidence. The interviews also provided insights into methods and processes utilized to ensure alignment which ar not necessarily documented in procedure, but have evolved over time.

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 2004.00955 Page 4 of 24 Mportant data from the Related Condition/OE Search In summary, the OE suggests that other plants also have MS1Vs with tight tolerances and our peers highlighted the fact that valvelactuator alignment is critical to ensuring score-fire operatio. Alignment means actuaodvalve stem alignment, bonnet centering, valve stem/disc/body bore alignment, verifying body bore concentricity, verifying stem clearances in the stuffing box, backseat, and packing follower. Most plants using the Rockwell Edwards Y-pattern globe valve still use the cadmium plated carbon steel packing spacer.

Equipment Failure Evaluation The key insights from the Equipment Failure Evaluation are detailed below.

1. A deficiency in past maintenance was identified where the stuffing box bore was machined without adequate control of the activity. The resulting oversized bore in the valve bonnet allowed for metal-to-metal contact of the packing spacer to the valve stem.

2. Previous correcdve actions associated with a stem galling event (CR-VTY-1998-00476) did not adequately address actions to prevent recurrence. The CR resolution focused on resolution of the leak rate failure problem.

3. Current work practices are not procedurized and rely heavily on the experence and skill of the craft and may allow for excessive discretion of the craftsman and/or supervisor and may reqire personal familiarity with the specific attributes of particular valves to ensure adequate alignment and assembly (ie. bonnet centering, stem centering, and valve pacldng techniques.)

4. The are design deficiencies which contribute to the failure including certain valves have conditions which are outside design tolerances. These conditions are not reflected on controlled drawings. The valve vendor has reviewed conditions outside of tolerance (not the stuffing box bore, however) and has accepted it, but the deviations are not reflected on controlled documents. Maintenance has developed work-arounds to cope with certain conditions (Bonnet centering using dial indicators where the bonnet rabbet fit is outside of design tolerances.)

Root Cause of the Condition:

RC [U.1*] Risks and consequences associated with change not adequately reviewed or assessed.

(Corrective Maintenance)

Prior to 1999, when the V2-80B stuffing box was 'cleaned up" or honed", tbe stuffing box dimensions were not verified to check the packinglstg box clearance against design ullowables. The significace of the tolerance in this

ROOT CAUSE ANALYSIS REPORT CR Numben CR-VTY- 2004-00955 Page 5 of24 area was not recognized. The oversized condition allowed the packing spacer to contact the sem leading to stem galling and ultimately to binding. There was an opportunity to verify that the stuffing box dimensions were correct in 1989 (VYV-89-201), when the packing configuration was changed. At that time, the vendor had advised that the stuffing box dimensions should be verified.

Contributing Cause(s)

CC-1 [N.v.21 Not per desgn fabrication.

V~anufacftuirng)

The bonnet rabbet clearances are not within vendor specifications for all of the valves. Ths condition creates the need for special alignment techniques using dial indicators to ensure that the bonnet is centered. These techniques are not procedralized, but are performed in the field. This condition can lead to scoring on the top of the valve stem at the bonnet barelbackseat location similar to that found on V2-8OA and V2-86D. Additionally, the V2-80B valve stem (design 1.498-1.500 inch di&) was 0.001 inch oversized. By design, there is only 0.001 inch of radial clearance betwem the packing spacer and the stem if the stem is 1.500 inches and the spacer is located at an extreme end of the stuffing box. Tis made te available radial clearance cven if th stuffing box bore was correct only 0.0005 inches.

CC-2 [L2a.2] Causes of a known problem were not determined.

(Corrective Action)

No CR was gencaed in 1998 to specifically address the galling problem with the V2-80B valve.

The CR generated in 1998 (CR-VTY-1998-00476) to address the failure of V2-BOB to pass itfs As-Found leak rate test briefly discussed the stem galling problem with the valve but focused primarily on the failed leak rate corrective actions.

Response to Specific CRG InstructonAddional Considerations:

N/A Extent of Condition:

Effects on Equipment:

The condition associated with the over-bored stuffing box may impact any of the eight MSIVs at ENVY. Including V2-80B, five of the valve stuffing boxes have been inspected during RFO24. No others hadfe overbore to the extent of V2-80B. V2-86C was 0.002 oversized at the top, but this condition should not adversely affect the stem as the bottom of the box is within specification. No scoring was evident on the V2-86C stem. All five valves have been equipped with custom fit packing spacers during RF024.

The other three will be equipped similarly wben they ae scheduled for repack

ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY. 2004-00955 Page 6 of 24 There is no Immediate nuclear safety or risk to generation concer associated with this condition for the valves which have not been worked in RF024, V2-86B, V2-80C, and V2-80D. This modified packing is considered to be an enhancement Based on document review, none of these three valves is known to have an oversized stuffing box and there is no evident or reported stem scoring on any of these valves.

There are no other large Air-Operated Rockwell-Edwards Y-Pattern Globe valves at ENVY. There are two large Motor-Operated Rockwell-Edwards Y-Pattern Globe valves which may be affected by similar conditions; these ar th RHR Outboard Injecdon Valves VIO-27A and V10-27B. A corrective action has been initiated to review the mainte e history of V10-27A, B to ascertain whether there has been stem galling or binding and whether the stuffing box configuration is vulnerable and to review the e processes which would be used to control work activities on these valves to ensure tight tolerances are maintained.

The IAF MSIVs are a similar valve and would be vulnerable to similar conditions.

Presently, one JAF MSIV has stem scoring.

Effects upon Pracessvlprrams:

The effected processes include the work control process, the configuration control process, and the corrective action program process. No other processes ae impacted.

Baniers from each of these three processes should have prevented the stem galling problem described herein. Corrective Actions will address necessary changes to the work control and configuration control processes associated with these valves. The changes to those processes will effectively ensure that the Corrctive Action proess is Invoked when necessary.

Effects upon Human Performance:

The lack of process controls (specific procedures) allowed for errors to be made and go undetected. There was a component of the "Pollyannae effect where maintenance was conducted without awareness that critical design clearances could be affecteda There was lack of a "Questioning Attitude" with respect to work performace which was not controlled in procedure or even in work order step tet. Another human factor obseation is that when worling MSrVs after a failed LLRT, the focus of the team is to correct the leaking valve. Side issues, such as scored stems, have a secondary level of concern. Therefore, the scored stem would tend to be "corrected" by replacement or buffing/polishing and valve reassembly and alignment. The cause of the stem score would tend to go uninvestigated.

ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY. 2000095 Page 7 of 24 Related Operating Experience:

Previous Related Conditions (ENVY):

Identify results of Related Conditions/OH Search (see Appendix 0).

CR-VTY-1998-00476 describes a similar galling event associated with V2-80B. In that event the pacldng junk ring was found to have been undersized The same clearance discrepancy which caused this event had cause that event, ie HD of stuffing box to OD of junk ring or packing spacer, resulted in stem galling. A root caused evaluation was performed in 1998 to address the multiple failed leak rate tests, but there were insufficient corrective actions to prevent the stem galling froi recuring.

CR-VTY-2004-00918 was initiated during RF024 to address an adverse trend in ENVY MSIV as-found leak rate test results.

Related Idustrv OE:

Important data from the Related Condition/OE Search In summary, the OE suggests that other plants also have MSIVs with tight tolerances and our peers highlighted the fact that valvelactuao alignment is critical to ensuring wcore-free operation. See Attachment G.

J

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 2004-00955 Page 8 of 24 Corrective Actions:

Immediatellnterim Actions Completed Item I Action Taken N/A In'tiated the planned contingency diagnostic Inspectlonlrepalrkefurbish plan for Waes falling LLRT. WO 04-001270-000 for V2-806 CC-i Install custom packing spacers (Carbon) Inaccordance with WOSE 2004-030 Which include specific clearance tolerance for as-bunltas-found stem and Stuffing box dimensions on valves which are worked during RF024.

NN Nuclear Network entry # OE18258 was submitted. (CR-VTY. 2004-00955 CA 2C)OOOO2 ProposedlAssigned Corrective Actions Item I Action CAType Assigned Due Date CA#

._ Department RC-1 Develop acontrolled MSIV LT-CR Mantenance 3131,2005 CR-VTY- 2004-CCo- maintenance procedure that Support 00955 CA00004 CC-2 adequately describes the required as-found dimensions, the acceptance criteria for the dimenslons, the action required for out-cf-tolerance dimensions, the disassembly process, the refurbishment process, the reassembly process, and the post-maintenance test requirements and acceptance criterlL This procedure will address both packing and valve Internals maintenance. Itis Intended that this procedure will be a stand-alone reference for all MSIV maintenance work.

RC-1 Update MSIV drawings, as LT-CA Design May30, 2005 CR.VTY, 2004-CC;1 necessary, to specifically Engineering 00955 CA0000S identify critical design -

dimensions. For example, the bonnet rabbet clearances, the stuffing box clearances, allowed stem diameter, and valve lhtemal dimensions.

CC1 Initiate WOR to Install custom EN System 613012004 CR-VTY. 2004-packing spacers (Carbon) in Engineering 00955 CA00006

ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY. 2004-00955 Page 9 of 24 Proposed/Asslgned Corrective Actions Item# Action CA Type Assigned Due Date CA A

. Department accordance with WOSE 2004-030 (or equivalent ER) which Inludes specifi clearance tolerance for as-bullt stem and stuffing box dimensions on three valves which are not worked during RF024 (V2-80C, V2-80D, and V2861B).

These upgrades to be Installed at the next occasion to repack the valves.

CC-1 Provide a recommendation to EN System 6i3aW2O04 CR-VTY- 2004.

Outage Management Engineering 00955 CA00007 specfying recommended Scope of work for RF025 on the three MSIVs not worked during RF024. This recommendation should be In the form Qf a memo. .

RC-1 Reviewldocument all known EN System 12/3&00D4 CR-VTY. 2004-CC-1 deviating conditions for all Engineering 00955 CA00008 MSIVs and develop a restoration plan.

Common Review the maintenance EN System 12O2004 CR-VTY-2.4-history of V1 0-27A, Bto Engineering 00955 CAOOOO9 ascertain whether there has been stem gafling or binding and whether the stuffing box confiratlon Isvulnerable and to review the maintenance processes which would be used to control work activities on these valves to ensure tight tolerances are maintained.

Identytevlewldocunent all known deviating conditions for these valves, Ifany, and develop a restoration plan.

CC-2 Initiate TCR to provide training EN System 6/30l2004 CR-VTY. 2004 to Mechanical Maintenance Engineering 00955 CAI)01 personnel with respect to potential signifficance of dimensional tolerances which can be Impacted by maintenance activIty.

CC2 Initiate TCR to provide training EN System 6/O2004 CR-VY 2004-to Engineering populatlon Engineering 00955 CAOOO11

ROOT CAUSE ANALYSIS REPORT CR Number CR-VTY- 2004-0095S Page 10 of 24 ProposedlAsslgned Corrective Actions Item I Action CAType Assigned Due Date CA#

Department during ESP cycle training with respect to potential signficance of dimenslonal tolerances which can be Impacted by maintenance

_ __ __ _ _ a c ti iy _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

Personnel Interviewed:

Larry Doucette Rich Booth John Apostoles Ron Scheman Joe Boivin SL Adams - Flowserve Corporation (Formerly Rockwell Edwards)

Keywords:

Stem Scoring Stem offing MSIV LIRT Attachments:

A. Barrier Analysis Worheet B. Timullneof Events C. VYAPF 0009.06 Equipment Failure Evaluation Checklist D. VYAPF 0009.05 ENVY Human Performance Evaluation Form E. Sketch of Stuffing Box Configuration which allowed galling F. Sketch of As-found stem condition G. Summary of Industry Operating Experience telephone conversations IL Mechanistic Root Cause Evaluation L Independent Peer Review by White Plains Metallurgist J. Historical MSIV Leak Rate Test Results

ROOT CAUSE ANALYIS REPORT CR Number: CR-VTY- 2004-00955 Page 11 of 24 Attachment A BARRIER ANALYSIS WORKSHEET BARREER ASSESSMENT CONSEQUENCE(S) BARRIER(S)THAT SHOULD HAVE [WHYTHEBARRIER(S)

PRECLUDEDIFALD MSIV V-80B stuffing box OP-5281 ValvePacling idelines Proedureprovides only inside diameter generic guidelines for measurement out of genring andgrecording tolerance. Reading 2.299 esuring b and packiring Specification 2.284-2290 cOmponents. MSIV BOB requires precision measuents to ensure proper fit and clearances. Refer to

___ VYAPF 5281.01 V2-80B pacldng stuffing Work ControlPocess AP-0168 Specific guidance frpacking bodxelarged s to larger stuffing box dimeasions not design 5pecfficflonls .included in the work order nor are they recorded in the work order notes. No statement of caution in WO to address the critical nature of packing components dimensions and

_ required clearances.

Corrective action pocess Corrective action process AP-0009 No cmrective action to failed to address and incorpoate 1998 analysis of correct V280B stem paclkng component g ng. dimensions, as written in WO notes 98-002519-000, into pffit procedures and processes.

Corrective actions focused on LLRT failure not stem galling issue. 1998 failure similar in nature to current 2004 failure.

Pailure to recognize the Plant Drawing 5920-2038 Best estimate of time stuffing stuffing box and packing Work order process AP-0168, bo for V280B t was oeursized spacer clearances Plant procedure OP-S281 cleanrot stuffing boxc not identified in a work order, and not included in the work order notes. Hand written drawing used to identify components

_ and clearances instead of using

ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY- 200440955 Page I1 of 24 CONS]EQLTENCE(S) lBARRIER(S) THAT SHO D HAVE PELDDWHT.HE ECLUDEDFAILE]

I BARRIER ASSESS NT BARRIER(S) poper drawings. Pacling done in accordance with OP-5281 which is generic In nature.

Risks and consequences Corrective action program AP-0009 Out of specification (Bore associated with change not diameter 2.299 instead of adequately reviewed or 2.2842.290 not entered into the assessed corrective action program i.e.

corrective action not utilized eliminating b~st chance of evaluating risks earlier.

Risks and consequences OP-5281/Work order process Procedurl guidance in OP associatedAwith change not 5281 does not specifically adequately reviewed or address MSIV packing assessed component dimensions and allowable clearances.

Additionally it does not discuss the potential consequences of improper clearances. Also Work Orders regarding MSIV repacking do not identify dimensions or consequences associated with out of specification measurements.

Work orders step text refers -

back to OP-5281 and VYEM-0079 Risks and cosequences Com nicatons; Verbal and written Direction to clean stuffing box associated wit change not by mechanical means done adequately reviewed or verbally by project lead to assessed mechanic. Communication based on no formal written

- communication e.g. procedure, work order. Discussion of change in stuffing box inside diameter and associated consequences of that change not discussed. Additionally no formal written communication

ROOT CAUSE ANALYSIS REPORT CR Nunmbe CR-VTY. 2004-00955 Page 13 of 24 BARRIER ASSESSME3NT CONSEQUENCE(S) T SHOUDHAVE YBARRTR(S)HT lEBARRIER(S)

_________________IFARME]

exists to idetfypoesse used and before and after dimensons. lnvestigation of event revealed hand written notes done by project supervisor and interview with individual. Action considered skill of the craft without consideration of clearance dimensions.

Failure to recognize the 1989 Design Change process when Vendr reormmnded verifying significanceof tndpi packing configuration was chaigedL stuffing box dimensions, staffing box and pacln.sgeting ltcgou space clea~cessuggesting that cleaning out space cleaancesbox tends to open ul tolerances over time. There is no evidence that this reconmenidation was heeded.

%(Lst one at a time; Need (Identify all applicable Physical and (Identify if barrier was missing, not be In sequential order.) Administrative bamriers for each -weak,or Ineffctdyi d why.)

consequence.)

ROOT CAUSE ANALYSIS REPORT CR Number CR-VTY. 2004.00955 Page 14 of 24 Attachment B Event Time Une At time of modifying the valve packing 1012511989 configuration, vendor provides advice tt verifying the dimeons of the stuffing box bore is important as the bore can become oversized over tm as box is cleaned out.

V2-80B Fails As-Fbund Local Leak Rate test 3/2211998 (CR-VTY-1998-00476)

V2-80B has stem galling attributed to junk ring 3t23/1998 out of tolerance allowing junk ring contact with stem (OD of junk Ring small)

(98-002519-000) .__

Stuffing Box machined such tdat ID is over- -3123/1998 sized V2-80B Fails As-Found Local Leak Rate test 10107/2002 (CR-VTY-2002-02211)

Valve repaired under WO 01-004406-000, 1018QOO2-1/13t2002 stem reported in good condition, no scoring, installed new packig spac=r V2M80B Fails As-Left Local Leak Rate test -10114=2002 CR-VTY-2002-02503)

Valve repaired under WO 014M406-004, 10116/02 - 10t2Q including replacing stem Valve identified to have scoring evident (CR- 10/1;2003 VTY-2003-02165)

Failed As-Found Leak Rate Test (CR-VTY- 415/0 2004-00841)

Stem Bound in Packing Follower (CR-VTY- Reported 4/812004 2004-00955) II

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 2004-00955 Page 15 of 24 Attachment C ENVY EQUIPMENT FAILURE EVALUATION CHECKUIT CR Np: Dso~lnn et CR-VTsY- 2004-0095S System EngneeringI Equlpmentl~omponent Being Evraluated:

Main Steam Isolation Valve V2.801B Info

- Response l Included In hem 4 Question for Review j Disposition YES NO WA Report I Was the equipment failure expected? X l 2 Had any associated PMs been missed or otherwise delayed?

Isthere any aspect of the PM Program X that appears to be hadequate? _ _ _

4 Are there deficiencies Inthe perforance x MA of past manteriance? X ._MBA 5 Are there system or component X Performance Monitoring deficiencies?

6 Does anyOE exstthat is applicable to x MGA

_ this component? _

7 Does any associated component OE

. ____ relate to this type of component ailure? X MGA a Are there any previous cor ie actions that appear to have been Inadequate? X MBA B Are there deficiencies Incurrent operating X

_ __ procedures or practices? -

10 Are there deficiencies incurrent maintenance practcesibehavlors or In X MGA associated training? .

11 Does there appear to be any deficiency In x MGA

____ ____ design?

12 Isthere any concem with quality of parts, x shippLng, or handli?

13 Does this appear to be a Malntenance x MOA Rule Functional Failure?

Evraluiatlon Completed By-.. Oae Ullchael Annett 4M6/2004

.D

ROOT CAUSE ANALYSIS REPORT CR Numben CR-VTY- 2004.00955 Page 16 of 24 ENY EHMAN PERFORMANCE EVALUATION FORM CR No: IDispositloning Dept:

CR-VT- 200400955 I System Engineering 1Cause IMaintenance Department.

Applicable HU TRAPs:

[l Thne Pressure ED Vague Guidance [3 Physical Environment O Dlstra0ionrtnterruptlon [3 First ShltLate ShIfl [3 Mental Stress O MultIple Tasks [3 Peer Pressure o Overconfidence 0 ChangefM-Normnal Description of Inappropriate Act(s): Assoc Process/Prog/Org Issue(s): __NA The stuffing box bore for V2-840 was honed The work activity was not controlled by (or otherwise machined) such that the work order step text or procedure and the Inside diameter was not In conformance significance of the stuffing box clearance with the vendor specified tolerances. tolerances was not recognized. A change was made wtthout recognizing that there was a change or without understanding the significance of the change.

Worker Behaviors:

l Procedure Use/Adherence [3 Seff-Checidng [3 Fitness for Duty o Placekeeping 3 Peer Checking O3 Turnover/Handof.

1 Spoken Communication El Knwledge 0 Problem Solving Method 19Written Communkcaton a kKl Supervisor Behaviors:

IManagement Spoken Communicaion El Task Aocation E3 Pe-Job Briet 0Written Communication I Clear Expectations Behaviors:

El Communications 0 Change Management a3 IProcess/Promrm Scheduling/Sequencing 1 Resource Allocation _ Conservative Decislon Mk Q ClearExpectaffons.

Issues:

El ErgonomicHuman Factors a3 Housekeeping 0 ProcedureN~k Pkg Qualty O Environmental Conditlons E Equpment Labeling 0 Training Di _ tloner.

J Date Completed:l Mlchael Annett _ _47102004

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 20040095S Page 17 of 24 Attachment E Sketch of Suffing Box Configuration which allowed galling

,5e, 0#t£q 17A/

1s VER1ICAL PACKING CR-VTY-2004-00955 DImenslons 80B MI UP 10/02 (Dspe toStem 0: 12 o'cdcm - In Contact Specertn Bonnet - 0.015.

(spamer to Bonnet B ForbIfe m0.004" GAP (2) Folower to Stem o 0.0065 (centered)

Spryto Bonnetn "p6de - 0.011- ()Foower to Bonnet a 0.0135 (centered)

OSpacer to Stem n X.011l Foltower Recess to Bonnet - 0.094

©BonnettoStem - 0.045 OspaCel t Doet - In COAM

ROOT CAUSE ANALYSIS REPORT CR Number. CR.VTYY 200400955 Page 18 of 24 Attachment F Sketch of As-found stem condition

,5u, pigte IV4

P~t)V, 154 4- 1.498n-1.500" 17 5/8 - MARK @ 12 O'clock position (Top of Valve Stem as Installed)

M}UJOR GALL 13 7/16-FOLI LOWER 53/4n -6 1/8" 9

- 4 5/8"

-BONNET 518" onf

'~ SPACER

- BACKSEAT

-1.75"

• - PILOT PLUG CR-VTY-2004-0955 V2-801 STEM GALLING CONFIGURATION

ROOT CAUSE ANALYSIS REPORT CR Number: CR-VTY. 2004-00955 Page 19 of 24 Attachment G Summary of Operating Experience Telephone Conversations:

Operating Experience concerning en1V packing and stem galling.

04/13/04 Susquehanna 1/2 - Steve Kartchner (Main Steam System Engineer) ph 570-542-3196 These plants use Atwood-Morrill MSMVs. They have 2 inch stems. The packing configuration is a 3 inch stellite bushing having 0.018 Inch diametrical clearance to the stem, 2.5 inch carbon bushing, 5 ring square ring graphite stack (Argo packing), 1 inch carbon bushing and then the follower. The bushings have standard Argo clearances which are:

1) for <2 inch stems 0.010+/- .003 both inside (stem) and outside (stffing box)

2) for>= 2 inch stems 0.010+/- .003 for stem andO.015+-.003 forthestuffingbox They had a recent problem with stem gaffing caused by gross actuator misalignment- the stems were side loaded enough to contact the stellite bushing and became galled. The actuators were not properly aligned upon assembly. The carbon bushings were pressed to the side of the stuffing box and had to be chipped out. They are generally pleased with the performance of this packing arrangement and intend to continue with it. Steve believed that the close tolerance stellite bushing was required by design (GE Design Spec) and the basis for it was stem buckling when the stem was loaded forcing the plug into the seat.

Brian Willy- AOV Engineer and packing expert- Susquehanna (ph x3809). The best ARGO product to adss unevenness in the bottom of the stuffing box is a 1116 thick graphite gasket-washer. It is soft enough to accommodate stress risers and small enough so that if it gets wasted away the live-load Is not totally lost. He validated the ARGO carbon bushing sing 04/13/04 Duame Arnold, Eric Sorenson, ph 319-851-7469 Project Engineer MSIVs. They have the same Rockwell-Edwards y-globe valves as VY. Their'stems are 1.5 inch. The dimensiona info is the same as VY.

Stems are 17-4 PH and design size is 1A92 to 1.500 OD, stuffing box is 2.285-2.290 ID, junk ring spacer is 2282-2284 OD and 1.510-1.512 ID. Backseat bore is 1.590-1.595 ID (was opened up), packing follower is 1.510-1.512 ID.

They installed new bonnets in 1990 and the stuffing box depth was shallower due to having to accommodae only one set of packing. Theirjunk ring is only 0.4 inch in height They pack the valves with Oarlock 0700 square graphite rings. If the bonnet rabbit fitup becomes loose beyond design tolerance they weld buildup the bonnet to restore tolerances.

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY- 2004-00955 Page 20 o24 04113104 Brunswick, Boyd Stanley, Main Steam System Engineer, ph 910-457-2386.

Brunswick has Rockwell-Edwards y globe MSIVs with 2 inch stems. The packing is 3.04 OD and 2.00 ID, they ise a5 ring squae shapedgraphite dieformed stack precompressed to 170ft-lb and then left at 40ft-lb final torque. Stem design is 1.999-2.000 OD, stuffing box is 3.040- 3.035 ID. The carbon steel junk ring is 3.032-3.034 OD and 2.20-2.10 I). The metal junk ring section is the same I shape as the VY design (having the inside lower leg missing). They have experienced stem galling which was caused by a metal washer locited in the bottom of the staffing box below the junk ring.

The gland follower ID is 2.012-2.020.

04114/04 Nine Mile Point, Steve Heimewitz, Valve Engineering (ph 315-349-4702)

Nine Mile Point Unit 2 utilizes Rockwell-Edwards Y-globe valves for their MSIVs.

They utilize the Edwards provided cadmium plated Carbon Steel packing spacer and have had no stem scoring Issues. Steve did not have the specific stiffing box clearance information, but believed that the clearances for ENVY were quite tight compared to NW. NMP uses ARGO packing in their MSIV.

ROOT CAUSE ANALYSIS REPORT CR Number. CR-VTY. 2004-00955 Page 21 of 24 Attachment H Mechanistic Root Cause Evaluation A dimensional analysis was performed on the V2-80B valve packing subcomponents Including the design specifications for the subcomponents. Field dimensional data for valves V2-86C and V2-86D was included in the analysis for comparison value.

Valve packing subcomponents as-found dimensional data is summarized below.

Packing Packing Spacer Stuffng Box ED Backseat Bore Packing Stem OD Spacer OD ID Gland ED nw= sys ID _ _ _ __2 -L Design 1.510- 2.282- 2.284 2.285- 2.290 1.590 1.510- 1.498-Dim1do 1.512 1.512 1.500 V2-80B 1.514 2.285 top 2.299 1.590 AL 1.517 1.501 2.283 bottom 1.533 AP V2-86C 1.515 2.284 top 2.292 top 1.590 1.516 1.497-2.286 bottom 2.290 bottom 1.499 V2-86D 1.516 2.278 top 2.290 top 1.548 AF 1.5125 1A97 2.284bottom 2.289 bottom 1.5905 AL .

V2-80A Not 2.282 top 2.290 top 1.525 AF 1516 1.4995 Taken 2.284 bottom 2.287 bottom 1.592 AL V2-86A Not Not Taken 2.289 top 1.592 AL 1.527 1.500 Taken 2.288 bottom AL Peforming a diiensional evaluation of the V2-80B packing subcomponents in the as-left condition from the October 2002 repacking, the following is evident- see attached drawing of MSIV stuffing box-Attachment E.

The BOB stem material is 174 PH stainless steel whose hardness value is approximately Rockwell C 34-36. The packing spacer is carbon steel A10S-cadium platedL The hardness of the packing spacer (A 15) is a imately HB 187 corresponding to a Rockwell B of 91 which is softer thin the stem. The silver cadmium plating is relatively soft compared to te stem. Ihe packing follower is made of ASTM A331 OR4140HT which is as hard or harder than the stem depending on the specific heat treatment.

During ihe first 6-8 strokes of the valve following the repack, the stem was rubbing off the cadmium coating and the carbon steel spacer and thus, when observed by the VY Valve Engineer at 20% power during plant startup, no galling was Indicated. The 80B valve was stoked for Tech Spec Surveilaces each quarter and stem scoring noted during a Oct 2003 shutdown. The rubbing and friction between the carbon steel spacer and the stainless stern resulte in an acculation omaterial on the stem. This material builtup in size and by sticking to the stem was transported to the follower and when it exceeded ffie radial clearance (.O05inch) was stopped by the follower. The hardness of tihe carbon steel was increased by work hardening until it resulted in the observed score

ROOT CAUSE ANALYSIS REPORT CR Number. CR.VTY- 200400955 Page 22 of 24 marks on the stem The scaring accumulated additional material frm the stem into the ball of material being held by the follower on the upward stroke and tearing through the spacer on the down stroke. The material from the gall finally wedged between the follower and stem causing the follower to cock grabbing the opposite side of the stem and stopping stem movement. Evaluation of tew stem marldnglgalling of the BOB MSWIV stem (see attached drawing- Attachment F) reveals that the gall runs for the approximate distance of the valve stroke (12 inches). From interviews it was established that the score mark was at the 12 o'clock position - top of the stem as instalied in the valve.

This evaluation concludes that the SOB stem galling was initiated by the interation of the packing spacer contacting the valve stem during stroking of the valve due to the ID dimension of the stuffing box being oversize out of specification.