ML20059C784

Intervenor Exhibit I-MFP-91,consisting of 930212 Rept Ncr DC1-92-MM-N021, Snubber at Pipe Support 1-171SL Damaged

ML20059C784
Person / Time
Site:	Diablo Canyon
Issue date:	08/19/1993
From:	AFFILIATION NOT ASSIGNED
To:
References
OLA-2-I-MFP-091, OLA-2-I-MFP-91, NUDOCS 9401060086
Download: ML20059C784 (32)

Text

- 2 75 23-6dA-E 9

e I-MFP4/ Mff { g 7 3 NCR DCl-9 J -1 February 12, 1993 1 Rev. 00 gh m "

7 MANAGEMENT

SUMMARY

1 On May 14, 1992, with Unit 1 in Mode 1 (Power Operation) at 100% power, during a walkdown of components on the unit 1 pipe rack, a snubber at pipe support 1-171SL was found )

damaged. l Investigation concluded that the snubber had locked in place I during normal plant operation, and the compressive thermal i loading that occurred as a result of a subsequent plant trip caused the snubber to buckle. The rack and the shroud were bent approximately 30 degrees and the annular stroke position indicator was cracked. The snubber is an Anchor Darling 501L with a 10" (nominal) stroke. It was determined that the failure of an internal part (verge whcal) had caused the snubber to lock in place. An evaluation of the failed verge wheel conducted by TES determined that the failure was caused by stress corrosion cracking resulting from the excessive hoop stresses generated by an out-of-tolerance pinion hole.

Further investigation and evaluations made by NECS Engineering determined that the line was operable. The snubber has been replaced.

Root cause:

The out-of-tolerance condition which resulted in an undersized pinion hole created high hoop stresses in the verge wheel. The verge wheel material (440C SS) is I susceptible to stress corrosion cracking when combined with the outdoor marine environment (i.e., salt water) and high tensile stresses.

i Corrective actions to prevent recurrence:

During 1R5 and 2RS, all Anchor Darling snubbers located out of doors will be removed and functionally tested to verify past operability. By 2R6, these snubbers will be replaced with PSA snubbers more suited for the service conditions.

The replacement PSA snubbers will be fitted with protective boots to preclude any potential for effects from the outside environment.

The TRG reconvened on February 12, 1993, reviewed and commented on the write up and signed off this NCR.

92NCRWP\92MMN021.PGD Page 1 of 30 9401060086 930819 PDR C ADOCK 05000275 PDR

4 4

NCR DC1-92-MM-N021 Rev. no i February 12, 1993 NCR DC1-92-MM-N021 SNUBBER AT PIPE SUPPORT 1-171SL DAMAGED '

I. Plant Conditions Unit 1 was in Mode 1 (Power Operation) at 100% power.

II. Descrintion of Event A. Summary: '

On May 14, 1992, with Unit 1 in Mode al at 100 percent power, during a PG&E structural inspection walkdown of components on the Unit 1 pipe rack, a snubber at pipe support 1-171SL on a main feedwater flow control bypass line for Steam Generator (SG) 1-1 was found damaged. ,

B.

Background:

t Technical Specification (TS) 3.7.7.1 requires that all snubbers shall be operable. The only snubbers excluded from this requirement are those installed on non safety-related systems, and then only if their failure, or failure of the system on which  !

they are installed, would have no adverse effect on any safety-related system.

This TS requirement is applicable in Modes 1, 2 (Startup) , .3 (Hot Standby), 4 (Hot Shutdown), 5 (Cold Shutdown) and 6 (Refueling) for snubbers located on systems required to be operable in those Modes.

TS 3.7.7.1 requires that with one or more snubbers '

inoperable on any system, within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> replace or restore the inoperable snubber (s) to operaale status and perform an engineering evaluation on the attached component or declare the attached system inoperable and follow the appropriate action statement for that system.

C. Event

Description:

l On May 14, 1992, with Unit 1 in Mode 1 at 100%

power, during a walkdown of components on the unit 1 pipe rack, a snubber at pipe support 1-171SL was found damaged.

• 92NCRWP\92MMN021.PGD Page 2 of 30 t

A A= - - en4+a, - Jm.--4 > +JJa> .4,-.' > a M l. a -au A.e s W m.2 a=.-a6a-@ .muAid4-.. e==ea.42 m4.,A.;.a.-4= + , - . -

aa,h- 'h.-

J -Ja.mm E-J.-u-.-E =c-E--->=a.- .h *.- >- a w

e N

1 I

1 J

a ,

l 4

e W

Y P,'('* {/ Q M[!)',,I89$Y CC'Id'$$'O*E

-?$

_[(f]j;(f.A_ '.n'tw h* t::l si r emi e u

. ..vo.c.i,me .c-y-

_.-  ?

,Wmomm mo n M , _%- - -

W f

'2 , h 6 ,V . a

. . i I ()..

1  !

e3 i

' %g '

1

(

b$}

4 g

l.

4

)

]

f i

R 9

3 I

i

, - . - . w. , - , _ ..,.._,m. _

,,_m.., ,,,.. ~ y,_,_,.. , ,_ .,. , . , , y. y m,,,,,,,%,.ng...-3.-,..,,w,.+g

._--. . . - . . . . . - , . _ - - . . , . - -- ~ _ _ . _ . - - .

6 NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 Investigation concluded that the snubber had locked in place during normal plant operation. Subsequent compressive thermal loading that occurred during system cooldown as a result of a plant trip caused the snubber to buckle. The rack and the shroud were bent approximately 30 degrees and the annular stroke position indicator was cracked. The snubber  ;

is an Anchor Darling (A/D) 501L with a 10" (nominal) stroke. It was determined that the failure of an internal part (verge wheel) had caused the snubber to lock in place. An evaluation of the failed verge wheel conducted by TES determined that the failure was due to stress corrosion cracking (SCC). In order for this condition to accur three factors must be present.  !

1. Material. The verge wheel is made from 440C stainless steel, heat treated to h.gh strength level. This material is susceptible to stress corrosion cracking (SCC), when exposed to a l contaminated environment and high tensile [

stress.

2. Environment. The snubber was located outdoors, in the coastal atmosphere which is humid and contains chloride salts. Water entered the <

snubber, as evidenced by the rust found on some inside components which include the verge wheel. Chloride in solution (in salt water, for example) can cauce stress, corrosion l cracking in high strength steels. l i

3. Stress. The pinion hole in the failed whee) l was found to be smaller than the din ~nsion specified by Anchor Darling, and the diameter of the pinion was found to be at the maximum tolerance specified by Anchor Darling. This I

resulted in a large amount of interference which created a high hoop (tensile) stress in the wheel. The stresses in the' failed wheel were calculated to be 250% higher than the maximum stresses expected based on the manufacturer's specifications.

92NCRWP\92MMN021.PGD Page 3 of 3n

, - - - -, r- -m-,- -m e -, , ...-m-.-,-.- , , . . , , - ,.- - - . . - . , . , - - - ~ , , _ , . . ,- . , - - ,

- . -- .-. - _-__ -. . . . . _ _ - . _ . . ~ . - _ . - ._. . -

4 e

~

NCR DC1-92-MM-N021 Rev 00 February 12, 1993 It is concluded that the subsequent plant trip (April 1992) which resulted in a full thermal cycle of the feedwater line caused the buckling of the snubber. There is no damage to any other components in the area. This is a class 1 support on line K-16-5266-6.

Further investigations demonstrated that the line was operable. The snubber has been replaced.

Refer to further dic ssion in section IV.A below,

" Safety Analysis."

It has been concluded by PG&E and Anchor Darling (ref. 12) that the failure of this particular snubber was unique and an isolated case. The Anchor Darling quality control inspection program in place at the time of manufacture of the subject snubber allowed the potential for an out-of-tolerance condition. Although statistically i

unlikely, this snubber and others manufactured at the same point in time could potentially be susceptible to a similar failure if the three factors were to combine. Since all of the Anchor Darling snubbers located out of doors are of a similar, susceptible style, it was decided to replace all of the snubbers at DCPP with a PSA brand, which are more suitable for the service

conditions.

j D. Inoperable Structures, Components, or Systems that Contributed to the Event:

None.

't E. Dates and Approximate Times for Major Occurrences:

1. May 14, 1992; Event / discovery date.

Snubber at pipe support 1-171SL found  !

damaged.

l F. Other Systems or Secondary Functions Affected:

None.

4 i '

I l

4 92NCRWP\92MMN021.PGD Page 4 of 30 i l

l

_ _ _ . . __ _ _ _ _ ._ _ ___ .._ ~ ~ . . _ - _. _.. -..-._. . - - , _ . . . . . , _ . - - . _ _ _ _ _ , _, -

4 i

NCR DCl-92-MM-N021 Rev. 00 February 12, 1993 i G. Method of Discovery:

A snubber at pipe support 1-171SL was found damaged during a walkdown of components on the unit 1 pipe rack.

H. Operator Actions:

N/A I. Safety System Responses:

N/A III. Cause of the Event  !

A. Immediate Cause l Failure of the verge wheel caused the snubber to lock-up in an extended position. The cool down of the line caused the snubber to buckle during the  ;

thermal line movement. ,

B. Determination of Cause: l See attached Root cause Analysis.  ;

C. Root Cause:

The out-of-tolerance condition which resulted in an undersized pinion hole created high hoop stresses in the verge wheel. The verge wheel material (440C SS) is susceptible to stress corension cra. king when combined with the outdoor marine environment (i.e., salt water) and high tensile stresses.

D. Contributory Cause:

The material and environment contributed to the l failure of the snubber. The failed verge wheel in this snubber is made from 440C stainless steel, heat treated to high strength level. This material is susceptible to stress corrosion cracking (SCC),

when exposed to a moist environment containing contaminants. This snubber was located outdoors, in the coastal atmosphere which is humid and contains chloride salts. Water / moisture entered the snubber, as evidenced by the rust-found on some inside components which include the verge wheel.

92NCRWP\92MMN021.PGD Page 5 of 30

d

=

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 Chloride in solution (in salt water, for example) can cause stress corrosion cracking in high strength steels when subject to high tensile stresses.

Therefore, these two elements contributed to the failure of the verge wheel, when combined with the stress caused by the out-of-tolerance pinion hole.  ;

IV. Analysis of the Event A. Safety Analysis:

An analysis was performed to determine whether associated piping were permanently damaged due to this event. The governing load case was the thermal stress induced in the system assuming the snubber froze at the hot setting and inhibited the pipe from reaching the cold position during the last cooldown. The stress, strains, and loads calculated were not high enough to prevent the system from continuing to perform its design function. The locations of the high stress points were inspected by NDE and no indications of damage ,

were found. The system was demonstrated to have remained operable. The snubber has been replaced and the engineering evaluation satisfied the requirements of T.S. 3.7.7.1. Therefore, this event has in no way affected the health and safety of the public. .

B. Reportability:

1. Reviewed under QAP-15.B and determined to be non-conforming in accordance with section  !

2.1.8.

1

2. Reviewed under 10 CFR 50.72 and 10 CFR 50.73 per NUREG 1022 and determined not to be reportable based on past practices for operability. The system was demonstrated to have remained operable. The snubber has been replaced and an engineering evaluation was conducted. This evaluation satisfied the requirements of T.S. 3.7.7.1. However, as a I 1

prudent measure, a voluntary LER has been l

submitted to the NRC. This issue was discussed l with the DCPP managers and it was decided, based upon precedent, that the Technical 1 l 92NCRWP\92MMN021.PGD Page 6 of 30

. .. .- ~- _. . . . - . ~ . .- _ - - - - ~

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 Specification clock starts at the time of discovery of the damage, even if evidence indicates that the failure may have occurred at a point in time earlier than the 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> TS Action Statement time requirement. (see reference 5).

The report associated with this NCR is voluntary LER 1-92-016-00.

3. It has been determined that this problem will not require a 10 CFR Part 21 report by PG&E since no defects have been identified on uninstalled snubbers at DCPP.

4. This problem has been reported via INPO Nuclear Network entries Design Engineering (DE) 3830 and Maintenance Information (MI) 1.302.

5. Reviewed under 10 CFR 50.9 and determined not to be reportable under this section since this event does not have a significant implication for public health and safety or common defense and security.

6. Reviewed under the criteria of AP C-29 requiring the issue and approval of an OE and it has been determined that an=OE is not required since it is an isolated case. See '

voluntary LER 1-92-O'_6-00.

V. Corrective Actions A. Immediate Corrective Actions:

i The entire line was visually inspected, Subsequently, NECS Engineering performed a stress analysis of the line and determined that it was j operable. ISI personnel performed an NDE on some

'l sections of the line, and the damaged snubber was then replaced with a new one. All immediate actions were performed within the Limiting '

Condition for Operation of the applicable TS.

92NCRWP\92MMN021.PGD Page 7 of 30

,.-nr

_ _ _ . . _ . . . _ _ _ _ _ _ . . . _ . . . . . - _m ___ _ _ _ ,

1 NCR D01-92-MM-N021 Rev. 00 February 12, 1993 <

B. Investigative Actions: ,

1. Send the broken verge gear from the damaged '

snubber (1-171 SL) to TES for failure analysis.

TES to also evaluate if there was any previous flaw and estimate the load and failure mode.

RESPONSIBILITY: N. Jahangir DEPARTMENT: NECS OPEG Tracking AR: '72665E3, AE #01 STATUS: RETURN Final report from TES was issued on 8/18/92 (ref. 10).

2. TES to determine the broken verge gear material from damaged snubber 1-171 SL.

RESPONSIBILITY: S. J. Quaintance DEPARTMENT: Mechanical Maintenance Tracking AR: -A0266563, AE # 02 STATUS: RETURN

3. Contact Anchor Darling to determine if they had seen similar snubber problems in the past.

RESPONSIBILITY: N. Jahangir DEPARTMENT: NECS OPEG Tracking AR: A0266563, AE #03 STATUS: RETURN i

4. Perform an NPRDS search for similar snubber problems.

l RESPONSIBILITY: D. Christensen/B. Patton '

DEPARTMENT: OSRG/Rel.Eng l Tracking AR: A0266563, AE # 04 l STATUS: RETURN ,

1 l

5. Research system history regarding events that i could contribute to a snubber failure of this type.

RESPONSIBILITY: Micki Dare. 3 DEPARTMENT: NECS OPEG l Tracking AR: A0266563, AE # 05 STATUS: RETURN.

92NCRWP\92MMN021.PGD Page 8 of 30

NCR DC1-92-MM-N021 Rev. 00 ,

February 12, 1993 i

6. Based on upon the TES report determine if the postulated overextension of the snubber will have caused failure of the verge gear.

l RESPONSIBILITY: N. Jahangir DEPARTMENT: NECS OPEG l Tracking AR: A0266563, AE # 06  :

STATUS: RETURN .i

7. Research teardown reports on previous snubbers that failed prior to this failure, and since 1988, in that location. Provide information on previous history, testing and vendor reports.

RESPONSIBILITY: S. J. Quaintance DEPARTMENT: Mechanical Maintenance Tracking AR: A0266563, AE # 07

> STATUS: RETURN See AE # 9.

8. Determine where the damaged snubber (1-171 SL) came from.

RESPONSIBILITY: S. J. Quaintance DEPARTMENT: Mechanical Maintenance Tracking AR: A0266563, AE # 08 STATUS: RETURN

9. Research teardown reports for this snubber.

Provide information on previous history, testing and vendor reports.

RESPONSIBILITY: S. J. Quaintance ,

DEPARTMENT: Mechanical Maintenance Tracking AR: A0266563, AE # 09 STATUS: RETURN

10. Provide a list of locations of any 501 L snubbers deleted and subsequently installed.

State the time they were installed and their present location. Determine the deleted l snubbers that existed in the plant prior to this event.

i RESPONSIBILITY: Nozar Jahangir DEPARTMENT: NECS OPEG ,

Tracking AR: A0266563, AE # 10 STATUS: RETURN 92NCRWP\92MMN021.PGD Page 9 of 30 i 7 w v + . - - p .? --

.m --ee , _ .--pn- . - , - -yw--. ;e -

's

~

NCR DC1-92-MM-N021 Rev. 00 '

February 12, 1993

11. During 1R5 perform a full stroke test of all .

Anchor Darling 500 series snubbers located in the pipe rack.

RESPONSIBILITY: S. J. Quaintance l DEPARTMENT: Mechanical Maintenance '

Tracking AR: A0266563, AE # 11 STATUS: COMPLETE The TRG has determined that this AE is no loncer necessary because the stroke test will be performed after identification and removal of the snubbers (see AE # 13).

12. During 2R5 perform a full stroke test of all Anchor Darling 500 series snubbers located in the pipe rack.

RESPONSIBILITY: S. J. Quaintance DEPARTMENT: Mechanical Maintenance  ;

Tracking AR: A0266563, AE # 12 STATUS: RETURN.  !

NRC Commitment? Yes.  !

The TRG has determined that this AE is no loncer necessary because the stroke test will be performed after identification and removal of the snubbers (see AE # 13).

13. Make a list of all Anchor Dar, ling smell and medium size snubbers located in Unit 1 and 2 outdoor environment. Update the list to include all Anchor Darling large size snubbers in outdoor environment.

RESPONSIBILITY: Nozar Jahangir i DEPARTMENT: NECS Engineering Tracking AR: A0266563, AE # 13 STATUS: RETURN.

14. Look at other mechanical snubbers that might be susceptible to stress corrosion cracking.

RESPONSIBILITY: Nozar Jahangir DEPARTMENT: NECS Engineering Tracking AR: A0266563, AE # 14 STATUS: RETURN 92NCRWP\92MMN021.PGD Page 10 of 30

_.- --. ~ . .

l 4 l NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 l

15. TRG chairman to discuss with management j possible reportability of this event under part  :

21.

RESPONSIBILITY: R. Waltos.

DEPARTMENT: Mechanical Maintenance.

Tracking AR: A0266563, AE # 15 STATUS: RETURN.

16. Regulatory Compliance tc report this snubber i event via the INPO network.

RESPONSidILITY: P. Dahan.

DEPARTMENT: Regulatory Compliance. #

Tracking AR: A0266563, AE # 16 STATUS: COMPLETE.

17. Determine if the past Anchor Darlir.g Quality program could be a contributor to this event.

RESPONSIBILITY: C. Hartz.

DEPARTMENT: Quality Assurance.

Tracking AR: A0266563, AE # 21.

STATUS: RETURN.

C. Correc ve Actions to Prevent Recurrence:

Summary:

During 1R5 and 2RS all Anchor Darling snubbers located outdoors will be removed and a functional test conducted to verify past operability. Those removed snubbers will be replaced with PSA snubbers 3

as available.

During 1R5, all existing Anchor Darling snubbers ,

that cannot be replaced by a new PSA snubber will be overhauled, i.e., the interior surfaces will bc cleaned and greased, and the snubber will be sealed and then reinstalled. By the end of 1R6, all Anchor Darling snubbers will be replaced.

During 2RS, the same process will occur with the exception that the Anchor Darling snubbers for which there are no replacements will only be  !

d functionally tested. Shortly after 2RS, these snubbers will be replaced with the Unit at power in accordance with applicable Tech. Spec. actions. '

This plan has been discussed with NECS Engineering i

92NCRWP\92MMN021.PGD Page 11 of 30 1

- - . - . - = . . _ - . - _ ___ _ ___ _ _- _ _ _ _ _

- - -- - - . .~ - . .. -- . _ . - -

~ '

NCR DC1-92-MM-N021 Rev. nn February 12, 1993 and is acceptable based upon the fact that Anchor Darling concludes that functional testing would Identify those snubbers in which stress corrosion cracking has significantly progressed. The slow j process of stress corrosion cracking is not expected to progress to failure in the short time between the test and replacement of all snubbers. ,

All replacement PSA snubbers will be fitted with ,

protective boots to p. ;1ude any potential for '

effects from the outside environment. This action  :

is recommended by snubber manufacturers in general for all snubbers located out of doors.

Corrective actions: 4

1. During 1R5 remove all Anchor Darling snubbers which are located outdoors and perform a functional test. Install new PSA snubbers in locations where replacements are available.

Install boots and seal areas on the new ,

snubbers as identified by NECS Engineering.

NOTE: During the 09/02/92 TRG meeting, it was decided to replace the Anchor Darling snubbers with PSA snubbers, and to install rubber boots.

However, this replacement could not be accommodated at all locations. As a result, all existing Anchor Darling snubbers that  !

cannot be replaced by new PSA snubbers during 1R5 will be overhauled, a rubber boot installed, and sealed and then the snubber will be reinstalled. By the end of 1R6, all A... hor d

Darling snubbers will be replaced.

RESPONSIBILITY: S.J. Quaintance.

DEPARTMENT: Mechanical Maintenance.

Tracking AR: A0266563, AE # 17 Outage Related? Yes OUTAGE: (1RS)

OE Related? No NRC Commitment? Yes CMD Commitment? No STATUS: COMPLETE.

92NCRWP\92MMN021.PGD Page 12 of 30 l l

NCR DC1-92-MM-NO21 Rev. 00 February 12, 1993

2. During IR5, disassemble all Anchor Darling 5500 series snubbers, that are to be reused on the pipe rack, within the bounds of existing procedures. Clean the verge wheel, regrease and reassemble.

RESPONSIBILITY: S.J. Quaintance.

DEPARTMENT: Mechanical Maintenance.

Tracking AR: A0266563, AE # 26.

Outage Related? Yes OUTAGE:(1RS)

OE Related? No NRC Commitment? No CMD Commitment? No STATUS: COMPLETE.

3. NECS to identify the areas that need to be sealed on the new Anchor Darling replacement snubbers.

RESPONSIBILITY: N. Jahangir.

DEPARTMENT: NECS Engineering Tracking AR: A0266563, AE # 19.

Outage Related? Yes OUTAGE:(2RS)

OE Related? No NRC Commitment? No CMD Commitment? No STATUS: RETURN.

NOTE: Per the TRG held on 09/10/92, this actiori is no longer necessarv.

4. During 2R5 remove all Anchor Darling snubbers which are located outdoors an d perform a functional test. Install new PSA snubbers at locations where replacements are available, and Install boots on the new snubbers.

RESPONSIBILITY: S.J. Quaintance ECD: 04/30/93.

DEPARTMENT: Mechanical Maintenance.

Tracking AR: A0266563, AE # 18 Outage Related? Yes OUTAGE:(2R5)

OE Related? No NRC Commitment? Yes CMD Commitment? No 92NCRWP\92MMN021.PGD Page 13 of 30

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993

5. During 2RS, reinstall all the Anchor Darling l snubbers (removed under Corrective Action to Prevent Recurrence #4 above (AE#18)) that cannot be replaced by new PSA snubbers, i

RESPONSIBILITY: S.J. Quaintance ECD:04/30/93.

DEPARTMENT: Mechanical Maintenance.

, Tracking AR: A0266563, AE # 22.

Outage Related? Yes OUTAGE:(2RS) l OE Related? No NRC Commitment? Yes l CMD Commitment? No

.l

6. Submit an AT EWR to NECS Engineering to change l where possible the Anchor Darling snubbers to l PSA snubbers for Unit 2.

l RESPONSIBILITY: S.J. Quaintance. l DEPARTMENT: Mechanical Maintenance. l Tracking AR: A0266563, AE # 23. )

Outage.Related? No  !

OE Related? No l NRC Commitment? No CMD Commitment? No STATUS: COMPLETE.

AT-EWR AR is A0277553.

7. Facilitate a PSRC interpretation of Tech. Spec.

3/4.7.7 that provides basis / guidance for starting of the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> clock for replacement or evaluation and subsequent potential reportability of inoperable snubbers.

RESPONSIBILITY: P.G. Dahan.

DEPARTMENT: Regulatory Compliance.

Tracking AR: A0266563, AE # 24.

Outage Related? No OE Related? No NRC Commitment? No CMD Commitment? No STATUS: RETURN.

NOTE: Per the TRG of February 3, 1993, this action is no longer necessary.

1 l

j 92NCRWP\92MMN021.PGD Page 14 of 30 l

______.__..-__,._.......,_,__......,,._,..,_._--..__'I 4

. .. .- .- - = - . . - . . - - . - . . . . . ~ . . - - . _ - -

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993

8. In the event that the snubber replacement described in-CAPR 1 (AE #17) is not completed during 1RS, complete the replacement of all remaining Anchor Darling snubbers located outdoors with PSA snubbers and install boots on the new snubbers.  ;

RESPONSIBILITY: S. Quaintance ECD: 06/30/93.

DEPARTMENT: Mechanical Maintenance.

Tracking AR: A0266563, AE # 28. ,

Outage Related? No OE Related? No NRC Commitment? No  :

CMD Commitment? No

9. In the event that the snubber replacement described in CAPR 4 (AE #18) is not completed  :

during 2RS, complete the replacement of all Anchor Darling snubbers located outdoors with PSA snubbers and install boots on the new snubbers.

RESPONSIBILITY: S. Quaintance. ECD: 06/30/93. [

DEPARTMENT: Mechanical Maintenance.  ;

Tracking AR: A0266563, AE # 27.

Outage Related? No l OE Related? No t NRC Commitment? No CMD Commitment? No i

10. Prepare a voluntary LER on damaged snubber 1-171SL.

l RESPONSIBILITY: P.G. Dahan. '

DEPARTMENT: Regulatory Compliance.

I Tracking AR: A0266563, AE # 25 Outage Related? No OE Related? No NRC Commitment? No ,

CMD Commitment? No STATUS: COMPLETE

, 92NCRWP\92MMN021.PGD Page 15 of 30 G

- . - .- ., -- r - _ , . , . . - , - - -. - . . , . < - - . - ,r ,-e ,

. - . - _ . - ~~ .. . - . - _ _ . .- _-. . .- . - -

NCR DCl-92-MM-N021 Rev. ^^

February 12, 1993 D. Prudent Actions (not required for NCR closure)

1. Mechanical Maintenance to establish a periodic Preventive Maintenance (PM) program to assure the adequacy of the installed boots and sealed  ;

areas as required by CAPR's 1, 2, and 3. PM to  !

include all snubbers located outdoors as '

identified by NECS Engineering.

RESPONSIBILITY: v.

Quaintance ECD: 12/31/93.

2 DEPARTMENT: Mechanical Maintenance.

Tracking AR #: A0288825.

Outage Related? No OE Related? No NRC Commitment? No CMD Commitment? No i

2. Determine if revision of the applicable ISI procedure is required in light of the installation of boots on the outdoors snubbers.

This procedure may need to include precautions i to appropriately assure that the boots are left intact during In Service Inspection activities.

RESPONSIBILITY: D.A. Gonzalez ECD: 01/15/93.

DEPARTMENT: ISI/NUCLENG.

Tracking AR #: A0288825, AE #02 '

Outage Related? No OE Related? No NRC Commitment? No CMD Commitment? Yes

3. NECS Engineering to identify all snubber types that require the installation of a boot and the vehicle by which to implement the installation.

Please identify the design class of the boots.

RESPONSIBILITY: N. Jahangir ECD: 01/15/93.

DEPARTMENT: NECS Engineering. l Tracking AR #: A0288825, AE #01 Outage Related? No '

OE Related? No  !

NRC Commitment? No I CMD Commitment? No j 92NCRWP\92MMN021.PGD Page 16 of 30

i i

. I NCR DC1-92-MM-N021 Rev. 00 ,

February 12, 1993 VI. Additional Information ,

j A. Failed Components:

None.

i B. Previous Similar Events:

1. NCR DC2-88-MM-N120 I On September 24, 1988, snubber 412-82SL on SG blowdown line 1043 was discovered broken. This snubber had been removed for functional testing per STP M-78B. The root cause was due to failure to promptly and effectively implement the corrective actions contained in NCR DC2 MM-N048 to prevent water hammer during syster realignments. AP C-23 was revised to include a requirement for TRGs which identify procedure

, revision (s) as corrective actions (s) to determine if that procedure revision must be approved and issued before the next evolution or task or if that procedure revision would be '

only a programmatic enhancement. The Operations department was to determine what actions were necessary to prevent recurrence of water hammers when commencing the SG blowdown.

A shift briefing was held to emphasize '

prevention of water hammers. These corrective actions would not have prevented this event.

2. NCR DCO-89-OA-N011 Audits of suppliers by PG&E QA identified that Anchor Darling and some other suppliers of safety-related components and equipment were not implementing control over sub-suppliers or i

dedicating sub-supplier commercial parts to the extent appropriate for compliance with PG&E's interpretation of,10 CFR appendix B, ANSI N- ,

452.2.13 and guidance contained in EPRI/NCIG 07 l and NUMARC letter dated June 26, 1989. One or i more of the following weaknesses were typically noted in the supplier programs or the  ;

implementation thereof: (1) Reliance on documentation furnished by sub-suppliers whose programs do not comply with 10CFR50, Appendix B ,

and reliance on sub-suppliers programs, even though the sub-supplier does not subscribe to

92NCRWP\92MMN021.PGD Page 17 of 30

. - - . =. ~_ . . . .- - . ~- - . - .

1 NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 -

10CFR21; (2) Reliance on commercial-grade components without performance of all  !

inspections and testing necessary to dedicate l parts for safety-related application; (3)  !

Reliance on sub-supplier quality program  !

without adequate auditing to assure adequate sub-supplier quality program implementation.

PG&E QA, based on their audit results, identified these problems with several suppliers and in particular Anchor Darling.  ;

On 09/07/89, the PG&E QA department confirmed f that Anchor Darling would remain off the QSL list based on re-audit and further review.

The root cause was identified as lack of specific guidance and requirements in industry 1 and NRC guidelines on the subjects of .i procurement of non-ASME parts and dedication of l commercial grade parts.

The corrective actions to prevent recurrence included: (1) Previous QA audits found  !

deficient were to be reperformed. As  !

applicable, future supplier's audits were to i specifically address sub-supplier quality programs and commercial-grade dedication activities; (2) QA procedures, including QAP's, i QADP's and NEMP's, were to be reviewed and l revised, as appropriate,.to assure that clear  :

criteria are in place for determining if a supplier program and program implementation are acceptable for inclusion of the supplier its the QSL; (3) To provide additional support to the  !

resolution of this issue, PG&E was to continue l its active participation in industry groups. i i

These corrective actions could not have prevented this event since they address i l corrections of the OA audits. l l

3. NCR DC2-90-MM-N016 l i

l On March 12, 1990, during the performance of i STP M-78B a snubber at location 72-78SL failed l

, to meet that STP. The snubber was replaced, '

disassembled and found to be damaged due to overloading. The root cause was personnel j error, failure to communicate effectively 92NCRWP\92MMN021.PGD Page 18 of 30 i

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 between the aux operator and the control room.

The corrective actions to prevent recurrence included (1) an engineering evaluation on previous Unit 2 water hammer's event, (2) a '

revision of operation procedure OP D-2:V, (3) a review of the blowdown piping, (4) recommendation of possible solutions, (5) training regarding the event, (6) revision of operation procedures regarding proper system operation, and (7) issuance of an event summary report by operations. These corrective actions would not have prevented this event since the corrective actions address a different root .

I cause.

C. Operating Experience Review:

1. NPRDS:

None.

2. NRC Information Notices, Bulletins, Generic  ;

Letters:

A search of the Operating Experience Assessment database revealed the following:

None.

3. INPO SOERs and SERs:

A search of the Operating Experience Assessment

! database revealed the following:

None.

D. Trend Code:

Responsible department XX (Manufacturing /

l Supplier), and cause code C2 (Material / Equipment Deficiency, Manufacturing / Supplier).

E. Corrective Action Tracking:

1. The tracking action request is A0266563.

2. Are the corrective actions outage related?

Yes.

92NCRWP\92MMN021.PGD Page 19 of 30

.- . - - . - - _ . . .. . _ . ~ . .-- .-.. . . - . - . . .

NCR DCl-92-MM-N021 Rev. 0:

February 12, 1993 F. Footnotes and Special Comments:

None.

G.

References:

1. Initiating Action Request A0266177.

2. QE Q0009735.

3. Tracking Action Request A0266563.

4. Tech. Spec. 3.7.7.1.

1

5. Memo regarding " Actions to be taken following discovery of an inoperable snubber."

6. NCR DC2-88-MM-N120

7. NCR DC0-89-QA-N011

8. NCR DC2-90-MM-N016.

9. Memo from TES to NECS Engineering dated 08/18/92.

Subject:

Evaluation of 1-171 snubber failure.

10. NRC IE NOTICE 88-95 " Inadequate procurement requirements imposed by licensees on vendors.

11. Letter from Anchor Darling's QA manager dated 12/23/1988. '

12. Letter from Anchor Darling Engineering Manager, dated 1/14/93, and enclosed A/D Engineering Report No. ER-92196-01, Rev. 3.

13. Magnetic Particle Examination report.

H. TRG Meeting Minutes:

1. On May 19, 1992, the TRG convened and considered the following:

A review of the events leading to the discovery of the damaged snubber was made. Based on their review of the condition of the damaged snubber and in light of the technical specs, NECS 92NCRWP\92MMN021.PGD Page 20 of 30

i e

NCR DCl-92-MM-N021 Rev. 00 February 12, 1993 Engineering evaluation concluded that the line was capable of performing its design function.

It was established that the TRG was convened to  ;

respond to management concerns regarding repetition of a previous snubber failure.

Discussion on what happened this time to 1 determine if it relates to previous occurrences.

A piece of the damaged snubber was examined.

Discussion on the scaffolding which was in the immediate vicinity of the snubber. Concern was voiced about the snubber point of impact which is the feedwater line piping.

When the line is heated, the snubber is fully ,

extended. When the line cooled down the snubber buckled.

Discussed what we know about the previous failure.

When the snubber was inspected the collar was cracked. The snubber was extended to 9 5/16" +

8/16" = 9 13/16". Can be as low as 9 11/16".

The snubber may have been overextended by 3/16".

It was reasoned that when the gear cracked the snubber locked up and buckled. l l

Did not find anything to indicate when this .

event happened.

Looked at the feed line movement and compared #

it to the feed line movements from other analyses. No abnormality was noted. ,

Looked at snubbers during previous outages and no visual indications were noted.

I Clean break on the gear. No indication of fatigue.

l During transients on early events in previous outages other snubbers were found broken. This 92NCRWP\92MMN021.PGD Page 21 of 30 l l

.. - - . - - - - -. .- ._ ~ .. . . _ -

3- .

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 could be a potential root or contributory cause.

Gear to be sent to the metallurgical lab to

, determine the failure mechanism.

The immediate cause was stated as follows: i The failure of the verge wheel caused the snubber to lock up in its ful.'y extended position. The cool down of the line caused the

• snubber to buckle up during the thermal line movement.

It is assumed that this snubber was swapped in 1991. Need to do some research to determine where the snubber was before being installed in this location.

Immediate corrective action: i Looked at the line, analyzed it, looked at its new position, installed a new snubber. The '

, snubber presently in place is operable.

The root cause and corrective actions to I prevent recurrence have not been determined as -

yet.

i NECS did an analysis that looked at the worst j

case condition and determined that the line 1

would have performed as it was supposed to l during a seismic event. Investigative actions 1-j 12 were assigned. '

i j A prudent action was assigned to Mechanical Maintenance that reads as follows:

! Mechanical Maintenance to issue an AT AWR to 4

NECS to evaluate the current snubber -

configuration for potential enhancement.

l Another prudent action which has not yet been (

assigned, but will be considered at the next ,

TRG reads as follows:

Should there be a trip due to a transient, a

} walkdown of the snubbers should be performed to -

look for any problem.

2

[ 92NCRWP\92MMN021.PGD Page 22 of 30 i

d 4 _.

e NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 At the next TRG meeting determine if it is within the scope of this TRG to address this prudent action.

TRG TO RECONVENE ON 7/21/92 TO DETERMINE THE ROOT CAUBE AND CORRECTIVE ACTIONS,

2. On July 24, 1992, the TRG reconvened and considered the following: ,

Micki Dare, NECS OPEG did not find any contributor to the snubber failure.

Review of TES findings on the verge wheel indicated that the:

Material 4 s A-276 TY 440C Stainless Steel.

Certificate of Conformance certifies the heat treat to be OK.

Material to be quenched and tempered Stainless Steel. There was some Chlorides on the

? surface. The internal grease was analyzed for chlorides. There was some slight surface corrosion. The pinion and gear are pressed

) together. There is no sign of fatigue. i Indication of overload due to stress could be due to stress corrosion cracking.

There are some generic implications. The ,

vendor has never seen this type of failure.

These snubbers have been salt sprayed and dust tested. Three different sizes of snubbers.

There are 15 snubbers on the pipe rack. 6 out '

of the 15 snubbers have been tested under rated

! load. They were random tested every outage.

This snubber in question was brand new, and was never used. It sat in the warehouse for 7 years before being installed.

It was agreed that the verge wheel failure caused the snubber to lock-up.

The snubber user group should verify the vendor statements on this snubber failure.

92NCRWP\92MMN021.PGD Page 23 of 30

-,.= -

-NCR'DC1-92-MM-N021 Rev. OC February 12, 1993 Need to receive more information from TES. The l draft from Anchor Darling will stay in draft form until it is received from TES.

TES is trying to establish the threshold limits on the 40 KSI.

If it is determined that the grease is a contributor to this failure, other snubbers will be looked at.

Several new investigative actions were initiated. See AE 12 through 14.

TRG TO RECONVENE ON 08/20/92 TO REVIEW THE INVESTIGATIVE ACTIONS AND ESTABLISH THE ROOT CAUSE AND CORRECTIVE ACTIONS.

3. On August 20, 1992, the TRG reconvened and considered the following:

Review of the previous investigative actions.

The verge wheel was dimensionally inspected by TES and found to be outside the manufacturer's tolerance. The center hole dimension was .001" smaller that allowable. .

Anchor Darling could not come up with anything specific. They need three days to talk to their people.

Anchor Darling will do 100% tcsting of the grease for chloride, fluoride, and Di-sulfate.

Chevron said that there are chlorides in there grease and all the snubbers have this grease.

Although, it is believed that grease was not the reason for the failure.

The vendor had performed failure mechanic analyses on the verge wheel. The vendor performed some hoop stresses which would cause

, the gear to crack.

It was determined that the failure of the verge wheel was due to stress corrosion cracking. To have this condition three factors are needed which are: electrolyte, material and high 92NCRWP\92MMN021.PGD Page 24 of 30

NCR OC1-92-MM-N021 Rev. 00 February 12, 1993 i

stress. In this case, stress due out-of-tolerance condition was the major contributor.

3 It was decided to replace all this type of snubbers located outdoors on Unit 1 and Unit 2.

There are 19 small, medium and large snubbers of this type on Unit 1 and 11 on Unit 2, the majority of which are large snubbers.

The new snubbers to be installed will have '

boots and they will be sealed in the areas designated by NECS Engineering to prevent '

moisture to enter.

All the snubbers that will be replaced, will be full load functional tested to determine past ,

operability.

A list of snubbers to be tested during 1R5 will be provided.

Two new investigating actions were initiated.

See items B.15 and B. 16 of this NCR for more information.

Three corrective actions to prevent recurrence were initiated. See items C.1, C.2 and C.3 of this NCR for more information.

TRG TO RECONVENE ON TUESDAY AUGUST 25 AT 2:30 PM.

4. On September 2, 1992, the TRG rm.cnvened and considered the following:

Summary of the event. The investigation and evaluation conducted by TES shall be summarized in the description of the event.

Concurred that stress corrosion caused the failure of the verge wheel.

Agreed that the pinion hole in the failed wheel being smaller than the dimension specified by ,

Anchor Darling and the diameter of the pinion being at the maximum tolerance specified by A/D specifications resulted in a large amount of interference which created a high hoop stress in the wheel. This out-of-tolerance condition F

92NCRWP\92MMN021.PGD Page 25 of 30 i

. .. , , - . . - , , ,, m -

NCR DCl-92-MM-N021 Rev. 00 February 12, 1993 was caused by the dimensional inspection on those parts on a sampling bases MIL-STD-105D.

The vendor is to write a 10CFR part 21 report based on the lubricant used in the snubber which contains chlorides and molybdenum disulfide which in combination with water can cause stress corrosion cracking in high strength steel such as the 44CC stainless steel used in the failed verge wheel. '

It was agreed that this event is reportable under 10CFR Part 73 and therefore an LER will be submitted to the NRC.

An investigative action was initiated to QA to research if Anchor Darling was in the Qualified Supplier List (QSL) in 1985.

A corrective action to prevent recurrence was  !

initiated to Mechanical Maintenance to issue an '

AT EWR to NECS Engineering in order to replace where possible all the Anchor Darling snubbers with PSA snubbers.

TRG TO RECONVENE ON WEDNESDAY BEPTEMBER 9, 1992 AT 3:00 PM.  !

5. On September 10, 1992, the TRG reconvened and  !

considered the following:

After a long discussion of the reportability, it was decided that although this event is not c reportable under 10 CFR part 21 or 10 CFR 50 72 and 10 CFR 50 73, a voluntary LER will be submitted to the NRC.

It was decided that Regulatory Compliance will facilitate a PSRC interpretation of Tech. Spec.

3/4.7.7 that provides basis / guidance for starting the 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> clock for replacement or evaluation and subsequent potential reportability of inoperable snubbers.

This will create a precedent when inoperable snubbers are found, i There will be further talks to the vendor about reportability.

92NCRWP\92MMN021.PGD Page 26 of 30

,f

4 NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 It was agreed that the grease by itself could not cause a failure of the snubber.

It was pointed out that NCR DCO-89-QA-N011 has '

indicated problems with Anchor Darling controls l of sub-suppliers. Anchor Darling was qualified ,

and misclassified parts, they supplied.

One investigative action was issued as follows:

i

1. Item / 17. QA to investigate if the past  !

Anchor Darling Quality program could be a contributor to this event. l

2. In the Corrective Actions to Prevent Recurrence (items 1 and 2) AEs # 17 and 18 change the words " full stroke test" to

" functional test" and add a note to the effect that NECS Engineering will provide  :

the sealing instructions in the DCN. l Three new corrective Actions to Prevent Recurrence were initiated as follows:

1. Delete item # 3, AE # 19 and replace with the following:

During 2R5, Mechanical Maintenance to overhaul all Anchor Darling snubbers that cannot be replaced by new Anchor Darling or PSA snubbers and thoroughly clean and degrease all interior surfaces. Install '

boots and seals on all areac identified by NECS Engineering.

2. Mechanical Maintenance to submit an AT EWR  !

to NECS Engineering to replace where possible, the Anchor Darling snubbers to  !

PSA snubbers on Unit 2.

3. Regulatory Compliance to facilitate a PSRC f interpretation of Tech. Spec. 3/4.7.7 that  ;

provides basis / guidance for starting of 1 the 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> clock for replacement or i evaluation and subsequent potential  ;

reportability of inoperable snubbers.

A prudent action was initiated as follows:

1 92NCRWP\92MMN021.PGD Page 27 of 30 i

e

,-.r* ' ' - ' - -

. - . . - . - . . . . ~ ~.

4 NCR DCl-92-MM-N021 Rev. 00 February 12, 1993 Mechanical Maintenance to establish a periodic Preventive Maintenance (PM) program to assure the adequacy of the installed boots and sealed areas as required by CAPR's 1, 2, and 3.

This NCR shall be sent to PSRC before the next TRG.

TRG TO RECONVENE AFTER 1R5 REFUELING OUTAGE TO REVIEW THE FINAL WRIAn UP AND RESULTS OF THE INVESTIGATIVE ACTIONS.

6. On December 18, 1992, the TRG reconvened and considered the following:

Review of the write-up.

I Review of the corrective actionL.

Review of the Prudent action.

Two new prudent actions were established as follows.

1. ISI to determine if revision of the i applicable ISI procedure is required to '

consider the installation of boots on the outdoors snubbers. This procedure may need 3 to include precautions to appropriately  ;

assure that the boots are.left intact '

during In Service Inspection activities. l 1

2. NECS Engineering to identify all snubber types that require the installation of a l boot and the vehicle by which to implement j the installation. Please identify the j design class of the boots. i Two new corrective actions were established as f follows:

1. Mechanical Maintenance, during 1R6 to replace all snubbers located outdoors that i

could not be replaced during 1RS. 3

2. Mechanical Maintenance during 2R6 replace all snubbers located outdoors that cannot ,

not be replaced during 2R5. '

92NCRWP\92MMN021.PGD Page 28 of 30 1

I

- , -. - , - .- - ... - ~.

t NCR n"1-92-MM-NO21 Rev. 00 February 12, 1993 NCR CLOBURE ECD: 12/31/94 TRG TO RECONVENE ON THE THIRD WEEK OF JANUARY 1993 TO REVIEW THE ROOT CAUSE ANALYSIS.

7. On January 22, 1993, the TRG reconvened and considered the following:

During 2RS, A/D snubbers will be replaced with PSA snubbers as available. Instead of waiting until 2R6 to finish the rest, we may be able to replace them under the TS action allowed outage time after 2RS, as the remainder of the PSA snubbers are expected to have arrived within 1-2 months after the end of the outage. Removal and testing of the existing A/D snubbers will still have to be done during the outage, however. After the testing, the existing snubbers should not be susceptible to failure due to SCC for the short time until they are replaced with PSA snubbers. P The TRG reviewed and finalized the root cause analysis (attached). Final TRG comments on the writeup are due by Wed. 1/27/93, and the TRG will reconvene to sign the NCR (to be scheduled). Voting / signing members should include the required members and the representative from OPEG.

8. On February 3, 1993, the TRG reconvened to

. sign-off this NCR.

However, several items needed to be cleared before the NCR could be signed off.

The investigative and corrective actions were i reviewed and corrected to reflect the latest decisions by the TRG.

There was some discussions on the sealing of the snubbers. The PSA snubbers are not required to be sealed.

QA brought up the subject of reportability citing NUREG 1022. It was decided that a PSRC interpretation was not required based on the position letter presented by Regulatory Compliance. (See reference 5).

92NCRWP\92MMN021.PGD Page 29 of 30

NCR DC1-92-MM-N021 Rev. 00 February 12, 1993 IT WAS DECIDED THAT THE TRG NEEDED TO RECONVENE ONE MORE TIME, ON FEBRUARY 10, 1993 AT 10:00 AM FOR SIGN-OFF.

9. On February 12, 1993, the TRG reconvened, reviewed and commented on the write up and signed off this NCR. '

' THIS NCR WILL NOW BE BENT TO THE PSRC FOR APPROVAL AND BUBBEOUENTLY TO OA FOR CLOBURE.

I. Remarks: None.

J. Attachment (s):

, None.

i b

92NCRWP\92MMN021.PGD Page 30 of 30 i

6

NCR DCl-92-MM-NO21 Rev. 00 February 12, 1993 ROOT CAUSE ANALYSIS DC 1-92-MM-N021 Snubber at pipe support 1-171SL damaged.

Page 1 of 1 EFFECT CAUSE EVIDENCE BARRIER A snubber at pipe support 1- The snubber had locked The snubber was 171S1 was found damaged. in plice prior to a found buckled recent plant trip. during a ,

structural inspection walkdown.

WHY? The snubber had locked in The failure of an Verge wheel l place prior to a recent plant internal part (verge found trip. wheell- fmdmd during the disassembly of the snubber.

WHY? The failure of an An evaluation Evaluation internal part (verge wheel). determined that the made by TES failure of the failed verge wheel was caused and Anchor by the hoop stresses Darling (ref.

! generated bf an out-of A/D report tolerance pinion hole- dated '

1/14/93).

WHY? An evaluation determined The out-of-tolerance Metallurgical The out-of-tolerance condition that the failure of the failed condition which evaluation of the pinion hole (i.e., 3 verge wheel was caused by the [*n"e p nion hole Performed by undersized) created the high hoop stress generated by an created high hoop TES. tensile stress. Elimination of t out-of-tolerance pinion hole. stresses in the verge any of the three factors would whet L. The verge wheel prevent stress corrosion material (440c SS) is cracking.

susceptible to stress corrosion cracking when '

combined with the outdoor marine environment (i.e., salt water) and high tensile stresses.

Root Cause.

CC = Contributory Cause ( broken barrier) l

, ._ . . . . _ , - . _ _ _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _