05000395/LER-2004-001

LER-2004-001, Reactor Trip Due to Valve Failure During Forced Shutdown

Docket Number
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown 10 CFR 50.73(a)(2)(ii)(A), Seriously Degraded 10 CFR 50.73(a)(2)(iv)(A), System Actuation
Initial Reporting
ENS 40628	10 CFR 50.72(b)(2)(iv)(B), RPS System Actuation, 10 CFR 50.72(b)(3)(iv)(A), System Actuation, 10 CFR 50.72(b)(2)(i), Tech Spec Required Shutdown, 10 CFR 50.72(b)(3)(ii)(A), Seriously Degraded
3952004001R00 - NRC Website

PLANT IDENTIFICATION

Westinghouse - Pressurized Water Reactor

EQUIPMENT IDENTIFICATION

Reactor Coolant Pump "C" Seal Injection Nozzle Feedwater Regulating Valve IFV-498

IDENTIFICATION OF EVENT

On March 30, 2004, while at power, VCSNS personnel were performing a reactor building inspection to identify the source of reactor coolant system unidentified leakage that was within Technical Specification (TS) limits. At 1129 hours0.0131 days <br />0.314 hours <br />0.00187 weeks <br />4.295845e-4 months <br />, a pressure boundary leak was identified at the seal injection line nozzle weld to reactor coolant pump "C". Pursuant to TS 3.4.6.2, Action a., VCSNS commenced a controlled reactor shutdown at 1410 hours0.0163 days <br />0.392 hours <br />0.00233 weeks <br />5.36505e-4 months <br /> on March 30, 2004.

During the shutdown, the main turbine experienced higher than normal vibration. At 1517 hours0.0176 days <br />0.421 hours <br />0.00251 weeks <br />5.772185e-4 months <br />, the turbine was manually tripped at approximately 43% reactor power. Subsequent to the manual turbine trip, feedwater regulating valve IFV-498 failed in the closed position while in automatic with a full open demand signal. The reactor automatically tripped at 1521 hours0.0176 days <br />0.423 hours <br />0.00251 weeks <br />5.787405e-4 months <br /> due to lo-lo level in the "C" steam generator with reactor power at approximately 9%.

This event was reported under Event Notification EN #40628 at 1738 hours0.0201 days <br />0.483 hours <br />0.00287 weeks <br />6.61309e-4 months <br /> on March 30, 2004 in accordance with 10CFR50.72(b)(2)(i), 50.72(b)(2)(iv)(B), 50.72(b)(3)(ii)(A), and 50.72(b)(3)(iv)(A).

EVENT DATE

03/30/04

REPORT DATE

05/27/04

CONDITIONS PRIOR TO EVENT

Mode 1 , 100% Power - 1129 hours0.0131 days <br />0.314 hours <br />0.00187 weeks <br />4.295845e-4 months <br /> through 1410 hours0.0163 days <br />0.392 hours <br />0.00233 weeks <br />5.36505e-4 months <br /> - Pressure Boundary LeakfTS Shutdown Initiation Mode 1, 9% Power - 1521 hours0.0176 days <br />0.423 hours <br />0.00251 weeks <br />5.787405e-4 months <br /> - Reactor Trip

DESCRIPTION OF EVENT

On March 30, 2004, while at power, VCSNS personnel were performing a reactor building inspection to identify the source of reactor coolant system unidentified leakage that was within Technical Specification (TS) limits. At 1129 hours0.0131 days <br />0.314 hours <br />0.00187 weeks <br />4.295845e-4 months <br />, a pressure boundary leak was identified at the seal injection line nozzle weld to reactor coolant pump "C" (RCP C). This nozzle weld was part of the repair performed for a previous leak at this location during refueling outage 14 (RF14) [Reference LER 2003-004]. Pursuant to TS 3.4.6.2, Action a., VCSNS commenced a controlled reactor shutdown at 1410 hours0.0163 days <br />0.392 hours <br />0.00233 weeks <br />5.36505e-4 months <br /> on March 30, 2004. The average rate of the shutdown was 1% per minute.

Main Turbine Bearing 5 vibration started to increase at approximately 70% reactor power and continued to rise as the shutdown progressed. At 40% reactor power, Bearing 5 vibration had increased to 10 mils, and the operators manually tripped the Main Turbine from the control room. A reactor trip was not required since reactor power was below the P-9 setpoint of 50%. The manual turbine trip occurred at 1517.

Because of the turbine trip, Steam Dump Banks 1 and 2 opened to the condenser and Control Bank D was automatically inserted from 120 steps to 60 steps. The Steam Generator Water Level Control System modulated the Feedwater Regulating Valves to maintain Steam Generator water level. Feedwater response was normal after the turbine trip, but IFV-498 appeared to go closed at 1518 as indicated by decreasing Feedwater flow to Steam Generator C. The lower limit switch for IFV-498 gave a closed indication to the plant computer shortly after 1519. Steam Generator C level decreased to the Low-Low Level Reactor Trip setpoint of 35%, and an automatic reactor trip occurred at 1521.

All control rods fully inserted and all safety systems responded normally. Both motor driven emergency feedwater pumps started as required. The plant stabilized in mode 3.

Condition Evaluation Report C-04-0879 was generated to address the RCP C seal injection line leakage, evaluate the cause, and develop effective corrective actions. This report was reclassified as a non-conforming condition and transferred to Non-Conformance Notice NCN 04-0879.

Condition Evaluation Report C-04-0884 was generated to address the failure of IFV-498 and provide the corrective actions necessary to preclude reoccurrence.

CAUSE OF EVENT

RCP C nozzle weld failure was caused by high-cycle, low amplitude vibration induced fatigue. The weld repair performed on the RCP C seal injection line during RF14 created a hot tear starter crack, which significantly reduced the time required for crack initiation, resulting in a rapid tearing of the weld once fatigue loading was applied. Fatigue loading on "C" seal injection nozzle was more severe than on the eight other RCP nozzles based on measurements of line vibratory displacements and natural frequencies. A RF 15 modification was planned to reduce the high cycle, low amplitude vibration induced fatigue loadings on the C RCP.

CAUSE OF EVENT (Cont'd) The cause of the failure of IFV-498 to operate properly is attributed to the pilot valve stem of the Bailey AV-1 positioner sticking due to fretting. The sticking of the stem prevented control air from opening the valve upon demand.

ANALYSIS OF EVENT

The "C" RCP seal injection nozzle exhibited a through wall leak on March 30, 2004 which resulted in a forced shutdown of the plant. This same nozzle had previously leaked in 1987 and again in 1994. These leaks were resolved by local weld repairs as allowed by the ASME Code to restore the welds. A third leak occurred in October of 2003 during RF 14 that resulted in the complete replacement of this nozzle and weldment.

SCE&G performed detailed inspections of the existing nozzle to support root cause evaluation. After all visual examinations, measurements and vibration testing had been completed, a metallurgical sample of the complete weld was removed for testing. The sample was transported to the hot cell metallurgical laboratory of BWXT Services, Inc. of Lynchburg, Virginia for analysis. The purpose of this testing was to determine the nature and physical characteristics of the failure.

The test results clearly indicate that the failure of the nozzle weld was entirely low-amplitude high cycle fatigue.

The weldment was found to be sound. Three factors appeared to dominate the fatigue process as follows:

1. The geometry at the weld root creates an inherent stress riser that intensifies the cyclic vibrational loads at the weld root.

2. The end of the nozzle impacted the bottom of the socket over a portion of the circumference. Also at this location the inner surface of the pipe was displaced inward towards the pipe axis. This action likely was a result of the shrinkage forces during welding coupled with the forces generated by the bottomed-out condition. This produces residual tensile forces at the root notch that opens the root notch and produces a crack starter (hot tear).

3. The crack starter effectively eliminated the crack initiation portion of fatigue life.

These three factors concurrent with the pump vibration produced sufficient stress to drive the crack completely through wall over a distance of about 0.44 inches in only 3 months. Fatigue propagation was entirely transgranular and there was no evidence of corrosion processes having influenced the fatigue propagation.

During a controlled shut down to repair the seal injection leakage on "C" Reactor Coolant Pump (RCP), the "C" Feed Water (FW) Regulating Valve, IFV-498, closed after a manual turbine trip and resulted in a reactor trip on Lo-Lo Steam Generator (SG) level. Initial troubleshooting by VCSNS Instrumentation & Control personnel did not reveal any failed components that lead to the FW Regulating Valve closure.

ANALYSIS OF EVENT (Cont'd) A Failure Modes Analysis (FMA) was initiated to determine the potential valve failure mechanisms, to obtain the necessary evidence to support/refute each potential mechanism and to provide the corrective actions necessary to preclude reoccurrence. The FMA narrowed the potential failure mechanisms to the Bailey AV-1 positioner.

A recent external Operating Experience (OE) Report 0E17276 identified industry issues with the pilot valves in these positioners. Quality issues in the manufacturing of the pilot valve were found to produce service induced fretting on the pilot valve stem. The fretting between the stem guides and the body can cause the stem to stick, which in turn can cause erratic valve operation including oscillations, valve failure to respond to demand and/or uncontrolled movement of the valve.

Inspection of the VCSNS pilot valve stem guides showed service induced fretting on the guides removed from the IFV-498 positioner. The cause of the failure is attributed to the pilot valve stem sticking due to fretting, thus preventing control air from opening the valve upon demand.

CORRECTIVE ACTIONS

The Station has taken the following corrective actions:

This section summarizes the corrective actions performed on the RCP C seal injection line under NCN 04-0879 and ECR 50547 to address the failure mechanisms identified and ensure safe restart.

Completed Corrective Actions

• Removed and replaced nozzle and weld, including all fillet, partial penetration, build-up, and heat affected zone weld material.

• Installed new nozzle design on "C" RCP seal injection nozzle using SCH160 pipe and a larger weld.

• Use of 1/8" pullback on new nozzle fit-up, followed by verification of pullback during and after welding using boroscope.

• PT of each weld pass (which exceeds the ASME code requirements) and an informational post-weld ultrasonic ID examination of the weld.

• Installed a new rigid pipe support on "C" seal injection line just upstream of nozzle flange that is supported by the RCP main flange. Pipe stress analysis predicts that the additional support will increase the resonant frequency and reduce vibration loading on the nozzle.

• Installed a new spring can pipe support on "C" seal injection line.

Post-modification measurement of "C" seal injection line natural frequency and vibration performed at Mode 3 during startup verified that displacements had been significantly reduced and natural frequency shifted away from pump driving frequency.

CORRECTIVE ACTIONS (Coned) Corrective Actions identified from the FMA for I FV-498 are as follows:

Completed Corrective Actions

• Installed new positioners on IFV-478, 488, 498 that include the quality improvements made at Bailey after January 30, 2004. Note these improvements are only in inspection/assembly not actual hardware improvements or changes.

• Inspected and replaced the pilot assemblies on the remaining valves that are equipped with a Bailey AV-1 positioner. Affected valves are as follows:

I FV02006-0-MB �I PV02000-0-MS HCV00936-0-SI I FV02016-0-MB I PV02010-0-MS LCV01003-0-WL I FV02106-0-MB �I PV02020-0-MS

• Inspected warehouse stock for all Bailey AV-1 model positioners for quality. Two problems were identified. The roller bearing in one positioner was found seized and one positioner had a leaky diaphragm.

• Established a PM to periodically inspect and/or replace the in service pilot valve assemblies.

Long Term Corrective Actions

• Develop a receiving and inspection procedure to inspect the Bailey AV-1 positioner to insure quality.

• Evaluate the need to replace the AV-1 positioner with a different brand.

• Establish a link between OE and the PM Program in the Equipment Reliability Improvement Program (ERIP) and evaluate component level assignments within Engineering.

VCS Engineering has concluded that the corrective actions to replace the Bailey AV-1 Positioners on the Feed Water Regulating Valves combined with the inspection and replacement of all other installed Bailey AV­ 1 pilot valve assemblies is adequate to operate the plant safely and reliably until RF-15.

PRIOR OCCURRENCES

There are no prior occurrences of failure for the feedwater regulating valves and/or their positioners.

This is a repetitive failure of the RCP seal injection nozzle. There were events in 1987 (LER 1987-013, June 25, 1987), 1994 (LER 1994-006, January 3, 1995), and 2003 (LER 2003-004, December 17, 2003) where the plant experienced pressure boundary leakage from the reactor coolant system from this same weld area.