05000247/LER-2004-002

LER-2004-002, Manual Reactor Trip Due to Decreasing 23 Steam Generator Level Caused by Feedwater Regulating Valve Closure Due to a De-energized Solenoid Operated Valve from Wiring Failure

Docket Number
Event date:	09-24-2004
Report date:	11-24-2004
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation
2472004002R00 - NRC Website

Note: The Energy Industry Identification System Codes are identified within brackets { }

DESCRIPTION OF EVENT

On September 24, 2004, at approximately 11:55 hours, while at 100% steady state reactor power, Operations manually tripped {JC} the reactor {RCT} as a result of decreasing 23 Main Feedwater (FW) {SJ} flow and 23 Steam Generator (SG) {S13} level. Prior to the event at 11:53 hours, a balance of plant (BOP) Control Room (CR) Operator observed decreasing 23 SG level and received an alarm for SG level control deviation. CR operators attempted manual control of 23 FW regulating valve FCV-437 and opened the low flow regulating valves for the 23 SG without success in restoring SG level. The level for the 23 SG continued to lower and as directed, the CR operator tripped the reactor at 15 percent SG level. All control rods fully inserted and all primary systems functioned properly. The plant was stabilized in hot standby with decay heat being removed by the main condenser.

There was no radiation release. Offsite power remained available therefore the emergency diesel generators did not start. The Auxiliary FW system automatically started as a result of a SG low level normally experienced on trips from full power (shrink effect). The 23 FW regulating valve FCV-437 Solenoid Operating Valve (SOV) FCV-SOV-437-E is a 125 VDC solenoid valve, Model No. HC8210C35, manufactured by ASCO. CR operators observed the rod bottom lights, Reactor Trip (RT) First Out Annunciator (Manual Trip). CR Operators entered procedure ES-0.1, secured the Main Boiler FW Pumps and subsequently transitioned to procedure POP- 3.2. The plant was stabilized in hot standby with decay heat being released to the main condenser via the steam dump valves 06. At 1320 hours0.0153 days <br />0.367 hours <br />0.00218 weeks <br />5.0226e-4 months <br />, a 4-hour non- emergency notification was made to the NRC for a reactor trip while critical under 10CFR50.72(b)(2)(iv)(B) and an AFW actuation under 10CFR50.72(b)(3)(iv)(A) (8- hour) (Incident Log No. 41066). Operations recorded the RT event in the corrective action program (CAP) as Condition Report CR-IP2-2004-04522. A post transient evaluation was performed on September 24, 2004. An investigation discovered that the FW regulating valve solenoid valve FCV-437-SOV-E had a failed electrical connection. An extent of condition investigation was performed for conduit/wiring connections to the remaining three FW regulating valve SOV's.

Engineering investigations were performed for all CAT-A Copes-Vulcan reverse acting AOV solenoids and a representative sample of the remaining non-reverse acting CAT-A AOV solenoid valves to verify proper configuration/orientation. The inspections were completed on November 6, 2004, and two discrepancies were identified (SOV-896A and SOV-896B). CR-IP2-2004-05743 recorded the condition and Work Orders were initiated. Operations determined there were no operability concerns with the conditions. An investigation into the cause of the FCV-437-SOV- E connection failure determined that during Unit 2 refueling outage cycle 15 (2R15) in October 2002, a design change package installed quick disconnects on Air Operated Valve (AOV) FCV-437 vent line. During the installation, the orientation of the SOV in that line was changed from its original design position (vertical) to a horizontal position without the required engineering analysis being performed. The FCV-437 actuator is a reverse acting type and therefore the actuator and the SOV move up and down when the valve is stroked lending itself to greater stresses when in the incorrect orientation.

Stress on the SOV was experienced on September 1, 2004, due to FW flow oscillations and flow perturbations that caused a thermal-hydraulic transient and FW pipe movement as a result of failed FW regulating valve FCV-427. After the RT for the September 1 event, a walk down discovered a broken conduit/condulet fastener for FCV-437-SOV-E associated with an adjacent AOV (FCV-437), and repairs performed. A CR recorded the broken conduit fastener, which was fixed immediately, but no degradation was noted for the SOV at that time. Following startup from the trip, two additional FW thermal-hydraulic transients/shutdowns occurred due to problems with FCV-427. On September 20, an inspection discovered the L-Tab of the FCV-437- SOV-E valve was pulled out of its housing. The condition was assessed and determined by the valve engineer to be degraded but still functional and repair should be deferred to the upcoming outage due to the trip risk for repair at power.

Subsequently, the conduit and its wiring pulled away from the SOV de-energizing it.

CAUSE OF EVENT

The cause of the manual reactor scram was decreasing 23 SG FW flow and level.

The cause of the decreasing 23 SG FW flow and level was a de-energized solenoid operated valve FCV-SOV-437-E as a result of a SOV wiring disconnection to the SOV coil which caused the SOV to fail open. The failed open FCV-SOV-437-E vented air off the air operated FW Regulating valve FCV-437 causing it to close and stop FW flow to SG 23. The direct cause of the wiring disconnection to the SOV was the failure of the housing L-Tab of FCV-SOV-437-E due to an improper change to the orientation of the SOV during a quick disconnect modification installation in 2002. The improper horizontal orientation of the SOV allowed the lateral forces from one of several thermal-hydraulic transients that preceded the failure, to negatively impact the integrity of the tab causing it to fail. There were two root causes (RC) identified. RC-1: The design modification and its associated work package for installation of the SOV quick disconnects did not contain sufficient detail to ensure the proper installation due to failure to follow Planning and Engineering procedures. Previous requirements of maintaining the SOV in the vertical orientation were not carried through from the original requirement. Only the impact of the air supply to the AOV was evaluated. RC-2:

The improper installation of quick disconnect modification to FCV-SOV-437-E was also due to inadequate verification of the accuracy of the change. Although the modification package was weak, a change to the orientation of the SOV from the as-found condition should have been checked and validated. The improper installation resulted in an orientation of the SOV that was not in accordance with the original intended position of the valve. The improper orientation of this SOV caused stresses to the structural components of the SOV that were greater and different than those seen by the vertically mounted valves. These stresses were sufficient to cause the L-Tab to be pulled outside of its housing and allowed it to be subjected to the movement of the pipe and actuator, now wholly unsupported, leading to the eventual failure of the L-Tab. A contributing cause was lack of attention to detail/questioning attitude in modification installation and review. During the installation of the SOV modification, the solenoid orientation was changed to a horizontal position from the as-found vertical position but was not questioned.

This new orientation was outside the original design for the component. This change to the orientation of the SOV was not recognized by the work planner, the installation workers, the contract supervisor overseeing the installation of the modification, the Quality Control inspector, or the design engineer verifying completion of the installation. A number of other additional enhancement corrective actions were taken as recorded in the CAP CR for this event.

CORRECTIVE ACTIONS

The following corrective actions have been or will be performed under the CAP to address the causes of this event and prevent recurrence.

1.Communicated to all site personnel the importance of maintaining a questioning attitude towards all work performed and discussed the event via a Red Memo and reset the Station Event Free Clock on September 27, 2004.

2.The FCV-437-SOV-E solenoid was replaced and returned to the proper vertical orientation and conduit/wiring properly connected. Repair was completed on September 25, 2004.

3.Engineering performed an extent of condition inspection of the FRV's following the Unit 2 trip. No other issues with tabs outside of the SOV housing were noted. The SOVs for the remaining FRV's showed no signs of metal fatigue or cracking. Four work orders were prepared to inspect and tighten any loose conduits. Action completed September 24, 2004.

4. IPEC Design Engineering (DE) performed a review of all open modifications where the work is not complete that could directly or indirectly affect the seismic evaluation, configuration or orientation of the solenoids associated with the FCV's. No modifications were found.

5.The current IPEC Design Engineering procedures will be reviewed and revised as necessary to ensure proper carry through of important design criteria for all Engineering Requests (ERs) and to ensure there's a requirement that all modification packages record the before and after orientation of all safety related components and equipment to ensure proper orientation is maintained unless otherwise specified by the Modification. Scheduled completion is March 1, 2005.

6.IPEC Planning, Scheduling and Outage (PS&O) procedures will be revised to include a requirement to ensure that all work package step lists record before and after orientation of all safety related components and equipment to ensure proper design orientation is maintained, unless otherwise specified by the Modification. Scheduled completion is January 3, 2005.

7.Project Managers and Engineers will be coached on the importance of field verifying Modification completion prior to signing off work packages as complete.

Scheduled completion is March 1, 2005.

8.A Supervisor tool kit was provided to Maintenance management for improved briefings containing a detailed discussion on verifying component orientation and maintaining a questioning attitude through use of proper Human Performance Tools and expectations on their use.

EVENT ANALYSIS

The event is reportable under 10CFR50.73(a)(2)(iv)(A). The licensee shall report any event or condition that resulted in manual or automatic actuation of any of the systems listed under 10CFR50.73(a)(2)(iv)(B). Systems to which the requirements of 10CFR50.73(a)(2)(iv)(A) apply includes the reactor protection system (RPS) including reactor scram or reactor trip, and AFWS. This event meets the reporting criteria because the RPS was actuated by manual trip and the AFWS actuated on low level due to steam generator level changes in response to the manual RT, which occurs after a RT from full power as a result of SG shrink.

PAST SIMILAR EVENTS

A review of the past two years of Licensee Event Reports (LERs) for events that involved a RT caused by FW flow transients identified LER-2004-001 dated October 25, 2004. This LER reported a manual reactor trip as a result of oscillating FW flow and SG level caused by a failed FW regulating valve FCV-427. The direct causes for these LERs are different however both have related causes of procedure deficiencies although the specific procedures involved are different.

SAFETY SIGNIFICANCE

This event had no effect on the health and safety of the public. There were no actual safety consequences for the event because the event was an uncomplicated reactor trip with no other transients or accidents. Required safety systems performed as designed when the manual RT was initiated. The AFWS actuation was an expected reaction as a result of decreasing SG water level due to the reduction of SG void fraction (shrink), which occurs after automatic RT/TT from full load.

There were no significant potential safety consequences of this event under reasonable and credible alternative conditions. The low SG level due to closure of the FCV as a result of the failure of the FCV SOV housing L-Tab and associated wiring/conduit is bounded by the worst case FW transient that could be a reasonable and credible alternate condition per the analyzed events described in FSAR Section 14.1.9, Loss of Normal Feedwater, and FSAR Section 14.1.10, Excessive Heat Removal due to a FW System Malfunction. The plant performed as expected and the event was bounded by the FSAR analysis. Redundant safety SG level instrumentation was available for a low SG level actuation which automatically initiates a RT and AFWS start providing an alternate source of FW.

For this event rod control was in automatic and the reactor scrammed immediately upon a manual reactor trip. RCS pressure remained below the set point for pressurizer PORV or code safety valve operation and above the set point for automatic safety injection actuation.