05000266/LER-2002-003

LER-2002-003,

En Docket Number 121 Page 131
Event date:
Report date:
Initial Reporting
2662002003R01 - NRC Website

NAME

Charles Wm. Krause, Senior Regulatory Compliance Engineer TELEPHONE NUMBER (lrciude Area Code) (920)155-6809 COMPLETE ONE UNE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT (13)

CAUSE

r

SYSTEM COMPONENT

MAN Li-

REPORTABLE

TO EMI( t l

• - 1 . , 1

CAUSE SYSTEM COMPONENT

MANU- FA C'TURER

TO EPIC

.- -.

SUPPLEMENTAL REPORT EXPECTED (14) ,

SUBMISSION

DATE (15)

MONTH DAY YEAR

I YES (Byes, complete EXPECTED SUBMISSION DATE). [ X I NO ABSTRACT (Limit to 1400 spaces, i.e., approximately 15 sirvale-spaced typewritten Him) (10 On October 24, 2002, while conducting post maintenance testing of the P-38A motor driven auxiliary feeclwater pump, recirculation flow was slightly reduced. Initial Investigations discovered that the flow restricting orifice in the recirculation line from the pump discharge to the condensate storage tanks was partially restricted. Following engineering analyses and evaluations, NMC concluded on October 29, 2002, that the potential existed under specific conditions for a common mode failure mechanism to result in the plugging of the orifices in the recirculation lines for all four AFW pumps. This could result in the loss of safety function if the discharge from the AFW pumps were throttled and inadequate flow existed in the recirculation line to preclude pump failure due to overheating. An ENS notification was made on October 29, and interim compensatory measures consisting of crew briefings and information posting were completed which restored the AFW system to operable status. Additional actions included temporary procedure changes and training for the operators. Laboratory testing of the orifice design concluded that the installed recirculation orifices were susceptible to clogging. A plant modification was completed that replaced the orifices with an improved design that is not susceptible to clogging. During the interval that these orifices were installed, there was no plant event which resulted In an actual failure of the AFW system.

1 1 YEAR � SEQUENTIAL 1 REVISION ■

• 1111

• ■ �

Event Description:

At 10:27AM CST on October 29, 2002, with both units operating at full power, the auxiliary feed water (AFW) system PM for the Point Beach Nuclear Plant (PIMP), Units 1 and 2, was declared inoperable. The system was declared inoperable due to a concern for a single mechanism to result in the failure of all four of the AFW pumps [P] under specific conditions. The concern for this common mode failure was identified by NMC personnel while conducting an extent of condition evaluation for the reduced recirculation flow on the P-38A AFW pump. This reduced flow was observed during post maintenance surveillance testing of the P-38A motor driven AFW pump on October 24, 2002.

During that test, the recirculation line flow was observed to be approximately 64 gpm. Normal flow for this recirculation line is approximately 75 gpm. The minimum acceptable flow for this pump by the test procedure is 70 gpm. This discrepancy was documented and entered in the PBNP corrective action program (CAP 29906). After additional venting of the flow transmitter and recalibration of the flow instrument Fri, the P-38A AFW pump was started and tested again; however, the observed recirculation flow was essentially unchanged. Following that test run, the recirculation flow orifice jOR) was removed and inspected. The inspection revealed debris In the flow restricting orifice that appeared to be corrosion particles.

The AFW flow restricting orifices use a multi-stage anti-cavitation trim package installed in the body of a globe valve (FCVI to limit flow. This style of orifice or flow restrictor was installed in the AFW recirculation lines by plant modifications over the past few years to rectify piping problems related to cavitation at the old orifices. This type of flow restrictor uses very small channel shaped holes (approximately 15 mils by 90 mils) in each stage along with a torturous path to limit flow and prevent cavitation. After removal of the orifice internals, partial blockage was observed in 24 of the 54 holes in the outermost sleeve. No additional particles were found on any of the inner sleeves. Samples of the particles removed from the orifice were retained for analysis. A boroscope inspection of the recirculation piping at the orifice location revealed no evidence of debris. Following cleaning and reassembly, the orifice was reinstalled and the P-38A AFW pump retested using procedure IT-10, -Test of Electrically Driven Auxiliary Feed Pumps and Valves (Quarterly)." This test was successfully completed with an indicated recirculation flow of about 75 gpm.

Testing was also successfully completed on the other three AFW pumps to verify acceptable recirculation flow. All four pumps were back in service at 1206 on October 25, 2002.

During the next several days NMC personnel evaluated the implications of the orifice plugging event. An apparent cause evaluation was initiated with specific directions to assess and evaluate the potential extent of condition. An action plan was developed to identify the source of the foreign material found in the flow orifice and to determine what other testing or flushing would be required to assure that future plugging did not occur. At this time the operability of the AFW system was not in question because of the recent operating experience with the pumps, and the successful verification of recirculation flows on October 25, 2002.

As the investigations continued, questions developed concerning the operability of the AFW system while supplied by its safety related water supply, the service water system [1311. Although the service water supply is provided through a basket strainer [ST% it was recognized that the strainer mesh is 1/8 inch and the orifice channel holes are much finer.

These concerns culminated in a meeting early on October 29 at which NMC concluded that there was no longer a reasonable assurance that operation of the AFW system on its safety related suction source of service water would not result in potential AFW recirculation orifice clogging from service water debris. In a worst case scenario, NMC determined that it may be possible, although unlikely, for each of the four flow control orifices, each associated with one of the four AFW pumps, to restrict the flow through the associated recirculation line. Under such conditions, it is hypothesized that if the discharge valves for the AFW pumps are throttled, adequate flow may be unavailable through the recirculation line to avoid over heating the pumps, and pump damage could occur.

At 1027 on October 29, 2002, all four AFW pumps were declared inoperable. Both units entered TSAC 3.7.6.E and required action 3.7.5.E.1 which directs immediate action to restore an AFW system to operable status. Immediate corrective actions consisted of briefing the on-shift MIN of the potential consequences of restricted recirculation flow.

I 1 Point Beach Nuclear Plant Unit 1 � 05000266 DOCKET NUMBER FACILITY (2) NAME (1) OF 6 The operators were also directed to secure a running AFW pump if the pump discharge flows should be decreased to less than 50 gpm for the motor driven pumps or 75 gpm for the turbine driven pumps. These flow rates are substantially above the point at which pump damage could occur. Information tags were placed at the AFW pump flow Indicators tFlj on the main control boards to convey that information. These actions were completed at 1305 CST. At that time, with these administrative controls in place, NMC declared the AFWS operable. An incident investigation was initiated to collect and confirm the facts of this event description beginning with the discovery of the P-38A AFW pump degraded recirculation flow during post maintenance testing and concluding with the decision to declare the AFW system Inoperable.

In accordance with 10 CFR 50.72(b)(3)(v) an eight hour ENS notification (EN#39330) was made at 1711 CST on October 29, 2002, for: "Any event or condition that at the time of discovery could have prevented the fulfillment of the safety function of structures or systems that are needed to : (B) Remove residual heat ... or (0) Mitigate the consequences of an accident.

Cause:

A multi-discipline event resolution team was appointed to identify and resolve the issues associated with the discovery of this condition. Activities Included Initiation of a root cause evaluation (RCE) to determine the root and contributing causes for the postulated common mode failure that would render all AFW pump recirculation lines with restricted flow rates. The RCE concluded that this event had a direct root cause and an organizational root cause.

The direct root cause was the failure by design engineering to properly evaluate the potential for orifice plugging within the design process. Instead of revisiting the design for adequacy and evaluating the potential for plugging of the proposed orifices within the rigor of the design process, the 10 CFR 50.59 safety evaluation was revised to justify the proposed design.

The organizational root cause was less than adequate management oversight of the design modification process.

Specifically, management did not

• Assure that a formal technical evaluation be performed for the potential plugging issue rather than allowing it to be resolved via the safety evaluation.

• Assure that independent reviewers are provided with clear and concise expectations for the performance of reviews, are truly independent of the person performing the original work and are held accountable for an effective review.

• Assure that management oversight of critical activities includes second level and higher management personnel from engineering and from other departments.

Significant contributing causes included:

• The vendor information concerning its design to preclude orifice plugging was misapplied and not verified

• Information on the design functions of the Appendix R flow path and throttling of AFW flow during emergency operations was omitted from the safety evaluation

• Inadequate knowledge of AFW recirculation line design functions existed

• Inadequate Independent verification resulted from Inadequate program management

NRC FORM MA

� U.S. NUCLEAR REGULATORY COMMISSION (7-2001) FACILITY NAME (1) � DOCKET NUMBER (2) � LER NUMBER (6 � PAGE is) Point Beach Nuclear Plant Unit 1 05000266 �

Corrective Actions:

Immediate compensatory actions to restore the operability of the AFW system were identified in the Event Description Interim Corrective Actions included:

• On coming operating crews were briefed on this potential failure mode and the necessity to maintain adequate AFW pump forward flow or to secure the pumps.

• An operability determination was completed at 1850 on October 30 which concluded that the AFW pumps were operable but non-conforming because the AFW pump recirculation paths described in the FSAR may not be available under all operating transient conditions.

• Changes were made to affected AOPs, EOPs and other critical procedures to ensure that minimum forward flow is maintained through the AFW pumps or the pumps were secured. These changes were completed on November 7, 2002.

• Just-in-time operator training on the procedure changes to establish appropriate operator guidance for accident sequences of Interest was conducted for selected licensed operators prior to assuming the watch starting on November 9, 2002.

• An independent evaluation of the procedure changes, the adequacy of the briefings and training provided to the crews, and the effectiveness of the temporary information tags was completed.

• An objective evaluation of the decision to declare the AFW system operable after taking the immediate actions on October 29 was also completed.

• An assessment of flow restricting devices in use in safety related applications at PBNP was completed. None of the other safety related flow restricting devices at PBNP were found to be susceptible to small particle clogging.

• An evaluation of the AFW system to identify the sources of potential corrosion products was completed. (CA 26914)

• An engineering evaluation concerning the potential for plugging the restricting orifice with debris from the Service Water system was completed. Subsequently laboratory testing of the susceptibility of the orifices to plugging has also been completed. The results are addressed in the Safety Assessment (CA 26911)

• Revised training materials to ensure that they accurately describe the AFW recirculation line design functions during accident conditions. (CA 27370)

• Conducted a design and licensing basis review of the AFW System. (CA 27171)

• Completed design and installation of new recirculation line orifices with an aperture size of greater then 1/8 inch. These orifices have been tested and are not susceptible to clogging by service water debris. (CA 26918 and MR 02-039 A/BC/D.

Other corrective actions Identified in the completed RCE have been entered into and will be tracked to completion in the PBNP corrective action program.

Component and System Description:

The following component and system description comes from Section 10.2 of the PBNP FSAR.

The auxiliary feedwater system consists of two electric motor-driven pumps, two steam turbine-driven pumps, pump suction and discharge piping, and the controls and instrumentation necessary for operation of the system. Redundancy Is provided by utilizing two different pumping methods, two different sources of power for the pumps, and two sources of water supply to the pumps. The AFWS is categorized as seismic Class I and is designed to ensure that a single fault will not obstruct the system function.

One AFWS water source uses a steam turbine-driven pump for each unit with the steam capable of being supplied from either or both steam generators [SG]. Each turbine driven pump is capable of supplying 400 gpm of feedwater to its dedicated unit, or 200 gpm to each steam generator through normally throttled motor-operated discharge valves.

The feedwater flow rate from the turbine-driven auxiliary feedwater pump depends on the throttle position of these motor operated valves (MOVs). Each pump has an AOV controlled recirculation line back to the condensate storage tanks to ensure minimum flow to dissipate pump heat. The pump drive is a single-stage turbine, capable of quick starts from cold standby and is directly connected to the pump. The turbine is started by opening either one or both of the isolation valves between the turbine supply steam header and the main steam lines upstream of the main steam isolation valves. The turbine and pump are normally cooled by service water with an alternate source of cooling water from the firewater system [T].

The other AFW source is common to both units and uses two similar motor-driven pumps each capable of obtaining its electrical power from the plant emergency diesel generators. Each pump has a capacity of 200 gpm with one pump capable of supplying the "A' steam generator in either or both units through an AOV back-pressure control valve and normally closed MOVs and with the other pump capable of supplying the "13' steam generator in either or both units through an AOV back-pressure control valve and normally closed MOVs.

Both back-pressure control valves fail open when Instrument air to the valves is lost The discharge valves are provided with a backup nitrogen supply to provide pneumatic pressure in the event of a loss of instrument air. This backup supply assures that the discharge valves do not move to the full open position which, combined with low steam generator pressures, may cause the pump motor to trip on over-current due to high flow conditions. Each pump has an AOV controlled recirculation line back to the condensate storage tanks to ensure minimum flow to prevent hydraulic instabilities and dissipate pump heat The discharge headers also provide piping, valves, and tanks for chemical additions to any steam generator. The pump bearings are ring lubricated and bearing oil is cooled by service water.

The water supply source for the auxiliary feedwater system is redundant. The normal source is by gravity feed from two nominal capacity 45,000 gallon condensate storage tanks (CST) fiiq, while the safety-related supply is taken from the plant service water system whose pumps are powered from the diesel generators [EN if station power is lost Safety Assessment:

In early January 2003 a report was evaluated regarding the potential for plugging of the restricting orifices with debris from the Service Water (SW) system. That report, based on analytical results, concluded that the fluid velocities in the orifice holes and channels would be sufficiently high at pump flowrates to prevent any significant fouling of the orifices by the SW sediment. The report recommended that a series of laboratory tests be conducted to confirm the FACILITY NAME (1) � DOCKET NUMBER (2) Point Beach Nuclear Plant Unit 1 � 05000266 LER NUMBER (6, � REVISION NumseR � NUMBER PAGE 13) _ NRC FORM 3$GA � U.S. NUCLEAR REGULATORY COMMISSION (7-2001) NRC FORM MA (7-2001) FACILITY NAME (1) � DOCKET NUMBER (2) LER NUMBER (6 � PAGE (3) Point Beach Nuclear Plant Unit 1 05000266 _ predictions of the analytical model. Subsequently, in February 2003 full-scale testing of an orifice assembly was conducted using the types and quantities of debris found to reside in the SW system. This testing demonstrated a high likelihood that the recirculation flow control orifices installed in the AFW system recirculation lines would have plugged if the system was operated with suction from the SW system.

Given this high potential for plugging of the orifices, and the original procedural directions that could have permitted manual operator action to reduce pump forward flow below levels required to assure continued pump operability, it Is possible that one or more AFW pump failures would have occurred. As discussed In the Event Description and Corrective Actions sections of this report, immediate compensatory actions were taken to restore the operability of the AFW system. Those actions restored the AFW system to operability on October 29, 2002. Subsequently, the recirculation orifices have been replaced in all four pump recirculation lines with an improved design that is not susceptible to this common mode failure mechanism.

In June 2003, the NRC was provided a summary of our preliminary determination of the increase in core damage probability due to internal and seismic events. Since then, corrections were made to a few failure probabilities and to the system success criteria for one initiator. With work completed as of September 18, the change in core damage probability (CDP) for Unit 2 ( the bounding unit) 151.7E-05 for internal events and 9E-06 for seismic events. A significant effort was also expended evaluating the impact that this issue had on fire events. This analysis included scenario development and detailed fire models using state-of-the-art software and techniques. The current core damage probabilities from the detailed fire models for the most risk significant compartments were summed with the estimated values determined for the remaining compartments. The current total change in core damage probability results from fire events for Unit 2 due to this issue is between 1.1E-04 and 2.2E-04. Additional details concerning these assessments was provided to the NRC in a letter dated September 18, 2003. An independent review of these results is scheduled for October 2003.

NMC has determined that during the interval that these orifices were installed there was no plant event which resulted in an actual failure of the AFW system.

Although the administrative controls and corrective actions discussed in this event report restored the AFW system to operability shortly after the condition was identified, it is possible that this sequence of conditions and potential events could have impacted the operation of all four AFW pumps. The result could have been the complete loss of the AFW safety related function. Accordingly we have also identified this event as possible safety system functional failure.

Similar Occurrences:

A review of recent LERs (past two years) identified the following events which involved the potential for a loss of safety function involving the auxiliary feedvrater system:

LER NUMBER

266/2001-006-00 Title PRA Assessment of Auxiliary Feedwater System Reveals Procedural Inadequacy Related to Loss of Instrument Air Appendix R Requirements Not Satisfied for Unanalyzed Fire Induced Damage to the Auxiliary Feedwater System