05000289/LER-2005-001
| Docket Number | |
| Event date: | |
|---|---|
| Report date: | |
| 2892005001R01 - NRC Website | |
EVENT DESCRIPTION
Plant Conditions before the event:
Babcock & Wilcox — Pressurized Water Reactor — 2568 MWth Core Power Date/Time: August 20, 2003/approximately 1600 hours0.0185 days <br />0.444 hours <br />0.00265 weeks <br />6.088e-4 months <br /> Power Level: 100% Mode: 1, Power Operations On February 3, 2005, it was discovered that a condition, which existed from August 20, 2003, until September 29, 2004, should have been reported under the following criteria: 1) a condition which was prohibited by the plant's Technical Specifications (TS), (10 CFR 50.73 (a)(2)(i)(B)), and 2) a condition that could have prevented the fulfillment of the safety function related to Post Accident Control Room habitability, 10 CFR 50.73 (a)(2)(v).
The condition involved Control Building Ventilation Return Air Fan "B" (AH-E-19B11/1/FAN)). AH-E-19B is a vane-axial fan, model number 48-21-900AP, manufactured by Howden Buffalo. An evaluation of past operability concluded that the Control Room Emergency Filtration System would not have been able to meet it's mission time of 30-days during several time intervals within the above specified time frame, when its redundant fan was not available. There were no structures, systems, or components that were inoperable at the start of the event and contributed to the event.
The Control Room Emergency Ventilation System has two trains, both of which are required to be operable by TS 3.1.15. With one train inoperable, unit operation can continue for 7-days. If the inoperable train is not returned to operable status within seven days, the unit is to be shutdown within 48 hrs. With both trains inoperable, unit shutdown is required within 48 hrs. Included in the operability of the Control Room Emergency Ventilation System is the requirement for the Control Room Emergency Ventilation Supply Fans (AH-E-18 fans) to develop a flow rate of 40,000 +/- 4,000 cfm through the filter banks. The AH-E-19 fans are not specifically required by the TS, but are required to support the flow requirements of the AH-E-18 fans. The AH-E-19 fans are 100% capacity and either fan will met the suction flow requirements of the AH-E-18A or AH-E-18B fans. Both sets of fans are vane-axial fans. This type fan has the motor connected directly to the fan and both the fan and motor are located in-line inside the ventilation duct. There were seven instances, for a total time interval of approximately eleven days, when AH-E-19B was the only fan available to provide suction flow to AH-E-18A/B.
On December 12, 2001, AH-E-19B was identified as being noisy by Operations.� The source of the noise could not be identified at this time. The fan was considered operable by Operations, but the noise required investigation and evaluation. The investigation and evaluation was routine work that followed the normal schedule process.
In February 2002, the Component Maintenance Optimization (CMO) Group collected diagnostic vibration data on AH-E-19B. The fan was not instrumented with normal vibration collection points, since vibration data was not normally taken on vane-axial fans. CMO collected vibration data on the motor connection box and the external ductwork. Since there was no base line data, CMO utilized general guidance provided in Maintenance Procedure E-1, 'Vibration Monitoring for Rotating Equipment", to analyzes the data. CMO concluded that the vibration data was below the alert level, and deferred additional troubleshooting to the Air Handling Equipment Inspection, Preventive Maintenance (PM) task.
The PM on AH-E-19B was completed on April 19, 2002. The PM task done in accordance with Maintenance Procedure E-16, "Air Handling Equipment Inspection", did not specify performing a visual inspection of the fan blades and hub for cracks, because this required coordination to remove the fan from service and entry into a TS time clock. The visual inspection steps of the work order that performed the PM were marked as "not applicable", since inspection of the hub required entry into the ventilation duct. If the visual inspection of the hub and blades had been performed, it is unlikely that the hub cracks would have been found, since the cracks were located where the hub bolted to the motor shaft. An inspection of this bolted connection requires removal of the hub cover. Visual inspections, as part of the PM Program, did not require disassembly. The noise issue was not resolved during the performance of the PM.
On January 18, 2003, Operations issued another work order to investigate and repair the noise at AH-E-19B. The fan was determined to remain operable in its current condition by Operations. The job order was sent to CMO for investigation and no work was done on the fan for approximately seven months.
On August 19, 2003, Operations identified the fan as noticeably noisier. CMO conducted troubleshooting, but did not utilize the process described in MA-AA-716-004 "Conduct of Troubleshooting". CMO attempted to reduce the noise by greasing both motor bearings. The sonic and ultrasonic indications on AH-E-19B did not change when the grease was applied. CMO Group concluded that the bearings were the beyond the repair stage and bearing failure was indicated. Additional thermography and ultrasonic data was obtained to support or refute motor bearing failure as the source of the noise. The thermography data when compared to AH-E-19A was consistent thereby indicating no adverse effects due to overheating. The ultrasonic data was inconclusive in part due to limited base line data. The vibration data did not indicate any frequencies that were indicative of a bearing failure. Based on the data, CMO concluded that the source of the noise could not be positively identified. CMO recommended motor bearing replacement and documented the findings in the work order. The information in the two work orders was combined into the original work order and the second work order was rejected.
CMO communicated the results of the troubleshooting to System Engineering. The System Engineer and CMO were unaware of any other failure modes that could be causing the noisy condition. In fact, motor bearing failures fit perfectly within their mental model since past fan failures were due to motor bearings. Instead of pursuing further troubleshooting, Engineering and CMO agreed to use motor bearing replacement as the vehicle for entry into the fan internals. Their intent was to evaluate other failure mechanisms during motor bearing replacement and fan disassembly. Neither of these intentions was documented. From this point forward, a motor bearing defect was incorrectly considered the failure mode of AH-E-19B, and it was not questioned by Engineering, Operations, or CMO. Had the conduct of troubleshooting process, per procedure MA-AA-716-004, been utilized, System Engineering and CMO would have been prompted to pursue additional troubleshooting, to review operating experience, and to consult a vendor and corporate component expert. Hub cracking would likely have been identified as an additional vane-axial fan failure mode if these industry tools were utilized.
On August 20, 2003, Operations placed AH-E-19B on "Non-Preferred Use" based on recommendations from CMO and Engineering. Operations did not perform an operability determination, when AH-E-19B was listed as "Non-Preferred". Nor did they request or identify the need for an Operability Evaluation or an Operational Technical Decision Making paper in accordance with LS-AA-105, "Operability Determinations," or OP-AA-106 101-1006, "Operational and Technical Decision Making Process," respectively. However, some discussion on operability did occur based on the following entry into the work order. "Experience has shown that bearings with identified defects will typically last five to six months before failure. With no identifiable bearing defects and given the fact that the bearing were greased, a high confidence exists whereby this unit will be able to perform its function for a specified mission time of 30 days.
On April 2, 2004, Condition Report (CR) 212805 was generated in response to NRC discussions regarding the "non-preferred use" status of the AH-E-19B fan. The NRC resident specifically questioned the amount of time a safety-related piece of equipment should remain on "non-preferred use." Procedure OP-AA-108-105, "Equipment Deficiency and Documentation", does not have a procedural limit for time equipment can remain on "non preferred use" status. The work order for the replacement of AH-E-19B required Engineering and CMO concurrence prior to extending the work. Contrary to the request, the motor replacement work order was rescheduled from 0350 to 0429 without documented Engineering and CMO concurrence. CR 212805 was to provide the documented assessment of new schedule date with respect to equipment performance.
In the Review of CR 212805, CMO provided a more detailed basis for why AH-E-19B was capable of meeting its specified mission time of 30-days. Because the basis assumed a bearing failure mode, CMO documented the information supporting bearing defects as the AH-E-19B failure mode. The Balance of Plant (BOP) Systems Manager recognized the lack of quantitative data supporting bearing failure due to the unreliable spectral vibration data. As a result, he concluded that the source of elevated noise could not be positively identified through the limited diagnostic capabilities available and initiated an action for CMO to perform additional troubleshooting.
However, the BOP Systems Manager chose to aggressively pursue motor replacement rather than additional troubleshooting. Had the troubleshooting been performed in accordance with procedure MA-AA-716-004 "Conduct of Troubleshooting," complex troubleshooting would have been pursued and the cracked hub would have been identified as a possible failure mode.
The AH-E-19B motor replacement task was rescheduled four times due to manpower and parts concerns. The motor/fan unit was replaced on September 29, 2004. During replacement, AH-E-19B hub cracking was identified.
The past operability review documented the as-found condition. CMO and Engineering recognized the hub cracking as the source of AH-E-19B vibration and noise. The past operability review determined that AH-E-19B had been inoperable since being placed on "Non-Preferred Use" status, in that it would not have been capable of performing its 30 day mission time if required.
CAUSE OF EVENT
The root cause of the event was Engineering, Maintenance, and Operations did not use the correct processes to ensure adequate rigor was applied to the troubleshooting, operability determination, and decision-making regarding component health. This prevented timely identification of the inoperable condition.
Engineering and CMO supplemented inconclusive vibration results and Tech Spec surveillances with inadequate evaluation to support operability. Engineering and CMO relied solely on personal experience when providing critical information to Operations, rather than utilizing the multiple industry tools to learn of other vane-axial fan failure modes. Internal and external operating experience, vendor input, and corporate component expertise were not utilized. Had the listed industry tools been incorporated into the evaluation, or the Exelon processes properly utilized, the degraded condition would have been more rigorously evaluated and the inoperable condition would have been identified prior to component disassembly on September 29, 2004.
ANALYSIS / SAFETY SIGNIFICANCE
The plant condition evaluated for reportability involves several instances during the time period (August 2003, to September 2004) where AH-E-19B would not have been expected to support 30-days of continuous operations.
During the instances where AH-E-19B was the only available Control Building Ventilation Return Fan, the requirements for operability for the Control Room Emergency Filtration System were not met. During these instances, the limiting condition of operation (LCO) for TS 3.15.1, were not met. AH-E-19A was operable throughout this time period except for several short periods.
The Control Building Ventilation System is designed to continuously maintain the control building ambient conditions within the desired limits of temperature, humidity, and radiation. A special or infrequent operating mode for this system is its emergency mode. In the emergency mode of operation associated with a high radiation signal from the Control Room, the system normal operating supply fans (AH-E-17A and 17B) are shutdown and the system emergency recirculation fans (AH-E-18A and 18B) are manually started to recirculate air from elevations 322', 338', and 355' through the emergency filter banks (AH-F-3A and 3B). The Control Building Ventilation Return fans (AH-E-19A and 19B) return air from these elevations to the suction of the supply fans.
The emergency system (AH-E-18's) is required by the Technical Specification to maintain a specified flow rate through the filter banks. The AH-E-19's are not specifically identified in the Technical Specifications. However, the TMI-1 accident analysis uses specific Control Building Ventilation System airflow rates through the filter banks to calculate the dose values to the Control Room Operators. Therefore, the AH-E-19 fans are considered as required supporting components to the AH-E-I8AIB supply fans. It has not been demonstrated that the AH-E-18 fans can develop the required flow rate without an AH-E-19 fan operating.
AH-E-19A and 19B are 100% capacity fans, thus only one is required to provide the necessary flow to fulfill the suction requirements of the supply fans. AH-E-19B was identified as noisy on December 12, 2001 and was placed in non-preferred use status on August 20, 2003. During the December 12, 2001, through August 20, 2003 time period the fan was run alternately with AH-E-19A. AH-E-19B ran approximately 50% of the time during this period and therefore met its operational requirements in this time frame. It has been judged that the fan would have met its mission time requirement during this time frame based on the amount of time the fan actually operated. The fan ran for approximately nine months. While 30-days of continuous operation was not completed, actual run time provided adequate demonstration that the fan could have met its mission time.
For the period after AH-E-19B was placed in non-preferred status (August 2003), until it was replaced in September 2004, a review identified 7 instances when AH-E-19A was out of service and AH-E-19B was relied on.
The total time duration of these 7 instances was 11 days, 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />, and 15 minutes. AH-E-19A was operable for all other time periods. AH-E-19B functioned continuously during the periods the "A" Train was out of service.
During the non-preferred status period, AH-E-19B was also run a minimum of 10-hours a month for routine surveillance testing.
During a loss of off-site power (LOOP) accident one of the AH-E-19's would be available. Each fan is powered from a separate Emergency Power Bus via an Emergency Diesel Generator. The Station Blackout Diesel can be manually started and loaded on either Emergency Power Bus on loss of the associated Emergency Diesel Generator. The lower tiered 480 V Emergency Buses also have the ability to be cross-tied, making AH-E-19A or AH-E-19B available post LOOP.
The safety significance of the isolated instances where only AH-E-19B was available from August 20, 2003 through September 29, 2004 is minimal. The total time when AH-E-19B was the only fan available to meet the required 30-day mission time was 11 days, 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />, and 15 minutes. The time period AH-E-19B was the only return fan available was approximately 3% of the time. AH-E-19A was available for 97% of the time the "B" fan could not be depended upon to meet its mission time. During the time the "A" fan was not available, the "B" fan would have operated for a minimum of ten hours and continued until it failed. Also, AH-E-18 fans were available through out the time period and would have operated as designed. The AH-E-18 fans may not have reached the flow rates required to be operable, but they would have provided a significant amount of protection at a reduced flow rate.
During the August 20, 2003, to September 2004, time period, the Control Building Emergency Filtration system was not called upon to perform any emergency functions. Thus, there were no actual safety consequences, and this event did not adversely affect the health and safety of the public or the employees.
The impact of the reduced flow rate through the filters in a post-accident scenario could impact the habitability of the Control Room. For the instances when AH-E-19B was the only return fan available and it failed to operate, the post-accident dose value to the Control Room Operator may have exceeded the 5 rem TEDE maximum for the duration of the accident (reference 10 CFR 50.67).
Establish and communicate clear expectations for entry into the Troubleshooting, Operability Evaluation, Non- Preferred Use, and Operational and Technical Decision Making processes for Engineering, Maintenance, and Operations personnel.
Establish and implement a mechanism for periodic reinforcement of expectations established in above Corrective Action. This action will include training needs analyses so that the expectations and mechanism for periodic enforcement are included in the appropriate initial and continuing training programs
PREVIOUS OCCURENCES
There were two previous events that identified hub cracking on non-safety related vane-axial fans. Auxiliary & Fuel Handling Building Exhaust fan (AH-E-14A) was discovered to have a cracked hub in September of 2000 when the motor was replaced. The Control Building First Floor Exhaust fan (AH-E-20A) was discovered to have a cracked hub in March 2003 when it was replaced due to a noisy condition. The corrective actions associated with these previous occurrences were too limited in scope and assigned too low a priority to identify the AH-E 19B hub crack in a timely manner.
ADDITIONAL INFORMATION
TMI Unit 1 has four safety-related vane-axial fans: the Control Building 2nd Floor Booster fans(AH-E-95A/B), and the Control Building Return Air fans(AH-E-19A/B). AH-E-95A/B are not susceptible to the failure mechanism of hub cracking because the hub is of a different design. The AH-E-19B fan was replaced with a different design hub that reduces its susceptibility to cracking. The AH-E-19A fan is scheduled to be replaced with the different design hub in May 2005. Preventative maintenance tasks will be revised via the Corrective Action Process to ensure performance of visual inspections of the AH-E-19A/B fan hubs to identify hub cracking in its incipient stages.
- Energy Industry Identification System (EIIS), System Identification (SI) and Component Function Identification (CFI) Codes are included in brackets, [SI/CFI] where applicable, as required by 10 CFR 50.73 (b)(2)(ii)(F).