05000296/LER-2010-004
| Browns Ferry Nuclear Plant | |
| Event date: | 12-26-2010 |
|---|---|
| Report date: | 02-24-2011 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| 2962010004R00 - NRC Website | |
I. PLANT CONDITION(S)
Browns Ferry Nuclear Plant (BFN) Unit 3 was at approximately 100 percent power (3458 MWT) at the onset of the event. Reactor power had been reduced to approximately 88 percent prior to initiation of the manual scram.
II. DESCRIPTION OF EVENT
A. Event:
On December 26, 2010, at 1615 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.145075e-4 months <br /> Central Standard Time (CST), an alarm for Main Turbine Vibration High 3-VA-47-15 was received in the Unit 3 control room on annunciator panel 3-XA-55-7B Window 32. Control room operators responded using Unit 3 Alarm Response Procedure (ARP) 3-ARP-9-7B. Exciter rotor inboard journal bearing vibration level indicated 8.0 mils and rising, and the outboard journal bearing indicated 5.5 mils and rising. At 1617 hours0.0187 days <br />0.449 hours <br />0.00267 weeks <br />6.152685e-4 months <br />, an Upper Power Runback was initiated per the ARP. It was noted that vibration levels initially lowered then continued rising. At 1620 hours0.0188 days <br />0.45 hours <br />0.00268 weeks <br />6.1641e-4 months <br />, control room operators initiated a manual reactor scram in accordance with Abnormal Operating Instruction (A01) 3-A0I-100-1, Reactor Scram. All plant systems responded as required to the manual scram signal. No Emergency Core Cooling System (ECCS) initiations occurred as a result of the manual scram signal.
Emergency Operating Instruction (E0I) 3-E0I-1, RPV Control, was entered because the reactor water level lowered below Level 3 (+2") as a result of the scram. Reactor water level was restored to the normal control band by the Reactor Feedwater Pumps (RFPs) [SK]. At 1647 hours0.0191 days <br />0.458 hours <br />0.00272 weeks <br />6.266835e-4 months <br />, Unit 3 operators exited 3-E0I-1. The expected Primary Containment Isolation System (PCIS) [JM] group 2 (Residual Heat Removal System (RHR) [BO]), group 3 (Reactor Water Cleanup System (RWCU) [CE]), group 6 (ventilation), and group 8 (Traversing Incore Probe System (TIP) [IG]) isolations were received due to reactor water level lowering below Level 3 (+2") with all systems isolating as required. There were no indications of any Main Steam Relief Valves (MSRVs) [SB] opening.
Tennessee Valley Authority is submitting this report in accordance with 10 CFR 50.73(a)(2)(iv)(A), as any event or condition that resulted in manual or automatic actuation of any of the systems listed in paragraph 10 CFR 50.73(a)(2)(iv)(B) (i.e., Reactor Protection System including reactor scram or trip, and general containment isolation signals affecting containment isolation valves in more than one system).
B. Inoperable Structures. Components. or Systems that Contributed to the Event:
None
C. Dates and Approximate Times of Maior Occurrences:
May 2008 During Unit 3 refueling outage 13, the exciter was disassembled, inspected, and reassembled. Post Unit 3 refueling outage 13, exciter inboard journal bearing vibration had a step change increase compared to the value prior to the refueling outage Problem Evaluation Report (PER 145841).
September 2010 December 26, 2010 at 1615 CST December 26, 2010 at 1620 CST December 26, 2010 at 1953 CST January 6, 2011 at 1656 CST The bearing vibration instrumentation had been replaced during the outage. Upon startup from the outage, the 'B' cooler became air bound and the differential temperature between coolers 'A' and 'B' spiked to approximately 50 degrees Fahrenheit. The outlet of the 'A' cooler was throttled in the closed direction, and the exciter air differential temperatures returned to normal. The outlet of the 'A' cooler was left in this throttled position until the event in December of 2010.
Unit 3 exciter air cooler temperatures began to diverge. A review of data indicated that the 'A' cooler was air bound during the summer of 2010. As the river temperature decreased in the fall of 2010, the 'B' cooler followed the river temperature profile while the 'A' cooler did not. With the 'A' balancing valve throttled and with changes in river temperatures, the 'A' cooler continued to be air bound resulting in differential temperature development between coolers 'A' and 'B' as the river temperature decreased.
An alarm for Main Turbine Vibration High control room on annunciator panel 3-)GA-55-7B Window 32.
After an Upper Power Runback reduced reactor power to approximately 88 percent, operations personnel initiated a manual reactor scram due to high vibrations on generator exciter inboard and outboard journal bearings.
Operations personnel made a non-emergency phone call to the NRC in accordance with 10 CFR 50.72(b)(2)(iv)(B) and 10 CFR 50.72(b)(3)(iv)(A).
Operations personnel performed Main Generator Synchronization and closed Main Generator Breaker.
D. pther Systems or Secondary Functions Affected None E. Method of Discovery The exciter rotor journal bearing high vibration levels were identified by an alarm for Main Turbine Vibration High 3-VA-47-15 on annunciator panel 3-XA-55-7B Window 32.
F. Operator Actions Operations personnel responded to the alarm per 3-ARP-9-7B and initiated a manual reactor scram in accordance with 3-A0I-100-1, Reactor Scram.
G. ,safety System Resnonses All plant systems responded as required to the manual scram signal. No ECCS initiations occurred as a result of the manual scram signal. Reactor water level lowered below Level 3 (+2") as a result of the scram and was recovered to the normal water level band by the RFPs.
The expected PCIS groups 2, 3, 6, and 8 isolations were received due to reactor water level lowering below Level '3 (+2") with all systems isolating as required. There were no indications of MSRVs opening. All control rods fully inserted. The plant was in a normal post-scram electrical alignment. Decay heat was being removed through the turbine bypass valves to the main condenser.
III. CAUSE OF THE EVENT
A. immediate Cause The immediate cause for this reportable condition was high vibrations on generator exciter inboard and outboard journal bearings that led to a manual reactor trip. , .
B. Root Cause The root cause was inadequate procedural guidance for monitoring the exciter air cooling system and prescribing mitigation actions to be taken based on differential temperature limits.
Specific instructions did not exist for the operations and engineering personnel responsible for maintaining system parameters within acceptable limits.
C. Contributina Factors 1. System monitoring did not identify the exciter air temperature imbalance prior to the event. The role of system monitoring and trending is to set in motion the mitigating steps necessary to prevent adverse effects on plant equipment.
2. Flow blockage (i.e., outlet valve to the 'A' cooler throttled 80% closed) and air binding (no capability or process to vent air from coolers) were contributors to this event. With limited capability or process to remove air from the exciter coolers, it is inevitable that air pockets will develop in the coolers during operation. The air pockets effectively reduce the performance of the cooler(s) which can lead to unbalanced air temperatures.
3. A clearance issue (light rub) between the exciter rotor and bearing oil/air deflector shield was identified in 2008. The Work Order (WO) to inspect and clear the rub was deleted from the spring 2010 refueling outage scope based on the fact that vibration levels throughout the operating cycle were not above established limits.
IV. ANALYSIS OF THE EVENT
During this event, high vibration on the generator exciter inboard and outboard journal bearings was caused by an air temperature imbalance within the exciter. The air temperature imbalance caused the exciter frame to "twist" by thermal growth. The movement of the exciter frame combined with a pre-existing condition (i.e., light rub identified in 2008) at the bearings resulted in a rub (i.e, a stationary component just touching the rotating shaft) which resulted in high exciter journal bearing vibrations.
Following the manual scram, the outlet valve of the 'A' cooler was found to be in the throttled position. An attempt was made to open the valve; however, the valve was stuck in its as-found position and would not move. A decision was made to replace the valve and, upon removal, the valve was found to have been throttled approximately 80 percent closed. This condition was determined to have existed since May of 2008. Upon startup from the Unit 3 refueling outage 13 in May of 2008, the 'B' cooler appeared to have been air bound (air trapped inside the cooler) based on subsequent increasing exit air temperatures. The difference between the 'A' and 'B' coolers exit air temperatures peaked at 50 degrees Fahrenheit. To balance the air cooler exit temperatures, the outlet valve of the 'A' cooler was throttled. Throttling of the outlet valve of the 'A' cooler reduced the amount of raw cooling water flow through the 'A' cooler and increased the amount of cooling water flow through the 'B' cooler. The extra flow through the 'B' cooler assisted in pushing the air out of the cooler. This manipulation was successful as the differential temperature returned to within the normal range. There was no process to evaluate the long term effects of leaving this valve throttled nor was there any configuration control as system 024 valve line-ups are not performed unless work is performed on the system Having this valve in the throttled position was considered a significant contributor to the development of the unbalanced air temperature within the exciter coolers.
When BFN has experienced air entrapment in the exciter coolers following refueling outages, the removal of the trapped air from the coolers was accomplished by opening the temperature control valve or throttling the cooler exit shut off valves. Opening up the temperature control valve was the most common way to remove trapped air. Throttling the cooler exit shut off valves is very rarely performed. These temporary maneuvers allowed more flow through the coolers and pushed the air out of the system. Guidance on when and how to remove air from these coolers is not documented in any raw cooling water or exciter operating instructions and relies largely on historical knowledge as to when it needs to be performed. During startup from the spring 2008 refueling outage, operations personnel noticed a step change in the exciter inboard journal bearing vibration amplitude. PER 145841 was initiated to investigate the change in vibrations.
The original equipment manufacturer's, GE, vibration data collection system was used to collect data on December 12, 2008. This data was delivered to the GE Region Vibration Specialist. GE stated that based on past experiences the vibration anomaly appeared most likely to be a light rub on the exciter shaft. This may be a deflector (i.e., seal) that was just touching or lying down refueling outage 13, it is most likely that these low clearances (i.e., light rub) were introduced during that time.
Following startup from refueling outage 13, the Unit 3 exciter bearing vibration was never above 5 mils and Nuclear Electric Insurance Limited Rotating Equipment Vibration Guideline, 0-TI-230V, states that vibration level less than or equal to 6 mils is acceptable for continuous normal operation. The Unit 3 exciter journal bearing vibration step change from the previous operating period was acceptable for continuous operation until the exciter could have been disassembled, inspected, and cleared of the rub.
A WO was initiated to disassemble and inspect the exciter to clear the rub. The WO was scheduled to be completed in the Unit 3 refueling outage 14 (March 2010), but the exciter disassembly inspection was removed from the refueling outage scope and not performed based on the fact that vibration levels throughout the operating cycle were not above established limits.
Unit 3 exciter air cooler temperatures began to diverge in September 2010. A review of data indicated that the 'A' cooler was air bound during the summer of 2010. As the river temperature decreased in the fall of 2010, the 'B' cooler followed the river temperature profile while the 'A' cooler did not. With the 'A' balancing valve throttled and with changes in river temperatures, the 'A' cooler continued to be air bound resulting in differential temperature development between coolers 'A' and 'B' as the river temperature decreased.
The contributors to the sustained differential temperature of greater than 10 degrees Fahrenheit in the months prior to the event are the outlet valve from the 'A' cooler throttled approximately 80 percent closed, and the lack of a formal method or process to vent the coolers. The sustained differential temperature above the 10 degrees Fahrenheit limit (set by BFN system engineering based on hydrogen cooler limits) combined with the decreased clearances on the exciter rotor caused a hard rub on the exciter rotor. The hard rub caused high temperatures to develop where the rotor was contacting the oil and air deflectors. As a result of the high temperatures, thermal expansion of the rotor material further compounded the problem causing vibrations to increase until the vibration alarm alerted operations personnel to take action in accordance with the ARP.
V. ASSESSMENT OF SAFETY CONSEQUENCES
generator exciter inboard and outboard journal bearings were not significant. All plant systems responded as required to the manual scram signal. No ECCS initiations occurred as a result of the manual scram signal. Reactor water level lowered below Level 3 (+2") as a result of the Scram and was recovered to the normal water level band by the RFPs. The expected PCIS groups 2, 3, 6, and 8 isolations were received due to reactor water level lowering below Level 3 (+2") with all systems isolating as required. There were no indications of MSRVs opening. All control rods fully inserted. The plant was in a normal post-scram electrical alignment. Decay heat was being removed through the turbine bypass valves to the main condenser.
VI. CORRECTIVE ACTIONS
A. Immediate Corrective Actions
The rub was corrected during the forced outage. The exciter was decoupled from the generator and disassembled for inspection. The exciter was shipped to the TVA Power Service Shop where repairs were made to the rotor shaft to remove the indications caused by the rub. The exciter inboard and outboard journal bearing air and oil seals were replaced. Electrical testing of the exciter was performed, and no anomalies were found.
The manual valve on the outlet of the 'A' cooler was replaced after inspections identified the valve was stuck throttled in the 80 percent closed position and could not be operated in the open or closed direction. A start-up monitoring plan was developed and implemented to monitor exciter parameters during following the forced outage.
B. Corrective Actions to Prevent Recurrence
1. Install vents on the Unit 2 and Unit 3 exciter air coolers for use in minimizing air binding.
(Unit 1 currently has vents installed.) 2. Establish a process and frequency for venting each of the exciter air coolers.
3. Increase the controls and documentation on the manipulation of the manual "balancing" valves.
4. Increase rigor and oversight of system monitoring process.
5. Perform a training analysis for inclusion of relevant aspects of this root cause into the Operations and Engineering training materials.
VII. ADDITIONAL INFORMATION
A. Failed Components
None B. Previous LERs or Similar Events None
C. Additional Information
The corrective action document for this report is PER 301505.
D. Safety System Functional Failure Consideration:
This event is not a safety system functional failure according to NEI 99-02.
E. Scram With Complications Consideration:
This event was not a complicated scram according to NEI 99-02.
VIII. COMMITMENTS
None