05000296/LER-2010-004, Re Manual Reactor Scram Due to High Vibration on the Generator Exciter Inboard and Outboard Journal Bearings
| ML110610339 | |
| Person / Time | |
|---|---|
| Site: | Browns Ferry  |
| Issue date: | 02/24/2011 |
| From: | Polson K Tennessee Valley Authority |
| To: | Document Control Desk, Office of Nuclear Reactor Regulation |
| References | |
| LER 10-004-00 | |
| Download: ML110610339 (9) | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(i) 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability 10 CFR 50.73(a)(2)(ii)(A), Seriously Degraded 10 CFR 50.73(a)(2)(viii)(A) 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition 10 CFR 50.73(a)(2)(viii) 10 CFR 50.73(a)(2)(iii) 10 CFR 50.73(a)(2)(ix)(A) 10 CFR 50.73(a)(2)(x) 10 CFR 50.73(a)(2)(v)(A), Loss of Safety Function - Shutdown the Reactor 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat 10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown 10 CFR 50.73(a)(2)(v), Loss of Safety Function 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 2962010004R00 - NRC Website | |
text
Tennessee Valley Authority, Post Office Box 2000, Decatur, Alabama 35609-2000 February 24, 2011 10 CFR 50.73 ATTN: Document Control Desk U.S. Nuclear Regulatory Commission Washington, D.C. 20555-0001 Browns Ferry Nuclear Plant, Unit 3 Facility Operating License No. DPR-68 NRC Docket No. 50-296
Subject:
Licensee Event Report 50-296/2010-004-00 The enclosed Licensee Event Report provides details of a manual reactor scram due to high vibration on the Unit 3 generator exciter inboard and outboard journal bearings. The Tennessee Valley Authority (TVA) is submitting this report in accordance with 10 CFR 50.73(a)(2)(iv)(A), as any event or condition that resulted in manual or automatic actuation of any of the systems listed in paragraph 10 CFR 50.73(a)(2)(iv)(B) (i.e., Reactor Protection System including reactor scram or trip, and general containment isolation signals affecting containment isolation valves in more than one system).
There are no new regulatory commitments contained in this letter. Should you have any questions concerning this submittal, please contact J. E. Emens, Jr., Nuclear Site Licensing Manager, at (256) 729-2636.
Respectfully, Vice President Enclosure: Licensee Event Report - Unit 3 Manual Reactor Scram Due to High Vibration on the Generator Exciter Inboard and Outboard Journal Bearings cc (w/ Enclosure):
NRC Regional Administrator - Region II NRC Senior Resident Inspector - Browns Ferry Nuclear Plant
ENCLOSURE Browns Ferry Nuclear Plant Unit 3 Licensee Event Report - Unit 3 Manual Reactor Scram Due to High Vibration on the Generator Exciter Inboard and Outboard Journal Bearings See Attached
NRC FORM 366 U.S. NUCLEAR REGULATORY COMMISSION APPROVED BY OMB NO. 3150-0104 EXPIRES 10/31/2013 (10-2010)
, the NRC may not conduct or sponsor, and a person is not required to respond to, the information collection.
- 3. PAGE Browns Ferry Nuclear Plant Unit 3 05000296 1 of 7
- 4. TITLE: Manual Reactor Scram Due to High Vibration on the Generator Exciter Inboard and Outboard Journal Bearings
- 5. EVENT DATE
- 6. LER NUMBER
- 7. REPORT DATE
- 8. OTHER FACILITIES INVOLVED I SEQUENTIAL REV M
D Y
FACILITY NAME DOCKET NUMBER MONT YEAR YEAR NUMBER NO.
MONTh DAY YEAR N/A 05000 FACILITY NAME DOCKET NUMBER 12 26 2010 2010 004 00 02 24 2011 N/A 05000
- 9. OPERATING MODE
- 11. THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR §: (Check all that apply) 0] 20.2201 (b)
[I 20.2203(a)(3)(i)
[I 50.73(a)(2)(i)(C)
[I 50.73(a)(2)(vii) 0] 20.2201 (d) 0] 20.2203(a)(3)(ii)
[I 50.73(a)(2)(ii)(A)
[I 50.73(a)(2)(viii)(A) 0] 20.2203(a)(1) 0] 20.2203(a)(4) 0] 50.73(a)(2)(ii)(B) 0] 50.73(a)(2)(viii)(a) 1 20.2203(a)(2)(i) 0 50.36(c)(1)(i)(A)
[I 50.73(a)(2)(iii)
[I 50.73(a)(2)(ix)(A)
- 10. POWER LEVEL 0
20.2203(a)(2)(ii) 0 50.36(c)(1)(ii)(A) 0 50.73(a)(2)(iv)(A)
[I 50.73(a)(2)(x) 0] 20.2203(a)(2)(iii) 0] 50.36(c)(2) 0] 50.73(a)(2)(v)(A)
[]73.71 (a)(4)
[] 20.2203(a)(2)(iv)
[I 50.46(a)(3)(ii) 0] 50.73(a)(2)(v)(B)
[]73.71 (a)(5) 100 0
20.2203(a)(2)(v) 0 50.73(a)(2)(i)(A) 0 50.73(a)(2)(v)(C)
- - 0 OTHER 0 20.2203(a)(2)(vi)
[0 50.73(a)(2)(i)(B)
[0 50.73(a)(2)(v)(D)
Specify In Abstrac below or in The bearing vibration instrumentation had been replaced during the outage. Upon startup from the outage, the 'B' cooler became air bound and the differential temperature between coolers 'A' and 'B' spiked to approximately 50 degrees Fahrenheit. The outlet of the 'A' cooler was throttled in the closed direction, and the exciter air differential temperatures returned to normal. The outlet of the 'A' cooler was left in this throttled position until the event in December of 2010.
September 2010 December 26, 2010 at 1615 CST December 26, 2010 at 1620 CST December 26, 2010 at 1953 CST January 6, 2011 at 1656 CST Unit 3 exciter air cooler temperatures began to diverge. A review of data indicated that the 'A' cooler was air bound during the summer of 2010. As the river temperature decreased in the fall of 2010, the 'B' cooler followed the river temperature profile while the 'A' cooler did not. With the 'A' balancing valve throttled and with changes in river temperatures, the 'A' cooler continued to be air bound resulting in differential temperature development between coolers
'A' and 'B' as the river temperature decreased.
An alarm for Main Turbine Vibration High 3-VA-47-15 was received in the Unit 3 control room on annunciator panel 3-XA-55-7B Window 32.
After an Upper Power Runback reduced reactor power to approximately 88 percent, operations personnel initiated a manual reactor scram due to high vibrations on generator exciter inboard and outboard journal bearings.
Operations personnel made a non-emergency phone call to the NRC in accordance with 10 CFR 50.72(b)(2)(iv)(B) and 10 CFR 50.72(b)(3)(iv)(A).
Operations personnel performed Main Generator Synchronization and closed Main Generator Breaker.
D. Other Systems or Secondary Functions Affected
None
E. Method of Discovery
The exciter rotor journal bearing high vibration levels were identified by an alarm for Main Turbine Vibration High 3-VA-47-15 on annunciator panel 3-XA-55-7B Window 32.
F..Oerator Actions Operations personnel responded to the alarm per 3-ARP-9-7B and initiated a manual reactor scram in accordance with 3-AOl-1 00-1, Reactor Scram.
G. Safety System Responses All plant systems responded as required to the manual scram signal. No ECCS initiations occurred as a result of the manual scram signal. Reactor water level lowered below Level 3
(+2") as a result of the scram and was recovered to the normal water level band by the RFPs.
The expected PCIS groups 2, 3, 6, and 8 isolations were received due to reactor water level lowering below Level'3 (+2") with all systems isolating as required. There were no indications of MSRVs opening. All control rods fully inserted. The plant was in a normal post-scram electrical alignment. Decay heat was being removed through the turbine bypass valves to the main condenser.
Ill. CAUSE OF THE EVENT A. Immediate Cause The immediate cause for this reportable condition was high vibrations on generator exciter inboard and outboard journal bearings that led to a manual reactor trip.
B. Root Cause The root cause was inadequate procedural guidance for monitoring the exciter air cooling system and prescribing mitigation actions to be taken based on differential temperature limits.
Specific instructions did not exist for the operations and engineering personnel responsible for maintaining system parameters within acceptable limits.
C. Contributing Factors
- 1. System monitoring did not identify the exciter air temperature imbalance prior to the event. The role of system monitoring and trending is to set in motion the mitigating steps necessary to prevent adverse effects on plant equipment.
- 2. Flow blockage (i.e., outlet valve to the 'A' cooler throttled 80% closed) and air binding (no capability or process to vent air from coolers) were contributors to this event. With limited capability or process to remove air from the exciter coolers, it is inevitable that air pockets will develop in the coolers during operation. The air pockets effectively reduce the performance of the cooler(s) which can lead to unbalanced air temperatures.
- 3.
A clearance issue (light rub) between the exciter rotor and bearing oil/air deflector shield was identified in 2008. The Work Order (WO) to inspect and clear the rub was deleted
from the spring 2010 refueling outage scope based on the fact that vibration levels throughout the operating cycle were not above established limits.
IV. ANALYSIS OF THE EVENT
During this event, high vibration on the generator exciter inboard and outboard journal bearings was caused by an air temperature imbalance within the exciter. The air temperature imbalance caused the exciter frame to "twist" by thermal growth. The movement of the exciter frame combined with a pre-existing condition (i.e., light rub identified in 2008) at the bearings resulted in a rub (i.e, a stationary component just touching the rotating shaft) which resulted in high exciter journal bearing vibrations.
Following the manual scram, the outlet valve of the 'A' cooler was found to be in the throttled position. An attempt was made to open the valve; however, the valve was stuck in its as-found position and would not move. A decision was made to replace the valve and, upon removal, the valve was found to have been throttled approximately 80 percent closed. This condition was determined to have existed since May of 2008. Upon startup from the Unit 3 refueling outage 13 in May of 2008, the 'B' cooler appeared to have been air bound (air trapped inside the cooler) based on subsequent increasing exit air temperatures. The difference between the 'A' and 'B' coolers exit air temperatures peaked at 50 degrees Fahrenheit. To balance the air cooler exit temperatures, the outlet valve of the 'A' cooler was throttled. Throttling of the outlet valve of the
'A' cooler reduced the amount of raw cooling water flow through the 'A' cooler and increased the amount of cooling water flow through the 'B' cooler. The extra flow through the 'B' cooler assisted in pushing the air out of the cooler. This manipulation was successful as the differential temperature returned to within the normal range. There was no process to evaluate the long-term effects of leaving this valve throttled nor was there any configuration control as system 024 valve line-ups are not performed unless work is performed on the system Having this valve in the throttled position was considered a significant contributor to the development of the unbalanced air temperature within the exciter coolers.
When BFN has experienced air entrapment in the exciter coolers following refueling outages, the removal of the trapped air from the coolers was accomplished by opening the temperature control valve or throttling the cooler exit shut off valves. Opening up the temperature control valve was the most common way to remove trapped air. Throttling the cooler exit shut off valves is very rarely performed. These temporary maneuvers allowed more flow through the coolers and pushed the air out of the system. Guidance on when and how to remove air from these coolers is not documented in any raw cooling water or exciter operating instructions and relies largely on historical knowledge as to when it needs to be performed. During startup from the spring 2008 refueling outage, operations personnel noticed a step change in the exciter inboard journal bearing vibration amplitude. PER 145841 was initiated to investigate the change in vibrations.
The original equipment manufacturer's, GE, vibration data collection system was used to collect data on December 12, 2008. This data was delivered to the GE Region Vibration Specialist. GE stated that based on past experiences the vibration anomaly appeared most likely to be a light rub on the exciter shaft. This may be a deflector (i.e., seal) that was just touching or lying down on the rotating shaft. Since the exciter was disassembled and reassembled during Unit 3 refueling outage 13, it is most likely that these low clearances (i.e., light rub) were introduced during that time.
Following startup from refueling outage 13, the Unit 3 exciter bearing vibration was never above 5 mils and Nuclear Electric Insurance Limited Rotating Equipment Vibration Guideline, 0-TI-230V, states that vibration level less than or equal to 6 mils is acceptable for continuous normal operation. The Unit 3 exciter journal bearing vibration step change from the previous operating period was acceptable for continuous operation until the exciter could have been disassembled, inspected, and cleared of the rub.
A WO was initiated to disassemble and inspect the exciter to clear the rub. The WO was scheduled to be completed in the Unit 3 refueling outage 14 (March 2010), but the exciter disassembly inspection was removed from the refueling outage scope and not performed based on the fact that vibration levels throughout the operating cycle were not above established limits.
Unit 3 exciter air cooler temperatures began to diverge in September 2010. A review of data indicated that the 'A' cooler was air bound during the summer of 2010. As the river temperature decreased in the fall of 2010, the 'B' cooler followed the river temperature profile while the 'A' cooler did not. With the 'A' balancing valve throttled and with changes in river temperatures, the
'A' cooler continued to be air bound resulting in differential temperature development between coolers 'A' and 'B' as the river temperature decreased.
The contributors to the sustained differential temperature of greater than 10 degrees Fahrenheit in the months prior to the event are the outlet valve from the 'A' cooler throttled approximately 80 percent closed, and the lack of a formal method or process to vent the coolers. The sustained differential temperature above the 10 degrees Fahrenheit limit (set by BFN system engineering based on hydrogen cooler limits) combined with the decreased clearances on the exciterrotor caused a hard rub on the exciter rotor. The hard rub caused high temperatures to develop where the rotor was contacting the oil and air deflectors. As a result of the high temperatures, thermal expansion of the rotor material further compounded the problem causing vibrations to increase until the vibration alarm alerted operations personnel to take action in accordance with the ARP.
V. ASSESSMENT OF SAFETY CONSEQUENCES
The safety consequences of the manual reactor scram due to high vibrations on the Unit 3 generator exciter inboard and outboard journal bearings were not significant. All plant systems responded as required to the manual scram signal. No ECCS initiations occurred as a result of the manual scram signal. Reactor water level lowered below Level 3 (+2") as a result of the scram and was recovered to the normal water level band by the RFPs. The expected PCIS groups 2, 3, 6, and 8 isolations were received due to reactor water level lowering below Level 3
(+2") with all systems isolating as required. There were no indications of MSRVs opening. All control rods fully inserted. The plant was in a normal post-scram electrical alignment. Decay heat was being removed through the turbine bypass valves to the main condenser.
VI. CORRECTIVE ACTIONS
A.
Immediate Corrective Actions
The rub was corrected during the forced outage. The exciter was decoupled from the generator and disassembled for inspection. The exciter was shipped to the TVA Power Service Shop where repairs were made to the rotor shaft to remove the indications caused by the rub. The exciter inboard and outboard journal bearing air and oil seals were replaced. Electrical testing of the exciter was performed, and no anomalies were found.
The manual valve on the outlet of the 'A' cooler was replaced after inspections identified the valve was stuck throttled in the 80 percent dosed position and could not be operated in the open or closed direction. A start-up monitoring plan was developed and implemented to monitor exciter parameters during following the forced outage.
B.
Corrective Actions to Prevent Recurrence
- 1. Install vents on the Unit 2 and Unit 3 exciter air coolers for use in minimizing air binding.
(Unit 1 currently has vents installed.)
- 2. Establish a process and frequency for venting each of the exciter air coolers.
- 3. Increase the controls and documentation on the manipulation of the manual "balancing" valves.
- 4. Increase rigor and oversight of system monitoring process.
- 5. Perform a training analysis for inclusion of relevant aspects of this root cause into the Operations and Engineering training materials.
VII. ADDITIONAL INFORMATION
A.
Failed Comoonents None B.
Previous LERs or Similar Events None C.
Additional Information
The corrective action document for this report is PER 301505.
D.
Safety System Functional Failure Consideration:
This event is not a safety system functional failure according to NEI 99-02.
E.
Scram With Complications Consideration:
This event was not a complicated scram according to NEI 99-02.
VIII. COMMITMENTS
None