05000390/LER-2012-001
| Event date: | 02-13-2011 |
|---|---|
| Report date: | 06-28-2013 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability |
| 3902012001R01 - NRC Website | |
On 11/21/2011 (the date Vital Battery III (VB3) failed the capacity test conducted in accordance with Surveillance Requirement (SR) 3.8.4.14), Unit 1 was operating in Mode 1 at 100% power.
On 01/17/2012 (the date of discovery of the Vital Battery IV (VB4) inoperability), Unit 1 was operating in Mode 1 at 100% power.
For the time period between 02/13/2011 and 12/03/2011, the following table establishes when VB4 was required to be operable to comply with Technical Specifications 3.8.4 and 3.8.5, the associated Mode or specified condition of operation, and applicable Rated Thermal Power (RTP).
Table 1 — Plant Conditions Associated with VB IV Inoperability 04/16/2011 at 0925 through 04/29/2011 at 2124 No Mode 000 VB4 not required to be operable No Condition Prohibited by TS Existed regarding VB4 06/27/2011 at 1647 through 07/09/2011 at 0501 1 Various VB4 removed from service — VB5 aligned to Vital Battery Board IV No Condition Prohibited by TS Existed regarding VB4 07/09/2011 at 0501 through 12/03/2011 at 0408 1, 2, and 3 100 - 000 3.8.4 — VB4 required to be operable Condition Prohibited by TS Existed Note: VB4 was not required to be operable for the time periods of: 1) 04/16/2011 at 0925 through 04/29/2011 at 2124 when the plant was in the "No Mode" condition, and 2) 06/27/2011 at 1647 through 07/09/2011 at 0501 when VB4 was removed from service and Vital Battery V (VB5) was aligned to Vital Battery Board IV.
DESCRIPTION OF EVENT
A. Event
VB3 Description of Event On 11/21/2011, a battery capacity test was completed for VB3 [Energy Industry Identification System (EIIS) Code EL] in accordance with SR 3.8.4.14. A battery capacity of approximately 70% was recorded, which did not meet the acceptance criterion of 80% for SR 3.8.4.14.
At the time of discovery, VB3 was removed from service, and was not being credited to meet TS 3.8.4. VB5 was aligned to Vital Battery Board III.
NUREG-1022 provides the following guidance regarding discrepancies identified during "...discrepancies found in technical specifications surveillance tests surveillance tests:
should be assumed to occur at the time of the test unless there is firm evidence, based on a review of relevant information (e.g., the equipment history and the cause of failure) to indicate that the discrepancy occurred earlier.
For VB3, the previous capacity test conducted in accordance with SR 3.8.4.14 on 10/31/2005 established a battery capacity of 109.85%. A review of maintenance history and the equipment failure analysis did not identify a specific event or action that occurred between 10/31/2005 (previous successful capacity test for VB3) and 11/21/2011 (failed capacity test for VB3) that rendered VB3 inoperable when aligned to a Vital Battery Board.
Despite the lack of firm evidence, on 03/14/2012, TVA conservatively concluded that VB3 may have been inoperable at some point in time prior to 11/21/2011. However, WA cannot identify when this inoperability occurred. Thus, for an undefined period of time that VB3 was credited as one of the channels of vital DC to comply with the requirements of LCO 3.8.4 or LCO 3.8.5, a condition prohibited by Technical Specifications may have existed.
During the time periods discussed above, VB3 was capable of performing its safety function (See Section V of this LER for detailed information).
VB4 Description of Event On 11/30/2011, Problem Evaluation Report (PER) 468950 was initiated to determine the cause of the unexpected degradation of VB3 and VB4.
On 02/10/2011, a battery capacity test was completed for VB4 [EIIS Code EL] in accordance with SR 3.8.4.14. A battery capacity of 82.5% was recorded. The battery was considered operable, because the battery capacity exceeded the acceptance criterion of 80% for SR 3.8.4.14.
In addition to this unknown period of time, Section I of this LER establishes periods of time when VB4 was inoperable, and it was required to be operable to comply with TS 3.8.4 and TS 3.8.5.
Thus, during periods of time that VB4 was inoperable and it was credited as one of the channels of vital DC to comply with the requirements of LCO 3.8.4 or LCO 3.8.5, a condition prohibited by Technical Specifications existed.
During the time periods discussed above, VB4 was capable of performing its safety function (See Section V of this LER for detailed information).
Conditions Reportable in accordance with 10 CFR 50.73(a)(2)(i)(B) The following events are reportable as an LER in accordance with 10 CFR 50.73(a)(2)(i)(B):
Failure to meet SR 3.8.4.14 for VB4 within the specified frequency.
When VB4 was required to be operable during the time periods identified in the Table in Section I of this LER, WBN, Unit 1 did not enter the applicable Conditions and Required Actions of the applicable TS (i.e., TS 3.8.4 or TS 3.8.5), and did not perform the applicable Required Actions within the applicable Completion Times.
WA conservatively concluded that VB3 and VB4 may have been inoperable for an did not enter the applicable Conditions and Required Actions of the applicable TS (i.e., TS 3.8.4 or TS 3.8.5), and did not perform the applicable Required Actions within the applicable Completion Times.
As a result of the above, multiple inappropriate Mode changes occurred due to the unknown inoperability of VB4 and may have occurred due to the unknown inoperability of VB3, when WBN, Unit 1 entered a Mode or specified condition of applicability for TS 3.8.4 or TS 3.8.5 while ascending in Modes following the refueling outage.
12/02/2011 2326 VB3 replaced with new battery and returned to service. VB3 tested in accordance with service test SR 3.8.4.13. VB3 met the acceptance criterion 105 Volts (Vdc)). Factory capacity test performed by C&D Technologies was credited for meeting the acceptance criterion for SR 3.8.4.14.
D. Other Systems or Secondary Functions Affected
There were no other systems or secondary functions affected.
E. Method of Discovery
The VB3 inoperability was discovered on 11/21/2011 following performance of a battery capacity test in accordance with SR 3.8.4.14.
During the performance of the root cause analysis for PER 468950 to determine the cause of the unexpected degradation of VB3 and VB4, an independent engineering analysis of the completed surveillance package for VB4 determined that the recorded results for the VB4 battery capacity test were incorrect, and that the battery capacity for VB4 was actually 79.87%.
F. Operator Actions
No Operator actions were required.
At the time of discovery of the VB3 inoperability (11/21/2011), VB3 was removed from service, and was not being credited to meet TS 3.8.4. VB5 was aligned to Vital Battery Board III. VB3 was subsequently replaced with a new battery that met the surveillance requirements prior to being restored to service.
At the time of discovery of the VB4 inoperability (01/17/2012), VB4 had been replaced with a new battery that met the surveillance requirements.
G. Safety System Responses
At no time during this reporting period was Vital Battery Board III or Vital Battery Board IV incapable of performing its design bases function (See Section V of this LER for detailed information).
VB3 and VB4 are only required to support plant safety loads if: 1) its associated battery charger fails; 2) a loss of offsite power occurs; or 3) a station blackout occurs. No plant transient occurred that would have required VB3 or VB4 to perform its intended safety function.
CAUSE OF EVENT
Cause of Unexpected Degradation of VB3 and VB4 TVA determined that the direct cause was due to the aggregate effects of lower battery temperatures, high electrolyte specific gravities, and marginal float voltages which over time caused the loss of battery capacity. This conclusion was primarily based on the ability to recover the capacity of the weak cells that were sent back to the vendor for analysis.
1) The organization's Risk Assessment System was less than adequate to assess the operability of the station's vital batteries and failed to identify the degraded capacity of VB3 and VB4; and 2) The organization did not promptly identify, fully analyze and resolve in a timely manner unexpected safety significant trend and test data concerning vital battery operability.
3) Inadequate design of vital battery room temperature controls which resulted in low vital battery room temperatures and inadequate charging currents.
Cause of Conditions Prohibited by Technical Specifications Associated with VB4 Inoperability VB4 was inoperable during the time periods identified in Sections I and II of this LER due to unidentified errors in the battery capacity calculations that were completed on 02/10/2011. The errors were discovered as part of an independent engineering analysis of the completed surveillance package for VB4 completed on 01/17/2012. The independent engineering analysis indicated that the actual battery capacity was 79.87%. This is less than the SR 3.8.4.14 acceptance criterion of 80% of manufacturer rating.
There were several problems with both the test equipment and procedure that caused VB4 recorded capacity (82.5%) to be in error. The errors were due to inaccurate calculation of the time interval that battery cell No. 54 was jumpered out of the circuit during the four hour capacity discharge test and rounding errors where times were not calculated to the nearest second. These errors resulted in greater recorded amp-hour capacity than what actually existed. Battery Cell No. 54 was jumpered out due to low cell voltage.
IV. ANALYSIS OF THE EVENT
VB3 and VB4 Unexpected Degraded Capacity WBN's oversight of the vital battery surveillance program was less than adequate. WBN's Risk Assessment System with respect to the vital batteries consists of implementation of the surveillance program. Additional elements that are used to monitor and assess the condition of the vital battery program are the System Health Report, the Operational Experience program, and the Margin Management program.
WBN's organizational response to the VB4 events in February and June 2011 was less than adequate.
Site management failed to recognize or understand the potential significance of vital battery degradation.
The organizational response to the 02/10/2011 VB4 capacity test failure was less than adequate to ensure VB4 would not fail the 06/27/2011 service test in accordance with SR 3.8.4.13. The organizational response to the 06/27/2011 VB4 service test failure was inadequate to ensure VB3 would not fail the 11/21/2011 capacity test in accordance with SR 3.8.4.14.
The computer system used for battery testing is obsolete. New test equipment was available, but training and test procedures to allow the use of new equipment had not been completed. Use of the obsolete test equipment and procedure deficiency led to inaccurate test results. Specifically, the test procedures did not specify that the time required to jumper out a defective cell(s) must be subtracted from the total discharge time when calculating battery capacity. In addition, WBN non-conservatively rounded test data during the VB4 capacity test.
The original determination of operability for VB4 was based on the original capacity test results for VB4. If the test procedure had determined VB4's capacity correctly, VB4 would not have been returned to service until the issue was resolved. Errors in the testing practices as described above caused TVA to not recognize that VB4 was outside the TS acceptance criterion of SR 3.8.4.14.
V. ASSESSMENT OF SAFETY CONSEQUENCES
Introduction The vital 125V DC power system is a Class 1 E system composed of four redundant channels (Channels I and III are associated with Train A and Channels II and IV are associated with Train B).
Each channel consists of a lead-acid-calcium battery, battery charger, distribution board, and the required cabling, instrumentation and protective features. These four channels provide control power to the Class 1 E 6.9 kV shutdown boards, 480V motor control centers, inverters and emergency DC lighting systems.
Each vital battery has adequate storage capacity to carry the required load continuously for at least 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> in the event of a loss of all AC power (station blackout) without an accident or for 30 minutes with an accident considering a single failure. Each battery board can also be aligned to the fifth vital battery system. The fifth 125V DC Vital Battery System can serve as a replacement for any one of the four 125V DC vital batteries during testing, maintenance, and outages with no loss of system reliability under any mode of operation.
The vital batteries preventative maintenance program has been maintained in accordance with manufacturers recommendations. In addition to the capacity and service tests, battery voltage is checked daily and the battery pilot cell temperature is checked weekly. Individual cell voltages and specific gravity checks are performed quarterly, and the battery circuit connection resistances are checked annually.
Surveillance testing is performed for all vital batteries in accordance with SR 3.8.4.13 and SR 3.8.4.14.
The test conducted in accordance with SR 3.8.4.13 is called the service test in this LER. It is a timed 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> discharge test that uses a load profile based on WBN's design basis event's equipment load requirements for safe shutdown. The result is the battery's voltage reading at the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> mark which is compared to the SR 3.8.4.13 acceptance criterion of 105 Vdc.
The test conducted in accordance with SR 3.8.4.14 is called the capacity test in this LER. It is a timed discharge at a constant load rate and is terminated when the overall cell voltage reaches 105 Vdc. The results are given as a capacity percentage value with respect to the designed capacity of the battery. The SR 3.8.4.14 acceptance criterion is 80%.
VB3 Discussion On 12/19/2011, a service test was performed on the VB3 that was replaced on 12/02/2012. This loads, common loads, and loads transferred from Unit 2 to Unit 1. VB3 met the acceptance criterion 105 Vdc) with a terminal voltage of 115.1 Vdc. This test demonstrated that VB3 had sufficient stored energy to meet design bases accident load demands for the time frame between 02/25/2011 when the last service test was performed and 11/21/2011 when VB3 was removed from service for replacement.
This test demonstrated VB3 was capable of performing its safety function for the worst case scenario which is the four-hour station blackout.
VB4 Discussion As indicated in Section II.0 of this LER, VB4 met the acceptance criterion of SR 3.8.4.13 on 07/06/2011 with a margin of 0.4 Vdc. This service test used a two unit load profile. A two unit load loads. Unit 2 is currently under construction, and all of its loads have not been transferred to VB4.
On 12/07/2011, a service test was performed on VB4 with the eight new cells that had been installed on 07/06/2011. This service test was a battery discharge test using a single unit load profile, which consists of Unit 1 loads, common loads, and loads transferred from Unit 2 to Unit 1. VB4 met the acceptance criterion 105 Vdc) with a terminal voltage of 114.9 Vdc. This test demonstrated that VB4 had sufficient stored energy to meet design bases accident load demands for the time frame between 07/09/2011 and 12/03/2011.
On 01/06/2012, a service test using a single unit load profile was performed on VB4 using the eight weak cells that had been removed on 07/06/2011. VB4 met the acceptance criterion (.?_ 105 Vdc) with a terminal voltage of 111.9 Vdc. This test demonstrated that VB4 had sufficient stored energy to meet design bases accident load demands for the time frame between 02/13/2011 and 06/27/2011.
These tests demonstrated VB4 was capable of performing its safety function for the worst case scenario which is the four-hour station blackout.
Based on the findings regarding the VB4 capacity test, extent of condition was considered for previous capacity tests performed on VB1, VB2, VB3, VB5 and the Emergency Diesel Generator (EDG) batteries. The 02/10/2011 capacity test for VB4 was the only time a battery cell was jumpered out resulting in amp-hour capacity errors. Rounding errors were discovered where computer printouts were not available to record the test's total discharge times. None of the rounding errors identified by this review affected the acceptance criterion in the applicable test packages. The rounding errors have been corrected in the permanent records.
The vital batteries and the EDG batteries were replaced or are scheduled to be replaced as follows:
VB1 and VB2 were replaced in 2009 VB3 was replaced in 2011 VB4 was replaced in 2012 VB5 is scheduled to be replaced in late 2012. On 02/22/2012, a capacity test in accordance with SR 3.8.4.14 established that VB5's capacity was 106.19%.
The EDG batteries were replaced in 2006.
VI. CORRECTIVE ACTIONS
A. Immediate Corrective Actions
There were no immediate corrective actions required for the VB3 inoperability. At the time of discovery of the VB3 inoperability (11/21/2011), VB3 was removed from service, and was not credited to meet TS 3.8.4. VB5 was aligned to Vital Battery Board III. VB3 was subsequently replaced with a new battery that met the surveillance requirements prior to being restored to service.
There were no immediate corrective actions required for the VB4 inoperability. At the time of discovery of the VB4 inoperability (01/17/2012), a new battery that met the surveillance requirements was installed as VB4.
B. Corrective Actions to Prevent Recurrence of Unexpected Battery Degradation 1. NPG-SPP-06.9.2, Attachment 2, "SI Scheduling Exception Form," was revised to require Plant Manager or Shift Manager approval prior to scheduling surveillance instructions deep into the grace period (> 50% of grace period).
2. Perform the modified performance discharge test specified in SR 3.8.4.14 on an 18 month frequency instead of a 60 month frequency. A License Amendment Request to modify SR 3.8.4.13 would be required to change Note 1 of SR 3.8.4.13 to permit the modified performance discharge test of SR 3.8.4.14 to be performed in lieu of SR 3.8.4.13 every 18 months.
3. Incorporate a Case Study of the 2011 events involving the vital batteries with specific examples of the organization's lessons learned into the department's recurring continuing training program.
3. Revise existing vital battery surveillance instructions to ensure that the battery capacity calculation accounts for periods when the discharge test is interrupted and that all values are recorded to the nearest second.
4. Implement the design change to control the Vital Battery Room temperatures to a nominal 77 °F.
VII. ADDITIONAL INFORMATION
A. Failed Components
VB3 and VB4 failed the SR 3.8.4.14 acceptance criterion. The batteries are C&D Technologies Model LCUN-33. TVA determined that the direct cause was due to the aggregate effects of lower battery temperatures, high electrolyte specific gravities, and marginal float voltages which over time caused the loss of battery capacity. This conclusion was primarily based on the ability to recover the capacity of the weak cells that were sent back to the vendor for analysis.
B. Previous LERs on Similar Events had a similar event regarding an unexpected degradation of the capacity of VB1. In May 2009, VB1 was replaced due to a capacity degradation trend. Prior to replacement, the battery capacity remained above the acceptance criterion of SR 3.8.4.14. The vendor concluded that improper curing of the battery cell plates was the likely cause of the earlier than expected loss in capacity.
C. Additional Information
None D. Safety System Functional Failure This event did not involve a safety system functional failure as defined in Nuclear Energy Institute (NEI) 99-02, "Regulatory Assessment Performance Indicator Guideline," Revision 5.
I