05000341/LER-2004-003
| Docket Number11 02 2004 20040 - 0030 - 00 12 21 2004 05000 | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 3412004003R00 - NRC Website | |
Initial Plant Conditions:
�Mode 1 �Reactor Power 100 percent
Description of the Event
The Standby Liquid Control (SLC) [BR] Pump B gear reducer [RGR] was inspected on November 2, 2004 as a result of high vibration and abnormal oil analysis results from a previous run on October 24, 2004. The pump was considered operable after the October 24, 2004 run, because it met all of its surveillance test acceptance criteria, including vibration. The vibration spectrum analysis from the October 24, 2004 run was reviewed and determined not to be indicative of gear reducer damage. Subsequent additional chemical analysis of the October 24,2004 oil sample resulted in a recommendation to open and inspect the SLC Pump B gearbox. The gearbox was disassembled and inspected on November 2, 2004, at which time only about ten ounces of oil were drained from the reservoir versus the approximately 80 ounces expected. The main gear teeth showed signs of discoloration (bluing), and a small amount of discoloration was noted on the worm gear. Both gears were wetted with oil, and no other abnormal characteristics were noted. The gear teeth were inspected, and no damage was detected. The vendor was contacted and indicated that the main gear bluing was most likely due to the flame hardening that was used to heat treat the gear during original manufacture. Based on the discoloration observed, it was determined that the gears did not reach temperatures that would adversely affect the ability of the gears to perform their intended function.
The gearbox oil analysis of the October 24, 2004 sample indicated a higher than normal severe wear index (SWI) so further analysis was performed. Normal rubbing wear and bits of plastic and aluminum silicate particles were found. The aluminum silica particles were large and were thought to have been present in the bottom of the sump for a very long time, and to have no effect upon oil condition or bearing performance. The source of the plastic particles is not known and they also could have been in the sump for a long time. These small pieces of plastic were thought to have a density similar to the oil and are much softer than the gearing. When these pieces of plastic pass between the gear teeth they would be shredded with no effective wear occurring on the gear teeth.
The gear inspections performed on November 2, 2004 confirmed that there were no perceptible signs of damage to the gears.
An operability evaluation was performed, and it was determined that SLC Pump B would not have been capable of performing its intended function following the oil sample taken after its July 24, 2004 surveillance test. It is thought that the oil sample drawn was not replaced at that time, resulting in the very low gearbox oil level. The surveillance test performed before that sample was taken was normal with flow, pressure, and vibration levels observed to be normal at that time. Although SLC Pump B was run and performed its intended function for 22 minutes on October 24, 2004, pump gear case vibration was higher than normal. The inspection performed on SLC Pump B on November 2, 2004 determined that there was no gear damage or signs of significant wear or excessive overheating. However, it could not be ascertained that the pump would be capable of performing its intended function for the time required to perform those functions. Thus, SLC Pump B was considered to have been inoperable from July 24, 2004 to November 2, 2004 at 2245 hours0.026 days <br />0.624 hours <br />0.00371 weeks <br />8.542225e-4 months <br /> when the proper amount of oil was replaced in the SLC Pump B gear reducer reservoir and the pump was tested and declared operable.
Two conditions were required to obtain a low oil level in a SLC Gear Reducer. There has to be a method to reduce the oil inventory, and a condition had to exist such that a lowering oil level would not be detected during operator rounds.
There were no signs of oil leakage on or in the vicinity of the gear reducer. The configuration, service temperatures, and run times were also such that oil consumption during operation would be negligible. However, investigation determined that on multiple occasions samples were taken from the gear reducer after quarterly pump runs without replenishing the sample and purge volumes. Each oil sample and purge consumes a total of 11.5 ounces of gearbox oil, and if not replaced, over time, this would result in a low oil level.
The only level indicator for these pump gear reducers is configured as a 90-degree pipe elbow with a downward slope from the gearbox such that some oil is retained in the bowl of the horizontal run, even with a totally drained reducer gearbox. The location of the SLC Pump B is such that the approach to view the level is only from the top. From that angle, it is difficult to see a difference between low and high levels in the level indicator cup, and the level must be near the top of the cup in order to ensure the proper oil level in the gearbox. Thus, it was concluded that the oil level could have been low for an extended period of time, even though it had been checked.
The same level indicator on the SLC Pump A is more accessible, and can be viewed from more than one angle.
Thus, a low level condition was more detectable on the SLC Pump A which is why it was found to have the proper oil level.
Technical Specification 3.1.7 requires two SLC Subsystems to be operable when in Modes I and 2. Action A allows one SLC subsystem to be inoperable for up to 7 days, and Action B allows both SLC subsystems to be inoperable for up to 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />. When either required Actions A or B and associated completion times are not met, Action C must be entered which requires the plant to be brought to Mode 3 in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. Since it was subsequently determined that the SLC Pump B subsystem was inoperable from July 24 to November 2, 2004, the 7 days of allowed inoperability under Action A ended on July 31, 2004. Since required Action C was not entered at that time, the plant was operated in a condition prohibited by Technical Specifications from July 31, 2004 until SLC Pump B inoperability was corrected on November 2, 2004.
The pump was reassembled, lubricated, and tested on November 2, 2004. Vibration returned to normal levels during this test, and the pump was declared operable.
Cause of the Event
It was determined that the gear reducer oil was not replenished in all cases after quarterly oil samples which resulted in a reduction in SLC Pump B gear reducer oil level over time. This was coupled with the difficulty of determining the oil levels during operator rounds. The installed gearbox oil level indicators appeared to indicate a satisfactory oil level when the oil level was actually low. This was especially true for the SLC Pump B gear reducer which could only be read from the top. The procedural requirements for these activities were inadequate in that they were subject to misinterpretation. The procedures did not discuss that there were two reservoirs for each SLC pump, that the indicator cups need to be nearly full to ensure adequate lubrication, that a 17-hour precaution on adding oil to the reservoir applied only to the pump crankcase, and that the 17-hour precaution did not apply when replacing the precise amount of oil removed when purging and sampling. Marking "N/A" on the step to replace the purge and sample volume was also not prohibited by the procedures.
There are two oil reservoirs per pump. One is for the pump crankcase, and one is for the gear reducer. The pump crankcase has a sight glass level indicator with "green bands" indicating the proper oil levels for operating and standby conditions. The system operating and surveillance procedures contained a precaution that indicated that 17 hours1.967593e-4 days <br />0.00472 hours <br />2.810847e-5 weeks <br />6.4685e-6 months <br /> is required after a pump run for the oil to drain back to the crankcase, and that oil should not be added during that time. That precaution was intended to apply only to the pump crankcase, but was interpreted by the operators to also include the gear reducer reservoir. It was also not intended to apply when replacing a sample drawn from either reservoir. Since the 17 hours1.967593e-4 days <br />0.00472 hours <br />2.810847e-5 weeks <br />6.4685e-6 months <br /> occurred over two shifts, this required a subsequent shift to verify that the pump / reducer oil levels were adequate. The easily readable oil levels on the pump crankcase would indicate that oil needed to be added, whereas the harder to read level in the gearbox indicator could be mistakenly determined to be adequate because some of the oil is always retained in the cup. Additionally, operator interviews indicated that some operators were not aware that there were two reservoirs per pump. The instructions for drawing a sample required refilling the reservoir as required by the operating procedure, however this step was found to be marked "N/A" in several instances, apparently due to the procedural 17-hour precaution.
Analysis of the Event
This event had no impact on the health and safety of the public. The SLC System is manually initiated from the main control room, as directed by the Emergency Operating Procedures (EOPs), if the operator believes the reactor cannot be shutdown or maintained shutdown with the control rods. Such a situation constitutes a beyond design basis Anticipated Transient Without Scram (ATWS) event. The ATWS evaluation assumes the beyond design basis failure of the Reactor Protection System (RPS) to initiate a reactor trip in conjunction with the anticipated operational occurrences described in the UFSAR. The primary success path for an ATWS event is the automatic Recirculation Pump Trip (RPT) and Alternate Rod Insertion (ARI) System. The evaluation indicates that RPT results in an immediate substantial reduction in power. ARI provides a diverse method from RPS for initiating control rod insertion. SLC System would only be necessary if ARI was unsuccessful. The evaluation conservatively assumes ARI failure and relies on the SLC System as a backup.
The SLC System is designed to provide the capability of bringing the reactor, at any time in a fuel cycle, from full power and minimum control rod inventory (which is at the peak of the xenon transient) to a subcritical condition with the reactor in the most reactive, xenon free state without taking credit for control rod movement.
The SLC System is used in the event that enough control rods cannot be inserted to accomplish shutdown and cooldown in the normal manner. The SLC System injects borated water into the reactor core to add negative reactivity to compensate for all of the various reactivity effects that could occur during plant operations. To meet this objective, it is necessary to inject a quantity of boron, which produces a concentration equivalent to 720 ppm of natural boron, in the reactor coolant at 70°F. To allow for potential leakage and imperfect mixing in the reactor system, an additional amount of boron equal to 25% of the amount cited above is added.
The volume versus concentration limits for the SLC tank in Technical Specifications Figure 3.1.7-1 and the temperature limit in SR 3.1.7.2 are calculated such that the required cold shutdown concentration is achieved accounting for dilution in the RPV with reactor water level at Level 8 and including the water volume in the residual heat removal shutdown cooling piping and in the recirculation loop piping. This quantity of borated solution is the amount that is above the pump suction shutoff level in the boron solution storage tank. No credit is taken for the portion of the tank volume that cannot be injected. It takes about 20 minutes at minimum boron concentration and flow rate for one pump to bring the reactor to hot shutdown (dilution water in the recirculation and cold shutdown lines not assumed). It takes about 48 minutes of SLC pump run time to inject the cold shutdown boron weight, and about another 12 minutes (or 60 minutes total) to provide an additional 25% boron weight to account for system leakage and imperfect mixing, as discussed in the UFSAR.
The SLC System is also used to maintain suppression pool pH at or above 7 following a Loss of Coolant Accident (LOCA) involving significant fission product releases. Following a LOCA involving significant fission product release, offsite doses from the accident will remain within 10 CFR 50.67, "Accident Source Term," limits provided sufficient iodine activity is retained in the suppression pool. Credit for iodine deposition in the suppression pool is allowed as long as suppression pool pH is maintained at or above 7. The SLC post LOCA pH control function is ensured if the minimum SLC tank volume of 2715 gallons of minimum concentration boron solution is pumped into the reactor at the minimum flow rate of 41.2 gpm. This takes 66 minutes of run time from one SLC pump and must be completed within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> of a LOCA to ensure adequate pH control.
SLC Pump A remained available during the 14-1/2 week period of SLC Pump B inoperability and was capable of performing the SLC injection if needed. However, the overall system reliability was reduced because a single failure in the remaining subsystem could result in reduced SLC System shutdown capability. Even if SLC Pump A were lost for some reason, the 22 minute SLC Pump B run on October 24, 2004 demonstrated that Pump B was capable of delivering enough boron to bring the reactor to hot shutdown. If the B pump were to have immediately failed after 22 minutes of operation, because the hot shutdown condition was achieved, time would be gained to either resolve the SLC-A pump functionality, find a way to insert some rods into the core, or add additional boron using the emergency operating procedure for alternate boron injection using the plant's standby feedwater system. The overall effect of SLC Pump B inoperability on plant risk over the 14-1/2 week period has been reviewed and determined to be of very low safety significance.
Corrective Actions
The pump was reassembled, lubricated, and tested on November 2, 2004. Vibration returned to normal levels during this test, and the pump was declared operable.
Immediate actions included extent of condition walkdowns to look for equipment important to safety with similar oil level indicators. None of the pumps observed had signs of low oil level. However, a number of pumps were identified where an easily readable level indicator was not provided.
A night order was issued to share the lessons learned from this event with Operations personnel. The night order requires: 1) that operators verify that the fluid level is near the top of the standpipe (cup) on the SLC gear
- reducers to ensure reservoir oil level is adequate, 2) that Performance Surveillance and Tracking (PST) event forms requiring oil to be added to replace an oil sample may not be marked "N/A" and replacement oil must be added, 3) that entries must be made to the oil use database anytime oil is added or removed from a plant component, and 4) that the SLC pump 17-hour precaution on oil addition applies only to the SLC pump reservoir (not the SLC gear reducer) and does not preclude adding the same volume of oil back to the pump that was removed for sampling and purge.
This event has been documented in the Fermi 2 corrective action program, CARD 04-25097. Procedures and instructions for maintaining oil level are being evaluated to remove any ambiguity as to the intent of adding oil to replace sample oil volumes, properly recording such events on the oil use database, and for no longer allowing the marking of "N/A" oil replacement steps. Changes to the training program, dissemination of lessons learned from this event, replacement of the SLC reducer level indicators, and the duration of the night order are also being evaluated under this CARD. Any further corrective actions identified as a result of these evaluations will be tracked and implemented commensurate with the established processes and priorities of the corrective action program.
Additional Information
A. Failed Components:
Component: Standby Liquid System Pump Gear Reducer ' Function: Provide boron injection to the reactor Manufacturer: Union Pump Company Model Number: Type TD-60 Failure Cause: Less than adequate lubrication B. Previous LERs on Similar Problems:
LER-2001-01 describes an event where an Emergency Diesel Generator was determined to be inoperable for a period longer than that allowed in the technical specifications as a result of low bearing oil level. In that case, it was determined that even though the oil level was maintained in the normal range as indicated by "green bands," the as-installed bands did not reflect the vendor oil level recommendations. Corrective actions associated with that event focused on verifying the adequacy of the indicated normal range "green bands" on equipment with oil level sight glasses. Since the SLC pumps had sight glasses for the pump crankcase, those level gauges were included in the reviews performed at that time. However, since the SLC Gear Reducers did not have typical sight glass level indicators, they were not included in the corrective actions associated with this event.