05000333/LER-2009-007
| Docket Numbersequential Revmonth Day Year Year Month Day Yearnumber No N/A 05000 | |
| Event date: | 07-07-2009 |
|---|---|
| Report date: | 09-08-2009 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability |
| 3332009007R00 - NRC Website | |
BACKGROUND
The Emergency Onsite Power System (El IS System Identifier: EK) Loss of Power (LOP) instrumentation monitors 4.16KV Emergency Buses 10500 and 10600 to ensure timely disconnection from their preferred power sources and re-connection to their onsite emergency power sources in the event of a loss of voltage or a degraded voltage condition. The 4.16 KV Emergency Bus Degraded Voltage Time Delay Relays (EllS Component Identifier: 2) delay disconnection long enough to allow time for the preferred sources to recover to normal voltages, but soon enough to ensure that power remains available to equipment necessary for safe shutdown and accident mitigation. The 4.16 KV Emergency Bus Degraded Voltage Time Delay Relays are "on-delay" type Agastat relays [Model E7012 (004)].
Upon eriergization, the relays delay the transfer of loads from one source to another for a preset period of time. The units feature electro-pneumatic timers that are adjusted by turning a dial-head to the desired value.
4.16 KV Emergency Bus Degraded Voltage Time Delay Relays, 71-27T2-1H0EA03 and 71-27T2-1HOEB03, are the degraded voltage time delay relays for Emergency Buses 10500 and 10600 for a coincident Loss of Coolant Accident (LOCA) condition. The Technical Specification (TS) allowable values range from >1= 8.40 seconds to 4.16 KV Emergency Bus Degraded Voltage Time Delay Relays, 71-27T3-1H0EA03 and 71-27T3-1 HOEB03, are the degraded voltage time delay relays for Emergency Buses 10500 and 10600 for non-LOCA conditions. The TS allowable values range from >1= 41.0 seconds to In October, 2004, a preventative maintenance task replaced all four Agastat Model E7012 (004) degraded voltage time delay relays with new relays of the same model. Prior to their replacement, the relays had a reliable performance history with only one relay exceeding the TS allowable value during "as-found" testing within the previous six years. In October, 2006, "as-found" calibration testing revealed that one relay exceeded the allowable TS limit. In September, 2008, "as-found" calibration testing revealed that three relays exceeded their allowable TS allowable value. All three relays were adjusted back within tolerance as required by the calibration procedure.
A detailed operability evaluation, completed on October 7, 2008, determined that if the drift trends remained constant, 71-27T2- 1HOEB03 would remain in calibration for JAF's next refueling outage (R19). However, 71-27T3-1HOEB03 should be replaced prior to R19 because the drift trend was outside the allowable band. Subsequently, relay 71-27T3-1HOEB03 was replaced during the 2008 refueling outage and the removed relay sent back to the vendor for testing. The review of trend data during preparation of the operability determination revealed that relay 71-27T3-1HOEB03 had a history of exceeding level one acceptance criteria. As such, it was reported under LER 2008-004 on December 5, 2008.
An apparent cause evaluation (ACE) was performed on October 29, 2008 to evaluate the two relays described above in addition to relay 71-27T2-1H0EA03. The ACE identified that the Agastat Model E7012 relays used as the 4.16 KV Emergency Bus Degraded Voltage Time Delay Relays were not the optimal component for this application and are subject to drift outside of TS limits. This conclusion was based on data provided by the vendor which determined that these relays are temperature and voltage sensitive. As temperature and voltage increases, the time delay would increase. Furthermore, Agastat did not detect any drift in the relay sent back. The vendor stated that drift is not a required specification and the relay was still considered acceptable for use.
During the ACE development, a drift rate calculation was performed to predict when the relays could be expected to exceed TS allowable values. The calculation was based on a straight line extrapolation of the worst case observed drift. This yielded an increased calibration frequency of approximately 14 months before TS level one acceptance criteria would be exceeded. JAF's Corrective Action Review Board further reduced the interval to 9 months to allow for additional conservatism.
EVENT DESCRIPTION
On July 7, 2009, with the plant operating at 100% power (Mode 1), an instrument calibration conducted at JAF on the 4.16 KV Emergency Bus Degraded Voltage Time Delay Relays, revealed that the "as-found" delay time for relays 71-2772-1HOEB03, 71-27T3- 1HOEB03, and 71-27T3-1H0EA03 exceeded the TS allowable values. As a result, multiple independent channels of the Emergency Onsite Power System LOP Instrumentation were inoperable and the plant exceeded TS 3.3.8.1 CONDITION A, COMPLETION TIME during the surveillance interval. The Emergency Onsite Power System LOP instrumentation is required to maintain operability of the emergency diesel generators (EDG). Losing both channels of instrumentation caused both trains of EDG to become inoperable for longer than the TS allowed COMPLETION TIME. TS 3.8.1 requires the plant to be in MODE 3 within 14 hours1.62037e-4 days <br />0.00389 hours <br />2.314815e-5 weeks <br />5.327e-6 months <br /> if both EDG's are inoperable.
The event is reportable pursuant to 10 CFR 50.73(a)(2)(i)(B), "Any operation or condition which was prohibited by the plant's Technical Specifications", and 10 CFR 50.73(a)(2)(vii), "Any event where a single cause or condition caused at least ...two independent trains or channels to become inoperable in a single system designed to ...(A) Shutdown the reactor and maintain it in asafe shutdown condition".
An instrument channel calibration of the LOP degraded voltage time delay relays is performed every 24 months in accordance with plant procedure, ISP-91-1, 4KV Emergency Bus (Buses 10500 and 10600) Degraded Voltage Timer Instrument Calibration. During the performance of this procedure, the relay trip and reset functions are verified to be within the "as-found" zone (AFZ) and re-adjusted, if necessary, to within the "as-left" tolerance (ALT), respectively. Relay 71-27T3-1HOEB03 was returned to the vendor for testing in November, 2008. Vendor testing indicated that these relays are temperature and voltage sensitive. As temperature and voltage increase, the time delay would increase. The vendor did not detect any drift or other problems with the relay. The vendor stated that drift is not a required specification and if the relay passed production tests, they would still ship it. Engineering subsequently evaluated this vendor input and concluded that the existing compensatory measure (i.e. increased surveillance frequency) was adequate.
Because the relays are not designed nor tested to be resistant to setpoint drift and the vendor does not consider drift to be a critical parameter, the apparent cause evaluation determined that the relays are not the optimal component for their application in the EDG logic circuit. As part of the corrective actions for LER 2008-004, a modification to replace the relays with solid-state components was planned for installation by August 31, 2009.
On July 7, 2009, during the performance of ISP-91-1, three of the degraded voltage timer relays again exceeded TS allowable values.
This prompted JAF to perform another ACE to investigate the cause. This ACE, completed on August 19, 2009, focused on the organizational and programmatic aspects of the corrective actions developed for the September, 2008 event. The ACE determined that the corrective actions to prevent recurrence were based on less than adequate data. As such, they were ineffective in preventing future failures of the relays due to the same cause. JAF uses a procedure to address the evaluation of plant data obtained from calibrations and surveillance testing. This procedure also addresses the method of data collection. A proper application of the drift rate calculation would have required statistically significant data derived from 10 or more data points. In the October, 2008 drift rate calculation, only three data points were available to be used. Therefore, the calculated 14 month interval was not statistically significant. It was based on a linear extrapolation of the worst case observed drift and therefore not an optimal indicator of drift performance. As a consequence of this, neither the calculated frequency nor the administratively reduced frequency was adequate to prevent recurrence.
CAUSE OF EVENT
The apparent cause of the relay failures prior to the next refueling outage was attributed to inadequate worker practices in implementation of the JAF setpoint drift monitoring process. The decision to wait until the next window of opportunity to replace the time delay relays was based on a drift rate calculation which projected a 14-month interval before any credible expected drift could occur.
The limited amount of data in the sample size required a correction factor to be used which would have resulted in a 4-month interval.
However, even using the correction factors, the calculated value would not have resulted in a high degree of confidence because the sample size was less than 10.
While the assigned engineer attempted to use a logical approach for drift rate projection, the results were based on an insufficient sample size that resulted in data that was not statistically significant. This was attributed to not having sufficient knowledge and experience with the JAF drift evaluation process. A contributing cause to this event was inadequate management practices.
Supervision improperly assigned a task to an individual who was not aware of the various procedures which address the evaluation of plant data during calibrations and surveillances.
EXTENT OF CONDITION
An extent of condition (EOC) review previously performed found that several additional Agastat Model E7012 (004) relays are susceptible to setpoint drift. The EOC review was completed on November 5, 2008. It stated that there was no reason to believe that any of the relays in the identified group had drifted out of tolerance. Performance of the identified relays was found acceptable and no further actions were necessary.
FAILED COMPONENT IDENTIFICATION
Manufacturer:� New Amerace Corporation (Agastat) (Tyco) Model No.:� Model E7012 (004) NPRDS Manufacturer Code:� A348 NPRDS Component Code:� Time Delay Starting (Closing) Relay FitzPatrick Component Id:� 71-27T2-1H0EA03 71-27T3-1H0EA03 71-27T3-1HOEB03
Immediate Corrective Actions:
1. 71-271-2-1H0EA03 and 71-27T3-1H0EA03 were replaced on July 8, 2009 with solid-state electronic timers.
2. 71-27T2-1HOEB03 and 71-27T3-1HOEB03 were replaced on July 9, 2009 with solid-state electronic timers.
As such, all LOP Degraded Voltage Timer Relays have been replaced with suitable relays.
Planned Corrective Actions:
1. Coaching of the individual who performed the drift analysis.
2. Provide a briefing for managers and supervisors on the proper methods in determining appropriate resources when assigning tasks.
ASSESSMENT OF SAFETY CONSEQUENCES
Though the as-found measured values for the degraded time delay relays exceeded the TS upper allowable value, and hence were inoperable, the relays remained capable of performing their safety function. The JAF 4KV Emergency Bus Loss of Voltage, Degraded Voltage, and Time Delay Relay Uncertainty and Setpoint Calculation sets calculated allowable values (CAV) for the LOCA and non- LOCA degraded voltage time delay relays of 9.94 seconds and 58.5 seconds, respectively. The TS allowable value is set lower than the calculated allowable value to provide additional margin to analytical limits. The "as-found" values measured in July, 2009 for the LOCA and non-LOCA time delay relays were within these bounding limits:
Component TS Allowable Value (sec) As-Found Value (sec) Calculated Allowable Value (sec) 71-27T2-1H0EA03 8.95 +1- .55 9.37 9.94 71-27T3-1H0EA03 43.8 +1- 2.8 47.09 58.50 71-27T2-1HOEB03 8.95 +1- .55 9.83 9.94 71-27T3-1HOEB03 43.8 +1- 2.8 47.04 58.50 Therefore, plant barriers providing safety to the public were not compromised and the safety significance of this event is considered low.
SIMILAR EVENTS
LER-2008-004-00 CR-JAF-2008-03546 CR-JAF-2008-03796
REFERENCES
1. JAF Condition Report CR-JAF-2008-03286, "During the performance of ISP -91-1 for the 10600 Bus, 71-27T2-1HOEB03 and 71-27T3-1HOEB03 were found outside (HI) their NO ADJUST range.
2. JAF Condition Report CR-JAF-2008-03546, "During performance of WO 51193792; ISP-91-1 timer was high out of tolerance.
This is a level 1 criteria.
3. JAF Condition Report CR-JAF-2008-03796, "CR-JAF-2008-03286 failed to identify that the effected components failed LEVEL 1 acceptance criteria and the CR was screened incorrectly.
4. JAF Condition Report CR-JAF-2008-04044, "The Extent of Condition (EOC) for CR-JAF-2008-03796 and CR-JAF-2008-03546 found that several additional model Agastat E7012 relays are susceptible to the apparent cause and need operability reviews.
5. JAF Condition Report CR-JAF-2009-02397, "During performance of WO 176967 for ISP 91-1 10600 bus timing for comp ID 71- 27T2-1HOEB03 & 71-27T3-1 HOEB03 were out of tolerance. Adjustments made per procedure sat. This is a level 1 acceptance criteria.
6. JAF Condition Report CR-JAF-2009-02403, "During performance of ISP 91-1 for 10500 bus for comp ID 71-27T3-1H0EA03 was out of tolerance for the as found condition. Adjustments made per procedure sat. This is level 1 criteria.
7. JAF Engineering Calculation JAF-CALC-ELEC-01488, Revision 5, 4KV Emergency Bus Loss of Voltage, Degraded Voltage and Time Delay Relay Uncertainty and Set-Point Calculation.