ML102250279

Operating Experience Smart Sample (OpESS) FY2010-01 Recent Inspection Experience for Components Installed Beyond Vendor Recommended Service Life

ML102250279
Person / Time
Site:	Hatch, Saint Lucie, Surry
Issue date:	08/13/2010
From:	Jacqueline Thompson NRC/NRR/DIRS/IOEB
To:
Thompson, John, NRR/DIRS/IOEB, 415-1011
References
Download: ML102250279 (9)

Text

OPERATING EXPERIENCE SMART SAMPLE (OPESS) FY 2010-01 Recent Inspection Experience for Components Installed Beyond Vendor Recommended Service Life Note: This is a public NRC document.

Highlighted hyperlinked documents should have active links.

OBJECTIVE:

To provide support and focus for NRC inspector efforts during the conduct of baseline inspections in the area of preventive maintenance (PM) activities. This OpESS provides additional guidance and examples of inspection findings of components that: 1) failed as a result of exceeding vendor-recommended service life, or 2) failed prior to reaching their recommended service life.

NOTE: This OpESS is a voluntary inspection sample performed under the Reactor Oversight Process (ROP) Baseline Inspection Program. When performed, it should be documented in an inspection report under the baseline inspection procedure selected as described in the REPORTING INSPECTION RESULTES/TIME CHARGES/ADDITIONAL ISSUES section below.

BACKGROUND:

Inspection experience over the past five years (2005-2010) has revealed approximately 30 inspection findings and non-cited violations (including 2 greater than green inspection findings) where the root or contributing causes of the failure(s) involved exceeding the vendor recommended service life of components. For some of these failures, the root causes were also attributed to accelerated aging due to environmental conditions or unforeseen causes. The frequency for these types of events that resulted in documented inspection findings doubled over the past 12-month period ending in June 2010.

Operating experience has shown that non-metallic components, such as rubber diaphragms, hoses, cables, lubricants, O-rings, have finite lifetimes that may not be specifically included in licensees periodic PM programs. Inspectors should be aware that Regulatory Guide 1.33, Quality Assurance Program Requirements, Appendix A (Rev. 2), Section 9.b states that:

Preventive maintenance schedules should be developed to specify lubrication schedules, inspection of equipment, replacement of such items as filters and strainers, and inspection or replacement of parts that have a specific lifetime such as wear rings.

Electrical components, such as electrolytic capacitors and relays, often have been in service beyond vendor-recommended life times, where either: 1) no additional evaluation has been completed to justify continued in-service performance, or 2) the component has not been included in the licensees PM program. Capacitors are energy storage devices that are widely used in electronic and electrical power circuits. PM programs are necessary to address adverse 1

effects of aging electrolytic capacitors in equipment circuitry. Licensee failure to do so may lead to operational issues for plant equipment. Aside from the effects of aging, electrolytic capacitors may also exhibit tendencies to be (electronically) noisy; leak, or drift, due to varying environmental conditions (i.e., shifts in temperature, humidity levels, etc.). Extreme temperature conditions can be problematic for capacitors containing aluminum electrolytes. At lower temperatures, capacitance falls off rapidly. At higher temperatures, the electrolyte may be lost from evaporation, thereby accelerating leakage. This may result in premature circuit damage or malfunction. Higher-quality aluminum electrolyte parts have better performance and lifetime characteristics compared to consumer-grade parts.

On June 8, 2010, Surry Unit 1 experienced a reactor trip and safety injection (SI), due to complications associated with the loss of a 120 VAC Vital Bus. The loss of the Vital Bus occurred after the alternate power source had been de-energized for planned maintenance. In addition, it appeared that aged electrical components caused a small fire and smoldering in nuclear Instrumentation (NI) cabinets. EN 45986 On June 16, 2010, St. Lucie Unit 1 operators manually tripped the reactor following two dropped control rods. The cause of the dropped control rods was discovered to be malfunctioning upper gripper coils from failure of two power circuit cards. Root cause indicated capacitor failures on both power circuit cards. The Licensee concluded the capacitors failed due to high temperatures and humidity levels because the (non-safety related) air conditioning system was taken out of service for repairs. The air conditioning system functioned to cool the rod control cabinets which contained the circuitry for the failed power cards. Plant restart conditions created higher than normal humidity levels, causing a "corona effect" of the capacitors, and ultimate failure. EN 46018 Hatch (Unit 2) Failure to Establish Appropriate Preventive Maintenance Program Led to Effects of Aging Electrolytic Capacitors (White Finding, EA-10-009), was issued internally to heighten the awareness to operational impacts on safety-related equipment due to aging of electrolytic capacitors.

Examples of components types that inspectors have found to exceed vendor recommended life times, or were degraded due to accelerated aging beyond reasonable expectations are listed below.

Component types:

Electrical:

1. Capacitive coupled potential device (CCPD) used in high current, low frequency electrical circuits, such as power supply filters and controller circuits.

2. Surge arresters (associated with station transformers)

3. Molded case circuit breakers (MCCBs) and 480 V motor control unit MCCBs (Westinghouse Tech Bulletin TB-04-13)

4. Electrolytic capacitors installed in battery chargers

5. Electrolytic capacitors in safety related SCI-model inverters

6. Relays and Tyco/Agastat E7000 time delay relays used in Engineered Safety Features Actuation System (ESFAS) circuits and in other safety-related electrical circuits (e.g., EDG output breaker closing circuit for EDGs) 2

7. Electrolytic capacitors installed in 120 VAC instrument inverters

8. Submerged safety related cables not evaluated for extended submergence Pumps:

1. Pump couplings

2. 5 KV cables for containment spray and Low Pressure Core Injection (LPCI) pump motors (increased temperatures from conductor temperature rise not accounted for resulting in reduced service life)

3. Safety related service water pump motor controller (effects of temperature not accounted for causing wiring and coil insulation degradation, reduced service life and repetitive failures)

Valves:

1. Safety-related outlet control valve for component cooling water (CCW) heat exchanger (actuator arm assembly not scoped into PM program - vendor identified as a consumable)

2. Manual valve operators that are infrequently operated (i.e., with no preventive maintenance schedule that creates conditions permitting stem rust to occur)

3. Hardened greases, or mixing of greases in motor-actuators Lubricants:

1. Snubber lubricant degradation in high-temperature environments (IN 94-48)

2. Lubricants used in circuit breakers (IN 99-13)

Spent Fuel Racks:

Degraded neutron absorber in the spent fuel pool racks (IN 2009-26)

Fans:

Control room ventilation exhaust fans stretched belts Batteries:

Safety related batteries (inadequate design capacity resulted in reduced service life due to non-conservative design inputs)

Seals:

1. Deck-O-Seal sealant for flood doors and panels.

3

2. Diaphragms in Safety Related Tanks. Referring to Callaway finding of 2002 involving Condensate Storage Tank (CST) diaphragm seal (Conservaflote-vendor, Teflon-coated fabric and inner foam material) degradation causing auxillary feedwater (AFW) pump inoperabilility. Vendor did not specify a seal lifetime. Inspectors determined that there was a finite lifetime for the diaphragm.

3. Asbestos-filled spiral wound gaskets used in pressurizer manway covers.

4

EDGs:

1. Air start hoses

1. Upper O-rings on the fuel oil injection pumps

2. Emergency Diesel Generator (EDG) day tank low level switches

3. T3A time delay relays

4. Buna-N nitrile rubber seals

5. Thermostatic control valve (AMOT) sensing elements for Transamerica Delaval (TDI) EDGs CORNERSTONES: Initiating Events (25%), Mitigating Systems (75%)

APPLICABILITY: All licensed operating commercial nuclear reactors. This OpESS provides additional Operating Experience information and follow-up inspection related to events/ issues involving Components Installed Beyond Vendor Recommended Service Life that could initiate events or result in issues impacting mitigating systems.

INSPECTION GUIDANCE:

The following inspection procedures (IP) include modules that may be applicable to this OpESS:

Baseline inspection program guidance IP- 71111.12 Maintenance Effectiveness IP-71152 Problem Identification & Resolution IP-71111.21 Component Design Bases Inspection General guidance/inspection reference documents (not part of the baseline inspection program)

IP-35743 QA Program (Maintenance)

IP-62700 Maintenance Program Implementation Note: The following inspection guidance may be applied in part or as appropriate to support any of the baseline inspection activities. Resident inspectors should use judgment when determining audit sample size and appropriate hours based on site specific knowledge of known problems or other unique circumstances that may affect sample size, depth, and scope.

1. Ensure that the licensee has procedures for establishing, implementing, and maintaining PM requirements associated with safety related equipment. Deficient maintenance procedures can potentially be a violation of Technical Specifications.

2. Ensure that PM activities are performed as scheduled. When not performed as scheduled, ensure that management controls are followed to defer and/or reschedule the PM. Any 5

equipment failure should be evaluated to determine if the PM program could be changed to prevent future failures.

3. Verify that the licensee was in compliance with these procedures for components that have exceeded vendor recommended life times.

4. Use the Q-list (or the licensees equivalent list of safety-related components that must meet 10 CFR Part 50, Appendix B requirements) to identify safety-related components and sub-components. From the Q-list, conduct an audit to verify comparable components are included in a periodic PM program.

5. Review past equipment failures of the audited components (from the Q-list) for root causes attributable to components or sub-components being left in a system beyond their intended service life.

6. For those components that are beyond vendor-recommended life, use licensee procedures governing PM practices for safety-related components to verify that the licensee has:

a PM program that includes these components, the PM program is adequate and robust and incorporates accepted industry practices (e.g., R.G. 1.33), and the licensee has conducted an appropriate assessment for age-related issues for components installed beyond vendor-recommended life through periodic testing or an engineering evaluation that has accounted for environmental effects (elevated temperatures, humidity, harsh environments).

7. For components that do not have vendor-recommended life times, ensure that the licensee has addressed possible issues with aging of these components through engineering evaluations and has incorporated any concerns into periodic surveillance and maintenance programs.

Note: some equipment, such as batteries, cables, and other electrical components, have calculations that estimate expected service life. If elevated temperatures and other hazardous conditions, such as submergence, or unusual operational demands (i.e., abnormal or asymmetric loading), have not been properly accounted for, then estimated service life can be reduced and result in situations where the components may fail earlier than predicted.

The following are examples of inspection findings from previous inspection reports that are representative of the types of problems seen by inspectors for components that are installed beyond vendor recommended life.

Examples of Findings Example 1: White Finding and Technical Specification Violation 4.4.1 The NRC identified an apparent violation of Technical Specification 4.4.1 for the failure to establish, implement, and maintain preventive maintenance requirements associated with safety related relays. The team identified that [the licensee] did not implement a performance 6

monitoring program specified by the Licensee in Engineering Service Package [X] in lieu of a previously established (in 1987) 10 year service life replacement PM requirement for the EDG T3A time delay relay. As a consequence, the EDG failed to run following a demand start signal.

Following identification of the failed T3A relay, it was replaced and the EDG was satisfactorily tested and returned to service. In addition, time delay relays used in other EDG protective circuits, that also exceeded the vendor recommended 10 year service life, were replaced. The

[licensee] entered this issue, including the evaluation of extent-of-condition, into the corrective action program. This finding is more than minor because it is associated with the equipment performance attribute of the Mitigating Systems Cornerstone and adversely impacted the objective of ensuring the availability, reliability, and capability of the safety related EDG to respond to a loss of normal electrical power to its associated safety bus. This finding was assessed using IMC 0609, Appendix A and preliminarily determined to be White (low to moderate safety significance) based upon a Phase 3 Risk Analysis. The cause of this finding is related to crosscutting area of Human Performance, Resources aspect H.2(a) because preventive maintenance procedures for the EDGs were not properly established and implemented to maintain long term plant safety by maintenance of design margins and minimization of long standing equipment issues.

Example 2: Green Finding and Non-Cited Violation of 10 CFR Part 50, Appendix B The team identified a Non-Cited Violation of 10 CFR Part 50, Appendix B, Criterion XI, Test Control, having very low safety significance for failure to implement a testing program to ensure that the installed safety-related and important-to-safety molded case circuit breakers (MCCBs) will perform satisfactorily in service. This issue was entered into the licensees corrective action program. The licensee was planning to purchase new test equipment and commence testing a statistical sample of the installed MCCBs to corroborate MCCB operability. The finding was more than minor because not testing installed MCCBs that were beyond the manufacturers design life would result in the inability to detect for breaker degradation or failure and would affect availability and reliability of safety systems that are needed to respond to initiating events.

This condition could affect breaker coordination, over-current protection, fire prevention, and multiple other safety-related and important to safety functions. The finding was of very low safety significance because the licensee determined the issue was a qualification deficiency confirmed not to result in loss of operability per Part 9900, Technical Guidance, Operability Determination Process for Operability and Functional Assessment. The cause of the finding was related to the cross-cutting aspect of problem identification and resolution.

Example 3: Green Finding and non-cited TS Violation of 5.4.1.a The inspector identified a non-cited violation of Technical Specification 5.4.1.a for the failure of operations and engineering personnel to establish and implement maintenance procedures for inspection and replacement of items that have a specific lifetime. Specifically, operations and engineering personnel failed to inspect or replace the emergency diesel generators fuel oil injection pump upper O-rings prior to the end of their service life resulting in fuel leakage and increased unavailability and unreliability of Unit 1 Train A, Unit 2 Train B, and Unit 3 Train B emergency diesel generators. This issue was entered into the licensee's corrective action program. This finding is greater than minor because it is associated with the equipment performance attribute of the mitigating systems cornerstone and affects the cornerstone 7

objective of ensuring the availability and reliability of systems that respond to initiating events to prevent undesirable consequences. Using the Manual Chapter 0609, "Significance Determination Process," Phase 1 Worksheets, the finding is determined to have very low safety significance because it did not represent a loss of system safety function, an actual loss of safety function of a single train for greater than its technical specification allowed outage time, or screen as potentially risk-significant due to a seismic, flooding, or severe weather initiating event. This finding has a crosscutting aspect in the area of problem identification and resolution associated with operating experience because the licensee failed to use available operating experience, including vendor recommendations, to implement and institutionalize operating experience through changes to station processes, procedures, equipment, and training programs.

Relevant Generic Experience IE Bulletin 76-06, Diaphragm Failures in Air Operated Auxiliary Actuators for Safety/Relief Valves IN 96-04, Boraflex Degradation in Spent Fuel Pool Storage Racks IN 91-85, Revision 1, Potential Failures of Thermostatic Control Valves Fore Diesel Generator Jacket Cooling Water IN 99-13, Insights from NRC Inspections of Low - and Medium - Voltage Circuit Breaker Maintenance Programs IN 2005-21, Plant Trip and Loss of Preferred AC Power From Inadequate Switchyard Maintenance IN 94-71, Degradation of Scram Solenoid Pilot Valve Pressure and Exhaust Diaphragms IN 95-22, Hardened or Contaminated Lubricants Cause Metal-Clad Circuit Breaker Failures IN 87-66, Inappropriate Application of Commercial-Grade Components IN 88-89, Degradation of Kapton Electrical Insulation IE Bulletin 86-02, Static O Ring Differential Pressure Switches IN 92-81, Potential Deficiency of Electrical Cables with bonded Hypalon Jackets REPORTING INSPECTION RESULTS / TIME CHARGES / ADDITIONAL ISSUES:

Document any inspection result findings, as applicable, in an integrated inspection report (i.e.,

quarterly inspection report/ PI&R or CDBI report) and reference the title/ OpESS number (example: Operating Experience Smart Sample (OpESS) FY 2010-01 Recent Inspection Experience for Components Installed Beyond Vendor Recommended Service Life.

8

If no findings are identified, document completion of the OpESS using the OpESS number/ title under the applicable inspection attachment (i.e., 1R12, 4AO2, or CDBI report) stating that no findings of significance were identified.

Inspection time for this OpESS is to be charged to the normal baseline procedure under which it is being documented (along with any routine preparation and documentation charge times).

CONTACTS:

For questions regarding the performance, documentation or additional issues related to this OpESS, discuss with your branch chief, or if necessary contact:

John Thompson (NRR/ADRO/DIRS/IOEB), Operating Experience Branch at telephone:

(301) 415-1011, or by e-mail at John.Thompson@nrc.gov Mark S. King (NRR/ADRO/DIRS/IOEB), Operating Experience Branch at telephone:

(301) 415-1150, or by e-mail at Mark.King@nrc.gov 9