05000483/LER-2009-001
| Docket Numbersequential Revmonth Day Year Year Month Day Yearnumber No. | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| 4832009001R01 - NRC Website | |
All times are approximate and Central Standard Time unless otherwise stated.
I.� DESCRIPTION OF THE REPORTABLE EVENT
A. REPORTABLE EVENT CLASSIFICATION
10CFR50.73(a)(2)(i)(A) requires reporting of the completion of any nuclear plant shutdown required by the plant's Technical Specifications. This Licensee Event Report (LER) is submitted accordingly.
B. PLANT OPERATING CONDITIONS PRIOR TO THE EVENT
The plant was in MODE 1 (Power Operation) at 100-percent reactor power at the time the event occurred.
C. STATUS OF STRUCTURES, SYSTEMS OR COMPONENTS THAT WERE INOPERABLE AT THE
START OF THE EVENT AND THAT CONTRIBUTED TO THE EVENT
No structures, systems, or components were inoperable at the start of the event which contributed to the event.
D. NARRATIVE SUMMARY OF THE EVENT, INCLUDING DATES AND APPROXIMATE TIMES
At 0228 on 2/19/2009, multiple annunciators actuated in the control room in response to receipt of a containment purge isolation [EllS system: JE] signal (CPIS), control room ventilation isolation [EIIS system: JE] signal (CRVIS), fuel building isolation [EIIS system: JE] signal (FBIS), steam generator blowdown and sample isolation [EIIS system: WI] signal (SGBSIS), as well as signals indicative of an auxiliary feedwater pump [EIIS system: BA, component: P] suction transfer to the essential service water (ESW) [EIIS system: BI] and a motor-driven auxiliary feedwater pump start. Proper actuations were verified for the CRVIS, CPIS, and FBIS per plant operating procedure OTO-SA-00001, Engineered Safety Feature Actuation Verification and Restoration, which was entered at 0231. Both trains of CRVIS, CPIS, and FBIS actuated per design.
Investigation revealed that the lights on power supplies PS-1, PS-3, and PS-4 [EMS system: JE, component: RJX] were extinguished on cabinet SA036D [EllS system: JE, component: CAB], Balance of Plant (BOP) Engineered Safety Features Actuation System (ESFAS) channel 1 logic cabinet (located in the back of the control room). Although the annunciator actuations were indicative of these three power supply failures, PS-1 directly feeds the other two downstream power supplies, PS-3 and PS-4, such that there was actually only one power supply failure, i.e., PS-1.
The BOP ESFAS system functions to provide certain ESFAS actuation signals including a CPIS, CRVIS, FBIS, SGBSIS, and Auxiliary Feedwater Actuation Signal (AFAS). In this instance, both trains of CPIS, CRVIS, and FBIS logic actuated (such that the affected dampers [EllS system: JM, component: DMP] repositioned, etc.).
The system isolation/actuations were invalid because they were not initiated in response to actual plant conditions, nor were they initiated due to parameters satisfying the requirements for initiation of the safety functions of the systems.
Although the isolation/actuations were invalid, the system isolation and actuations that occurred did occur per design in response to the initiating condition (i.e., the noted power supply failure). However, the inadvertent isolation/actuations only affected containment isolation dampers in one system, Containment Purge, and therefore these inadvertent isolation/actuations are NOT reportable per 10CFR50.73(a)(2)(iv)(A) & (B).
While there was annunciation that the MDAFP had started and that AFW suction had switched to ESW, these alarms were caused by the power supply failure; the actuations normally indicated by these alarms did not actually occur, as previously noted.
In response to the failure of PS-1, and based on the identified actuation/isolation functions affected, the following Technical Specification (TS) Conditions were entered:
AFAS:� 3.3.2.J, 3.3.2.0, 3.3.2.P, 3.3.2.Q, 3.3.2.R and 3.7.5.0 CPIS:� 3.3.6.B and Final Safety Analysis Report (FSAR) Specification 16.11.2.4 CRVIS:� 3.3.7.A FBIS:� 3.3.8.A Shortly thereafter, the Operations crew realized that they failed to take Action per Condition 3.3.2.0, which requires tripping an inoperable LSP channel within 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />. Entry into TS Limiting Condition for Operation (LCO) 3.0.3 was thus made and logged. However, it was subsequently confirmed that the loss of power to the bistable module placed it in a tripped condition, and therefore, entry into LCO 3.0.3 was not required.
In addition, the Operations crew should have entered TS 3.3.6 Condition A for CPIS; however, after the expiration of its 4-hour Completion Time, default Condition B is required to be entered to assure the specified safety function is satisfied. TS 3.3.6 Condition B was entered immediately as discussed above.
The Required Actions of Condition 3.3.2.Q require the plant to be in MODE 3 (Hot Standby) in 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and MODE 4 (Hot Shutdown) in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. Load reduction began at 0530 on 2/19/2009, with MODE 3 being reached at 0817. The PS-1 power supply was replaced and the system was restored to Operable status at 1009 on 2/19/2009. TS Condition 3.3.2.Q was thus exited at that time.
The failed PS-1 power supply was shipped offsite to a vendor for failure analysis. The failure mechanism was determined to be an end-of-life failure of the power supply due to failed and degraded capacitors [EllS system: JE, component: CAP].
Due to non-related, equipment issues experienced during the plant shutdown, and in order to perform additional discretionary work to enhance unit reliability, a forced outage ensued and the plant was maintained in a shutdown condition, i.e., stable in MODE 3. The plant entered MODE 2 (Startup) at 1556 on 3/1/2009, and MODE 1 at 2218. At 1312 on 3/3/2009, the main electrical generator [EMS system: TB] was paralleled to the grid and Callaway Plant was returned to service.
E. METHOD OF DISCOVERY OF EACH COMPONENT, SYSTEM FAILURE, OR PROCEDURAL ERROR
Given the multiple annunciators at the onset of this event, the condition was self-revealing. Causal factors, as well as a root cause, were discovered through the use of a seven-step root cause analysis, with assistance from the vendor who performed the failure analysis.
II.0 EVENT DRIVEN INFORMATION
A. SAFETY SYSTEMS THAT RESPONDED
Not applicable for this event.
B. DURATION OF SAFETY SYSTEM INOPERABILITY
No structures, systems, or components were inoperable during the event which contributed to the event.
C. SAFETY CONSEQUENCES AND IMPLICATIONS OF THE EVENT.
This event was determined to be of very low risk significance. At no time was the plant in an unanalyzed condition. The events described in this LER involve a reactor shutdown required by TS. The shutdown was completed without incidence within the allotted time. During the shutdown, no equipment malfunctions or human performance errors were encountered that would have placed the plant in a condition outside of its safety analysis. Therefore, this event has been determined to be of low safety and risk significance.
III. CAUSE(S) OF THE EVENT AND CORRECTIVE ACTION(S)
This event was evaluated using a root cause analysis (RCA) process. An RCA team was assembled to review this event, determine the cause of the event, and develop corrective action to prevent recurrence.
The RCA team identified two Causal Factors (CFs) with three Root Causes (RCs). This issue has been entered into Callaway's corrective action program. One Corrective Action to Prevent Recurrence (CATPR) and two Corrective Actions (CAs) were identified.
The first CF identified was that no actions were taken in response to the existing condition monitoring preventive maintenance (PM) results (i.e., voltage and ripple checks). According to EPRI guidelines, PM includes predictive and periodic actions to improve equipment reliability and availability. The goal of predictive maintenance activities is to collect, trend, and analyze equipment operating data and process parameters to initiate maintenance activities for degrading equipment prior to failure, thereby minimizing unplanned corrective maintenance activities. Callaway's PM strategy for these power supplies has been based solely on the predictive approach, which was consistent with industry and EPRI guidance several years ago. The predictive approach is predicated on effective trending and monitoring of the data collected.
No periodic replacement strategy had been implemented. The RCA team determined that the condition monitoring data for these power supplies has not been adequately trended or monitored. Therefore, inadequate trending and monitoring of the existing PM data was determined to be a RC for this power supply failure. The CA for this RC is to incorporate System Engineer trending and monitoring of BOP ESFAS and LSELS power supply data into System Trending and Monitoring Plans (STAMPs) to include monitoring frequency and action levels.
The second CF identified was that the power supply was not replaced prior to exceeding its expected service life. The power supply service time exceeded approximately 25 years, which is beyond any reasonable service life expectation. The RC was that Callaway did not have a time-based PM strategy in place to replace these critical power supplies prior to failure. The power supply was allowed to run to failure without any time-based replacement planned. The CATPR for this RC is to generate new PM tasks in order to implement a time-based replacement and refurbishment program for critical power supplies based upon EPRI guidelines and Callaway's Power Supply Life Cycle Management (LCM) study. A CA is to reverse engineer and replace all power supplies currently installed in the BOP ESFAS and Load Shedding and Emergency Load Sequencing (LSELS) system [EIIS system: JE] cabinets. A second RC for this CF was a failure to recognize the generic issues with power supply failures and implement corrective actions. Callaway has had previous opportunities to prevent this power supply failure, but adequate corrective actions were not implemented.
The extent of condition review determined the scope to include a population of fourteen power supplies in the BOP ESFAS and LSELS cabinets. These power supplies are all of similar design, used in similar Consolidated Controls cabinets, and have similar TS implications. Based upon a review of the failure history of the applicable power supplies, the most common failure mechanism appears to be failed or degraded capacitors. The capacitors have failed by opening, shorting, or leaking causing the output voltage to have increased ripple, lower voltage output, or no output.
Until the power supplies identified by the extent-of-condition review can be replaced/refurbished in accordance with the above-noted replacement and refurbishment program, and in light of the limited number of available spares for the current power supplies, contingency actions have been established for the current operating cycle. Work Management has developed on-demand PM work documents to replace any failed BOP ESFAS and LSELS power supply. In addition to these work documents, l&C has staged one spare of each type of power supply such that the spare is maintained energized in the shop. These contingency actions will facilitate prompt replacement if the need arises (i.e., within the most limiting TS 3.3.2 action time of 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br />).
IV. PREVIOUS SIMILAR EVENTS
A search of the Callaway corrective action request system (CARS) identified five occurrences of power supply failures similar to the event on 2/19/2009.
- LER 1992-011: In October 1992, all four field contact power supply output fuses blew causing all main control board annunciators [EllS system: IB] to become inoperable during restoration from replacement of a failed power supply. The cause of the initial failure of the power supply was a short in the power transformer internal to the field power supply. An extent-of-condition review was performed for annunciator power supply internal components. Corrective action called for establishing a replacement program for those power supplies.
- In October 2002, a power supply in the ESFAS channel 4 logic cabinet was found failed. It was suspected that an electrolytic capacitor on the regulating board was degraded and caused the failure. An extent-of-condition review was performed for the power supply components in the BOP ESFAS and LSELS cabinets. As corrective actions, the need for developing a replacement program was identified. Since this program was implemented only a month before the event described in this LER, this program could not have prevented this failure.
- In October 2002, an LSELS power supply failed. The cause of the failure was indeterminate. The power supply age was 20 to 25 years. The corrective actions were to continue voltage and ripple check PMs on these power supplies in order to detect some of the incipient failures. An extent-of condition review was performed for the power supply components in the BOP ESFAS and LSELS cabinets, and the need for a replacement program was identified. Again, since this program was implemented only a month before the event described in this LER, this program could not have prevented this failure.
- In February 2005, a maintenance preventable functional failure for the plant computer system due to inadequate PMs for installed power supplies was identified. Corrective actions included creating a power supply PM document under the Equipment Reliability Improvement Project (ERIP) based on identifying the actions that should be taken to monitor and replace supplies to reduce service failures. These corrective actions were not implemented. The extent of condition and cause were not identified.
- In October 2008, a power supply in the ESFAS channel 4 logic cabinet failed, which led to CPIS, CRVIS, and FBIS inadvertent actuations. The cause of this power supply failure was age-related.
Enhancements included revising the PM basis document for BOP ESFAS and LSELS power supply monitoring, incorporating EPRI recommendations based on replacement frequency and the results of the LCM plan. This enhancement has not yet been implemented. The extent-of condition review included the power supplies in the BOP ESFAS and LSELS cabinets within the identified scope.
V. ADDITIONAL INFORMATION
The power supply that failed is a Sorenson (EPIX code S257) model STM 48-14M20. This is a DC/DC power supply with the following ratings:
Input�105-145 Vdc / 6.7 A Output Nominal 48 Vdc / 10 A The system and component codes listed below are from the IEEE Standard 805-1984 and IEEE Standard 803A-1984 respectively.
System:�BA, Auxiliary/Emergency Feedwater System (PWR) Component: P, Pump System:�BI, Essential Service Water System System:�IB, Annunciator System System:�JE, Engineered Safety Features Actuation System Component: CAB, Cabinet CAP, Capacitor RJX, Power Supply, Electric, Regulated System:�JM, Containment Isolation Control System Component: DMP, Damper System:�TB, Main Generator System System: WI, Steam Generator Blowdown System