05000440/LER-2002-002

LER-2002-002, Failure of the High Pressure Core Spray Pump to Start

Docket Number
Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(v)(D), Loss of Safety Function - Mitigate the Consequences of an Accident
4402002002R01 - NRC Website

12. LICENSEE CONTACT FOR THIS LER

NA/61E Dale L. Miller, Compliance Engineer TELEPHONE NUMBER (retude Area Code) (440) 280 - 5466

13. COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT

CAUSE SYSTEM COMPONENT IdiANU- FACTUREN

REPORTABLE

TO EFIX - i CAUSE SYSTEM COMPONENT 11,kNU-

TO EPI X

X BG SWGR ABB Y 14. SUPPLEMENTAL REPORT EXPECTED 15. EXPECTED

MONTH DAY YEAR

YES (If yes. complete EXPECTED SUBMISSION DATE). X No 16 ABSTRACT (Limit to 1400 spaces, I.e., approximately 15 single-spaced typewritten lines) On October 23, 2002, at

• 1113 hours, the High Pressure Core Spray (HPCS) pump failed to manually start during routine testing of the HPCS room cooler at the Perry Nuclear Power Plant. At the time of the failure, the HPCS ---syStem Was-inoperable- fd(the-conduct of the't6Sting-and required-Technical Specifitibri -actions-had -already ---- : - been completed. The HPCS pump breaker did not close because the circuit breaker cell switch was cut of adjustment. The root cause of the event was determined to be procedural inadequacy. The maintenance instruction for adjusting the cell switch did not provide adequate criteria for the inspection of the circuit breaker cubicle cell switch when performing maintenance. Following adjustment of the cell switch linkage and verification of proper operation, HPCS was returned to operable status at 1529 hours0.0177 days <br />0.425 hours <br />0.00253 weeks <br />5.817845e-4 months <br /> on October 24, 2002. Redundant Emergency Core Cooling Systems remained available during the unavailability of HPCS, but HPCS is significant from a risk perspective. Therefore, this event is considered to have a low to moderate safety significance. This event was reported to the NRC via the Emergency Notification System (ENF#39311) on October 23, 2002, at 1216 hours0.0141 days <br />0.338 hours <br />0.00201 weeks <br />4.62688e-4 months <br />, in accordance with 10CFR50.72(b)(3)(v)(D) as a condition that could have prevented the fulfillment of a safety function needed to mitigate the consequences of an accident. This LER is submitted in accordance with 10CFR50.73(2)(2)(v)(D) as an event or condition that could have prevented the fulfillment of a safety function of systems, structures or components needed to mitigate the consequences of an accident.

I) INTRODUCTION The High Pressure Core Spray (HPCS) system [BG] is a single train safety system and is one of four Emergency Core Cooling Systems (ECCS) at the Perry Nuclear Power Plant (PNPP). The primary purpose of the HPCS system is to maintain reactor vessel [RPV] water inventory following a small reactor coolant boundary break, which does not depressurize the reactor vessel. HPCS also provides spray cooling during reactor coolant boundary breaks that result in uncovering the reactor core.

- The HPCS system includes a motor-driven centrifugal prienTY[BG:PrOo–vreldd from a safety-related, diesel backed electrical bus. The pump discharges into the reactor vessel through a spray sparger located above the reactor core using associated system piping, valves, controls, and instruments.

The plant was in Mode 1 at 98.3 percent of rated thermal power and at normal operating pressure and temperature at 1113 hours0.0129 days <br />0.309 hours <br />0.00184 weeks <br />4.234965e-4 months <br /> on October 23, 2002. The HPCS system had been administratively declared inoperable to support periodic testing of the HPCS pump room cooler [BG-CLR]. Technical Specification required actions for inoperability of the HPCS system had been completed. When the pump failed to start, the event was reported to the NRC in accordance with 10CFR50.72(b)(3)(v)(D), as an event or condition that could have prevented the fulfillment of the safety function of structures or systems that are needed to mitigate the consequences of an accident. This LER is submitted in accordance with 10CFR50.73(a)(2)(v)(D) as an event or condition that could have prevented the fulfillment of a safety function of systems structures or components needed to mitigate the consequences of an accident.

II) EVENT DESCRIPTION Testing of the HPCS room cooler on October 23, 2002 had progressed to the step that called for starting the HPCS pump. HPCS system indications appeared normal prior to the attempted pump start. The control room staff attempted to start the pump in accordance with the System Operating Instruction (SDI). When the pump did not initially start, the Unit Supervisor directed the operator to make a second attempt to start the pump, believing that the control switch may not have been held in the start position long enough. Again the pump did not start.

The pump circuit breaker [BG-BKR] EH1304 failed to close when the pump start switch [BG-HIS) was taken to the start position.

A visual Inspection of the breaker and cell switch LBG-SWGR] was conducted.- Thecircuit breaker-was checked _— for proper position in the cubicle, status of the closing springs, and any other readily apparent conditions. Breaker position indication, that is provided through a different set of contacts in the cell switch, indicated that power was available to start the pump. The switchgear cubicle was also inspected for anything that may have caused the failure. All wiring, the auxiliary switches and the cell switch appeared to be normal. Results of this initial inspection were reported to the Control Room. The area adjacent to the switchgear was then roped off to preserve the as-found conditions.

A Condition Report (CR 02-03972) was written to document the event in the PNPP corrective action program, and a work order containing a troubleshooting plan was generated to determine and correct the cause of the failure.

Troubleshooting activities included switch contact voltage checks, breaker/cubicle interface checks, a visual inspection of the breaker, and an operational test of the breaker using a test box. Troubleshooting identified that one of the rotary contacts of the breaker closure circuit cell switch had not engaged to make a proper connection and resulted in an open pump start circuit. The cell switch is a unit that contains eight individually adjustable double-break rotary contacts mounted on a shaft. A mechanical linkage driven by the racking action of the breaker positions the contacts. It was determined that the proper course of action was to restore the breaker to service by performing an adjustment of the cell switch operating linkage. Following adjustment, operation of the cell switch was verified by moving the circuit breaker from the test to the connected position and back. The cell switch contact make up of all contact pairs was verified both visually and electrically.

III) CAUSE OF EVENT The root cause for the failure of the HPCS pump to start is procedure inadequacy. Maintenance instruction GEl- 0135, revision 1, 'ABB Power Circuit Breakers 5kV Types 5HK250 and 5HK350 Maintenance,"did not provide adequate criteria for the Inspection of the cubicle cell and auxiliary switches when performing breaker maintenance. GEI-0135, revision i provided guidance on what to do In the event that a switch requires adjustment; however, the procedure lacked the direction to reach the conclusion that adjustment was required.

--Personnel performing the frispections-were-instrated to-eniiiielif5t-th—e—coTitaCts—iiiEr-FinthEfrat, horizontal position. A note contained in the procedure (GEI-0135, Revision 1) as part of step 5.14.3 stated, "It may be acceptable for contact bars to not be perfectly straight vertical or flat horizontal aligned provided a clear make/break Is observed." This minimizes the significance of the flat horizontal position and allows the use of subjective judgement to exit the step. This inspection was performed in 1994, 1998, and 2002 and did not identify a discrepancy with the position of the cell switch contacts.

As determined during the course of the investigation, the only way to be certain of positive contact following rotation is to ensure that the normally open contacts are in the horizontal position, completely within the viewing window, which equates to approximately ± 100 from horizontal, and observe the contacts during racking of the breaker. Following racking of the breaker, visual observation ensures that the contacts rotate approximately 900 so that the normally closed contacts are in the horizontal position and are completely within the viewing window.

"Completely within the viewing window" means no part of the contact extends beyond the viewing window. The instruction did not provide this objective criterion, and therefore left room for subjective observation resulting in Inconsistent application of a weak standard.

A contributing cause to this event is the variation in the dimensions of the replacement breaker that was installed in 1994 during the performance of a breaker refurbishment program. This conclusion is based on the degree of misalignment observed during the investigation. it is physically impossible for this amount of misalignment to occur during normal operation. The location of the operating pin on the breaker truck and variations in the size of the truck can change the alignment of the operating linkage. This condition was not recognized as a potential problem during performance of the breaker refurbishment program. Satisfactory completion of the maintenance instruction followed by a successful post-maintenance test was determined to be sufficient to ensure reliable performance of the equipment, when in reality, it was not. The true verification is the alignment of the cell switch.

IV) SAFETY ANALYSIS This event is considered to have a low to moderate safety significance.

The Emergency Core Cooling Systems are designed to provide protection against postulated loss of coolant accidents (LOCA) caused by ruptures in primary system piping. They are also designed to ensure that no single active component failure will prevent automatic initiation and successful operation of the minimum required ECCS. The ECCS uses two independent methods (flooding and spraying) to ensure that the reactor core is adequately cooled in the event of a LOCA.

The ECCS consists of three divisions, any two of which have been evaluated to be capable of mitigating the spectrum of breaks in the accident analyses. The ECCS include the HPCS system, the Low Pressure Core Spray (LPCS) system IBM], and the Low Pressure Coolant Injection (LPCI) [80] mode of the Residual Heat Removal (RHR) system. The Automatic Depressurization System (ADS) is also considered an ECCS.

For accidents that do not quickly depressurize the reactor. if HPCS is not available to maintain water level, mitigation includes automatic initiation of ADS in combination with LPCI and LPCS. In such a situation, the ADS is IV) SAFETY ANALYSIS (continued) designed to open the selected safety/reef valves, depressurizing the Reactor Coolant System (RCS) and allowing the LPCI and LPCS to overcome RCS pressure and inject coolant into the vessel. if the break is large, RCS pressure initially drops rapidly, allowing the LPCI and LPCS systems to inject coolant into the core.

The HPCS pump had been operated satisfactorily on August 28, 2002. The failure was discovered on October 23, 2002, at 1113 hours0.0129 days <br />0.309 hours <br />0.00184 weeks <br />4.234965e-4 months <br />, and the system was restored to operable status on October 24, 2002, at _1529 hours,— - resulting in twenty-eight hours and -sikte-e-ri-rniffetet-Of ilnairaiiiliiiitTfr-d-m-raisEavery until return to service. During this period while the system was unavailable, the other ECCS were available.

To assess the impact of the failure on the probability of core damage, an Incremental Conditional Core Damage Probability (ICCDP) was calculated. The unavailability exposure used for this calculation was derived by using half of the time period between the last successful surveillance performance and the time of discovery, after subtracting the time HPCS was not required as a result of a plant outage during the same period. The calculated ICCDP was 3.32E-6. Using Regulatory Guide 1.174, "An Approach For Using Probabilistic Risk Assessment In Risk-Informed Decisions on Plant-Specific Changes To the Licensing Basis,' as a reference, this event is considered to have a low to moderate safety significance since the calculated ICCDP was greater than 1E-6.

V) CORRECTIVE ACTIONS The cell switch linkage for the HPCS pump breaker was adjusted on October 23, 2003 to obtain proper alignment of the switch contacts. Adjustment of the cell switch to obtain proper alignment included shortening the cell switch linkage actuator arm. Proper cell switch operation was verified, both visually and electrically, during breaker racking evolutions. Proper operation of the system was confirmed on October 23, 2002, at 1848 hours0.0214 days <br />0.513 hours <br />0.00306 weeks <br />7.03164e-4 months <br />, when the HPCS pump was started in accordance with the system operating instruction. The system was declared operable at 1529 hours0.0177 days <br />0.425 hours <br />0.00253 weeks <br />5.817845e-4 months <br /> on October 24, 2002.

A walkdown of 5KV safety-related and nonsafety-related switchgear involving 73 breaker cubicles was completed to identify any breaker cell switch contacts not centered in the viewing window. Cell switches that did not satisfy this criterion were documented to require additional adjustment to optimize the cell switch contact alignment. The walkdown of 40 safety-related switchgear cubicles was completed on November 18, 2002. Work requests were submitted for 10 safety-related breakert-that required cell switch adjustmentr--Voltage checks for the-10 safety-------- related breakers that were identified to require cell switch adjustments were completed on December 17, 2002, to verify they were providing circuit continuity. Voltage readings on each side of the switch contact were taken to confirm the breakers were ready to operate. The voltage checks verified cell switch contacts that permit breaker closure were making electrical contact even though their contacts were not optimally centered in the viewing window.

The walkdown of nonsafety-related switchgear involving 33 breaker cubicles was completed on December 4, 2002. A few nonsafety-related breaker cell switches were identified for adjustment and work requests were submitted. In addition, work requests were submitted for breaker auxiliary switches that were identified to require additional adjustment.

Maintenance Instruction GEI-0135 was revised (revision 2) to provide objective criteria for the inspection of the cubicle cell switches when performing breaker maintenance. This procedure revision was made effective on February 25, 2003. The procedure revision clarified instructions, by revising step 5.14, to observe that the breaker auxiliary and/or cat switch contacts are centered in the viewing window prior to installation of the breaker into the cubicle. Guidance in Attachment 9, "Auxiliary Switch and Cell Switch Linkage Adjustment Guide" was also enhanced to include the criteria to ensure the contacts are properly aligned. Guidance for shortening the V) CORRECTIVE ACTIONS (continued) actuator arm was also included in Attachment 9. Revisions 3, 4, 5 and 6 to GEI.0135 were completed to further clarify and correct cell switch adjustment methods. Revision 6 to GEI-0135, made effective on September 12, 2003, established the acceptance criteria for the cell switch contact position as 'completely within the viewing window:

Revision 8 to SOI-R22, "Metal Clad Switchgear5 — 15 KV," was completed and.made effective March 27;2003,---- --- 0 liictu—des-duii.lance to plant operators for visual checks of cell switch contact position when racking in breakers. The revised procedure also states when racking in class 1E 5KV breakers, the breaker, 'may not be declared operable until the breaker is cycled per an approved instruction: Training was conducted for non- licensed operators as documented in CR 02-03972. Discussion included the draft revision of the procedure SOI- R22 and an actual cell switch for hands on application. The non-licensed operator training was completed on March 25, 2003, Revision 12 to S01-R22 clarified the indication to verify proper change-of-state of the cell switch contacts for 5KV breakers as "completely within the viewing window:

Cubicle cell switches for five Division 2 breakers and one Division 3 breaker were adjusted during the ninth refueling outage that was completed May 31, 2003. Two of these breakers provide an automatic closure to support safety system functions. Two additional safety-related breaker cell switches were adjusted in July 2003 and one was adjusted in October 2003. The remaining cubicle cell switch Is scheduled for adjustment during the normal maintenance schedule and will be completed by the end of the tenth operating cycle. The remaining cubicle cell switch is for an electrical bus source breaker.

Condition Report 03-01546 was initiated on March 27, 2003, to address questions that required clarification or additional information that was not provided in the original investigation for CR 02-03972. As a result, root cause revision 1 was prepared and approved on July 22, 2003. The generic implications portion of the root cause was found to be inadequate during an NRC inspection conducted the week of July 28, 2003. Revision 2 of the root cause was prepared to re-examine the generic implications associated with the root and contributing causes of the HPCS pump failure to start. Revision 2 of the root cause was approved on September 13, 2003. Revision 3 of the root cause investigation report was prepared to ensure the completeness of the document and to clarify some details contained in the report. Revision 3 was approved on September 26, 2003.

Electrical maintenance personnel were provided training on adjusting 5KV cell and auxiliary switches. Training was completed in September 2003.

Revision 2 of the root cause, to re-examine generic implications, identified that additional procedures needed revision to ensure the criteria for inspection of adjustable cell and auxiliary switches is consistent. In addition to procedures GE1-0135 and SCI-R22, procedures GEI-009, 'ABB Low Voltage Power Circuit Breaker Types K-600 & K-6003 through K-3000 & K-3000S Maintenance,' GEI-012, "Inspection and Cleaning of Electrical equipment:

and GE1-0136, "ABB Power Circuit Breakers 15KV Type 15HK1000 Maintenance," were revised to incorporate the wording "completely within the viewing window' as the objective criteria to evaluate acceptability of cell switch adjustment. The revised procedures GEI-012 (revision 5), GEI-0135 (revision 6) and GEI-0136 (revision 2) were made effective on September 12, 2003. Procedure 6E1-009 (revision 8) was made effective on September 16, 2003. Procedure $01-R22 (revision 12) was made effective on September 29,2003.

A modification to the HPCS pump circuit wiring has been approved to provide positive Control Room indication through the pump control switch green indicating light of breaker closing circuit integrity. Implementation of this modification is being planned for November 2003.

V) CORRECTIVE ACTIONS (continued) The PNPP 345 KV switchyard utilizes the same type of rotary switch in auxiliary switch applications. During extent of condition reviews, the switchyard auxiliary switches were walked down. As a result, PNPP will coordinate with the FirstEnergy Corporation Northern Region management to communicate the findings of this investigation with regard to the required maintenance of ABB Type U rotary switches and review switches identified In the switchyard that potentially require adjustment. These actions are being addressed in the PNPP corrective action program.

VI) PREVIOUS SIMILAR EVENTS A search of Licensee Event Reports over the past 3 years from the PNPP found that no similar events had been reported. A review of the PNPP corrective action program identified three events.

The HPCS pump failed to start during testing following routine breaker maintenance in January 1998 (reference Potential Issue Form 98-125). The cause of the event was not determined because the breaker was racked out by Operations following the failure to start, and the as found condition of the breaker was lost. When the breaker was racked in, the pump was started successfully and the test of the pump was completed.

The Division 2 Emergency Diesel Generator output breaker cell switch did not change position when the breaker was racked out in March 2001 (reference CR 01-01347). The corrective action for that event would not have been expected to prevent the current event.

A Reactor Feed Booster Pump failed to start in June 2001 (reference CR 01-2441). The investigation found the breaker cell switch was not making good contact. This was corrected by adjusting the switch linkage. A corrective action for that event was to provide awareness training for electrical maintenance personnel. This corrective action by itself was not sufficient to prevent the event that occurred on October 23, 2002.

The events in January 1998 and June 2001 are considered missed opportunities to identify and prevent the failure experienced in October 2002.

Vii) COMMITMENTS No regulatory commitments were identified in this report.

Energy Industry Identification System (EIS) Codes are identified in the text by square brackets p0(1.

Perry Nuclear Power Plant Unit 05000