05000440/LER-2009-003

From kanterella
Jump to navigation Jump to search
LER-2009-003, Reactor Recirculation Pump Failure Results in Manual Reactor Protection System Actuation
Docket Number
Event date: 10-16-2009
Report date: 12-14-2009
Reporting criterion: 10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown
10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
10 CFR 50.73(a)(2)(iv)(A), System Actuation
Initial Reporting
ENS 45434 10 CFR 50.72(b)(2)(i), Tech Spec Required Shutdown
4402009003R00 - NRC Website
v • d • e

Energy Industry Identification System Codes are identified in the text as [XX].

INTRODUCTION

On October 15, 2009, at 0718 hours0.00831 days <br />0.199 hours <br />0.00119 weeks <br />2.73199e-4 months <br /> the Division 2 Emergency Service Water (ESW) [BI] pump [P] B tripped after 77 minutes of operation. Testing confirmed initial breaker indications that there was an electrical problem with power to the associated pump motor. At approximately 1225 hours0.0142 days <br />0.34 hours <br />0.00203 weeks <br />4.661125e-4 months <br />, with the plant in Mode 1 (i.e., Power Operation) and the reactor operating at 100 percent rated thermal power (RTP), a planned shutdown was commenced due to Division 2 ESW inoperability. A four-hour non­ emergency notification was made in accordance with 10 CFR 50.72(b)(2)(i)(A) at 1447 hours0.0167 days <br />0.402 hours <br />0.00239 weeks <br />5.505835e-4 months <br /> for the initiation of a Technical Specifications (TS) required shutdown and is documented in Event Notification number 45434.

On October 16, 2009, at 0048 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, with the reactor operating at approximately 30 percent of RTP, shutdown of the plant was completed by manual actuation of the Reactor Protection System (RPS) [JC] when the Mode switch was placed in Shutdown in accordance with plant procedures. The plant remained in Mode 3 until 0429 hours0.00497 days <br />0.119 hours <br />7.093254e-4 weeks <br />1.632345e-4 months <br /> on October 19, 2009, when the plant entered Mode 4.

The plant was brought into compliance with TS 3.4.10 Action A.1 at 0045 on October 21, 2009, when an alternate method of decay heat removal was established. Following repairs and testing, the Division 2 ESW pump B was restored to service at 0515 and declared operable at 1358 hours0.0157 days <br />0.377 hours <br />0.00225 weeks <br />5.16719e-4 months <br /> on October 28, 2009.

This event is being reported in accordance with 10 CFR 50.73(a)(2)(i)(A) for completion of any nuclear plant shutdown required by the plant's TS. Additionally, with the loss of the Division 2 ESW B loop, Residual Heat Removal Shutdown Cooling System B was unavailable and during the shutdown TS 3.4.10 Condition A Required Action A.1 could not be complied with, therefore, this event is also being reported in accordance with 10 CFR 50.73(a)(2)(i)(B) as any operation or condition which was prohibited by the plant's TS.

EVENT DESCRIPTION

On October 12, 2009 at 1122 hours0.013 days <br />0.312 hours <br />0.00186 weeks <br />4.26921e-4 months <br />, the failure of a vacuum breaker line on the Fire Protection System resulted in numerous components being wetted in the ESW pumphouse, including the Division 2 ESW pump B motor.

On October 14, 2009, at 1747 hours0.0202 days <br />0.485 hours <br />0.00289 weeks <br />6.647335e-4 months <br />, the Division 2 ESW System was declared inoperable and unavailable for planned maintenance work. The following TS were entered:

  • TS 3.8.1 for Division 2 Diesel Generator
  • TS 3.6.2.3 for Suppression Pool Cooling B
  • TS 3.6.3.3 for Combustible Gas Mixing System B
  • TS 3.7.10 for Emergency Closed Cooling B
  • TS 3.7.4 for Control Room HVAC train B.

� As a preventive measure, the ESW pumps were tested to verify that there was no consequential damage as a result of the wetting. On October 15, 2009, the Division 2 ESW pump B cabling and motor were tested to verify functionality beginning at approximately 0230 hours0.00266 days <br />0.0639 hours <br />3.80291e-4 weeks <br />8.7515e-5 months <br />. Following these tests, at 0601 hours0.00696 days <br />0.167 hours <br />9.937169e-4 weeks <br />2.286805e-4 months <br /> the Division 2 ESW pump B was started for a planned maintenance run. At 0718 hours0.00831 days <br />0.199 hours <br />0.00119 weeks <br />2.73199e-4 months <br />, the ESW pump B tripped for unknown reasons. A decision was made to commence a controlled plant shutdown due to the anticipated investigation and expected repair time exceeding the applicable TS Limiting Condition for Operation completion time. At approximately 1225 hours0.0142 days <br />0.34 hours <br />0.00203 weeks <br />4.661125e-4 months <br />, with the plant operating in Mode 1 at 100 percent RTP, a planned shutdown was commenced due to Division 2 ESW System inoperability.

At 2241 hours0.0259 days <br />0.623 hours <br />0.00371 weeks <br />8.527005e-4 months <br />, during the shutdown evolution, reactor recirculation pump A failed to transfer to slow speed during pump downshift and subsequently tripped. On October 16, 2009, at 0048 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, with the reactor operating at approximately 30 percent RTP, shutdown of the plant was completed by manual actuation of the RPS when the Mode switch was placed in Shutdown in accordance with plant procedures (reference LER 2009-004, "Reactor Recirculation Pump Failure Results in Manual Reactor Protection System Actuation").

The plant remained in Mode 3 until 0429 hours0.00497 days <br />0.119 hours <br />7.093254e-4 weeks <br />1.632345e-4 months <br /> on October 19, 2009, when the plant entered Mode 4.

The plant was brought into compliance with TS 3.4.10 Action A.1 at 0045 on October 21, 2009, when an alternate method of decay heat removal was established. Following repairs and testing, the Division 2 ESW pump B was restored to service at 0515 and declared operable at 1358 hours0.0157 days <br />0.377 hours <br />0.00225 weeks <br />5.16719e-4 months <br /> on October 28, 2009.

CAUSE OF EVENT

Based on cable failure analysis, the cause of the Division 2 ESW pump B trip was determined to be a preexisting manufacturing defect in the power supply cable. The location of the failure confirmed that the failure was not related to the wetting of plant equipment in the ESW pumphouse. The location of the cable failure was approximately 1100 feet from the ESW pumphouse.

The cable that failed was a 5kV rated safety-related Anaconda Uniblend with black Ethylene Propylene Rubber insulated shielded cable [CBL5]. A contributing cause to the cable failure is the effects of the predictive maintenance testing (Direct Current megger/Polarization Index) methodology.

Three tests were conducted within a 30 minute timeframe during troubleshooting. This, coupled with the AC current on motor start, accelerated the degraded cable failure.

Another potential root cause was identified as "water treeing". Water treeing is described as a series of tiny hollow channels that can develop in insulation exposed to water and electrical stress. This was determined based on known submergence of the cable either between manhole dewatering periods or from groundwater intrusion into the conduit. The failure analysis results do not indicate water intrusion affected overall cable insulation integrity. However, industry documents indicate that a wet condition can allow a defect to proceed to failure faster than if it had been dry. It is indeterminate whether this damage mechanism would have caused a failure by itself without a preexisting cable insulation defect.

_NRC FORM 366A (9-2007) PRINTED ON RECYCLED PAPER

EVENT ANALYSIS

The plant's ESW System is comprised of three independent subsystem loops A, B, and C. These open subsystem loops take suction from and return to Lake Erie. Each loop contains a separate pump, which is operated from a preferred power source or a standby diesel generator (DG) [EK].

The B loop supplies cooling water to Division 2 residual heat removal (RHR) [BO], standby DG, and emergency closed cooling water heat exchangers, and a fuel pool cooling heat exchanger. The ESW loops run intermittently for plant evolutions such as waste discharges, chemical treatments, ESW testing, and testing of supported systems. The ESW loops only normally run for extended periods to support plant operations such as shutdown cooling during outages and when necessary to support operation of control complex chillers and ventilation trains.

The following Systems were considered inoperable as a result of the failure of ESW pump B and consequently the Division 2 ESW System inoperability:

  • Low Pressure Coolant Injection C
  • Control Room HVAC Train B
  • Hydrogen Analyzer B
  • Emergency Closed Cooling B
  • Combustible Gas Mixing System B.

Mode 4 shutdown cooling TS require two available methods of decay heat removal. With the loss of the Division 2 ESW capability, RHR B was unavailable for decay heat removal, leaving only one credited decay heat removal system, using RHR A in the shutdown cooling mode. As a result, the plant operators used the full timeframe authorized by TS prior to entering Mode 4 to utilize the decay heat removal capabilities of the Main Condenser for as long as practicable.

The plant operating crews were trained on this method of decay heat removal prior to the shutdown.

The plant remained in this condition until the Main Condenser was verified to be a valid alternative for decay heat removal capabilities while remaining in cold shutdown. Per the plant's defense-in-depth philosophy, the main method of decay heat removal capability was posted as protected equipment, including the associated RPS isolation function and all support systems. During this timeframe, the plant was maintained at a heightened awareness state with work activities strictly controlled. No work was performed in the switchyard, and maintenance that could challenge shutdown cooling was limited and direct oversight was provided. Prior to the subsequent reactor startup, extent of condition testing was performed on Division 1 and 3 ESW trains to verify that a common cause condition did not exist in those trains.

A bounding probabilistic risk assessment was performed for the loss of Division 2 ESW System using the average unavailability model. For this event, the probabilistic risk assessment calculated the change in Core Damage Frequency (CDF) to be 8.56E-07. The Large Early Release Frequency (LERF) is on the order of 1E-08. Events with changes in CDF of less than 1.0E-06 and a LERF of less than 1.0E-07 are not considered to be significant risk events. Based on the probabilistic risk assessment results, this event is considered to be of low safety significance.

�

CORRECTIVE ACTIONS

The failed cable section was replaced with a Unit 2 installed spare. The good sections of the faulted cable were spliced at two locations, thereby eliminating the underground faulted location. Corrective actions were initiated to replace the Division 1 and 2 ESW motor cables, revise cable testing procedures, implement a modification to install a drain path in elastomer seals installed in safety­ related penetrations, generate a preventive maintenance task to replace similar cables at a 20 year frequency, and share lessons learned with engineering and maintenance personnel. Additionally, the lessons learned will be used in testing a Service Water motor cable or a Circulating Water motor cable to determine if water treeing exists, and based on these results, the population of cables will be expanded to test the remaining non safety 5kV and 13kV Anaconda shielded cables in 5kV or 13kV non safety applications.

PREVIOUS SIMILAR EVENTS

A review of Licensee Event Reports and the corrective action program database for the past 3 years did not identify any previous similar events.

COMMITMENTS

There are no regulatory commitments contained in this report. Actions described in this document represent intended or planned actions, are described for the NRC's information, and are not regulatory commitments.

� collection does not display a currently valid OMB control number, the NRC may(See reverse for required number of not conduct or sponsor, and a person is not required to respond to, the digits/characters for each block) information collection.

4. TITLE

Reactor Recirculation Pump Failure Results in Manual Reactor Protection System Actuation 5. EVENT DATE 6. LER NUMBER 7. REPORT DATE 8. OTHER FACILITIES INVOLVED

FACILITY NAME DOCKET NUMBER

MONTH DAY YEAR YEAR MONTH DAY YEAR

FACILITY NAME DOCKET NUMBER

10 16 2009 2009 - 004 - 00 12 14 2009 9. OPERATING MODE 11. THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR 10 CFR : (Check all that apply)

  • 20.2201(b)
  • 20.2203(a)(3)(i)
  • 50.73(a)(2)(i)(C)
  • 50.73(a)(2)(vii) 1
  • 20.2201(d)
  • 20.2203(a)(3)(ii)
  • 50.73(a)(2)(ii)(a)
  • 50.73(a)(2)(viii)(A)
  • 20.2203(a)(1)
  • 20.2203(a)(4)
  • 50.73(a)(2)(ii)(B)
  • 50.73(a)(2)(viii)(B)
  • 20.2203(a)(2)(i)
  • 50.36(c)(1)(i)(A)
  • 50.73(a)(2)(iii)
  • 50.73(a)(2)(ix)(A)
  • 20.2203(a)(2)00
  • 50.36(c)(1)(ii)(A) 1 50.73(a)(2)(iv)(A)
  • 50.73(a)(2)(x)10. POWER LEVEL
  • 20.2203(a)(2)(iii)
  • 50.36(c)(2)
  • 50.73(a)(2)(v)(A)
  • 73.71(a)(4) 30
  • 20.2203(a)(2)(iv)
  • 50.46(a)(3)(ii)
  • 50.73(a)(2)(v)(B)
  • 73.71(a)(5)
  • 20.2203(a)(2)(v)
  • 50.73(a)(2)(i)(A)
  • 50.73(a)(2)(v)(C)
  • OTHER Sp in AFbsm ac low
  • 20.2203(a)(2)(vi)
  • 50.73(a)(2)(i)(B)
  • 50.73(a)(2)(v)(D) or icNR 6 6A

12. LICENSEE CONTACT FOR THIS LER

FACILITY NAME TELEPHONE NUMBER (Include Area Code) Perry Nuclear Power Plant, Lloyd Zerr, Compliance Engineer (440) 280-5274

13. COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT

MANU- REPORTABLE MANU- REPORTABLE CAUSE SYSTEM COMPONENT CAUSE SYSTEM COMPONENTFACTURER TO EPIX FACTURER TO EPIX A AD CNTR G080 Y 14. SUPPLEMENTAL REPORT EXPECTED 15. EXPECTED MONTH DAY YEAR

SUBMISSION

  • YES (If yes, complete EXPECTED SUBMISSION DATE).[ I NO DATE ABSTRACT (Limit to 1400 spaces, i.e., approximately 15 single-spaced typewritten lines On October 15, 2009, at approximately 1225 hours0.0142 days <br />0.34 hours <br />0.00203 weeks <br />4.661125e-4 months <br />, a planned shutdown was commenced due to Emergency Service Water System pump B inoperability. At 2241 hours0.0259 days <br />0.623 hours <br />0.00371 weeks <br />8.527005e-4 months <br />, reactor recirculation pump A failed to transfer to slow speed during pump downshift and subsequently tripped. On October 16, 2009, at 0048 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, with the reactor operating at approximately 30 percent rated thermal power, shutdown of the plant was completed by manual actuation of the Reactor Protection System (RPS) when the Mode switch was placed in Shutdown in accordance with plant procedures.

The reactor recirculation pump A failed to transfer to slow speed because the low-frequency motor­ generator output breaker did not close. This resulted from an intermittent connection of an improperly seated contactor retaining clip on the contactor, caused by a skill-based human performance error during a preventive maintenance task.

Recently related corrective actions have been focused on addressing the inconsistent use of human performance tools by individual workers and the lack of reinforcement of tool usage by supervisors and management. The failed contactor has been replaced and tested. Proper assembly of the contactor was verified.

The safety significance of this event is considered to be low. This event is being reported in accordance with 10 CFR 50.73(a)(2)(iv)(A) for any event resulting in manual actuation of the RPS.

Energy Industry Identification System Codes are identified in the text as [XX].

INTRODUCTION

On October 15, 2009, at approximately 1225 hours0.0142 days <br />0.34 hours <br />0.00203 weeks <br />4.661125e-4 months <br />, a planned shutdown was commenced due to Emergency Service Water (ESW) system [BI] pump [P] B inoperability (reference LER 2009-003, "Completion of Technical Specifications Required Shutdown due to Division 2 Emergency Service Water (ESW) Inoperability"). At 2241 hours0.0259 days <br />0.623 hours <br />0.00371 weeks <br />8.527005e-4 months <br />, reactor recirculation (RRC) [AD] pump [P] A failed to transfer to slow speed during pump downshift and subsequently tripped. On October 16, 2009, at 0048 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, with the reactor operating at approximately 30 percent rated thermal power (RTP), shutdown of the plant was completed by a manual actuation of the Reactor Protection System (RPS) [JC] when the Mode switch was placed in Shutdown in accordance with plant procedures. A four­ hour non-emergency notification in accordance with 10 CFR 50.72(b)(2)(iv)(B) was made at 0148 hours0.00171 days <br />0.0411 hours <br />2.44709e-4 weeks <br />5.6314e-5 months <br /> for actuation of the RPS with the reactor critical and is documented in Event Notification number 45440.

The safety significance of this event is considered to be low. This event is being reported in accordance with 10 CFR 50.73(a)(2)(iv)(A) for any event resulting in manual actuation of the RPS.

EVENT DESCRIPTION

On October 15, 2009, at approximately 1225 hours0.0142 days <br />0.34 hours <br />0.00203 weeks <br />4.661125e-4 months <br />, with the plant in Mode 1 (i.e., Power Operation) and the reactor operating at approximately 100 percent RTP, a planned shutdown was commenced with the intent of completing the shutdown by manual reactor scram in accordance with plant procedures and the Evolution Specific Reactivity Plan. The plant was performing a Technical Specifications required shutdown due to the anticipated time necessary to troubleshoot and repair the cause of the ESW system pump B inoperability.

At 2241 hours0.0259 days <br />0.623 hours <br />0.00371 weeks <br />8.527005e-4 months <br />, with the reactor operating at approximately 40 percent RTP, plant operators attempted to downshift RRC pumps A and B from fast to slow speed. Per the plant's system operating instructions, following the low-frequency motor-generators (LFMGs) [MG] start, the transfer sequence was initiated by simultaneously taking both RRC pump breaker controls 5A and 5B to transfer.

Breakers 5A and 5B opened and RRC pump B Breaker 2B closed as expected and pump B continued in slow speed. RRC pump A Breaker 2A never closed to re-energize the A pump in slow speed. After an approximate 43 second time delay, Breaker 1A tripped open to shutdown LFMG A.

No abnormal conditions were noted by visual inspection at the breaker cubicles or the LFMG Auxiliary Relay Panels immediately following the event.

At 2248 hours0.026 days <br />0.624 hours <br />0.00372 weeks <br />8.55364e-4 months <br />, Feedwater Heater 4 isolated which created some balance of plant perturbations. The Evolution Specific Reactivity Plan contained reevaluation criteria for the trip of RRC pump A.

However, due to the current plant conditions and the time required to revise the Evolution Specific Reactivity Plan, preparations were made for a manual reactor scram. On October 16, 2009 at 0048 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br />, a manual actuation of RPS occurred when the Mode switch was taken to Shutdown at approximately 30 percent RTP vice the planned 19.5 — 22.5 percent RTP with the generator offline.

Following the RPS actuation, the turbine and generator tripped as expected. All control rods were fully inserted into the core. No automatic Emergency Core Cooling Systems or Reactor Core Isolation Cooling system response was required and level control was maintained using the feedwater system with the main condenser being used to remove decay heat. The plant remained in � Mode 3 until 0429 hours0.00497 days <br />0.119 hours <br />7.093254e-4 weeks <br />1.632345e-4 months <br /> on October 19, 2009, when the plant entered Mode 4. Overall level control systems responded as expected and operator response to the scram was consistent with the training that was conducted prior to the planned plant shutdown.

CAUSE OF EVENT

The cause of the RRC pump A failure to transfer to slow speed was due to the LFMG output breaker failing to close. This was caused by an intermittent connection of the retaining clip on the contactor [CNTR] due to not being properly seated. The retaining clip was not properly installed on the 1B33K0131A [GE CR305] contactor because a skill-based error occurred during a preventive maintenance task that inspected the contactor in April of 2007. As part of this maintenance task, the contactor coil is inspected, which would require the retainer clip to be removed from its normal position. No additional work activities were identified after April 2007.

Therefore, the root cause was determined to be less than adequate use of human performance tools in that self-checking was not effectively applied to ensure the intended actions were correct. This was determined to be a skill-based error that involved less than adequate self-checking, peer­ checking, and pre-job briefs of important steps on critical equipment. The preventive maintenance task does not require verification of the retaining clip installation. Additionally, the post maintenance test and 32 additional starts of the LFMG A set had no anomalies.

A contributing cause was determined to be a too narrowly focused operating experience screen which missed a similar industry event that that occurred on February 22, 2006, related to an improper installation of a retaining clip on a similar General Electric Contactor (CR105 versus CR305) within the RPS.

EVENT ANALYSIS

The RRC system provides a forced coolant flow through the core to remove heat from the fuel to allow operation at significantly higher power than would otherwise be possible. The system consists of two recirculation pump loops external to the reactor vessel. A bounding probabilistic risk assessment was performed for the manual scram using the average unavailability model. For this event, the probabilistic risk assessment calculated the change in Core Damage Frequency (CDF) to be 1.69E-07. The Large Early Release Frequency is on the order of 1E-08. Events with changes in CDF of less than 1.0E-06 and a LERF of less than 1.0E-07 are not considered to be significant risk events. Based on the probabilistic risk assessment results, this event is considered to be of low safety significance.

CORRECTIVE ACTIONS

The 1B33K131A LFMG contactor was replaced and proper assembly of the contactor was verified.

A separate root cause investigation was recently performed to determine the causes of undesirable individual and organizational behaviors related to human performance issues. The Human Performance error identified in this event is similar to the errors investigated in the root cause.

Corrective actions were focused on addressing the inconsistent use of human performance tools by individual workers and lack of reinforcement of tool usage by supervisors and management.

� A review of these corrective actions was performed and it was concluded that those were adequate to address the human performance aspects of this event; therefore, no additional human performance corrective actions were initiated.

Corrective actions were initiated to revise the Maintenance Plans for these and similar contactors to add an operation to review/discuss the relevant operating experience as part of the pre-job brief and to add a step to perform peer verification that the retaining clip has been properly seated on the detent. Additionally, corrective actions were developed to address the screening weakness of the operating experience with the section operating experience coordinators and to revise the RRC system operating procedure to add a step to confirm the LFMG Voltage Regulator Output voltage after the supply breaker is closed. Maintenance training will be revised to incorporate lessons learned.

PREVIOUS SIMILAR EVENTS

A review of Licensee Event Reports and the Corrective Action Program database for the past three years was completed for conditions written for RRC pump failures to transfer to slow speed and improper installation of equipment.

Actuation," describes a condition where the RRC pump B failed to transfer to slow speed and subsequently tripped on June 22, 2007. The cause of the RRC pump B failure to transfer to slow speed was a malfunction of the low-frequency motor-generator set control and interlock circuit Agastat time-delay relay. This was due to the auxiliary contact failing to open as a result of less than adequate programmatic controls for relay checkout and calibration. A contributing cause was less than adequate implementation of testing controls for installing relays.

A review of corrective action program documents over the last three years found only condition reports associated with this event and the events reported under LER 2007-007. The corrective actions taken for this previous event could not reasonably be expected to prevent the occurrence of this event.

COMMITMENTS

There are no regulatory commitments contained in this report. Actions described in this document represent intended or planned actions, are described for the NRC's information, and are not regulatory commitments.

�

Retrieved from "https://kanterella.com/w/index.php?title=05000440/LER-2009-003&oldid=202251"