ML17306B330

LER 93-001-00:on 930204,reactor Trip Occurred Due to SG 2 Water Level Reaching RPS Trip Setpoint for Low SG Water Level Following Loss of Main Feedwater Pump A.Caused by Failure of Circuit Board.Investigation Underway

ML17306B330
Person / Time
Site:	Palo Verde
Issue date:	03/04/1993
From:	Bradish T ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:
Shared Package
ML17306B329	List: ML17306B328 ML17306B330
References
LER-93-001-02, LER-93-1-2, NUDOCS 9303150145
Download: ML17306B330 (20)

Text

LICENSEE EVENT REPORT (LER)FACILITY NAME{1)TITLE (4)Reactor Tri Due to Loss of M i DOCKET NVMSER (2)0 5 0{)0 PAOE OF EVENT DATE{6)LER NUMBER d)REPOR t DATE{7)OTHER FACILITIES INVOLVED (8)MONTH DAY YEAR YEAR NUMBER NUMBER MONTH DAY YEAR FACIUTY NAMES DOCKET NUMBER(S){)5{)0{)OPERATINO MODE (0)POWER LEVEL (10)NAME THIS REPORT IS SUBMITTED PURSUANT 20.402($)20A05(a)(1)g) 20.405(a)(I)gl) 20A05(a)(1)(ie) 20 405(a)(1)(iv) 20.405(a)(1)(v) 51k 73(a)(2)(tv) 50.73(a)(2)(v) 50.73(a)(2)(v6) 50.73(a)(2)(vB)(A)50 73(aX2)(van) 50.73(a)(2)(rr) 20A05(c)5046(c)(I) 5086(c)(2) 50.73(a)(2)(i)50.73(a)(2)gi)50.73(a)(2)(iii)LICENSEE CONTACT FOR THIS LER (1 2)TO THE REOVIREMENTS OF 10 CFR I: (Check one or more of the tolbwtng)(11){)5{)0{)73.7t(tr)73.71(c)OTHER (Specliyln Abed ect belorv anti ln Terr{NRC Form 366A)TELEPHONE NUMBER Thomas R.Brad'.sk'OMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCR ED IH THIS REPORT (13)CAUSE SYSTEM COMPONENT htAN VFAC-TURER SYSTEM COMPONENT MANUFAC-TURER X S J RG 60 8 0 N SUPPLEMENTAL REPORT EXPECTED (14)EXPECTED SVBLISSION DATE (15)MONTH DAY YEAR YES (II yea, complete EXPECTED SUBMISSION DATE)NO ABSTRACT (IJrrit ro f400 epecee, I e., approxlmatety EIteen elnJIe-apace type vrif iten Inc e]{1 6)On February 4, 1993, at approximately 1523 MST, Palo Verde Unit 3 was in Mode 1 (POWER OPERATION), operating at approximately 100 percent power when a reactor trip occurred due to Steam Generator Number 2 (SG-2)water level reaching the Reactor Protection System trip setpoint for low steam generator water level following the loss of Main Feedwater Pump A.Innnediately following the reactor trip, a low steam generator water level alarm annunciated for SG-1, followed by the Engineered Safety Feature Actuation System (ESFAS)actuation of both Auxiliary Feedwater Actuation Systems (AFAS-1 and AFAS-2)on low-low steam generator water level for both steam generators.

The Steam Bypass Control System responded as designed to control the secondary system pressure.The injection of auxiliary feedwater to the steam generators, combined with the quick-opening of the steam bypass control valves (SBCVs), caused reactor coolant temperature to decrease, which resulted in a primary system pressure decrease below the low pressurizer pressure setpoint of 1837 pounds per square inch absolute (psia).Valid ESFAS actuations of the Safety Injection Actuation System (SIAS)and the Containment Isolation Actuation System (CIAS)occurred due to low pressurizer pressure.The event was diagnosed as an uncomplicated reactor trip: By approximately 1559 MST on February 4, 1993, the plant was stabilized in Mode 3 (HOT STANDBY).There have been no previous similar events reported pursuant to 10CFR50.73.

This LER also serves as a Special Report prepared and submitted pursuant to Technical Specification (TS)3,5.2 ACTION b and TS 6.9.2.9'303150145 930304 PDR ADOCK 05000530 S'DR

LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACiLITY NAME Palo Verde Unit 3 TEXT DOCKET NUMBER osooo53093 LER NUMBER gP SEOUENTIAL

<x'UMBER 0 0 1 REVISION 52 NUMBER 0 0 PAOE OF 1 0 DESCRIPTION OF WHAT OCCURRED: A.Initial Conditions:

At 1523 MST on February 4, 1993, Palo Verde Unit 3 was in Mode 1 (POWER OPERATION) operating at approximately 100.percent power.B.Reportable Event Description (Including Dates and Approximate Times of Major Occurrences):

Event Classification:

An event that resulted in automatic actuation of any Engineered Safety Feature (ESF)(JE), including the Reactor Protection System (RPS)(JC)At approximately 1523 MST on February 4, 1993, a reactor (AC)trip occurred when Steam Generator Number 2 (SG-2)(AB)water level reached the Reactor Protection System (RPS)trip setpoint for low steam generator water level following the loss of the steam-driven Main Feedwater Pump (MFWP)A (SJ)(P)..Immediately following the reactor trip, a low steam generator water level alarm annunciated for SG-l, followed by the Engineered Safety Feature Actuation System (ESFAS)actuation of both Auxiliary Feedwater Actuation Systems (AFAS-1 and AFAS-2)(JE)(BA)on low-low steam generator water level for both steam generators.

The Steam Bypass Control System (SBCS)(JI)responded as designed to control the secondary system pressure.The injection of auxiliary feedwater to the steam generators, combined with the quick-opening of four steam bypass control valves (SBCVs), caused reactor coolant (AB)temperature to decrease, which resulted in a primary system pressure decrease below the low pressurizer (AB)pressure setpoint of 1837 pounds per square inch absolute (psia).Valid ESFAS actuations of the Safety Injection Actuation System (SIAS)(JI)(BP)and the Containment Isolation Actuation System (CIAS)(JI)(BD)occurred due to low pressurizer pressure.The Control Room Supervisor (CRS)(utility, licensed)diagnosed the event as an uncomplicated reactor trip.By approximately.

1559 MST on February 4, 1993, the plant was stabilized in Mode 3 (HOT STANDBY).Prior to the reactor trip, at approximately 1522 MST, Control Room (NA)personnel (utility, licensed)observed that the MFWP A high vibration alarm flashed, that the MFWP A speed decreased rapidly from approximately 4900 revolutions per minute (rpm)to approximately 1000 rpm, and that both steam generator water levels were decreasing rapidly.Following an evaluation of plant conditions, the Shift Supervisor (utility, licensed)directed

LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER g~: SEQUENTIAL NUMBER gP<REVISION I NUMBER PAOE o 5 o o o 530 9 3 00 1 0 0 0 30F Control Room personnel to manually trip MFWP A in order to initiate a reactor power cutback (RPCB)(JD)(i.e., an automatic pover reduction from 100 percent to belov 70 percent).MFWP B continued to operate properly.However, before Control Room personnel could complete the MFWP A trip directive, at approximately 1523 MST, a reactor trip occurred when SG-2 water level reached the RPS trip setpoint for low steam generator water level.All control element assemblies

{CEA){AA)inserted as designed.Immediately.

following the reactor trip, a low steam generator water level alarm annunciated for SG-1, followed by AFAS-1 and AFAS-2 ESFAS actuations on low-low steam generator water level for both steam generators'he AFAS actuations initiated auxiliary feedvater injection to both SGs.The SBCS responded as designed to control the secondary system pressure.The injection of auxiliary feedwater to both SGs, combined with the quick-opening of four SBCVs, caused reactor coolant temperature to decrease which resulted in a primary system pressure decrease below the low pressurizer pressure setpoint of 1837 psia.At approximately 1524 MST, valid SIAS and CIAS HSFAS actuations occurred due to low pressurizer pressure.Following'the reactor trip, pressurizer pressure decreased to"a minimum value of approximately 1821 psia, while pressurizer level decreased to approximately 15 percent with pressurizer heater cutout occurring, at approximately 25 percent.as designed.A review of data acquired from the plant monitoring system (IQ)indicated that the SIAS and CIAS ESFAS actuations occurred 41 seconds after the reactor trip.Pressurizer pressure decreased below the discharge head of the safety injection pumps (BP){P)resulting in the injection of borated water into the reactor coolant syst'm (RCS)(AB).Following the reactor trip, pressurizer pressure and level vere restored[i.e., per procedure, in response to the SIAS ESFAS actuation, Control Room personnel manually tripped two of the four reactor coolant pumps (RCP 1B and RCP 2B)(AB)(P), one per steam generator, and pressurizer pressure and level started to recover at a faster rate].The injection of borated water into the reactor coolant system was less than 100 gallons.At approximately 1552 MST, safety injection was throttled and shutdown li.e., per procedure, Control Room personnel stopped Trains A and B Containment Spray pumps (P)(BE), High Pressure Safety Injection pumps (P)(BQ), and Lov Pressure Safety Injection pumps (P)(BP)].The steam generator water levels were restored using auxiliary and main feedwater.

The CRS diagnosed the event

~c LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER o s o o o 53 093 LER NUMBER M SEOUENTIAL Fg NUMBER 001 REVISION NUMBER 0 0 PAOE 0 40F as an uncomplicated reactor trip in accordance with the emergency plan implementing procedures.

By approximately 1559 MST on February 4, 1993, the plant was stabilized in Mode 3 (HOT STANDBY).No other safety system responses occurred and none were required.At'pproximately 1546 MST on February 4, 1993, Unit 3 declared'a Notification of Unusual Event.The Notification of Unusual Event was declared pursuant to Emergency Plan Implementing Procedure (EPIP-02)for'an event resulting in a SIAS ESFAS'actuation caused by a valid low pressurizer pressure.At approximately 1700 MST on February 4, 1993, the Notification of Unusual Event was terminated in accordance with EPIP-03.By approximately 1811 MST on.February 4, 1993, the SIAS, CIAS, AFAS-1, and AFAS-2 ESFAS actuations were reset.CD Status of structures, systems, or<<components that were inoperable at the start of the event that contributed to the event: Not applicable

-no structures, systems, or components were inoperable at the start of the event which contributed to this event.D~Cause of each component or system failure, if"known: An independent investigation of this event (i.e., a reactor trip on low steam generator water level following the loss of feedwater flow from MFWP A)is being conducted in accordance with the APS Incident Investigation Program.As part of the investigation, an equipment root cause of failure analysis (ERCFA)of MFWP A is being performed by APS Engineering personnel (utility, nonlicensed).

Following extensive troubleshooting, a preliminary ERCFA determined that the apparent failure mechanism'as related to the MFWP A main control system circuit board.When the MFWP A main control system circuit board was removed during troubleshooting and reinserted without being well-seated, the MFWP A exhibited similar symptoms (i.e., speed reduction)

.No significant evolutions, maintenance, or troubleshooting activities were in progress that contributed to the failure of MFWP A.Since no other problems were found that could have contributed to the MFWP A coastdown, the MFWP A main control system circuit board was replaced, omnilight recorders were installed on the MFWP A control system to monitor specific test points, and MFWP A was returned to service.

LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITy NAME Palo Verde Unit 3 DOCKET NUMBER 0 5 0 0 0 5 30'ER NUMBER yEAR'>'EQUENTIAL P NUMBER 93 0 01 REVISION NUMBER PAOE OF At approximately 1354 MST on February 17, 1993,,Control Room personnel (utility, licensed)observed that the MFWP A speed decreased rapidly and that both steam generator water levels were'ecreasing rapidly.Control Room personnel manually tripped MFWP A in order to initiate an RPCB.The plant was stabilized in Mode 1 at approximately 65 percent power.Following extensive analysis of the data retrieved from the omnilight recorders, APS Engineering personnel determined that the MFWP A electronic speed governor control system circuit board's-12 vdc voltage regulator failed when the temperature surrounding the circuit board reached approximately 140 degrees Fahrenheit.

The voltage regulator is ,rated to approximately 250 degrees Fahrenheit.

The MFWP A electronic speed governor control-system circuit board was replaced.APS Engineering has determined that voltage regulator failure is limited to the replaced circuit board and that the fail'ure is not generic to Units 1 or 2.Failure mode, mechanism, and effect of each failed component, if known: The ERCFA determined that when the MFWP A electronic speed governor control system circuit board's-12 vdc voltage regulator failed, the MFWP A control valves were driven closed.When the control valves closed, steam to the steam-driven MFWP A stopped, and MFWP A experienced a rapid reduction in speed from approximately 4900 rpm to 1000 rpm (i.e., coastdown).

Because MFWP A did not trip (NOTE: the steam-driven MFWP is designed to trip on low suction or high discharge pressure, not on loss of steam flow), an RPCB (i.e., an automatic power reduction from 100 percent to below 70 percent)signal was not initiated.

Both steam generator water levels decreased rapidly on loss of feedwater flow from MFWP A.Before Control Room personnel'ould complete the MFWP A trip directive, a reactor trip occurred when SG-2 water level reached the RPS trip setpoint for low steam generator water level.For failures of components with multiple functions, list of systems or secondary functions that were also affected: Not applicable

-no failures of components with multiple functions were involved.For a failure that rendered a train of a safety system inoperable,'stimated time elapsed from the discovery of the failure until the train was returned to service: All equipment responded as designed to the RPS and ESFAS actuations with the exception of the SG-1 Train B Auxiliary

LtCENSEE EVENT REPORT (LER)TEXT CONTINUATlON FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER LER NUMBEA 5~~I SEQUENTIAL NUMBER'EVISION I NUMBER PAGE 0 5 0 0 0 9 3 0 0 1 0 0 0 6OF Feedwater regulating valve (AFB-HV30).

AFB-HV30 experienced a loss of power and failed to fully open following the receipt of the ESFAS AFAS-1 actuation.

At approximately 1537 MST on February 4, 1993, Control Room personnel declared the Train B Auxiliary Feedwater pump (AFB-P01)inoperable and entered Technical Specification Limiting Condition for Operation (TS LCO)Remote Shutdown Systems 3.3.3.5 ACTION a and TS LCO Auxiliary Feedwater System (AFWS)3.7.'1.2 ACTION a.Although Train B AFWS was declared inoperable, the AFWS was still able to perform its safety function (i.e., maintain feedwater inventory to the affected SGs during operation when the main feedwater system is inoperable)

~Upon receipt of an AFAS-l, both essential AFWPs (AFA-P01 and AFB-P01)start and discharge through downstream motor operated crossover valves (regulating valves AFA-HV32 and AFB,-HV30) into the MFWS downcomer supply lines.The crossover valves allow each AFWP to supply the affected SG (i.e., SG-1).The two motor operated regulating valves,(AFA-HV32 and AFB-HV30)were designed'o open automatically upon receipt of an ESFAS AFAS-1 actuation signal.The Train A AFWS regulating valve (AFA-HV32) continued to operate properly and supply feedwater flow to SG-1.AFB-HV30's motor contiol center (MCC)breaker and motor operated valve (MOV)were quarantined.

An ERCFA was initiated.

The valve's MCC breaker was found in the trip position and the valve was found to be approximately five percent open.The preliminary ERCFA determined that the breaker's instantaneous overcurrent magnetic trip device had operated and apparently tripped the breaker and that the MOV was stroking in the open position when the breaker tripped.Following pulse current injection testing, the magnetic trip devi.ce's phase C trip setting of 29.5 amps was found to be below the acceptance criteria of 31.5 amps.Phase A and phase B settings were found at 33.0 amps.The breaker's instantaneous overcurrent magnetic trip setpoints were initially set on March 18, 1986, during plant startup.The three phases A, B, and C were set at 38, 40, and 43 amps, respectively, The settings have not been adjusted since plant startup.However, on October 31, 1989, during the AFB-HV30 MCC monthly relay functional test, AFB-HV30's valve motor faulted (i.e., the motor was found to be solidly grounded), and the MCC breaker tripped and would not reset, until the ESFAS individual subgroup pushbutton was released.Following troubleshooting, the motor was replaced.A visual inspection of the breaker was performed and no problems were found.The postulated degradation of the magnetic trip settings may have occurred when the motor faulted.

.0 LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER LER NUMBER'j"I SEQUENTIAL P REVISION NUMBER PAGE TEXT 05ooo OF No other problems (i.e., Phase C below acceptance criteria)have been found that could, have contributed to the, failure of AFB-HV30.to fully open.The MCC breaker was replaced.In addition, weekly valve stroking is being performed until the ERCFA is completed.

APS Engineering is expected to complete the ERCFA by April 15, 1993.If the completed ERCFA results differ significantly from this determination, a supplement to this report will be submitted to describe the final ERCFA.Additional problems discovered during the performance of the AFB-HV30 ERCFA are being investigated under the APS Incident Investigation Program.These include discrepancies found in the MCC breaker overcurrent magnetic trip relay setpoints and the possible replacement of MCC breakers following a fault interruption.

The investigations and corrective actions will be tracked under the Commitment Action Tracking System.At approximately 1116 MST on February 7, 1993, Control Room personnel declared the Train B Auxiliary Feedwater System operable and exited TS LCO'emote Shutdown Systems 3.3.3.5 ACTION a and TS LCO Auxiliary Feedwater System (AFWS)3.7.1.2 ACTION a.H: Method of discovery of each component or system failure or procedural error: As discussed in Section I.D, the failure of the MFWP A electronic speed governor control system circuit board's-12 vdc voltage regulator was discovered during troubleshooting following a recurrence of the February 4th event.As discussed in Section I.G, the failure of the AFB-HV30 to fully open following the receipt of the ESFAS AFAS-1 actuation was discovered following the event.At approximately 1537 MST on February 4, 1993, Control Room personnel declared the Train B Auxiliary Feedwater pump (AFB-POl)inoperable and entered TS LCO 3.3.3.5 ACTION a and TS LCO 3.7.1.2 ACTION a.There were no procedural errors which contributed to this event.Cause of Event: An independent investigation of this event (i.e., a reactor trip on low steam generator water level following the loss of feedwater flow from MFWP A)is being conducted in accordance with the APS Incident Investigation Program.As part of.the investigation, an ERCFA of MFWP A is being performed.

by APS Engineering personnel.

As discussed in Section I.D and I.E, the ERCFA has'determined that the apparent failure mechanism is related to a failure of the MFWP A electronic speed governor control system circuit board's-12 vdc l

LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER LER NUMBER<%~SEOUENTIAL Fj NUMBER REVIBION NUMBER PAGE osooo OFI P voltage regulator (SALP Cause Code E: Component Failure).No additional problems were found that could have contributed to the MFWP A coastdown.

No significant evolutions, maintenance, or troubleshooting activities were in progress that contributed to the failure of MFWP A.No unusual characteristics of the work location (e.g., noise, heat, poor lighting)directly contributed to this event.There were no personnel or procedural errors which contributed to this event.J~Safety System Response: The following safety systems actuated automatically as a result of the event: Emergency Diesel Generators (EK), Trains A and B, Essential Spray Pond Systems (BS), Trains A and B, Essential Chilled Water System (KM)', Trains A and B, Essential Cooling Water System (BI), Trains A and B, High Pressure Safety Injection (BQ), Trains A and B, Low Pressure Safety Injection (BP), Trains A and B, Containment Spray System (BE), Trains A and B,.Essential Auxiliary Feedwater System (BA), Trains A and B, Containment Isolation System (JM), Control Room Essential Heating, Ventilation and Air Conditioning (HVAC)System (AHU), Trains A and B, Auxiliary Building Essential HVAC System (AHU)(VF), Trains.A and B, and Fuel Building Essential HVAC System (AHU)(VG), Trains A and B.K.Failed Component Information:

The MFWP A electronic speed governor control system circuit board's-12 vdc voltage regulator is manufactured by General Electric.The model number is MDT-20.II.ASSESSMENT OF THE SAFETY CONSEQUENCES AND IMPLICATIONS OF THIS EVENT: l A safety limit evaluation was performed as part of the APS Incident Investigation Program.The evaluation determined that the plant responded as designed, that no safety limits were exceeded, and that the event was bounded by current safety analyses.The event reported by this LER (530/93-001) is bounded by the Palo Verde Updated Final Safety Analysis Report (FSAR)Chapter 15 accident scenarios concerning decreases in heat removal by the secondary system.In addition, the Updated FSAR Chapter 6 scenarios concerning loss of coolant accidents were not challenged by this event.

I I~LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER YEAR LER NUMBER SEQUENTIAL NUMBER 5'EVISION~c4 NUMBER PACE'5 0 0 0 OF The impact of the transients (i.e., concurrent decrease in primary system temperature and pressure and pressurizer level)posed no threat to fuel integrity as adequate subcooling margin and RCS inventory were maintained throughout the event.The maximum RCS pressure recorded during the event was 2300 psia, which did not exceed the 2750 psia safety limit.There were no Departure from Nucleate Boiling Ratio (DNBR)related fuel failures since the Specified Acceptable Fuel.Design Limit (SAFDL)for DNBR was not exceeded during the event.Therefore, there were no safety consequences or.implications as a result of this event.This event did not adversely affect the safe operation of the plant or health and safety of the public.III.CORRECTIVE ACTION: Immediate:

MFWP A and AFB-HV30 were quarantined pending troubleshooting.

B.Action to Prevent Recurrence:

An independent investigation of this event is being conducted in accordance with the APS Incident Investigation Program.As part of the investigation, an ERCFA of MFWP A is being"p'erformed by APS Engineering personnel.

APS Engineering personnel determined that the'FWP A ele'ctronic speed governor control system circuit board's-12 vdc voltage regulator failed.The MFWP A electronic speed governor control system circuit board was replaced.APS Engineering has determined that voltage regulator failure is limited to the replaced circuit board and that the failure is not generic to Units 1 or 2.I APS Engineering is expected to complete the ERCFA by April 15, 1993.If the, completed ERCFA results differ from this determination, a supplement to this report will be submitted to describe the final ERCFA.The corrective actions for the failure of AFB-HV30 to fully open are discussed in Section I.G.IV.PREVIOUS SIMILAR EVENTS: Although reactor trips related to MFWPs have been previously reported, no other previous events have been reported pursuant to 10CFR50.73 where a MFWP did not trip as the pump speed, flow, and discharge pressure l

Vt LICENSEE EVENT REPORT (LER)TEXT CONTINUATION FACILITY NAME Palo Verde Unit 3 DOCKET NUMBER YEAR SEQUENTIAL NUMBER REVISION NUMBER PAOE TEXT 0 5 0 0 0 5 3 0 001 001 oF10 approached zero (NOTE: the steam-driven MFWP is designed to trip on low suction or high discharge pre'ssure, not on loss of steam flow).Because MFWP A did not trip, an RPCB (i.e., an automatic power reduction from 100 percent to below 70 percent)was not initiated and the plant tripped on low steam generator water level.V.ADDITIONAL INFORMATION:

Based on reviews by the Plant Review Board (PRB), the Management Response Team, and the Incident Investigation Team, unit restart was authorized by the Plant Manager in accordance with approved procedures.

Based on PRB approval, the unit was restored to 100 percent power with both MFWP A and MFWP B in service.On February 7, 1993, Unit 3 entered Mode 2 (STARTUP)at approximately 1453 MST and Mode 1 at approximately 1832 MST, and was synchronized on the grid at approximately 1512 MST on Febiuary 8, 1993.VI.SPECIAL REPORT:, This LER also serves as a Special Report prepared and submitted pursuant to Technical Specification (TS)3.5.2 ACTION b and TS 6.9,2 to describe the circumstances of the Emergency Core Cooling System (ECCS)actuation and the total accumulated actuation cycles to date.The circumstances of the ECCS actuation are described in Section I.B of this report.In Palo Verde Unit 3, there have been 3 total accumulated actuation cycles of the ECCS to date.