ML17313A413

LER 98-006-00:on 980507,determined That Plant Was Outside Design Basis Due to SI Discharge Check Valve Reverse Flow. Check Valve Was Disassembled,Examined & Reassembled, Whereupon Valve Met Acceptance Criteria

ML17313A413
Person / Time
Site:	Palo Verde
Issue date:	06/05/1998
From:	Marks D ARIZONA PUBLIC SERVICE CO. (FORMERLY ARIZONA NUCLEAR
To:
Shared Package
ML17313A412	List: ML17313A411 ML17313A413
References
LER-98-006, LER-98-6, NUDOCS 9806160057
Download: ML17313A413 (16)

Text

LICENSEE EVENT REPORT (LER)

ACILflYNAME ('I ) DOCI(ET NUMBER (2) PAGE (3)

Palo Vercfe Unit 1 0 5 0 0 0 5 2 8 1oF08

(<)

Safety Injection Discharge Check Valve Reverse Flow Causes Condition Outside Design Basis EVENT DATE 6 LER NUMBER 6 REPORT DATE OTHER FACIUTIES INVOLVED 6 YEAR YEAR SEQUENTIAL REVISION 0 5 0 0 0 5 2 9 0 5 0 7 8 9 8 - 0 0 6 - 0 0 0 6 0 5 9 8 Unit 3 0 5 0 0 0 5 3 0 OPERATING IS REPORT IS SUBMITTED PURSUANT TO THE REOUIREMENTS OF 10 CFR g: (Check one or more ot the foaorrln9) (11)

MODE (9) 20A02(b) 20.45(c) M.73(aX2XN) 73.71(b)

POWER 20.45(sX1Xi) 50.36(cX1) 50.73(sX2Xv) 73.71(c)

LEVEL(to) $ P P 20.45(aXt~i 50.36(cX2) 50.73(aX2Xvi) OTHER (SPeci/7 in Abstract 20.45(sX1Xi) X 50.73(aX2Xi) 50.73(aX2Xvw+A) bekxtr and in Text, NRC Form 20.45(aXt Xiv) X 50.73(aX2Xi) SO.73(aX2XviXB) 2IL45(sXIXv) 50.73(aX2Xi) 50.73(aX2Xx)

LICENSEE CONTACT FOR THIS LER (12)

ELEPHONE NUMBER CODE Daniel G. Marks, Section Leader, Regulatory Affairs 6 0 2 3 9 3 - 6 4 9 2 COMPLETE ONE UKE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT (13)

CAUSE SYSTEM COMPONENT MANUFAC REPORTABLE CAUSE SYSTEM COMPONENT MANUFAC REPORTABLE TURER TO NPRDS TURER TO NPRDS A B Q V B 3 5 0 B B Q V B 3 5 0 SUPPLEMENTAL REPORT EXPECTED (16) EXPECTED MONTH DAY YEAR SUBMISSION YES (IfYes, ccmpkrte EXPECTED SUBMISSION DATE) DATE (15) 0 8 0 7 9 8 TRAOT (Lama to 1600 spaces, Le., approxsmately fsteen se9~ Ypeevntten ines) (16)

On May 7, 1998, Palo Verde Units 1, 2 and 3 were in Mode 1 (POWER OPERATION),

operating at approximately 100 percent power when engineering personnel determined there was sufficient evidence to conclude the Unit 1 "A" train High Pressure Safety Injection pump discharge check valve would not have performed its intended function from October 17, 1996, until April 11, 1998.

Engineering personnel believed, at that time, that the use of enhanced assembly instructions during the Unit 1 seventh refueling outage had corrected the condition. However on May 13, 1998, it was determined that current valve alignment was suspect, and the valve was declared inoperable. Subsequent testing of the "A" train valve revealed that reverse flow through the check valve was sufficient to cause less than minimum injection flow from the redundant train "B" HPSI system. After testing the check valve it was disassembled, examined and reassembled, whereupon it met acceptance criteria.

Based on the "as-found" condition of'he Unit 1 "A" train check valve, the Unit 2 "B" train check valve was tested on May 14, 1998, and it also demonstrated excessive reverse flow. The Unit 2 valve was reworked and further engineering examination revealed no other check valves were rendered inoperable by the condition.

No previous similar events have been reported pursuant to 10CFR50.73 in the last three years.

'7)806l60057 67)80605 PDR ADGCK 05000528 8 PDR

JA LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AGILITY NAME DOCKET NUMBER LER NUMBER, PAGE SEOUENTIAL REVISION NUMBER NUMBER Palo Verde Unit 1 0 5 0 0 0 5 2 8 9 8 - 0 0 6' 0 0 0 2 of 0 8

1. REPORTING REQUIREMENT:

This LER 528/98-006-00 is being submitted'ursuant to the following 10 CFR 50.73 criteria. In addition, a RETRACTION of one of the reporting criteria used during related Emergency Notification System (ENS) reports 34227 and 34246 (made pursuant to 10 CFR 50.72) is included.

10 CFR 50 '3(a)(2)(ii)(A and B)

Due to disc misalignment of the Unit 1 "A" and Unit 2 "B" train High Pressure Safety Injection (HPSI)(ECCS)(BQ) pump discharge check valves (V), the design basis minimum flow would not have. been met during a Loss of Coolant Accident (LOCA) design basis event (DBE).

If one HPSI pump is assumed to fail, the redundant HPSI train could not produce required minimum flow due to reverse flow through the opposite train's check valve, a condition where the Units were outside of the design basis and in an unanalyzed cond'ition.

10 CFR 50.73(a)(2)(i)(B)

The discs in the Unit 1 "A" train, and Unit 2 "B" train HPSI discharge check valves were misaligned for extended periods of time and resulted in a condition where Limiting Conditions for Operation (LCO) Allowed Outage Times (AOT) were unknowingly exceeded and that resulted in an operation or condition prohibited by the plant's Technical Specifications (TS). In addition, TS 3.0.3 was entered for brief periods from when the suspect valves were declared, inoperable based on engineering judgment, until they could be isolated from the redundant HPSI trains.

10 CFR 50.73(a)(2)(vii)(B and D)

The failure mechanism (immediate cause) of the Unit 1 "A" train and Unit 2 "B" train HPSI discharge check valves,was vertical misalignment of the disc which resulted in interference between the disc and valve body and incomplete valve closure. The failure mechanism was attributed to a common-cause error in assembling the valves which was a result of inadequate vendor and work instructions and/or personnel errors. Therefore, the assembly error led to multiple failures in systems designed to remove residual heat and mitigate accidents.

S l'l C

)

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION DOCKETNUMBER LER NUMBER PAGE SEQUENTIAL RENSION NUMBER NUMBER Palo Verde Unit 1 0 5 0 0 0 5 2 8 9 8 - 0 0 6 - '0 0 0 3 of 0 8 ENS 34227 and 34246 On May 14 and 15, 1998, PVNGS reported that the Unit 1 "A" train 2 "B" train HPSI pump discharge check valves had back-leakage and'nit in excess of acceptance criteria, which indicated design basis minimum flow might not be met (ENS 34227 and 34246 respectively).

The condition was reported as being outside the design basis for an .

extended period time,, and the system did not have suitable redundancy (50.72(b)(1)(ii)(B)). In addition, since the check valves could potentially divert 'flow from the redundant ECCS system, a condition that could have prevented the fulfillment of a safety function, the condition was also reported under 50.72(b)(2)(iii)(D).

Subsequent review of NUREG 1022, Revision 1, has revealed that it is not necessary to assume an additional random single failure in systems reported under 50.72(b)(2)(iii)(D) and therefore, this portion of the ENS reports is hereby RETRACTED.

2. EVENT DESCRIPTION:

On March 12, 1998, just prior to the beginning of Unit 1',s seventh refueling outage, the surveillance test procedure for the HPSI pump discharge check valves was revised to include new acceptance criteria for reverse flow testing.. The Unit 1 check valves were the first to be tested using the new acceptance criteria and on April 9, 1998, the Unit 1 "A" train check valve failed to meet the acceptance criteria. Upon disassembly, engineering personnel (other utility personnel) concluded that the valve disc was vertically misaligned high.

Engineering and Maintenance (other utility personnel) personnel believed, at that time, that the vertical misalignment had been corrected during rework of the valve on April 11, 1998, because revised instructions had been used to assemble the valve and post maintenance testing demonstrated acceptable reverse direction flow. A significant condition investigation was initiated to determine the root cause of the surveillance test failure. At this time, engineering personnel evaluated other HPSI pump discharge check valve surveillance test records and determined the inadequate HPSI delivery was not a concern, based on the test results.

On May 7, 1998, Palo Verde Units 1, 2 and 3 were in Mode 1 (POWER OPERATION), operating at approximately 100 percent power when engineering personnel determined there was sufficient evidence to conclude the Unit 1 "A" train check valve would not have performed its intended function from October 17, 1996, until April 11, 1998, when the valve was, reworked.

Although not able to confirm at the time, engineering, personnel suspected the valve disc may have been misaligned as early as 1992.

LlCENSEE EVENT REPORT (LER) TEXT CONTINUATlON ACIUTYNAME DOCKETNUMBER LER NUMBER PAGE SEOUENTIAL REVISION NUMBER NUMBER Palo Verde Unit 1 0 5 0 0 0 5 2 8 9 8 - 0' 6 - 0 -0 0 4 of 0 8 EXT Engineering concluded, based on a review of the Unit 1 "A" train test data, that the Unit 1 "B" train HPSI delivery flow would not have been met for certain design basis events. If, during these events, the "A" train HPSI pump is assumed to flail, the redundant "B" train HPSI system would not meet design basis minimum flow due to reverse flow through the "A" train check valve.

As the root cause investigation was proceeding, on May 13, 1998, engineering personnel suspected, based on measurements taken from a spare valve, that the Unit 1 "A" HPSI discharge check valve may not have been assembled correctly on April 11, 1998, as previously thought. Engineering personnel also suspected that if the valve disc was posi'tioned too low in the valve body it could result in a condition where the outside, upper edge of the disk could get caught under the inside upper edge of the seat causing the disk to "cock" open, similar to events described in Information Notice 89-62.

Engineering personnel informed Unit 1 Operations management (other utility personnel) of their suspicions regarding the check valve's condition and the valve was declared inoperable on May 13, 1998, at 1432 MST., The Unit 1 "A" HPSI system was already inoperable and TS 3.5.2.(a) entered at this time due to maintenance activities unrelated to the check valve condition.

Unknowingly, when the "A" HPSI check valve was declared inoperable, the effect was to make inoperable the "B" train HPSI flowpath,, which in effect, was entry into TS 3.0.3. At 1545 MST on May 13, 1998, .the "A" train HPSI pump was isolated, thereby effectually exiting TS 3 .3 condition.

On May 14, 1998, at 0615 MST, the Unit 1 "A" train HPSI discharge check valve was tested using a new. test procedure and the valve failed to meet reverse direction flow acceptance criteria. The NRC was notified (ENS 34227) of the test failure. Work began immediately to disassemble and inspect the valve, which confirmed the suspected vertical misalignment of the valve disc. The apparent cause of the misalignment was attributed to a measurement error that occurred during the April'9, 1998, disassembly of the valve. The valve was re-assembled, correcting the misaligned disc condition and when tested met the acceptance criteria, with no observable leakage. Operations personnel returned the valve to an operable status and exited the TS LCO 3.5.2(a) at 1756 MST.

7

-

~

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AGILITYNAME DOCKETNUMBER LER NUMBER PAGE yEAR sEaUENTIAI. RensIoN NUMBER NUMBER Palo Verde Unit 1 0 5 0 0 0 5 2 8 9 8 - 0 0 6 - 0 0 0 5 of08 Based on the dimensional data from the spare check valve and the Unit 1 "A" train valve, engineering personnel initiated external dimensional checks on the remaining HPSI pump discharge check valves. Dimensional data, maintenance work history and surveillance test records were used to create a matrix which identified valves potentially misaligned and susceptible to "cocking". Engineering personnel established a testing sequence for the remaining valves, based upon this matrix.

Data indicated the Unit 2 "B" train check valve had previously passed surveillance testing requirements, but had exhibited elevated reverse direction flow and external measurements indicated that the disc might be misaligned. Engineering personnel recommended to Unit 2 Operations management that the Unit 2 valve be declared inoperable based on their suspicions regarding the check valve's condition and the "B" train HPSI pump was declared inoperable on May 14, 1998, at 2155 MST and actions were taken to test the valve.

Unrealized by Unit 2 Operations personnel at the time, was that when the "B" HPSI pump was declared inoperable, the effect was to make the "A" train HPSI flowpath inoperable, which in effect, was entry into TS 3.0.3.

This condition existed until 2235 MST on May 14, 1998, when the "B" HPSI injection valves were isolated.

On May 15, 1998, at 1322 MST, the Unit 2 "B'rain check valve failed to meet acceptance criteria. The NRC was notified (ENS 34246) of the test failure. The valve was re-assembled, correcting the misaligned disc condition and when tested met the acceptance criteria, with no observable leakage. Operations personnel returned the valve to an operable status and exited the TS LCO at 0915 MST, on May 16, 1998.

To provide additional assurance that the remaining HPSI pump discharge check valves (Unit 3. "B" train, Unit 1 "B" train, Unit 2 "A" train and Unit 3 "A" train) were operable, each was tested in the order prescribed by the engineering matrix. By May 17, 1998, each valve had been tested and had demonstrated acceptable performance in accordance with the surveillance test acceptance criteria. However, the Unit 1 HPSI "B" train and the Unit 3 HPSI "B" train valves had dimensional values which suggested they may be susceptible to the disc cocking condition in the future and they were reworked by May 26, 1998 to optimize valve alignment.

41 LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AGILITYNAME DOCKETNUMBER LER NUMBER PAGE YEAR SEQUENTIAL REVISION NUMBER'UMBER Palo Verde Unit 1 0 5 0 0 0 5 2 8 9 8.- 0 0 6 - 0 0 06of 0 8 EXT 3 ~ 'ASSESSMENT OF THE SAFETY CONSEQUENCES AND IMPLICATIONS OF THIS EVENT:

The event did not result in any challenges to the fission product barriers or result in any offsite releases. Therefore there were no actual adverse safety consequences as a result of the event. However, it is known that design basis minimum flow could not have been maintained due to the reverse flow through the opposite train's check valve, a condition where the Units were outside of the design basis and in an unanalyzed condition.

The safety significance of the failed HPSI check valves was evaluated by reviewing possible failure modes. The limiting failure mode has been determined to be degraded HPSI flow delivery of the operating train as a result of reverse flow through the failed check valve. Safety evaluations are being conducted to assess the potential impact at this time.

A determination has been made that Updated Final Safety Analysis Report (UFSAR) Chapter 15 Design Bases Event (DBE), Main Steam Line Break Return to Power (All Modes), and the UFSAR Chapter 6 ECCS Performance Analysis need further evaluation to determine the potential safety impact due to the degraded HPSI flow. Additionally Fire Protection events that are impacted by the degraded HPSI condition also require further evaluation.

All other Chapter 15 DBEs were determined to not be impacted by this condition. A supplement to this LER will report the evaluation conclusions.

The Probabilistic Safety Analysis (PRA) group (other utility personnel) performed a preliminary assessment of the degraded HPSI flow condition.

Initiating events that were impacted were identified and a review of operator responses was conducted.

PRA's review revealed that Operator response to this event is covered by existing plant procedures and training. Emergency procedures 40EP-9E003, "Loss of Coolant Accident" and 40EP-9E009, "Functional Recovery" address identification of the degraded HPSI flow condition and the required actions to recover the Inventory Control Safety Function. Adequate instrumentation exists for the Operating staff to diagnose the degraded HPSI flow condition.

l LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AOII.AYNAME DOCKETNUMBER LER NUMBER PAGE SEOUENTIAL REVISION NUMBER NIIMBER Palo Verde. Unit 1 1 0 5'0 0 0 5 2 8 9 8 - 0 0.6 - 0 0 07 of 0 8 HPSI pump reliability, which is also used by PRA in determining the safety signi;ficance of this condition has historically been very good. In reviewing the history of failure and demands being tracked for the Maintenance Rule, from the period 1994 to present, there have been no HPSI pump or motor failures in approximately .614 demands. This supports the current estimated failure probability of 6.73 E-4 for the HPSI pump. Now that the final HPSI degraded flow condition is determined, the PRA group will perform an assessment of the risk increase associated with this condition.

4. STRUCTURES,, SYSTEMS, OR COMPONENTS INFORMATION:

The valves affected by the described condition are manufactured by Borg-Warner and are ASME Class 2, 4 inch, 1500 pound, bonnet pressure seal swing check valves. The disc assembly is suspended from the underside of the valve bonnet.

5. CAUSE OF THE EVENTS:

The Unit 1 Train "A" and Unit 2 train "B" HPSI pump discharge check valves failed because the valve discs became "cocked" under the top of the valve seat, preventing full closure. The cause for the valve discs being cocked open is due to vertical misalignment which was attributed to inadequate maintenance instructions. The-primary contributor to the inadequate maintenance instructions was inadequate vendor information.

On April 9, 1998, when the Unit 1 "A" train HPSI was disassembled the as-found measurements were incorrectly recorded which led to vertical disc misalignment when the valve was reassembled on April 11, 1998. This was attributed to personnel error.

Missed opportunities to identify the condition included: 1) Surveillance test procedures did not ensure the valve discs were seating and 2) lessons learned from in-house, and industry operating experience reports were not effectively incorporated into maintenance and testing procedures.

The cause of the TS 3.0.3 entries is being evaluated and the cause. of the condition will be provided in the supplement to this LER.

6. CORRECTIVE ACTIONS TO PREVENT RECURRENCE:

Immediate corrective actions were implemented to restore the affected valves to an operable condition. All HPSI discharge check valves discs have been determined to be assembled correctly.

LICENSEE EVENT REPORT (LER) TEXT CONTINUATION AC flAYNAME DOCKETNUMBER LER NUMBER PAGE SMUENTIAL REVlSION NUMBER NUMBER Palo Verde Unit.1 EXT

'0 '5 0 0 0 5 2 8 9 8 - 0 0 6 - 0 0 08of08 The maintenance instructions in use for check valve assembly had been revised on November 7, 1994 and are currently considered adequate to perform the activity. However, the check valve maintenance instructions will be revised to include more detailed installation instructions and drawings'ngineering has completed transportability reviews for all other Borg-Warner. bonnet hung pressure seal check valves susceptible to vertical disc misalignment caused by retaining ring position.

Engineering is evaluating transportability to other Inservice Test'ing (IST) program check valves with closure functions and this action will be completed by July 31, 1998.

Surveillance Test procedures for IST program check valves with closure functions are being reviewed to confirm that the acceptance criteria is appropriate and this action is expected to be completed by, July 17, 1998.

An evaluation will be conducted to determine if other industry operating experience information on complex component assemblies has been properly incorporated and this action will be August 31, 1998.

Engineering Support Personnel will be briefed on this event during quarterly industry events training this action will be completed by December 31, 1998.

A Safety Analysis summary report will be completed by July 24, 1998 and will, be included in the LER supplement.

"Late entries" were documented in the Unit 1 and 2 "Unit Logs" to note the entries into TS 3.0.3 Condition Report Disposition Requests were written

~

and the entries are being evaluated in accordance with the corrective action program.

7 ~ PREVIOUS SIMILAR EVENTS:

No other previous events have been reported pursuant to 10 CFR 50.73 in the last three years.

!I 0