05000529/LER-2000-005

LER-2000-005,

Event date:	10-27-2000
Report date:	05-00-0529
Reporting criterion:	10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications 10 CFR 50.73(a)(2)(v), Loss of Safety Function 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition
Initial Reporting
5292000005R01 - NRC Website

NAME

Daniel G. Marks, Section Leader, Regulatory Affairs TELEPHONE NUMBER (Include Area Code) 623-393-6492 COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT (13) == CAUSE SYSTEM COMPONENT MANUFACTURER REPORTABLE TO EPIX CAUSE SYSTEM COMPONENT MANUFACTURER ==

TO EPIX

SUPPLEMENTAL REPORT EXPECTED (14) EXPECTED == SUBMISSION ==

DATE (15) == MONTH DAY YEAR ==

YES

(If yes, complete EXPECTED SUBMISSION DATE). I X NO ABSTRACT (Limit to 1400 spaces, i.e., approximately 15 single-spaced typew itten lines) (16) On October 23, 2000, at approximately 1600 Mountain Standard Time, Palo Verde Unit 2 was in Mode 6 (REFUELING) when surveillance testing of the "B" train high pressure safety injection (HPSI) pump discharge check valve (2PSIBV405) revealed back-leakage in excess of surveillance I test acceptance criteria. Further investigation revealed the cause of the back-leakage was due to a defective manufacturer's weld that caused the valve disk to become cocked.

APS engineering has determined that while 2PSIBV405 was in this condition the HPSI system minimum flow requirements would not have been met by the redundant "A" train HPSI system during certain events when the "B" train HPSI pump was not operating. The internals of the check valve were replaced and the valve was returned to operable status on November 3, 2000.

A previous similar event was reported in LER 50-528/1998-006-01.

1. REPORTING REQUIREMENT(S):

APS is reporting this condition based on the assumption that the "B" train High Pressure Safety Injection system (HPSI)(EIIS: BQ) discharge check valve (2PSIBV405)(EllS: V) was degraded for some period of time prior to discovery during 18 month surveillance testing. Based on this assumption APS is reporting this condition in accordance with:

10CFR50.73(a)(2)(i)(B) because it is probable that the condition existed for a period of time in excess of the Limiting Condition for Operation (LCO) Allowed Outage Times (AOT).

10 CFR 50.73(a)(2)(ii)(B) because the minimum high pressure safety injection system flow could not have been maintained which is an unanalyzed condition.

10CFR50.73(a)(2)(v) because the degraded condition of 2PSIBV405 alone may have prevented the fulfillment of the safety function to mitigate the consequences of an accident.

2. DESCRIPTION OF STRUCTURE(S), SYSTEM(S) AND COMPONENT(S):

2PSIBV405 2PSIBV405 was manufactured by Borg-Warner (BW) and is an ASME Class 2, 4 inch, 1500 pound, bonnet pressure seal, two piece welded body swing check valve. The disc assembly is suspended from the underside of the valve bonnet. After failing surveillance testing, 2PSIBV405 was discovered to have excessive freedom of movement between the swing arm and the valve disc.

Emergency Core Cooling System (ECCS) The ECCS, of which the HPSI is a subsystem, is designed to provide core cooling in the unlikely event of a loss-of-coolant accident (LOCA). The ECCS prevents significant alteration of core geometry, precludes fuel melting, limits the cladding metal-water reaction, == DOCKET ==

removes the energy generated in the core and maintains the core subcritical during the extended period of time following a LOCA.

The ECCS accomplishes these functional requirements by use of redundant active and passive injection subsystems. The active portion consists of high and low pressure safety injection pumps and associated valves. The passive portion consists of pressurized safety injection tanks (SIT)(EIIS: BP). During an event requiring ECCS actuation, a flow path is required to supply water from the refueling water tank (RWT)(EIIS: BQ) to the reactor coolant system (RCS)(EIIS: AB) via the HPSI pumps and their respective supply headers to each of the four cold legs.

3. INITIAL PLANT CONDITIONS:

On October 23, 2000, at approximately 1600 MST, Palo Verde Unit 2 was in Mode 6 (REFUELING) when testing began on 2PSIBV405.

On November 3, 2000 at 0500 MST, Palo Verde Unit 2 was in Mode 5 (COLD SHUTDOWN) when 2PSIBV405 was returned to an operable status.

Reverse flow testing of selected Unit 1 and Unit 3 HPSI discharge check valves was conducted between October 27, 2000 and October 28, 2000, during which time Palo Verde Units 1 and 3 were in Mode 1 (POWER OPERATION) at approximately 100 percent power.

There were no inoperable structures, systems, or components that contributed to the event.

4. EVENT DESCRIPTION:

On October 23, 2000, at approximately 1600 MST, the 18 month surveillance testing of 2PSIBV405 was commenced in accordance with the inservice testing program. The surveillance testing results revealed back-leakage in excess of the acceptance criteria of conditions for operation were entered as the ECCS was not required in Mode 6.

DOCKET

On October 24, 2000, 2PSIBV405 was tested again using higher capacity testing equipment and again failed to meet acceptance criteria.

On October 25, 2000, APS engineering personnel visually examined the valve internals with a boroscopic camera and determined that the valve disc had become cocked under the top of the valve seat.

On October 26, 2000, at approximately 2240 MST, a troubleshooting and diagnostic procedure was implemented which uses the "A" HPSI pump to provide a pressure force and flow to seat 2PSIBV405. Results of the troubleshooting and diagnostic procedure indicated 2PSIBV405 remained cocked open as indicated by a pressure of 460 pounds per square inch gauge (psig) at the suction of HPSI pump "B.

On October 27, 2000, at approximately 1400 MST, 2PSIBV405 was disassembled and engineering personnel observed a bent disc stud and excessive freedom of movement (articulation) between the swing arm and the valve disc. At approximately 1500 MST, APS made notification of the findings regarding 2PSIBV405 to the NRC via the Emergency Notification System (reference ENS# 37465).

I On October 27, 2000, at approximately 1715 MST, further investigation of the reverse flow test failure of 2PSIBV405 revealed that disc articulation had not been measured during the initial valve alignment of Unit 1 HPSI discharge check valves 1PSIAV404 and 1PSIBV405, or Unit 3 HPSI discharge check valve 3PSIBV405. Although these three valves had previously passed surveillance testing, they were reverse flow tested again to provide additional assurance they would perform their design function. By October 28, 2000, at approximately 0632 MST, the Unit 1 and Unit 3 HPSI discharge check valves had successfully completed reverse flow testing.

On November 3, 2000 at 0500 MST, upon completion of rework activities, 2PSIBV405 was returned to an operable status.

5. ASSESSMENT OF SAFETY CONSEQUENCES:

The degraded HPSI flow condition did not result in any challenges to the fission product barriers or result in any offsite releases. Therefore there were no actual adverse safety I NRC FORM 366A (6-1998) U.S. NUCLEAR REGULATORY COMMISSION LICENSEE EVENT REPORT (LER) == DOCKET ==

consequences as a result of this condition. However, it is known that the design basis minimum flow could not have been maintained by the opposite HPSI pump alone due to the reverse flow through the subject check valve, a condition where the Unit was in an unanalyzed condition.

An evaluation was performed to determine if the 1998 event and evaluation bound the current event. This evaluation was performed by comparing the identified leakage rates and pressurization characteristics of the events.

In both the 1998 event and the October 2000 event, reverse flow testing of the check valves was performed using demineralized water. The leakage rates measured from these tests are as follows:

• 1PSIAV404 33 gpm (1998 test)

• 2PSIBV405 37.5 gpm (1998 test)

• 2PSIBV405 30 gpm (2000 test) From this data it can be seen that the 1998 event was worse from a leakage standpoint than the 2000 event. Troubleshooting and diagnostic testing following discovery of the leakage was performed in both events. This testing consisted of running the opposite train pump to try and seat the leaking check valve. In 1998 this testing resulted in the Unit 1 'A' Train HPSI pump suction being pressurized to approximately 690 psig. Similar testing for the 2000 event resulted in the suction pressure for the Unit 2 B' Train pump not exceeding 500 psig. This again confirms that the leakage in 1998 was more significant and bounded the 2000 event.

In 1998, the safety significance of the failed HPSI check valves was evaluated by reviewing possible failure modes. Design engineering completed calculation 13-MA-SI-982 (Evaluation of allowable leak rate test criteria for SIAV404 and SIBV405/Assessment of as-found leakage for 2PSIBV405 and 1PSIAV404). This 1998 calculation determined 2PSIBV405 leakage with various RCS pressures. From these leakage values, flow delivered to the RCS for the 2000 event was determined and compared to the delivery flow values from the 1998 event. The delivery flow from the 1998 event was bounding; therefore, the LOCA analysis and safety significance determination from the 1998 event (LER 50-528/98-006-01) are applicable for the 2000 event.

DOCKET

1 Combustion Engineering (CE) performed a small break LOCA analysis in 1998 taking into account the cocked check valve. The evaluation demonstrated that the failure of the HPSI discharge check valve results in sufficient HPSI flow to the RCS to meet the ECCS acceptance criteria of 10CFR50.46 for the limiting Small Break LOCA (SBLOCA). This conclusion is based on methods that make use of more realistic models (1979 ANS decay heat standard and CE's Realistic Evaluation Model for hot rod heatup) than allowed by Appendix K to 10CFR50. The results of the analysis were a peak clad temperature of 1742 °F, maximum cladding oxidation of 6%, maximum core-wide cladding oxidation of analysis using the conservative models required by ABB CE's NRC accepted Appendix K SBLOCA evaluation model would not meet the ECCS acceptance criteria of 10CFR50.46.

Additionally, in 1998 several best estimate simulations of the SBLOCA were performed in the PVNGS simulator to assess the effectiveness of the emergency operating procedures and operator actions for dealing with the effect of the degraded HPSI flow on SBLOCA long term cooling. These simulations demonstrated that the emergency operating procedure (EOP) guidance contained in the functional recovery procedure would enable the operators to recognize inadequate flow and take corrective actions to successfully cool the RCS with sufficient time despite the degraded HPSI condition.

APS Probabilistic Risk Assessment (PRA) personnel performed an assessment of the degraded HPSI flow condition in 1998. This analysis was performed assuming the degraded HPSI flow would have resulted in core damage and does not consider the results of the deterministic analysis discussed above. PRA's review revealed that existing plant procedures and training covers Operator response to this event.

Emergency procedures 40EP-9E003, "Loss of Coolant Accident" and 40EP-9E009, "Functional Recovery" address identification of the degraded HPSI flow condition and the required actions to recover the Inventory Control Safety Function. Adequate instrumentation exists for the operating staff to diagnose the degraded HPSI flow condition. PRA analysis of the risk associated with the degraded HPSI flow condition, assuming this condition leads to core damage, resulted in a core damage frequency increase of 3.3 E-05/yr, approximately a 100% increase in the baseline value. However, when the results of the deterministic analyses discussed above are considered, then the == DOCKET ==

increase in core damage frequency related to the degraded HPSI flow condition is negligible.

APS determined that the reverse flow surveillance test on the 2PSIBV405 check valve had been successfully performed twice since the valve was aligned vertically in May of 1998. Additionally, APS noted that there were two instances in which the header downstream of the check valve was pressurized indicating the check valve was seating tightly. Pressure down stream of the check valve was observed to be above the RWT pressure in November 1999 and June 2000, both were documented in the corrective action program. As such, there were no indications of check valve degradation until the Unit 2, ninth refueling outage in October 2000, when the reverse flow surveillance test failed, two years after the vertical height had been aligned. The planned and unplanned unavailability that resulted in fault exposure for the High Pressure Safety Injection B Pump, conservatively assuming the check valve was open during the period between June 2000 and October 2000, was 42.67 hours. Therefore, this is the maximum possible exposure time to the unanalyzed condition for the degraded check valve.

This is considered a Maintenance Rule Repetitive Functional Failure for 2PSIBV405 and results in mandatory (a)(1) placement for the HPSI system. Unavailability goals were not exceeded.

6. CAUSE OF THE EVENT:

An investigation of this condition has been conducted in accordance with the APS corrective action program. APS has concluded the equipment root cause of the valve failure was a manufacturing defect. Specifically, the manufacturer fabricated the disk assembly with non-uniform weld build-up at the stud to disk weld. The weld buildup as well as other contributing factors resulted in an excessively large gap between the swing arm and washer. The non-uniform weld caused the spherical bearing to be tilted, which allowed the disk to articulate further in one position. The gap (washer position) is essential in controlling and limiting disk articulation. When the gap is too large, as in this case, the washer is unable to limit the disk articulation because the swing arm does not contact the washer as the disk tilts. When the gap is not properly controlled, disk tilt becomes a function of the interface between the bearing and the bore of the swing arm.

Palo Verde Nuclear Generating Station Unit 2 05000529 LER NUMBER (6) == DOCKET ==

Check valve 2PSIBV405 also had a spherical bearing that was longer than the latest manufacturer's design, which contributed to the excessively large gap. Any changes or variances in these components affects disc tilt. Collectively, each contributing factor eventually led check valve 2PSIBV405 to fail.

7. CORRECTIVE ACTIONS:

Based on this and previous events, APS initiated multiple corrective actions to provide assurance that similar events do not occur. Check valves that are determined to be susceptible to this failure were categorized based on the safety and production significance and an inspection plan will be developed accordingly. Maintenance procedures were revised to require measuring valve tilt and recording the maximum tilt, as well as the vertical alignment setting. The procedure also provides alignment criteria for the acceptance or rejection of the valve's internals.

As an immediate action, the internals of HPSI discharge check valve 2PSIBV405 were replaced during Unit 2's ninth refueling outage and the valve was returned to an operable status on November 3, 2000. Based on the new information gained from the completion of the Equipment Root Cause of Failure Investigation, and the washer gap measurements of 2PSIAV404 and 2PSIBV405, both Unit 2 valves are considered to be acceptable and do not have excessive disk stud to disk fillet welds. The Unit 2 valves will however, be disassembled and inspected during the next Unit 2 refueling outage to verify excessive disk articulation does not exist.

The damaged 2PSIBV405 components were sent to Southwest Research Institute where APS engineering personnel observed the testing of the components.

Following the October 23, 2000 event, the initial transportability review identified three other HPSI pump discharge check valves, 1PSIAV404, 1PSIBV405 and 3PSIBV405 that were potentially susceptible to the identified failure mode. The valves were potentially susceptible because tilt measurements had not been taken on these valves. Unit 1 and Unit 3 HPSI discharge check valves 1PSIAV404, 1PSIBV405, and 3PSIBV405 were reverse flow tested and confirmed operable by October 28, 2000.

DOCKET

outage (February 2001) and both were verified to be acceptable. However, check valve 3PSIAV404 was found with the spherical bearing installed upside down. Therefore, the check valve internals on 3PSIAV404 were changed out. During the Unit 1 refueling outage (April 2001), valve 1PSIAV404 was disassembled, inspected and verified to be acceptable.

Check valve internals for 1PSIBV405 were also found to be acceptable; however, a small hole was discovered in the spherical bearing. Subsequent investigation determined the assembly was received with the defect from the manufacture. The internals of the check valve 1PSIBV405 were replaced, inspected and found acceptable.

8. PREVIOUS SIMILAR EVENTS:

train "B" HPSI pump discharge check valves failed because the valve discs became cocked under the top of the valve seat, preventing full closure. The cause for the previous event was vertical misalignment, which was attributed to inadequate maintenance instructions.

APS personnel investigating the current event reviewed the previous event and subsequent procedural enhancements regarding disc articulation. Based on the findings to date, the condition reported in this LER does not appear to be the result of vertical misalignment and therefore, the underlying concern or sequence of events is not the same as the previously reported condition.