ML20244A755

Insp Rept 50-382/89-09 on 890308-09.Violations Noted.Major Areas Inspected:Licensee Action on Previous Insp Findings Associated W/Testing,Low Recirculation Flow Condition & Corrective Maint of HPSI Pump B

ML20244A755
Person / Time
Site:	Waterford
Issue date:	04/03/1989
From:	Chamberlain D, Howell A NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20244A753	List: ML20244A751 ML20244A755
References
50-382-89-09, 50-382-89-9, IEIN-89-008, IEIN-89-8, NUDOCS 8904180178
Download: ML20244A755 (8)

See also: IR 05000382/1989009

Text

n --

". .

b

g- 't

. APPENDIX ~

'

'

U.'S. NUCLEAR REGULATORY. COMMISSION

REGION'IV

NRC' Inspection Report: 50-382/89-09 Operating License: NPF-38-

3:

Docket: 50-382

'

Licensee: Louisiana Power & Light Company (LP&L)

317 Baronne, Street

P.O.' Box 60340-

New Orleans, Louisiana 70160

<

Facility Name:' Waterford Steam Electric Station, Unit 3

Inspection At: 'Taft, Louisiana-

Inspection Conducted: Marc 8-9, 1 89

. Inspector: -

N _ [ _

, III, Pro 5eH Engineer, Reactor

h8

Date

, A. T. Howe

. Project Section A

Approved: M'8-8Y

D. DV Chamberlain, Chief', Reactor Project Date

Section A

Inspection Summary

Inspection Conducted March 8-9, 1989 (Report 50-382/89-09)

Areas Inspected: Special, announced inspection of licensee action on previous

inspection findings associated with the testing, low recirculation flow

condition, and the corrective maintenance of the "B" high pressure safety

injection (HPSI) pump.

Results: Potential violations involving two examples of failure to test the

"B" HPSI pump in accordance with some of the provisions of Article 3000 of

Section XI of ASME Boiler and Pressure Vessel Code were identified. These two

examples are apparent violations of Technical Specification (TS) 4.0.5.a. Also

a potential violation of failure to properly implement a procedure during

corrective maintenance of the "B" HPSI pump motor was identified.

These potential violations are indicative of three broad problem areas. First,

the licensee relies almost exclusively on surveillance procedure acceptance

criteria to demonstrate component / system operability rather than emphasizing

890418017e s9040g_

PDR ADOCK 050003B2

G PNV

- _ _ _ _ _ _ _ - - - _ _ _ - - - - - _ _ _ _

_ _ - _ - _ - - _ - _ _ - _ _ _ _

. .

I . ..

2

.

\

sound engineering judgement when evaluating components and systems for operability.

Second, the failure to find and correct the cause of low "B" HPSI pump recirculation

flow is indicative of continuing problems in the area of corrective action.

Third, the heavy reliance on job specific work instructions and vendor technical

manuals rather than component specific maintenance procedures (where appropriate

to the circumstances) increases the likelihood of personnel error during the

maintenance process. This could adversely impact the operability and reliability

.of safetyrrelated. components and systems.

.

I

1

i

I

_ - - - _ _ _ - - - _ . - . - - _ )

.- _ _ . ._

__

.

1

... E. .f

1

'

3 ,

i

.

)

DETAILS

1.. -Persons Contacted l

l

Principal Licensee Employees

• D. F. Packer, Assistant Plant' Manager, Operations and Maintenance

• P. V. Pransankumer, Assistant Plant Manager, Technical Support

J. J. Zabritski, Manager, Nuclear Quality Assurance (Acting)

'*G. M. Davis, Manager, Events Analysis Reporting and Responses

• L. W. Laughlin,' Site Licensing Supervisor

• D. E. Marpe, Lead Maintenance Engineer

• W. E. Day, Trending, Compliance and Response Supervisor

• P. N. Backes, Assistant to Plant Manager

R. B. Hereford, Systems Engineer

• G. F. Koehler, Quality Assurance Supervisor

A. M. Ciluffa, Maintenance Engineer

D.' A. Shultz, Assistant Operations Superintendent

B. G. Morrison, Licensing Engineer

• Present at exit interview

2. Followup of Previously Identified Items (92701)

!

a. (Closed) Unresolved Item 382/8901-02 - Failure to Perform

Surveillance Testing of the "B" HPSI Pump in Accordance With

TS 4.0.5.a and 4.0.3  ;

HRC. Inspection Report 50-382/89-01 documented that the licensee had

requested relief from Article IWP-3000 of the ASME Boiler and

Pressure Vessel Code,Section XI (IWP-3100), that requires' that the

resistance of the system shall be varied until either the measured

differential pressure or the measured flow rate equals the

corresponding reference value. Specifically, the licensee stated

that test flow was approximately the same for each test because a

flow restricting orifice was installed in the "B" HPSI pump test

line. The licensee proposed that an alternate measurement of '

. differential pressure would be compared to the allowable ranges in

lieu of the flow rate comparison.

This written relief request was denied by the Office of Nuclear

Reactor Regulation (NRR) in a May 20, 1988, letter to LP&L. The NRC

inspector reviewed this letter and noted that the letter implicitly

acknowledged that the flow of the system could not be varied, per .

IWP-3100, because of the flow restricting orifice; however,

differential pressure as well as flow could be measured and compared

to allowable ranges. Specifically, the letter required the recording

of all pararoeters in Table IWP-3100-2 in order to determine pump

operability. With respect to flow, Table IWP-3100-2 requires the

measuring of Alert and Required Action Ranges. In order to establish

_ __ _

. .

. i

4  !

• i

.

Alert and Required Action Ranges, a reference value for flow (Qr)

must first be established in accordance with IWP-3110.

Discussions with licensee personnel revealed that not all of them

were aware of the requirements of the May 20, 1988, letter. They

noted that they were, in fact, measuring pump flow per Table IWP-3100-1  ;

and that IWP-3100 only required varying the resistance of the system '

until either the measured differential pressure or the measured

flow-rate equals the corresponding reference values. They'noted that

this was being performed for differential pressure.

The NRC inspector informed the licensee that flow had to be recorded

in accordance with Table IWP-3100-2 as well as Table IWP-3100-1.

Additionally, while IWP-3100 requires varying the resistance of the

system until either measured differential pressure or the measured

flow rate equals the corresponding reference value, it further

requires that the test quantities in Table IWP-3100-1 be measured or

observed, and that any deviations determined shall be compared with

the limits given in Table IWP-3100-2 and the specified corrective i

action taken. The NRC inspector noted that even though flow was

measured per Table IWP-3100-1, it was not being measured per

Table IWP-3100-2 since the denial of the relief request on May 20,

1988.

'Further discussions.with licensee personnel revealed that LP&L

responded on July 8, 1988, to the May 20, 1988, letter from NRR. In

this letter, LP&L reiterated its request not to vary the resistance

of the system since'the system design prevents varying the flow rate. i

However, LP&L noted in the letter that for each surveillance test

"the flow rate, differential pressure and vibration, are measured and i

'

compared to reference values." The NRC inspector noted that flow was

not being compared to reference values during surveillance testing of

the "B" HPSI pump or the two other HPSI pumps and the two low

pressure safety injection (LPSI) pumps. In the February 7,1989,

transmittal of the Safety Evaluation of the Waterford 3 inservice

testing program for pumps and valves through Revision 5, NRR again

reiterated the requirement to " measure and record all of the

parameters in Table IWP-3100-2 in order to determine pump

operability."

The NRC inspector noted two instances, since the issuance of the

July 8,1988, LP&L letter to the NRC, of "B" HPSI pump surveillance

testing in which flow data was not recorded in accordance with

Table IWP-3100-2. These tests occurred on August 30, 1988, and

November 22, 1988. Failure to perform surveillance testing in

accordance with the Article IWP-3000 standard is an apparent

violation of TS 4.0.5.a. TS 4.0.5.a requires, in part, that

Surveillance Requirements for inservice testing of ASME Code Class 1,

2, and 3 components shall be applicable as follows: inservice

testing of ASME Code Class 1, 2, and 3 pumps shall be performed in

- _ _

n .

.

>-

.

, ,

E , ,

,, ~4 .

'

. .

.

A

'

e

t, -

5-

- ,

e

accor' dance with= Section XI of the ASME- Boiler and Pressure Vessel .'

,

Code, except where specific written relief has been granted by the'

- Commission.

i; b. .(Closed)' Unresolved Item 382/8901-03 - Failure to Take Prom)t

- Corrective Action Following the Identification of Low "B" H)SI

~

Pump Recirculation Flow

- A'n NRC maintenance' team inspection-(NRC Inspection Report 50-382/89-01)

noted that a low recirculation flow condition.for the "B" HPSI pump -

was discovered by the licensee during a routine surveillance test of-

the "B" HPSI pump on November 22, 1988. :The recorded value of-

recirculation' flow was 19.0 gpm. . Although no acceptance criteria

-

was listed on-the' surveillance test data sheet, the vendor technical

manual recommended a minimum recirculation. flow'of 25 gpm. . During the

second week of the ~NRC maintenance team inspection of Waterford 3

(January 31' through February 3,1989) the team noted that the cause of

the recirculation flow condition had not been determined'or. corrected.

The team further noted et that time the "B" HPSI pump had not been-

- declared-inoperable as a ~ result of the low recirculation flow

condition.

As a result of NRC concerns about'the operability of the "B" HPSI-

- pump,,the licensee performed a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> and 15 minute. surveillance test

of -the pump on- January 31,_1989. During this test, the I&C

technician observed that the flow orifice associated with Flow

Instrument SI-IFI-7121 in the return line to the refueling water

storage pool (RWSP) had apparently been installed backwards during'

~

maintenance in December 1987. 'The orifice was'placed in the proper

.

position and the pump was retested. Recirculation flow indication

increased approximately 4.5 gpm to a value of 24.5 gpm.

The N'lC inspector' concluded that 'the reversed. orifice could not be

the sole cause of the low recirculation flow condition since the

"B" HPSI pump recirculation flow rate 'was still. several gpm less than

the average flow that was recorded prior to. December 1987. Additionally,

flow ~had increased from 20 gpm in' December 1987 to an average rate of

25.5 gpm during the next 3 surveillance tests but.had decreased to

19.0 gpm on November 22, 1988. The reversed flow orifice does not

account for these apparent flow anomalies.

The ."B" HPSI pump.was declared inoperable on February 1,.1989, due to

high ' axial vibration. On February 2,1989, the licensee initiated

Work Authorization (WA) 01031752.to determine the cause of excessive

wear and to repair the "B" HPSI pump inboard and outboard thrust

bearings.

Discussions with licensee personnel revealed that the accelerated

thrust bearing failure was caused by improper balance drum clearance.

One of the purposes of the pump balance drum is to absorb axial pump

forces during low flow conditions (particularly when the pump is

!

<

w____---.___.___--___ ___ __ -- - _ . _ . _ . - _ _ - - . . _ _ _ . .

- - _ _ _ _

. .

.

,

i

6

.

.

operating in the recirculation mode). Improper balance drum-

clearance resulted in the pump's thrust bearings absorbing a

disproportionate amount of the pump's axial forces during low flow

conditions. This resulted in an accelerated deterioration of the

thrust bearings. The balance drum clearance was apparently

improperly set following replacement of the "B" HPSI pump rotor in

April 1986. The licensee attributed the improper clearance to

inadequate guidance in the vendor technical manual. A review of _  ;

Ingersoll-Rand HPSI Instruction Manual 457000272, revealed that the

balance drum clearance settings specified were correct, but that the

level of detail for properly setting the balance drum clearance was

lacking. The NRC inspector concluded that this aspect of the repair

activity could not be considered within the skill of craft for

maintenance personnel. The NRC inspector further concluded that this

improper repair due to inadequate technical manual guidance could

have been avoided if the licensee had developed a detailed corrective

maintenance procedure for the HPSI pumps. This is discussed in

further detail in paragraph 2.c of this report.

The licensee returned the "B" HPSI pump to service following repair

of the pump and motor bearings. The NRC inspector noted, however,

that recirculation flow was still significantly less than the

pre-December 1987 values and less than the 25 gpm minimum specified

in the vendor technical manual. The "B" HPSI pump recirculation pump i

flow was recorded as 22 gpm during a February 24, 1989, surveillance

test of the pump. This flow was evaluated as acceptable by the

licensee. The NRC inspector made the following observations:

(1) In a' minimum flow evaluation report, "High Pressure Safety

Injection Pump: Ingersoll-Rand 4x9c-9, Motor Driven,"

(January 27,1989) performed by the Ingersoll-Rand Company at

the request of LP&L (which was collecting data for NRC

Bulletin 88-04, " Potential Safety-Related Pump Loss"), the

minimum flow for pump starting / stopping was 25 gpm.

Ingersoll-Rand defined pump starting / stopping as pump operation

of 15 minutes or less. For pump operation greater than ,

15 minutes, Ingersoll-Rand specified significantly higher

minimum flow requirements.

(2) In the same report, Ingersoll-Rand also stated that "in order to

realize the pump life projected in this evaluation, periodic

maintenance must be provided. As a minimum, the pump must be j

'

operated in accordance with the guidelines and recommended '

maintenance schedules as presented in the Ingersoll-Rand Pump

Instruction Manual and good industry practice." The NRC

inspector noted that one of the guidelines in the Pump

Instruction Manual was to keep the recirculation line open when

starting and stopping and at capacities less than 25 gpm.

(3) The licensee primarily based its "B" HPSI pump operability

determination on verbal information provided by the

- - - - _ _ - - _ _

l

-

. .

. .

7

.

'

.

Ingersoll-Rand Company. The basis for this determination was

that the pump parameters (noise, vibration, and temperature

readings) all remained within acceptable limits during a 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br />

and 15 minute run of the pump in recirculation mode. One of the

primary considerations was the stabilization of pump

temperatures (including bearing temperatures) taken at various

.

l

! locations on or near the pump. The NRC inspector noted,

however, that in NRC Not. ice 89-08, " Pump Damage Caused by

Low-Flow Operation," early component degradation from low

recirculation flow may not be detected by routine inservice

i tests. Furthermore, a review of the licensee's Inservice Test

l Program, Revision 6, revealed that LP&L was requesting relief

from the requirement to measure bearing temperature annually.

If this exemption were granted, the licensee would not have the

benefit of valuable operating temperature information in

assisting in the determination of pump operability.

(4) The cause of the low recirculation flow for the "B" HPSI pump

has not been determined and corrected. The licensee plans to

investigate the cause of the low flow condition once it obtains

a highly accurate ultra sonic flow measuring device that is

expected to be available in late March or early April. The NRC

inspector noted, however, that had a reference value for the

"B" HPSI pump been established on the basis of flow data

obtained prior to July 8,1988, it appears that the '19.0 gpm

value recorded on November 22, 1988, would have been lower than

the " Low Value Required Action Range" limit. The NRC inspector

further noted that IWP-3230(b) requires that if deviations fall

within the " Required Action Range" of Table IWP-3100-2, the pump

shall be declared inoperablo and not returned to service until

the cause of.the deviation has been determined and the condition

corrected. Discussions with licensee personnel revealed that

they believed that their analysis of the low recirculation flow

condition met the requirement of IWP-3230(c) which states, in

part, that " correction shall be either replacement or

repair . . . or shall be an analysis to demonstrate that the

condition does not impair pump operability and that the pump

will still fulfill its function." However, the NRC inspector

noted that the licensee had not yet determined the cause of the

deviation.

Failure to determine the cause of the low flow condition prior to

returning the "B" HPSI pump to service is a second exeinple of failure

to meet the Surveillance Requirements for the "B" HPSI pump in

accordance with TS 4.0.5.a.

c. NRC Inspection Report 50-382/89-06 - Repair of the "B" HPSI Pump and j

Pump Motor

As previously noted in paragraph 2.b, the HPSI Pump Instruction

Manual provided insufficient guidance for setting HPSI pump balance

i

_ _ _ _ _ _ _ _ _ _ _ _

_ _- .

,

' '

'

. . . ,

y.. >

a

8

.

.

drum clearance. ! Because of the licensee's heavy reliance on the use

.of work instructions and vendor manuals in lieu of Plant _ Safety

Committee (PSC) reviewed and approved component specific corrective

maintenance procedures,' the NRC inspector was concerned. that future

instances of inadequate maintenance on safety-related. equipment was

likely to occur., This concern was also documented in NRC Inspection

Report 50-382/89-01.

NRC Inspection Report 50-382/89-06 documented an instance in which

the "B" HPSI pump motor oil slinger ring was improperly installed and

that the bearing antirotation pins were not transferred from the old

bearings to the newly-installed bearings. Two uncoupled motor runs

were attempted.on February 20, 1989, but the motor had to.be secured

because.cf excessive vibration. Upon removal of the bearing covers,

bearing damage was observed. The damage was attributed to the.

improper installation of the oil slinger ring.

A review of Event Summary Report-(ESR)89-003, revealed that a

contributing cause of the "B" HPSI pump motor bearing damage was the

omission of the checks for proper oil slinger ring operation which-

are in the vendor instruction manual, but was not included in the

work instructions of WA 01032220. Furthermore ESR 89-03 documented

that the motor technical manual was.not the latest revision and did

not provide guidance for the movement of the antirotation pins from

the.old bearing to the new bearing. As noted in NRC Inspection

Report 50-382/89-06, failure to install the antirotation pins would

cause motor failure, but may not be detected during normal

surveillance testing.

Step 5.2.7 of Administrative Procedure MD-1-014, Revision 2. " Conduct

of Maintenance," . requires, in part, that "if, during the performance

of any activity requiring the use of vendor technical manuals,

deviations fromithe recommendations or procedures in the_ manuals are

necessary, such deviations shall receive a documented technical

review by Maintenance Engineering and be a

MAS [ maintenance assistant superintendent]pproved by the

." The NRC inspector noted cogniza

that the omission of the checks for proper oil slinger _ ring operation

from WA 01032220 was not reviewed by Maintenance Engineering and

approved by.the Electrical MAS. This is an apparent violation of

MD-1-014.

3. Exit Interview

l

The inspection scope and findings were summarized on March 9, 1989, with

those persons indicated in paragraph 1 above. The licensee acknowledged

the NRC inspector's findings. The licensee did not identify as

i

proprietary any of the material provided to, or reviewed by, the NRC

I inspector during this' inspection.

!

,

-

w. .

,