ML20199H452

Insp Repts 50-361/98-18 & 50-362/98-18 on 981109-1218. Violations Noted.Major Areas Inspected:Operations, Maintenance & Engineering

ML20199H452
Person / Time
Site:	San Onofre
Issue date:	01/15/1999
From:	NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20199H447	List: ML20199H442 ML20199H448 ML20199H452
References
50-361-98-18, 50-362-98-18, NUDOCS 9901250212
Download: ML20199H452 (19)

See also: IR 05000361/1998018

Text

__

i.l' .

l

4

I'

ENCLOSURE 2

U.S. NUCLEAR REGULATORY COMMISSION

l

REGION IV

!

l

Docket Nos.:

50-361

i

50-362

License Nos.:

NPF-10

NPF-15

Report No.:

50-361/98-18

I

50-362/98-18

i

I

f

Licensee:

Southern California Edison Co.

Facility:

San Onofre Nuclear Generating Station, Units 2 and 3

'

Location:

5000 S. Pacific Coast Hwy.

)

' San Clemente, California

i

Dates:

November 9 through December 18,1998

)

Inspectors:

J. J. Russell, Resident inspector

i

J. G. Kramer, Resident inspector

>

'

.

Approved By:

L. J. Smith, Acting Chief, Branch E

'

Division of Reactor Projects

e

.

.

,

ATTACHMENT:

Supplemental Information

l

l

I

l

,

i

1

l

1

9901250215 990115

.

gDR

ADOCK 05000361

'

PDR

i

'

.

1

- .

. ._ . _ _ _.._ _ . _ _ _ _ _ _

- . _ _ _ _ _ _ . _

_ _ . _ _ . _ _ . . _

>

.

l

EXECUTIVE SUMMARY

f

San Onofre Nuclear Generating Station, Units 2 and 3

NRC Inspection Report 50-361/98-18; 50-362/98-18

l

'

Operations

!

A faulty temperature control unit, an incorrectly set electrical demand switch, and

i

noncondensable gases rendered Chiller E335 inadvertently inoperable for 20

!

consecutive days in August 1998. Similarly, a miswired low chilled water temperature

cutout switch or a combination of noncondensable gases, oil in the refrigerant, and an

!

electrical demand setting of 80 percent rendered Chiller E335 inoperable for 22

consecutive days in September 1998. The time periods for these inoperabilities

exceeded the Technical Specifications Limiting Condition for Operation 3.7.10 allowed

outage time and was identified as an apparent violation (Section 08.1.b.3).

.

l

The emergency chilled water system operating procedure specified that the electrical

!

demand be limited to 80 percent, which reduced chiller capacity below that assumed in

i

design bases document. The failure to translate the design basis requirement for

emergency chiller capacity in the operations procedures for setting electrical demand

contributed to emergency Chiller E 335 in operability during August and September

1998. This failure was identified as a noncited violation of 10 CFR Part 50, Appendix B,

i

l

Criterion Ill, " Design Control." However, this deficiency alone did not render the

!

emergency chiller inoperable. Tims nonrepetitive, licensee-identified and corrected

l

violation is being treated as a noncited violation consistent with Section Vll.B.1 of the

NRC Enforcement Policy (Section 08.1.b.5).

Operators demonstrated poor awareness of the effects of compressor oil pump

operation on a standby emergency chiller. Operators operated the oil pump with the

compressor in standby for greater than the time allowed by procedure in attempts to

!

lower a high oil level. The inspectors identified this as a failure to follow a procedure

required by Technical Specification 5.5.1.1.a. Since the licensee implemented

i

appropriate corrective actions, no response was required (Section O8.2).

Maintenance

Postmaintenance testing, as prescribed by operations work control, for scheduled

emergency chiller maintenance was not adequate because work that affected the start

circuit for the chiller was not tested. This deficiency was identified as a noncited

violation of 10 CFR Part 50, Appendix B, Criterion V, consistent with Section Vll.B.1 of

the Enforcement Policy. This deficiency contributed to inoperability of Chiller E335 for

22 days in September 1998 (Section 08.1.b.2).

Maintenance personnel demonstrated poor control of lifted leads and of emergency

chiller switch settings. An instrumentation and control (l&C) technician failed to

l

properly reconnect wires to an emergency chiller low temperature cutout switch,

j

rendering the chiller inoperable. The failure to follow procedure was identified as a

noncited violation of Technical Specification 5.5.1.1.a consistent with Section Vll.B.1 of

'

l

the Enforcement Policy. During the same maintenance activity, the chiller was left with

!

an improper thermostat setting and the sequence for the low chilled water and low

i

l

(

.

-

._

-

.

.

.

--

.

.

>

. -__ ..

_ . _ - . . . _ -

.

_ . _ . _ _ _ . _ _ . _ _ . _ _ _ _ . _ _ . . _ _ . _ _

.

4

l

-2-

,

refrigerant temperature chiller trips was not in accordance with procedural

recommendations. These deficiencies reflected inattention to detail by an l&C

technician (Section M8.1).

Enoineerina

i

Engineering personnel demonstrated poor assessment of equipment operability and

i

understanding of information contained in a vendor manual. A flawed operability

!

assessment when an emergency chiller could not achieve design cooling resulted in an

!

inadvertent inoperability of the chiller for approximately 20 days during August 1998.

!

The failure of engineers to take appropriate actions to demonstrate operability of the

Train B emergency chiller, as specified by the corrective action program, was identified

as a noncited violation of 10 CFR Part 50, Appendix B, Criterion XVI, " Corrective

Action," consistent with Section Vll.B.1 of the Enforcement Policy. Had the licensee

,

implemented the vendor manual recommended performance monitoring of the chillers,

l

the buildup of noncondensable gases in the refrigerant during August and

September 1998 could have been detected (Section E8.1).

An engineering evaluation of Chiller E335 operability from September 3 to 25 was

generally thorough. However, some of the reasons for the chiller inoperability, including

introduction of oil and noncondensable gases into the refrigerant, were not completely

understood by the licensee until questioned by the inspectors (Section E8.2).

Licensee use of equipment specific configurations and human reliability analysis in the

=

risk assessment associated with the inadvertent Train B chiller inoperability was a

strength. The overall increase in risk because of the inoperable chiller was potentially

risk significant (Section E8.3).

i

i

,

,

,

,

,

,,v--

-

. - . - - . _ .

. -

..

. - - - .-.

- . - _ . - - -

_ ~ . - - - . ~ - .

I

!

-

Report Details

Summary of Plant Status

l

Units 2 and 3 operated at essentially 100 percent power during this inspection period.

J

l

l. Operations

-

j

08

Miscellaneous Operations lasues (92700)

,

08.1

f, Closed) Licensee Event Reoorts 50-361: 362/98-020-00 and 98-021-00: emergency

)

chilled water (ECW) inoperable because of faulty temperature control unit and ECW

!

inoperable because of an incorrectly wired switch.

a.

Inspection Scope

'

The inspectors reviewed the circumstances conceming two periods of inadvertent

inoperability of Units 2 and 3 Train B ECW, as pertains to operations. The inspectors

reviewed Procedure SO23-1-3.1," Emergency Chilled Water System Operation,"

Temporary Change Notice 10-2, and Design Bases Document SO23-800, " Auxiliary

Building Chilled Water System," Revisions 0 and 2. The inspectors reviewed portions of

Work Action Request (WAR) C-9802021 and a work schedule for implementing this

WAR prepared on August 13,1998. The inspectors reviewed portions of the chiller

- vendor manual Technical Manual SO23-410-7-164-2, " Operating Instructions for Carrier

Centrifugal Refrigeration Machines," Revision 2. The inspectors also interviewed

operators and operations management personnel,

b.

Observations and Findinos

b.1

Seouence of Events

As described in the licensee event reports, Chiller E335, Train B emergency chiller

' (common to both units), was inadvertently inoperable for 20 continuous days in

August 1998 and for 22 continuous days in September 1998.

On August 6, Chiller E335 was operated to support a monthly control room

essential air cleanup system surveillance. Operators observed that the chilled

water temperature leaving the chiller decreased to approximately 51 *F; however,

the normal chilled water outlet temperature was 43'F. Operators considered

Chiller E335 operable; however, they generated an action request (AR) to have

j

engineering assess operability.

'

On August 26, operators removed Chiller E335 from service for scheduled

maintenance and declared the emergency chiller inoperable.

On August 28, I&C technicians replaced the temperature control unit. Operators

performed a postmaintenance test that demonstrated satisfactory performance

3

and declared Chiller E335 operable.

l

I

,-

- , _ _

-

_

.

_

_

>

i

.

I

-2-

On August 31, operators removed Chiller E335 from service for scheduled

maintenance.

,

On September 3, an l&C technician disconnected the leads from a low

temperature chilled water cutout switch, in order to perform a dynamic calibration

j

of the switch setpoint. During the dynamic calibration, the l&C technician

'

stopped the calibration after hearing the emergency chiller make a rumbling

noise.

On September 4, operators declared Chiller E335 operable after verifying that

the chiller continued to operate satisfactorily; however, the operators did not start

Chiller E335.

1

On September 25, Chiller E335 failed to start when operators attempted to

perform a monthly control room essential air cleanup system surveillance.

Subsequently, the licensee determined that the l&C technician had incorrectly

wired the low chilled water temperature cutout switch for Chiller E335, and the

postmaintenance test failed to detect the error. This error rendered Chiller E335

inoperable from September 4 to 25.

During troubleshooting on and after September 25, the licensee determined that

noncondensable gases present in Chiller E335 from August 6 to September 25

degraded the capability of the chiller condenser. In October 1998, as part of the

operability review, the licensee determined that Chiller E335 had been degraded

from August 6-26, because of several factors: (1) a faulty temperature control

unit, (2) an incorrectly set electrical demand switch in the control circuit, and (3)

the presence of noncondensable gases in the refrigerant. The licensee

determined that these factors combined with a 51"F chilled water outlet

temperature rendered Chiller E335 inoperable under design bases load

conditions.

b.2

Postmaintenance Testina

Chiller E335 was removed from service for scheduled maintenance on August 31.

WAP, C-9802021 and a work schedule prepared on August 13 directed that Chiller E335

be operated in accordance with Procedure SO23-1-3.1 to verify operability following the

maintenance. Concurrent with the operability test, a dynamic calibration of low chilled

water and low refrigerant temperature cutout switches was scheduled to be performed.

The design chilled water outlet temperature for Chiller E335 is 43'F. The low chilled

water temperature cutout switch has a nominal setpoint of 38'F. The switch contacts

are normally closed above 38*F and normally open below 38'F. When the switch

contacts open, the chiller compressor trips and the chiller shuts down. The switch

contacts were installed in the start circuit for the chiller, such that the chiller would not

start manually or automatically with the contacts open. During the dynamic calibration,

an I&C technician disconnected the wires from the low chilled water temperature cutout

switch and installed a jumper to disable the switch cutout from tripping the chiller.

Because of personnel error, an l&C technician had reterminated the switch wires

_

. _ _ _ _ _ . _ _ . _ _ _ _ _ _ _ _ . _

_ _ _ _ . _ _ . _ _ _ . _ _ _ _ . _ _ _ _

.

-3-

incorrectly. In addition, although the dynamic calibration affected contacts in the chiller

start circuit, the postmaintenance test did not test the capability of the start circuit to

start the emergency chiller. Failing to provide instructions for postmaintenance testing

appropriate to the circumstances was a violation of 10 CFR Part 50, Appendix B,

,

Criterion V, " Instructions, Procedures, and Drawings."

i

The licensee initiated the following corrective actions to prevent recurrence: (1) add a

postmaintenance test requirement to start the chiller after a dynamic calibration and

l

(2) assess the adequacy of the retest program and evaluate the committee that met to

approve retest requirements. This nonrepetitive, licensee-identified and corrected

i

violation is being treated as a noncited violation, consistent with Section Vll.B.1 of the

l

NRC Enforcement Policy (50 361; 362/98018-01).

)

b.3

Technical Specifications

The licensee determined that Chiller E335 was inadvertently inoperable from August 6

l

to 26, and from September 3 to 25 (periods of 20 days and 22 days, respectively).

!

Unit 3 was in Mode 1 operation for this entire period. Unit 2 was in Mode 1 for 22

consecutive days in August and the first 15 of the 22 consecutive days in September.

Technical Specification Limiting Condition for Operation 3.7.10 directs that, with one

r

train of ECW inoperable, each unit shall restore the train to operability in 7 days. If the

train is not restored by the end of the 7 days, then the limiting condition for operation

requires operators to place the unit in Mode 3 in 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and be in Mode 5 in 36 hours4.166667e-4 days <br />0.01 hours <br />5.952381e-5 weeks <br />1.3698e-5 months <br />.

Because operators were not aware that Train B ECW was inoperable, these completion

i

times were not complied with, which resulted in an apparent violation of Technical

Specification Limiting Condition for Operation 3.7.10 (50-361; 362/98018-02).

i

The inspectors verified that Train A ECW remained operable throughout the 2-month

,

period. Operations had declared Train A ECW inoperable from September 15 to 16,

i

because of an air leak on the expansion tank, which caused the tank to fill with water

,

and pressure to decrease. Also, the operators had declared the Train A Emergency

l

Diesel Generator (EDG) 2G002 (the power source Chiller E336, Train A emergency

chiller) inoperable on September 15 because the fuel transfer pump was returned to

service without the required inservice test. As discussed below, the Train A chiller and

l

the Train A EDG were shown operable throughout the period the Train B emergency

chiller was inoperable,

b.4

Train A Operability Assessments

l

For the occurrence with the Train A expansion tank full of water, the expansion tank

pressure increased to approximately 15 psig, which was the hydrostatic pressure that

l

resulted from 70 feet of water in the chilled water piping above the expansion tank. The

>

chilled water pump required approximately 4 psig suction pressure to avoid cavitation

,

from a loss of net positive suction head. This 15 psig hydrostatic pressum exceeded the

4 psig required suction pressure. The chilled water pump would have provided 54 psig

discharge pressure, given an expansion tank at 15 psig. The 54 psig discharge

pressure would have been sufficient to pump chilled water throughout the chilled water

i

!

piping. The licensee demonstrated, via analysis in AR 980900850, that some gas

!

i

i

_. - . -

_ _

.

-4-

pockets and separation of water from gas may have occurred on the chilled water pump

suction side high points; however, the licensee determined that the chilled water flow

would have been sufficient to collapse or mix these gas pockets, such that flow through

all sections of piping would be maintained. Consequently, the inspectors found that the

Train A emergency chilled water system could perform its intended function under these

conditions.

A second occurrence resulted with Train A chilled water powered by EDG 2G002, which

was declared inoperable because both fuel transfer pumps were inoperable. Each EDG

has two fuel transfer pumps that transfer fuel from the EDG fuel storage tank to the day

tank. On September 15, one EDG fuel transfer pump (Pump 2P096) was removed from

service for maintenance. Subsequently, this fuel transfer pump was declared operable,

and the second EDG fuel transfer pump (Pump 2P093) was removed from service for

maintenance. While the second fuel transfer pump was removed from service,

engineers decided that the retest for the first fuel transfer pump should have included an

inservice test because the pump had been decoupled from its motor during the

maintenance. Consequently, for a period of time, operators declared both fuel transfer

pumps inoperable. The first fuel transfer pump passed an inservice test that was

subsequently administered. Since one fuel transfer pump had remained available, the

inspectors found that the fuel transfer system for EDG 2G002 could perform its intended

function.

b.5

Electrical Demand

Train B Chiller E335 had an electrical demand setting on the control module that could

limit the amount of current that the chiller compressor motor would draw. Prior to

September 25, Procedure SO23-1-3.1 directed operators, during prestart checks of the

chiller, to set electrical demand to 80 percent. This electrical demand setting of

80 percent limited the Chiller E335 to 85 percent of nameplate capacity and contributed

to Chiller E335 inoperability during August and September 1998 because this limited the

capacity of the chiller.

Setting the electrical demand to 80 percent was not referenced in any design

documents, and the licensee could not present a bases for this setting. The vendor

manual stated that the variable electrical demand limited chiller capacity during colder

periods, so that the chiller would not unnecessarily consume electrical power. The

design bases for Chiller E335, as stated in Design Bases Document SO23-800,

6

indicated that the chiller would provide 4.8 x 10 BTU / hour cooling. However,

6

Procedure SO23-1-3.1 limited the chiller capacity to approximately 4.1 x 10 BTU / hour

when the electrical demand was set at 80 percent. While less than the design basis

specification, the licensee stated that the limited chiller capacity was still sufficient to

cool design basis heat loads, if the chiller was functioning properly.10 CFR Part 50,

Appendix B, Criterion 111, " Design Control," states, in part, that the design bases for

components shall be correctly translated into procedures. The failure to translate design

basis requirements for chiller capacity into operations procedures for setting electrical

demand is a violation of NRC requirements. Upon discovery of this condition, the

licensee revised Procedure 8023-1-3.1 to require setting electrical demand at

100 percent. The inspectors found this corrective action satisfactory. This

.

. ___

__

_ _ . _ _ _ . _ _ _ _ _ - _

_ _ _ _ ___._..____.__-

.

'

-5-

i

l

i

nonrepetitive, licensee-identified and corrected violation is being treated as a noncited

'

,

l

violason, consistent with Section Vll.B.1 of the NRC Enforcement Policy

(50-3611362/98018-03).

Also, on September 30, operators had started the Train A Chiller E336 with the electrical

,

l

demand set at 80 percent; however, the motor current slightly exceeded the nameplate

l

l

value, indicating that electrical demand switch was not properly limiting current. The

!

licensee calibrated the electrical demand for Train A Chiller E336 and verified that the

electrical demand for Train B Chiller E335 met specifications. Although the vendor

i

manual did not have a recommended calibration interval for operational settings, such

as electrical demand, the licensee had calibrated electrical demand on 6-year intervals.

From August 28 to 31, the inspectors determined that I&C technicians had set electrical

demand at 100 percent for Chiller E335 after replacing the temperature control unit.

Although this setting met the design bases and maintenance procedure requirements,

the inspectors noted that the setting conflicted with the requirements specified in

i

Procedure SO23-1-3.1. Consequently, the inspectors found that control of chiller

'

,

electrical demand was poor and contributed to the noncited violation of design control

(

since the maintenance and operations procedures conflicted.

l

c.

Conclusions

Postmaintenance testing, as prescribed by operations work control, for scheduled

emergency chiller maintenance was not adequate because work that affected the start

circuit for the chiller was not tested. This deficiency was identified as a noncited

violation of 10 CFR Part 50, Appendix B, Criterion V, consistent with Section Vll.B.1 of

the Enforcement Policy. This deficiency contributed to inoperability of Chiller E335 for

22 days in September 1998.

1

A faulty temperature control unit, an incorrectly set electrical demand switch, and

noncondensable gases rendered Chiller E335 inoperable for 20 consecutive days in

August 1998. Similarly, a miswired low chilled water temperature cutout switch or a

combination of noncondensable gases, oilin the refrigerant, and an electrical demand

,

setting of 80 percent rendered Chiller E335 inoperable for 22 consecutive days in

September 1998. The time periods for these inoperabilities exceeded the Technical

Specifications Limiting Condition for Operation 3.7.10 allowed outage time and was

identified as an apparent violation.

The emergency chilled water system operating procedure specified that the electrical

demand be limited to 80 percent, which reduced chiller capacity below that assumed in

design bases document. The failure to translate the design basis requirement for

emergency chiller capacity in the operations procedures for setting electrical demand

contributed to emergency Chiller E 335 in operability during August and

September 1998 was identified as a noncited violation of 10 CFR Part 50, Appendix B,

'

Criterion ill, " Design Control." However, this deficiency alone did not render the

emergency chiller inoperable. This nonrepetitive, licensee-identified and corrected

'

violation is being treated as a noncited violation consistent with Section Vll.B.1 of the

o

NRC Enforcement Policy.

i

!

,

-

-

-

--

- - - _ ,

l

.

!

{

i

-6-

,

08.2 Chiller Oil Pumo Ooeration

)

a.

Inspection Scoce (92700)

The inspectors reviewed the circumstances surrounding operators lowering Chiller E335

oillevel. The inspectors reviewed AR 980900388 and portions of

Procedure SO23-1-3.1 and interviewed station technical personnel.

b. - Observations and Findinos

On September 7,1998, operators noted that, with Chiller E335 in standby, compressor

oil level was out-of-sight high in the upper sight glass. Chiller E335 has both upper and

lower bulls-eye sight glasses vertically aligned to indicate compressor oil level.

l

Procedure SO23-1-3.1, step 6.1.10.3, directed for compressor oil level out-of-sight high

!

that operators run the lube oil pump for approximately 15 seconds to separate

refrigerant from oil and then recheck the oillevel. Operators followed this procedure

step and confirmed that the oil level lowered within the top sight glass and met the

specification of Procedure SO23-1-3.1, which required the oil level to be less than the

top sight glass and above one-half of the lower sight glass. Subsequently, operators

ran the oil pump a second time for approximately 3-4 minutes, which decreased the oil

level to approximately seven-eighths of the lower sight glass,

j

Operators generated AR 980900388, which documented the oil pump starts and

recommended procedure changes to incorporate operating the oil pump for

'

3-3% minutes in order to more completely lower the oil level. Engineers rejected the

procedure change during the AR disposition because the additional oil pump operation

introduced oilinto the cooler and degraded Chiller E335 performance (refer to

Section E8.2).' The inspectors found that the operators involved failed to recognize that

excessive oil pump operation, with Chiller E335 in standby, could degrade chiller

performance when the chiller was started.

The licensee initiated the following corrective actions to correct this deficiency:

(1) changed Procedure SO23-1-3.1 to emphasize the limits on oil pump operating time

and frequency with the chiller in standby, (2) scheduled additional operator training in

chiller dynamics, and (3) initiated a Level 3 event report to assess corrective actions for

operators failing to correctly interpret what "approximately" meant. Failure to follow

Procedure SO23-1-3.1 violated Technical Specification 5.5.1.1.a; however, the

inspectors found the corrective actions for this deficiency to be adequate, consequently

no response to the violation is required (50-361; 362/98018-04).

c.

Conclusions

Operators demonstrated poor awareness of the effects of compressor oil pump

operation on a standby emergency chiller. Operators operated the oil pump with the

compressor in standby for greater than the time allowed by procedure in attempts to

lower a high oil level. The inspectors identified this as a failure to follow a procedure

required by Technical Specification 5.5.1.1.a. Since the licensee implemented

appropriate corrective actions, no response was required.

..

_

._.

.

.

-

1

!,

t

.

-

-7-

]

11. Maintenance

y

,t-

M8

Miscellaneous Maintenance issues (92700)

M8.1 Maintenance of Emeroency Chiller E335

i

l

a.

Inspection Scope

The inspectors reviewed the circumstances concerning two periods of inadvertent

inoperability of Units 2 and 3 Train B ECW, as pertains to maintenance. The inspectors

l

reviewed Licensee Event Reports 50-361; 362/98-020-00 and 98-021-00. The

'

inspectors reviewed Procedure S023-11-8.25 and interviewed l&C technicians, as well as

'l

,

maintenance supervisors. The inspectors reviewed portions of

I

Procedure SO123-ll-15.3," Temporary System Alteration and Restoration Form,

j

Revision 7, and portions of Technical Manual SO23-410-7-164-2, Revision 2.

i

b.

Observations and Findinas

b.1

Lif ted Lead Control

,

On September 3,1998, an l&C technician disconnected the leads from the Chiller E335

low chilled water temperature cutout switch, as instructed by Procedure SO23-ll-8.25, in

i

order to perform a dynamic calibration of the switch setpoint (nominally 38"F) and reset

point (nominally 48'F). As directed by Procedure SO23-il-6.25, a day shift l&C

]

technician jumpered out the low refrigerant temperature and low chilled water

i

temperature cutouts and disconnected the wires from both of these switches. A swing

shift l&C technician continued with the calibration by lowering thermostat temperature,

until the refrigerant switch actuated. When the thermostat was lowered further, to

provide approximately 38"F chilled water temperature (nominal switch setpoint), the

chiller began to make loud, rumbling noises. In addition, cooler pressure lowered to

about 30 psi, the minimum cooler pressure allowed by Procedure SO23-ll-8.25. The

I&C technician decided to back out of the calibration and restore the chiller.

I

Procedure SO23-ll-8.25, step 6.3.3, directed that the wires removed from the low chilled

water temperature cutout switch be reconnected. The low chilled water temperature

switch contact is normally closed above switch setpoint (chilled water temperature above

38"F) to allow operation. The low chilled water temperature switch contact opens to

either trip the chiller and prevent chiller automatic and manual start, when the switch is

I

below setpoint (chilled water temperature below 38"F). The switch has four connection

points for the two wires. The technician reconnected the wires to the connection point

screws that corresponded to a closed contact by using a voltmeter to establish these

points. The technician then reported to operations work control that he was unable to

l

complete the dynamic calibration.

I

!

The licensee determined that, on September 25, when Chiller E335 failed to start, the

I

l&C *echnician had connected the wires to the wrong screws. The low temperature

cutout switch had changed state, such that the I&C technician had unknowingly

I

.

-

- - - . - - - - - - - - - -

-

-

.=

l

.

i

'

,

~,

)-

-8-

1

!

connected the wires to the normally open contacts. The emergency chiller had not

i

tripped because the reset point of the switch had not been reached. The failure to

l

reconnect the low temperature cutout switch wires did not meet the requirements of

i

l

Procedure SO23-Il-8.25, step 6.6.3; consequently, the failure to properly implement a

{

!

procedure recommended in Regulatory Guide 1.33 was a violation of Technical

!

Specification 5.5.1.1.a.

i

l

As described in Procedure S0123-11-15.3, the inspectors noted that the licensee

controlled jumpers and lifted leads connected or disconnected during maintenance

activities in three primary ways: (1) used an alteration form, (2) controlled in a

l

procedure, or (3) verified by a test or other indication that would demonstrate adequate

i

!

restoration. The I&C technician's activities described above were controlled by

i

procedure, as allowed by the lifted lead and jumper program. The inspectors found that,

!

to an extent, the technician demonstrated poor skill-of-the-craft by not recognizing that

l

the low temperature switch may have changed state. The inspectors also found that, to

l

an extent, the program for controlling lifted leads and jumpers was weak. The exact

i

location of jumpers and positions that leads were lifted from was not required to be

l

recorded as work progressed when these activities were controlled by procedure and

when a functional test would be performed. Both the procedure weakness and

i

i

personnel error contributed to the violation described above.

i

Following identification of this deficiency, the licensee implemented the following

corrective actions: (1) conducting a site-wide standdown on November 20 to discuss

j

this occurrence with all site personnel, (2) requiring the use of an alteration form for all

I

jumpers and lif ted leads as an interim measure, while conducting a review of their

program for controlling lifted leads (3) assessing the adequacy of the alteration form,

and (4) reviewing the occurrence for disciplinary action. The inspectors found these

corrective actions adequate. This nonrepetitive, licensee-identified and corrected

violation is being treated as a noncited violation, consistent with Section Vll.B.1 of the

NRC Enforcement Policy (50-361; 362/98018-05),

b.2

Thermostat and Temperature Switch Settinos

Procedure SO23-il 8.25 directed that personnel: (1) lower the chiller thermostat setting

l

until the low refrigerant and low chilled water temperature switches actuated, (2) raise

the chiller thermostat setting to check the reset points of the switches, and (3) return the

chiller thermostat to the position corresponding to 43'F. The chiller thermostat was

controlled by a dial on the temperature control module. However, no markings existed

on the dial face or the panel on which the dial was mounted to identify the correct

setting. When the l&C technician attempted the dynamic calibration on September 3,

the procedure step that returned the chiller thermostat to 43*F had been marked *NA,"

and the chiller thermostat was lef t at a lower setting. Based on review of data for the

operation of the chiller on September 4, the inspectors found that operations had

i

corrected the chiller thermostat setting after noting a low chilled water temperature

during operation.

l

The inspectors also noted that the vendor manual and Procedure SO23-ll-8.25

{

recommended that the low refrigerant temperature switch be set such that, as

,

l

l

-.

.

-

- -

-

- _ .

-

,

_-

.,_

l

__ _

_ . -

__.

.

9

refrigerant and chilled water temperature lowered, the low chilled water temperature

switch would actuate before the low refrigerant temperature switch. This was because

the low refrigerant temperature switch had to be locally reset, while the low chilled water

switch would reset when chilled water temperature was 48'F and a 15-minute time

delay had been met. The inspectors found that on September 3 the I&C technician

achieved the low refrigerant temperature switch setpoint before he had achieved the low

chilled water temperature switch setpoint, as he was lowering the thermostat. This

indicated that the switch setpoints were not left in accordance with the recommendations

noted above. The inspectors found that, on September 3, when the l&C technicians

informed operators that Chiller E335 was restored from maintenance, two settings were

left in nonrecommended positions, indicating poor attention to detail.

c.

Conclusions

Maintenance demonstrated poor control of lifted leads and of emergency chiller switch

settings. An l&C technician failed to properly reconnect wires to an emergency chiller

low temperature cutout switch, rendering the chiller inoperable. The failure to follow

procedure was identified as a noncited violation of Technical Specification 5.5.1.1.a.

consistent with Section Vll.B.1 of the Enforcement Policy. During the same

maintenance activity, the chiller was left with an improper thermostat setting and the

sequence for the low chilled water and low refrigerant temperature chiller trips was not

in accordance with procedural recommendations. These deficiencies reflected

inattention to detail by an I&C technician.

lil. Enaineerina

E8

Miscellaneous Engineering issues (92700)

E8.1

Enaineerina Assessment and Support of Emeraency Chiller E335

a.

Insoection Scope

The inspectors reviewed the circumstances concerning two periods of inadvertent

inoperability of Units 2 and 3 Train B ECW, as pertains to engineering. The inspectors

reviewed Licensee Event Reports 50-361; 362/98-020 00 and 98-021-00. The

inspectors reviewed portions of Procedure SO123-XX-1," Action Request / Maintenance

Order initiation and Processing," Revision 9, Issue 2. The inspectors reviewed portions

of Technical Manual SO23-410-7-164-2 and Procedure SO23-1-3.1. The inspectors

reviewed data for Chiller E335 operation at various times from February 1997 until

September 1998 and interviewed station technical and design engineers and

'

supervisors.

.

_

__

__.

._ _ __

_.

.

t

.

-10-

b.

O.bgervations and Findinas

b.1

Ooerability Assessme_n_t

"

On August 6,1998, operators started Chiller E335 to support a control room essential

air cleanup system surveillance. Operators generated AR 980800326 because

Chiller E335 was maintaining 51 *F outlet chilled water temperature, instead of the

design 43*F. The cognizant engineer performed an operability assessment, approved

by an acting supervisor, which concluded the chiller remained operable. The engineer

concluded in the operability assessment that 51 *F chilled water outlet temperature was

acceptable to cool the actualloads on the chiller. However, the operability assessment

did not address increased loading on the chiller that would result from a design basis

accident condition (one unit in a loss of coolant accident and the other shutting down).

The licensee subsequently determined that noncondensable gases in the refrigerant

and problems with the temperature control unit had rendered the chiller inoperable on

August 6. The inadvertent inoperability remained until August 26, when Chiller E335

was declared inoperable for scheduled maintenance, which included replacing the faulty

temperature control unit.

Procedure SO123-XX-1, step 6.6.2.2.5, states, in part, that to accept a condition as-is

and justify that the configuration continues to satisfy design requirements, a

nonconformance report is required. The operability assessment was not accomplished

in accordance with Procedure SO123-XX-1. Utilization of a nonconformance report, in

principle, would have caused increased engineering attention to the design of the chiller

and the effect of the degradation on the chiller. The failure to promptly identify and

correct a condition adverse to quality is a violation of 10 CFR Part 50, Appendix B,

,

Criterion XVI," Corrective Action." The licensee had implemented corrective actions that

included: (1) developing a station technical reading assignment, (2) counseling

personnelinvolved, and (3) considering enhancing standard questions used during

operability assessments. This nonrepetitive, licensee-identified and corrected violation

is being treated as a noncited violation, consistent with Section Vll.B.1 of the NRC

Enforcement Policy (50-361; 362/98018-06).

b.2

Vendor Information

The inspectors noted that the chiller vendor manual recommended that a set of

operating logs be taken when operating the chiller in order to be able to detect

degradation in chiller performance. The recommended logs included chilled water

,

temperature entering and leaving the chiller, refrigerant temperatures and pressures,

'

and component coolirig water temperatures entering and leaving the chiller. The

inspectors determined that Procedure SO23-1-3.1 did not require operators to monitor

J

any of these parameters in order to analyze for performance degradation,

i

In addition, the chiller vendor manual provided saturation tables for refrigerant

temperature and pressure to be used in order to detect noncondensable gases in the

,

f

refrigerant. A refrigerant temper 0ture 2'F below the saturation temperature for any

9 ven pressure was indicative of noncondensable gases. Licensee personnel had not

i

'

j

utilized this information on September 3. nor during other past attempts at dynamic

-

_ .

- .

__

- _

.-

-.

. . _ .

-

.

-11-

i

I

calibration, when refrigerant pressure (limited to 30 psi) prevented completing the

dynamic calibration. This information was not used until September 25 when a vendor

representative suggested that noncondensable gases were present in the refrigerant.

Noncondensable gases had degraded chiller performance for at least 2 months prior to

September 25. The inspectors concluded that station technical engineers demonstrated

weak performance monitoring of the chillers, in that personnel failed to make full use of

the information available in the chiller vendor manual.

c.

Conclusions

Engineering demonstrated poor assessment of equipment operability and understanding

of information contained in a vendor manual. , A flawed operability assessment when an

emergency chiller could not achieve design cooling resulted in an inadvertent

inoperability of the chiller for approximately 20 days during August 1998. The failure of

engineers to take appropriate actions to demonstrate operability of the Train B

emergency chiller, as specified by the corrective action program, was identified as a

noncited violation of 10 CFR Part 50, Appendix B, Criterion XVI, * Corrective Action,"

consistent with Section Vll.B.1 of the Enforcement Policy. Had the licensee

implemented the vendor manual recommended performance monitoring of the chillers,

the buildup of noncondensable gases in the refrigerant during August and September

1998 would have been detected.

E8.2

Effect of Noncondensable Gases on Chiller Performance

a.

Inspection Scope

The inspectors reviewed data of Chiller E335 performance to assess operability,

b.

Observations and Findinas

Licensee Event Reports 50-361; 362/98-020-00 and 98-021-00 described that the

chillers have a 35 percent capacity margin and that the noncondensable gases

degraded the chiller performance by 11 percent. The capacity margin was based on

calculations of design load (one unit in a loss of coolant accident and one shutting

down). The loads on the chiller were divided by the chiller capacity to determine the

excess capacity the chillers possessed. The licensee based the 11 percent degradation

on comparisons of chiller performance when equipment status as well as the presence

of noncondensable gases was known.

The inspectors identified that chilled water exit temperature (as a function of time) data

recorded for September 25,1998, and chiller operation with control room essential air

cleanup system loading contradicted either or both the 35 percent capacity margin or

the 11 percent degradation resulting from the noncondensable gases. The inspectors

noted that, in principle, for the chiller operation on September 25, the chiller should have

had a 9 percent excess capacity for design loading. Specifically, operators had set the

current limiter to 80 percent, which provided 85 percent available capacity. Since

noncondensable gases were present, this 85 percent capacity should have been

reduced to no lower than 74 percent chiller output (based upon the 11 percent

.

- - - . - - - - . - -

- . - - -

- - - - . - - -.- - --

-

.

-12-

degradation). The 74 percent chiller output exceeded the 65 percent chiller output by

.

9 percent, which the licensee stated would maintain the chiller capability within design.

On September 25, Chiller E335 only achieved 49"F chilled water outlet temperature with

significantly less than the design loading; however, the chiller should have easily

i

achieved 43*F chill water outlet temperature. Normally, chiller operation provided for a

relatively sharp drop in chilled water temperature in the first 10 to 20 minutes of chiller

i

operation, as the chiller " worked" hard to lov.er the outlet temperature to the setpoint.

After the setpoint was reached, the chiller would draw less amperage since it would be

easier to mcintain the chill water outlet temperature at the 43*F setpoint. In this

,

instance, for the first hour (the extent of the chiller operation with current demand at

>

80 percent), the chiller operated at the maximum amperage for 80 percent electrical

i

'

demand. Even though the temperature difference between setpoint and actual chilled

i

water outlet temperature (43 versus 49*F) was not that great, the capability of the chiller

"

unit was exceeded, as demonstrated by the controller continuing to generate a full open

l

signal for the refrigerant guide vanes. Consequently, Chiller E335 had operated at

maximum capacity for the conditions reported (74 percent) instead of functioning

1

normally even though, in principle, the chiller capacity exceeded the design capacity

(65 percent) by 9 percent.

In response to the above information, a reanalysis determined that the 11 percent

degradation resulting from noncondensable gases was dependent on current available

to the compressor motor and component cooling water temperature. The degradation

would increase above 11 percent if the chiller was current-limited or component cooling

water temperature was elevated. Consequently, for a current-limited setting of

80 percent, degradation resulting from noncondensable gases would exceed

11 percent. The licensee also stated that, for the inoperability from September 3 to

25, Chiller E335 would have been inoperable because of the noncondensable gases

and the 80 percent current limited setting, independent of the miswired low temperature

cutout switch.

On December 1 station technical engineers informed the inspectors that Chiller E335 oil

pump operation on September 6 (as described in Section 08.2) had introduced

approximately 1 gallon of oilinto the Chiller E335 cooler. This oil had been absorbed by

the refrigerant and, during Chiller E335 startup on September 25, had degraded

Chiller E335 performance. The chiller was designed to remove oilin the refrigerant

through a weep hole that directed oil separated from refrigerant by the action of the

compressor to be returned to the oil reservoir. This would occur during the first 2

to 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> of Chiller E335 operation such that the performance degradation would

j

correct itself.

i

Based on licensee statements, the inspectors found that Chiller E335 was operable from

August 28 to 31. Any additional Chiller E335 pedormance degradation, resulting from

oilin the Refrigerant, had occurred after August 31. The inspectors also noted that

operating the Chiller E335 compressor tube oil pump, in order to lower oil level with the

chiller in standby, was not an action that was mentioned in the chiller vendor manual.

However,15 seconds of operation, as specified in Procedure SO23-1-3.1, should not

adversely affect chiller performance,

l

e

y

~

g

.

k

.

-13-

1

!

Based on the above, the inspectors found that the retrospective evaluation of operability

of Chiller E335, as stated in Licensee Event Report 50-361; 362/98-021-00, was

,

accurate and generally thorough. However, the reasons for Chiller E335 inoperability

i

from September 3 to 25 included more than the miswirirg of the low chilled water

,

temperature cutout switch, independent of the switch miswiring, Chiller E335 was also

l

inoperable because of the combination of noncondensable gases and oilin the

l

refrigerant. The chiller performance degradstion caused by the oil and noncondensable

gases was compounded by the chiller electrical demand being set at 80 percent from

i

September 3 to 25, but a setting of 100 percent would have still rendered Chiller E335

inoperable based on the effects of the oil and noncondensable gases in the refrigerant.

!

j

c.

Conclusions

An engineering evaluation of Chiller E335 operability from September 3 to 25 was

generally thorough. However, some of the reasons for the chiller inoperability, including

,

introduction of oil and noncondensable gases into the refrigerant, were not completely

!

understood by the licensee until questioned by the inspectors.

E8.3 Licensee Assessment of Risk

a.

Inspection Scope

The inspectors reviewed the licensee assessment of the increase in risk that resulted

from the inadvertent chiller inoperability. The inspectors reviewed NSG/PRA

i

'

Report NSG-98-021,"The unavailability of Emergency Chiller SA1513ME335," dated

October 23,1998, and had discussions with licensee representatives.

.

b.

Observations and Findinas

The risk assessment determined that the Train B chiller inoperability during August and

September 1998 increased the risk in Unit 2 by 8.9E-06 and Unit 3 by 9.6E-06. The

licensee concluded that the increr.ses in risk were small when using the criteria of

Regulatory Guide 1.174, "An Approach for Using Probabilistic Risk Assessment in

Risk informed Decisions on Plant-Specific Changes to the Licensing Basis." The

inspectors noted that the risk analysis was comprehensive in that specific plant

configurations had been used to assess the impact of the chiller outage. Additionally, a

human reliability analysis had been performed to evaluate the operator performance

issues associated with the chiller inoperability.

The inspectors noted that Regulatory Guide 1.174 was intended to evaluate plant-

specific changes to the licensing bases and not necessarily to evaluate the risk

significance of plant configurations or events. In addition, the inspectors noted that,

even when using Regulatory Guide 1.174, the overall risk bordered on the region of

potential risk significance.

--

-

-

-

._.

-

-. .

---

,

-

- _

.

.

..

.

.

_.

.. .

_

_ _ _ _ _ _ _

.

.

.

&

-14-

c.

Conclusions

Licensee use of equipment specific configurations and human reliability analysis in the

risk assessment associated with the inadvertent Train B chiller inoperability was a

strength. The overall increase in risk because of the inoperable chiller was potentially

risk significant.

V. Manseement Meetinas

X1

Exit Meeting Summary

The inspectors conducted a status meeting with members of licensee management on

November 13,1998.

The inspectors presented the inspection results to members of licensee management at

the exit meeting on December 18,1998. The licensee acknowledged the findings

presented.

The inspectors asked the licensee whether any materials examined during the

inspection should be considered proprietary. No proprietary information was identified.

.

1

i.,,

- .

.

. . . . .

. ..

.

_ . _ , _

__

_.

. . .

-

.-

-

. - . ~ .._

- -

- .

. - . _

. - . -

_ . - - . . - .

. . _

-

i

!

'

f

'

ATTACHMENT

l

SUPPLEMENTAL INFORMATION

i

!

EARTIAL LIST OF PERSONS CONTACTED

Licensee

j

D. Brieg, Manager, Station Technical

J. Fee, Manager, Maintenance

i

D. Herbst, Manager, Site Quality Assurance

i

R. Krieger, Vice President Nuclear Generation

A. Scherer, Manager, Nuclear Regulatory Affairs

!

D. Nunn, Vice President, Engineering and Technical Services

!

T. Vogt. Plant Superintendent, Units 2 and 3

R. Waldo, Manager, Operations

!

INSPECTION PROCEDURE USEiQ

l

<

IP 92700:

On Site LER Review

i

!

IIEMS OPENED AND CLOSED

Ooened

!

361; 362/98018-02

eel Technical Specification LCO completion times not complied with

J

(Section 08.1.b.3)

Closed

361;362/98-020-00

LER ECW inoperable due to faulty temperature control unit switch

I

(Section 08.1)

)

i

361; 362/98-021-00

LER ECW inoperable due to incorrectly wired switch (Section 08.1)

Opened and Closed

361; 362/98018-01

NCV inadequate postmaintenance test (Section 08.1.b.2)

361;362/98018-03

NCV emergency chiller capacity reduction not in accordance with

design basis (Section 08.1.b.5)

361;362/98018-04

VIO failure to follow procedures (Section 08.2.b)

. _ -

.

--

-

--

.

.

_

.

.

2

361;362/98018-05

NCV failure to reconnect low temperature cutout switches

(Section M8.1.b.1)

361;362/98018-06

NCV failure to perform adequate operability assessment

(Section E8.1.b.1)

LIST OF ACRONYMS USED

AR

action request

CFR

Code of Federal Regulations

ECW

emergency chill water

-

EDG

emergency diesel generator

I&C

instrumentation and control

NRC

Nuclear Regulatory Commission

PDR

Public Document Room

WAR

work action request