IR 05000498/1993003

Insp Repts 50-498/93-03 & 50-499/93-03 on 930112-29. Apparent Violations Noted.Major Areas Inspected:Circumstance Surrounding Drift of Nuclear Instrumentation Setpoints & Failure of Unit 1 EDG 13 to Start

ML20128P030
Person / Time
Site:	South Texas
Issue date:	02/17/1993
From:	Stetka T NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20128P016	List: IR 05000498/1993003 ML20128P010
References
50-498-93-03, 50-498-93-3, 50-499-93-03, 50-499-93-3, NUDOCS 9302240065
Download: ML20128P030 (10)

Text

_

-

,

. .

.

A>PENDIX A U.S. NUCLEAR REGULATORY COMMISSION ,

REGION IV  ;

NRC Inspection Report: 50-498/93-03 50-499/93-03 Operating License: NPF-76 NPT-80

'

Licensee: Houston Lighting & Power Company P.O. Box 1700 Houston, Texas 77251 facility Name: South Texas Project Electric Generating Statio Units 1 and 2 lospection At: Matagorda County, Texas inspection Conducted: January 12-29, 199?

inspectors: J. 1. Tapia, Senior Resident inspector R. J. Evans, Resident inspect',r

/ proved: &/l? !'f"3

~

T. f. Stetka,' Thief, Project Section D Date

'

in_spe ion Summary Areas inspected: A special inspection was conducted to determine the circumstances surrounding the drift of nuclear instrumentation setpoints and the failure of Unit-1 Emergency Diesel Generator (EDG) 13 to start. The inspection also reviewed previously identified problems with personnel error Results: >

• One apparent violation was identified that involved eight examples of a failure to follow procedural requirements for performing self-verificatio These examples, of which seven were previously identified and documented as unresolved items in NRC inspections, represent instances in which work was performed on the wrong component, wrong train, and, in one case, on the wrong unit (Sections 1.3, 3.1, and 3.2).

• An unresolved item was identified involving EDG availability and mode change instruction '

• The verificatien process associated with setpoints and the lack of procedural requirements for. assuring independent verification of the 9302240065 930218" PDR ADOCK 05000498 C PDR .i

.. - .. _

- _ ___- _ _ _ -- _ -_ . - _ _ - - _ _ _ _ _ - _ - _ _ _ _ _ _ _ . __ _ _ _ _ _ _ _ _ . . _ _ _ _ _ _ _ _ - _ _ _ - - _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ - _ _ _ _ _ -

t 4 .

.

..

,

e-2- ,

l

nuclear instrumentation system are considered a weakness and a i contributing cause of the apparent violation identified in Section ,

(Section 1.4).

$unanary of Inspecti.on findings:

• Unresolved item 499/9229-03 was closed (Section 3.1), f

Unresolved item 498:499/9232-02 was closed (Section 3.2).

• Apparent Violation 498 499/9303-01 was opened (Sections l'.3)'.- -I

• Unresolved item 498/9303-02 was opened (Section 2.3).

Attachments and/or [nclosures: ,

• Attachment 1 - Persons Contacted and Exit Meeting i I

. '

& .

.

-!

-i

. I-

,

!'

,

?$

. - ,, , ,. ..

l

. .

.

-3-DETAILS 1 HIGH FLUX TRIP SETPOINT ERRORS 1.1 Seouence of Events On January 4, 1993, during power ascension from 60 to 77 percent power, the Unit I neutron flux high setpoint was adjusted in accordance with TS 2. from 84.5 to 109 percent after exceeding 75 percent rated thermal power. All four channels of power range neutron flux high setpoint were adjusted in accordance with Procedure OPSP02-HI-0040, Revision 0, " Power Range Channels N-0041, N-0042, N-0043, and N-0044 Overpower Trip High Range Sotpoint Adjustment." Procedure OPSP02-N1-0040, Step 7.2.4, requires recording the as-found trip value, monitored at the percent full power meter, on form (-01).

On January 8, 1993, when reactor power was reduced below 75 percent to allow for power range nuclear instrumentation (NI) and incore detector work, the neutron flux high setpoint was again adjusted from 109 to 84.5 percen During this adjustment, the as-found setting for Channel N-0044 was found at 112 percent, 3 percent higher than the required value. The trip setpoint was reduced to 84.5 percent within 10 minutes and the requirements of TS 3. were satisfied. The licensee initially suspected electrical drift as the cause of the misadjusted setpoint. The instrumentation and controls (l&C)

technician issued Station Problem Report (SPR) 930076 to document the identified condition and initiate an investigation to determine the caus On January 9,1993, following completion of the Ni and incore detector work, and increasing power level to greater than 75 percent, the Unit I neutron flux high setpoint was again adjusted from 84.5 to 109 percent on all four channel On January 11, 1993, the licensee elected to reduce reactor power from 90 to below 75 percent rated thermal power as a result of problems with a potentiometer card that serves to bias the output _of Power Range-Channel N-0042 prior to being . input as a flux penalty to the OverTemperature-Delta'lemperature reactor trip circuitry. As a result of reducing power, the neutron flux high setpoint was again adjusted from 109 to 84.5 percent in order to preclude having to perform an incore flux map to determine the quadrant power tilt ratio every 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> in accordance with TS Surveillance Requirement 4.2. On January 11,-1993, the as-found overpower trip high range setpoint for Channel N-0041 was 99 percent. The other three channels were found at the correct setting of 109 percent. The licensee initially suspected that the channel had experienced drift since last being adjusted on January 9, 199 An I&C_ technician issued SPR 930100 to further investigate the cause of the misadjusted channel- setting.

h

_ _ _ _ _ _ . _ _ _ _ _ _ _ _ ._ . _ _ _ _ - _ - _ _ _ - _ - _ _ _ _-_

. ,

• .

i

!

-4- l I

!

1.2 Licensee's investigative Results l

i Subsequent investigation by the licensee included a review of potential common i denominators in equipment, personnel, procedures, and measuring & test ,

equipment for both events. The licensee's review resulted in_ preliminary root- '

causes for both event i

!

With respect to Channel N-0044 being found at 112 percent, it was determined  :

that previous corrective maintenance on the rod stop bistable had resulted in i a static charge causing an Ni panel meter drag of 3 percent and subsequent- -!

erroneous setting of the voltage comparator. The bistable setting was verified to be correctly set by voltage equivalency measurements. . When the ,

neutron flux high setpoint was adjusted on January 8,1993, the reading of *

112 percent was the resulting indication. The licensee is continuing to-evaluate methods for eliminating-the effect of static charge on meter faces, in order to provide added assurance that the erroneous setting was in fact due__

to static diag of the meter, the licensee is performing increased frequency  !

testing of the high flux trip s tpoint bistabl The inspectors will monitor the licensee's efforts in this are l

,

i The licensee's review disclosed that Channel N-0041 had been misadjusted as a i result of personnel error. Several factors were initially considered as possible contributors to the execution of the error. These factors included _ 3 the rapidity of performance of the surveillance procedure, the adequacy of the '

procedure, and possible schedule pressure The inspector independently assessed each of these factor .3 Procedurejmplementation A review was conducted of all instances when Surveillance  !

Procedure OPSP02-NI-0040 was implemented in Unit 1. Twenty-two surveillances +

- from May 1989 to January.1993 were reviewed. The time to implement the l surveillance ranged from a maximum of 124 minutes on_ June 30, 1990, to a- i minimum of 44 minutes on August 30, 1990. The average time to perform the surveillance was 82.5 minutes. The performance on' January 9, 1993,_ required 47 minutes, the second fastest time. Both the fastest time of August _30, .

1990, and the January 9,1993, time were 1erformed by. the same technicia .

From this analysis, it is apparent that' tie performance _of this surveillance-  ;

procedure, by the technician who performed it during the occasion when the

'

error occurred, was atypical inasmuch as it.was aerformed in a very rapid manner. The licensee does not consider this to Je a contributor to the execution _ of the error _ by the technician because of his experience 1and:

familiarity with the procedure. . However, the inspector ' concluded that '

experience and familc arity with a procedure do not1 provide adequate assurance that an error will rat occur if the procedure is implemented in a rapid' ,

- manne According to the. licensee, the technician did not. adequately perform; self- *

'

'

verification of the procedural step where'the panel meter was adjusted. Plant Procedure OPGP03-ZA-0010, Revision 15 " Plant Procedure Adherence and ,

--

_

-- - i r < ,. : __. , i;,..w. ,.,n.,a,.,,._.%,.,-, J.--,,,,,,u , . - ,- .--w _. , , .

. _ _ _ _ . - _ _ _ _ _ _ _ _ _ __ __

. .

.

-.

'

5-Implementation and Inde)endent Verification," Step 4.2, " Implementation of l Procedures," requires taat, for the performance of procedures which require .

manipulation of plant equipment, the individual performing that step shall ~

first implement the seven-step self-verification program. The ,

self-verification program was dafined in the procedure and requires that an '

employee stop and think about the task, locate the device to be operated, l touch the device, verify the correctness of the device, anticipate the expected and unexpected responses and the actions to be taken, manipulate the component, and observe for expected and unexpected responses. The technician failed to adhere to the procedural requirements for self-verification. This is an example of a failure to follow procedures and is considered to be an '

apparent violation (498;499/9303-01), edeauacy of Procedure The surveillance procedure was reviewed to determine its possible role in causing the error. The procedural steps to adjust a high flux trip bistable relay essentially consists of the following: the power range channel being aligned is bypassed, the operation selector switch is taken from normal to the-DET A & B position, the TRIP ADJ potentiometer is rotated to zero, the front panel DETECTOR A and B TEST SIGNAL potentiometers are rotated as necessary to obtain the desired percent indication reading (required trip value) on the PERCENT FULL POWER meter, then the TRIP ADJ potentiometer is rotated until the OVERPOWER TRIP HIGH RANGE indicator becomes illuminate These steps did.not require independent verification. The technician made the error when he .

erroneously set the required trip value at 99 instead of 109 percent and did not practice self-verificatio Subsequent to the identification of the error, the licensee issued field Change 93-0019 to Procedure OPSP-N1-0040 to provide independent verification steps for the as-found and as-left trip setpoints. The procedure does require independent verification of various other steps and the field change is

-

considered a further enhancement. However, the procedure that requires independent verification. Procedure OPGP03-ZA-0010. Revision 15, " Plant Procedure Adherence and Implementation and Independent Verification," does not list the NI system in Addenoum 2, " Plant Systems _ Requiring Independent Veri fication. " This oversight is considered a weakness. The ommission is being evaluated by the licensee. The licensee's evaluation will also address whether an enhancement of the verification of setpoints is appropriate...In 1 the interim, the Manager of Maintenance issued a memorandum-on January 16, 1993, to all 1&C and electrical maintenance personnel describing a potential weakness in the verification process associated with surveillances and requiring several actions to assure that critical settings, which provide-inputs for reactor trip and engineered safety features (ESF) actuations as well as as-left data required to meet surveillance test acceptance criteria, are verified as correct. These actions include: review of all surveillances

- by the performers and their supervision prior to work in the field, evaluation-of the procedure. steps to assure that the as-left setting is checked r verified by observation of a second person or by reperforming the as-lett setpoint, requiring independent verification where calculations are. performed

.

-- __-_

. .

.

.

-6-as part of a surveillance test, and issuing field changes to procedures prior to implementation to include second verification attribute .5 Reactor Trip Prevention Reviews On March 26, 1992, the Plant Manager issued a directive aimed at preventing reactor trip The actions required in his directive included power level reductions to perform several surveillance tests and the implementation of additional administrative controls. These administrative controls included performing thorough and complete prejob briefings; performing a complete and detailed talk-through of the work instructions with the unit or shift

',

supervisor; ensuring that communications were maintained and were clear and concise; ensuring strict c ap-by-step adherence to the procedure or work instruction; keeping tL cr e l room informed of the status of the job and of any problems; and provic e pu m nal, direct, and on-scene supervision of the complete job or surveillans.e :<e.t. The specific _ surveillance tests to which ihme controls applied were listed in an attachment, with Proceoure OPSP02-N1-0040 being on3 of those liste On April 10, 1992, the Plant Manager amended the instructions to provide less restrictive requirements for power levels in order to perform some surveillances. On November 6, 1992, the Plant Manager again issued a revision to the March-26, 1992, directions for' administrative controls and power restrictions for work or testing, which has the potantial to cause a reactor trip. This document eliminated 1&C surveillances from the list of those work activities .that were required to be performed under direct, on-scene, and continuous. supervisio This action was taken based on a recommendation from the Trip Provention Task force. The task force believed that existing controls for the implementation-of the surveillances and the use of dedicated maintenance teams provided adequate protection against a reactor tri The inspector determined that, on January 9, 1993, the existing controls for the implementation of the surveillance were less than adequate. The foreman simply gave the technician the surveillance work package and told him to go in the field and perform it. A prejob briefing did not occur, The inspector obtained security access log reports and determined'that no field supervision-occurred during the performance of the surveillance procedure. Although not specifically required by procedure, an adequate prejob brief and field supervision would have enhanced the quality of the work performed and may have precluded the error from occurrin The lack of these elements .is considered -

a contributing cause of the apparent violation identified in Section .6 potential Schedule Pressure -

The inspector reviewed the Daily lime Report records in-order to ascertain th z hours scheduled and worked by the technicians on January 9, 1993. .On that

.

day, the reguarly scheduled-hours for the 1&C crew in question were from 7 a.#.~to 7:30 p.m. The surveillance was begun at'4:01 p.m., adequate time before the end of the shift to. complete the job. The technician did not l

- _ _ _ _ _ - _ _ _ _ _ _ _ _ _

. ..

N' b-7- '

perform concurrent jobs or an excessive number of jobs on the day in questio ;

Schedule pressures were, therefore, not a contributor to the personnel erro !

,

1.7 Conclusions The cause of the misadjusted setpoint on January 9, 1993, was the result of several factors. The primary cause was a failure of the technician to practice self-verification as required by procedures. In addition, the procedure for performing the surveillance did not require independent verification of the setting of the trip values. This was determined to be a potential weakness with the verification process associated with surveillances in general. Other factors which contributed to the misadjustment were a less-than effective prejob briefing and no field supervision of the work in progress, r 2 UNIT 1 EMERGENCY DIESEL GENERATOR AVAILABILITY 2.1 Event Description in an effort to upgrade plant housekeeping and appearance, the licenseo started to repaint various areas and components within the plant. On December 29, 1992, contract painters started to reaaint EDG 13 (Train C) and finished the job 2 days later. The recoating of t1e exterior of EDG 13 was performed using Work Request XG-ll638 The work request, which was- originated on. february 2, 1991, included the following instructions in the work request statement: (1) precautions shall be taken to protect moving-parts that could bind during operation of the diesel, if painted, and (2) an operability test is required after coating of the equipmen On December 29, 1992, a prejob briefing was held between the contract painters and mechanical maintenance personne The maintenance personnel discussed and pointed out to the painters areas not to be painted. These areas included +

moving parts, stainless steel, and greased components. A contract foreman was present during the painting process. The EDG remained in service during the painting evolution and no postmaintenance test of the engine was performed _to

- .

verify operability. The po,tmaintenance test requirements were deleted by the system engineer on December 28, 1992. The system engineer, with operations department _ input, decided that a postmaintenance test was not necessary. in part because the machine was not declared inoperable, _ This was contrary to the corrective actions recommended in an industry experience report that was issued in 1990. The inspectors considered this to be a weaknes On January _20, 1993, a regularly scheduled surveillance start of EDG 13 was .

performed using the EDG 13 operability test and Train C EDG slave relay test  :

surveillance ~ procedures. As part of the test. process, the EDG was given an emergency start signal at 6:27 a.m. - The EDG 13 engine rolled to 100 rpm _ but failed to start. The EDG was declared inoperable and' remedial actions, such as verifying ESF power availability and EDG_11 and 12 operability, were commence .- - - .- - - - _

'

.

-8-An inspection and troubleshooting of EDG 13 were performed in accordance with Service Request DG-17022 The fuel starting solenoids were initially considered to be the source of the problem; however, further investigation revealed that 11 of 20 injection puups (seven on right bank, four on left bank) had fuel metering racks that were stuck because of paint. The paint that was applied to the fuel injection pumps had dripped and entered the back side of the fuel rack openings. This paint prevented the fuel racks from traversing through the injection pumps, preventing the injection of fuel into the cylinders. The areas were cleaned and maintenance personnel manually cycled the fuel rack linkage to ensure the racks moved freely in both directions.

,

EDG 13 was subsequently started in accordance with the system operating procedure for a postmaintenance run. While unloading EDG 13, the engine experienced load swings of about 1000 kilowatts while the engine was operating at 4200 kilowatts. The engine was secured and Service Request DG 175439 was issued to troubleshoot the problem. The cause of the problem was determined to be a faulty motor-operated potentiometer located in the electrical speed / load control circuitry. A load sensor was visually observed to have a questionable solder connection and was also replaced. After replacement of both parts, EDG 13 was satisfactorily tested in accordance with the operability test and slave relay test surveillance procedures and returned to-service on January 22, 199 EDG 13 was the first of the plant's six EDGs to be painted. The licensee plans to paint the remaining EDGs in the future. Adequate precautions to prevent EDG operability concerns will be implemented prior to initiating this wor ,

2.2 Conclusinns lhe failure of the EDG to start resulted from less than adequate review and oversight of work activities which had the potential to negatively affect operability. Specifically, the activities to be performed did not appear to be well planned or implemented and the licensee did not provide adequate oversight of the_ activities to ensure that the paint was being applied in a careful and controlled manner. This matter concerning EDG operability is considered an unresolved item (498/9303-02) pending further review by NR FOLLOWUP (92701) (Closed) Unresolved Item 499/9229-03: Work Start on incorrect _E_quipment (Unit 2)

On October 19, 1992, electrical maintenance personnel obtained work start authorization from the shift supervisor to perform Procedure 2 PSP 06-RC-0005, Revision 3, "Underfrequency Reactor Coolant Pump Trip Actuating Device Operational Test," on Reactor Coolant Pump 2C. The maintenance personnel erroneously started work on Reactor Coolant pump 2A. Further review of this incident has disclosed that the maintenance personnel did not verify the-correctness of the device prior to starting work in noncompliance with

,

w 4----- e , y- ,----p ,, - , , , , y- , , - ~ - , , - - - ,y.s g w - -

e ,v-

_ .-

f

- -

.  :

-.

-

9- j

-;

correctness of the device prior to starting work in noncompliance with  !

Procedure OPGP03-ZA-0010, Revision 15, " Plant Procedure Adherence and -j 1mplementation and Independent Verification." 1his Unresolved item is  !

considered one example of a failure to' follow procedures and is included in i apparent Violation 498;499/9303-0 ^!

3.2 (Closed) Unresolved item 498:499/9232-01: Errors _in_ Work Performance f

'

(Units I and 21 This unresolved item addressed an adverse trend involving errors resulting from work being performed on the wrong component, wrong train, and wrong uni t Six examples were identified under this one unresolved item. Further review ,

of each example during this-inspection has disclosed-that licensee personnel  !

did not verify the correctness-of the device-being worked on prior _ to starting '

work in each case. These are further examples of noncompliance with Procedure OPGP03-ZA-0010, Revision 15. " Plant Procedure Adherence and Implementation and Independent Verification." This unresolved item is-considered to be six examples of a failure to follow procedures and is .;

included in apparent Violation 498;499/9303-0 '

Er t

.

>

I u

i

- n - ,, ,,-g-, .&. -,.,:- , - ~ ,.n ,, A--- , - . -,

. ,

..

O ATTACHMENT 1 1 PERSONS CONTACTED licensee Personnel S. Bollinger, Consulting Engineering Specialist, Corrective Action Group C. Bowman, Corrective Action Group Administrator K. Christian, Operations Manager R. Dally, Engineering Specialist, licensing R. Helton, Senior Staff Specialist, Maintenance 1. Jordan, General Manager, Nuclear Engineering W. Jump, General Manager, Nuclear Licensing W. Kinsey, Vice President, Nuclear Generation M. Kanavos, Mechanical / Nuclear Engineering Manager D. Leazar, Plant Engineering Manaaer J. Ledgerwood, Consulting Engineering Specialist, Corrective Action Group M. Ludwig, Nuclear Training Manager 1. Meinicke, Planning & Assessments Manager G. Parkey, Plant Manager J. Pinzon, Senior Licensing Engineer R. Rehkugler, Quality Assurance Director The personnel listed above attended the exit meeting. In addition to the personnel listed above, the insoectors contacted other personnel during this inspection perio EXIT MEETING An exit meeting was conducted on february 1, 199 During this meeting, the inspectors reviewed the scope and findings of the report. The licensee did not identify as properietary any information provided to, or reviewed L,,, the inspectors.

L