IR 05000498/1993054

Insp Repts 50-498/93-54 & 50-499/93-54 on 931213-17. Violations Noted.Major Areas Inspected:Inspection to Determine Effectiveness of Licensee Actions to Improve Station Problem Report Process & Corrective Action Program

ML20059H057
Person / Time
Site:	South Texas
Issue date:	01/20/1994
From:	Johnson W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION IV)
To:
Shared Package
ML20059H046	List: IR 05000498/1993054 ML20059H042 ML20059H054
References
50-498-93-54, 50-499-93-54, NUDOCS 9401260219
Download: ML20059H057 (25)

Text

. . . - ,

-

i

.

APPENDIX B U.S. NUCLEAR REGULATORY COMMISSION

REGION IV

,

Inspection Report: 50-498/93-54

.

50-499/93-54 Licenses: NPF-76 NPF-80 Licensee: Houston lighting & Power Company P.O. Box 1700 Houston, Texas facility Name: South. Texas Project Electric Generating Station (STPEGS), o

'

Units 1 and 2 Inspection At: Matagorda County, Texas Inspection Conducted: December 13 through 17, 1993

'

Inspector: T. O. McKernon, Reactor Inspector, Project Section A, Division-of Reactor Projects Accompanying Personnel: D. Desaulniers, Human Factors Engineer, Human Factors Assessment Branch, Office of Nuclear :

Reactor Regulation ,

D. Schultz, Consultant Approve : / / ko /9V W. D. Johnson, CEief, Project Section A Date

' '

Division of Reactor Projects

.

Inspection Summary Areas Inspected (Units 1 and 2): Routine, announced inspection to determine the effectiveness of the licensee's actions to improve the station problem >

report (SPR) process and the corrective action program. In addition,

' corrective actions related to postmaintenance testing (PMT) were reviewe Results (Units 1 and 2):

• Two examples of failure to follow the work process program were identified. The cited examples involved failure to conduct a prejob briefing when required and failing to verify the correct station component identification tag prior to working on motor-operated valve actuators. Generic corrective action may be necessary to prevent future work on wrong components (Sections 2.1 and 2.2). i 9401260219 940120 PEN 1 AIX}CK 05KNDO498 .

'

G PER

,

. - - . .

.

.

-.

-2-

,

.- The licensee has made significant improvements in the area of SPR ,

investigction and the corrective action program (Sections 2 & 3).

]

• The licensee has implemented corrective actions related to PMT .

weaknesses identified in'NRC Inspection Report 50-498/93-46; 50-499/93-46 which appear to programmatically resolve inconsistencies in :

the implementation of the PMT program (Section 6).

Summary of Inspection Findings: ,

• Violation 499/9354-01 was opened (Sections 2.1 and 2.2).

• Inspection Followup Items (IFIs) 498;499/9331-18, -23, -26, -27 remain '

open pending any adverse input from the operational readiness assessment team inspection (Section 3).

'

• Violations 498/9235-02; 499/9235-02 and 498/9224-01; 499/9224-01 were closed (Section 3.3 & 3.4).

'

• IFIs 498;499/9331-06, -28, and -67 were closed (Section 4).

l Attachments:

• Attachment 1 - Persons Contacted and Exit Meeting l

• Attachment 2 - Events and Casual Factors Diagrams

F I

?

!

,

>

>

,

r

!

,

.

.

t-3- ,

,

'

DETAILS

.

I BACKGROUND Both units at STPEGS were shut down in early February 1993, and remain shutdown as a result of numerous broad scoped problems identified by the NRC and the license NRC Inspection Report 50-498/93-31; 50-499/93-31, issued on October 15, 1993, identified 16 Restart Issues that required resolution prior to the restart of Unit 1. In addition to these Restart Issues, a number of items related to these Restart Issues were identified. The purpose of this inspection was to !

determine the licensee's effectiveness in resolving Restart Issue No. 2,

" Station Problem Reporting Process and Corrective Action Program," and to establish a basis for concluding that this Restart Issue has been adequately resolved by the licensee. In addition to reviewing specific items related to .

'

the Restart Issue, the inspector assisted by a human factors engineer and a consultant, reviewed the licensee's root cause analyses and human factors evaluation related to two recent events (wrong component worked and swapped temperature switches) for adequacy and completenes .

2 ONSITE LICENSEE REVIEW ACTIVITIES ASSOCIATED WITH EVENTS (92720)

During the period December 13-17, 1993, a human factors engineer from the office of Nuclear Reactor Regulation Human Factors Branch and a consulting engineer evaluated several recent adverse human performance events at the STI EG The inspectors focused on two events related to performance of maintenance on the wrong train and wrong component category. The inspectors performed an independent evaluation of the events through record reviews and interviews of the maintenance technicians and their supervisors involved in the event. The inspectors also evaluated the licensee's investigation of the event, the licensee's root cause analysis, and the licensee's corrective action l

.l Event No. 1 - Improper Temperature Switch Installation On December 6, two instrument and control (I&C) contract technicians (Technicians 1 and 2) removed the two switches from their wall mountings, and transferred them to the I&C shop for calibration. Although Calibration Procedure OPMP08-ZI-00ll, " Generic Temperature Switch Calibration (Filled Element)," permitted removal of the switches, the procedure allowed the switches to be calibrated in place and included a note indicating that calibration in place was the preferred method. Technician 1 elected to remove ,

the switches to the shop for calibration based on a previous experience of l having difficulty obtaining the proper measuring and test equipment in the ;

fiel !

Calibration of one switch was performed on December 6, by Technician On l December 7, Technician 1, was assigned to a different activity. A new l

t

. .- . .

.

.

.

-4-  ;

r contractor technician, Technician 3, assisted Technician 2 in calibrating the second switch. Neither the original two technicians, nor the replacement ;

technician were provided a formal pre-job brief by their supervisor as required by Procedure OPGP03-ZA-0090, " Work Process Program," Revision 7, ,

dated August 24, 1993, paragraph 3.7.2.5, l

"The Work Supervisor shall conduct a pre-job brief prior to work start (SPR 910422)."

As a consequence, the pre-job brief did not address: (1) the issue of calibrating the switches in place or in the I&C shop, (2) the different quality classes of the two switches and the reasons for keeping them segregated, and (3) precautions related to working the switch in the diesel generator room without electrical isolatio Upon completion of calibration, Technician 3 unsuccessfully attempted to obtain a torque measuring screwdriver from a tool issue point. As a consequence, the technicians used a socket driven, wide blade (5/8" vs. 1/4" required), torquing device. The technicians practiced torquing the switch (wrong switch) in the shop to demonstrate their ability not to disturb the calibration. It appears that a consequence of the practicing was that the technicians related the wrong switch to the wrong field installation positio l The reconstituted, two-person crew then proceeded to Diesel Generator Room 2 with both work packages and both switches and installed the wrong switch. The technicians installed the Quality Class 4 switch previously removed from the Mechanical Auxiliary Building (MAB), Room M226. Prior to the installation, l the technicians made a comparison check of the Total Plant Numbering l System (TPNS) tag at the conduit location with the work package TPNS numbe However, the technicians failed to compare the TPNS tag on the switch with the Work Package TPNS number. As a consequence, the barrier to properly identify the component in accordance with the requirements of OPGP03-ZA-0090, paragraph 3.7.2.16, was also breache " Work group personnel shall verify that station component identification matches the component specified as requiring maintenance in the work document."

The technicians then proceeded to MAB Room M226 and installed the Quality Class 7P temperature switch previously removed from the diesel generator room'

location. Technician 2 identified the location for installing the switch based on his memory of from where he had removed it the previous day. Again, a comparison check of the TPNS tag on the switch with the work package TPNS number and the TPNS tag at the conduit location was not made prior tc the installation. However, a comparison check of the TPNS tag n the switch with the TPNS tag at the conduit location was not possible due to absence of a TPNS tag. Supervision was not notified of this problem as should have been done in accordance with the requirements of OPGP03-ZA-0090, " Work Process Program,"

Revision 7, dated August 24, 1993, paragraph 3.7.2., that stated, i

j

._ . , m . . . _ . ._

..

s

.

-5- l

"If the configuration of the equipment does not match the ,

configuration of the work documents, the craft shall notify the Work Supervisor (SPR 910484; DR 91-027; SER 84-069)." .

Thus, another procedural barrier was breached that could have prevented the event from proceeding furthe j

The inspectors noted that Technician 2 had been trained in the Houston '

Lighting and Power (HL&P) Work Process Program in March and April and was certified as I&C technician, Level II on November 9,1993, and as Basic -

Journeyman - I&C technician on November 10, 1993. Technician 3 was trained (24 hrs) in " Work Control for I&C" in September 1990; was certified to I&C '

.

Level 11 technician status on March 17, 1992; and, certified as " Basic Journeyman - I&C Technician" on August 31, 1993. Since the work supervisor made no assignment of the lead technician, Technician 2 became the self-appointed lead by virtue of having worked the job the day befor [

!

Upon completion of the switch installation in the diesel generator room, Technician 3 summoned the Quality Control (QC) inspector for completion of the required inspections. Upon performing the required inspections, the QC inspector noted that the switch was improperly located when performing the

TPNS comparison checks, issued a stop work order that was not properly complied with (see below), and notified the control roo Several other deficient conditions were noted during the review of the two +

preventive maintenance (PM) work packages that were not directly related to ..

'

the human performance evaluation issues. The diesel generator room switch '

package stated in Section 2.0, " Precautions," that "None" were required. The inspectors learned from interviews with the technicians that the PM activity was performed with the switches energized, and required lifting and landing-energized leads. This practice was contrary to the requirements of .

Procedure OPGP03-ZI-0021, " Electrical Safety," Revision 1, dated September 24, l

1992, which stated:

"4.3, All electrical work will be performed when equipment is de-energized when possibl t

'

"4.4, Working on energized circuits or equipment may be done only when '

absolutely necessary."

The diesel generator room switch was worked energized since it would have .

required placing the diesel engine out of service. However, no l

'

rationalization was included in the work package reflecting the necessity for working the equipment energized. The inspectors concluded from this condition alone that a detailed.prejob briefing was necessar .

The failure to follow procedures, in that, no prejob briefing was conducted for both work activities was the first example of a violation of Technical '

Specification 6.8.1.a (499/9354-01).

,

m Y

-

,

. . . .. _

-.

'

-6-  ;

In addition, the " SUMMARY OF WORK PERFORMED" blocks in both work packages did not reflect that the switches were exchanged to the proper position subsequent to the event. Although the corrective maintenance action of restoring the switches to their proper position was clearly not a preventive maintenance ;

'

action, based on the information provided the inspectors, neither a service request nor a "one-time" amendment to the PM was issued for both packages to perform the exchange. Instead, at least one restoration exchange was execute under the existing PM activity in spite of the QC stop work order. The entries in the PM packages did not permit tracking of the equipment .

i configuration during the entire activit l

'

Although the 6esel generator room switch was seismically mounted with bolt I sizes having torquing requirements (1/4") according to the work package, no documentation was available in the work package to verify that the switch had been properly installed for its final installation. The included data sheet, !

step 7.2.16, stated, " Instrument mounting bolts torqued (See NOTE 1 & 2)."

Notes 1 & 2 required signatures if the switch was removed and verification of 3 torquing of mounting bolts when a specific torque value was given. For r

'

reasons not recalled by the technicians, they were initially confused relative to torquing requirements, and therefore conferred with QC (but not Plant ,

Engineering because of previous experience in obtaining answers taking long time periods) on what was required to obtain a QC sign-off. Consensus was ,

achieved that torquing was appropriate for the diesel room switch. Subsequent l investigation by the licensee, prompted by the inspectors, identified that the ;

calibration procedure was inconsistent with Specification No. 5Z689ZS1027, !

General Requirements, Instrumentation Construction, which stated' at l paragraph 6.6.10, "All instrument fasteners, seismic and nonseismic, shall be !'

wrench tight." The licensee agreed to take corrective ution relative to the calibration procedur . Adequacy of Licensee Investigation, Corrective Actions and Conclusions ;

i The Station Pruoiem Report had not been closed on this event since the SPR had I a due date of January 5, 1994. The investigation, however, was considered -

reasonably complete at the time of this inspection in that interviews had been compided, statements had been taken, and at least partial corrective action of term,nating I&C Techre .'.ans 2 and 3 had been taken, j Based on the records obtained by the licensee to date including the statements of Technicians 2 and 3, the QC inspector, and interviewer notes, the  ;

inspectors concluded that some potential causal factors were either not l

'

-identified or not documented. For example;

• The work supervisor directed the switches to be removed due to absence i of necessary measuring and test equipmen * The absence of proper torquing devices (tool problem) may have become a j distraction to the technicians later when preparing to install the i

-

g y , - - - -

,

, - . _ _ . - . _-

'

..

.

-7- ,

l

,

switches. These adverse conditions of tool availability were not :

documented in the SPR records to dat ;

• There was no evidence that a thorough pre-job briefing had not been !

conducted for any of the technicians. There was no evidence that pre . :

job briefings were not initially conducted, nor later with the personnel !

chang * There was no evidence that the missing TPNS at the MAB conduit location :

was noted by the investigato :

The licensee investigator did note, " working 2 pkgs concurrently", " swapping personnel around from job to job", "no tern-over" and " Cal switches in _

place - cal gear designed to be used in field" as causal factors in the even l The inspectors' evaluation of this event was early in the SPR process, thus the adequacy of all corrective actions could not be determined. However, '

based upon the information available to the inspectors, they concluded that several necessary details were absent upon which a thorough corrective action :

'

program could be based. It was not clear whether all investigative action had been empleted since licensee management had not reviewed the SPR in final form, at based on immediate. corrective action taken by the licensee, most of >

l the ir stigation had been complete .2 b ..,it No. 2 - Maintenance Performed on Improper Valve Actuators l Contract Order No. ST-400619, dated March 24, 1993, retained ITI M0 VATS In l as the contract vendor to perform diagnostic motor-operated valve (MOV) _

testing, and subsequently by contract amendments actuator and valve l refurbishment, during STPEGS unit outages. The contract was managed by the STPEGS Design Engineering Department (DED). The scope of work provided, in part, that, "All work shall be performed by using HL&P procedures . . . or by using contractor prepared procedures which will be reviewed and approved by HL&P." The contract also provided that the contractor would provide current training certification records, and their related training programs for their personnel. The records were to demonstrate that personnel training was equal to or more comprehensive than the corresponding HL&P training program, and that HL&P would identify and provide any training that must be provided after record review. Some HL&P training was provided to the initial cadre of MOV vendor personnel in the administrative areas of maintenance performance because the personnel did not possess the equivalent skills from factory trainin In addition to these contract specifics, Procedure OPGP03-ZP-0013, Revision 0, dated December 1, 1992, " Purchase Order / Contract Management, Monitoring, Reporting and Rating," required a Contract Technical Coordinator (CTC) be l

'

assigned by the originating department (Design Engineering Department), and be charged with the responsibility for contract management, monitoring J

_ ,

.- -- . . _

-

.

.

-.

r-8- 0 i

performance, et In accordance with Procedure OPG03-ZP-0013, the CTC generally had: ,

"3. . . . the duty to manage and monitor the day-to-day coordination of Vendor work activities . . ."

and specifically was supposed to perform Performance Monitoring to assure:

"4.1.2.3 Vendor personnel receive required training . . . and are knowledgeable of site procedural requirements such as security, safety practices, housekeeping, system clearance, and rigging."

The CTC was provided specific management guidelines that required, prior to the vendor starting work, that,

" Addendum 1, 2.3 Qualification of personnel to STPEGS and/or vendor procedures. Schedule training and/or review training '

.

records and certifications."

Additionally, Procedure OPMP01-ZA-0035, Revision 0, dated June 2,1987, i

" Qualification and Certification of Maintenance Personnel," required contract 't personnel performing maintenance activities affecting the quality of safety-related structures, systems, and components to be qualified and certifie This procedure was amplified with the issuance of Maintenance Department ;

Standing Order MG - 22, Revision 0, " Contractor Qualification Requirements,"

that provided interim guidance for determining, evaluating, and documenting !

Maintenance contractor qualification requirements. However, the requirements [

were made applicable only to maintenance contractors, and did not affect any ,

maintenance contracts then in effect. Because of the caveats, the M0V vendor contractors were unaffected. An extensive record search by the licensee indicated that the two MOV vendor technicians and one MOV vendor QC inspector ,

had not been trained nor certified in HL&P work control processe P These procedures provided guidelines that should have acted as multiple barriers against the assignment of noncertified personnel performing field ,

work and were considered causal factors in the even t G

Monitoring of the qualification status of MOV vendor replacement personnel was ;

faulty in the beginning, and began to deteriorate further in June 1993 becau :

the comparison of contract personnel certification status to HL&P requirements ;

was not implemented in DED, and DED did not comply with Maintenance Department j procedures. 'DED rotated staff that had been doing the informal certification evaluation without recognition that the function required continuation. Thus several replacement MOV vendor personnel who were well qualified technically, !

began performing maintenance functions on MOVs, but _they were not qualified ,

administratively in the HL&P work control process. The inspectors considered this to be a causal factor for the event occtrrenc On or about November 21, 1993, work orders were generated and approved to inspect " c limit switch compartment components of the Unit 2, Trains A and B

>

P

, v --n -- n , . -

,

.. ,

s

.

-9- ,

containment spray-(CS) pump discharge isolation valve actuators (A2CSMOV0001A '

and B2CSMOV0001B). The work orders and associated work travelers were properly prepared, and contained adequate detail, including the component ,

~

location On November 22, MOV vendor day shift personnel were assigned the work packages for the two valves. The day shift completed the inspection, corrected minor deficiencies, but was unable to complete the final close-out inspection. The close-out inspection was left for the night shift to complete. The inspectors -

learned through interviews with day shift personnel that a near miss of the same event occurred related to not identifying the correct components, but was caught by a maintenance member. The near-miss was not communicated to the on-coming shift verbally or in writin The evening shift work supervisor ieceived a face-to-face turnover from the day shift work supervisor but the evening shift personnel did not overlap with .

'

day shift personnel. Thus turnover was by log entries reflecting corrective actions on compartment deficiencies taken and the remaining close-out ,

inspection required. The evening shift work supervisor accepted the equipment

-

clearances on containment spray valves by acceptance of the work package However, the work supervisor did not assure that the crew adequately ,

understood the task location as evidenced by the crew performing work outside the isolation boundaries. The work supervisor assigned two contract M0V vendor technicians the two work packages, but did not perform a formal pre-job briefing. Although station Procedure OPGP03-ZA-0090, Revision 7, dated August 24, 1993, " Work Process Program," required, l

"3.7. The Work Supervisor shall ensure personnel have the necessary skills, experience and training certification specific i to the work task (OMR 85-21).",

f the Work Supervisor had not been previously required to perform such an ,

evaluation, did not have the tools (training /certificv ion matrix) to perform the certification evaluation at the time of job assignment, and therefore did ;

not perform the task. No pre-job brief required by OPGP03-ZA-0090, paragraph i 3.7.2.5, was performed for the cre Procedure OPGP03-ZO-0039, Revision 4, dated February 27, 1993, " Operations Configuration Management," required, ,

"9.1.19 Signing an EC0 as " Acceptor" ensures the SCOPE of the ECO J adequately fulfills the clearance requirements of the Acceptor's work within a Work Package AND establishes clear boundaries." i lhe clearance procedure further required that, i

"9.1.2 There should be only one Acceptor per craft per work j package. Shift turnover SHALL include a turnover of acceptance of i

'

any ECOs accepted during the shift. When the work package is signed by the relieving work supervisor, THEN the acceptance o l J

.

!

l

.

-10-i

,

.

l the clearance is considered turned over to the oncoming Work Supervisor. This does not prevent an EC0 walkdown on turnover, but does limit the number of Acceptors per craft on a work package to one."

There was no indication that the Work Supervisor understood his responsibilities as a Clearance Acceptor and formally assured _that adequate protection was available for the work crew. Clearance adequacy was treated as !

a crew responsibilit For other reasons, 2SI-MOV-0001B had been danger-tagged, but 2SI-MOV-0001A was not. This_went undetected by the crew since they observed the danger tags hanging on power supplies to the CS valve s These procedural requirements could have acted as barriers to the event had they been performed, and their nonperformance was perceived by the inspectors as event causal factor The two evening shift MOV vendor technicians notified the necessary personnel i to perform the close-out inspection, namely a MOV vendor QC inspector, an HL&P QC inspector, and a design (contractor) engineer, and obtained work start authority from the control room. All five personnel met at the Radiological Control Area (RCA) boundary sign-in and control desk, and signed in on a clean area Radiological Work Permit (RWP) that permitted work in street clothes. An MOV vendor technician (Technician 1) assumed the role at lead technicia Technician 2 had recently started work at STPEGS and Technician 1 believed he knew exactly where the specified valve actuators were bued on past ,

experience. During interview, Technician 1 indicated tt:at the 1 0001A(B) designators reminded him of previous work he had conducted and the specific room location Upon arriving at the Unit 2, Trains A and B Safety Injection (SI) Valves 0001A and 0001B, that Technician 1 assumed to be CS valves, they discovered that the valves were within an additional local area RCA denoted- by magenta and yellow rope. Technician I called a health physics technician and the health physics ,

supervisor and they collectively agreed that_ tt a work could be performed ;

within the boundary without oressing out in protective clothing or modifying i the boundary or modifying the RWP if they did not step on the floor under the 1 (SI) valves. Although working within an RCA boundary contrary to their RWP ,

'

and contrary to the posted requirements was inconsistent with plant procedures, and discovery of the condition was inconsistent with the crew's .

expectation, the crew did not notify their respective supervision. This lack of action was inconsistent with Procedure OPGP03-ZA-0090 as follows

"3.7.2.2 If the configuration of the equipment does not match the '

configuration of the work documents, the craft shall notify the Work Supervisor (SPR 910484; DR 91-027; SER 84-069)." l These procedural requirements could have acted as barriers to the event had ,

they been performed, and their nonperformance was perceived by the inspectors ;

as event causal factor .

.

_4,

-11-MOV vendor Technician 1 verified the last five characters on the valve TPNS tag as 0001A, not notiy the first three characters of "2SI." Three of the other four personnel observed the same numbers and concluded they were at the correct valve, ignoring consideration of what system they were looking a The work package did not have signature verifications required of the relieving crew (such verification had been required and signed off by the day crew). Furthermore, although Procedure OPGP03-ZA-0090 stated,

"3.7.2.16 Work group personnel shall verify that station component identification matches the component specified as requiring maintenance in the work document."

all five personnel failed to perform the task, notwithstanding that the correct TPNS number of 2CS0001A(B) was on the work package cover, a number of the work package pages contained the number, and the valve and its actuator were properly labeled with a TPHS tag. The failure of the technicians to verify the complete component number on the identification tag was a second example of a violation of Technical Specification 6.8.1.a (449/9354-01). In addition to this specific violation, a number of barriers that could have precluded the errors were identified as having been misse These items were of regulatory significance but will not be cited individually. However, the licensee's response to this violation should include a discussion of the causes and corrective actions for the failure of each barrier. The work packages properly reflected the.CS valve room location, different from the SI valves. The station also has a vis.ibly emphasized program of self-checking and self-verification that is both written and verbally delivered to all station personnel. Although the TPNS tag was difficult to read, these procedural and policy requirements could have acted as barriers' to the event had they been performed, and their nonperformance was perceived by the inspectors as event causal factor The inspectors identified several factors which contributed to the failure of Technician 1 to properly verify that he was inspecting the correct componen As previously noted, Technician 1. had a developed a strong expectation concerning the location of the valve because of past work on similarly numbered valves. Other factors reinforced or failed to correct this incorrect expectatio * The valve identification tags were either deeply stamped or lightly '

etched and the work area was not well illuminated. These conditions made the tags difficult to rea * The actual room where the CS valves were located is generally referred to by plant staff as the SI valve room. As a consequence the actual location of the CS valve was not normally associated with valves other than those related to the SI syste * The work package identified the room locations for the specified Trains A and B CS valves as F007 and F008, respectively. The room

_ _ . . _ __

-

!

i

'

!

-12-  :

containing the SI valve that was assumed to be the Train A CS valve was ;

labeled " TRAIN A PUMP ROOM 005." Following the 005 was "/007" written in magic marker. It was not clear from.this marking that the "/007" referred to the mezzanine, where the specified CS valves were actually -

located, though it did suggest past difficulty in identifying the roo The same magic marker labeling was used for the room containing.th ,

i Train B CS valve. The inspectors also noted that the CS valve actuators that were supposed to be inspected had magic marker labels on the actuator compartments, indicating a past need for aids, in addition to the TPNS tags, in their identificatio ,

The inspectors also identified factors which contributed to the failure of other members of the work group to adequately verify the identity of the component Interviews with Technician 2, the QC' personnel, and DED engineer involved in the inspection. activity indicated that their verification efforts i were inadequate because they lost their independence due to arriving at the site as a group and observing the verification activities of the individuals who preceded the Some individuals made comments in interviews which suggested that they had become focused on only their specific tasks and :

abdicated the responsibility of verifying the component identif.ication to :

other individuals in the grou >

Technician 1 removed the limit switch cover from 2SI0001A and proceeded with l the close-out inspection in accordance with the work package, noting that the {

same discrepancies described by_the day shift as corrected in the turn-over ,

log were still eviden The technician made no electrical safety checks to :

assure absence of electrical potential between terminals, and between !

terminals to oround. As above, upon discovery of discrepant conditions that ,

were inconsistant with the crew's expectation, the crew did not notify their -

respective supervision. Furthermore, the crew should have assured the equipment was not energized in accordance with station i Procedure OPGP03-ZI-0021, Revision 1, dated September 24, 1992, as follows:

_

,

" Unless they are proven to be de-energized, all circuits and ;

electric apparatus shall be considered energize !

i

"4.3 All electrical work will be performed when equipment is de-

-

energized when possible."

Similar requirements were noted in the contractor's' industrial safety ,

handbook. These procedural requirements could have acted as barriers to the ,

event had they been performed, and their nonperformance was perceived by the inspectors as event causal factor A reactor plant operator (RPO) accompanied the work crew on the elevator in i the fuel handling building to the scene at about 9 p.m., on November 22, and i inquired of the inspectors what they were going to do, and whether they had a [

clearance on the valves. He then made some equipment checks in the local :

area, and returned to review the work package. He observed that a clearance ,

had been set for the work, but did not note that the clearance was applicable l

l

-

,_ .

- - -.

,_ -

J

_ ~ - - -. -. .. -- . . - .-. - .. - ..

..

.

.

!

-13-

"

to containment . spray valves instead of the safety injection valves being worked. He then went on the rest of his tour, and returned to the control room at approximately 11:30 p.m. He observed that the control room board ,

safety injection valves were not tagged, and in -fact had energized position l lights. Upon inquiring of control room supervision whether work authority had i been granted on SI valves, he learned that it had not, and was directed to  :

return to the scene and return with the workers. On arrival at the scene, the l RPO found no workers (the MOV vendor inspectors had completed the effort.on the Si valves and departed). He returned to the control room and reported his findings, and was dispatched for his subsequent watch station round ;

After completing his watch station tour, the RP0 was ordered into the control i

'

room at about 4 a.m., for purposes of lifting the CS valve clearance, .and noted immediately that the workers had probably performed maintenance on the ;

wrong pair of valves. He immediately notified shift supervision who caused ,r'

the workers to come to the control room and report what had happened. Upon learning the wrong actuators had been worked, shift supervision initiated SPR 933299. The RP0 received commendatory entries to his personnel file for i his part in identifying the proble !

Further discussion of this event is included in NRC Inspection  :

Report 50-498/93-45; 50-499/93-45. In that inspection report the inspectors i

'

expressed concern that the lack of self-verification, as viewed by this and several other recent events, was not restricted to the specific contracto The licensee should evaluate recent wrong component events and discuss in

their response to the violation the need for more generic corrective actions to address problems with self-verification as conducted by South Texas Project ;

,

employees and contract personne . Adequacy of Licensee investigation, Corrective Actions and Conclusions !

'

At the time of this inspection, the SPR had not been closed on this event since the SPR had-a due date of December 23, 1993. .The investigation was, :i however, reasonably complete and draft, short term corrective actions were l'

provided the NRC inspectors before departing the site. Interviews with the involved individuals had been completed, statements taken, and at least partial corrective action of terminating the two MOV vendor technicians and-the MOV vendor QC inspector were complet Based upon records obtained-by the licensee to date, the inspectors concluded that some potential causal factors were either not identified or not documented. However, there was indication that most causal factors were identified based upon corrective actions formulate For example:

• There appeared to be no consideration of the workers' certifications and qualifications relative to the work control process. Several days were required for the licensee to finally conclude that the technicians had not been trained and were not certified for the wor i r

-- ,

,

. .

.

-14-

• The work supervisor was not provided with_ records, nor required to assign only properly qualified and certified personnel to field task The Work Supervisor did not perform a formal pre-job briefing that included discussion of clearance boundaries for the work to be performed to assure worker safet e Work group personnel did not perform electrical safety checks to assure adequacy of the equipment clearanc * Day to day management monitoring of vendor services was not performe The licensee determined the root cause of the event to be " INATTENTION TO DETAIL" for the following areas:

• Workers did not recognize system identification as importan * Self-verification and self-checking were not employe * Workers had an incorrect mental picture of the task locatio * Barriers were negotiated without raising warning signal The inspectors prepared an Event and Causal Factor tree (Attachment. 2) to determine if all causal factors had been identified, if the root cause(s) were identified, and to determine if corrective actions contemplated by the licensee were adequate. The tree clearly indicated a preponderance of inadequate training as a causal factor. Lack of management and supervision, and organizational factors were significant contributors. The inspectors noted also a significant absence of procedure related causal factors, (i.e.,

if all persons concerned had complied with the HL&P procedures) the event would not have occurred. This emphasis on causal factors was absent from the SPR documentation provided the inspectors, however, it was obvious that the corrective actions were properly focused. Increasing field supervision, management oversight on procedural adherence, etc. were found to be proper corrective action '

2.3 Events Review Conclusions The inspectors noted a significant improvement in the root cause analysis process since the spring of 1993. Many more pertinent details were uncovered, during the investigative process, more causal factors identified (although perhaps not documented), and appropriate corrective actions formulated. In the past, the NRC has noted that many corrective actions, although formulated, were not implemented. In contrast, the licensee now appeared committed to implementing required corrective actions.

t a~.,-.- ,n - n . . - - - - - - _ . .-.____k - - - - - _ - - - - - - - -

.. . . _ - - . .- - ._ -

,

.

l

. 1-15-l 3 LICENSEE ACTIVITIES ASSOCIATED WITH STATION PROBLEM REPORTING AND THE ,

CORRECTIVE ACTION PROGRAM (92720) l Ineffective Corrective Action Processes as Major Obstacles to Irprovement During this inspection, the inspectors reviewed various changes to the 3 corrective action program from streamlining the problem identification process, improving information systems, involving management at a working level in'c the corrective action program, to the allocation of long-term budgetary resources to resolve plant problem As part of the licensee's corrective action improvement process, the responsibility for station problem report investigation had been assigned.to the applicable departments (e.g., Central Maintenance, Operations, and  ;

others). This action was taken in order to have the line management ,

organizations take ownership of the corrective action process. As such, each major plant department had SPR investigators trained in root cause analyses and human performance evaluation The licensee has improved their RCAs through additional RCA training and 'a :

planned increase in the number of investigators. The Business Plan Items E and E3.1 further defined initiatives to foster a culture that promotes continual self-assessment and problem correction by the line organizations and .

training for station personnel on the importance of identifying and effectively correcting problem ;

A sampling review of 15 SPRs from November 12, 1993, to the present indicated ;

that generally RCAs were performed acceptably and to a depth consistent with ;

identifying the potential root causes of the problem. However, some ,

'

weaknesses still existed with the licensee's ability to perform human performance evaluations and identify all potential root causes to problems (see discussions for Events 1 and 2 above). l 3.2 -Management's Support to Effectively Correct Program and Component Problems i

The licensee has taken corrective actions to address the specific examples 3 given in the diagnostic evaluation team report. A labeling program '

incorporating industry guidelines has been initiated and the first package ,

installed on the diesel generator systems. The entire labeling program is scheduled for completion by the end of 199 The locked valve program has been revised. The-lice ree's design engineering i department implemented design change documents whio 9 moved approximately 772 valves per unit from the program and revised the administrative *

Procedure OPOP01-ZA-001, " Plant Operations Administrative Guidelines," -

Revision 3, to delete the valves from the progra The licensee has completed Modification 92043 which installed 39 orifices in ,

the main steam system drain lines. Installation of orifices in Unit 2 will be

- - . - - . - - -- . . .- . .-

~ . - - . . . . . - .. - - - - -

a

l l

r

.

-16-

!

accomplished prior to Unit 2 restar The modification wa; implemented to assist in precluding post-trip overcooling through secondary plant heat losse :

Further, during this inspection, the inspectors reviewed the corrective action ,

program under Revision 3 to Procedure OPGP03-ZX-0002. The licensee had '

established a problem review group (PRG) as a review check by plant managers and senior management members from operations, engineering, quality assurance, .

licensing, corrective action group and others. The PRG is responsible for the t review of each SPR and assignment of category level and responsibility to the applicable department (s). The corrective action group's multidiscipline group is responsible for reviewing SPRs for generic implications, repetitiveness of the condition and mitigating corrective actions taken or initiated. The multidiscipline group's recommendations would then be reviewed by the PR The SPR screening process ensures management awareness of station problems.and ;

provides direct management support and input into correcting program and  ;

component problems.

-

In addition to the above, plant walkdowns have indicated greater management presence in overseeing and observing key maintenance, operations, and other ,

activities being performed in the plant. This heightened sensitivity to  :

problem awareness by upper management and the increased willingness by plant personnel to identify problems has greatly improved management's support in correcting program and component problem .3 Corrective Actions Associated with Essential Chilled Water Flow Switches l During the inspection, the inspectors reviewed the licensee's corrective '

actions in response to Violation 498/9224-01; 499/9224-01, " Failure to take adequate corrective action to preclude essential chilled water switch  ;

malfunctions during valving-in processes following maintenance."

'

,

' '

The licensee corrected programmatic weaknesses by. implementing changes to Preventive Maintenance Procedure OPGP03-ZM-0002 adding precautions to preclude overpressurizing the switch bellows during postmaintenance valve-in.and revised the station corrective action Procedure OPGP03-ZX-0002 to require the responsible department obtain concurrence from the initiator prior to .

rejecting a maintenance feedback request. In addition, the. licensee has  ;

implemented a modification that provides an equalization valve. and piping to ,

the flow switch configuration. At the time of this inspection, the modification had been completed on Unit I and was awaiting installation on Unit 2 during the next equipment' train outage. The corrective actions were adequate to address the concerns of the violatio .4 Corrective Actions Associated with the Failure to Initiate'an SPR During the inspection, the inspectors reviewed the licensee's corrective actions in response to Violation 498/9235-02; 499/9235-02, " Failure to promptly initiate an SPR."

,

t

., ,s.- ,- r- '

-w

. _ . . . - . _ .

.

[

,

9 '

-17- i j

!

I This violation involved 4 examples of a failure to promptly. initiate an SPR,

(1) to document foreign material found on the Emergency Diesel Generator 23 fuel oil strainers, (2) to promptly correct equipment deficiencies with Essential Chiller 21A, (3) failure to take corrective actions for repeated overspeed trips on the Auxiliary Feedwater Turbine 24, and (4) a repeat J violation involving individuals working excessive overtim During this inspection, the inspectors verified that the licensee had taken _

corrective actions for the above cited examples. The appropriate plant l procedures regarding conformance with overtime policy were issued, j OPGP03-ZA-Oll6, Revision 0, " Overtime,"-and records of personnel training were :l verified. Other actions to incorporate examples of nonconforming service i request forms and specific instructions for dealing with nonconforming service requests incorporated into the work process Procedure OPGP03-ZA-0090 were also .]

verified complete. Corrective actions to finalize modifications to the ,

essential chillers had been completed at the time of-this inspection and the ;

essential chillers were scheduled for testing during the week of January 3, i 1994. Additional discussions related to the essential chillers are included i in NRC Inspection Report 50-498/94-04; 50-499/94-04. For the Auxiliary Feedwater Pump 24 overspeed trip mechanism, the licensee cleaned the tri reset plunger and changed from lube oil from Mobil Vaprotec Light oil to Mobil i DTE 79 The testing of the auxiliary feedwater pumps will be conducted in j Mode Additional discussions on this issue are provided in NRC' Inspection i Report 50-498/93-38; 50-499/93-3 ]

In addition to the above, the licensee's training and emphasis on plant personnel self-identifying problems has been reflected in a culture change currently taking place. Through discussions with plant personnel and observation of management meetings to review and disposition SPRs there :

appeared to be an increased scrutiny of plant problems. Pending successful l testing of the auxiliary feedwater pump in Mode 3, the licensee's corrective l actions were adequate in addressing not only the specific hardware issues but l the broader generic concerns of the violatio !

4 CLOSED ITEMS RELATED TO RESTART ISSUES (92701) (Closed) Violation 498:499/9224-01: Failure to take adeauate corrective ;

action to preclude essential chilled water switch malfunctions durinq !

'

valve-in processes following maintenance

!

This item was closed based on the licensee's corrective action described in paragraph 3.4 of this repor ]

,

4.2 (Closed) Violation 498:499/9235-02: Failure to Initiate an SPR l This item was closed based on the licensee's corrective action described in -

paragraph 3.3 of this repor l

2 f e g - -

4

.-

-18- i (Closed) Inspection Followup Item 498:499/9331-06: Ineffective management support to correct program & component problems This item was closed based on the licensee's corrective action described in i paragraph 3.2 of this repor .4 (Closed) Inspection followup Item 498:499/9331-28: Corrective Actions and Root Cause Analysis for the Diesel Generator Fuel Pump Holddown Studs This item was closed based on the licensee's corrective actions described in NRC Inspection Report 50-498/93-44; 50-499/93-4 .5 (Closed) Inspection Followup Item 498:499/9331-67: Line Management q

<

Ownership of the Corrective Action Process

!

This item was closed based on the licensee's corrective actions described in paragraph 3.3 of this repor ,

5 OPEN ITEMS RELATED TO RESTART ISSUES (92701) ,

The following items related to Restart Issue No. 2 were statused concerning i the manner in which the licensee had resolved the issue within the-scope o :

improving the SPR process and the corrective action program. Several of these

'

issues were broad scoped; however, it is acknowledged the.t significant improvements have been made in this area. These items will remain open pending further NRC inspection effort (i.e., operational readiness assessment ;

'

team inspection results). (0 pen) IFis 498:499/9331-18. -23. -26. -27: Effectiveness of the Corrective Action Process

The inspectors concluded that the licensee has made significant improvements to the SPR process and the corrective action program which pending'any adverse input from the operational readiness assessment team inspection would be adequate to recommend closure of these items. These items are statused in paragraphs 3.1-3.4 of this repor l 6 CORRECTIVE ACTIONS ASSOCIATED WITH THE POSTMAINTENANCE TESTING PROGRAM (92720)  !

,

During a prior NRC inspection, 50-498/93-46; 50-499/93-46, certata

.

implementation weaknesses in the post-maintenance testing program were identified. These weaknesses included inconsistencies in the maintenance planners filling out the work order sheet to inconsistent usage of the new PMT <

reference manual in specifying PMT requirement During this inspection, the inspectors reviewed corrective actions taken by the licensee to resolve the above inconsistencies. The inspectors noted that '

the licensee has established a PMT group responsible for reviewing new work

i

'

T

.

.

-19-packages for consistent identification of PMT requirements. The PMT group was also responsible for review of previously planned work packages for identification of PMT in accordance with the PMT reference manual, and for assisting planning, craft, or work start authorities as necessary with the identification / implementation of PMT. In addition, the licensee provided further training to maintenance planners on PMT and filling out the work order instruction LICENSEE'S A9,ILITY TO IDENTIFY, PURSUE, AND RESOLVE PLANT PROBLEMS (92720)

The liciasee has demonstrated aggressive problem resolution particularly in the area of hardware problem correction in preparing for the Unit I startu The goals for service request backlog are being met, however the SPR backlog remains consistently high. The licensee's independent assessment findings for .t the week ending December 2, 1993, indicated the effectiveness for corrective actions associated with backlogs, management oversight and ownership, communication, and timely and effective correction of identified problems still require further evaluatio I

!

l l

l

__ - _.-- .

l

..

,

'

ATTACHMENT 1 i

1 PERSONS CONTACTED

> Licensee Personnel

,

• Berg, Manager, Design Support, HL&P

• Butterworth, Unit 1, Ops Manager, HL&P

• Cloninger, VP Nuclear Engineering, HL&P

• Coates, Unit 2 Manager, HL&P Maintenance

• Conly, licensing Engineer, HL&P 3

• Cottle, Group VP Nuclear, HL&P

• Cumeaux, Consultant Engineer, ISEG ,

• Dahl, Consultant, Enercon  !

• Daniels, Administrator Corrective Action Group e

• Ferguson, Consultant / Licensing Engineer, HL&P

• Fisher, Spur Engineering Specialist, HL&P ,

• Groth, VP, Nuclear, HL&P i

• Helton, Unit I Manager Assistant, HL&P '

• Johnson, Supervisor QA, HL&P/QA

• Jordan, Manager, Systems. Engineering, HL&P

• Jump, Director, Regulatory, HL&P

• MacKenzie, Senior Consulting Engineer, HL&P/DED

• McIntyre, Manager, Engineering Support, HL&P

• Pacy, Manager, Engineering Prog., HL&P

• Parrish, Senior Specialist, HL&P/NL '

• Parthasarathy, Support Engineer /DED, HL&P

• Rehkugler, Manager QC/MT, HL&P/QC/MT

• Rencurrel, Unit 2 Maintenance Manger Assistant, HL&P  :

• Selman, Consultant, Selman .;

• Sheppard, GM, Nuclear Licensing, HL&P

• Smith, Senior Consultant, HL&P ,

'

• Soward, Nuclear Assessment Support, HL&P

• Stephenson, Licensing. Engineering, Enercon

• Stonestreet, Outage Manager, HL&P/0utage

• Tapplett, Manager, NSAP, HL&P

• Thomas, Manager DED, HL&P 'i

• Towler, QA Ops Supervisor, HL&P/QA >

l

• Underwood, Support Manager, HL&P

• Walker, Licensing Engineer, HL&P l

• Wittman, Work Control Supervisor, HL&P j In addition to the personnel listed above, the inspectors contacted other i personnel during this inspection perio ; NRC Personnel  ;

• Goldberg, Reactor Inspector, Division of Reactor Safety

• Johnson, Chief, Project Section A, Division of Reactor Projects  ;

• Keeton, Resident Inspector I

• Loveless, Senior Resident Inspector -l

• McKernon, Reactor Inspector, Division of Reactor Safety l

• Vickrey, Reactor Inspector, Division of Reactor Safety )

• Westerman, Chief Engineering Section, Division of Reactor Safety )

• Denotes personnel attending the exit meetin ,

,

_.,-g --

w-n , m " ,, ns, + w ~ + x-

(- . . . . . - - - -- - .- - . . .

.

s ,

-2-2 EXIT MEETING An exit meeting was conducted on December 17, 1993. During this meeting, the inspectors reviewed the scope and findings of this report. The licensee did 1 not take exception to any of the inspection findings, and did not identify as '

proprietary any information provided to, or reviewed by, the inspector ;

,

i

,

,

!

l

)

l

.

I

!

.

!

- . - . . -- . - . .-. - - - - - .. -

- - . . .

.

r i

• ,

.. .:

ATTACHMENT 2 l i

EVENTS AND CASUAL FACTORS DIAGRAM The attached Events and Causal Factors diagrams depict the Wrong Valve Actuator Work Event sequence and cause . The boxes contain information concerning the event sequenc ,

2. The ovals describe conditions that may have influenced human performance i

in the event sequenc ,

I

~ The text under each condition identifies general areas considered for additional root cause investigatio ,

l l

I

,

!

i

'j

_ . . . .

.

.

a-21-ATTACHMENT 2 EVENTS AND CASUAL FACTORS DIAGRAM

- The attached Events and Causal Factors diagrams depict the Wrong Valve Actuator Work Event sequence and cause . The boxes contain information concerning the event sequenc . The ovals describe conditions that may have influenced human performance in the event sequenc . The text under each condition identifies general a eas considered for additional root cause investigatio .

ATTACHMENT 2 DED Evaluated HLLP Provides Service Reauest Dew ShtFt form HLEP and ITI Movets Ovel Movets & Trevelor Suce - Receives Movete agree Identseses Technicians Precered re- t/P- Crew A on contreet freinsng Station Cnt Scr ew Completes fork Needs Procedure teng Actuator Ins Exc C/0 Inso-I T  ; i

!

'

!  !  !

r--- i  :  :

!  ; 1 DED Dverlooks Deu Shirt has HL&P & Movets so*e reoutred Not all Rea'd Trng Nee ~ Rise on # coag F-

'

understand training in Provided all Movet Comconcat - Not coctreet maintenance peccess Teebs Tvreed Over to Night Shift

-

TRAINING TRAINING COM"UNICAT!DNS SUPERVISION p- DED CTC bed not Night Crew Night Crew Crew DreviousIw Receives Receives Inittetes performed contract r--------------------* Written W/P Deviews tork Pecutred B monitoring '

Turn-over er Work & bg W/P Pe.eining Work Cleerex es

. .

TRAINING l .' '

CRGANIZRTIONRL FRCTCRS  !

? .

Crew NOT l

N2ght t oc k identary Compenent Crew NOT Identtry Sveer Assigas Swstem. NOT vertfw Swste & Component i

'-

sock Crew Cleerence on Co*oonent CED Menegement not ensure croced re <

edhee.nce TRAINING TRAINING SUPERVISION HUMRN ENGINEERING SUPERVISION Forest Pre-seb ORGANIZATIONAL FACTORS SUPERVISION Brser Not Given to Crew CCMMUNICRTION ORGANIZATICNRL FACTORS TRAINING

,

ORGANIZATIONRL' FACTORS R ---------- a EVENTS & CAUSRL FRCTORS SUPERVISION IMPROPER MOV MRINTENANCE Night Work Svoer Receives SHEET ore Fece-to-Face T/0. Recepts e/P

!

e .. .

. - _ _ _ _ _ - _ - - - . - - - - . - . - - - - - - - - - - -- --- - - - - - - - - ~'

ATTACLV4ENT 2 Crew Fanda Crew Crew

'

Comooaents in Crew Stects Co-eletes Co+oletes O Conte tnoted w or k on Dock on e-eag s/P. Reouests Arees. Receives Retuato- Rctuator Cles ence be foek Rooroval leaves Scene Lifted i T l l

,.....  ; ___

t  !

C-e, Notes Work Crew Notes Work Conditsens 02rr e ert Conditters Darrereat TFen E=cecte Then Encected. NOT Nottfies HP. NOT Nottrw Suce visor i Supeevisoe i

, I

.

TRAININb TRAININ3!

Stetton ORGANIZh' TIONAL FRCTORS CAGANIZRTI'CNRL FACTORS Preblem _"

Report .

Prece e h H,elth Physses Crew NOT Perform Perests Work w/o Electrice! Sere ty RTP Mo Checks IRRINING HUMAN ENGINEERING TRAIN!NG PROCEDURES RPO Returns Fron RPO Discusses SI Rounds. D1-ected to Clearance w/ Unit Lset Clee-ence on Supervision - CS Velves Notes

- Lee-rs there is --* Problem. Notteses NCNE for SI wor Unit Supervision-Goes for Werker Brings Workers to Reector Plent But Can' t F in Control Room for Operator (RPO) RPO Coe lotes Investigatio Accompenses Crew to local C eck Dock Scea Looks 9 9/P --

Inauiree Rbout Crew Cles-ence Having Clearance T

,.4 EVENTS E CRUSRL FACTORS RPO NOT Note Cleerence for CS IMPROPER MOV MAINTENANCE Velves vs. SI SHEET it0 HUMAN ENGINEERING

. .. ,

• +-- _ - . _ - _ _ _. ___