ML20072P562

an Assessment of Hl&P Management Prudence at South Texas Project

ML20072P562
Person / Time
Site:	South Texas
Issue date:	07/31/1994
From:	HOUSTON LIGHTING & POWER CO.
To:
Shared Package
ML20072N121	List: ML20072N157 ML20072P544 ML20072P562
References
NUDOCS 9409080046
Download: ML20072P562 (302)

Text

.-

Figure Id_5-2 An Assessment of HL&P Management Prudence at the South Texas Project 1

THE T,TRERTY O

CONSULTING  !

4 GROUP ,i M'$lkik.

250 West Pratt Street Suite 2201 Baltimore, Maryland,21201 (410) 625-0990 July 1994 p

9409080046 940830 PDR ADOCK 05000498 T PDR

1 I

1 This page intentionally left blank

)

i i

i l

I I

RIS '

O

) A)

'ud i Table of Contents 2 Table of Contents i 3 Glossan- v 4 Chapter One - Introduction 11 5 i Purpose . II 6 ll Criteria 1-3 7

Ill. Methodology 1-4 8 IV. STP's History I5 9 A. Economic Performance I-5 10 B Regulatory Performance I-6 11 C. Recent History I8 12 V. He DET Evaluation 19 13 VI. Summary Conclusions 1-12 14 15 Chapter Two - Operations 11- 1 16 I. Introduction and Summary of Findings 111 17  !!. Operations Staffing 113 18 A. Administrative Tasks . 11 19 B. Operations Staff Rotation 118 20 C. Shutdown From Outside the Control Room . 11 10 21 D Three Train Design 11- 1 1 22 1. Surveillance Testing . 11- 1 1

23 2. Limiting Conditions for Operation 11 13

\  !

24 E. Dual-Unit Outage and Operations Overtime 11-1 6 l 25 F. Independent Evaluations of Operations Staffing 11-1 8 1 26 111. Support to Operations . 11- 2 0 27 A. Introduction 11 20 28 B. Automatic Controls 1121 l 29 C. Use of Computer Information Systems 11-2 7 l

30 D. Tecimical Specifications 11-3 0 i 31 IV. Communiutions with the Control Room Staff 11 35 32 A. Testing of Manual Trip Circuitry Il 35 33 B. Control Room Written Guidance 1139 34 V. Operator Performance 11-4 3 35 A. Introduction 11-4 3 )

36 B. Unmanned Control Room 11-4 4 37 C. Boron Dilution Event 11 46 38 D. Incorrect Valve Manipulation . 11-4 8 39 -

E. Other NRC Evaluations of Operator Performance  !!-49 40 VI. Problem Identification and Resolution by Operations . 11-5 1 41 A. Plant Labeling 11 51 42 B. Locked Valves ,

11 52 43 C. OERs and SPRs 11-5 3 D

Page i The Libery Consulting Group

R!s-:

A 1 Chapter Bree - Niaintenance 1111 2 I Introduction and Summary of Findings 1111 3 11. Corrective hiaintenance . .1113 4 A Introduction ,1113 5 B Feedwater Isolation Bypass Valve Position 1114 6 C Diesel Water Jacket .1115 7 D Qualified Display Processing System .Ill7 8 E Steam Generator Access Covers III.10 9 "

111. Preventive hiaintenance lil-12 10 A. STP's Preventive Niaintenance Program History III-12 1i B. Industrv Perspective . Ill.14 12 C. STP's Phi Program Development . 111-1 6 13 D Examples Used by the DET 111-1 7 la IV. hiaintenance Training 111-1 9 15 A. Introduction .111 19 16 B. Niaintenance Training issues . .111 20 j 17 1. Accreditation . ((120 l 18 2. Journeyman Qualification and Use of OR Form . 111-2 6 19 3. Work Direction of Unqualified Personnel 111 26 20 C DET Examples . 111-2 8*

21 1. hioided Case Circuit Breakers . .111 28 22 2. Other DET Examples . III-30 23 V. Replacement Parts . III-32 24 A Availability of Replacement Parts . !!!.32 C 25 26 B. hiOV T. Drains C. Use of Correct Pans .

. III.34

.111 36 27 VI. Support to hiaintenance III.38 28 A. hiaintenance Backlog . Ill.38 29 B. Staffing and Overtime .111 41 30 C. Vibration hionitoring Program .111 48 31 VII. Work Control Process . . 111-5 1 32 A. Background and Perspective . . 111-5 1 33 B. Other NRC Evaluations . 111-5 4 34 C. Independent Arsessment . 111-5 5 35 D. Efficiency of the Work Control System .111 56 36 E. Specific DET Comments on Efficiency of Work Control 111-5 7 37 F. Conclusions .

.111 60 38 Vill. Post hiaintenance and Periodic Testing . . 111-6 1 39 A. Post hiaintenance Testing . .111 61 40 1. Introduction , .11161 41 2. Standby Diesel Generator .III-61 42 3 Diesel Breaker . 111-6 3 43 4. Chiller Breaker . . 11!- 6 4 44 5. HL&P's Actions . . Ill.64 45 6. Other Evaluations of STP's PhiT . !!!-65 46 B. Periodic Testing . 111-6 7 Pagen The Libem Consultmg Group

l l

U s.. .

,m

( l IX Turbine Drisen Auxiliarv Feeduater Pumps ill-70

\ 2 A introduction 111 70 l 3 B. Overspeed Trip initiatmg Es ent 111 70 I 4 C History of Overspeed Trip Problems . Ill-71 5 D Corrective Actions . 111 73 6 E Evaluation . . 111- 7 4 7 F Conclusions . 111- 7 8 8

Chapter Four - Engineering IV-1 9 I. Introduction and Summary of Findings IV-1 10 II. Engineering's Resolution of Plant Problems IV-5 11 A, Root Cause Analyses and Corrective Actions IV 5 12 B Temporary Modifications IV-7 13 C. Injector Pump Hold-Down Studs IV-11 14 D Toxic Gas Monitors . IV-14 15 E. Solenoid Operated Valves . IV.16 16 F. Startup Feedwater Pump IV-18 17 G Steam Generator Feed Pump Turbine . IV-20 18 H. TSC Diesel Generator IV.24 19 1. Torque Setting on Motor-Operated Valves IV 26 20 J. Flow Noise in the Auxiliary Feedwater System IV-28 21 III. System Engineering IV.30 22 A. Introduction IV-30 m 23 B. Resources IV 32 24 C. Information Systems (VI 25 D. Training IV-33 IV.34 26 E. Conclusions IV-35 27 IV. Engineering Work Load . IV-37 28 A. Backlog IV-37 2v B. Emergent Work IV-41 30 C. Work Control IV.41 31 D Work Load and Overtime IV-43 32 E. Conclusions IV-46 33 V. Use cf Openational Experience IV 48 j 34 A. Reactor Coolant Pump Bearing Failure IV-48 l 35 B. Diesel Rocker Arms . IV-52 1 36 C. Vendor Equipment Technical Infonnation Program IV-54 I 37 D. Application of Risk Assessment Technology IV 57 38 VI. Support to Engineering IV 61 39 A. Management Informnion Systems SCS) IV-61 1

40 B. Personal Computers . IV-62 41 C. Increasing Backlog of Work IV-63 )

42 D. Training IV-63 l 43 E. Application ofimpros ement Programs IV-64 j 44 F. Timeliness of Modifications IV-69

-15 G Independent Studies of Engineering . IV 71 16 H. Conclusions IV.73 47 Vll. Configuration Control . IV 75 f .r ] l 18 A. Background and Perspective .

i; IV 75 Page ni The Dberty Consulnng Group

RL.\ '

1 B. DET Examples - Conficuration Control 2

~

tv-79

' 1. hioided Case Circuit Breakers CICCB/ IV-79 3

2. Installation of SDG Rocker Arms IV.79 4

• 3. Environmental Qualification of Valve Actuator hiotors IV 81 5 4 Vendor Drawings IV 81 6 C Conclusions 7

IV-82 VIII. Essential Chilled Water System IV-84 8 A. Introduction IV-84 9 B. DET Examples - Essential Chilled Water System .

• IV 84 10 C. Conclusions IV 92 11 IX. Fire Protecti n Systems IV-94 12 Chapter Five - Afanagement and Organization . V1 I 13 l. Introduction and Summary of Findings V.1 14 II. Direction and Oversight . V-3 15 III. Support and Resource Utilization V.I?

16 A. hianagement Practices . V-12 17 B. Funding V 14 18 C. Staffing . V-19

, 19 IV Communications and Teamwork . V 21

,, 20 A. Task Forces V-21 21 B Speakout Program

' V-23 22 V. Corrective Action Process . V 26 -

, . - 23 A. Background and Perspective .

/ V-26 24 B. Outage Planning . V.2 7 25 VI. Self Assessment and Quality Oversight V-30 26 A. Introduction V-30 27 B. Nuclear Safety Review Board (NSRB) V-31 28 C. Quality Assurance (QA) . V-33 29 D. Independent Safety Engineering Group (7SEG) . V-39 30 VII. Information Systems . V-41 31 A. Infonnation Systems Planning and Progress V-41 32 B. DET Examples V-44

(

Pagenv The Libery Consulting Group

an.:

)

i^ -

(s 1 Glossary l l

l 2 AEOD NRC Office for Anal) sis and Es a!uation of Operational Data i 3 AFW Autliarv Feedwater 4 AIT Augmented luspection Team ,

5 ASP AuVliary Shutdown Panel l

6 BOP Balance of Plant 7 BTRS Boron Thermal Regeneration System 8 ,CAG Correctise Action Group 9 CAL NRC Ccnfirmatorv Action Letter 10 CCW Component Cooling Water 11 CH Essential Chilled Water System 12 CMAP Cooperative Management Audit Program 13 CPL Central Power and Light la DBA Design Basis Accident 15 DBD Design Basis Document 16 DCN Design Change Notice 17 DED Design Engineering Department 18 DET Diagnostic Evaluation Team 19 DG Diesel Generator -

20 DR Deficiency Report 21 EAP Employee Assistance Program 22 ECN Engineering Change Notice -

g 23 ECW Essential Cooling Water s J 24 EDSFI Electrical Distnbution System Functional Inspection V 25 EPRI Electric Power Research' Institute 26 EQ Equipment Qualification 27 ESF Engineered Saferv Feature 28 FSAR Final Safety Analysis Report 29 FW Feedwater 30 GET General Employee Training 31 HHSI High Head Safety Injection 32 HL&P Houston Lighting & Power Company 33 HVAC Heating, Ventilation, and Air Conditioning 34 IC Instrumentation & Controls 35 IE Inspection and Enforcement 36 ILRT Integrated Leaf Rate Test 37 INPO Institute for Nuclear Power Operations 38 IPE Individual Plant Evaluation 39 ISEG Independent Safety Engineering Group 40 IST Inservice Test Program 41 JUMA Joint Utility Management Audit 42 KV Kilovolt 43 LCO Limiting Conditions for Operation 44 LER Licensee Event Report 45 Libeny The Liberty Consulting Group 46 LOCA Loss of Coolant Accident

(

i 47 48 LRISP MCCB Long Range Information Systems Plan Molded Case Circuit Breaker Pages The Libern Consulnng Group

iu 8 :

O I MDP ModiScation Design Package 2 MIS Management Information S3 stem 3 MOP Master Operating Plan 4 MOV Motor Operated Vahe 5 MPE Maintenance ProSciency Evaluations 6 MTl Maintenance Team Inspection 7 MWH Megawatt Hour 8 MWR Maintenance Work Request 9 ,NCR Non Conformance Report to NNAB National Nuclear Accreditation Board 1I NRC U S. Nuclear Regulatory Commission 12 NRR NRC Office of Nuclear Reactor Regulation i 13 NSRB Nuclear Safety Review Board la NUMARC Nuclear Utilities Management and Resources Council 15 OER Operating Experience Review 16 01 NRC Office ofInvestigations 17 OIP Operational Improvement Plan 18 OJE On the Job Experience 19 OJT On the Job Training 20 OSTI NRC Operational Safety Team Inspection -

21 OTL Operability Tracking Log 22 P&lD Piping and Instrumentation Diagram 23 PC Personal Computer

{ 24 25 PCF PDC Plant Change Form Project Document Control

( 26 PDI Performance Data Inc.

27 PED Plant Engineering Department 28 PI Performance Indicator 29 PM Preventive Maintenance 30 PMT Putt-Maintenance Testing 31 PORC Plant Operating Review Committee 32 PRA Probabilistic Risk Assessment 33 PUC Public Utility Commission 34 PWR Pressurized Water Reactor 35 QA Quality Assurance  !

36 QC Quality Control 37 QDPS Qualified Display Processing System I 38 QR Qualification Record 39 RCA Root Cause Analysis 40 RCM Reliability Centered Maintenance 41 RCP Reactor Coolant Pump 42 RFA Request For Action 43 RHR Residual Heat Removal 44 RMA Responsible Maintenance Authority 45 RPO Non-licensed Reactor Plant Operator 46 SALP Systematic Assessment of Licensee Performance 47 SDG Standby Diesel Generator 48 SER j']

Safety Evaluation Report g 49 SOFPT Steam Generator Feedwater Pump Turbine Pagesi The Lsberty Consulnng Group

!$!.N=.'

n 1 SOV Solenoid-Operated Valve 2 SPR Station Problem Report 3 SR Sernce Request a SRO Senior Reactor Operator 5 SSFA Saferv System Functional Assessment 6 STA Shift Technical Advisor STP South Texas Project Electric Generating Station 8 STPEGS South Texas Project Electric Generating Station 9 SLTP Stanup Feedwater Pump to " TCl Technical Specification Clarification 1I TCV Temperature Control Valve 12 TDAFWP Turbine Dnven Auxiliary Feedwater Pump 13 TN1 Temporary hiodification 14 TS Technical Specifications 15 TSC Technical Suppen Center 16 TSI Technical Specification Interpretation 17 VDC Volts-DC 18 VETIP Vendor Equipment Technical Infonnation Program 19 Whis Work hianagement System O

l I

i l

l l

L Page s ti The Liberg Consulnng Group I

ce ae.g,ama.L41 -- - ae"*6A-h--e4 - + - - 4.- --4 -- m-Im4-- s+-4 m-mA h-wa m a /- - a s ., *J- pA ++-.md ---4m---h--W+4 d-. 4 de4-.>-+medMe5 4-AEm'-44&4-,_-.,A aL__m.J.-a.-AA. 4#-wh 3

9 l

i I

I 4

4 1

i i

a f "

i J

f f

i 1

I b

J h ,

i d

4 l  !

. r l

c l

j 1

s ii r

a J

e i

l 1

. - - -- v - ,.n.-., .- -, , -,,. , . -- , , - , ,r, e? , . e- ,--,,

RD ' l Chapter One - Introduction (3 i O i Chapter One -Introduction l 2 I. Purpose 3 The staff of the U S. Nuclear Regulatorv Commission /XRC/ issued the Diagnostic Evaluation 4 Team (DET) Repen (DETreport) on the South Texas Project Electric Generating Station /S7P/

5 in early June 1993 The NRC staff placed STP on its Watch List later that same month. Prior to 6 the DET's review, in February 1993, mechanical problems with the turbine-driven auxiliary 7 feedwater pumps caused both units at STP to be shut down The units were shut down at the time 8 of the DET's review and remained shut down for a considerable period after the NRC issued the o repon.

10 STP's regulatory and operational record had been solid and so the announcement of the NRC's 11 plans to conduct a DET came as a sugrise to STP's owners. The DET repon was an even greater 12 shock. Although each substantive issue discussed in the DET repon had been previously identified 13 by STP's management, the discussions of these issues were surprisingly harsh given the NRC's prior assessments of the plant.

14 ih b) 15 Houston Lighting & Power Company (HL&P) requested The Liberty Consulting Group /Libernj 16 to perform a prudence evaluation focusing on the issues, areas, and topics addressed in the DET 17 repon The DET report was very critical of the management of STP. However, the methodology, 18 purpose, and standards used by the NRC in its capacity as a safety regulator were not the same as 19 those ordinarily used by an economic regulator in a prudence review. The most significant 20 differences between the DET's evaluation and a pmdence review are that the former focuses on 21 results and takes full advantage of hindsight in identifying weaknesses and areas needing 22 improvement in licensee performance. A pmdence review of management, on the other hand, must 23 evaluate the reasonableness of decisions and actions of management only in light of that 24 information reasonably available at the time of those decisions and actions 25 To perform its evaluation, Libeny reviewed contemporaneous documentation and interviewed 26 many of the key managers involved in the decision-making process. In addition, Liberty evaluated 27 the decisions made and actions taken at STP in light of Libeny's knowledge of industry conditions  !

l 28 existing at the time. i 29 Liberty concluded that STP had been managed by competent and experienced individuals who j

\ 30 made reasonable and pmdent decisions regarding actions to be taken. While some decisions may Page I-l The Liberry Consultmg Group 1

1 l

in s.: j Chapter One - Introduction l i

/^N l h I not have turned out as expected and some actions may have failed to achieve the desired results.

2 Liberty found no instance in which STP's management failed to select a course of action from 1 among reasonable options The NRC criticized some of these actions, but it is the NRC's role to 4 demand perfect performance from its licensees and to encourage licensees to strive for perfection.

5 It was not Liberty's intention to be critical of the NRC's ongoing exhortation for improved 6 performance. Rather Liberty's intention was to evaluate the issues contained in the DET report 7 in recognition of the sharp distinction between the NRC's role as a safety regulator and the criteria 8 by which it evaluates nuclear plant operators and the PUC's role as an economic regulator and the 9 criteria it must apply.

i 10 The purpose of this report is to present the results of Liberty's prudence review of the issues that 11 were raised in the DET report A

r I

l l

l l

,/'%

(

's Page I-2 The Libern Consulting Group i

IU 3. '

Chapter One - Introduction f

( i II. Criteria 2

In assessing management prudence relative to the issues raised in the DET report. Liberty adopted 3 the prudence standard employed by the Public Utility Commission of Texas This standard a

requires the exercise of thatjudgrnent and taking of that action which a reasonable person or entity 5 would exercise or take in the same or similar circumstances, given the information or alternatives 6 available at the point in time such judgment is exercised or action is taken.

7 STP was shut down for a lengthy period of time in 1993 and early 1994. However, this 8

undesirable end result alone cannot constitute grounds for concluding that utility management was 9

imprudent. Rather, >;ndesirable results serve primarily to highlight areas requiring further 10 investigation and analysis Similarly, the criticisms of STP's management presented in the DET 11 report should not be taken to be dispositive of the question of whether management was 12 imorudent The NRC applied a different standard as a basis for those criticisms However, the 13 criticisms in the DET repon should be seen as having identified cenain significant issues that are' 14 worthy of a review to independently assess the reasonableness of management's decisions and actions on the basis of the prudence standard.

15

[G 16 Moreover, the DET report, which is a snapshot of the project, cannot be read in isolation. To do 17 so could give the impression that STP had a historf of poor performance and management 18 difTiculties. Yet, in the years before the 1993 outage, STP had been a solid performer with a good 19 regulatory history and an excellent record for producing power economically. Tu evaluate 20 properly the issues raised in the DET report, Libeny viewed those issues in the context of not only 21 what was happening at STP during the applicable time frame but what had gone on before This 22 historical perspective, as well as the industry perspective atTorded by Libeny's knowledge of other 23 plants was of little or no importance to the NRC. The NRC is concerned with performance The 24 NRC uses hindsight to form its conclusions on the basis of results or outcomes In a prudence 25 review, a PUC is concerned with whether management acted reasonably given what was known 26 at the time.

1 l

l I

[\

N Page 1-3 The bberty Consulnng Group l

(U .8 .

Chapter One - Introduction

~ fh i III. .\lethodology '

2 Liberty's basic methodology was to (1) review a section or conclusion in the DET report, C) 3 gather and evaluate the information available on the subject, and (3) evaluate the prudence of 4

HL&P's management with respect to that issue. Liberty particularly sought to find information 5

that related to what HL&P had known and done with respect to a panicular issue prior to the 6

DET s evaluation. Liberty gathered information, in addition to that provided by the DET report, 7

that could be relevant to a pmdence inquiry. This was important not only because the DET did not 8

perform a pnadence review but also because the NRC purposely tries to make repons by DETs 9 concise and might not, therefore, have discussed information relevant to the reasonableness of to management's decisions and actions.'

11 Much of the information available to Liberty was provided to the DET. The DET asked HL&P 12 to respond to nearly 1,400 questions during the course of its review at STP The informatiot>

13 provided in response to those questions occupied about 150 feet of shelf space. This material was la made available to Liberty and constituted a primary information source. Liberty also interviewed

[N 15 personnel at STP, as the DET had done.

16 Liberty also used additional information that bad been available to the DET. This included things 17 like prior NRC inspection reports, SALP reports, Licensee Event Reports fLER) IhPO (Institute 18 for Nuclear Power Operations) evaluations, and correspondence between the NRC and HL&P.

19 There was some information available to Liberty that had not been available to the DET simply 20 because Liberty's review was performed later. This included things like correspondence and 21 repons issued after the DET issued its report. Finally, Liberty used historical documents fred STP 22 and material from other plants to add perspective to the review of management actions 23 Liberty structured its work efforts along the same lines as the organizatior, cf the DET report That 24 is, four primary areas ofinquiry were established: (1) operations, (2) maintenance and testing, (3) 25 engineering, and (4) management and organization. The results of Libe ty's evaluation are 26 presented in chapters corresponding to these same four areas.

f~%

) '

NRC Diagnostic Evaluation Propam Handbook. 8.7, p. 5.

Page 14 The Liberty Consulnng Group

RLs .'

Chapter One - Introduction (3

C) i IV. STP's History 2 A. Economic Performance 3 HL&P declared STP Unit I to be in Commercial Operation on August 25,1988. Ten months later.

4 on June 19,1989, HL&P declared Unit 2 to be in Commercial Operation.

5 During its Orst four years of operation, Unit I had a cumulative capacity factor of 65.6 percent 6 During its 52-month history through the end of 1992, Unit I had a cumulative availability factor 7 of 68 5 percent, a cumulative capacity factor of 62.0 percent, and a forced outage rate of 14 4 8 percent In the last quarter of 1992, HL&P completed the fourth refueling outage on Unit 1.

9 During Unit 2's Erst four years of operation (which includes a ponion of the 1993 outage), the 10 cumulative capacity factor was 64.0 percent. Through the end of 1992, Unit 2 had a cumulative 11 availability factor of 77.5 percent, a cumulative capacity factor of 69.7 percent, and a forced 12 outage rate of 11.8 percent. The units at STP were on an 18-month refueling cycle and Unit 2 did 13 not have a major outage in 1992. Unit 2 set records by producing over 10 million MWH net O 14 electrical in 1992, with the generator being on line for 8,549 of the 8,784 hours0.00907 days <br />0.218 hours <br />0.0013 weeks <br />2.98312e-4 months <br /> in the year Unit h 15 2 had an 85-day refueling outage scheduled to begin on February 27,1993.2 16 The following table shows the annual capacity factors for the years 1990 through 1992.3 17 STP Annual Capacity Factors (%)

18 1090 .Lol .Lo2 19 Unit 1 54.8 65.8 66.1 20 Unit 2 58.7 66.2 94.I 21 Power production costs at STP dropped each year of its operation through 1992. In 1989, 22 production costs were 18.0 mils (1.8 cents) per net kwh (kilowatt-hour). In 1990,1991, and 1992 23 production costs were 16.9 mils,16.3 mils, and 14.1 mils, respectively. Nuclear industry average 24 production costs were over 21 mils during this period.

2 Monthly Operating Repon to the NRC, ST lE AE-4302, January 15,1993.

/ '

( Monthly Operating Repons to the NRC, ST-lR AE 3669, January 10.1991 ST-lE AE-3973. January 14.

(, 1992, and ST-lE-AE-4302, January 15,1993.

PageI 5 The Liberry Consulting Group

RL.8-:

Chapter One-Introductmn i

,V 1 B. Regulatorv Performance l

2 STP had shown consistently strong regulatory performance, as documented by the NRC SALP 3 repons The SALP (Systematic Assessment of Licensee Performance) is "an integrated NRC staff 4 effort to collect available observations and data on a periodic basis and to evaluate licensee 5 performance based upon this infonnation "'In the first SALP report after Unit I began operations, 6 the 'iRC said 5 7 "While some apparent weaknesses were identined, it is my view that the overall 8 performance at South Texas Project has been satisfactory with apparent 9 improvement in specified areas.. . Strong management involvement, good to operating experience with Unit 1, and the ability to apply lessons learned 11 characterize the licensee's performance at STP, Units 1 and 2."

12 The NRC began its overview of the STP assessment for the period from January 1,1989 through 13 January 31,1990, with the following statement.6 .

14 "STP is a plant with strong management involvement, good operating experience 15 with Unit I and stanup experience with Unit 2, and a strong commitment to

( 16 safety."

17 The next SALP covered the period from February 1,1990 through May 31,1991. The initial 18 overview comment in that repon was?

19 "Overall, licensee performance was good and improvements were noted in some 20 areas. However, the licensee was unable to sustain the superior level of 21 performance that was achieved in the previous SALP assessment period in the area 22 of plant operations, maintenance, and surveillance."

23 The next and final SALP covered the period from June 2,1991 through August 31,1992. The 24 corresponding overview statement in that report was.'

Re introductory section of all SALP reports.

' NRC SALP Report, ST-AE-EE-92053 March 20,1989, cover letter p 1 and report p. 3.

NRC SALP Report, ST-AE-lE-92567 June 20,1990, p. 3.

f] '

NRC SALP Report, ST AE-EE-92831. Septernber 6,1991, p. 2.

NRC SALP Report ST-AE-lE-93239, Novernber 18,1992,p.2.

Page I-6 The L berry Consulnng Group

R1 \.:

Chapter One - Introductinn

(*

1 "Overall. licensee performance was good, how ever. a decline in performance was 2 noted m some areas The NRC's SALP program assigned numerical ratings to seven functional areas A "1" rating 4 indicates " superior" performance and possibly reduced levels of NRC inspection effort A "2" 5 rating indicates " good" performance and normal inspection efforts, and a "3" rating indicates o " acceptable" performance but possibly increased inspection effons in each of its SALP evaluations since the beginning of operation, STP always received either a "1" or a "2" rating in 8 all functional areas. Most other plants that had been placed on the NRC's Watch List had received 9 at least one "3" rating before the NRC decided the plant needed extra regulatory attention lo Although HL&P and the other owners were not notified until February, the NRC decided in 11 January 1993, to perform a DET review of STP. The NRC generally does not perform these 12 evaluations unless it has reason to believe that performance problems might affect plant safety.

13 Given STP's solid regulatory performance, the selection of STP for a DET review came as s 14 surprise to STP's management. The DET report later cited the decline in SALP performance as 15 a primary reason for conducting the evaluation. However, the decline did not appear to be

[h 16 extremely significant, in fact, the SALP report had many positive comments about STP and the 17 SALP average numerical ranking was the same as that given when the NRC issued STP's full 18 power license. Moreover, prior to the time the DET was announced HL&P had informed the NRC 19 of substantial actions it was taking in response to the SALP's assessment and such actions had 20 clearly not had time to take effect. The DET report also said that hardware problems ar.d personnel 21 erTors had resulted in reactor trips and plant transients. However, STP's performance in this area 22 had improved considerably from its initial years of operation. The DET report noted that an NRC 23 Operational Safety Team Inspection (OSTI) conducted in December 1992 found that issues 24 associated with hardware problems had not been resolved. While the OSTI had expressed concerns 1 25 related to the identification and resolation of hardware deficiencies, it also found that STP was 26 staffed by " competent knowledgeable personnel who executed their duties in a professional  !

27 manner. Several notable strengths were identified in the area of plant operations."' 1 l

l l

NRC Inspection Report No. 92-35 ST AE-HL-93325, March 3,1993.

PageI 7 The Liberry Consulnng Group

a RIM:

Chapter One - Introduction

()

V i C. Recent History 2 Soon after the NRC decided to perform the DET review. STP experienced overspeed trips on the 3 turbine-driven auxiliary feedwater pumps of both units. "Overspeed trip" is an automatic 4 shutdown of the pump to protect the pump from damage due to excessive speed The trip on Unit 5 2 coincided with a unit trip, and the problem on Unit I was not completely resolved before a 6 Technical Specification limit required that unit to be shut down also. HL&P told the NRC that it 7 would not start up either unit at STP until it had assured itself that the problems that had caused 8 the overspeed trips had been resolved and it had briefed the NRC on the resolution The NRC

9 confirmed HL&P's intended actions in a Confirmatory Action Letter /C.fL) dated Februarv 5 10 1993 Subsequently, the NRC sent a letter dated February 12,1993 to HL&P that said it was 11 planning to conduct the DET review. Beth units at STP were shut down when the DET arri,ed 12 on site on hiarch 29,1993.

13 On hiay 7,1993, HL&P got the first taste of the results of the DET's evaluation. In a telephone

• I 14 conversation and a written supplement to the CAL, HL&P was told that several matters, in 15 addition to the auxiliary feedwater pump trips, had to be resolved before either unit could be 16 restarted. These matters included the process for reporting problems, the size of the service request 17 and engineering bacidogs, the adequacy of staffmg and training, and the corrective action process 18 On June 3 the NRC conducted an exit meeting in which it discussed the results of the DET's 19 review. The NRC issued the DET report one week later.

20 The repon by the DET, like those issued on other plants in the past, was very negative. The DET 21 was critical of STP in operations, reuntenance, engineering, and management. It determined that 22 declining performance had been caused by inadequate management support, ineffective 23 management oversight, and ineffective self-assessments and corrective action processes.

24 HL&P developed an operational teadiness plan and a business plan to make both the near- and 25 long-term changes that were required as a result of the DET's review. HL&P also performed 26 maintenance and modifications on both units while they were shut down. The NRC performed 27 detailed inspections of the changes and confirmed HL&P's demonstration that plant systems were 28 ready to operate after the extended shutdown and maintenance period. Unit I started up in 29 February 1994 and the main generator breaker was closed on February 25. Unit 2 staned up in 30 hiay 1994 and the main generator breaker was closed on hiay 30. l O

V '

Page I-8 The Libery Consulting Group

R1.u.

Chapter One - Introduction

/%

i V. The DET Evaluation 2 Twice a year the senior management of the NRC staff meets to discuss the status of operating 3 reactors The purpose of these meetings is to help ensure that the staffis focusing its attention and a resources on the plants and issues with greatest safety significance " Following the meeting, the 5 stafTbriefs the NRC Commissioners. In the briefing of February 9,1993, the NRC disclosed that 6 it planned to perform a DET review at STP.

7 " Commissioner Remick: I have one question. In reading the background 8 information, I see that there will be a diagnostic evaluation at South Texas Is that 9 correct?

10 Mr Taylor: Yes. We weren't prepared to announce that at this stage, but that's the 11 case ""

12 The NRC informed HL&P that it planned to conduct the DET evaluation shonly thereafter." The 13 letter that announced the DET's review contained some imponant information about the purpose 14 of the assessment.

Oi V 15 16 "This will provide an independent assessment of Houston Lighting & Power Company (HL&P) performance at South Texas. The evaluation is intended to 17 supplement information from the Systematic Assessment of Licensee Performance 18 (SALP) and Performance Indicator programs, and other assessment data."

19 The letter did not contain specific reasons why the DET was to be performed.

20 "My decision to conduct a Diagnostic Evaluation at South Texas Project resulted 21 from a recent meeting of 5%C senior managers At this meeting we conducted a ,

22 detailed review of the regulatory and operational performance history of South  !

23 Texas Project, as well as other licensed nuclear facilities. During these discussions,  !

24 it was concluded that additional information regarding South Texas Project would 25 be needed for NRC senior management to more fully evaluate ovmil plant 26 performance."

• Transcript of Staff Briefing of NRC Commissioners, February 9.1993, statement by James Taylor.

Executive Director for Operations, p. 4. 4 fx "

Transcript of Staff Briefing of NRC Commissioners, February 9,1993, p. 57.

" Letter, NRC to HL&P. Taylor to Hall ST AE-HL-93309, February 12.1993.

Page I-9 The Liberry Consulnng Group

ilL% .'

Chapter One - Int roduction V 1 The DET repon described the conduct of the evaluation "

2 "The diagnostic evaluation team (the team) consisted of 15 technical members and 3 an administrative assistant and was organized with four team leaders reporting to 4 a team manager The team devoted several weeks to preparation that included team 5 meetings and brieEngs by representatives from Region IV, the Office of Nuclear 6 Reactor Regulation (NRR), and the Of6ce for Analysis and Evaluation of 7 . Operational Data (AEOD). On March 29, 1993, the team began a 2-week 8 evaluation at the facility. This Grst onsite evaluation period wu followed by a two 9 week inofEce review of data collected The team returned to the site on April 26 10 1993 for an additional week of onsite evaluation."

11 The NRC conducted an exit meeting on June 3.1993, in which the results of the DET review were 12 made public The report was issued on June 10 13 The NRC has prepared guidelines that are to be followed in the preparation for and conduct of all 14 diagnostic evaluations. Of particular signiEcance in the context of the present prudence review' 15 is the NRC's direction to DETs concerning the documentation of those issues of which the 16 licensee was not aware.

O 17 "The evaluation should focus on those safety and performance issues which 18 warrant senior NRC and licensee management attention and should clearly identify 19 those issues for which this licensee was not aware or had not taken appropriate 20 corrective action.""

21 Essentially all of the substantive issues discussed in the DET report had previously been identified 22 by STP. The executive summary of the report acknowledged that "most of the performance issues 23 observed by the team had also been identified by the licensee's own assessment effort "" The 24 body of the report contained a similar statement?

25 " Prior to the NRC's decision to conduct a diagnostic evaluation, the licensee had 26 ident16ed and acknowledged most of the problems identined by the team and had 27 made several attempts to improve performance, including implementation of an )

28 operations improvement program." l

" DET Repon. p. 2.

" NRC Guidelines for the Diagnostic Evaluation. May 1991. Revision 2. p 24.

O "

DET Report. p x.

DET Report, p. 38. .

1 Page1-10 The bberty Consultmg Group

RLS-Chapter One - Introduction O 1 In a number of cases the DET tried to show that corrective actions had not been effective, but for 2 the most pan the DET did not discuss actions that had been taken or were underway to address 1 3 the DET's issues 4 An additional aspect of the DET repon posed a challenge to determining its relevance to 5 management pmdence. As noted earlier, the SRC guidelines for a DET evaluation require that the 6 repon be concise. "The DET repon content will be controlled to provide only the detail necessary 7 to support the findings and root cause determinations."" The main body of the DET report is 8 called " Evaluation Results." The NRC's guideline for the preparation of that section says that 9 "[t]ypically, this section contains fewer than 20 pages "" While the NRC's desire to limit the size 10 of the repon is reasonable m light ofits purposes, the limitation of the description of evaluation 11 results, panicularly those of a broad-based review, can result in information not being discussed 12 that is relevant to a pmdence inquiry or necessary to place such results in context for the benefit 13 of readers not steeped in the technology of nuclear plants or the procedures of the NRC.

14 The agenda ofissues that is relevant to the pmdence inquiry is the same agenda that is followed 15 in the DET repon. As noted above, at the senior management meeting in June 1993 that followed

[ ^

16 17 the DET evaluation, the NRC Staff decided to put STP on its Category 2 Watch List. When the NRC Staff briefed the Commissioners about STP, the bulk of the discussion concerned the 18 weaknesses identified by the DET and the root causes of problems documented in the DET repon.

19 Thus, it is clear that the findings of the DET were significant to the decision to place STP on the 20 Watch List. Furthermore, the issues that the NRC required to be resolved before STP staned 21 operation again were for the most part issues covered by the DET.

NRC Diagnostic Evaluation Program Handbook. 8.7, p. 6.

NRC Guidelines for the Diagnostic Evaluation. May 1991 Revision 2. p 32.

Page I-II The Liberty Consultmg Group

RLs:

Chapar One - Introduction v i VI. Summary Conclusions 2 Libeny concluded that the NRC's criticisms of STP contained in the DET report did not show that 3 the actions taken or decisions made at STP prior to the DET's review had been unreasonable or 4 that management had been imprudent. While the NRC, in its role as a safety regulator. may have 5 questioned the effectiveness and results of certain actions and decisions, it was Liberty's 6 conclusion that STP had been managed prudently While achieving excellent performance, STP had continued its process of self-assessment and identification of areas that could be enhanced or 8 required improvement. In fact, prior to the DET's review, HL&P had identified all valid areas for 9 improvement that were later highlighted by the DET report By the time of the DET's review, lo HL&P had made progress in these areas and had plans for additional improvement.

11 The language used in the DET report was particularly harsh and the steps that HL&P took to 12 satisfy the NRC were extensive. However, prior to the DET's review, the circumstances undet 13 which STP's management made decisions would not have indicated that STP was in line for 14 severe regulatory criticism or in need of a significant mid-course correction. The plant was 15 running well. Results of the NRC's SALP, routine and special inspections, and other assessments L/ 16 were solid While areas that required attention had been identified, there were also clear 17 indications that improvement was being made and that STP was "on the right track." STP simply 18 did not appear to be a plant headed for a DET or the Watch List Other than the fact that prior 19 DET repons had been very negative, HL&P had no reason to expect the kind of harsh criticisms 20 contained in the DET report. To the contrary, HL&P had every reason to believe that the regulator 21 would continue to indicate that STP was a solid performer.

22 A review of the facts associated with the issues raised in the DET report led Liberty to conclude 23 that the management of STP had been reasonable and prudent. There were several reasons why 24 Liberty reached conclusions that seemed so different from those of the DET. Unlike the DET.

25 Liberty applied the prudence standard and thus sought to determine whether HL&P's actions and 26 decisions had been reasonable given the information available at the time of those decisions and 27 actions The DET used hindsight and applied a very different standard of review. Although the 28 DET's standard was not clearly defined, it is clear that it demanded a much higher level of 29 effectiveness of STP than those used in prior NRC evaluations of STP. The evidence of this lies 30 in the fact that the DET's conclusions about STP were inconsistent with those reached by prior  ;

31 NRC inspectors and evaluators of STP.

V i

l PageI.12 l The Liberty Consuhmg Group j l

l l l j

I 1

l

/$/ .\-2 Chapter Two - Operation, O

'd i Chapter Two - Operations 2 I. Introduction and Summary of Findings 3 The DET's review of STP in the operations area focused on staffing, the support provided to 4

operations from management and from other parts of the organization, and operator performance 5 Prior to the DET's review, evaluations of operations at STP had been very positive And while 6 the DET acknowledged that STP had a dedicated operations staff with good morale, it was critical of most of the other areas examined in operations.

8 Liberty found that the DET's criticisms of plant operations did not demonstrate that the decisions 9 made and the actions taken in the operations area of STP had been unreasonable or imprudent For 10 example, Liberty concluded that the decisions made regarding operator staffing were ressonable 11 Such decisions used as a basis not only the experience and judgment of STP's management but 12 also independent staffmg studies that were regularly conducted in order to provide furthe'r 13 guidance on staffing issues. NRC evaluations conducted prior to the DET's review, as well as 14 independent staffing studies performed for STP, did not show staffing deficiencies. STP assigned 15 licensed operations personnel to other parts of the organization, and there was substantial 16 Operational experience in the organizations that interfaced with operations. The issues discussed 17 in the DET report did not demonstrate that surveillance testing or the plant design had created is demands on operators that had not been reasonably factored into current staffing levels.

19 Support to operations was not unreasonable. For example, STP had placed emphasis, and had 20 achieved progress, on reducing the number ofinoperable automatic functions The " problems" that 21 were attributed to the use of a computer system were more likely caused by conflicting 22 requirements in Technical Specifications issued by the NRC than by inadequate support to i 23 operations.

l 24 In May 1992, STP took aggressive actions to improve communications between management and I 25 the control room staff after the discovery of an incomplete test procedure was not immediately 26 reported to the plant operators.

27 The DET used the example ofinadequate labeling of plant equipment to support several of its

S conclusions. However, Liberty found that, prior to the DET's review, HL&P had applied

, O 29 considerable resources to labeling of plant equipment after it was discernible that labeling could 30 have contributed to personnel errors. -l l

Page 11-1 l The Libery Consulnng Group l

IU.s:

Chapter Tw a - Operations O

1 Operator performance at STP was consistently praised by the NRC and other evaluators The

. 2 DETs conclusion was not consistent with these other assessments and prior NRC inspection 3

repons Liberty's conclusion, which was based upon a detailed review of the examples used by a

the DET, was that in light of the positive evaluations of the NRC and others the DET repon's 5 criticisms of operator performance did not show that management had made unreasonable 6 decisions or taken unreasonable actions.

7 Finally, the DET did not identify any substantive issues that STP had not already identified.

8 started corrective actions, or made progress toward resolving by the time the DET performed its

9 evaluation. With respect to some matters, Liberty found that more accurate or complete to information led to conclusions significantly different from those contained in the DET report (d

i Page Il-2 The Lsberry Consultmg Group

. _ _ _ _. . _ __ =. _

RLS-:

Chapter Tw o - Operations f

\

i II. Operations Staffing

A. Administrative Tasks 3 One of the principal concerns expressed by the DET about STP's operations was the 4 administrative burdens that were placed on the control room operators. The DET report stated:

5 "The shift supervisors were not maintaining a broad plant perspective because their -

6 attention was frequently consumed with administrative duties and resource-7 intensive surveillances."'

8 "The shift supervisors and their control room staff could not effectively maintain 9 the proper focus and overview of plant operations because of their participation in to administrative programs and resource intensive surveillances."2 11 "StafEng of senior reactor operator (SRO) positions was most affected by ,

12 administrative burdens. Each of the units' crews contained two SROs, the shift i 13 supervisor and unit supervisor. The shift supervisor spent the majority of his time 14 performing a number of administrative duties, including reviewing work packages i

15. for work start authority and again at closecut for post-maintenance test adequacy.  !

16 The team also confirmed through interviews that there was a heavy administrative 17 burden placed on the shift supervisors during power operations. This situation was 18 exacerbated during refueling outages. One shift supervisor reviewed 22 procedure 19 field changes during a dayshift watch, taking a significant portion of his shift.

20 Additionally, the team observed that the shift supervisor was routinely involved  ;

21 in providing the maintenance craft personnel with general information, such as 22 plant status and schedules, that could have been obtained elsewhere. This 23 responsibility left the unit supervisor to monitor the control room and any plant 24 tests or evolutions."'

25 "Although these LCO entries and exits were appropriate, they placed a substantial 26 administrative burden on the operations staff"'

27 Prior to obtaining an operating license for STP, HL&P was required to file a Final Safety Analysis 28 Report (FSAR). In the FS AR, HL&P described an organizational position called Administrative T

' DET Report, p. 5.

2 DET Report, p. 6.

' DET Report, p. 6.

O.

• DET Report, p. 7.

Page 11-3 The Libertv Consulang Group

Ri ? .'

Chapter Two - Operations

[D 1 .-lide The purpose of this position was to relieve the shift supervisor of administrative burdens 2 through delegation of non-safety duties STP's nuclear plant organization policy described the 3 position The FSAR stated that "[a]n Administrative Aide has been assigned to perform routine 4 administrative duties and processes such as routing records, logs and correspondence for the 5 Control Room Operations staff as required."'

c The BTRC's evaluation of STP's FSAR is contained in a document called the Safety Evaluation ,

7 'epott (SER). The SER for STP said: "Each shift will have administrative aides to relieve the shift 8 supencisor of routine administrative duties and to process and route various records, logs and 9 correspondence "6 10 Initially, the administrative aide was used on all shifts on Unit 1. As the startup activities began 1I to peak on Unit 2, additional administrative aides were employed to handle the work load for both 12 units. Once both units had completed startup testing and begun regular power production, the 13 administrative work load decreased significantly. HL&P concluded that the administrative aide

• 14 was not needed for all shifts, and coverage was reduced to Monday through Friday,6 a m. to 10 15 ptn.'

O 16 During an inspection in November 1990, the NRC noted that STP's practice deviated from the 17 NRC's SER in that the administrative aide was not provided on each shift ' HL&P responded to 18 that inspection report and provided an explanation of why the position was not required on back 19 shifts and weekends. STP procedures were followed to ensure that the deviation from the SER did 20 not involve any unreviewed safety questions, and HL&P formally requested a change to the SER.'

21 The NRC reviewed this matter and in an inspection repon issued in May 1992 said: "The 22 licensee's response to the deviation was reviewed and was determined to be acceptable. The 23 change in shift coverage poses no hazard to plant operations. This deviation is closed.""

' STP Updated Final Safety Analysis Report Section 13.1 2.2.1.

• NRC SER for STP, NUREG-0781 Section 13.51.1.

DET response item #1205.

' NRC Inspection Repon No. 90-36, ST-AE-HL-92631. December 17,1990.

Letters. HL&P to NRC, ST HL AE-3667, Janumy 17,1991 and ST HL-AE-3691, March 8.1991.

r (q) i NRC Inspection Repon No. 92-08 ST AE HL 93077, May 22.1992.

Page II-4 The Liberg Consuhung Group

RL> .'

Chapter Two- Operations t 1 Two years after HL&P's request for a change. in May 1993, the NRC formally approved HL&P's v

2 request for an SER change "

"The staff [NRC Staff] notes that administrative aides relieve the shift supervisor 4 as required. but it is generally not necessary on backshift and weekend time 5 periods when the volume of work is generally low. Removal of the administrative 6 aide requirement for "each shift" is administrative in nature and does not impact 7 - plant safety or operations. The staff agrees that the original safety evaluation is 8 currently unduly restrictive. However, the staff maintains that administrative aide 9 coverage shall be required during the normal work week, which is roughly denned to as day shift Monday through Friday. Additionally, the staff maintains that the 1i administrative aide should be available to accommodate any activities that take 12 place during that portion of the evening shift as necessary to accommodate day 13 shift carryover activities and evening surveillance activities."

14 Liberty examined the NRC inspection reports that covered day-to-day operations over a long 15 period of time to ascertain another view on the administrative burdens on the control room staff.*

16 In an inspection report that covered the period of February 2 through March 14,1992, the h3C 17 said:':

V 18 "The ir.spectors visited the control rooms on a routine basis and veri 6ed that 19 control room stafSng. operator decorum, shift turnover, adherence to TS, and 20 overall personnel performance within the control room was in accordance with 21 NRC requirements."

22 This statement was repeated in inspection reports that covered the periods March 15 through April 23 25,1992, April 26 through June 6,1992, June 7 through July 4,1992, July 5 through August 1, 24 1992, and others."

25 in late 1992 the NRC conducted a special Operational Safety Team Inspection (OSTI) The report 26 from that inspection included the following."

" Letter, NRC to IE&P, Kokajko e Cottle, ST AE-lE-93425, May 24,1993.

" NRC Inspection Report No. 92-05, ST AE lE-93037, April 8,1992, p. 9.

" NRC Inspection Report No. 92-08, ST AE-1E-93077, May 22.1992, p.10; NRC Inspection Report No 92-

14. ST-AE 1E-93122. July 8,1992, pp 6-7; NRC Inspection Report No. 92 21. ST AE lE-93145. August q l1.1992, and NRC Inspection Report No. 92 24, ST-AE-lE-93184. September 10.1992.

NRC Inspection Report No 92-35 ST-AE lE-93325, March 3,1993.

Page 11-5 The Liberty Consultmg Group

RLv:

Chapter Tw o - Operations i "The operators maintained excellent control of equipment status. Equipment 2 clearance orders were well documented and appropriately implemented The 3 operators logs accurately reDected plant evolutions and equipment status.

4 Inoperable safety-related equipment was accurately documented in the operability 5 tracking logs " (page iii) 6 "The team concluded that the plant was staffed by competent, knowledgeable 7 personnel who executed their duties in a professional manner. Several strengths 8 were noted in the area of plant operations Control room decorum and operator 9 professionalism were good. Excellent communication was noted between the to operators and plant personnel. Operator response to alarms was very good.

11 Excellent control of equipment status, including authorization to begin work 12 activities, was observed. The team noted that the recent changes to the corrective 13 action program for the identification and resolution of hardware and programmatic la deficiencies was well de6ned The nuclear review board oversight function was 15 noted to function very effectively."(page 2) 16 These numerous NRC inspections, which occurred over extended periods of time, covered a.

17 variety of operating conditions, and were speci6cally aimed at operational safety, did not observe 18 an excessive administrative burden on the control room staff. Numerous other NRC and IhTO evaluations clearly had the opportunity to comment on any observed administrative distractions

{s 19 20 that may have been present but did not do so. In contrast, the DET only observed activities for a 21 limited period during an unusual two-unit outage. Thus, prior the DET's review, STP had no 22 reason to believe outside evaluators thought the control room operators had excessive 23 administrative tasks Nor had STP's own assessments identified the administrative burden on 24 control room operators as a concern.

25 An additional observation by the DET blamed the operators' administrative burden on the lack 26 of" operational experience" elsewhere in the organization. The DET first said. "The near absence 27 of operational experience outside the operations organization placed an excessive reliance on shift 28 supervisors to screen work packages for safety impact and selection of appropriate post-29 maintenance testing."" The next page of its report used slightly milder language: "The limited 30 operational experience throughout the site organization placed an excessive reliance on the shift 31 supervisor to screen work packages for safety impact and selection of appropriate PMT." As 32 explained below, Liberty concluded that HL&P's approach to staffing and its efforts to 33 incorporate operational experience in its staff were reasonable.

I DET Report, p. 5.

Page 11-6 The bberty Consuhmg Group l

l

o RD .'

Chapter Tw o - Operations

~)

(G 2

1 After the accident at TMI-2 in 1979, the education, training, and abilities required of nuclear plant operators increased The demand for qualified operators rose significantly throughout the industry 3 Those qualified enjoyed a heightened demand for their sersices as on-shift operators The fact that 4 there was not an abundance of reactor operators and senior reactor operators throughout the 5 organization at STP was expected and not unreasonable. Nevertheless, personnel in positions that 6 interfaced with operations on a regular basis, panicularly those that dealt with work packages, had 7 consrderable nuclear experience.

8 .-is shown in the next section of this report, an SRO (Senior Reactor Operator) was rotated to the

9 Work Control Group, an RO (Reactor Operator) was serving as a Senior Outage Coordinator in 10 the Integrated Plannmg Group, and another SRO was serving as the Work Control Interface in 11 Engineering at the time of the DET's evaluation. This last individual had over 20 years of 12 professional experience, had a master's degree in nuclear engineering, and had served as the 13 Manager of Operations Support at STP 6 The Division Manager of the Work Control Center had 14 over 15 years of experience at the time of the DET evaluation, including seven years at other i 15 nuclear plants, and had operator training at a Westinghouse nuclear prototype. He also had worked 16 for General Electric at the following nuclear plants

Vermont Yankee, Monticello, Brunswick.

,O 17 Hatch, Arkansas One, and Grand Gulf." The maintenance manager had over 23 years of V 18 professional experience, had 16 years of nuclear experience, had also worked at a Westinghouse 19 nuclear prototype, and was a graduate of the operator training facility in Zion, Illinois."

20 The four top managers in the Planning & Scheduling Depanment averaged 16 years of 21 professional nuclear experience, including an average of seven years at nuclear facilities other than 22 STP The depanment manager and his six division managers in the Maintenance Department 23 averaged over 20 years of professional experience, including an average of 19 years in the nuclear 24 industry. This experience was gained not only at STP but also at other facilities; on average, 25 nearly 14 of their 19 years of experience were obtained elsewhere."

" DET response item #0020, rdsums of Victor Simonis.

DET response item #0020. job description and rdsums of David Musick.

DET response item #0020, risumd of Thomas Underwood.

O "

\] DET response item *0020.

Page Il 7 The Libern Consulang Group

R1 S .'

Chapter Tw o - Operations 1

1

[%}

1 On the basis of the results of many NRC inspections other evaluations, and the considerable I 2 actual operational experience outside the operations organization, Liberty concluded that HL&P's 3 management of staffing and experience in the organization was reasonable and prudent. l l

4 B. Operations Staff Rotation 5 The DET said that " strained operator staffing had prevented a normal progression of operations 6 personnel into other parts of the site organization ": The DET was given a list of 16 individuals 7 who had been in the Operations Department and who were then serving in key roles in other parts 8 of the organization. Twelve of the 16 had been SRO-licensed, and the other four had been licensed 9 as reactor operators 22 As shown in the table below, there was a regular progression of people 10 moving from operations; four of these individuals moved from operations to other parts of the 11 STP organization in 1990, four in 1991, and six in 1992.

12 In addition, at the time of the DET evaluation, there were six people who were not part of the 13 Operations Department and who were attending licensed operator training. Of these six, two were u in the Engineering Depanment, two were in the Training Department, one was in Nuclear

/~~T i 15 Assurance, and one was in Technical Services.2:

k,j 1

1

" DET Report. p. 6.

" DET r:sponse item #1133.

) l (y 22 DET response item #4018.

Page Il-8 The Liberty Consultmg Group

R1 %:

Chapter 'l u n - Operations A. Date Lett Current Work Former Orierannns M Grn e Iicense

! I 3'87 Manager Nuclear secunts SRO 2 2 I 89 St QA Specialist QA SRO 3 3 4'90 Sr. QA Specialist QA RO 4 4, 6/90 Sr Consult. Eng. Eng - Work ControlInterface SRO 5 5 11/90 Sr. Outage Coordinator Integrated Planning RO 6 6 12/90 Sr Staff Specialist Emergency Planning SRO 7 7 3 91 Sr. Training Instruct Operations Traming SRO 8 8 7/91 Staff Specialist Work Control SRO 9 9. 9,91 Consulting Eng. Spec. Corrective Action SRO 10 10 9/91 Contractor Procedures Enhancement SRO 11 11 1/92 Training Instructor Operations Training SRO -

12 12. 1/92 Training Inst 1uctor Operations Trainmg RO 13 13. 1/92 Training Instructor Operations Training RO Admmistrator Corrective Action SRO

(%)O) 14 14. 2/92 4/92 Sr. Training Instruct. Operations Training SRO 15 15.

16 16. 12/92 Consulting Eng Spec. Licensing SRO 17 The DET did not state the grounds for its conclusion that there was not a " normal" progression 18 into other parts of the organization. At the time of the DET inspection, STP's Operations 19 Department included 37 SROs 23 in 1991 and 1992 over 10 percent of this number transferred to 20 other parts of the STP organization. Given the industry-wide demand for SRO-qualifted 21 individuals, this rate of rotation was significant. In Liberty's view, the facts show that there had 22 been a regular progression, that the individuals were serving in key roles, and that operator 23 training was being given to personnel in other parts of the organization.

I l

i s

,. " DET response item #1098.

Page 11-9 The Liberry Consulnng Group

1 4

A RD :

Chapter Tu o - Operations C%

V) i C. Shutdown From Outside the Control Room 2 In its evaluation of operations staffing, the DET said 3 "The team reviewed the staffing required to mitigate a resource-intensive accident 4 (reactor shutdown outside the control room) and concluded that the existing 5 staffing would be strained to handle such a scenario."2'

~

6 The control rooms of nuclear power plants have multiple and redundant design features to assure 7 that they will be habitable under most any conceivable problem or accident scenario. Nevertheless, 8 the NRC requires that nuclear plants be capable of shutting down under a scenario that involves 9 evacuation of the control roorn STP's FSAR describes the equipment, controls, and 10 instrumentation that would be used in the unlikely event that the plant had to be placed in a safe 11 shutdown condition from outside the control room? It also provides an analysis of a shutdown 12 from outside the control room 26 13 The NRC reviewed and approved the STP design and analysis In 1986, the NRC's evaluation 4 was?

\

15 "If temporary evacuation of the control room is required because of some 16 abnorTnal station condition, the operators can establish and maintain the station in 17 a hot standby condition from outside the control room through the use of controls 18 and indicators located at the auxiliary shutdown panel (ASP), transfer switch 19 panels, and other local control stations."

20 The NRC specifies the minimum shift crew composition for STP in the Technical Specifications?

21 STP's staffing has always exceeded the NRC minimum because STP maintains a separate shift 22 supervisor for each unit rather than just one for both units.

23 In addition to its review of the design and staffing, the NRC required STP to perform both pre-24 operational and initial startup testing to demonstrate that it was able to establish and maintain safe 3'

DET Report, p. 8.

" SIT' Updated Final Safety Analysis Report. Section 7.4.19.

• STP Updated Final Safety Analysis Report. Section 7.4.2.7.

" NRC SER for STP. NUREG-0781, Section 7 4.12.

f']

's / I V 2' STP Technical Specifications. Table 6 2-1. j Page tI-10 The Liberry Consulnng Grcup l

RI .s .'

Chapter Tw o - Operationi n

(v) I shutdown conditions from outside the control room 3 HL&P performed that testing and the NRC 2

2 reviewed the test procedures and results 3 An accident requiring a shutdown from outside the control room is indeed a resource-intensive a event In fact. from the point of view of operator staffing, it may be the most resource-intensive 5 event that any nuclear plant must consider, and it is likely that staffing for such an event would 6 be Strained at any nuclear plant. If on-shift stafSng were more than just adequate to accommodate 7 such a scenario, it might have to be regarded as excessive. Having just enough operators to 8 perform this function was evidence of adequate, cost-effective stafEng. The DET's criticism in 9 this regard did not show that STP's management had made unreasonable staf6ng decisions 10 D. Three-Train Design 11 1. Surveillance Testing .

12 The DET used the fact that STP has a three-train safety-system design to bolster its arguments 13 about operator stafGng In one case the 1 SET said.

14 "The surveillance test program was also a significant resource burden on the

3 control room staffin general and the SROs in particular Each unit has three-trains of safety equipment, thus adding a third more surveillances than the conventional two train design."

18 It is a fact that STP has more safety equipment (50 percent more, not a third) that requires 19 surveillance tests than a typical nuclear plant that has two trains of safety systems However, this 20 fact by itself does not show that surveillance testing was a signincant resource burden on the 21 control room staff. One of the ways in which the DET sought to examine the burden from 22 surveillance testing was to ask HL&P how many surveillances had been overdue or missed within 23 the last three years.32 HL&P's response listed ten missed surveillances. Two of the ten did not 24 merit the submission of LERs. A review of the LERs associated with four others indicated that

" NRC SER for STP, NUREG-0781, Section 7.4.2.2.

STP Updated Final Safety Analysis Repen. Section 14.2.12 2 test 98 and 14 2.12.3 test 25.

" j DET Repon. p. 6.

, I t '

\ " DET request #1017.

Page 11-11 The Liberg Consult:rg Group l

I

. _ _ _ . - - - _ _ _ _ . - . - - - _ _ _ _ _ _ _ - _ - _ _ _ _ - _ _ _l

RLs. ,

Chapter Tw o - Operations

] 1 they were not actually missed surveillances One, for example, was a missed post-maintenance l

2 test. Of the remaining four, one was caused by a misinterpretation of the Technical Specifications 3 issued by the NRC, one was a 1990 event in which a daily power channel calibration was 4 performed 38 minutes late, one was a 1991 event in which a chemistry sample was taken 55 [

5 minutes late, and one was a 1992 event in which ar equipment failure prevented a regular channel  !

6 check. Thus, missed or overdue surveillances were not the result of any excessive suneillance  ;

7 testirag burden This conclusion had also been reached and documented by the NRC in the SALP  !

8 report issued just prior to the DET evaluation: "The missed surveillance rate was extremely low "" i i

9 The other way in which the DET tried to get quantitative information to suppon the surveillance-  :

10 burden hypothesis was to request a breakdown of the opr. rations staffing required to perform the ,

1I more complex logic and system surveillance tests." HL&P provided a list of 13 sun eillances that 4 12 required multiple reactor operations personnel. Two of the 13 would only be performed during l 13 or before core alterations (i.e., refueling). Another was only performed during stanup prior to l

• f 14 entenng Mode 2. The remaining ten tests were periodic; eight were performed quarterly and two 15 were performed monthly. The following table is a summary of those ten periodic tests. f i

t 16 Periodic Tests Requiring Multiple Operations Personnel t Tests Total Mhrs/ Test Mhrs1 SRO  :

17 Frecuenev Ed SED.1 EQs Erds Isti Lsch Memh Mrs /Wnth l 18 1. Quarterly 3 1 3 8 50 6 50 6 [

2. Quanerly 4 15 3 20 4 19 1 3 5 f

20 3. Quarterly 4 1 4 5 15 3 20 4 21 4. Quanerly 3 1 3 6 18 3 18 3 f

22 5. Quanerly 1 1 1 6 14 2 5 I s 23 6. Quarterly 3 1 2 4 16 4 16 4 24 7. Quarterly 3 1 1 4 12 3 12 3 f

25 8. Quanerly 3 0 1 7 56 8 56 0 26 9 Monthly 1 1 2 6 36 6 36 6 27 10. Monthly 1 1 2 6 36 6 36 i Total: 37 28 With two SROs on watch per unit at STP, there are 1,460 SRO-hours per month. The average 37  !

29 SRO-hours spent on these large logic and system surveillances represent only about 2 5 percent

- t

" NRC SALP Report. ST AE-HL-93239, November 18,1992,p.I1.  ;

l

" DET tequest #1073  ;

Page U-12 The Laherry Consulting Group ,

i 1

1 10.sl Chapter Tu o - Operatium I

of that total There are, of course, many other suneillance procedures that are performed

regularly However, the listed tests are the ones that would most likely demand an SRO's 3

attention On the basis of the data provided to the DET and prior NRC evaluations, Liberty a determined that there was no reasonable basis for STP's management to take action to address a 5 resource burden on the control room staff due to sun eillance testing.

6 2. Limiting Conditions for Operation 7

The Technical Specifications specify Limiting Conditions for Operation fLCO) A typical LCO 8 for STP is that "[ajt least three independent component cooling water loops shall be OPERABLE 9

With only two component cooling water loops OPERABLE, restore at least three loops to 10 OPERABLE status within 72 hours8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> or be in at least HOT STANDBY within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and 1I in COLD SHUTDOWN within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br /> "" This means that if anything is wrong 12 with a component in a cooling water loop, or if something is being tested or repaired in one of thq 13 loops, the reactor must be shut down unless the test or repair is completed within a few hours. The 14 Technical Specifications contain many LCOs. One of the responsibilities of the operators is to 15 keep track of these matters so that none of the many requirements in the Technical Specifications b 16 Lc violated.

17 The DET report included the following?

18 "The three-train design requirements and the history of material condition 19 problems frequently prompted the controi room staff to cause the plant to enter 20 limiting conditions for operation (LCO), For a 2-year period, ending February 21 1993, Unit 1 averaged 19 LCO entries each day while Unit 2 averaged 26 entries 22 each day. These figures did not include entries into an LCO for surveillances of ,

23 less than 8-hour duration. Although these LCO entries and exits were appropriate, 1 24 they placed a substantial administrative burden on the operations staff. On the basis 25 of a request by the team, the licensee performed a survey and concluded that the 26 plant entered LCOs at a rate greater than four times that of similar facilities." ,

l 27 The operations group had a computer system that helped keep track of the operability of the 28 plant's equipment. (Refer to the section below called " Support to Operations - Use of Computer 29 Information Systems.") HL&P provided the DET with a report from this system showing the i

STP Technical Specifications. 3/4 7-12.

(

DET Repon, p. 7.

Page 11-13 The Libmv Consuhmg Group

RL.%.

Ch tpter Tw o - Operations U 1 number of active entries (operability tracking tog entries. or OTLs) for each unit on a daily basis 2 for a 122-week period starting January 1.1991 These data are shown graphically below-  ;

1 3 Average Daily OTL Entries  !

i l

l 100 Urat 1 'k Refueling fi Udt2 3 go __ . . ._.Refueks .. ..

' i i '

U 1 I

,' , _# 65 t s \

so .... . .

.. ..l.. ..;. . . .. .. .l )

8 i

' 1 3

1 8

t  !

40 -" "l" I s i

I t I i ,

20 * ' ' " " '

f'"' **l" '

"T" * " ' ' '

I p /  ! L\ /1 4 D \,//' % g ,,/~ ,.,*g r, O o lllllllllllllllllllllllllllllllll 'llllllllllll 'llllllllllllllllllllllllllllllllllllllllllllllllllll':l!llllll.Llll Jan-91 May-91 Oct-91 Feb 92 Ju!-92 Dec-92

- Unit 1 - " Unit 2 4 Any analysis of the number of active OTLs should consider outage and non-outage periods. At 5 STP, or at other plants, there is a great disparity between the number of OTL entries made during 6 periods of significant maintenance and the number made during other periods. The actual daily 7 average number of active OTLs (which the DET equated to LCO entries) during the 122-week 8 period of the data were 21.1 for Unit I and 15.4 for Unit 2. During periods other than refueling 9 outages, the daily averages were 10.7 for Unit I (over 90 weeks) and 9.2 for Unit 2 (over 104 10 weeks).

. 11 The number of active OTLs could be used for some kind of trend analysis but is meaningless as 12 an absolute quantity unless put into perspective. To gain this perspective, the DET asked HL&P 1  ;

k [

! " DET response item W 1066.

Page U 14 The Libertv Consulting Group

l 10.ul Chapter Two - Operations

'y) I whether STP's number was high or low compared to peer group plants? HL&P responded as 2 follows 3 "DET information request #1066 2nd part in response to how we compare to peer a group plants on the number of LCOs entered. Only a small amount of comparable 5 data could be obtained Either they would not or could not give us the data .

6 requested Three (3) plants gave us "ballpark" numbers However, they do not 7 track LCO entries the same way we do. Plant #1 -2000/ unit / year Plant =2 -

8 1750/ unit / year. Plant #3 -315/ unit / year."

9 On the basis of this information, the DET reported that HL&P had concluded that STP's rate was to greater than four times that of similar facilities. There were only three data points for comparison.

11 The value of one was over six times that of another. The data were described not only as 12 "ballpark" but also as non-comparable because "they do not track LCO entries the same way we 13 do " The lack of comparable data renders this comparison virtually useless.

14 The DET concluded that the three-train design and material condition of STP had caused frequent 15 LCO entries As to the three-train design causing more LCO entries, it was a logical assumption.

16 The NRC's requirements for two-train plants are very similar to the typical STP LCO stated s 17 above, except that those plants would be required to maintain two loops of cooling water operable is instead of three. With more equipment it is obvious that there will be more testing and more 19 equipment malfunClions. Therefore, while the STP design provides an inherent safety and 20 reliability advantage, the Technical Specifications require STP to keep track of more equipment.

21 As to the second point, if a plant's material condition were poor, one would expect an increased 22 number of LCO entries. In STP's case, however, even the criticisms about its material condition 23 center on the secondary (BOP) side of the plant. This equipment generally would not cause LCO 24 entries because the BOP does not contain safety related equipment subject to the Technical 25 Specifications.

26 The DET's conclusion about STP's LCO entries being more than four times more numerous than 27 those at other plants seems insupportable. If a two-train plant had 4 entries per day (and other 28 factors being equal), STP's three-train design would be expected to cause 50 percent more LCO 29 entries, or 6 per day. Under the reasoning of the DET, the remaining difference between these 6 30 entries and STP's supposed rate of 16 (four times the other plants) would have to be caused by 31 a difference in material condition. There simply is no evidence, however, that the material b "

DET request #1066.

Page 11-15 The Liberry Consultmg Group

RL%'

Chapter Tw o - Operations I condition of STP's safety related equipment was any worse, let alone worse enough to cause this 2 magnitude of a difference, than that of other plants . l 3 Liberty found that the information available would not support a conclusion that staffing decisions 4 had been unreasonable because of LCO entries causing a substantial administrative burden on the 5 control room staff. STP kept detailed records of these entries, and, during periods other than 6 refueling outages, each unit averaged less than one every two hours. As expected, during  !

t, 7 maintenance periods (like the one in which the DET's evaluation was conducted), the numbers 8 were much higher. Although STP probably did have more LCO entries than other plants because 9 of the combination of its three-train design and the Technical Specifications, it is very unlikely 10 that STP's rate was several times higher than that of ccmparable plants.

i li E. Dual-Unit Outage and Operations Overtime 4 4

12 The DET conducted its review during a period when both units at STP were shut down. HL&P 13 does not, nor should it, base staffing on'the requirements of a dual-unit shutdown. Like all other h 14 15 two-unit nuclear plants, STP plans its outages so that only one unit is shut down at a time. When, due to unforeseen or unplanned circumstances, both units are shut down, the use of personnel must 16 be adjusted so that the existing staff can meet requirements. These adjustments typically take the 17 form of changes in working shifts and changes in support activities like training. It would be ,

18 unreasonable for HL&P to plan its staffing so that in the event of a dual-unit outage, there would 19 be no disruptions to training, work shifts, or the use of overtime. Any evaluation of the {

20 reasonableness of staffing levels must recognize the uncommon nature of the dual-unit outage i

21 The DET stated?

22 "The licensee funher strained staffing levels for the non-licensed reactor plant i

23 operators (RPOs) by implementing 12-hour shifts without margin above the 24 administrative staffing limit of 4 each shift. Thus, any delay in an RPO reporting 25 to work resulted in holding one of the onshift RPOs over past the normal 12-hour 26 shift and therefore, on occasions, exceeding the technical specification (TS) l 27 ovenime guidelines. Since January 1993, operators exceeded the TS guidelines for i 28 overtime on several occasions."  !

" DET Repon, p 7.

Page H.16 The Laberry consulting Group

_ _ - . l

l l

RD. l Chapter Tu - Operationi s

V 1 During its review, the DET requested any documentation that showed that operations staffing was 2 evaluated as a potential root cause of ovenime violations

• HL&P's response stated that since 3 January 1991 there had been seven problem reports associated with overtime violations. Only one 4 of the seven dealt with the Operations Department. That one was not even reponable to the NRC 5 The full text of that one problem repon was as follows.

6 - "At 1245 on 3/31/93 while reviewing RO timesheets it was noticed that 2 RO's 7 worked 13 hours1.50463e-4 days <br />0.00361 hours <br />2.149471e-5 weeks <br />4.9465e-6 months <br /> on 3/19/93 This caused them to exceed the tech spec limit on 8 work hours (24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> in a 48 hour5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> period) Some of this time may be turnover time 9 [that time is excluded from T/S requirements] but first review shows that the T/S lo limit was exceeded. Cause of the event was their reliefs did not show up on time 11 due to a schedule change, but came in one hour late "

12 A further examination of the use of overtime showed that HL&P had actively dealt with the 13 situation In May 1991, the NRC issued a Notice of Violation to HL&P because during February 14 and March of that year shift supervisors had exceeded the Technical Specification limit of 72 15 hours in a 7-day period without authorization of the Plant Manager HL&P's response to the 16 notice stated that in one case the necessary approvals had been obtained but the documentation 17 could not be located, and in the other case two shift supervisors had worked 80 hours9.259259e-4 days <br />0.0222 hours <br />1.322751e-4 weeks <br />3.044e-5 months <br /> in a 7-day 18 period because of an unforeseen emergency. Nevertheless, HL&P instituted additional 19 administrative Controls to assure that similar occurrences would not happen in the future 20 About a year later, the NRC followed up on the general concern about the amount of overtime 21 used during outages by both maintenance and operations personnel. The NRC's repon on that 22 follow-up gave a balanced discussion of HL&P's actions and the results achieved, and included 23 the following '

DET request #1167.

,]

Letter, HL&P to NRC, ST HL-AE-3792, June 20,1991, DET response item #1168.

(v/ '

NRC Inspection Report No. 91 34, ST AE4IL-92991, February 26.1992. p 7.

Page 11 17 The Lsbertv Consultmg Group

5 RLs.'

Chapter Tu o - Operations i

V 1 "[T]he licensee established a goal that scheduled overtime during the recently 2 completed llnit 2 outage would not exceed 50 percent. The inspectors reviewed the 3 actual hours worked during that outage This review indicated that control room 4 operators worked about 40 percent average overtime while nonlicensed operators 5 worked about 50 percent. The maximum average monthly overtime worked by 6 mechanical maintenance wa.; 64 percent for the craft and 72 percent for foremen During the nonoutage period of April through June 1991, maintenance personnel 8

worked less than 7 percent overtime while control room operators averaged less 9 than 15 percent overtime. While the nonoutage overtime is well within accepted to standards, outage overtime (for maintenance personnel] continues to exceed the 11 established goals."

12 Liberty concluded the use of overtime in operations was not an unreasonable management action 13 at STP. Two years before the DET's evaluation, the NRC issued a violation because shift 14 supervisors exceeded overtime limits. HL&P's actions in response to that violation were 15 reasonable and effective. When the DET first came on site, STP discovered that two reactor -

16 operators had exceeded a limit by one hour. This was an isolated case and one that occurred during*

17 a dual-unit shutdown, a situation that must be recognized as very unusual. Adjustments to training 18 and work schedules to accommodate this circumstance were expected and reasonable. During 19 every year ofits operations, STP had more licensed operators than the prior year, and prior to the ,

20 DET evaluation, the NRC had never found an operations staffing shortage. Therefore, STP's s 21 management had no reason to believe that action was required to aviod a determination that there 22 was an operator staffmg shortage.

1 i

23 F. Independent Evaluations of Operations Staffing i

24 STP's management regularly sought outside assistance in evaluating the size and capability of the l 25 STP staff. HL&P contracted for external, independent staffing studies for STP in 1989,1991,and 26 1992. The first two studies were performed by Tim Martin & Associates. In these studies, the STP l 27 staff were categorized into 45 work functions or functional groups and compared to inodels based l 28 on staffmg data from other nuclear plants. STP-specific characteristics such as unique work 29 activities and the site size and arrangements were taken into account. For operations, the fact that 30 STP has a third train of safety equipment, and the number of staff necessary to operate the 31 secondary plant, were specifically factored into the analysis.

Page H-18 The Libery Consulnng Group

I RLs.'

Chapter Two - Operations m

1 The 1989 Martin study showed that the operations function contained 166 personnel, whereas the 2 models and analysis showed an expected staff size of 150 for that function " The 1991 study 3 compared the actual operations function staff of 185 with an expected staff of 184." The 1991 1

4 study noted that while the current staf6ng in operations was close to the expected, the plans to add 5 apprentices would raise staffing above the expected number.

6 HL&P decided to conduct another stafimg study in the spring of 1992 This study was performed 7 by ASTA, Inc. during the period June 8 to September 25,1992 " The intent of this study was to 8 focus on the actual needs of STP instead of comparisons. For operations, this study considered the 9 unit staffs and shift crews, the training pipeline, the corrective action section, the technical section, 10 and the training section. It determined the staff number that would be required to perform these 11 operations functions for the long term in accordance with Technical Speci6 cations and industry 12 standards 13 The DET requested that HL&P compare the Februrv 1,1993 actual staf6ng at STP with the level 14 determined to be appropriate by the ASTA study. Actua; staf6ng was 140 compared to the ASTA 15 number of 141. (Note that these 62ures are not comparable to the figures obtained from the 1989 Q

i ,

16 and 1991 studies since ASTA used an organization-based approach and Martin used a function-

%/

17 based approach ) The DET made specific reference to these studies. "The decision to have several 18 station staffing studies conducted by outside consultants indicated senior managements' concern 19 over appropriate staffing levels."" In the context of the DET report, it could be implied that 20 management was concemed about inadequate stafUng levels. However, Liberty found no evidence 21 to support this implication. The facts are that three thorough studies of staf6ng showed that 22 operations was manned at the appropriate and expected levels. Reasonable management must be 23 concerned about assuring safe .md reliable operation of the plants but must also have some

4 concern for holding the line on costs. Liberty concluded that the results of the staffing studies 25 combined with the results of the many evaluations of operator performance demonstrated 26 reasonable management on the part ofIE&P.

" Nuclear Group Steady State Staffing Analysis, Tim D. Manin & Associates, Inc., May 5,1989.

" Staffing Analysis Report for Souc Texas Project. Tim D. Martin & Associates. Inc.. February 19.1991.

DET response item #0067.

" South Texas Project Staffmg Level Study. AST A. Inc. for Houston Lighting & Power Company, June 8 to

/' September 25,1992. DET response item #0066.

k "

\ DET Report, p. 41.

Page 11-19 The Liberry Consultmg Group

M.%:

Chapter Tw o - Operationi

! \

i f v

i III. Support to Operations 2 A. Introduction 3 The DET concluded that "[p]oor support to operations was adversely impacting their capability 4 to safely operate the plant " The DET discussed three areas in support of this conclusion. These 5 were.(1) longstanding design and equipment problems that contributed to operator workload, (2) 6 inadequate computer information systems, and (3) revisions to the plant's Technical 7 Speci6 cations 8 Concerning the Grst of these three areas, the DET cited three examples of"de6ciencies" that it 9 believed had contributed to the operator workload. The 6rst example concemed surveillance to testing equipment and is discussed in this subsection. The second concemed solenoid-operated 11 valves and is discussed in the section on engineering's resolution of plant problems in the chapter 12 on engineering. The third example concerned inoperable automatic controls and is discussed in 13 subsection B below. The second and third areas mentioned by the DET, computer information 14 systems and Technical Speci6 cations, are covered in subsections C and D, respectively, below.

G 15 The first example of an alleged de6ciency that contributed to the operator workload was:"

16 "The absence of permanently-installed Gow measuring devices required the use of 17 temporary test instrumentation to support routine pump Dow surveillances in 18 safety-related systems such as the essential chilled water, auxiliary feedwater, 19 RHR, and spent fuel cooling systems. Extended surveillance setup times had been 20 necessary to obtain accurate and meaningful surveillance results "

21 The DET asked HL&P to determine the number of manhours that technicians had spent installing 22 and removing the equipment and gauges used for surveillance testing the auxiliary feedwater, 23 spent fuel pool cooling, and essential chillers in all of 1992. HL&P infonned the DET that 776 24 manhours had been spent on these activities on both units in 1992." Liberty concluded that this l 25 amount of time, which is less than one-half of a person-year (for a considerable number of tests, 26 some of which were performed as often as monthly), did not indicate that test setup time was such DET Report, p. 8.

(n) v

" DET Report p. 8.

DET response item #2253.

Page H-20 The Liberty Consuhing Group

RD .'

Chapter Tu u- Operations n

k I a significant contributor to operator workload that the only reasonable course of action by STP's 2 management would have been to add staff to absorb this work load-3 B. Automatic Controls a The DET said that "[t]ongstanding design issues and degraded plant equipment contributed 5 significantly to the operator work load " One of the examples used to support this conclusion was e the following 'o 7 " Numerous automatic controls, such as temperature control valves (TCVs) had 8 been inoperable for a signiEcant period of time. Examples included the TCVs in 9 the BOP lube oil coolers, the seal cil coolers, and the hydrogen coolers on the 10 turbine generator These TCVs were oversized and had to be manually throttled, Ii along with the associated bypass valves, in order to control cooling for the various 12 systems." .

13 PROTECTED MATERIAL FOLLOWS O

14 HL&P was fully aware that inoperable automatic controls could adversely affect plant operators' 15 ability to respond to abnormal conditions. In its May 1991 evaluation of STP, INPO had a finding 16 to this effect." In response to the INPO finding, HL&P made the following commitment?

17 "In order to permit more effective system control and monitoring of important 18 parameters, equipment deficiencies such as those noted will be corrected on a 19 priority basis. The station will establish high visibility goals for both inoperable 20 automatic functions and control room instruments out of service "

21 When INPO returned to STP for another evaluation in May 1992, it addressed the finding from 22 the previous year, The evaluation was included in Appendix I to the INPO report.

" DET Report, p. 8 p)

" INPO Evaluadon of South Texas Project Electric Generating Station. May 1991 p. 25.

tb- " INPO Evaluation of South Texas Project Electric Generating Station. May 1991. p 27.

Page 11-21 The Lsberty Consulting Group

d RL%:

Chapter Tw o - Operatiom ,

l ' Appendix ! is a listing of 6ndings from previous evaluations where corrective 2 actions hase not been completed, but are progressmg on a reasonable schedule. A 3 current status, as determined by the WPO team, is also provided ""

4 In its " current status" repon on the finding, INPO said "

5 "Although progress has been made toward reducing the amount of compensatory 6 actions the operators must take, several of these actions must still be taken 7 ~ During May 1992, the total number ofinoperable automatic functions on Unit I

, 8 was 17 and on Unit 2,21 The total number of main control board denciencies on 9 Unit I was 39 and on Unit 2,29. One year ago, approximately 250 control board

10 de6ciencies were identiDed and progress continues to be made Several items are 11 scheduled for completion during each unit's next scheduled outage "

12 END OF PROTECTED MATERLAL 13 In August 1991, HL&P established the high visibility goals for inoperable automatic functions that 14 it had discussed with INPO. Starting with that month's Station Report, a separate page identified *

~,

15 the nur.ber of these functions and displayed the year-end goal. The following graph shows the numbers of these functions, as stated in the information given to the DET "

16 h

v

" INPO Evaluation of South Texas Project Electric Generatmg Station. May 1992. Appendix 1, p 1.

,/m INPO Evaluation of South Texas Project Electric Generating Station. May 1992, Appendix 1 p. 4

" Stanon Reports, August 1991 through February 1993. DET response item #0018.

Page H-22 The Libern Consuhing Group

RLS.:

Chapter Two - Operations 1 Inoperable Automatic Functions 1 n .

i i

U!-  ;

e.0 H ,

i >

l- _

. I I n .

1 n l 3lll1t.l ...  ;

s o

Aug-9 3 Oct 91 llll..l-Dec.91 Feb-92 l.' nit !

A pr.92 Unit 2 h 92 Aug.92 Common Oct 92 Dec 92 ll Feb-9) 2 HL&P established year-end 1991 goals ofless than or equal to 20,15, and 12 for inoperable 3 functions for Unit 1 Unit 2, and common, respectively. For 1992, the year-end goals were made 4 more aggressive and changed to 10,20, and 10. During the period from August 1991 through 5 February 1993, the total number of inoperable automatic functions went from a high of 81 in 6 January 1992 to a low of 24 in October and November of 1992. The direction of the overall trend 7 was clearly downward. The increase seen on the graph in January 1993 was not caused by a 8 degraded material condition but rather was the result of a review of outstanding senice requests 9 to identify any that could affect automatic functions. While the DET indicated that certain 10 automatic functions were inoperable, there had been clear progress and significant management 11 attention in this area. Given the fact that the plant could and in fact was operated reliably with 12 these functions operated manually, Liberty found that management took reasonable actions in this 13 area.

Page 11-23 The Liberry Consultmg Group l

l

lu S-:

Chapter Tw o - Operationi p

i The DET indicated that some automatic functions had been inoperable for "a significant period 2 of time ' A detailed review of the data led Liberty to a different conclusion In response to the 3 DET, STP's operations personnel identified .ne "most signi6 cant" inoperable automatic functions a This listing included 3 Unit i functions,16 Unit 2 functions, and I function common to both units 5 This distribution was as expected, since Unit I had just come out of a refueling outage in late 6 1992, and Unit 2 had been operating nearly continuously for over a year and was scheduled for 7 an outage in spring 1993 The average age of the four Unit I and common inoperable automatic 8 functions was two months. Because ofits extended operating nm, the average age of the 16 Unit 9 2 inoperable automatic functions was 9% months. While two of the Unit 2 functions had been 10 identified almost two years earlier, most had been identified since the last Unit 2 refueling outage 11 At the time of the status report on these items that was given to the DET, all except one were 12 scheduled to be resolved within the next two months. The exception had just been identified the 13 day before the status repon was prepared."

14 The tables at the end of this section show the data used in the above graph of inoperable automatic 15 functions and the status of the most significant operator inoperable functions given to the DET.

t 16 Inoperable automatic functions had been a concern at STP. However, the situation had been 17 identified before the DET's review and was receiving considerable management attention. STP 18 was making good progress toward reducing the number ofinoperable automatic functions. Most 19 of the automatic functions that were inoperable at the time of the DET's review had been present 20 only since the last major outage of each unit. These items were scheduled to be resolved in the 21 near future.

l 1

'M Q(' "

DET response items nos.1092 and 2069.

Page11-24 The Libery Consulting Group 1

I l

l RLs .'

Chapter Tw o - Operation, (m

\ l Operations Inoperable Automatic Functions 2 Ment Dull M Common M j i -\ugu3t 1991 24 35 16 75 l 4 September 1991 21 il 14 66 5 October 1991 24 22 11 57 6 Nosember 1991 18 19 10 47 7 December 1991 17 26 10 $3 8 January 1992 24 47 10 81 9 February 1992 16 40 10 66 10 March 1992 21 24 5 50 11 April 1992 17 23 0 40 12 May 1992 13 21 0 34 13 June 1992 14 16 0 30 14 July 1992 14 15 0 29 15 August 1992 II 21 0 32 16 September 1992 10 18 0 28 17 October 1992 7 17 0 24 18 November 1992 4 20 0 24 19 December 1992 5 20 0 25 20 January 1993 12 27 0 39 ,

21 Februarv 1993 18 22 0 40 l

.f -

)

I Q i I

1 Page 11-25 i The Lsbertv Consulting Group

!U. A..'

Chapter Tw n - ope rathirii (V 1 Status of Most Significant inoperable Functions (as of April 1.1993)

&heduled AR 8 Des. Ace <mne ' ComMetinn 3 Unit 2:

4 1 06l' Ntar 31. 93 o ..

5 158186 Jan 28. 93 21 Nia) 3n. 93 6 166451 Niay 27. 92 10 2 Apr 28. 93 7 160152 Feb 15,92 13 5 Apr 27. 93 x l'3479 Dec 28,92 3.1 Niay 19. 93 9 134854 Jan 6,92 14 8 Apr 16. 93 to 189340 Jan 14,93 2.5 Niay 27,93 Il 130646 Apr 19,91 23 4 Apr 24,93 12 189396 Nov 19,92 44 Apr 17,93 13 166694 Jun 29,92 9.1 Niay 17,93 14 146938 Feb li 92 13 6 htay 3. 93 O. 15 166153 Sep 17,92 64 hiay 9. 93 16 170837 Jan 8. 93 2.7 Niay 20,93 17 146936 Feb 12,92 13 6 Nfay 3,93 18 150280 Apr 22,91 23 3 Niay 13. 93 19 166260 Aug 26,92 72 N1ay 4. 93 Average: 9.4 20 Unit 1 & Common:

21 182470 Jan 8,93 2.7 Apr 1,93 22 179173 Feb 2,93 1.9 Apr 1,93 23 170662 Feb 12,93 1.7 -

24 179452 Feb 9,93 1.7 Apr 5,93 Average: 2 l

'sL Page 11-26 The Liberty Consulnng Group

RIS:

Chapter Tw o - Operationi v i C. Use of Computer Information Systems 2 The DET reported that the operations group at STP had developed computer information systems 3 to aid "in performing such functions as work control, equipment clearances, and reactor coolant a system leak-rate calculations. These systems were originally intended to aid the operators in 3 perfonning these operating functions. However, operators had come to rely on these information 6 systems as more than aids "" One such system was the Operability Tracking Log (OTL), which 7 helped to keep track of equipment operability and the resulting actions that must be taken by the 8 operators to comply with the plant's Technical Specifications. Since the system was intended to 9 provide information only, it did not have all of the controls found in computer software with an 10 intended safety-related operational function. Nevenheless, the use to which these systems were 11 placed represented a natural and helpful evolution in the use of technology available to the 12 operators.

13 While the DET also reported that " problems" had resuitec from the use of these systems, Liberty 14 found that only one minor incident had occurred that was blamed, in part, on the OTL. As a result 15 of that incident, which had no safety significance, HL&P suspended the use of the OTL pending 16 enhancement of the computer program."

17 The OTL had been a helpful source ofinformation during the Unit I refueling outage in the fall 18 of 1992. During the two-unit outage in March 1993, the OTL for Unit 2 was being maintained 19 outside the control area because of the plant status (shutdown and in the refueling mode) and the 20 amount of activity associated with the Unit 2 refueling outage. The system was being used and 21 maintained outside the control area to limit the administrative burden on the control room 22 operators. It is necessary to examine the incident that caused the suspension of the use of the OTL 23 in order to appreciate (1) the complex set of rules that the operators must comply with, (2) the lack 24 of safety significance of the incident itself, and (3) the aggressive actions that HL&P took to 25 assure that the causes of the incident did not result in any situations that could have had 26 significance.

] " DET Report, p. 9.

[V " LER 93-007, Unit 2, ST HL-AE-4405, April 7,1993.

Page 11-27 The Liberry Consultmg Group

l l l  !

RD-:

Chapter Two - Operations I r

1 The DET reponed "

)

l 2 " Problems resulted from operators using these uncontrolled computer information i 3 systems. For uample, a problem with the Operability Tracking Log (OTL) 4 software contributed to an event on h! arch 10, 1993, when the L' nit 2 heating.

5 ventilation, and air conditioning (HVAC) system was found to be incorrectly 6 aligned The OTL program tracked equipment operability to aid the operators in deciding on the limiting and applicable TS action requirements, based on 8 equipment operability. Due to an error in the OTL software report program, and

, 9 because the operators were inappropriately relying on the information in the 10 computer report, the operators were not alerted to the proper system lineup i li configuration for the HVAC system. Consequently, the operators did not 12 appropriately align the HVAC system "

13 On NIarch 3, one of Unit 2's three independent trains of control room heating, ventilation, and air-14 conditioning was declared inoperable so that planned maintenance could be performed. The 15 efTective time of the inoperability was 1:30 a m. STT)'s Technical Specifications require, for those 16 plant conditions, that either the inoperable train must be restored to an operable status within seven

^

17 days or the remaining trains must be placed in the recirculation and makeup air filtration mode '

18 This required action s ; placed in the OTL system with an action date of hiarch 10 at 130 a m.

19 Later on N1 arch 3, one of the two toxic gas monitors for the control room of Unit 2 was taken out 20 of service to accomplish a planned modification. The Technical Specifications required that with 21 one monitor inoperable, the operability needed to be restored within seven days or the control 22 room ventilation system had to be placed in the recirculation mode of operation. An appropriate 23 entry in the OTL was also made for this limiting condition.

24 On Afarch 5, the other toxic gas monitor was taken out of service to perform the same 25 modification. The Technical Specifications for this condition (i.e., both monitors inoperable) 26 required that the control room ventilation system be maintained in the recirculation mode of 27 operation.62 The ventilation system was placed in the recirculation mode of operation to comply 28 with this requirement. Once again the OTL was updated.

" DET Repon, p. 9.

STP Technical Specifications,3.7.7 a-f i

STP Technical Specifications,3.3.3.7 a.

'2 STP Technical Specifications. 3.3.3.7.b.

Page 11-28 The Liberry Consuhmg Group

RLs:

Chapter Two - Operations O

I Dunng the shift turnover on the night of March 9. the operations crew noted that the ventilation 2 system was in recirculation because of the ongoing modi 6 cations to the toxic gas monitors. A 3 printout from the OTL was used during shift turnover That printout did not indicate that any 4 Technical Speci6 cation action statements would be required of the oncoming shift The OTL 5 listed numerous outstanding items, but the oncoming crew did not recognize that one of those 6 items should have required an action Prior to 1:30 a m. the ventilation system should have been 7 changed from recirculation tofiltered recirculation to comply with the Technical Speci6 cation 8 pertaining to the inoperable train of the ventilation system About 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> later, it was recognized 9 that the ventilation system was not in the required mode. The remaining two operable trains were 1

to then placed in the Gltered recirculation mode of operation.')

11 The Technical Speci6 cations appear to have competing requirements. While the inoperable toxic 12 gas monitors required one mode of operation, the inoperable train of ventilation required another.

13 Operations personnel pointed out thi case of competing requirements to the DET."

l 14 In its initial reporting of this matter, he NRC said:"The consequences of this event had little p 15 safety signi6cance since the CRE [mmol sco~ envelope) HVAC was already in the recirculation

( 16 mode of operation and no s'lety d?ars, n. ' as chemical, radioactive, or smoke release, 17 occurred during the time th)f the systr:m _s required [to] be in 61tered recirculation.""

18 Nevertheless, HL&P suspendd use of the computer generated OTL index, maintained a manual 19 OTL within the control area, reinforced requirements concerning the control room ventilation 20 systems in operator requalincation training, and committed itself to review other computer 21 systems for similar dif6culties "

22 HL&P recognized that improvements were needed in STP's overall information systems and had l 23 plans in effect to bring about that improvement. The DET said that the support operations was 24 getting from other organizations was " adversely impacting their capability to safely operate the l

l " LER 93-007, Unit 2, ST-HL AE-4405, April 7.1993, NRC Inspection Report No. 93-11. ST AE HL 93424, hiay 21.1993, pp. 5 7.

" hiemorandum 1 Sikes to L Rompson (DET), April 5,1993, DET response item #1090.

" NRC Inspection Report No. 93-11 ST AE-lE-93424 hiay 21,1993, p. 7.

" LER 93 007, Unit 2. ST-lE AE-4405, April 7,1993.

" hiaster Operating Plans,1992 and 1993. Also refer to chapter five of this report.

Page H-29 The Liberry Consultmg Group

.. , - . -_ .- . - .. ~ - --

RD-Chapter Two - Operations O1 2 plant."" One of the support areas was computer information systems While it is true that operations developed its own systems to aid its operators, these systems performed the intended

'3 functions An error in a report from one system, combined with a complex set of circumstances 4 and operator failure to keep track of these competing requirements, caused a ventilation system 5 to be misaligned about 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> longer than a 7-day limit. Such an oversight could have just as 6 easily occurred using a manual system or a more controlled computer system. This one event with 7 the OTL did not sustain the conclusion that the use of computer information systems had had an .

8 adverse impact on the capability of the operators. Furthermore, management's response to the 9 event was reasonable.

io D. Technical Specifications ,

11 The final area of criticism by the DET related to support to operations dealt with revisions to 12 STP's Technical Specifications (TS). STP's TS are the primary set of rules for operation and 13 testing of the reactor and safety-related supporting systems. The TS are part of the licenses that 14 the NRC issued for operation of STP Units 1 and 2.

15 The DET report contained the following discussion related to the TS."

16 "The licensee had not aggressively pursued TS revisions to resolve the numerous 17 inconsistencies within the TS at STP. The licensee has written approximately 150 .

18 technical specification interpretations (TSIs) and clarifications (TCIs) to help 19 clarify some of these TS inconsistencies. These TSIs and TCIs were only intended ,

20 to provide short tenn guidance, with the eventual implementation of this guidance 21 to be provided in a more permanent document, such as a TS amendment. However, 22 some of the TSIs have been in effect for over 4 years. Particularly troublesome ,

23 areas regarding TS inconsistencies included the toxic gas and control room HVAC, 24- standby diesel generator (SDG), and RHR systems. Although the licensee has 25 pursued essential TS amendments needed to continue plant operation, the bulk of 26 the licensee's TS improvement effort was deferred with the understanding that a -

27 new standard TS would eliminate many of the TS inconsistencies."

28 These remarks were a criticism of the TS revision process, not of HL&P. In order to understand 29 why the quoted passage should be so viewed, it is necessary to understand the administrative 30 process for issuance of TS.

DET Repon, p. 8.

DET Repon, p. 9.

Page U-30 The Liberry Consuhing Group ~

I RLb .' i Chapter Tw o - Operations

/~N (Jl

\ The NRC started with standard technical speci6 cations applicable to Westinghouse PWRs.

allowed little variance from its standard wording, and issued the TS for STP. HL&P may request 2 [

changes to the TS, but only the NRC issues any changes The only reason that the operators at STP 4 might require interpretations or clari6 cations of the TS would be a lack of clarity or precision in 5 the language in the TS as it pertained to STP's design or methods of operation The NRC used the 6 same language for STP's three-train design that it typically uses for two-train designs The TS are 7 lengthy and complex, and therefore it is reasonable to expect that there may be occasional needs 8 for clari6 canon However, the language of the TS reDect the standardized specifications, and the 9 formality associated with the TS requires that only the NRC approves of6cial changes.

10 STP's Technical Specification interpretations (TSI) do not change TS requirements or wording ii The purpose of a TSI is to help assure consistent application among the operators, eliminate any 12 potential for confusion or error, and help clarify how the TS relate to STP's design basis. TSIs 13 provide interpretation of the TS in response to questions from station personnel?

14 Interpretations of the TS can take the form of either a TSI or a Technical Speci6 cation p 15 Clari6 cation. The primary difference between an interpretation and a clarification lies in the

'd 16 internal review process that is used before either is issued. Both are controlled by a procedure that provides for speci6c review and approval."

17 18 The DET asked HL&P to identify areas in the TS that cause confusion or contain conflicting 19 requirements. HL&P provided that information and it was repeated in the DET report. One of 20 those areas was the complex and competing requirements associated with toxic gas monitors and 21 control room ventilation that is discussed in subsection A above?

22 The DET concluded that HL&P had not " aggressively pursued TS revisions "" However, HL&P 23 had requested many changes to the TS, and the NRC had issued 50 revisions since May 1989. A 24 review of those revisions indicated that while some requests were approved within a short period, 25 18 took over 10 months, and 3 took two years from the time HL&P initiated the request until the

l DET response item #4009.

' DET response item #1161.

' DET response item #1090.

V 2 DET Report, p. 9.

Page 113 l The Liberry Consultmg Group 1

10 %:

Chapter Tw o - Operations

(~

( l time that the NRC issued the TS reusion The average response time for the NRC for the 50 1

i 2 revisions was over eight months At the time of the review. one HL&P request for a TS revision 3 was four : ears old and had yet to be acted on by the NRC. The table on the following page shows the dates for HL&P's initiation and the NRC's approval of changes to STP's Technical 3 Speci6 cations 6 The DET also pointed out that the bulk of HL&P's TS improvement efforts were deferred because of the understanding that new standard TS would be issued. However, the NRC had delayed 8 issuance of new standard TS many times.

1 9 Liberty concluded that STP's management had behaved reasonably in the area of Technical to Speci6 cations. STP had requested multiple revisions to the TS so that they would be more 11 specifically applicable to STP. To the extent that the NRC-issued TS caused a higher than 12 necessary work load on the STP operators, that situation cannot be attributable to unreasonable 13 management actions or decisions at STP.

O O

>O V

Page 11-32 The Dbery Consulang Group

l l

RL8-:

fg Chapter Tw o - Operations

, 1 Unit i Urut 2 Duration  !

I H1 A P imtiated NRC Arrrnved Amendment

• Amendment

• Imn* I 2 18 May 1988 24 May 1988 1 .

o2 3 8 Mar.h 1988 1 Nos ember 1988 2 . 7g 4 22 June 1988 18 Nosember 1988 3 .

49 5 7 November 1988 29 December 1988 4 .

1.7 6 17 January 1989 28 March 1989 5 .

2.3 7 25 January 1989 28 March 1989 6 -

20 8 8 March 1988 28 March 1989 7 .

12.7 9 24 February 1988 8 May 1989 8 .

Not TS Change 10 1 June 1989 31 July 1989 9 1 2.0 11 1 June 1989 31 July 1989 10 2 20 12 1 June 1989 5 September 1989 11 -

Not TS Change 13 IE April 1989 15 September 1989 12 3 Not TS Change 14 25 January 1989 3 Apnl1990 13 .

14.2 r

[%/\

\

15 7 March 1990 13 April 1990 14 4 12 16 25 January 1989 9 May 1990 15 5 15 4 17 I hluch 1990 11 June 1990 16 6 34 1

18 25 October 1989 22 June 1990 17 7 79 19 29 November 1989 19 July 1990 18 8 76 20 11 May 1990 31 August 1990 19 9 3.7 21 18 December 1989 14 November 1990 20 10 10 9 22 5 September 1990 4 March 1991 21 11 5.9 23 15 November 1990 11 March 1991 22 12 38 24 22 August 1990 15 April 1991 23 13 7.8 25 15 October 1990 17 May 1991 24 14 70 26 22 February 1991 26 August 1991 25 15 61 27 26 February 1991 26 August 1991 26 16 60 28 15 September 1989 9 September 1991 27 17 23 8 29 12 J%e 1990 18 September 1991 - 18 15 2 30 14 July 1989 26 September 1991 28 19 Not TS Change 31 15 October 1990 26 September 1991 29 20 11.4 V 32 25 September 1991 24 October 1991 30 21 10 Page 1133 The Liberrv Consulting Group 1

i RL% .'

Chapter Tw o - Operations O j, } {l& P Imtiated VRC Aerrns ed I'mt I E Duranon 2 21 December 1990 8 Nos ember 1991 31 22 10 6 3 8 Januarv 1991 13 Februan 1992 32 23 13 2 4 26 August 1991 25 Februarv 1992 33 24 6o 5 15 April 1991 12 hiuch 1992 34 25 to 9 6 30 August 1991 2 April 1992 35 26 7.1

,7 23 August 1991 6 hiay 1992 36 27 84 8 20 hiay 1992 2 June 1992 37 28 04 9 - -- 8 June 1992 38 29 Not TS Change 10 30 August 1991 18 August 1992 39 30 ti6 11 26 Niay 1992 18 August 1992 40 -

2.8 12 12 June 1990 19 August 1992 41 - Not TS Change 13 30 October 1990 21 August 1992 42 31 21.7 14 26 hiay 1992 25 August 1992 43 32 3.0 15 19 June 1992 27 August 1992 44 33 2.3

(

i 16 30 August 1991 5 November 1992 45 34 14 2

\

17 12 December 1990 24 November 1992 46 35 23 4 18 26 June 1991 21 December 1992 47 36 17.9 19 28 September 1992 9 h! arch 1993 48 37 5.3 20 30 August 1991 19 hiarch 1993 49 38 18 6 21 15 April 1991 18 hiay 1993 50 39 25 1 22 14 January 1993 25 hiay 1993 51 40 43 23 29 hiay 1993 29 June 1993 52 41 1.0 24 18 August 1992 12 July 1993 53 42 10 8 25 5 August 1993 4 October 1993 54 43 20 26 12 August 1992 7 October 1992 55 44 1.8 27 1 February 1990 Pending 28 27 hiav 1993 Pend'me i

V Page II-34 The Liberty Consulnng Group

i R[ s.:

Chapter Tu n - Operatinni O)

V i IV. Communications with the Control Room Staff 2 A. Testing of Manual Trip Circuitry 3 1. Introduction a The DET criticized STP management for sending confusing and conflicting guidance to the 5 control room staff The DET report also said that HL&P had attempted to address the issue, but 6 several events had occurred that had " undermined the credibility of site management with the 7 control room staff."' One of the three such events listed in the DET report involved the testing 8 of circuitry used to manually trip the reactor. The DET repon's summary of that matter was: '

9 "The licensee discovered that the Unit 2 reactor trip breaker shunt trips had never to been adequately tested. The breakers were declared inoperable and a TS shutdown 11 action statement was entered without informing or involving the control room 12 personnel that were licensed and responsible for operability decisions. When the 13 control room personnel were finally informed, they were told they had been in the la action statement for over 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> and would therefore need to perform an 15 accelerated shutdown."

t l

D 16 This matter was the subject of a prolonged and detailed NRC inquiry. It was finally closed in l 17 August 1993,15 months after the event, when the NRC's Office of Investigations ruled on l 18 whether HL&P had intentionally violated problem-reponing procedures?'

19 The technical explanation of the failure to test a portion of a control circuit is secondary in 20 importance to the sequence of actions that were taken in response to the discovery. The NRC's 21 concerns did not relate to technical matters. Rather, the NRC's focus was on the timing of 22 communications about the discovery. In a letter dated April 19,1993, exactly eleven months after 23 the incident and two months after the formal announcement of the DET, the NRC notified HL&P 24 that it would be fined 575,000 for violating NRC regulations related to this matter."

DET Report, pp. 9-10.

DET Report, p.10.

, (~N Letter, NRC to IE&P, Milhoan to Cottle, ST.AE lE 93503, August 5,1993.

(

b ~

Letter, NRC to HL&P, Taylor to Cottle, ST-AE-HL 93382, April 19,1993.

Page I1-35 The Liberry Consulting Group

!U.sl Chapter Two - Operationi 1 2. Description 2 A system engineer at STP was performing a periodic review of a surveillance test procedure 3 insolving the manual reactor trip circuitry on hiay 18,1992. He became concerned that a portion 4 of that circuit (i e., specific electrical contacts) possibly had not been individually veri 6ed to 5 operate as designed using the current test procedure. Early the next morning, STP management 6 personnel were informed of the potential inadequate test procedure, and a meeting on the subject 7 was started In the afternoon of Niay 19, the Plant hianager concluded that it was not possible to 8 serify that these contacts had been speci6cally tested. The lack of individual contact testing was 9 applicable to both units and was recognized as being applicable to both units from the beginning.

10 Two telephone conference calls were held with the bRC to inform them of the matter and to 11 request a temporary waiver of compliance with the plant's Technical Specifications During the 12 second call it became apparent that the control rooms of neither unit had been officially notified 13 about the issue. About two hours after the conclusion about the lack of specific testing had been 14 reached, the control rooms were notified and a shutdown of both units was started. Less than one 15 hour later, the NRC approved the temporary waiver, the shutdowns were terminated, and the units C/ 16 were returned to full power." The extensive documentation on this event did not confirm the 17 assertion that the control room staffs needed to perform an " accelerated shutdown "

18 In a letter dated hfay 21,1992, the NRC acknowledged the approval of the temporary waiver 19 request? Shonly thereafter the NRC issued an amendment to the STP operating licenses to permit 20 operation and to require verification of the trip circuit contacts prior to each unit's startup after 21 the first shutdown following hiay 19,19929 Neither of these NRC letters mentioned anything 22 about the circumstances surrounding this matter or, specifically, the timing of control room 23 notification.

24 The NRC evaluated whether STP could have invoked a section of the Technical Specifications 25 that would have given a 24-hour exception to any applicable action statement. The reason for the

' NRC Inspection Report 92 17, ST-AE-HL 93291. January 15,1993. Letter, HL&P to NRC, Hall to Milhoan, ST HL AE 4208, September 11,1992; Letter. HL&P to NRC, Hall to Milhoan, ST HL AE-4217, September 18,1992.

l /m

\

' Letter, NRC to HL&P. Virgilio to Hall, ST AE HL 93069, May 21,1992.

" Lener, NRC to HL&P, Dick to Hall, ST AE HL 93087, June 2,1992.

l Page II-36 The Liberty Consulting Group l

l

Ri %.' l Chapter Tw o - Operations i l

( l 24-hour exception is to balance the risks of uncertainties caused by untested components against 2 the potential for a plant upset caused by a forced shutdown The NRC's evaluation stated "

1 1

3 "In the case of the South Texas event of May 19, 1992, a temporary waiver of 4 compliance was granted. The utilization of the 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> provision of TS 4 0 3 5 would have prevented the initiation of the shutdown of both units. The temporary 6 waiver and the subsequent amendment of the Technical Specification provide 1 7 NRR's evaluation of the safety significance of the inadequately performed 8 surveillance of the manual shunt trip circuitry at South Texas."

9 In July, the NRC issued the Resident inspector's inspection report that covered activities at STP to for the period April 26 through June 6,1992. Relative to the reactor trip circuit incident, that 11 report only said that a special NRC inspection report would address the May 19 event. The 12 Resident Inspector's report made note of the fact that on May 28, the STP Plant Manager had been 13 replaced."2 In fact, the Plant Manager was replaced, in part, because STP's upper management 14 determined that the failure to inform the control room immediately after recognition that portions 15 of an important control circuit had not been tested was not in keeping with management's 16 expectations. There is no evidence to indicate that the NRC, throughout its extended investigation h

\

U 17 of this matter, acknowledged this fairly drastic action that HL&P had taken in response to an 18 admitted error.

19 On August 10, 1992, the NRC requested a management meeting with HL&P to discuss the 20 incident. The NRC also requested that HL&P provide information regarding the timing of events 21 related to discovery of the problem, notincation of the control room, noti 6 cation of the licensing 22 group at STP, and notification of the NRC." The meeting was held on August 28, and HL&P 23 provided written information on September 11 and September 18,1992 " Finally, in January 24 1993, the NRC issued its report on the inspection conducted in May of 1992. It contained a listing )

25 of five " apparent violations" that ranged from the failure to adequately test the trip circuit to the

" NRC Mernorandum. Virgilio to Beach, Response to Request for Assistance Concerning Interpretation of South Tem Project Technical Specification 4 0 3 June 10,1992.

" NRC Inspection Report No. 9214 ST AE HL 93122, July 8,1992.

" Letter, NRC to HL&P, Beach to Hall, ST AE HL 93152, August 10,1992.

A

" Letters, HL&P to NRC, Hall to Milhoan, ST HL-AE-4208, September 11,1992 rad ST-HL AE-4217 iV) September 18,1992.

Page 11-37 The Libern Consulang Group

iD.' .

Chapter Two - Operations A

1 failure to provide complete and accurate information to the NRC " The letter that forwarded the 2 inspection report also scheduled an enforcement conference for the next month. The enforcement

conference was held on h! arch 8,1993."

4 Ultimately, the NRC issued a Notice of Violation on two matters The Grst dealt with the failure 5 to promptly notify the control room, and the second concerned other, related incidents that had 6 occurred in June and September and had to do with the issuance of clari6 cations of Technical 7 Speci6 cations to the control room staff. This Notice included a proposed imposition of a 575,000 8 fine " In August 1993, the NRC sent HL&P the results of"an investigation conducted by the 9 NRC's Office ofInvestigations (01) to determine whether Houston Lighting & Power Company to (HL&P) ofDcials intentionally violated Station Problem Reporting procedures at the South Texas 11 Project Electric Generating Station (STP) on hfay 18-19, 1992. The evidence obtained in this 12 investigation was insuf6cient to conclude that STP procedures were intentionally violated.""

13 3. Conclusions p 14 HL&P admitted that it had been a mistake for the Plant hianager not to inform the control rooms V 15 until two hours after the determination had been made that the trip circuitry had not been fully 16 tested. The Plant hianager felt that the NRC would grant a waiver of compliance and that 17 shutdown of the units would not be required. On this question the Plant hianager was correct. The 18 NRC granted the waiver within a short time period and later issued a change to the operating 19 license. The NRC's interpretation of Technical Speci6 cations later said that STP had actually had 20 a day to pursue this matter, notjust a few hours. However, HL&P's and the NRC's concern did 21 not relate to whether the Plant hianager was correct in believing the NRC would approve of 22 continued operation but rather to whether in cases like these the responsibility for decisions 23 regarding operability and entry into the Technical Speci6 cation Limiting Condition for Operation 24 should rest with the licensed operators on shift.

" NRC Inspection Repon No. 9217, ST-AE EE-93291 January 15,1993.

" Letter, NRC to IE&P, Milhoan to Cottle ST AE lE 93382, April 19,1993.  ;

[^

(

Letter, NRC to HL&P, Milhoan to Cottle, ST AE FE 93382, April 19,1993.

Letter, NRC to IE&P. ST-AE HL-93503, Closure of Investigation Case No. 4 92 018, August 5.1993 Page 11-38 The Liberty Con.sulang Group I

l

l l

l RD-:

Chapter Tu o - Operations t

( l

\

l One cannot deny that the failure to promptly inform the control room may have. at least at first.

2 " undermined the credibility of site management with the control room staff." However, HL&P's 3 response to the incident should have relieved concerns about management, if any existed, along 4 these !ines HL&P's actions included (1) development of specific instructions for informing the 5 shift supervisors as soon as possible on potential operability problems, (2) development of formal )

6 policies and procedures dealing with actions to be taken when unresolved problems are brought  !

7 to the attention of the control room, management expectations regarding operability 8 determinations, and clarification of the authority to make operability determinations, (3) 9 discussion between the 'vice President of Nuclear Generation and all licensed operators of the 10 lessons learned from this event, and (4) remarks by the Vice President of Nuclear Generation at i1 a regular status meeting that emphasized executive management's suppon of the role of the shift 12 supervisor in making operability determinations." In add; tion to these announced corrective 13 actions, the Plant Manager involved in the incident was relieved of his position Neither the DET 14 report nor other NRC evaluations of this matter describe these actions, which are essential to an l5 evaluation of the reasonableness of management's actions in this regard.

( 16 B. Control Room Written Guidance 17 in discussing another example of" events and issues [that) occurred during the past year which 18 undermined the credibility of site management with the control room staff," the DET said:"

19 "A station problem report (SPR) and quality assurance (QA) audit of operations 20 in 1991 stated that hundreds of memoranda on various subjects, such as TS 21 interpretations and operations policies, were issued each year and that many of 22 them conflicted with procedures or with each other. Most of these memoranda 23 were not controlled documents because they had not been reviewed forn ally."

24 Although a rninor point, both the problem repon and the QA audit referred to were written in the 25 first half of 1991 (and the DET performed its evaluation in 1993), so neither had " occurred during 26 the past year." More importantly, Liberty's review produced a more complete set of facts.

" Station Problem Report 92-0201, Attachment B to letter,IE&P to NRC, ST4E AE 4208. September 11.

]) 1992.

U "

DET Repon. p 10.

Page 1139 The Liberty Comulang Group

d RD :

Chapter Tw o - Operations (q

V) 1 The Station Problem Repon (SPR) in question was written by an SRO m June 1991 At STP

anyone can write an SPR SPRs are part of a process used to help encourage the identification of

, 3 problems and areas that can be improved This SPR addressed the concern that the many a memoranda being written could connict with one another or conflict with procedures. The SPR 5 attached one example memorandum that the author of the SPR thought encouraged procedure 6 violations The DET used words from the SPR (e.g," hundreds of memoranda") and stated as fact 7 that "many conflicted with procedures or with each other." Yet there was no evidence that there 8 were any conflicts among the memoranda.

I v The Operations Supervisor at STP responded to the SPR. That response stated that the "use of to memoranda to give supplementary directions or orders relating to conduct of operations in and of it itself is an acceptable practice and is allowed by OPGP03-ZA-0010. It is not now nor has it 12 eve (r] been the policy of Plant Operations to operate the plant using memoranda or similar 13 documents in lieu of current and approved procedures."" As to the one specific example cited in

.' la the SPR, the response stated that there was no conflict with procedures and that there was no 15 reason to suspect other potential procedural violations.

A k 16 There was no specific evidence that there were any conflicts among the memoranda. The response 17 to the SPR did recognize, however, that there could be confusion about which memoranda were 18 in effect. To prevent any possible confusion, a Policy and Practices Manual was instituted with 19 the aim of providing "a single, controlled source of current policies and guidance to assist 20 operations personnel in performing their jobs." Libeny found that STP appropriately considered 21 the employee's concern and took reasonable actions to remedy any perceived problem.

22 The QA audit, to which the DET referred, was another example of a case in which a complete 23 review of the information showed HL&P to be pmdent in its actions. In January 1991 STP's QA 24 department performed a comprehensive audit of plant operations. To place the audit's results in 25 perspective, the following is the audit summary:"

[ " Memorandum, G N. MidkifT to C E. Keen. July 30.1991. DET response item #1141.

( " STP Q A Audit Report 91-02, p. 2. DET response item *1096.

Page 11-40 The Liberty Consulnng Group

l l

RL.% .'

Chapter Tw o - Operations O

l 1 ' Adequate and effective implementation of plant operation activities ensures the 2 safe, eflicient, uniform and reliable operation of the plant. Adherence to Technical

Specifications and operating procedures ensures the continued health and safety 4 of the public and plant personnel The identification of 14 deficiencies,3 concerns 5 and 5 recommendations does not detract from the overall effectiveness and 6 adequacy of implementing plant operation activities and programs."

7 Relative to the use of memoranda, the audit report said."

8 "The audit team has a concern with the number and subjects of memorandums 9 used to communicate management policy, interim guidance or technical direction 10 to Reactor Operators. They address subjects such as Technical Specification 1i clanfications, bo rg temperature limitations, pump runout information, and NRC 12 telephone minutes. Reactor Operators feel that they are accountable for any 13 information or management direction that they contain. We recommend that 14 management review these memorandums for their continued use and incorporation 15 into appropriate procedures, Technical Specification Interpretations, or Standing 16 Orders Management must avoid issuing memorandums that personnel can 17 construe as operating requirements or directions instead of adhering to the current is procedure or document requirements."

19 QA performed a comprehensive audit on operations the following year. In 1992, QA followed up 20 on the prior audit finding. The results were:"

21 " Evaluation of the audit results determined that the Operations Policies and 22 Practices Manual (including policies, practices, memorandum, Night Orders and 23 Condition Notification Forms) is appropriately used and effective in 24 communicating operating information to Plant Operations personnel.

25 Memorandums and Night Orders are not used to amend, revise or delete procedure 26 requirements. The corrective actions taken for DR No.91-005 and actions taken 27 for Audit Report 91-02 Concern No. I have been effective in preventing the use 28 of memorandums to amend or revise procedure requirements."

29 The facts showed that the two year old QA audit concern with the use of written guidance to 30 operators had been resolved. The most recent QA audit had found that this was a strength and was 31 proven to be an effective corrective action to a potential concern identified the year before.

" STP Q A Audit Report 91-02, p. 6, DET response item # 1096.

k "

STP QA Audit 92-01, Audit Checklist item #31 DET response item s1096.

Page 11-41 The Liberty Comulnng Group

RD ' .

Chapter Tu o - Operations (h)

(

'v i The DET indicated that "[m]ost of these memoranda were not controlled documents because they 1

l

had not been reviewed formally " In fact the documents were not controlled because they were 3 not required to be so " The DET also said that "[t]he licensee attempted to consolidate their a written guidance to the control rooms into a " Plant Policies and Procedures Manual." This effort 5 appeared to have been hampered by the inability of the licensee to determine the extent and 6 subject matter of the memoranda that had been issued."* The most recent QA audit found that the 7 manual had been quite effective. While the DET's discussion of the issue was off the mark in ,

8 many areas, it did provide further impetus to enhance the guidance provided to operators 9 Nevertheless, the actions of management had been reasonable in this area as clearly shown by the to 1992 QA audit. Liberty concluded that the information provided to the DET showed that there was ii no signincant difRculty with written guidance provided to the control room. Document control 12 was appropriate. HL&P's actions on a potential, but not actual, problem in 1991 were reasonable 13 and had been proven effective long before the DET's evaluation.

O

" Memorandum. Blair to Hamilton. April 5,1993, DET response item #1020-006

/

DET Report, p.10.

Page I1-42 The Liberry Consulang Group l

iD.:

Chapter Tw n - Operationi i V. Operator Performance  ;

2 A. Introduction 3 Section 214 of the DET repon," Inconsistent Operator Performance," stated?'

4

" Operators generally performed their duties in a competent manner. However, poor 5 communications, marginal staffing levels, and excessive operator distractions and 6 tasks contnbuted to inconsistent operator performance. Cumbersome programs and 7 procedures were also barriers to successful performance. A common element in 8 many previous operator performance problems was the failure to stop and 9 adequately focus on the specific task."

I 10 To support this conclusion, the DET used examples to show that work schedules, practices, and 11 staffing were significant contnbutors to " inconsistent" operator performance. Other matters were 12 also cited to support the proposition that cumbersome programs and procedures were barriers to l 13 successful operator performance.

n 14 Libeny examined some of the DET's examples to see if they showed that management had taken 15 unreasonable actions or inade unreasonable decisions. One example, post-maintenance testing of 16 a diesel, is addressed in the chapter on maintenance in this repon. One way to gauge management 17 action is to examine whether such action is consistent with the feedback HL&P had been receivine 18 from other inspection functions of the NRC. Subsection E below discusses other NRC evaluations 19 of operator performance at STP.

l 20 Liberty sought to determine whether the DET's conclusions about operator performance were 21 consistent with the feedback HL&P had been receiving from the NRC and, if so, whether HL&P's 22 actions in response to that feedback had been reasonable. In the complex and highly l 23 proceduralized operation of a nuclear power plant, operators will occasionally make mistakes. A 24 broad conclusion related to prudence cannot be made on the basis of a few individual events. Even 25 if these events were reported accurately and completely, the poper question for this evaluation 26 should be whether HL&P should have been taking different actens cr making different decisions 27 in light of all the information available to it regarding operator performance.

DET Repon, p.11.

Page 11-43 The Liberry Consulnng Group

l u s.: ,

Chapter Tw o - Operations l

,A (d

)

1 Liberty found that STP had consistently received reports from both internal and external 2 assessments, ir:cluding those from the NRC, that operator performance had been good. HL&P's 3 response to both individual events and broader assessments of the potential causes of mistakes by 4 operators were aggressive and reasonable. Libeny concluded that management was prudent in its 5 actions related to operator performance.

6 B. Unmanned Control Room 7 STP maintains two SROs on shift for each unit. The Technical Specifications require that at least 8 one SRO be in the control room while the unit is in Operating Modes 1 through 4. On Februarv 9 14,1993, while Unit 2 was in Mode 4 (reactor shut down and coolant temperature less than 350 10 degrees), one SRO (the Shift Supervisor) designated the other SRO (the Unit Supervisor) as 1I responsible for the control room command function and left the control room. The Unit Supervisor 12 was involved with a surveillance test and left the control room to observe a portion of the test at 13 cabinets adjacent to the control room. As soon as the Unit Supervisor left, he realized his mistake 14 and returned to the control room. Security records demonstrated that the Unit Supervisor was out

/] 15 of the control room for 41 seconds. There were two other licensed reactor operators in the control 16 room during this period."

17 An NRC inspection report that covered the period including February 14 reported on this event 18 as follows:"

19 "The cause of the event was human error. Although the unit supervisor was new 20 in the position (the individual had been a unit supervisor only about 2 months), the 21 inspectors determined that the event was not caused by a lack of operator training 22 in control room staffing requirements."

i

" LER 93-05. Revision 1 Unit 2, ST lE-AE 4438, May 14,1993.

V "

NRC Inspection Report No. 93-04, ST AE4E-93376, April 16,1993, p 14.

l j

l Page H-44 )

The Liberty Consulting Group

i l

i Rb. '.

Chapter Tu - Operationi V 1 The DET used the meident as ac example to support one of its conclusions The DET repon 2 stated '*

3 " Work schedule, practices, and staffing were signi6 car:t contributors to 4 performance problems in operations Examples reviewed by the team included the 5 following:

6 No SRO was in the Unit 2 control room for a short period because the unit 7 supervisor left the control room to participate in a surveillance activity 8 The licensee determined the root causes to have been a lack of self-9 veri 6 cation and de6ciencies in management guidance regarding command to and control. Contributing factors included the relative inexperience of the 11 SROs involved, shift rotation, and competing tasks that called the unit 12 supen ar out of the control room "

13 Liberty found that the incident was not related to work sche 61es or staffing It involved 14 " practices" only to the extent that STP employed a practice of having an SRO observe certain 15 critical surveillance tests in order to minimize the chances of getting an unwanted trip signal. The 16 licensee (HL&P) did not determine the root cause to have been a " lack of self-veri 6 cation and O)

V 17 deficiencies in management guidance regarding command and control." Rather, HL&P's 18 determination of the cause of the event was as follows: "

19 "The cause of the event was inappropriate actions by the Unit Supervisor. The Unit 20 Supervisor was cognizant of his responsibility to maintain control room command 21 function in the absence of the Shift Supervisor. However, his preoccupation with 22 the surveillance in progress caused his loss of focus (for a few seconds) on his 23 primary responsibility, resulting in a violation of Technical Specification."

24 HL&P's Station Problem Report (SPR) on this event suggested one of the causes was management 25 guidance during the advancement of personnel from an operational to a supervisory role '" This 26 conclusion was reached because the Unit Supervisor had recently been promoted. However, 27 HL&P did not find, as stated by the DET, that management guidance on command and control 28 had de6ciencies. The SPR also mentioned that the event had occurred on the night shift, and that DET Repon, p.11.

CN '"

LER 93-05, Revision 1,IJnit 2, ST HL-AE-4438, May 14,1993.

8 Station Problem Repon 930513, DET response item #1183.

Page 11-45 The Liberry Consultmg Group

4 W..

(. hapter Tw o - Operationi U l this was "a known factor associated with the ability to process information " The DErs use of 2 the term " shift rotation" was not really meaningful in this context.

3 Liberty concluded that the " unmanned control room" incident had nothing to do with work a schedules or staffing it was simply a mistake. Libeny found that HL&P's evaluation of the 5 incident was thorough and provided no basis for management to take funher action 6 C. Baron Dilution Event 7 To illustrate its findings related to operator performance, the DET also used an example 8 concerning a February 1993 boron dilution event. The DET said: 5 9 " Work schedule, practices, and =taffing were significant contributors to

. 10 performance problems in operations. Examples reviewed by the team included the 11 following-12 "a An inadvenent boron dih5 tion event occurred while the operators attempted t 13 to borate the reactor coolant system. The licensee determined that the event V 14 was caused by a deficient understanding of the system operation during 15 shutdown ccaditions. However, other contributing factors mentioned in the 16 licensee's assessment included an inadequate shift turnover, insufficient 17 crew experience, and the inability of personnel to properly focus on a 18 specific task."

19 Boron is used in the reactor coolant system as an absorber of neutrons to help control the nuclear 20 fission process. A mechanical system called the Boron Thermal Regeneration System fBTRS) in 21 each unit at STP can be used to change the boron concentration in the reactor coolant system. The 22 BTRS is a sub-system of the chemical and volume control system and uses a combinatior' of 23 different flow paths, varying temperatures, and demineralizers to either remove or add boron. Its 24 normal use is to allow for load following operations at power.*

25 On February 20,1993, Unit 2 was in Mode 4 (shut down with temperature less than 350 degrees),

26 and the BTRS was being used to add boron to the reactor coolant system. Over the course of two 27 shifts and several samples of boron concentrations, it was recognized that in fact boron was being

/ DET Report, p. I1. .

STP Updated Final Safety Analysis Repon. Section 9.3 4.

1 Page H-46 The Dberty Consulting Group

RLC.'

Chapter l u o - Operations O\

removed from the reactor coolant system instead of being added While the reactivity shutdown (V) I 2 margin was never reached (and would not have been reached even if the operators had not recognized the situation), the event was a serious concem to HL&P because it meant that operators a had not fully recognized the reactivity increase."5 5 The NRC also believed this event was a serious matter. An inspection repon issued in April 1993 6 gave the matter considerable attention The NRC summarized the extensive corrective actions that

HL&P either had taken or planned to take as a result of the event. The NRC also said '"

8 "The cause of the event was inadequate understanding of the BTRS during plant 9 shutdown operations. The plant operators did not expect the system to operate as 10 it did, in part, because of the experience level of the crew. A contributor to the 11 event was an inadequate understanding of where the boron concentration 12 monitoring system, which was monitored throughout the event, actually measures 13 the boron concentration."

14 HL&P also believed the root cause of the event was the lack of understanding of BTRS operation 15 during shutdown conditions (The system's normal use is during load following operation at 16 power.) In identifying the factors that had caused the event, HL&P noted that (1) the combined V 17 experience level on the dayshift operating crew had caused a slower response to the event, (2) the 18 oncoming shift supervisor had not conducted a thorough review of the BTRS system status, and 19 (3) the Shift Technical Advisors had not addressed the boration operation during shift turnover."'

20 Neither the NRC's inspection of the matter nor HL&P's investigation identified " work schedule, 21 practices, and staffing" as significant contributors to the event. HL&P did not conclude that 22 " inadequate" shift turnover had contributed to the event. HL&P did not conclude that 23 " insufficient" crew experience had contributed to the event. HL&P did not conclude that personnel 24 had been unable to " properly focus on a specific task." While the seriousness of this event itself 25 should not be minimized, Liberty found that the documentation associated with the event did not 26 indicate that management's approach to work schedule or staffing had been unreasonable or 27 contributed to operator performance issues.

NRC inspection Report No. 93-04, ST AE.Hl. 93376, April 16,1993, pp.14-16; Station Problem Report 930582, March 3,1993, DET response item #1231.

NRC laspection Report No. 93-04, ST AE-HL 93376, April 16,1993, p.15.

[j\

g d' Station Problem Repon 930582, March 3,1993, DET response item #1231.

Page 11-47 The Liberry Consulang Group

RL%:

Chapter Tu - Operations 1 D. Incorrect Valve .\lanipulation 2

The third example used by the DET to support its conclusion about inconsistent operator ,

3 performance concerned an incident that had happened shortly before the DET review started in l 4 .\tarch 1993 The DET reported

• 1 5

"During a periodic survei!!ance of the ECW system, the operator who was 6 performing the local valve manipulations had to leave the area to locate a valve e

lock key so he could throttle Dow to a heat exchanger. When he returned, he 8

throttled the valve to the wrong heat exchanger in a different train. The licensee 9

determined that the event resulted in part from inadequate self-veri 0 cation. The lo licensee stated that a contributor to the event was the insuf0cient number of 11 personnel available to perform the evolution. SROs who have performed this 12 surveillance in the past stated to the team that generally, four RP0s are required 13 to perfomi this surveillance, although the surveillance could have been performed 14 ef6ciently with three RPOs. In this case, only two RPOs performed this 15 surveillance which made it difficult to focus on the required specific tasks. The 16 three remaining RPOs on shift at the time were not available because they were 17 performing other duties."

18 While it is true that HL&P's evaluation of this event determined that a cause was a less than 19 adequate practice of self-veri 0 cation, this was described as a " secondary root cause." The primarv 20 root cause was determined to be the procedure that made it necessary to throttle now to obtain the 21 same pump discharge pressure each time the surveillance was performed. HL&P intended to 22 perform an evaluation of whether the procedure could be made easier to perform by using the 23 system operating curve instead of having to throttle now.*

24 HL&P's evaluation indicated that one of the contributing causal factors was less than adequate 25 staffmg for the performance of that particular evolution. The other RP0s on shift were occupied 26 because the reactor was defueled and the reactor coolant system was drained, requiring a sightglass 27 watch and an operator to perform valve manipulations. The fact that the plant was in "no mode" 28 made the immediate consequences of the event insignincant. The concern related to whether this 29 could happen when ECW Gow was really required. Also not mentioned in the DET report was the 30 fact that a reactor operator had offered to help with the evolution, but his offer had been tumed -

n*

I DET Report, p. I 1.

O '"

Station Problem Report 930906. DET response item #1252.

Page U-48 The Liberty Consulting Group

RL.% .'

Chapter Tw o - Operations h Q l down because he was not in the local Bargaining Unit. and the tasks to be performed were 2 identified as Bargaining Unit tasks "

3 The DET's summary of this event was accurate but incomplete. The DET did not repon what 4

HL&P had found to be the primary root cause of the event. The DET repon also did not explain 5 the significance of the plant's operating mode at the time of the event or the fact that additional 6

personnel had volunteered to help with the evolution. Liberty concluded that the event provided 7 no basis for a conclusions that management had taken unreasonable action or made unreasonable 8 decisions that led to inconsistent operator performance.

9 E. Other NRC Evaluations of Operator Performance 10 In the section of this report on operations staffing and the administrative tasks required of control 11 room operators, it was noted that numerous and regular NRC inspection reports had praised the 12 general performance of STP's operators. The NRC rated operator professionalism good and found 13 that overall personnel performance within the control room was in accordance with h1C r 14 requirements. That section also noted that in late 1992 the NRC's OSTI had reached favorable 15 conclusions about operator performance.

16 Prior to the issuance of the DET report, the NRC document that provided the most comprehensive 17 summary of the NRC's view of operator performance was the SALP repon that covered the period 18 from June 1991 through July 1992. The SALP report contained the following comments on the 19 area of plant operations;"!

20 "The previous SALP report noted strong performance by operators during the plant 21 transients, good operations support, and that the plant operating procedures, 22 housekeeping, and material condition of the plant had improved. During this 23 assessment period, enforcement history and reponable events in this area revealed 24 the continuation of the similar types of problems that were noted daring the 25 previous assessment period, but fewer in number. . Management involvement in 26 plant operations was generally good during this assessment period, with some 27 exceptions noted. The Unit 2 refueling outage and the Unit 1 maintenance outage 28 were both well managed and controlled. . As in the previous assessment period, I

Station Problem Report 930906 DET response item #1252.

~

NRC SALP Repon. ST AE-Hl. 93239, November 18,1992,pp.4-5.

Page H-49 The Laberry Consultmg Group

I RJ S:

Chapter T= o - Operations i operating crew performance remained good in response to most plant events and

transients, and licensed operator actions were consistently conservative in nature.

3 Plant operating procedures, including emergency operating procedures, system 4

operating procedures, and alarm response procedures, were upgraded during the 5 assessment period. Generally, adherence to procedures by operators has been 6 good Operating crew staffing to support routine operations was evaluated as 7

good Operations support staffing and assistance was determined to be superior.

8 The support staff has continuously provided good technical suppon in such areas y

as dispositioning station problem repons and upgrading procedures. Operations to personnel maintained a professional work environment in the control room 11 Communications between the control room operators and craft personnel during 12 the performance of maintenance and surveillance activities were good. The ability 13 to control and direct complex evolutions was evident during reduced inventory la operations and power changes."

15 Not everything about operations in the SALP report was positive. The NRC pointed out areas in 16 which HL&P needed to improve. Nevertheless, the SALP report provided no indication 17 whatsoever that STP's management needed to take significant steps to upgrade its operators' 18 performance. While HL&P was aware that there was room for improvement in operator 19 performance and support of the operations staff, the SALP report's comments represented the O 20 regulator's input to STP management regarding operator performance immediately prior to the 21 DET's evaluation. Therefore, Liberty concluded that STP management was prudent in its actions 22 lelated to operator performance.

v Page H-50 The bberry Consuhmg Group

_ _ . _ ~ __ _ _ _ . , . _ _ - - - __ _

RLS :

Chapter Tw o - Operations  !

i VI. Problem Identif'ication and Resolution by Operations -

2 A. Plant Labeling 3

The DET used the plant's labeling as an example of ineffective problem resolution and poor 4 support from management. In the section ofits report on operations, the DET noted that IA&P l

5 had conducted a labeling study." The results of that study were recommendations for a relabeling  ;

6 of all equipment (e.g.,25,000 valves,5,000 instruments) at a total project cost ofjust over $1 7 million."'

n 8

The DET noted that initially management had not approved the funding for the complete 9

relabeling effort. Instead, maintenance of existing labels and replacement of missing labels was ,

10 to continue."' In the section of its repon on management support and resource utilization, the 11 DET reported that operations had requested $220,000 for the relabeling program in the 1993 12 budget, but that management had approved only $12,000."5 Later, in the section on management -

13 corrective actions, the DET opined that the failure to fund the relabeling program had ,

14 "significantly contributed to the lack of station improvement.""' i 15 From a prudence perspective, the significance of management's decision to maintain existing 16 labels rather than to relabel the plant is questionable. Moreover, the DET report contained no  ;

17 discussion of IE&P's actions. The DET noted that in late 1992 and early 1993, component i 18 labeling had contributed to personnel errors. Then the DET stated: "At the end of the evaluation, i 19 the licensee informed the team that it was again reviewing the prioritization of the plant labeling 20 upgrade.""' i

)

"2 DET Report, p.13.

"5 Site Labeling Project Proposal, DET response item #1011. ,

DET Report, p.13.

DET Report, p. 42.

DET Report, p. 45.

DET Report, p.14 Page H-51 The Liberry Consultmg Group i

.- -- .. . _ - - . . . - _ . - - - ~ ~ . - .

l RLh '

Chapter Tw o - Opera tions 1

However, before the DET arrived at the STP site. management had incorporated the 2

comprehensive program for upgrading plant labeling into the Afaster Operating Plan for 1993

• 3 The DET was informed that this item had been included in the Afaster Operating Plan in a written 4

response dated April 5,1993, early in the DET's second week on site."' HL&P issued a 5

51,000,000 Change Notice to the budget that funded the relabeling effort. Libeny found that 6

after there were instances in which plant labels might have contributed to personnel errors. HL&P 7

had acted promptly and dedicated considerable resources to improving plant labeling.

8 B. Locked Valves s

9 The DET indicated that self-assessments on locked valves had " failed to find significant problems to that were later evident through events or reviews by other organizations."' ' The DET also said  ;

11 that "a weak locked valve program" was one of the challenges faced by the STP operators.':

12 During the DET's evaluation, HL&P performed a study of problem reports that may have been  ;

13 associated with locked valves. The study found that in 1990 five station problem reports (SPR) i 14 had documented problems with locked-valve components. Two of the five problems were 15 attributable to programmatic errors. The program was thereafter revised to clarify requirements.

16 In 1991, only one SPR concerned locked valves. However, in 1992, five locked-valve SPRs were  ;

17 written. Two concerned a locked valve deviation log (an administrative means to keep track of 18 l

valves purposely out of the normally locked position), and two others involved deficiencies 19 attributable to locking tabs (one type of physical device that locks valves in position).8 3 [

Master Operating Plan, Goal PM-3, Action #5, March 16,1993.

DET response item #4039.

82.

DET response item #1143.

DET Report, p.12.

DET Report, p 13.

Station Problem Report 930341. DET response item #1070.

t Page II-52 f The Liberty Consultmg Group

{

r

RLS.'

Chapter Tw o - Operationi 1

in December 1992, STP's Vice President, Nuclear Generation requested INPO to make a special 2

assistance visit to help improve STP's valve position control efforts. INPO performed this visit 3

in February 19939 INPO suggested ways to make improvements.

4 Also during February 1993, STP personnel discovered two valves unlocked that, according to 5

procedure, should have been locked? As a result of this discovery and the industry assistance, 6

STP took actions such as (1) replacing locking tabs with lock keys, (2) implementing procedure 7

changes, (3) carrying out a locked valve reduction program, and (4) revising training content.

8 STP's management thus had taken prudent action related to the configuration control and 9

administration oflocked valves. When the initial actions proved to be not as effective as HL&P 10 had hoped, additional substantial actions were taken. The DET report did not discuss any of  :

1i HL&P's efforts to make improvements in this area. Liberty concluded that HL&P's actions related '

12 to locked valves had been reasonable.

13 C. OERs and SPRs

14 x The DET said that operations' reviews of problem reports and operational events lacked depth and 15 breadth. To support this subjective opinion, the DET used quantitative information that implied 16 that there had not been enough people to perform the reviews adequately. The DET report 17 contained the following paragraph?

18 "Under the licensee's corrective action program, the CAG and the Independent 19 Safety Engineering Group (ISEG) assigned investigations and operating 20 experience reviews (OER) to the operations staff. The two SPR coordinators on 21 the operations staff were responsible for performing 8 to 10 OER and 20 to 30 22 SPR reviews a month. These individuals spent large amounts of overtime to 23 complete the sizeable workload as the volume of SPRs continued to grow."

24 Liberty reviewed the information given to the DET on the subjects covered by this paragraph.

25 Regarding the number of OERs and SPRs reviewed by operations, the DET's requested 26 information for the most part covered only those items that had not been completed by the Lener, INPO to NRC, Borer to Kinsey, March 8,1993.

8" Station Problem Repon 930341, DET response item #1070.

\

d "'

DET Report, p.13.

Page 11-53 The Liberty Consulting Group

RLS:

Chapter Tw o - Operations O

1 assigned due date 'r One response to the DET provided information on OERs that had been 2

reviewed by operations in 1992 and early 1993. It showed that operations had completed the 3 review of 68 OERs in all of 1992, fewer than 6 per month. For the first six months of 1993, 4 operations had been assigned to review 34 OERs, again fewer than 6 per month "' The DET asked 5 for the current status of all OER reviews The response showed that the entire Nuclear Generation 6 Depanment (of which operations is a part) had no overdue OFR reviews?

7 The DET may have been told in interviews that operations had reviewed 20 to 30 SPRs per month, 8 but quantitative data to support this figure could not be located. During the seven-month period 9 from September 1992 through March 1993, an average of 36 Category 1-4 SPRs per month were 10 generated at STP? Since many other groups, such as engineering and CAG, are actively involved 11 in the review of SPRs, it is clear that operations was assigned as many as the DET stated.

12 However, the exact number reviewed per month is not as significant as whether the operations 13 group handled the assigned workload. The monthly totals for Category 1-4 SPRs generated during 14 the same seven-month period were 49,36,27,40,38,30, and 34. These data do not support the 15 DET's assertion that the volume had been crowing. Furthermore, the total non-outage overtime

( 16 spent by the operations support group was less than 7 percent."2 As to both the stafnng and the 17 quality of the operations support, the NRC gave STP's management a different assessment just 18 a few months before the DET evaluation. In the S ALP repon issued in November 1992, the NRC 19 concluded.u 20 " Operations support staffing and assistance was determined to be superior. The 21 support staff has continuously provided good technical support in such areas as 22 dispositioning station problem reports and upgrading procedures "

l DET requests nos. 3678 and 3681.

DET response item #3702.

DET response item #3097.

l Station Reports, September 1992 through February 1993 DET response item #0018, and Station Report. I March 1993. I DET response item #4043. 1

\ 02 NRC SALP Report, ST AE IIL-93239, November 18,1992,p5.

Page 11-54 The Libertv Consuking Group

RI.V.:

Chapter Tw o - Operations r

/

(- 2 1

Libeny concluded that the facts penaining to OERs and SPRs would not have led management.

prior to the DET report. to believe that it had provided inadequate support to operations in this area or that significant management action was necessary to avoid serious regulatory problems i

i l

s

R' EA'b5 The Libem Con:ulting Group

- . _ - . - = _ . . - - . . . . - - . . - . . . - . . - - . _ . _. . _ _ .-

d 1

l@

i P

This page intentionally left blank I

1 r

en i

i e

' I

i RL.s .'

Chapter Three - \taintenance 1

I i Chapter Three -- Maintenance  !

2 1. Introduction and Summary of Findings 3 The DET report covered a broad range of maintenance and testing issues.

4 "The team reviewed; preventive maintenance, predictive maintenance, corrective 5 maintenance, periodic testing and post-maintenance testing to determine their 6 contributions to proper equipment performance. The team also reviewed the 7 number and experience of available personnel resources, the work control process, 3

8 maintenance training, maintenance facilities and the availability of spare pans as 9 they related to the performance of maintenance, the performance of tests, and the 10 maintenance request backlog. The team conducted formal interviews, informal 11 interviews, machinery history record reviews, and direct observations to evaluate 12 maintenance and testing performance."

1 13 The DET made favorable comments about STP's maintenance shop facilities, the Technical 14 Support Engineer Group, the General Maintenance Supervisor position, and the use of walkdown 15 crews The DET also acknowledged that "[t]he licensee had recognized the need for improvements 16 and had recently developed or was developing corrective action plans to address many of the 17 problem areas."2 Despite these positive general remarks, however, the overall report on 18 maintenance was critical. The DET concluded that corrective and preventive maintenance, craft 19 performance, staffing, and the work control process had not been sufficiently effective and they 20 had contributed to reduced reliability of safety-related and balance-of-plant equipment. Many of 21 these concerns appeared to have stemmed primarily from the size of the maintenance backlog.

22 Liberty found that STP's operating history, panicularly the record run that Unit 2 had in 1992, did 23 not suggest an unreliable plant, and that STP's overall safety record did not indicate a lack of 24 reliability in safety-reuted equipment. To a large degree, the DET supported broad conclusions 25 with a limited number of specific examples. Liberty examined the information presented to the 26 DET and analyzed the issues addressed in the DET report, including many of the examples used 27 to support the DET's conclusions to evaluate the reasonableness of management's decisions and 28 actions in the maintenance area.

' DET Report, p.14.

2 DET Report p.15.

Page III-l The Libery Consulnng Group

ich :

Chapter Threr \taintenance

/'

( i Liberty found that while the DET repon did identify areas in w hich improvement was needed. the

findings in the DET report would not support the proposition that STP's management had made 3 unreasonable decisions or taken unreasonable actions in the area of maintenance Particularly a notewonhy were HL&P's ongoing etTorts to identify areas that needed improvement, the measures 5 taken by HL&P to make such improvements, and the progress actually made Liberty also found 6 that prior NRC assessments and inspections would not have caused reasonable management to 7 conclude that significant changes in the way maintenance was managed were necessary in order 8 to avoid adverse regulatory action.

9 In some Cases, the DET reached conclusions on the basis of preliminary information. Facts that to became known after the DET's evaluation caused Liberty to reach different conclusions. In other 11 cases, a review of the facts related to a particular example resulted in Liberty drawing different 12 conclusions and fonnulating difTerent characterizations. The DET sometimes cited dated examples 13 from years past without discussing the corrective actions taken in the meantime. The way in which 14 the DET presented some information could give the impression that it had analyzed the issues at 15 STP in isolation and failed to take into account an industry-wide perspective. That perspective 16 often revealed that difficulties encountered at STP, with preventive maintenance, for example, v) 17 were not unique but characteristic of the nuclear mdustry as a whole.

18 HL&P's handling of the matters associated with the overspeed trips of the turbine-driven auxiliary 19 feedwater pumps was prudent. A review of the entire history of the pumps led to the conclusion 20 that STP had had reason to believe the pumps were reliable. When overspeed trips occurred in 21 early 1993, HL&P acted in L thorough and safety-conservative manner.

22 The following sections provide the bases for Liberty's conclusions These sections parallel the 23 DET report and address corrective maintenance, preventive maintenance, maintenance training, 24 replacement parts, support to maintenance, the work control process, and post-maintenance and 25 periodic testing. The fmal section of this chapter is devoted to the overspeed trip events associated 26 with the auxiliary feedwater pumps.

O O

Page UI-2 The Labery Consulting Group

RL%: l Chapter Three - Wintenance l O l V i II. Corrective Maintenance l A. Introduction 3 The section of the DET report on corrective maintenance consisted of brief descriptions of two 4 sets of examples cited as support for the DET's broad conclusien that "[i]neffective corrective 5 maintenance . adversely affected safety-related equipment performance."5 6 Liberty found that the issues raised by the DET did not demonstrate unreasonable management of STP's corrective maintenance. It is unreasonable to expect that mistakes will not occur in the 8 implementation of corrective maintenance. The DET did not specifically describe the standard by 9 which it measured the results of STP's corrective maintenance, but the comment that "[c] raft to personnel occasionally made mistakes during corrective maintenance" suggested a standard of 11 perfection

• Without an objective standard, STP's co Tective maintenance can only be judged by 12 the results. However, STP's overall results in terms of operating performance and safety record 13 did not suggest weaknesses in the effectiveness ofits corrective maintenance.

b 14 The initial set of examples was aimed at " poor" root cause determination and maintenance efforts.

15 The first of the DET's examples concerned a feedwater isolation bypass valve position. The DET 16 was only able to use preliminary (and ultimately superseded) information on this matter, as 17 discussed in subsection B below. The second example, diesel injector pump hold-down studs, is 18 covered in the engineering chapter of this report. A review of that issue showed that STP's root 19 cause determinations had been reasonable. A routine maintenance item relating to a diesel water 20 jacket leak is addressed in subsection C below. This matter raised the question of how much 21 significance in a pmdence evaluation should be attached to using a gasket of the wrong size to 22 repair a fhnge leak. The auxiliary feedwater pump overspeed trip is discussed in section IX below.

23 There it is pointed out that another NRC inspection yielded results much different from those of 24 the DET Another of the DET's examples involved the incorrect installation of a sight glass and 25 the resultant overfilling of a bearing with oil.

26 The second set of examples related to the timeliness and prioritization of corrective maintenance 27 actions. The first example in this set involved an instrumentation system referred to as QDPS.

' DET Report. p.15.

U

• DET Report, p.15.

Page III.J The Liberty Consulnng Group

RLh-;

Chapter Three %Iaintenance e 1 Subsection D below describes that issue The second example related to steam generator access 2 covers and is addressed in subsection E below.

3 B. Feedwater Isolation Bypass Valve Position 4 On April 8,1993, while the DET was on site at STP, an instrumentation and controls technician 5 was assigned to align the local position indicator on a feedwater isolation bypass valve When the 1

6 technician attempted to verify valve position by rotating the valve stem, there was a sudden

{

7 movement, and the local valve position indicator showed the valve to be shut The technician l 8

assumed that the valve had been mechanically bound and had not been completely shut. HL&P 9

followed procedures and notified the NRC that this valve had been found out of position. On the 10 basis of this information, the DET reported) 11 " A feedwater isolation bypass valve (a containment isolation valve) was found 12 partially open for over a year. Maintenance had been performed on the valve to 13 correct a failure to get a closed indication light in the control room. Maintenance la personnel stroked the valve several times and then adjusted the closed limit switch 15 to bring in the closed light without confirming the actual position of the valve.

16 Five months later the licensee issued another SR to correct an apparent discrepancy 17 between the control room indication and the local position indication. However, 18 the potential safety significance of this condition was not properly recognized and 19 the SR was worked six months later. At that time maintenance personnel 20 determined that the valve was only going 75 percent closed."

21 After notifying the NRC, STP hired Fisher Diagnostic Services to perform a valve diagnostic test 22 and inspected the valve with representatives of the valve manufacturer, Valtek. These 23 investigations found that the valve had not been open. The sudden movement noticed by the 24 technician had been caused by actuator movement, not valve stem movement.

25 After the DET completed its field work, the NRC conducted a special " reactive" inspection of this 26 incident. The NRC's report acknowledged the new information.'

l DET Repon, p.15.

NRC Inspection Repon No. 93 19, ST-AE4IL-93483, My 23.1993.

Page 1114 The Libertv Consulting Group

1 l

l Nb* ' l Chapter Three - Maintenance O 1 1 \

V 1 "The \'altek report dated May 27.1993. concluded that the valve was closed on 2 April 8, and when the stem clamp was loosened by the technician, the spring pressure forced the threaded actuator stem down over the threaded valve stem, 4 which appeared to the technician as valve travel "

5 The NRC's inspection indicated some problems associated with this valve incident For example.

6 the NRC determined that the technician's action to verify valve position had not been speci6ed 7 in the work request, and that HL&P had not fully evaluated the cause of the inaccurate local valve 8 position indicator. However, the NRC agreed that the containment isolation valve had not been 9 partially open for over a year.'

to C. Diesel Water Jacket 11 The DET used a minor difGculty that involved fixing a water leak as an example of " poor 12 maintenance etTorts" and of" attempts to install incorrect parts."' The facts showed that this matter 13 had been a typical maintenance problem that had been quickly resolved. While a mistake was p

14 made, its effect was to lengthen the time it took to 6x the leak, not to compromise safety or

\ 15 reliability. And while STP should strive to prevent such problems, it is unrealistic to believe these 16 kind ofincidents will never occur.

17 In its discussion of corrective maintenance, the DET said:'

18 "A SDG jacket water leak took four attempts to correct. The first two repair efforts 19 were unsuccessful because maintenance personnel installed the wrong size of 20 gasket. In a third repair attempt, the gaskets were made on site with material not 21 suited for that application."

22 In discussing the installation of replacement parts, the DET described the same occurrence."

NRC Inspection Repon No 93-19, ST AE-HL-93483, July 23,1993.

DET Repon, p.15 and p. 20.

DET Repon, p.16.

" DET Repon. p. 20.

Page 111-5 The Liberty Consulnng Group l

I

(U.s:

Chapter Three \laintenance V 1 "During repair activities to stop a jacket water leak on the inlet header of a SDG.

the discharge header gasket was installed This occurred twice before the 3 mechanics recognized that the gasket was not the correct size '

4 The issue involved maintenance work performed on a service request to correct a water leak at the 5 inlet Dange of thejacket water for the =13 standby diesel generator. On March 27,1993, after the o Dange gasket had been replaced and the system refilled, leakage was apparent and tightening of the Gange did not stop the leak. Initially, maintenance suspected that the gasket might have rolled 8 into the pon instead of maintaining alignment between the Gange faces, so the gasket was 9 removed, Dange faces were cleaned, and the gasket was replaced. This effort did not stop the leak.

10 On the next day, maintenance removed the gasket and discovered that it was too small to 6t 11 exactly over the port and bolt holes. Maintenance initiated a problem report to document the 12 situation and a plant change form to allow the use of gaskets fabricated at STP from another 13 gasket material, Permatex. Investigation by STP personnel showed that the service request had 14 initially specified the correct gasket, but that a revision had been made to specify a very similar 15 (but slightly smaller) gasket meant for the outlet flange. On March 30,1993, the correct gaskets 16 were obtained and used to successfully complete the repairs."

V 17 If the inlet and outlet gaskets are held side by side, the difference in size is small but visible.

18 However, looking at the two gaskets separately, it is easy to understand how one could be 19 mistaken for the other. The maintenance craft used the gasket that was specified in the work order.

20 When it did not work, they sought engineering help. What the DET called the use of" material not 21 suited for the application" was a request by maintenance personnel to engineering to evaluate the 22 use of material that would stop the leak. The cause of the difficulty in making this repair was the 23 error in the work order that had resulted in the wrong gasket being supplied. Installation attempts 24 were reasonable considering the similarity of the gaskets, and an attempt to remedy the matter 25 with an alternate material was a common-sense approach to solving a minor problem. Libeny I

26 concluded that this matter was not a reasonable example of" poor maintenance efforts." there was 27 no threat or significance to plant safety, and STP's handling of this routine maintenance issue was 28 reasonable.

l l

l S

\

" DET response item #2300, and SPR 930988 contained in DET response item #3612.

Page III4 The Libery Consulung Group i

. RLS.:

Chapter Three - Wintenance i- D. Qualified Display Processing System 2

The DET used an example involving some missing internal screws in a cabinet for a display 3

system in Unit I to support conclusions in the areas of maintenance, engineering, and management 4

and organization. After a complete review of the facts of this matter, Liberty concluded that this 5

was not an example of unreasonable actions or an unreasonable decision by management.

6 in the section of its report on maintenance, the DET stated."

7

' " Untimely ccrrective maintenance and poor prioritization resulted in delays in 8

restoring equipment to an operable status, allowed degraded equipment to 9

deteriorate until it was incapable of performing its intended safety function, and to resulted in site personnel being forced to work around the failed and degraded i1 equipment. Specific examples follow:

12 "- In August of 1992, the licensee discovered that seismic hold down 13 screws in the Qualified Display Processing System (QDPS) card 1 14 racks were missing but did not issue a SR to replace the missing 15 screws for four monthi. The team noted that the SR had not been p

16 implemented or evaluated for operability. At the request of the 17 team the licensee evaluated the situation. Consequently, QDPS was 18 declared inoperable affecting both units."

19 In the section on engineering, the DET stated:"

20 "The engineering staff did not always adequately evaluate equipment operability 21 as illustrated below:

22 "In August 1992, a system engineer discovered that seismic hold-down screws 23 were missing from the Unit I quality display parameter system [ sic] (QDPS) card 24 racks, but did not understand the seismic consequences and did not request an 25 evaluation for operability. The licensee did not properly evaluate the effect of the 26 deficiency on operability until so requested by the team in April 1993. The QDPS 27 was subsequently declared inoperable."

l DET Repon, p.16.

DET Repon, p. 28.

l Page III 7 The Libery Consulung Group ,

I

Rb-Chapter Three - Wintenance (m\

g i Finally. m the section on management and organization. the DET stated."

2 "The team identified several instances where inadequate safety evaluations resulted 3

in ineffective corrective actions For, example, the licensee failed to adequately 4

evaluate the impact of missing seismic hold down screws in the Unit 1 Quality 5 Display Parameter System (sic)(QDPS) card racks when discovered in August 6 1992. The QDPS was declared inoperable eight months later following an operability review requested by the team "

8 After the DET had completed its Held work, the NRC conducted a special inspection "to l

9 determine the circumstances surrounding the inappropriate dispositioning of a service request that I 10 had identi6ed deficiencies in the seismic qualifications of the quali6ed display processing 11 system "" The NRC's findings in this special inspection were different from those repeated three 12 times in the DET report. The later inspection confirmed that an error had been made but reported 13 the facts differently.

14 "The inspector determined that the condition concerning the seismic qualification 15 of the QDPS was identi6ed on January 4,1993, when the system engineer 16 identified the missing hold-in screws and generated the SR. On April 29,1993, 17 the SR was given to the Design Engineering Department to perform a conditional O(,/ 18 release on the effect of the rnissing hold-in screws. The licensee initiated 19 rework action on the missing hold-in screws on the QDPS cabinet. The missing 20 screws were replaced and the QDPS was declared operable on April 29,1993."

21 The later inspection confirmed that there had been a delay in assuring that the equipment 22 maintained its seismic quali6 cation but did not find that the delay had had anything to do with 23 " untimely corrective maintenance and poor prioritization." The inspection reported no evidence 24 that degraded equipment had been allowed to deteriorate or that site personnel had been forced 25 to work around failed equipment.

26 HL&P's review of the incident agreed with the NRC inspection.

DET Repon, p. 45.

A NRC Inspection Repon No. 93-21, ST AE-HL 93459, June 30,1993, p.1.

NRC Inspection Repon No. 93 21, ST-AE4fl. 93459, June 30, t993, p. 3.

Page 111-8 The Liberry Consulting Group

Rb :

Chapter Three - \taintenance f}

/ 1 "The Shift Supervisor reviewed the SR and annotated on the SR that an 2 engineering evaluation. called a Conditional Release Authorization, was necessary to determine if the QDPS could be relied upon to perform its design function under a all analyzed conditions. Neither the System Engineer or Shift Supervisor identified 5 the missing screws as a potential operability issue and the affected equipment was 6 not declared inoperable. The SR was delivered to Maintenance Planning Division for processing and data entry but following data entry, the SR was inadvertently 8 filed instead of being forwarded to the Technical Support Engineering Group for 9 the Conditional Release Authorization that the Shift Supervisor had determined 10 was necessary.""

11 The NRC also confirmed that the system engineer had not recognized the operability issue 12 involved but did not find that the " engineering staff did not always adequately evaluate equipment 13 operability." What the NRC's special inspection of this issue found was that the service request 14 had asked for a conditional release evaluation of the equipment. That evaluation was not 15 performed promptly because the service request was filed rather than sent to engineering.

16 "Neither the System Engineer or the Senior Reactor Operator that reviewed the SR 17 identified the condition as a potential operability issue, due to affecting the seismic O 18 qualification of the QDPS cabinet, and the QDPS was not declared inoperable. A k 19 conditional release authorization, which formally requires an operability evaluation 20 to be performed by the Technical Support Engineering Group, was requested in the 21 SR to obtain engineering concurrence that the missing screws did not affect the 22 seismic qualification of the QDPS. The SR was subsequently delivered to the 23 Maintenance Planning Department and was inadvertently filed rather than being 24 forwarded to the Technical Support Engineering Group for completion of the 25 conditional release authorization.""

26 In fact, once engineering had received the request to perform the conditional release, the 27 significance of the missing screws was recognized and they were replaced the same day.

28 The DET concluded that "the licensee failed to adequately evaluate the impact of missing seismic 29 hold down screws in the Unit 1 Quali[fied] Display P[rocessing) System (QDPS) card racks when 30 discovered in August 1992. The QDPS was declared inoperable eight months later following an 31 operability review requested by the team " Liberty's statement of this conclusion was: In January 32 1993, when the system engineer discovered that there were missing screws in the QDPS, neither A LER 93-016. Revision 1. Unit 1. July 15,1993, p. 2.

t i k "

NRC Inspection Report No. 93-21, ST-AE-HL-93459, June 30,1993, p. 3.

Page 111-9 The Libery Consultmg Group

Ki v:

Chapter Ihree - \taintenance i

/m b)\ I the system engineer nor the reviewing SRO concluded that the seismic quali6 cation of the QDPS 2 could have been affected. However, a service request was prepared to request an engineering 3 evaluation of the equipment Unfortunately, the service request was inadvenently filed instead of a

being fonvarded to engineering. It was nearly four months before the matter was brought to the 5 attention of the engineering staff. Engineering's evaluation and corrective actions were then 6 completed in one day.

7 Libeny concluded that this entire matter had been an isolated case in which personnel had not 8 fully recognized the significance of the missing screws, and an administrative error had occurred v in the 61ing of a form. It would not have signalled STP's management that there were serious i t

to problems in maintenance, engineering, or management or of a program that required a signi6 cant I i1 mid-course correction to avoid significant regulatory dif6culties.

l 12 E. Steam Generator Access Covers

~

13 One of the DET's examples related to corrective maintenance concerned the access covers on a A 14 steam generator in Unit 1. The DET's summary was?

-Y 15 "The steam generator primary side access covers on Unit I had known leaks for 16 two and a half years prior to being repaired. On four separate occasions licensee 17 personnel noted the leaks, however, corrective action was not implemented. These 18 leaks existed through two refueling outages. While numerous SRs were wTitten for 19 repairs, confusion concerning the status of the SRs resulted in the repair effons not 20 being performed."

21 The DET's brief summary of this example was fundamentally accurate. However, there were other 22 factors involved that affected the determination of management prudence. In retrospect, this was 23 clearly a case in which mistakes were made. It was not a case, however, in which HL&P simply 24 ignored leaks. Furthermore, it did not represent a case of allowing " degraded equipment to 25 deteriorate until it was incapable of performing its intended safety function," nor was it a case of 26 " sit'e persormel being forced to work around the failed and degraded equipment," as stated by the 27 DET.20

[~N "

DET Report, p.16.

(U ) " DET Repon. p.16.

Page III-10 The bberty Consuhing Group

ids :

Chapter Three \laintenance p)

(G 1 2

The NRC performed an inspection that focused only on this matterjust before the DET arnved on site The conclusions of that inspection presented a somewhat different view of the problems 28 3 Those conclusions included the following a "The boric acid corrosion prevention program appropriately addressed. with one 5

exception, the program critena articulated in Generic Letter 88-05. The exception 6 pertained to the absence ofguidance on engineering evaluation methods to be used 7

in determining the impact of identined leakage on the reactor coolant system 8 boundary.

9 "The program appeared to effectively provide for ongoing surveillance and to corrective maintenance of identi6ed leakage, but did not address responsibilities 11 and methods for detection and evaluation of recurring leakage conditions."

12 The NRC's inspection identified certain speci6c issues, namely, that STP had not issued particular 13 forms called for by an STP procedure for describing evidence ofleakage and had not promptly 14 corrected a situation where there was evidence of a very small amount of leakage. This was 15 caused, in pan, by an error on the part of a system engineer, who mistakenly thcught the manway 16 covers would be removed, inspected, and resealed during each refueling outage. The NRC (m

\

17 inspector also noted inconsistencies between the results ofinspections conducted by different 18 individuals.

19 Libeny concluded that STP had maintained an active inspection program for detecting evidence 20 of primary system leakage and that maintenance had corrected identi6ed leaks. The example cited 21 by the DET, when viewed in the context of all of STP's efforts in this area, was not indicative of 22 a lax effort or a willingness to accept degraded conditions. A system engineer made a logical 23 assumption but one that in hindsight was not correct. The results of this error were not signincant.

I "

t

, V) NRC Inspection Report No. 93-14, ST-AE HL-93371. April 13,1993.

Page 111-11 The Liberty Consulung Group

RL.5 .'

Chapter Three - \laintenance n

i III. Preventive .\laintenance 2 A. STP's Preventive Maintenance Program History 3 The DET report characterized STP's preventive maintenance (PAf) program as "less than fully a etTective" and attributed the shortcomings to " poor development of the Phi program in terms of 5 scope and procedure accuracy that were not properly addressed "

6 STP developed its initial Phi program on the basis of recommendations made by equipment 7 manufacturers. This was a common practice in the nuclear industry but one that proved to be 8 unnecessarily costly and deficient in terms ofits lack of focus on relative importance. STP began 9 its Phi program with about 33,000 tasks. Pan of the reason for the large number is that the 10 manufacturers' recommendations were generic and did not consider the standby status of 11 equipment in nuclear safety systems. This remited in excessive preventive maintenance 12 frequencies for equipment not operated frequently. Another reason is that STP has a three-train 13 design for safety systems and so has more equipment to maintain than other plants that have two 14 safety trains.

('N D'1 15 Because of deferred Phi rates in the 40- to 50-percent range after initial operation of the plant, 16 STP was encouraged by INPO to reduce the number of Phi tasks. The idea was to eliminate those 17 that were excessive or not imponant for safety or reliable operation. The NRC's SALP repon that 18 covered the period from January 1989 to January 1990 recognized STP for this important Phi 19 reduction effort?

20 "In order to focus maintenance initiatives, the licensee implemented a preventive 21 maintenance (Phi) program enhancement plan of action during this assessment 22 period. This program resulted in a reduced and more focused scope of the Phi 23 program and a reduced Phi deferral rate trend."

24 HL&P specifically categorized deferred Phis so that they could be recalled later for review and 25 possible reactivation prior to the date they were originally scheduled to be performed. A second 26 review of Phis was then done by maintenance. This second review included evaluation criteria 27 related to performance of the equipment in service and was an opportunity for STP to reconsider O DET Report, p.17.

i N " NRC SALP Report, ST-AE HL-92567 June 26,1990, p 12.

Page IU-12 The Liberty Consuhmg Group l

RL%.'

Chapter Three - \laintenance I the imponance ranking previously established for classifying PNis as active or inactive The review was appropriately performed by the Responsible hiaintenance Authority 6df4) in the 3 N!aintenance Department.

4 The DET attributed cenain equipment failures to lack of Phi This led to the DET's criticism that 5 the second review for reactivation of low priority Phis was not performed by personnel qualified 6 to make that determination. The DET said?

? "[T]he only review performed to determine which individual Phi tasks would be j 8 classified as inactive or active, was a non-technical one by maintenance 9 personnel."

to This statement was incorrect. The original technical review that determined Phi task priority used 1I the importance of the Phi to equipment reliability and plant safety as the basis for judgment. Low

. 12 priority Phis were made inactive in November 1988? The subsequent maintenance review 13 referred to by the DET occurred just prior to the scheduled conduct of Phis and was a check of 14 inactivated low priority Phis to determine whether any Phi' task had been inappropriately p 15 classified or whether maintenance had reason to perform the inactivated Phi on the basis of

( 16 operating experience, maintenance history, cost and time to perform, and a number of other 17 factors. Phis thought to have been inappropriately classiEed and determined to be of a higher 18 priority were reactivated by maintenance or became candidates for reactivation after funher 19 review. This was a reasonable process. The DET's commerit implied that (1) the second technical 20 review should have been performed by someone outside of maintenance and (2) the review could 21 result in the inactivation of high priority Phis. That was not the case The review was only of Ph!

22 tasks that had been assigned a low priority during a prior technical review and represented a 23 second chance to review the appropriateness of task inactivation.

24 The initial Phi reduction program was a simple and expedient way of dealing with a dilemma of 25 significant proportion. The Phi deferral rate was 40 to 50 percent, and some of the PNis on 26 imponant equipment had no specific importance ranking. The importance classification by STP i i

27 was a reasonable method used by qualified people but did not attempt to refine any of the vendor- l 28 recommended tasks or add any tasks that might have been overlooked by the vendors. It was a l 29 reasonable action to take to meet specific immediate objectives and was recognized as such by

" DET Report, p.17.

CI " Memorandum ST-PZ-HS 1301, Jump to Maintenance Division Managers. November 15.1988.

Page III-13 l The Liberrv Consulnng Group '

1

lus:

Chapter Three - \laintenance 1

NRC in their S ALP review. but it was never presumed to be a long-term substitute for a well.

2 developed and refined PM Program 3

The DET report also suggested that the PM feedback forms were the principal or only mechanism 4

intended to refine the PM Program, and that the backlog of PM feedbacks scheduled for review 5

was the principal impediment to enhancement of the Phi tasks that comprised the program The 6 DET said?

7 "The method for improving the PM program involved the use of PM " feedback" 8

forms to identify errors and refinements for incorporation into the program.

9 However, since 1991 a large backlog of PM feedback forms had accumulated."

10 PM feedbacks were only one element in the PM refinement process and did not affect the issues 11 raised by the DET regarding potentially overlooked or missing PMs for equipment important to 12 safety.

13 The PM feedback concept was implemented at STP to refine and correct what was specified to 14 be done in the vendor-based PM tasks. It refined work steps, corrected or added references to

'( / 15 procedures, modified parts lists, and served a useful purpose in optimizing the work process for 16 the PM task. It was not intended to optimize PM by identifying needed additional PM on 17 equipment not receiving PM, did not deve.Iop the appropriate PM task type (predictive, preventive, 18 periodic, or run to failure), and did not serve to measure the risk, cost, or consequences of PM or 19 the lack of PM.

20 B. Industry Perspective 21 In order to adequately evaluate STP's PM program, it is necessary to understand the broader 22 industry context in which that program has developed. The refinement of PM programs continues 23 to pose a complex challenge to the entire nuclear industry.

24 After the 1979 accident at Three Mile Island, the NRC became increasingly aware of the 25 importance of utilities' maintenance programs. Studies completed by the NRC in 1984 found that p

DET Repon, p.18 Page 111-14 The Liberry Consulnng Group

RL%.'

Chapter Three - \taintenance

( 1 maintenance was the root cause of 39 percent of safety-related events? In the following year the 2 NRC identified maintenance as a management issue and initiated a formal maintenance status j study of the industry. NUMARC (Nuclear Utilities h!anagement and Resources Council) and 4 EPRI (Electric Power Research Institute) began to focus their attention on the maintenance issue 5 as well What became evident was that a systematic process for identifying Phi tasks had not been 6 developed, and that the range and depth of Phi programs varied widely throughout the industry.

• 7 By 1989, when STP recognized that a vendor manual-based Phi program was unworkable, the 8 industry in general had reached a similar conclusion. Following prototype studies that had begun 9 in 1983 EPRI launched large-scale demonstration projects in 1988 at two utilities to define a new I

to process and practical approach to the optimization of Pht. STP's program at the time was typical 11 of Phi programs throughout the industry in that it was vendor-based and relied on experience, 12 judgment, and failure experience to develop improvements.

13 STP's management had been following the EPRI large-scale, reliability-centered maintenance la (RCM) demonstration projects and other RChi pilot project efforts started at other utilities and 15 recognized the need for a more systematic way to define the tasks that comprise the Phi program.

l 16 STP recognized that the RChi process could replace the reactivation review of the Phis that had i

17 been deferred and, in addition, could provide a way to logically identify where predictive l

a 18 maintenance should be used. This led to an evaluation of another utility's component-based Phi 19 program in 1990 and to STP's RChi pilot programs on manual valves and instrument air dryers.

20 The RChi process is time-consuming and resource-intensive, and STP's pilot project approach 21 was typical of how the industry was embracing the new idea. STP's implementation of the RChi 22 process was cautious because industry results were initially mixed, depending on the approach 23 used. One of EPRI's large-scale, two-year demonstration projects, an apparent success right up -

24 through completion, was transferred to maintenance from engineering following completion in 25 1990, the basic analytical process altered, and much of the work redone. hiany other utility 26 programs also faltered for a variety of reasons.

27 Everi today, with the EPRI large-scale demonstration projects completed, few if any of the nuclear 28 utilities have completed RChi or reliability-based PM programs and fully implemented the results.

l 29 The process is rigorous and has now been complicated by requirements of the new NRC 30 hiaintenance Rule. STP's vendor-based Phi program is typical of many industry programs in V) GAO Report, GAO'RCED 91-36,"NRC's Effetts to Ensure Effective Plant Maintenance are Incomplete?

December 1990, p. 8.

Page 111-15

) The Liberty Consulnng Group

l l

PLS.' .

Chapter Three - %!aintenance i transition to a risk and-reliability approach, in which the sersice requirements of the equipment. i 2 their dominant failure modes. the consequences of failure, and the most cost-effective way to 3 forewarn or prevent the occurrence of the failure determine the type and frequency of Pht.

4 C. STP's PM Program Development 5 In April 1991, in an effort to better match PM program enhancement needs with staff skills and 6 expertise. STP transferred maintenance engineering staff and responsibility for the Ph! program 7 to engineering. In engineering, system engineers with system performance and safety risk expertise 8 could use their knowledge to apply the more sophisticated RCM PM task development 9

methodology to refine STP's PM Program. Predictive maintenance program responsibility, already 10 in engineering, is integrally related to results of the RCM program and needs the RCM evaluation ii results to define the best applications for predictive maintenance. The system engineer also had ,

12 a personal incentive to maintain his system because of the ownership concept that is part of the 13 system engineer philosophy. In November 1991, to further focus the preventive maintenance 14 program effort and consolidate the various programs, engineering formed the Reliability ,

t5 Engineering Group. The system engineers, used as key resources by the Reliability Engineering

\

16 Group, would organize the various initiatives and refine the PM program. Engineering started 17 RCM studies in April 1992 on the Seal Oil System. Performing this task with its own staff, STP 18 gained first-hand knowledge of the level of effort required and how to judge the quality of results.

19 Considering industry experience with faltering programs and their expense, STP gained the 20 knowledge necessary to proceed with the larger analysis of all important plant systems.

t 21 Budget approval was obtained in late 1992, and HL&P hired contractors to work on the PM 22 program in April 1993. The effort is ongoing with project plans outlined in HL&P's 1994-1998 23 Business Plan.

i 24 STP moved more slowly than the industry leaders in RCM for a number of valid reasons:

25 o to avoid and learn from the mistakes made by others, 26 o to take advantage of RCM software development by EPRI and others, 27 o to take advantage of generic solutions and techniques developed by others, and '

28 o to ensure consistency with requirements of the NRC's Maintenance Rule.

O Page 111-16 The Liberty Consultmg Group l

l l

l

RL%2 Chapter Th ree - Maintenance

/*

l

( STP has now completed RCM analyses of several systems and has.a plan in place to complete ,

l 2 analysis of all the major systems by 1996. This effort is. integrated with STP's Maintenance Rule 3 Implementation Plan, its Probabilistic Risk Assessment (P/M) and Individual Plant Evaluation

~

4 ~(/PE) programs, and its effons to enhanc'e the 'use of predictive maintenanc'e. Each of these l

~

5 programs was'initiat'ed well before the DET's review.'

6 While RCM development efforts by'STP were being initiated,'.and responsibil'ity for.the.

7 maintenance program was being . shifted to engineering from maintenance, the NRC issued the .

8' Maintenance Rule on July 10,1991: This complicated the issue of how RCM analyses sh'ould be 9 ped'ormed and added new maintenance program responsibilities. By June 1992, NUMARC and 10 NRC had agreed on NUMARC's guideline for implementation of the Maintenance Rule and begun

]

11 a verification and validation effort to ensure that application of the Rule through the NUMARC 12 guideline would have the expected and desired results. STP monitored these activities closely. On 13 July 29,1992, one month afterNUMARC and NRC had ' reached agreement o'n application'of the 14 Rule, STP created the position of Maintenance Rule Program Manager and assigned the Manager 15 of Design Engineering to the position. HL&P responded to the new rule, and developed an action 16 plan for compliance and issued the plan in Febmary 1993, one month after receiving NUMARC's

,C% 17 updated and revised industry guidelines.

t U

18 STP is actively pursuing its RCM program, taking initial actions in accordance with its action plan 19 for the new Maintenance Rule, continuing its evaluation of PM Feedback Forms, integrating 20 results of its PRA and IPE Programs into the Maintenance Rule and RCM Programs, revisiting 21 inactive PMs for activation by engineering, and building its predictive maintenance capability for 22 applications defmed by new PM tasks resulting from the RCM Program. Actions taken in the past 23 to enhance or refme the PM tasks were reasonable. Development and implementation of a well-24 dermed, risk-and-reliability-based PM program will take significant time and effort, as it has for 25 other utilities. In recognition of the time and effon required for implementation of the new 26 Maintenance Rule, the NR'C has allowed five years from the date ofinitial issue on July 10,1991 27 for utilities to be in compliance.

28 D. Examples Used by the DET L 29 HL&P used engineeringjudgment and analysis to determine the relative importance of all of the 30 PM tasks originally included it its PM program. The existence of high PM deferral rates and

/%

( ) 31 missed PMs during operations in the late 1980s made it obvious that all of the PMs initially called N _./

Page III-17 The Liberty Consulting Group

RLS-2 Chapter Three-Maintenance 1 for could not reasonably be done, and the lower priority PMs were inactivated. The fact that this )

2 prioritization was performed was appropriate.

3 At the request of the DET, STP provided a list of SPRs that were associated with PM." Then, 4 using hindsight, the DET criticized STP's PM prioritization:"

5 "As a result of not performing these inactive PM tasks, the following preventable 6 events, equipment failures, and instances of poor assurance of operability (mostly 7 dealing with instmment calibrations) occurred."

8 Whether or not the DET's specific examples demonstrated that deferred PMs led to equipment 9 failures, it is important to recognize that the best PM program cannot guarantee that failures will 10 not occur. Specifying which PM tasks are required to safely and cost-effectively operate the plant 11 is a complicated process. Learning from equipment failure and taking action to prevent its 12 repetition are still part of the industry's PM program development and refinement. The DET did 13 not appear to take these factors into account in its evaluation. Many factors must be considered 14 by management in determining the speed with which the PM program is developed and PM tasks O 'S more precisely defined. Liberty concluded that STP's decisions and actions related to the PM .

i 16 program were reasonable and that the examples provided in the DET report did not indicate that i 17 management had made unreasonable decisions or taken unreasonable action in connection with i 18 the PM program.

A

DET sesponse item #3671.

DET Report, p.17.

Page III-18 The Libery Consulting Group

RLN-2 Chapter Th ree - Maintenance l

Vi IV. Maintenance Training 2 A. Introduction 3 The DET report included very broad criticisms of.STP's maintenance. training 8 4 "The training program established for maintenance craft pers,onnel was deficient.

5 T'his . contributed to ' numerous' instances of ineffective'niaintenan'ce'.and poor - ,

6 equipment performance.' Key maintenance support personnel such as maintenance

. 7 ' planners and procedure writers only rdceived limited formal technical trai'ning. The '

8 team noted .the procedures did .not include.enough detail an'd cautions to i 9 compensate for training deficiencies.""

10 However, the DET's view contrasted sharply with prior NRC assessments. For example, in May 11 1991, the NRC's SALP report included a generally positive appraisal: 33 12 " Inspections of routine maintenance and surveillance activities identified well 13 trained personnel. Training in the self verification process was strongly 14 emphasized.. . Overall maintenance training was considered good, but the licensee p 15 found that OJT requirements were not being uniformly implemented because of

('- 16 a lack of understanding of the requirements by maintenance department 17 personnel."

18 The NRC's SALP assessment of maintenance training in late 1992 was equally positive:32 19 "The licensee's preventive and corrective maintenance programs were considered 20 good. Several strengths were identified. The specific training given to 21 maintenance personnel on work processes was good, and the workers were suitably 22 tested to demonstrate their knowledge."

" DET Report, p.18.

" NRC SALP Report, ST AE-lE 92831, September 6,1991, p.13.

\d " NRC SALP Report, ST-AE-lE-93239, November 18,1992,p.9.

Page UI-19 The Liberty Consuhing Group

RLS-2 Chapter Three- Maintenance

/ s l

! I V i B. Maintenance Training Issues .)

2 1. Accreditation 3 The primary basis for the DET's criticism of maintenance training appeared to have been the 4 placement on temporary probation of three of STP's training programs. The key issue in this 5 matter was the method STP had used to allow experienced new-hire maintenance craftsmen to 6 meet journeyman qualification requirements.

7 The DET reported that "[i]n mid-1992, an industry organization determined the licensee's basic 8 maintenance craft skills training program was deficient."32 This was an issue that HL&P 9 management had recognized and addressed well before the DET arrived at STP. In fact, most of 10 the necessary corrective actions were initiated by HL&P in 1992 and completed by July 1993.24 11 HL&P's training program received full accreditation in July 1990.35 The issue that the DET 12 referred to involved HL&P's temporary loss ofINPO's National Academy for Nuclear Training 13 accreditation for its mechanical, electrical, and instrumentation and controls (I&C) training

'4 programs. The central issue was the method HL&P had used to qualify maintenance craftsmen as

~)

15 journeymen on the basis of their experience. The issue did not involve the fundamental quality of 16 STP's training programs, its training methods, or the additional knowledge gained by those being 17 trained. Three training programs were placed on probation because experienced new hire 18 joumeymen had been qualified without a performance-based assessment of their knowledge and 19 skills.

20 Many early STP maintenance department hires were part of the STP contractor construction work 21 force with startup and maintenance experience. HL&P thought that signed statements by these 22 experienced craftsmen on a Qualification Record (QR) form attesting to their job and task 23 competency was an acceptable and reasonable way to satisfy qualification requirements. This 24 - process allowed mai.ntenance craftsmen with prior STP constmetion'and startup experienc.e to '

~

25 achievejoumeyman qualification on the basis of their work experience as part of the construction' l

" DET Report, p.18.

" Accreditation Package, STT'EGS Presentation to NNAB, June 17,1993;IIL&P letter to NRC, ST-IIL-AE- l g

4295, January 14,1993.-

G " Letter, INPO to llL&P, Pate to Jordan, July 20,1990 Page III-20 ,

The Liberty Consulting Gmup

i l

RL%2 Chapter Three - Maintenance m

l T V

I 1

work force. In addition, STP hired other maintenance craftsmen who did not have prior STP work 2 experience and permitted them to meet basic joumeyman qualification requirements on the basis 3

of the results of technical interviews to determine their knowledge and skills and 'their entries on 4 the QR forms:These practices were thought to be reasonable by STP's management, and STP's

' ' ~

5 training programs receivediull accredit'ationby~the National Nuclear Accreditation Board (NNAB) 6 while they were in place.

~

7 In the original 1990 industry accreditation review report, the criterion for training exemptions and

. '8 . qualification vias: .

9 "1.6 Training to be completed prior to qualification is clearly defined. Exemptions 10 from training may be granted when justified and supported by a documented 11 assessment.of prior training and experience."

12 STP's management believed their practices were in compliance with this criterion. In evaluating 13 HL&P's overall train;ng program compliance with stated criteria for initial accreditation in 1990, 87 14 the Accreditation Review Team stated:

x 15 "A review of training records for individuals receiving exemptions from training 16 indicated that the basis for granting exemptions is well documented. Approved 17 exemptions include appropriate documentation of applicable experience, training, 18 or satisfactorily passing an equivalent written examination."

19 INPO documents that provided guidance to utilities regarding qualification and that were in effect 20 at the time STP received its initial accreditation also contained the following definition of 21 experience for the purposes of qualification:'"

22 "4.2 Excerience 23 The experience gained in complying with Section 4.3 Training. and 24 demonstration of the job / task competencies. The time required to become 25 a fully qualified craftsman will vary, dependent upon the utility's methods 26 of implementing the training and job / task competency demonstration 27 programs"

" Accreditation Report, STPEGS, Review con i m March 26-30,1990, p. EIM l-8.

/O s s' Accreditation Report, STPEGS, Review CorJuewd Mrrch 26-30,1990, p.1-14.

" INPO GPG-85, Revision 0, Guidelines for Mechanical Maintenance Personnel Qualification, July 31,1981.

Page 11121 .

The Liberty Consuking Group

PlS.2 Chapter Three- Maintenance O

V1 The need to determine the experience of the applicants for journeyman positions through a )

2 performance-based assessment of their knowledge and skills was not clear or well understood until 3 the events of mid-1992, when accreditation was temporarily suspended. " Demonstration of' 4 job / task competencies" was interpreted at STP to include r6 sum 6 information, knowledge of prior 5 work experience, technical interviews, and statements made by the experienced craftsmen on QR 6 forms.

7 INPO's current requirements for qualincation are much more clearly stated in a new guidance 8 document that was issued in May 1992, the same time thatjourneyman qualification practices 9 were being reexamined at STP. The requirements for quali5 cation on the basis of experience are 10 contained in sections 2.3 and 2.4 and include:"

11 "Section 2.3: Qualification to perform a job task or function independently 12 requires line manager approval. While some training may be exempted, task 13 performance as part of qualiEcation prior to assignment to complete a job 14 independently should not be exempted 15 "Section 2.4: . Experienced personnel may be considered for exemption from

'6 prerequisite training. Exemption from training requirements should not include

\._/ 17 exemption from qualiDeation requirements. Line manager review of an 18 individual's prior training and job performance history provides data for this 19 exemption. This review should consist of one or both of the following, or an 20 equivalent process:

21 -

an objective, technical. interview of the indiv.idual to determine..

22 work experience history, quali6 cations, and training - This process 23 validates resume history and may include contact with previous 24 work supervisors.

25 -

a review ~of previous training records and job performa'nce history. .

26 to determine training re'ceived and. work accomplish ~ed: -

27' "To Veri'fy tliat the. individual possesses' hd' equate knoviledge and . skill $,- the '

28 following should be conducted:

29 +

an examination. based on'th5 learning objectives being exempted -

~

30 It is not necessary' to test all learning' objectives.' Input'frohi the 31 training and experience review should be used to determine the 32 scope of the examination.

' " INPO Doctunent ACAD 92-004, May 1992.

Page III-22 The Liberty Consulting Group

l 1

RLS 2 Chapter Three - Maintenance

( l .- an evaluation to verify proficiency of skills necessary to perform ,

2 the tasks that are being exempted - This evaluation may be based 3 on the candidate performing or having performed the task 4 satisfactorily, or it may be a technical " walk-through" or " talk-5 through" of the~ tasks. Task perfo.rmance is'the preferred meth'od."

6 These new requirements are much more specific than those contained in.the INPO guidance 7- documents that preceded thlem. Th'e processes to be'used for.meciing qualification requirements 8 on the basis of exper.'ence are more detailed. The lack of specificity in prior guidance documents '

9 helps to explain.why BTP thought it was acting reasonably in using the practices th.at it did .

10 Even the new guidance provided by INPO contains some flexibility, although it states that task 11 performance is the preferred method of qualification. Notwithstanding this flexibility in the 12 methods, STP fully understood INPO preferences and expectations in 1992 and began qualifying

~

13 its journeymen accordingly.

14 Following INPO's May 1992 evaluation, HL&P conducted a comprehensive assessment and 15 developed an action plan. Results and actions planned were identified to the National Academy 16 for Nuclear Training in July 1992' and later to the NRC in a follow-up progress report on actions

\ taken in response to the most recent SALP review (even though no maintenance training problem 17 18 had been identified in the SALP report).' And while accreditation for three of STP's maintenance 19 training programs was temporarily placed on probation because of the issue,42 a prompt and to complete response by STP resulted in probation removal and full reinstatement of accreditation 21 by the NNAB on June 17,1993.'$

22 It may be a logical conclusion based on hindsight that STP's qualification program was a root 23 cause of workmanship and performance errors occurring in the workplace. However, none of the 24 contemporary documents listed journeyman qualification and training as a cause and there was 25 no discernible connection between the two. STP had first identified weaknesses with journeyman 26 skills in November 1990 and was taking a number of actions to better implement its work 27 direction procedures forjob oversight when journeymen performing tasks were not qualified for

" Letter, HL&P to National Academy for Nuclear Training, D.P. Hall to Terence Sullivan, July 15,1992.

Letter, IIL&P to NRC, ST-HL-AE-4295, January 14,1993, d'

National Nuclear Accreditation Board letter to HL&P, T. Jordan, November 23,1992.

Letter, HL&P to INPO, ITN 93 165, August 31,1993, p. 7.

Page III-23 The Liberty Consulting Gmuy

RIS 2 Chapter Three- Maintenance 1 independent work. These actions were believed by management to be reasonable and cost- ) ;

2 effective and were having some success."

3 3 Because of the NNAB's requirement for a performance-based assessment of qualification and 4 skills and the subsequent action taken with regard to accreditation of three of STP's training 5 programs, HL&P reassessed all journeyman and apprentice training program exemptions and ,

6 redetermined qualifications using performance-based evaluations and testing techniques.

7 The actions taken by HL&P in response to the NNAB's position on performance-based assessment 8 of qualifications and skills staned in 1992 and included:"

9 1. revision of the basic qualification record cards (GR) to improve accuracy and 10 applicability 11 2. reperform:w e of technical interviews with each craftjourneyman and crew leader, 12 using the revised basic certification forms to determine the need for training 13 3. implementation of maintenance proficiency testing, using performance-based skill G 14 evaluations for all incumbents and new hires 15 4. implementation of a maintenance task qualification process that ensures that only 16 qualified maintenance workers are assigned to appropriate tasks, and that only 17 .qualifiedjoumeymen are assigned to perforn$ or supervise specialty and advanced 18 tasks independently 19 5. conduct of remedial training for all journeymen as necessary on the basis of 20 technical interviews and performance-based skill evaluations.

21 By May 1993, HL&P had c'ompleted maintenan'ce proficiency eval.uations'(AEE) for.98 percent

, . 22 _ of the' direct-hire mechanical ma[ntenancejourneymen,,100 percent of the. ele'ctrical maintenance '

~ '

, 23 ' journeymen,100 p'erc'ent of the'I&C computs technicidris, aNd (peicent-of the I&C process' < G 24 tech'nicians. All proficiency evaluations were completed by the end of June 1993.

s a

s .'. ,

DET response item #0040-002, Memorandum, R.J. Rchkugler to W.ll. Kinsey/D.J. Denver, November 10,

' 1992.

f - .

( " Accreditation Package, STPEGS Presentation to NN AB. June 17,1993, and iILAP 1.cticr to NitC, ST-1II.-

AE-4295, January 14,1993.

Page HI 24 The Liberty Consulting Group

i RU-2 Chapter Th ree - Maintenance V 1 By June 10,1993, the date.the DET report was issued, HL&P had determined the additional 2- training needs for all incumbent journeymen and'had begun the additional training.".NNAB 3 accreditation probation was removed one week later on June 17,1993. All maintenance work' 4 performed after' June 1993 was performed by requalified maintenance personnel or.was performed 5 under the supervision of qualified personnel.

-6 ~ Liberty concluded that the D5T's portrayal of training program deficiencies as an.is'stie of concern 7 confused past issues with current capabilities. HL&P had effectively impleme' nted needed changes

8. .in the maintenancejoumeyman qualifica' tion' process 'by the time ~ the DET report 'was issued. The i- 9 DET report did no't document the source and ' nature of the training and qualification deficiencies 10 and the comprehensive actions taken by HL&P management to remedy the situation. Furthermore, 11 the DET report could be misinterpreted to imply that the issue was a current one that had not been 12 ' acted on by STP's management: .,

.> 13 "Beginning in 1990 and continuing well into 1993, SPEAKOUT, the QA program, 14 and the training department manager alerted senior management that there were 15 serious problems in maintenance training, maintenance qualifications, and (O

( ,)

16 17 maintenance performance. Management failed to effectively address the root causes and the corrective actions recommended by QA in these areas, even when

, 18 advised that unacceptable practices still existed. The result was a growing backlog 19 of work, reduced plant reliability, and an extended unplanned Outage."

20 The maintenance training, maintenance qualification, and maintenance performance issues 21 apparently referred to by the DET in this statement involved the methods STP had used to 22 determinejoumeyman qualification status and the difficulties it had experienced in implementing 23 work direction procedures for unqualified journeymen. These difficulties had been experienced, 24 but it was not accurate to say that "[m]anagement failed to effectively address the root causes and 25 the corrective actions recommended by QA in these areas, even when advised that unacceptable 26 practices still existed." The following discussion addresses this point.

(#) " STP Station Report, June 1993, p. J-6.

U

DET Report, p. 47.

Page LU-25 The Liberty Consulting Group

i l

\

R13-2 l Chapter Three - Maintenance 1 2. Journeyman Qualification and Use of QR Form )

2 The issue regarding the QR form originated with an October 1991 SPEAKOUT Concern." STP 3 made a commitment to the NRC to implement the QR form instead of a qualification matrix for 4 documentation of worker qualifications and continued OJT (on-the-job training) qualification. The 5 QR form listed tasks that the worker could attest to being qualified to perform. Worker concern 6 over ambiguity as to how much expertise was expected was the issue.

7 STP held meetings with workers in early 1990 to clarify the conditions under which the workers 8 would be expected to perform the tasks, revised and clarified the QR Forms in November 1991, 9 and then later issued another QR fonn with the additional clarifications.

10 The SPEAKOUT Concern was substantiated, but SPEAKOUT detennined that actions had been 11 taken by maintenance management to correct the situation. A second SPEAKOUT Concern was 12 raised in May 1992, but investigations revealed that the concerns were unsubstantiated."

13 STP believed the matter had been resolved by the actions taken, and neither QA nor any other (O

d4 1 independent internal or external assessment group identified the QR Form and its potential for overstating true worker skill or capability as a problem until the issue was raised by INPO and the

)

15

1. National Academy for Nuclear Training. STP's management took the actions it thought reasonable 17 in response to the SPEAKOUT Concerns. The basis for these actions was judgment as to the 18 significance of the situation, and this in turn was influenced by the fact that the issue had not been l'9 raised by any other independent internal or exte'rnal assessments. STP's actions were reasonable.

20 3. Work Direction of Unqualified Personnel 21 In employing the process of worker qualification through'OJT and OJE (on-the-job' experience)' . , ,

22 STP experienced difficulties in de'termining the amount of direct supervision (work direction) 23 required and in providing work .directiori orjob supervisi.on 100 percent of the time that it was '

24 required. The issue originated with a' SPEAKOUT Concern raised in November 1990 and'had 25 been the subject of a number of other SPEAKOUT Concerns, QA Audits; SP.Rs, and DRs? In-

" SPEAKOUT Concern 12178 - October 1991, action summarized in DET response item #0040-002  !

r " SPEAKOUT Concern 12283 May 1992, addressed in DET response dem #004'0'-002.

(

v "

1 DET response item #0040-002, p.13.

Page III-26 The 1.iberty Conwitmg Group

l Jus 2 i Chapter Th ree - Maintenance l

O k I response to the initial November 1990 SPEAKOUT Concern, QA verified corrective action to 2 ' ensure that workers understood and properly documented. work direction provided.5:

3' During the QA'Atidit performed in January 1991, the audit team identified weaknesses in erisuring

~

^

4 that only qualified personnel performed maintenance tasks independently, and that work packages 5 adequately , documented the work direction provided. hiaintenance revised the'STP procedure

~ ~

6 covering the ~OJT/ Qualification Program,EIP-8.18, to stren'gthen ' controls'in thi's area.- QA 7 performed a follow-up audit in 1992 and did not identify any continuing work direction problems.

8 ' Personnel performing tasks independently were either qualified or given properwork direction.52 9 In hfarch 1991, SPEAKOUT identified that certain personnel had been observed working without 10 adequate work direction, as required or intended by IP-8.18. A case of OJT documentation 11' practices in violation ofIP-8.18 was also identified. Corrective actions by management included 12 issuance of additional instmetion and guidance on work direction, strengthened policies for 13 uniform distribution of qualified personnel on each shift, and training on the importance of giving 14 work direction integrity priority over work schedules."

15 Later,in September 1991, QA issued a deficiency report to document a case of failure to provide 16 Work direction when a judgment was made by supervision that a pre-job briefing was adequate 17 for the particular task at hand. This led to corrective action in the form of revisions to two 18 procedures governing work implementation, qualification and certification of personnel, and 19 counseling of all maintenance department personnel by the hiechanical hiaintenance hianager.

20 In each of the cases described above, STP took prompt action to improve adherence to work 21 direction oversight and documentation requirements. When QA did identify a deficiency in the 22 process, corrective action was taken and a follow-up audit was performed to confirm adequate 23 resolution of the concerns. STP's management responded promptly to weaknesses as they were 24 identified and took reasonable actions to strengthen the work direction process. Liberty found that 25 the implication that no action had been taken as a result of QA findings and recommendations was 26 notjustified.

i DET response item #0040-002, p.13.

" I DET response item #0040-002, p.14.

m ;l f ) "

DET response item #0040-002, Attachment B to Memorandum, Rehkugler to Kinsey and Denver, V November 10,1992, pp. 5-6, i Page 111-27 The Liberty Consulting Group

RLS-2 Chapter Three- Maintenance 1 Fully accredited in 1990, STP's training programs received positive assessments through NRC )

2 SALP and Maintenance Team Inspection (MI7) reviews. It was reasonable for STP's management 3 to believe it was following accredited and approved practices in its initialjourneyman hiring and' 4 qualification procedures. STP's management responded reasonably with corrective action when 5 the need was indicated.

6 Liberty did not agree with the broad negative characterization of STP's work force and training 7 programs because it oversimplified and overstated the issue. The DET's examples of 8 communication or human errors did not support any claim that unreasonable management action 9 in the area of maintenance training 1.ad a significant impact on work backlog, plant reliability, or

~

10 the length of plant outages.

11 C. DET Examples 12 1. Molded Case Circuit Breakers p 13' The DET used an example involving molded case circuit breakers to support two different

\ 14 conclusions. However, the information available to the DET was preliminary. After the matter was -

15 fully investigated, it was learned that there was no safety significance to the physical problem 16 associated with the breakers. While the matter pointed out programmatic areas that could be 17 improved, its seriousness was overstated when only the preliminary information was available.

18 More specifically, the DET at one point stated that seven components had been rendered

. . ~

19 inoperable by the circuit breaker situation. In another part of the report, the DET stated that there 20 had been operability concerns about 10 breakers that provided power to motor-operated valves 21 (MOVs). In fact, no equipment was made inoperable due to the circuit breaker issue.

22 The DET report included the following:" ' , -

i 23' ' "The team noted the following deficiencies in the basic craft trainingjknowleilgeC 24 "The training for molded case circuit breakers did not include the correct method 25 for determining the breaker settings based on.the values (amperes) proyided in the 26 setpoint document. This' lack of training'and the complex procedural instructions' 27 for determining the breaker settings resulted in incorrect breaker settings rendering 28 .seven safety-related components inoperable." . ,

DET Report, p.19.

l Page III-28 The Liberty Comulting Group

W.&2 Chapter Th ree - Maintenance 1 The report used the same example to support its conclusion about engineering's configuration 2 control. It stated:"

.3 " Examples.of configuration control weaknesses include the following:

.4 ~"The Electrical Setpoint Index f'o'r molded case circuit breakers was' not properly 5 understood or impleme~nted in the field, resulting in operability concems.'While l 6 performing maintenance on molded case circuii breakers, the licensee discovered '

7 that (magnetic) trip settings were adjusted using the electrical penetration test point 8- calculations for permissible currents rather than trip values obtained from the

~ ~

9 index. The licensee later determined that the instantaneous trip magnetic settlngs 10 were improperly adjusted for approximately 30 breakers in Units 1 and 2. The 11 licensee found operability concerns with 10 breakers powering MOVs such as 12 containment and accumulator isolation valves. This condition may have existed 13 since startup. Although the index contained appropriate criteria, the licensee had

~

14 not prepared detailed work or procedural instmettons fo'r craft personnel to use in 15 interpreting or scaling the index guidance."

l 16 HL&P discovered a potential misapplication of the instructions contained in a document used, in 17 part, to establish the trip setpoints of a certain class of electrical circuit breakers. After a 18 preliminary evaluation and a complete walkdown of all the breakers, which was completed on i

19 March 31,1993, HL&P determined that a total of 23 breakers (out of a total population of 562 20 breaken of the same type) were set improperly, but that only 12 loads fed by these breakers might 21 need to be evaluated further for operability considerations. As a conservative measure, the 12 22 loads fed by these breakers were initially classified by STP as inoperable."

23 An engineering evaluation of those 12 breakers and associated loads ultimately showed that all I 24 loads were operable. One of the 12 breakers was found to have been set correctly, but a 25 typographical error in the setpoint index initially gave the impression of an incorrect setting. The 26 remaining 11 breakers were actually set inconectly, but the margin between the actual setting and 27 the locked-rotor (maximum) current of the associated load was substantial. Therefore, even with 28 a trip setting lower than optimum, the breaker would not have prevented the attached MOVs from 29 operating as designed.87

" DET Report, pp. 33 34.

" LER 93 012, Revision 1 Unit 1, July 13,1993, p. 2.

" LER 93-012, Revision 1. Unit 1, July 13,1993, pp. 5-6.

Page Ill-29 1he Liberty Consulting Group

RIS 2 l Chapter Three-Maintenance l l

r% 1 1 HL&P adjusted the breaker settings, clarified the procedure used to test the circuit breaker trip )

2 settings, and conducted training on the test methodology." While these corrective actions were 3 necessary, they were not required to ensure equipment operability. HL&P discovered this l 4 situation, analyzed its significance, and performed the corrective actions on its own. This issue did 5 not support the conclusion that STP's management made unreasonable decisions or that l

6 unreasonable actions were taken in connection with STP's training program for maintenance craft 7 personnel.

8 2. Other DET Examples 9 Other instances cited by the DET as examples of poor maintenance training included:"

10 "I&C technicians introduced air into essential chillers and flooded a control panel 1I with oil due to a lack ofunderstanding of how the chillers function under vacuum.

12 This contributed to degraded equipment performance and lack of equipment 13 availability."

g 14 The air introduction event occurred in 1990 during instrument calibration. The issue involved i

Q 15 communications between I&C maintenance craftsmen performing the calibration.' The corrective 16 action planned was a review of work instructions for needed improvement. Liberty concluded that 17 this three year old example did not show that management was taking unreasonable actions 18 regarding its training program in 1993.

19 The DET also said:'i 20 " Craft personnel were not trained on the need to expeditiously place battery 21 chargers into service after performing discharge testing of 125 VDC station 22 batteries. This lack of training and the omission from the testing procedure of this .,

23 critical element of battery testing could haveiesulted in perman.ent damage to the 24 .' station batteries." .

LER 93-012, Revision 1, Unit 1, July 13,1993, p. 6..

" DET Report, p.19.

i O

\

" -DEhesponse item #3023, pp.1245-1247, Chiller Task Force Niceting Minutes, August 3,1990.

DET Report, p.19.

Page 111 30 j The 1.iberty Consutung Group l

l

\

1 8

RL%2 Chapter Threc ~ Maintenance m

1 Liberty found that the STP Training Department lesson plans on battery maintenance specifically 2 included the requirement to " ensure that a battery equalize charge has been initiated following the 3 discharge test."62 The discharge test data sheet also containsd the requirement.

4 Iri ~the examples cited as showing l' imitations in specialized tr'ainirig, th'e'DET f'ocused on the-

5. ,TDAFWP.overspeed trip mechanism, the reactor coolant pump motors, the SDG governor and voltage. regulator,;and the~ security systemi No specific consequences were 'noted, however.

6^ ,

7 Although the DET expressed an opinion that a lack of certain specialized t' raining had contributed .

8. . to the numerous unsuccessful attempts to resolve' difficulties 'with the TDAFWPs, as discussed j 9 later iri this chapter, neither the AIT nor the follow-up' NRC inspection' on the TDAFWP trip 10 issues identified specialized training of any kind as a contributing cause of the overspeed trips or 11 the difficulty in determining the reason for such trips. STP's analysis of the overspeed trips did 12~ not identify specialized training as a contributing factor. Liberty's opinion, which was consistent 13 with the detailed analyses of the trip issues performed by others, was that the event did not 14 indicate a lack of effective specialized training at STP and that the actions taken and decisions 15 made by STP's management in the area of maintenance training were reasonable and prudent. ,

V b

Nucicar Training Department Lesson Plan No. EMT205.04, August 15,1990, p. 32.

Page 111-31 The Liberty Consulting Group

l RIS 2 Chapter Three- Maintenance n

i V. Replacement Parts )

2 A. Availability of Replacement Parts 3 The DET stated that the lack of replacement parts had an adverse impact on equipment 4 performance. Four examples were used to support this conclusion. In the first example the DET 5 highlighted an auxiliary feedwater turbine trip throttle valve leakage deficiency and associated the 6 Occurrence of numerous overspeed trips with the leakage. The overspeed trips of the turbine-7 driven auxiliary feedwater pump were found to be due to a number of contributing root causes, 8 but leakage in the trip throttle valve was not a significant cause. A detailed discussion of that 9 matter and its resolution is included in section IX of this chapter.

10 The second example concerned drains on motor-operated valves. This issue is discussed separately 11 in subsection B below. Another example was a case in which several switches had failed and had 12 been replaced with spare switches from the warehouse. The DET was concerned because if the 13 switches failed again, no replacement parts were in the warehouse. The switches did not fail and 14 there was no effect on equipment performance. This particular concern appeared to be related to l 's the DET's perceptions about the level of stock inventory of this one item. However, inventory -q L 16 management requires judgement as to the proper level of spare parts. Too few parts could affect 17 plant reliability. Too many spare parts represents an unnecessary cost. In this particular case there 18 was no evidence to indicate management had made an unreasonable decision.

19 - Another statement iricluded in the DET report, which attempted to quantify a parts availability 20 problem, asserted a direct connection between parts availability and work backlog. ~

21 "The lack of readily available parts contributed to the size of the maintenance

22. - backlog. ' Approximately . 25. percent of all . non ~ outage .related ' corrective.

'23 maintenance packages were routinely in a parts hold. status."

~

~

, 24 This statement implied that (1) if the parts were available, backlog wodld be redheed, and (2)'all -

25 corrective maintenance work packages in parts-hold status'would be worked if the parts were 26 available. While it is true that parts unavailabilit can dela work, the relationship.betweenlthe;

! 27 ava'ilability of parts, work backlog, and potential safety. impact is not a dirbct one.

l

( )

" DET Report, p.19.

l Page III-32 The Liberty Consulting Group

I RLV2 Chapter Three - Maintenance l

'~i 1 Classification of 25 percent of the work requests as being on hold because of parts did not mean 2 the work'would be performed if the parts were available. This figure related to the ability to.

3 supply parts for all work requests regardless of the date the work was scheduled to be performed.

4 At the time the DET performed its assessment, parts not in inventory were ordered to support tiie . l 5 s'cheduled work' start dates. This put the w~ork in aparts-hold status but did not mean that the work j 6 was being delayed by a lack of parts!

7 Moreover, even a lack of spare parts contributes to the size of.the maintenance backlog only if 8 there is no other work that can be performed. This was not the case at STP, where the non-outage 9 ' work backlog at the end of December 1992 included 3,467 service requests." A job prevented by .

10 an unavailable part simply meant that another job without constraints would be performed. A 11 consultant hired by HL&P in Ame 1991 to evaluate the overall maintenance process (Performance 12 Data, Inc.).reache'd a similar conclusion, finding that the Work Management System (IfMS) 13 indicated that spare parts had not been a source of delays."

14 Spare parts deficiencies existed at STP, as they do at other nuclear plants. For high priority 15 repairs, those that affect plant operability or safety, the issue is an important one. Liberty's 16 examination of STP's LER history for all of 1991 and 1992 revealed that no reportable events had 3 17 been caused by a lack of spare parts. This measure of the adequacy of STP's replacement parts 18 program provided a basis for assessing the results of STP's policies and actions and was consistent 19 with the performance-based assessment approach the NRC has recently implemented for its SALP 20 program and in the new Maintenance Rule.

h 21 While safety had not been adversely affected by any problems then present in determining a 22 source for parts or material availability, HL&P had identified ways to improve its spare parts 23 progmm prior to the DET evaluation. Among these improvements was a policy to procure parts 24 not in stock when the work was first identified, rather than tying the parts order lead time to the 25 date the work requiring the part was scheduled to be done. This action was aimed at reducing the 26 possibility that parts availability would become an issue in the future.

27 Until mid-April 1993 parts requirements for service requests requiring materials not in stock were 28 to be filled when the work was scheduled. Thisjust in time strategy is common practice in the

December 1992 Station Report, DET response item #0018.

Analysis of Maintenance Activities, Performance Data Inc., hme 1991.

Page III-33 The Liberty Consulting Grvup l l

I

l j

RLS-2 1

Chapter Three - Maintenance o

I

\ l industry in general and serves to minimize inventory and cash fiow requirements and thus overall 1 2 operating costs. The policy had been put in place in April 1991 by management when the 3 judgment was made that STP's inventory of stocked parts and supplies was adequate, and that it 4 would be prudent to defer further procurement of spare parts for shelf stock until the master parts 5 list was completed. STP's management believed this would not create a pans availability difficulty 6 because "[s] pare parts support of field work has been satisfactory; no major work has been 7 threatened by a lack of repair parts."" This was clearly one of the reasonable options available to 8 STP's management at the time.

9 The issue of spare parts inventory at STP had been a consideration of management, but ironically, 10 the issue was the large size of the inventory, which was valued at $97.6 million at the end of 1992, 11 not a lack of spare parts. Lack of spare parts was never demonstrated to be a factor that affected 12 plant safety. Acceleration of parts availability can have benefits, although there is a corresponding 13 cost. Liberty concluded that STP's management gave appropriate attention to the size and 14 adequacy of replacement parts. This was an area in which management sought improvements and 15 the optimum cost-benefit balance between inventory costs and parts availability.

O V

16 B. MOV T-Drains 17 The DET discussed the replacement parts program and listed "[e]xamples of unavailable parts 18 which adversely impacted equipment performance."" In one of the examples, the DET stated:"

19 "The lack of parts contributed to valves within the primary containment being 20 inoperable for a year. During the 1991 refuel outage, "T" drains were not available 21 for installation into some new valve motors. Without the "T" drains installed, .

22 moisture could.not drain from the motors and could damage the components.after.

23 an accident. A failure of the work . control system later resulted in.the."T" drains , ,

24 not being installed in'a timely manner."

" Memorandum, D.P. Itall to LB. llorrigan/S.L. Rosen, April 15,1991, DET response item #2160.

DET Report, p. 20.

Dl?T Report, p. 20.

Page 111-34 The 1.iberty C<msuhing Group

l l

RW-2 Chapter Three - Maintenance O

i While there was a deficiency with a single T-drain on one MOV, a further examination of the facts 2 disclosed a much less significant situation than the DET report implied and one in which there had 3 been no adverse impact on equipment performance.

4- Equipment important to safety in a nuclear plant must be capable of operation. iri the environment 5 that could be present if an accident happened T-drains are required on the motor casings for

'6 motor-operated valves (MOVs) inside:co'ntainment t6 ensure that, in the event of a pipe rupture, 7 i moisture'will have a path by which to escape from the motor casing. In November 1990, after the..

8 motor on an MOV was replaced, the correct size T-drain was not available. .The service request

~

9 (SR) for the motor replacement was closed after a new SR was written for installation oithe T-10 drain. The mistake that was made was that the people involved in the preparation and review of 11 this new SR did not recognize that the T-drain was required to maintain the environmental 12 qualification of the valve and thus failed to expedite the installation of the T-drain." Although the 13 MOV operated as required in actual practice, it was technically inoperable because it did not have 14 the required T-drain.

15 The MOV in question was in the Residual Heat Removal (RHR) system. STP has three trains of 16 RHR. Only one train is required to perform the system function. HL&P's analysis of the T-drain 17 matter concluded that "the failure to install the "T" drain in RH-0060B did not have significant 18 impact on the health and safety of the public."7" The issue here was not the reasonableness of 19 decisions made with respect to spare parts, but rather the adequacy of the SR preparation.

20 The DET also used this example to criticize engineering: 7 21 " Examples of configuration control weaknesses include the following:

22 "The licensee did not maintain the environmental qualification of valve actuator 23 motors in containment by installing "T" drains as required by design. A service 24 request submitted in November 1990 to install two "T" drains in Unit 2 train B 25 residual heat removal suction isolation valve was still open during the evaluation.

26 The team requested the licensee to determine which MOVs did not have installed 27 "T" drains. The licensee found five actuator motors that did not have "T" drains.

28 The engineering staff evaluated three of the five, concluded that no action was LER 93-008, Revision 1, Unit 2, July 14,1993.

l p LER 93-008, Revision 1, Unit 2, July 14,1993, p. 3.

t

\ " DET Report, pp. 33 34.

Page 111-35 The Liberty Consulting Group

i RIS-2 '

l Chapter Three- Maintenance

('N 1 required, and was evaluating corrective actions for the remaining two valve ) i 2 actuator motors."

3 The missing T-drain could be considered a conDguration control weakness in that the T-drain 4 should have been installed sooner. However, the documentation of the missing T-drain was clear, 5 so the problem did not reflect on configuration control insofar as that tenn applies to maintaining 6 accurate records of the plant configuration. Furthermore, HL&P performed a 100-percent 7 walkdown of harsh environment MOVs in both units and foutd that while four MOVs had T-drain 8 discrepancies, no other reportable conditions were identified.72 The NRC performed a special 9 inspection of this matter after the DET had completed its work. Its report on this inspection 10 stated:"

11 "The inspector performed visual inspections of a sampling of motor-operated 12 valves, transmitters, and solenoid-operated valves located in harsh environments 13 for the installation of T-drains, grease reliefs, conduit seals, and mounting, as 14 required. The inspector did not identify any discrepancies with any of the 15 components."

r.

( 6 HL&P also perfomied a review of open service requests "to determine if non-conformances that 17 potentially affect operability have been identified and resolved. No additional issues affecting 18 operability were identified."" Clearly, the results of HL&P's inspection and the NRC's follow-up 19 inspection did not point to serious configuration control weaknesses or demonstrate that the 20 actions taken and decisions made by management in this area had been unreasonable.

21 C. Use of Correct Parts 22 The DET attributed instances of incorrect parts use to a defective plant labeling system:and 23 incorrect computerized plant parts d'atabases. The evidence gathered from a study of STP's i,ERs .

.. ~

24 suggested 'thatAis wa.s a minor issue from the standpoint of plant safety: No reportable operational 25 events due to.use of incorre'ct pairts' occurred in th~e 'twd years preced'ngi the plant'shiftdo~wn in 26 February 1993 Over the six-month period fr'om October 21,1962 to April 21,1993, there were

~

27 14 minor occurrences of the use, or attempted.use, ofincorrectl parts.' Interestingly,'none.of these

" LER 93:008, Revision 1, July 14,1993, p. 3.

A 's I

! ' NRC Inspection Report 93-19, ST-AE-IIL-93483, July 23,1993, p.15.  :!

N l

" LER 93-008, Revision 1, July 14,1993, p. 3.  ;)

il Page Ill-36 'l The 1.iberty Consuhing Group .l l

RLS-2 Chapter Three - Maintenance f%

1 14 were traced to the plant labeling system or the parts databases. Nearly half were very minor 2 material discrepancies that were probably left over from construction or were vendor items, and 3 several were due to simple human error in reading parts numbers or in making material use 4 assumptions. Only one resulted in any signi6 cant follow-up action, arid that was . caused by

'5 flushing a hydraulic system with fiuid staged near the~e'quipment and assumed erroneously to b'e 6 - the flushing fluid."

7' The DET report offered two examples. .One involved the installation of a pressure switch in'a.

8 chiller. The correct part was installed but the particular switch had not been qualified as safety-j 9 related. Before retuming the chiller to service, however, the switch was replaced with one that was 10 quali6ed. The other example involved the attempted use of an incorrect gasket during a repair of 11 a water leak on a standby diesel generator. Again, personnel discovered the mistake and corrected 12 it. The DET also used this example as evidence of poor corrective maintenance and it is described 13 in more detail in the section of this chapter on corrective maintenance. Liberty concluded that

, 14 these instances taken individually or collectively did not indicate that unreasonable decisions were 15 made or actions taken by STP's management in the area of replacement parts.

, o l

l O

" DET response items nos. 2100 and 2100-001.

Page III 37 The Liberry Con.sulting Group

RLS-2 Chapter Three - Maintenance f

Vi VI. Support to Maintenance )

2 A. Maintenance Backlog 3 The DET considered the maintenance backlog at STP to be high. The DET indicated that the 4 maintenance backlog was the result of insufficient staffing. Furthermore, the size of STP's 5 maintenance backlog appeared to be the principal cause of the DET's negative findings or 6 perceptions in other areas, such as spare parts availability, management information system 7 adequacy, efficiency of the work control process, timeliness of resolution of problems, 8 productivity, and overtime.

9 The size of the maintenance backlog at a nuclear plant can be important because ofits potential 10 relationship to the safety of plant operations. The relationship can become obvious if some items 11 in the backlog act together to degrade the margin of safety. Thus, the nature of the items in the 12 backlog is crucial to this potential relationship. However, the number ofitems in the backlog is 13 easier to report and understand. Therefore, the use of absolute numbers is conunon.

14 While data are difficult to compare, STP's backlog might have appeared large. However, with one .

V 15 unit at the end of an extended run and with the additional safety train at STP, the STP backlog 16 could be expected to be higher than other plants. Moreover, backlog size comparisons are only 17 meaningful if the content of the backlog is comparable. For example, a large backlog of low 18 priority items may not be as significant as 'a small backlog of high priority safety-related items.

19 The issues of backlog size and makeup should be viewed together to put the potential effect on 20 safety in perspective. Other aspects of STP's backlog are also important injudging its absolute 21 size relative to the size of the backlog at other plants.

22 STP's Wes'tinghouse three-train safety system design.is unigtie. Its purpose.is to pr6 vide two-

~

.- 23 ; ' layers of redundancy in s'afety systems where other plailts have only'one'. This nicans that STP hasD 24 ~ a greater safety margin in most accident scenarios than a twlo-train design, but it also means that 25 it has more safety systein equipment'to maintain.

26 Anotiier. reason why simplistic comparisons can be misleading'is that STP's work backlog '

21 probab}y contains many more items of a minor nature than the backiogs at'oths plants. ThAre 'are 28 two reasons for this. Historically, STP's management encouraged formal work requests for all

[ 29 work, no matter how minor, that needed to be perfonned. Second, STP, unlike many otfier plants, 30 did not until recently have a process to correct minor deficiencies without first entering them on Page 111-38 1he Liberty Consulting Group

l RL.%2 Chapter Three- Maintenance

\, i the backlog. Both of these factors contributed to a backlog that may be larger than other plants but 2 contains a disproportionate share of minor items.

3 ,

This point became an issue during the NRC's Maintenance Team Inspection (MT/) of STP in late 4 ' Febmary 1990. MTIs were being performed at all operating nuclear plants at that time to assess 5 the adequacy of maintenance programs and.their implementation in connection .with NRC's 5 evaluation of the need to have a' maintenance rule. The MTI concluded that STP's maintenance

. 7" programs were " good" and that implementation was " satisfactory." With. respect to the backlog, 8 the MTI was initially concerned about size and commented that "the gross numbers and trends 9 were not optimal," with the total backlog at the end of 1989 at about 5,000 SRs. (At the end of to 1992, just prior to the events that led to the plant shutdown and DET evaluation, the backlog was 11 . again at about 5,000 but on the way down from a July 1992 peak of about 5,600.) In order to gain

~12 perspective on the significance of the backlog with respect to safety, the MTI team, with STP 13 assistance, reviewed a large number of the SRs. The MTI concluded:

14 "The team reviewed 50 percent of the backlogged quality MWRs [SRs] and 15 selected numerous activities that were questionable with the regard to continued 16 deferment of work. These activities were then discussed with the licensee's O' d 17 operational representative as to their potential impact on safety. In each case the technicaljustification was sufficient to warrant not completing the work until the

{ 18 19 next normally scheduled maintenance " window." The team was convinced that 20 the licensee was adequately tracking each outstanding work item and 21 scheduling its completion consistent with procedural criteria." (Emphasis 22 added.)

9 23 The MTI examined the substantive nature of the backlog in detail and concluded that the priority 24 assigned to work items was appropriate and that safety was not a concern. The DET, on the other 25 hand, looking primarily at the overall size of the backlog, reached a conclusion on the basis of that l 26 factor 27 Normally there is a direct relationship between the number ofitems in a maintenance backlog and 28 the operational history of the plant or plant site. Obviously, outage work items can only be 29 Completed during a plant Outage. While work classified as non-outage can be performed while a l 30 unit is operating, it is often more efficient and practical to perform this work during scheduled 31 maintenance periods. At a two-unit site, a maintenance outage on one unit will affect the amount

" NRC Maintenance Team Inspection Report, ST AE H1-92455, May 10,1990.

Page III-39 The Liberty Comuhing Group

)

RLS-2 ,

Chapter Three- Maintenance O

1 of work performed on the other because of constraints on available maintenance resources. The ,' ,

2 following graph shows the total number of open service requests for both units at STP for 1991 3 and 1992.7' Also displayed is the periodic site capacity factor, which is a measure of the outage 4 time at one unit or the other. The profile for STP was not unusual. The total number of open 5 service requests dropped during outage periods and grew during periods of consistent operation.

6 The profile also shows that the total number of items in a maintenance backlog can vary 7 significantly over time, and so in many ways the trend is more important than the absolute number 8 at any specific time.

9 The size of STP's SR backlog, and the effect of a large backlog on safety, were the basis of many 10 of the DET's findings regarding work processes, resources, and the adequacy of STP's work force.

11 STP's management recognized the value of reducing the backlog for reasons of operational 12 efficiency and was taking actions to reduce it. The difference between the DET and MTI views 13 demonstrates that the speed at which that backlog should be reduced is an issue upon which 14 reasonable minds might differ. Having examined the details of the issue, it was Liberty's 15 conclusion that the size of the backlog was not an indication of unreasonable or imprudent 16 management.

O

)

G "

Data obtained from DET response items nos. 0017 and 0018.

Page 111-40 The 1.iberty Consulting Group

RLS-2 Chapter Three - Maintenance U Open senice Requests Total sTP Site 6000 100

-90 *

"" V .so ,

./

3000

[ t -70 u" t i

'* ?, . ;

c'" w 1\ m::l  :

A V 30 l + Open SRs - Site C.F. l >

3500 10 Jan-91 Mar 91 91 M-91 sep.9i Ndp! J 92 Mar-92 Mh-92 M-92 Sep.92 Nov92 i

b

l. I B. Staffing and Overtime 2 1. Staffing  ;

3 The DET's findings related to staffing levels in maintenance resulted from the DET's opinions 4 about the size of the non-outage backlog and STP's resolution ofdeficiencies in certain equipment.

5 Those findings were summarized as follows: 78 6- "The staff size was insufficient to accomplish corrective maintenance given the 7 productivity achieved using the existing system, the unique three-train design of 8 the facility, and the untimely resolution of design deficiencies. The balance-of-9 plant corrective maintenance effort suffered mostly due to the lack of personnel 10 resources."

11 In order to provide a basis for its assessment, the DET focused on two particular staffing issues.

12 These were: 1) purported incorrect assumptions in a consultant's study on productivity that 13 concluded that staffing levels appeared reasonable and 2) the assertion that STP's staff levels s

)

l DET Report, p. 21.

Page 111-41 The Liberty Consulting Group

RLS-2 Chapter Three - Maintenance N

1 inappropriately assumed that the plant was in a stable operating condition. With regard to the first /

2 issue, the DET said:"

3 "The decision to have several station staffing studies conducted by outside 4 consultants indicated senior managements' concem over appropriate stafling levels.

5 However, the recommended staffmg levels in the most recent study was [ sic] based 6 on incorrect assumptions on productivity. The average time required to complete 7 a service request (SR) was estimated in this study as less than 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.

8 Information supplied by the licensee at the request of the team indicated that the 9 actual time required to complete a SR ranged between 42 and 50 hours5.787037e-4 days <br />0.0139 hours <br />8.267196e-5 weeks <br />1.9025e-5 months <br />. Additional to information found by the team indicated that the latter figures may be low because 11 the licensee's management information system (MIS) did not account for lost time 12 due to lack of coordination and parts availability, which were significant problems 13 at STP, and the time expended by one craft assisting another."

14 STP provided information to the DET that clearly supported the use of an average of about 30 15 manhours for maintenance activities.: This matched the 29.2 manhours used in the ASTA report.

16 ASTA used that figure as the average level of effort needed to complete a service request (SR) and 17 calculated staffing requirements using the number of SRs performed. ASTA stated in its report that O18 the 29.2 manhour average included lost time such as time spent waiting for parts.8' I 19 The consultant's study covered the period from June 8,1992 to September 25,1992. It concluded 20 that STP's total baseline maintenance staffmg requirement was 566 for Level II (sustained long-21 . term operations meeting industry standards) and 653 for Level III (additional personnel for short-22 term projects or programs that exceeded industry standards). d'ctual STP maintenance' staffing'in 23 February 1993 was higher than both at 705.82 24 STP's management had independent studies performed to support decision-making when the need..

. .. . . ,. ' 1 2'5 ' was indicated. This' was a reasonable management practice. The results of the . study. for .

26 maintenance dId not provide any reason'for management to 15e}ieve'that itaffing levels weritoo :

27 ' low.' StP's management . appropriately considered this conciusion in making decisions 'on'. .

DET-Repori, p. 41.' ,

" ~ DET response item #2315.

" DET response item #0066. '

%/ " DET response item #4068.

I age 111-42 The Liberty Consulting Group

4 RLV-2 Chapter Three - M aintenance a

lV) 2 I maintenance staffing and continued with its plans to improve work processes and overall efGciency, as discussed in the section below on the " Work Control Process."

3 The maintenance staff size was important primarily because ofits linkage to maintenance backlog.

4 Many factors contributed to the size of STP's non-outage backlog, and the level of staf6ng was 5 certainly one of them. But the core issue is the size of the backlog and whether or not it posed an 6 unacceptable risk to plant safety or operability.

7 The DET's second major statement on maintenance staf6ng was as follows:"

j 8 "In addition to staf5ng based on incorrect assumptions on productivity, the 9 licensee generally appeared to be stafUng based on levels predicated on the station 10 operating in a stable condition with only long term requirements and no signi6 cant 1I backlogs or emergent workloads. These assumptions did not reflect the state of 12 maintenance backlogs, improvement programs, and other post construction 13 transitional requirements that existed at STP."

14 STP's staf6ng levels in maintenance were established on the basis of management's assessment

/' 15 of the amount of work that needed to be accomplished, work force performance expectations, the 3

16 opinions of the many oversight groups and agencies both inside and outside of STP that oversee 17 activities at STP, and the obligation to work ef6ciently through effective cost control and 18 budgeting.

19 The ASTA report concluded that stafHng levels in maintenance were reasonable, considering the 20 workload. SALP assessments by the NRC were no different. In the SALP report issued for the 21 period February 1990 through May 1991, the NRC stated that, with respect to maintenance, 22 "[o]verall, stafHng was considered to be good."" In June 1991, a consultant's evaluation 23 concluded that total maintenance stafGng was about 5 percent lower than expected." The SALP 24 report for the following period (June 1991 through August 1992) made no comment on 25 maintenance staffing.

1

" DET Report, p. 41.

p\ NRC SALP Report, ST-AE-HL-92831, September 6,1991, p.13.

( " Analysis of Maintenance Acthities, Performance Data, Inc., June 1991. l Page IH-43 l The Liberty Consulting Group

1 i

RLS-2 Chapter Three- Maintenance O1 i

The DET's view of staffing levels in maintenance was directly linked to the size of STP's ) '

2 maintenance work backlog. However, when STP received a SALP rating of"1" in maintenance -

3 in January 1990, the backlog was 5,000 items. In August 1992, after STP was rated "2" and i 4 declining in maintenance, the backlog was about 5,700. STP's management took action to reduce . +

5 the backlog more aggressively, and before year's end it was reduced to under 5,000, the same l 6 level it was at when STP's SALP rating was a "1." The reduction was accomplished through a 7 combination ofincreased ovedime and the use of contractors." The correlation between NRC's .

8 SALP ratings and the raw numbers in a backlog is not a direct one, of course, but it indicated to i 9 STP's management that plant maintenance could be considered superior even with a maintenance 10 backlog of 5,000. This was significant because it provided a reasonable basis for management to 11 conclude that the maintenance backlog, at the 5,000 level, was not by itself a reason for r 12 heightened concern (from the viewpoint of the NRC) about the impact on plant safety.

13 With the backlog reduced to under 5,000 at the end of 1992, and a number ofinitiatives in place 14 to improve efficiency, information systems, work control processes, and the number of 15 maintenance journeyman qualifications, the need for increased stafrmg, overtime, and use of 16 contractors to further reduce backlog was weighed against needs, costs, and benefits. In early ,

17 1993, with both units in a forced shutdown, the backlog remained near the 5,000 level until it was 1 18 clear that a greatly reduced backlog would be a requirement for restart of the units. A mix of 19 increased staffing, overtime, and use of additional contractors was then used to reduce the backlog.

20 ,

A . report on. staffing levels in the nuclear industry, published in mid-_1992, provided som6- ,

21 adilitional perspective on the issue." The data providdd in the article for STP's~ peer group" of 22 new 3- and 4-loop Westinghouse plants showed that overall stafTing levels at STP were not low:

s . .

1 .!

4

• " ' Station Report for December 1992, DET response item #')018~.

i

" Nucicar Manpower, Electrical World, July 1992. (Calculated employees by using 2000 mh/yr/ employee and 4 . mh/MW reported in article { excluded single unit sites.) . +

I j '"'

Performancu lahak rs for Operating Commercial Nuclear Powcr Reactors (Data through March 1993). Part

1. U.S. NRC AEOD.

I Page III-44 The 1.iherty Consulting Group

RL%2 Chapter Three - Maintenance l

O Number of Emplovees ner Plant:

1 2 Braidwood - 733 3 Byron - 695 4 Catawba - 1065 5 Diablo Canyon - 1243 6 McGuire - 986 7 Sequoyah - 1467 8 South Texas - 1588 9 Vogtle - 1787 10 Average - 1196 11 STP's December 1992 staffing level was 1705, a 7-percent increase over the 1990 staffing level 12 of 1588. The June 1993 staffing level was 1770,1I percent above the 1990 staffing level. In the l 13 peer group, STP ranked second only to Vogtle in number of employees per two-unit site.

14 The DET said;"

15 "From the end of the Unit 2 refuel outage (December 1991) until the beginning of i 16 the Unit I refuel outage (September 1992) both units were essentially operating at power. However, during these 9 months, the backlog of nonoutage SRs increased

[ 17 f

( 18 by 1600, an increase of approximately 50 percent. Three fourths of the SRs were on balance-of-plant systems."

19 20 While it was true that non-outage backlog had increased by more than 1,600 SRs in the period 21 from December 1991 to September 1992, this was not evidence of insufficient personnel 22 resources. In the preceding 9-month period, the backlog was reduced by 1,600 SRs with lower ,

23 total STP staffinf The growth in the size of the backlog was not directly related to staffing l 24 levels, nor did it support the assertion that the staffing level was insufficient to manage the i 25 backlog.

26 A backlog ieduction also occurred from July 1992 to December 1992. During that period, backlog 27 decreased by 800 SRs, again with essentially the same maintenance staffing. The difference was 28 primarily attributable to unit operating mode. During 1992, when the backlog increased, STP 4

" DET Report, p. 21.

" Station Reports for Decembcr 1992, SPR Backlog (D-5) and Staffing (L-2) and November 1991, (E-3) and (L-2), DET response item #0018.

Page IH-45 [

The Liberty Consulting Group

RLS'-2 Chapter Three - Maintenance i Units 1 and 2 had high availability and only one unit refueling outage. In 1990 and 1991, when )

2 the backlog decreased, both units were down in each year for refueling.

3 The results ofindependent staffing studies showed an adequate maintenance staff size. Objective 4 data supported the accuracy of those studies. Liberty concluded that the relationship between 5 backlog and staffing did not, by itself, indicate a need to increase staffing or indicate that 6 management had made unreasonable decisions or taken unreasonable actions in the area of 7 maintenance staffing.

8 2. Overtime 9 The DET's criticism of STP's use of overtime during outages focused on three issues: (1) that STP 10 routinely used overtime when in an outage condition, (2) that Technical Specification guidelines 11 for overtime were exceeded in some instances without appropriate management review and 12 approval, and (3) that "by the first quarter of 1993, absenteeism for illness of craft workers had 13 increased in all three disciplines with mechanical maintenance [ absenteeism) almost doubling from p 14 the previous year."" No specific safety consequence was identified by the DET, and no ,

15 comparative data were provided that would allow a relative comparison of STP with other plants.

16 The use of overtime during an outage is a standard practice in the utility industry because it is 17 cost-effective. In prior refueling outages, STP's hourly overtime ranged from about 30 percent . i 18 for the second refueling outag'e of Unit 1 in the~ spring of 1990 to about 60 percent for'.the first 19 refueling outage of Unit 2 during the fall of 1990." During the latest refueling outage of Unit 1 20 in the fall of 1992, the average overtime rates for hourly personnel were 26 percent in September, 21 42 percent in October,48 percent in November, and 34 percent in December. The amount'of ,

22 overtime used during the last four refueling outages was not significantly different from these .

23 values." STP has ' committed itself to monitoring the use of overtime more closely," but its'use'

._ 24- is ultimately a business ~ decision based on.a balance of cost and efficiencyf, o . ,,

"- DET Report, p. 21.

~ '

l

" Calculated from Station Reports for May and November 1990.

" ~

Station Reports for December 1992, December 1991(contained in DET respotise item #0018')'and calculated

. from Station Report for December 1990.

1[1,&P Operational Readiness Plan Section V.IL2.c., p. 23.

Page Ill-46 The IJberty Comulting Gmup

RI.S-2 Chapter Three - M sintenance tQ l STP recognized that excessive overtime could lead to inefHeiency and safety concerns because 2 of worker fatigue. This point was stressed in HL&P's policies and administrative practices on 3 overtime." STP's Administrative Procedure on overtime was revised in 1992 on the basis of an 4 SPR investigation that had indicated the need for additional clarity in the deGnition of Technical 5 Specincation limitations?'

6 In a related criticism of the performance indicator (P1) STP had used for monitoring overtime, the 7 DET stated:"

8 "Many Pls were in error and were not indicative of the major performance 9 problems at STP. For example, Pls associated with ovenime indicated that the 10 overtime rate was less than 5 percent for salaried and less than 10 percent for non-11 salaried. Additionally, most maintenance craft and supervisory employees 12 worked approximately 50 percent ovedime."

13 Liberty found that the 10 percent Egure quoted by the DET was the projected figure, not the actual 14 values. The reported values in the PI for non-salaried employees, for the period between 15 September 1992 and June 1993, ranged from 13 to 48 percent with an average of 31 percent. This

( ) 16 was the period of time between the start of the refueling outage on Unit I and the conclusion of 17 the DET's evaluation. Moreover, there was no basis to compare the DET's 50-percent ovenime 18 worked Egure, which just covered "most maintenance craft and supervisory employees," with a 19 PI that covered all non-salaried employees. The P1 was not in error.

20 Absenteeism for illness for the Maintenance Department as a whole in the first quarter of 1993 21 was only 5.2 percent higher than in the same period in 1992 and was 2 percent lower than in the 22 same period in 1991. In terms of numbers, the average number of paid personal illness days 23 increased from 1.14 to only 1.20 days per employee for the three-month period.

24 On the subject of Technical SpeciDeation guideline limits on overtime being exceeded in some 25 instances, the DET's statement was accurate. Several individuals had inadvertently exceeded the

" DET response item #4045, Memorandum,11all to Kinsey, February 14,1992, and Memorandum, Kinsey to 11 alt, February 4,1992 on Drafl Administrative Practice on Overtime.

j'] Station Problem Report 921460, contained in DET response item #1167.

" DET Report, p. 48.

Page LU-47 The Liberty Consulting Gmup

-- . . - -. - . .. .- . . . ~ . - - . . _ - _

RLS-2 Chapter Three- Maintenance

\

1 limitation on the maximum number of hours that should be worked in a seven-day period without )

2 obtaining the necessary prior approval.

3 C. Vibration Monitoring Program 4 The DET portrayed STP's vibration monitoring program as understaffed because of the length of 5 time between vibration readings taken on several pieces of equipment. Liberty's review did not 6 result in similar findings. In one case the DET said there had been three years between vibration 7 readings, when the available data showed quarterly vibration readings had been taken.

8' The DET's stafling issue centered on whether to have one or two technicians in STP's Predictive 9 Maintenance Group take readings. However, this was not the only group to take such readings.

10 Other personnel in STP's Operations Group took vibration readings as part of the IST (Inservice

.11 Test Program). Furthermore, Liberty found that the DET's examples did not demonstrate that 12 management had made unreasonable staffing decisions or that indeed circumstances justified 13 ad.iitional staffing.

14 The DET cited certain adverse consequences ofinfrequent vibration readings. For example, the 15 DET stated:"

16 "During a vibration analysis in.May 1990, the Unit 1 main generator seal oil 17 . backup pump exceeded alarm limits However.over 2.1/2 years passed before th.e 18 next vibration readin's g were taken in January of 1993. Subsequen'tly,' the 19 deteriorated motor and pump bearing had to be replaced."

~

~

20 'To. evaluate this event, several important additional factsTmust be considered. Ainong these was -

21 :th'e fact that the pump in question is 'non'saf' e ty-related and runs infrequently on generhtor startup, 22 and shutdown: While it is' tiuf that th'e vibration readings'taken b'y 'STP's Predictive Maintenahce c 23' JGro'up were 'not frequent,'other inspections were made of tlie pump's. condition.These'Inchided w'

24 frequent IST vibratiori testing by' operators."

DET Report, p. 22.

DET response item #3250, p.10.

Page 111-4h The Libertv Consulting Group

, I RLV-2 Chapter Three - Maintenance Ql 2 Any implication that the infrequency of the vibration readings had something to do with a premature failure was incorrect. In fact, a failure never did occur and the repair action cited by the 3 DET was actually preventive maintenance. Vibration monitoring provides some of the data on the 4 basis of which a decision is made to repair or increase monitoring. The alarm and alen levels set 5 for comparison purposes are not based on facts related to the probability ofimminent failure but 6 rather on engineering judgment. Examination of the relevant data showed that vibration levels 7 were higher during the May 1990 test than in January 1993, when the DET determined that the 8 motor and pump had " deteriorated." In May 1990,4 of 10 readings taken were over alert levels; 9 in January 1993, only 2 of 10 were over. The numbers of readings increasing and decreasing were ,

i 10 the same. While taking readings more frequently might have been desirable, the fact that they 11 were not had no great significance.

12 In the second example cited by the DET, the issue was the infrequent vibration readings taken on 13 the HHSI pumps.*

14 "Since the plant began commercial operation the vibration of the Unit 1 HHSI 15 pump motors exceeded the alarm limits of the predictive maintenance program.

,'/G 16 However, more than 27 months passed between vibration readings on the 1C pump 17 and 18 months passed for the 1A pump. Eventually, unsatisfactory oil samples 18 were taken on the l A and IC motar bearings. The combination of high vibration 19 and oil sample results prompted maintenance personnel to inspect and replace the 20 bearings."

21 Vibration analysis wasjust one tool used in making maintenance decisions. Because all three new 22 pumps just marginally exceeded the alarm limits and operated well in all other respects with no 23 sign ofimminent failure, the appropriateness of the alarm limits, not the condition of the pumps, 24 was the issue. Inspections were taken more frequently after the initial readings and indicated 25 stable, not deteriorating, conditions.' 3 The infrequent monitoring was a consequence of prioritized 26 resource allocation, limited access to the units for testing, o2 and ajudgment that more frequent 27 monitoring was not a high priority. When oil analysis five years later indicated that bearing wear 28 might be beginning, a preventive maintenance action was taken to inspect and replace the 29 bearings. Subsequent vibration readings on the HHSI pump motors with the new bearings revealed

• DET Report, p. 22.

p

• DET response item #2353.

• DET response item #3250, SPR 930449, p. I 1, para. III.

Page 111-49 The Liberry Consulting Group

1 RLS 2 Chapter Three- Maintenance )

{

1 only a marginal decrease in vibration levels. A detailed examination of the bearings confirmed that )

2 only minor wear had in fact occurred.

3 In its last example of the consequences of the staffing of vibration monitoring programs, the DET 4 stated:"As much as a [ sic] three years passed between vibration readings on the Unit I auxiliary 5 feedwater pumps."*

6 This DET statement proved to be inaccurate. Available data clearly indicated that vibration 7 readings on the auxiliary feedwater pumps had been taken quanerly.* The DET examined those 8 data and concluded that two readings had exceeded alen levels with no annotation of a concern 9 or indication of a need for increased frequency of testing. Further examination however revealed to that the readings were "well below the alert levels."* Thus, this example did not demonstrate that 11 STP was not adequately staffmg vibration monitoring programs.

12 Vibratio. monitoring is a predictive maintenance diagnostic tool. It is far from an exact science, i 13 and the results of vibration monitoring must be used in conjunction with other performance, 14 inspection, and predictive maintenance monitoring techniques to formulate hypotheses regarding 5 the potential for imminent equipment failure. As a predictive maintenance technique, it is a 16 maintenance program enhancement. It is not a requirement except in specific cases. The resources 17 STP invested in enhancement programs depended on their anticipated overall value in maintaining 18 the safety and cost-effectiveness of operations. At the time of the DET review, only one technician 19 was assigned to collect data. HL&P believed one technician was all that was required at the time..

20 Libeny found that the frequency of vibration monitoring on cenain pieces of equipment did not 21 evidence a general level of understafTmg. Staffing levels in any organization depend upon a

'22 ' balancing of costs and needs. The examples in the DET report did not demons.trate that' staffing .

23 levels for the vibration technicians had in any way adversely 'affected the ' safety of operations at 24 STP or that' mana'gement had made unreasonable decisions or taken' unreasonable action in

- 4 25 establishing such s;taffing levels.#

i l .

• DET Report, p. 22.

t

• DET response item #3296.

\

'" DET response item #3296-o01.

Page 111-50 The Liberty Consuhmg Gmur l

l

._ ._ -_= _ -- _ _ _ . . _

RLS-2 Chapter Three - Maintenance C

i VII. Work Control Process 2 A. Background and Perspective 3 Work control refers to a set of steps in a process used to manage and control work from the 4 identification ofits need through its completion, including any follow-up checks to ensure it was 5 performed adequately. In a complex facility like a nuclear power plant, some of the basic steps in 6 the work control process for maintenance are: .

7 o documenting the need for the work 8 o identifying specifically what needs to be done 9 o checking the accuracy of the work request  ;

10 o determining the importance and relative priority of the work 1I o identifying the reason the workis required 12 o determining generic implications of the need for the work 13 o identifying prior work history and its significance 14 o identifying when the work should be performed

( 15 o determining the consequences of removing items from service 16 o identifying other work that can be coordinated with the new work 17 o estimating parts requirements and work procedures 18 o detemiining parts availability or procurement lead times 19 o evaluating nuclear radiation and contamination potential 20 o developing safety procedures 21 o defining quality control and inspection requirements 22 o identifying worker skill and qualification requirements 23 o determining the availability of required workers  ;

24 o estimating resources and work durations 25 o identifying required post-maintenance test requirements 26 o preparing detailed work instructions 27 o rescheduling when higher priority work emerges 28 o performing the work i 29 o performing post-maintenance tests b

30 o preparingjob closeout paperwork and completing feedback forms Page 111-51 The Liberty Consulting Group 5

RLS-2 Chapter Three - Maintenance r i 1 This sampling of the steps involved illustrates the breadth of the work control process. Quality )

2 requirements at a nuclear plant tend to require that each step be controlled by detailed procedures. l 3 Each step often requires both performance and independent checking or verification of the 4 performance. Efficiency is affected by how well each step is performed as well as the inherent 5 efficiency of the process, which is a function of the steps included, the order in which they are 6 performed, and the reviews performed to ensure each step is performed correctly. Changes in one 7 step may affect the performance of another, and gains in efficiency are normally achieved through 8 an incremental and evolutionary improvement process.

9 The following findings and conclusions summarized the DET's assessment:*

10 "The work control process was inefficient and manpower intensive. This resulted 11 in the inefficient use of staff which contributed to the poor material condition of the 12 plant, and the completion of only high priority work. Consequently, the high 13 maintenance backlog significantly stressed the maintenance department in the form 14 of emergent work, rendering the process more inefficient. Also, multiple barriers 15 to an efficient work control process existed within the planning, preparation, 16 scheduling and execution of work."

L)71 STP's work control process has been the subject ofINPO, NRC MTI, and independent consultant 18 reviews and a number of STP internal improvement initiatives since the start of commercial 19 operation. Originally, the work control process was covered by one procedure. In 1988, when the 20 procedure was thought to be too complex,it was split into six different procedures on the basis of 21 the different functions involved. In 1990, the procedures were combined again in an effort to 22 improve implementation of the work control process and better ensure compliance with all of the 23 detailed work control requirements. In 1992, the procedure was simplified to streamline and

~

24 improve the process and'to encourage ' owner' ship and individual responsibility.' The current c25 - simphfied procedure now contains 40 pages,'down from 160, and reflects comments," suggestions, .

26 and input from a wide range of plant, iridustry, an'd regulatory contributors.

s .: .

t ,

27 These initiatives demonstrate that the importance of th'e ' efficiency and accuracy'.of'the work 28 control process have been the subject of significant management attention at STP and have not -

29 been neglected or ignored.' And while the DET's- assessm'ent.that the work control process.

30 contained inefficiencies is not disputed, the fact is that at any nuclear plant, or any other facility

31. where maintenance is~ performed, there will be inefliciencies in the maintenance work control DET Report, p. 22.

Page 111-52 The Liberty Consultmg Group

RLS-:

Chapter Three - Maintenance O v 1 process. The continual effort to improve the process, however, showed that management was 2 taking reasonable actions.

3 STP's management received numerous comments and suggestions from INPO regarding the work a control process and took the appropriate action on the basis ofINPO's recommendations. In May 5 1991, TNPO found that deficiencies in planning and coordinating maintenance activities and the 6 resulting delays in repairing and retuming equipment to service had contributed to a degraded 7 material condition in the plant. The major recommendation was for STP to ensure that work 8 packages contained effective guidance and adequate work instructions.'" STP's management 9 promptly responded by:

10 o implementing a fonnal training program for planners that began in June 1991 11 o establishing senior maintenance planner positions to encourage greater work 12 package ownership responsibility 13 o proceeding with a new work process improvement effort ongoing since April 1991.

y/ 14 The complete implementation was targeted for January 1992, 15 INPO's September 3,1991, review made no mention ofits earlier findings of May 1991, signaling 16 that STP's management had taken reasonable action in response to its recommendations to 17 improve the work control process.

I8 However, in October 1991, INPO provided additional comments and suggestions for improvement 19 of the work control system.'" The issues then identified included weaknesses in work planning, 20 lack of detail, and poor resource requirements definition. STP responded in 1991-1992 with a 21 number of additional initiatives to improve work planning and work control:"*

1 22 o The responsible craft division was required to verify resource requirements and 23 adequacy of work instruction content, including concurrence that manhour 24 estimates were correct. ,

l i

INPO May 1991 Evaluation, Finding MA.3 1 (Related to MA.3.1,1989).

INPO October 1991 Outage Review, December 13,1991. ,

'\ l Letter, HL&P to INPO, responding to October 1991 outage review, February 11,1992. l Page 11153 The Liberty Consultmg Group i

R U .'

Chapter Three \1aintenance i o A maintenance feedback fonn was incorporated into all work packages. This 2 required craftsmen and foremen to provide written feedback to planners on all 3 aspects of work package suitability. including tests performed. The process a provided for direct confirmation of resources and manpower requirements 5 identified during the planning phase.

6 o The Maintenance Planning Division was required to conduct backend reviews of 7 all work packages. Backend review had previously been the responsibility of the 8 group performing the maintenance to ensure that the work and the post-9 maintenance testing were completed properly. The change afforded the planners 10 first-hand review of comments generated during work performance.

1i o The Planners Guide was revised to provide expectations for work package 12 preparation, including standardized formats and examples. The feedback and 13 backend review processes were used to determine the effectiveness of these 14 actions.

15 Each of these actions demonstrated that STP's management was aware of areas in which the work 16 control process could be improved and was taking actions to accomplished those improvements.

17 B. Other NRC Evaluations 18 In 1990 the NRC performed a Maintenance Team Inspection (MTI) of all maintenance activities 19 at STP. The MTI concluded that job plannmg and work control processes were strengths. Results 20 of the review were included in the NRC SALP report covering that period:"

21 "The MTI was performed at the beginning of the assessment period and found that 22 the licensee had a well developed maintenance program. The inspection identified 23 strengths in job planning, the work control process, PMT, ISEG oversight, 24 material storage, and the deficiency reporting system."(Emphasis added.)

O

\

\J "'

NRC SALP Report, ST-AE-HL 92831, September 6,1991.

Page 111-54 The Liberty Consulting Group

?U-:

Chapter Three \taintenance i

\ l This positive assessment was followed by the NRC's S ALP covering the period from June 1991

~

to August 1992. Again. the independent assessment of STP's work control process was positise?"

3 "Several strengths were identified. The licensee had a good maintenance work 4 control process that provided for the identification of equipment problems.

5 evaluation of equipment operability, work activity prioritization, conduct of 6 maintenance actisities, and proper closure of work packages."(Emphasis added.)

.., During 1991 and 1992, STP was responding to INPO suggestions to strengthen the work control 8 process, implementing improvements, and at the same time receiving NRC assessments and A

9 inspections that were positive. Liberty found that STP's management was reasonable in concluding 10 that. prior to the DET's review, significant changes in the pace of improving its work control 11 process was not required in order to avoid serious regulatory difficulty.

,, i: C. Independent Assessment

^

13 in order to gain a better perspective on needed changes and to help guide its intemal improvement f

\

la 15 efforts, STP's management, in June 1991, commissioned an independent third party analysis of STP's maintenance activities. The consultant concluded that the work control process was complex 16 and had certain inefficiencies. Areas for improvements were identified, and STP took additional n actions on the basis of the consultant's recommendations as well as INPO's May 1992 evaluation 18 and a January 1993 INPO special assist visit."2 The actions taken included:

19 o Issuance in April 1992 and subsequent revision and refinement of the Maintenance 20 Department Expectations and Assessment Manual 21 o Implementation of a new work scheduling process to improve schedule compliance 22 and enhance productivity 23 o Initiatives to improve communications of maintenance problems to management 24 by maintenance foremen 25 o Streamlining of the work control procedure on the basis ofINPO Good Practices 6 and a comparison of STP with plants with an NRC SALP rating of"1" p '"

NRC SA LP Report, ST-AE-HL-93239, November 18,1992.

"2 Lener, INPO to HL&P. Gillispie to Hall, February 8,1993.

Page 111-55 The Lsberry Consulang Group

m-L~ 4 em 4 - Lo*L A w J- e PL5 .*

Chapter Three - Wintenance t

1 o increasing the use of minor maintenance through the use of a Maintenance Ros er  ;

2 Program 3 o implementation of an Operations Work Control Group to review and prioritize a incoming service requests and improve problem descriptions.

5 STP's management consistently responded to each finding and suggestion and demonstrated full ,

6 recognition of the importance ofimproved performance and efficiency in its work control system.

7 D. Efficiency of the Work Control System 8 The adequacy of management's efforts to improve the work control system and to reduce the size 9 of the work backlog were the issues about which the DET expressed its concerns. While it is true j lo that certain inefficiencies existed in the process, recognition should be given to STP's management ii for its continued efforts to improve the process. Liberty also found that management had every-12 reason to believe that it was taking reas'onable steps in this area since the NRC's prior assessments 13 through the SALP program had reached completely different conclusions from those set forth in .

14 the DET report. Given the differences between the DET and other NRC evaluations, quantitative 15 measures of work control efficiency are important in establishing the significance of the issue' and 16 putting the DET's criticism in perspective.

17 One way to measure efficiency is to examine the time it takes to perform an activity. Turn-time l

18 from work origination to completion can provide an objective measure of the efficiency of a work 19 control system. Even though NRC assessments had been positive, STP's management identified 20 specific objectives in its Master Operating Plans, the attainment of which would have brought 21 about improvement in the efficiency of all ofits work processes. In 1992. the action plan for 22 improving work processes included monitoring a performance indicator measuring tum time. The 23 target was set at 50 days from identification of a non-outage maintenance service request until 24 completion."3 25 The average tum time for a service request was reported to be 66 days in June 1992. The reported 26 industry average was 45 days."' From January 1,1992 to December 31,1992, STP reduced turn-

"8 Master Operating Plan for 1992, Work Processes, p.10.

V "'

Analysis of Maintenance Activities, Performance Data Inc., June 1991.

Page 11156 The Luberty Consulting Group

- w

R!.5:

Chapter Three - \taintenance r\

I time for non-outage sersice requests by about 30 percent.'" This significant reduction

demonstrated that actions taken by STP were appropriate and were successful in achieving process 3 impros ement.

4 E. Specific DET Comments on Efficiency of Work Control 5 The DET report presented specific findings on work control that addressed what the DET believed 6 were barriers to improved effectiveness and efficiency. The DET report also included a number 7 of criticisms related to these barriers. Each criticism is quoted below and is followed by Liberty's 8 observations and conclusions."6 9 Emergent Work to "The large amount of emergent work significantly contributed to the inefficient i1 work control process. . The emergent work estimates of about 20 percent were 12 routinely exceeded with over 40 percent of routinely accomplished corrective 13 maintenance being emergent work. This stressed all aspects of the work control p 14 system by reducing the time in which personnel had to accomplish their assigned  ;

15 tasks. . The excessive emergent work, prompted the staff to postpone previously l l

16 planned or partially planned jobs, adding to the backlog."

17 Emergent work can disrupt and complicate work planning. However, it is also true that emergent l

18 work, by its very nature, is unanticipated and not something that can easily be controlled. STP 19 recognized the impact of emergent work on planned work and made improvements in various 20 processes to diminish its effect. Rese changes were documented in STP's Master Operating 21 Plan."'

STP Station Report, December 1992, D 6, DET response item #0018.

DET Report, p.12.

%./

}

Master Operating Plan for 1993, MNT-2.

Page 111-57 The Lsberty Comultmg Group

i l

pu-:

Chapter Three \laintenance i Equipment Degradation l

"[T]he large corrective maintenance backlog. . inhibited the timely repair of 3 deliciencies before their condition degraded.""'

4 Deficiencies included in any nuclear plant's backlog could result in equipment degradation before 5 work is perfonned. However, continued degradation is more a function of the length of time the 6 deficiency existed before repairs were made and the nature of the deficiency than it is of the size 7 of the backlog. A review of STP's performance through 1992 showed (1) a relatively small 8 percentage of the backlog was in high priority areas, i.e., areas in which equipment degradation 9 would be of concem, and (2) a significant reduction in the average time between identification of to the need for work and its performance. This reduced the potential for material condition 1i deterioration for work in the backlog and demonstrated that work processes at STP were being 12 improved.

13 Communication la " Coordination and communication weaknesses contributed to poor maintenance""'

r

( 15 16 The DET provided a number of examples to support this finding. Inadequate communications between maintenance workers was the primary cause of the difficulties represented by several of 17 the examples. Inadequate communications during a shift turnover, an inadequate pre-job briefing, 18 and a human error were the root causes. These events occurred, and poor communication was 19 clearly a contributing factor. Examples of communications weaknesses such as these, however.

20 are difficult to relate to a programmatic fiaw, a complex procedure, or other structural weaknesses.

21 For example, the failure of a system engineer to consult with a design engineer or the diesel vendor 22 when designing a stud driver tool is not relevant to the issue of an inefficient work control system.

22 It is therefore unclear why such examples were included in the DET report as part of the 24 assessment of the efficiency of the maintenance work control system.

- DET Repon, p. 22.

V "'

DET Repon, p. 23.

Page 111-58 The Liberty Consulting Group

PLS-:

Chapter Three - Slaintenance O

d 1 Component Identification

" Planner performance was inhibited in some cases by incorrect component 3 identifications within the facility on SRs. This necessitated walkdowns of all a equipment to verify the correct component number against design documents.

5 work package quality and parts availability deficiencies decreased efficiency.""

6 There were cases in which incorrect component identification numbers were entered on senice 7 requests, and clearly work package quality and parts availability affected schedules and efficiency.

8 Each of these issues was the subject of STP corrective action before the DET evaluation.

9 A consultant's study of maintenance activities in June 1991 reported that planner walkdowns at to STP were conducted for about 10 percent of the work while other utilities averaged 20 percent.':'

11 Planner walkdowns of work identified on SRs can improve the quality and detail of the work 12 package. Some utilities strive for 100 percent walkdowns by planners to ensure that work is 13 accurately scoped, components accurately identified, and work conditions identified that could 14 affect how the planner defines the way thejob should be performed. STP's Master Operating Plan q 15 for 1993 included this objective.':

16 Qualified Personnel 17 " Occasionally, the lack of certified workers required work activities to be 18 postponed until a qualified individual was available."m 19 By the time the DET report was issued in June 1993, STP was midway through a program to 20 increase maintenance craft certifications from 4,349 to 4,500 in pursuit of a goal of 5,007.*' The 21 desired increase in the number of journeyman certifications was in response to the training 22 accreditation issue, which arose in mid-1992 and was, in turn, the result of the method STP had 23 used to qualify experienced journeymen. The DET's comment, while true, will always be true to 24 a greater or lesser extent. The postponement of work until a qualified joumeyman is available will

' 2' DET Report, pp. 22-23.

'8' Analysis of Maintenance Activities, Performance Data Inc., June 1991.

1993 Master Operating Plan, MOP illB, Rev. 3. Section MNT-2 A, Item 2.b.

DET Report, p. 23.

' 2' STP Station Report, August 1993.

Page 11159 The Liberty Consulting Group

?:s.:

Chapter Three **f aintenance

(% i

,.- I always be necessary from time to time. .\foreos er. STP's management had responded effectively

to 2. maintenance training deficiency in 1992 and was successfully increasing the number of 3 qualified joumeymen. Liberty, however, found no evidence that the lack of qualified joumeymen a was a significant contributor to inefficiencies in the work control process.

5 F. Conclusions 6 Efficient and effective work control at a nuclear plant presents unique challenges. STP's 7 management was aware of the importance of efficiency in work control practices and was striving 8 to make improvements. There is no question that various specific processes and procedures were 9 overly cumbersome and complex. This reflects the very nature of work at a nuclear plant where 10 every step is written down and where steps are added to prevent recurrence of prior problems. The Ii resulting tension between thoroughness and streamlining, between exacting detail and unnecessary 12 steps is inevitable. Resolving this tension with an eye toward safety and efficiency is a continuing 13 management challenge, At STP, ways to improve were sought from self assessments, INPO 14 reviews, and independent third-party assessments. Some reports on STP's work control practices.

15 including those from the NRC, were positive, and there was objective evidence to demonstrate that d 16 progress had been made. Liberty found that STP's management acted reasonably in continuing to 17 pursue improvements despite the prior positive assessments.

(O /

Pa ge 111-60 The f.sberty Consultmg Group

i e

PLS.:

Chapter Three - \l.untenance n

i VIII. Post-Maintenance and Periodic Testing 2 A. Post-31aintenance Testing 3 1. Introd uction a Post-Niaintenance Testing (PMT) is conducted to help ensure that maintenance is performed 5 correctly and that affected components, equipment, and systems are operable after the 6 maintenance is completed. STP's Ph1T program is defined by several program documents and 7 procedures and a PhiT Reference hianual, j

8 The DET report characterized STP's PhfT as "not always effective,"'" i.e., not always achieving 9 the desired result. The DET said:

10 " Numerous weaknesses in the implementation and programmatic requirements for Ii post-maintenance testing (Ph1T) reduced assurance that equipment was operable

., 12 upon return to service. The PhiT [ manual) planners [used] to select the appropriate 13 test requirements did not specify appropriate detail and occasionally specified the 14 wrong test. The planners lacked appropriate training, expenence, and guidance that

()

(N is 16 17 would allow them to compensate for the manual's deficiencies. This resulted in planners listing all possible PhiT that might be necessary and specifying the PhiTs to be performed as "if required." This required the already heavily burdened shift 18 supervisor to review the scope of work completed in order to specify the l 19 appropriate post maintenance test to be performed. Periodically, the shift supen isor

{

20 selected inappropriate PhiT and in some instances inoperable equipment was not l 21 identified as such:.. " I 22 These comments were followed by taree examples, which are addressed below.

i 23 2. Standby Diesel Generator l 24 The first example used by the DET involved one of the standby diesel generators.

25 "SDG 13 was inoperable for 2 weeks because of the failure to perform adequate 26 PhiT after painting activities. The correct PhiT had been specified in the work 27 package but was inappropriately cancelled due to a concem over excessive SDG 28 stans."C6 DET Report p. 24.

U "'

DET Report p. 24.

Page 1114l The Liberry Consulnng Group

l l

i 1

Au . l Chapter Three - Wintenance g

\

t The DET's brief account of events oversimplified the decision-making process that had led to

deletion of the PMT and in so doing. implied that there had been no other issues considered or 3 actions taken by STP in recognition of the fact that painting could. if done improperly. affect SDG a operability. Liberty found that this had not been the case. The example was introduced with an 5 assessment that the PMT program was weak because planners occasionally specified too many 6 tests or the incorrect test or the shift supervisor was overburdened, which affected his decision-7 making capability. Liberty found that the SDG example was not one of unreasonable management 8 decision making.

9 Because painting activities were scheduled to take two to three weeks, STP's shift supervisor had 10 reason to be concemed about a PMT requirement that would require the diesel to be declared ii inoperable or taken out of senice for painting and then started daily after completion of the day's 12 painting activities. Frequent starts with short run times have been recognized by the nuclear 13 industry for some time as having the potential to increase wear and reduce SDG reliability. The la identified altemative, declaring the diesel inoperable for two to three weeks and waiting to perform 15 the PMT, was also judged not to be a viable one. As an attemative, STP took what it considered 16 to be reasonable precautions to eliminate the possibility that painting would affect operability.

/ 17 These included:

is o pre-job briefings to explain the issues and STP's concerns, 19 o a requirement for the contractor to use his best five painters and have them work 20 under the direct supervision of a contractor foreman, and 21 o assignment of the system engineer to oversee the painting and ensure that paint was 22 not applied in areas that might affect operability.

23 Because of these precautions, the shift supervisor was convinced that operability would not be 24 afTected and elimmated the requirement for the PMT. After the painting was complete, the diesel 25 failed to start during a regularly scheduled test run. The primary cause of the event was the lack 26 of proper application of work process controls. The decisien to delete the PMT and not perform j 27 the test immediately after painting was, in hindsight, an unfortunate one, but was reasonable at the j 28 time. Moreover, the incident did not demonstrate that the PMT program was weak or provide 29 evidence of PMT planner error or a compromise in shift supervisor judgment because of an i 30 excessive work burdcn.

s Page 111-62 The Liberty Consulnng Group  ;

l

RU .'

Chapter Three - \laintenance

(% l 1 PNIT was specified. but the decision made to delete the Ph!T was believed to be the selection of 2 a reasonable option that would best ensure SDG reliability and least affect plant safety. The shift 3 supersisor was assured that adequate work controls were in place to preclude the possibility that  ;

.: operability would be affected. Liberty concluded that the actions taken in this matter were 5 reasonable.

6 3. Diesel Breaker 7 In its second example the DET said;"7 8 "PhtT was not performed on a SDG output breaker after a fuel oil injector pump 9 was repaired. During that maintenance activity, the output breaker was racked out to to support work on the injector pump and later improperly racked in. For PSIT the 11 SDG was started but breaker closure was not tested. During a subsequent 12 surveillance test, the SDG output breaker would not close onto the bus."

13 This event occurred in September 1991,1 % years before the DET evaluation. While a question of 14 relevance is raised by using an old example to judge the adequacy of a current PhfT program. it O 15 was accurate to state that failure to specify the proper PhfT had contributed to the amount of time N 16 the SDG was inoperable."' Failure to follow procedures for racking in the breaker was the primary 17 cause of the event.

18 As part of the action taken to prevent recurrence, STP implemented an operations policy that 19 required electrical breaker continuity tests on all safety related 480v,4.16 kv, and 13.8 kv breakers 20 that were racked out for any reason. STP's corrective actions were reasonable and appropriate and 21 reflected proper concern for adequate testing after maintenance to ensure operability of equipment 22 important to plant safety.

DET Report, p. 24.

\ '"

NRC Inspection Report No. 91-25, ST AE-HL 92910, November 12,1991.

Page 111-63 The Liberty Consultmg Group

?L5 :

Chapter Three \taintenance V 1 4. Chiller Breaker '

2 in discussing its last example. the DET said:'"

3 "After work was performed on the feeder breaker for essential chiller 21C, no PSIT a was performed. yet the chiller was declared operable. The following day the 5 chiller's feeder breaker tripped during a routine start attempt due to breaker e problems."

7 The DET implied that a Ph1T program weakness had been the cause of chiller inoperability. The 8 reason PNiT was not performed had nothing to do with the Ph1T program. the proper specification 9 of PhiT by the planner, or a decision made by an overburdened shift supervisor. Rather, the failure 10 to perform PhiT was the direct result of a change in work scheduling. Two service requests for the 1I breaker were scheduled to have been performed concurrently. The two work packages had been 12 designed to be performed concurrently but were actually performed separately. One contained the 13 required PSIT and was completed earlier than originally scheduled. When the second service la request was performed, the required Ph1T was not performed because the second work package 15 made no mention of PhiT. The station problem report that documented the event correctly 16 attributed the oversight to a work organization and planning error, not the Ph1T program." The y) 17 event had no significant effect on safety and was not reportable. The other two independent ECW 18 trains were continually operable, as required by Technical Specifications.

19 5. HL&P's Actions 20 On the basis of recommendations made in two station problem reports, STP took action prior to 21 the DET' evaluation to identify PhiT work document and process improvements in order to 22 enhance and upgrade the tools used by the planners to identify the appropriate PhfT."'

23 STP organized a Task Committee to review testing programs and methodologies in order to 24 identify opportunities for making improvements and refinements in the PhiT program."2 After the 25 committee completed its work, and PhiT guidance was consolidated into one procedure, the PhiT DET Report, p. 24 Station Problem Report 920384, DET response item 83671-001.

A HL&P Memorandum Maint-93-9-0134, March 8,1993, DET response item #1102.

DET response item #1102.

Page 111-64 The Liberty Consulting Group

RLS-:

Chapter Three - %1aintenance

.0 Cf 1 N1anual was revised. and the assignment ot' responsibility for specification of tests required for 2 operability was assigned to the Operations Work Control Group."'

3 Recognition of a need for improvement in PNIT processes and work practices before the DET's 4 evaluation began reflected STP's commitment to improving the effectiveness and efficiency of 5 corrective maintenance. This is a normal enhancement process for any utility. STP's initiative in 6 idennfying opportunities for improvement and pursuing them was a reasonable action.

7 6. Other Evaluations of STP's PSIT 8 STP received the highest performance rating in the hiaintenance/ Surveillance area in the first 9 SALP after both units went into commercial operation. One of the strengths noted by the NRC was 10 STP's post-maintenance test program.

1I "The licensee has a strong program for determining the need for retest and 12 identification of approp-iate retest type. Also, good procedures were developed and 13 implemented for postmodifications and postmaintenance retests.""'

(

\ 14 In a special Maintenance Team Inspection review conducted in 1990, the NRC made a similar is assessment of STP's PMT program and its implementation.

16 " Post-maintenance testing was a strong program with adequate criteria, well 17 implemented at the planning level, and performed well in the field."'"

18 Unlike the DET's assessment, which covered a wide range ofissues, the NRC's MTI inspection 19 was a focused, in-depth evaluation of only the maintenance function and was performed over a 20 three-month period in early 1990. Its purpose was to determine the need for additional NRC 21 rulemaking to improve the conduct of maintenance at nuclear facilities whose performance was 22 poor. STP ranked " good" in maintenance programs and " satisfactory" in implementation, an 23 overall rating that put STP among the better plants with respect to its maintenance program.

'" HL&P Memorandum Maint 93-4-0139, J.M. Gmber to T.N. Lucas on SPR 931310, October 1,1993.

p NRC SALP Repon, ST AE-HL-92567, June 20,1990, p. 20.

kx '"

NRC Inspection Report No. 90-01, ST AE HL-92444, May 10,1990.

Page 111-65 The Libery Cortsulung Group

?L 5.:

Chapter Three \laintenance p

(j N i The MTl concluded that:"'

2 "The licensee had effectively implemented a strong work order planning program.

3 Work orders were comprehensive and accurate, included appropriate post a maintenance testing, and were subject to meaningful reviews during closecut.

5 The post maintenance program was well implemented."

6 The SRC's SALP report that covered the period from June 2,1991 through August 2.1992 did 7 not specifically cite PMT as a strength but did render a favorable finding on STP's maintenance 8 program as a whole:"'

9 "The licensee's preventive and corrective maintenance programs were considered 10 good. Several strengths were identified. The licensee had a good work control ii process that provided for the identification of equipment problems, evaluation of 12 equipment operability, work activity prioritization, conduct of maintenance 13 activities and proper closure of work packages."

14 Given the limited information in the DET report, a review of LERs is useful and can help to put 15 into perspective the ultimate effect of STP's PMT program on plant safety. Before the DET 16 inspection, the last LER that involved PMT as a contributing cause of a reportable event occurred 17 on January 20,1993 and related to painting work performed on a standby diesel generator /SDG/.

18 The next most recent LER that involved adequacy of PMT as a contributing cause of a reportable 19 event occurred in early 1991."' A required hydrostatic test of a steam line was not performed after 20 a weld repair of a steam cut. PMT had been specified and performed, but the test performed was 21 incorrect for the class of piping that required the repair. Upon discovery of the discrepancy, the 22 pump was declared administratively inoperable from the time the test should have been performed 23 until the test omission was discovered. Upon discovery of the omission, the proper test was 24 performed immediately and was successful. This confirmed that the pump had been physically 25 operable during the period ofits admmistrative inoperability, and that there was neither a reduction 26 in the capability of the safety system nor an adverse effect on plant safety.

NRC Inspection Report No. 90-01, ST AE-HL-92455, May 10,1990, p.16.

"' NRC SALP Report, ST-AE HL-93239, November 18,1992.

)

"' LER 9 l-002, Unit 2, March 4,1991.

Page 111-66 The Lsberry Consuhing Group l

4 RL5-:

Chapter Three - \taintenance O

V 1 Only two other LERs involving PMT performance have occurred at STP since the plant went into 2 operation. and both were in 1989? All three LER events involving PMT in the period 1989-1991 3 occurred at a time when STP's PMT program and its implementation were regarded by the NRC a as strengths. No LER after 1991 identified PMT as the primary cause of the event. L' sing LERs 5 as a measure. Liberty concluded that the effectiveness of STP's PMT program had not degraded.

6 In terms of results and effectiveness, there was no convincing basis for management to conclude

, 7 that the PMT program was less effective just prior to the DET than it was when it was 8 characterized as a strength by the NRC. Liberty found that the facts did not show that this area had 5 9 been managed imprudently. Comments made by the DET regarding difficulties with the PMT 10 work process should not be confused with unreasonable actions or imprudence. Over time.

Ii feedback from the PMT program had been used to improve its effectiveness, but at the expense 12 of making the PMT Manual used by the planners more complex and difficult to use. STP formally 13 recognized the evolving work process difficulties in 1992 and initiated actions to streamline and

, 14 improve the work documents used by the planners and to streamline the process.

A

( is B. Periodic Testing l

l 16 The DET's comments about STP's periodic testing involved two issues. First, the DET described 17 a program that STP had implemented to ensure that Technical Specification surveillance 18 procedures were adequate. Initially, the review included those procedures carried out by operations 19 and maintenance. STP then expanded the scope of the procedure enhancement program to include 20 those tests performed by engineering.

21 The second matter involved STP's discovery that in May 1990 the control room filtration system 22 had exceeded the surveillance limits specified in Technical Specifications. The DET described the 23 event as follows:"

l.ERs89-010 and 89-022.

] "

DET Report, p. 25. s hge 111-67 The Liberty Cortsultmg Group l

l i

PU-:

Chapter i hree - \laintenance i "[T]he team questioned the licensee conceming an engineering test of the control 2 room emergency ventilation recirculation charcoal adsorbers. Subsequently, the 3 licensee determined the surveillance requirements had not been satisfied in that a 4 defective method had been devised to determine when adsorber testing should be 5

performed. The failure to send the charcoal sample for testing within the required 6 inten al resulted in a 3-month delay in determining that the charcoal bed was below 7 require standards for iodine adsorption. If the adsorbers had been tested at the 8

proper test interval, the results may have indicated the degradation, prompting 9 adsorber replacement before the standards were not met."

10 STP's Technical Specifications require that a charcoal adsorber sample from the control room ti emergency ventilation system be analyzed for methyl iodine penetration every 720 hours0.00833 days <br />0.2 hours <br />0.00119 weeks <br />2.7396e-4 months <br /> of 12 adsorber operation. If the sample results are satisfactory, the absorber is considered operable until 13 the next sample is required. During normal operation, the hours accumulate very slowly, but la during an outage the service hours can accumulate rapidly. Also during an outage, one train of the 15 control room ventilation system is often out of senice for maintenance, and the Technical

~

16 Specifications do not allow taking another train out of senice to take the sample. The Technical 17 Specifications do allow for a 25-percent grace period on the 720 hours0.00833 days <br />0.2 hours <br />0.00119 weeks <br />2.7396e-4 months <br />, or a maximum of 900 18 hours. "'

[

\

19 STP's surveillance procedures were based upon the assumption that service hours on the adsorber 20 were the same as operating hours on the control room ventilation make up fans. However, because 21 of a Technical Specification change, a mode of operation called filtered recirculation was used that 22 operated cleanup fans without the makeup fans. The change in Technical Specifications had been 23 required to resolve a contradiction between two specifications. During a period in 1990, the 24 cleanup fans ran independent of the makeup fans for about 62 hours7.175926e-4 days <br />0.0172 hours <br />1.025132e-4 weeks <br />2.3591e-5 months <br />. These hours, when added to 25 the 866 hours0.01 days <br />0.241 hours <br />0.00143 weeks <br />3.29513e-4 months <br /> accumulated (before the next sample) on the makeup fans, produced a total that 26 exceeded the 900-hour lirnit by 28 hours3.240741e-4 days <br />0.00778 hours <br />4.62963e-5 weeks <br />1.0654e-5 months <br />. The analysis results were unsatisfactory and the adsorber 27 was replaced. STP discovered and reported the violation of Technical Specifications.

28 The cause of the technical specification noncompliance was the failure to calculate sen ice hours 29 considering the possible independent operation of the makeup and cleanup fans. Contributing to 30 this cause was that at the time of the Technical Specification change (May 1990), the procedure 31 for reviewing Technical Specification changes did not provide specific instructions for j LER 93-015, ST HL.AE-4450, May 21,1993 (included in DET response item #3201), and DET response

( item #2356.

Page 111-68 The Lsbeny Consultmg Group

RLS .

Chapter Three - \taintenance fh d i interdepartmental reviews. The example cited in the DET report was an isolated incident that took i

l 2 place in 1990 and thus was not a reflection of poor periodic testing practices at the time of the DET 3 evaluation." Indeed, subsequent to that event and prior to the DET's evaluation. a SALP a evaluation of STP's surveillance and testing program gave the following feedback to STP's 5 management:"3 6 "The surveillance and testing programs were effective. Surveillance tests were 7 . being scheduled and performed as required by TS. The missed surveillance rate 8 was extremely low. Overall, surveillance procedures were determined to be of high 9 quality."

10 Liberty concluded that HL&P's program to ensure that test procedures were adequate was a 1i positive and reasonable action. The specific example concerning the control room filtration system 12 did not reflect a weakness in the periodic test program that would alert management to a need for 13 significant changes in this area. Finally, the feedback STP received from the NRC prior to the u DET's evaluation indicated good surveillance test program performance. .

(G p)

(

U "2

DET response items nos. 2356,2356-001, and 3201.

NRC SALP Report, ST-AE-HL 93239, November 18,1992,p.I1.

Page 11169 The L&erty Consultmg Group

- . . _. . . . ._ _ . m. _ _ . . _

?L5-:

Chapter Three - 31aintenance

, - t IX. Turbine-Driven Auxiliary Feedwater Pumps 2 A. Introduction 3 The auxiliary feedwater system is a safety-related system common to pressurized water nuclear l 4 plants. The purpose of the system is to supply water to the seconday side of the steam generators ,

5 any time that the normal feedwater system cannot do so. For example, following a reactor trip, the 6 auxiliary feedwater system can add relatively colder water to the steam generators so that heat can ,

7 be removed from the reactor coolant system by heat transfer in the steam generators.

s Each unit at STP has an auxilian feedwater system. Each system has four pumps with independent ,

9 motive power. Three of the pumps are driven by ele:tric motors and the fourth is driven by a steam 10 turbine.'" The purpose of having a turbine-driven pump is that it will operate even if no electrical i

1i power is available. Each turbine-driven pump has an overspeed trip mechanism installed so that 12 the turbine and pump will not be destroyed if excessive steam drives the turbine too fast. -

13 STP's auxiliary feedwater systems experienced overspeed trips of the turbine-driven pumps. These  :

la overspeed trips caused HL&P to shut down Unit I and keep Unit 2 shut down in early February 15 1993. The NRC acknowledged HL&P's actions in a Confirmatory Action Letter that indicated 16 HL&P would not start up the plants until HL&P had briefed the NRC on the efforts to correct the i 17 overspeed trip condition of the turbine-driven auxiliary feedwater pumps."' Therefore, the -;

18 overspeed trip issue was significant in that it was the basis for the concem that had initially caused ,

19 STP to be shut down in 1993. It also resulted in special NRC inspections.

20 B. Overspeed Trip Initiating Event .

21 A surveillance test was conducted on the turbine-driven auxiliary feedwater pump (TDAFWP) of 22 Unit 1 on February 1,1993. The test resulted in an overspeed trip of the pump, which led to a 23 declaration that the pump was technically inoperable. This situation also required that the unit 24 enter *a Technical Specification 72-hour action statement during which either the deficiency had ,

25 to be resolved or the plant had to be placed in a mode in which the pump was not required to be 5

26 operable.

STP Updated Final Safety Analysis Report, Section 10.4.9.

A "' Letter, NRC to HL&P, Milhoan to Hall, Confinnatory Action Letter, February 5,1993.

Page 111-70 The Liberty Consulung Group l

RLS-:

Chapter Three - \laintenance C

( l 2

While the Unit I pump problem was being investigated. Unit 2 expurienced a reactor trip on February 3.1993. Following the reactor trip. the Unit 2 TDAFWP experienced an overspeed trip.

3 Because the cause of these TDAF%7 overspeed trips could not be readily determined. STP's management decided to shut down Unit I and to maintain Unit 2 in its shutdown condition until 5 the os erspeed problems could be identified and resolved.'" I

^

6 C. History of Overspeed Trip Problems 7 1. Unit 1 8 During the startup process of Unit 1 in 1988, the TDAFWP experienced a series of three overspeed 9 trips. HL&P concluded that these trips had been caused by excess condensate and modified a valve 10 line-up on a steam trap so that it could remove the condensate and thereby remedy the problem.

11 An augmented testing program over several months verified that the modification had been j 12 etTective. and the problem did not recur. Normal periodic testing was resumed in June 1988. Aftet 13 this modification, the Unit 1 TDAFWP responded as designed with no overspeed trips until June 14 11,1990, when a trip resulted from starting the pump too soon after it had been secured. During 15 1991 and 1992 there were at least 27 TDAFWP starts with no overspeed trips prior to December 16 27,1992."'

17 Following the 1992 Unit I refueling outage, a successful surveillance on the TDAFWP was 18 performed. The turbine trip throttle valve, MOV-0514, was known to be leaking and was 19 disassembled and inspected to see whether any improvements could be made; however, the stellite 20 disc and seat surfaces had been damaged by steam flow and could not be repaired. Replacement 21 parts for repairs were not available. The valve was therefore reassembled and returned to service.

22 Minor leakage by the seat of the trip throttle valve should not have impaired system operability.

23 Subsequently, however, an overspeed trip occurred on December 27 during a post maintenance .

24 test (PMT). After considerable additional successful testing, the pump was declared operable.'"

"* Special Report Concerning Recent Turbine Driven Auxiliary Feedwater Pump Overspeed Events prepared by the Corrective Action Group. 3/11/93, DET response item #2065.

i

"' Station Problem Report 930431, DET response item #2065.

"8 Station Problem Report 930431. DET response item #2065.  ;

Page 11171  ;

The Liberty Consuhmg Group I

PL 5-Chapter Three - \1aintenance A

i On January 28,1993, during a planned outage, work was performed on the pump and turbine. The 2 subsequent PMT resulted in an overspeed trip. A slow manual start was attempted with the same 3 result. At HL&P's request, the manufacturer of the turbine governor sent a representative to the a site on January 30. He determined that the govemor valve was not closing fully. Following repairs.

5 HL&P successfully conducted several starts, including a satisfactory PMT. The fact that the 6 governor was not closing fully was consistent with the overspeed trips, so after the adjustments.

7 HL&P had reason to believe that the deficiency had been corrected. The pump was declared 8 operable that same day. Then on February 1, HL&P decided to conduct another test, and an 9 overspeed trip occurred. An additional 19 tests were performed and 5 trips were experienced.'"

to it was determined that there was standing water in the turbine casing due to a plugged or partially 1i closed drain. The drain was fixed and a series of three tests was conducted; all of the tests were 12 successful. The pump was considered operable by February 4, but the 72-hour action statement u had already expired, and, therefore, Unit I was being shut down. Since Unit 2 had experienced la similar trips, HL&P decided to continue the shutdown of Unit 1."

Os is 2. Unit 2 V

16 During the startup of Unit 2 in February 1989, the TDAFWP experienced a series of 18 overspeed 17 trips. AAer a number of service requests had been completed, a successful test was performed on 18 March 3 and the pump declared operable on March 4. Between that time and late January 1993, 19 the pump was challenged 88 times, resulting in 5 overspeed trips. At least two of the five were 20 caused simply by starting the pump too soon after it had been secured. During 1992 there was a 21 minimum of 14 starts with no overspeed trips."'

22 On January 8,1993, a successful surveillance test was performed. On January 23 a reactor trip 23 occurred and the TDAFWP actuated and operated satisfactorily. When the pump was secured, a 24 mechanical overspeed indication was received in the control room. While the overspeed trip device 25 actuated, no physical overspeed took place. While this situation was being investigated and tested 26 on January 24, an overspeed trip occurred. It was concluded that the trip had probably occurred Station Problem Report 930431, DET response item #2065.

" Station Problem Report 930431, DET response item #2065.

]

Station Problem Report 930431, DET response item #2065.

Page 111-72 The Liberry Consuhmg Group l

I

a RU.

Chapter Three - %1aintenance b]

f 1 because of moisture buildup caused by closing a valve. HL&P also discovered that the pin in the

mechanical trip linkage was the wrong diameter ( % inch instead of the specified % inch). The

. 3 vendor recommended that the spring tension be increased to compensate. Finally, it was a determined that the seat drain steam trap bypass valve was closed, inhibiting the draining of 5 condensate. After several adjustments had been made, five tests were satisfactorily run, including 6 three PMTs. The pump was declared operable on January 30. On February I HL&P adjusted the 7 spring tension on the trip linkage. On February 3 Unit 2 had a reactor trip and the TDAF%7

. 8 experienced an overspeed trip. Because of similar difficulties with the pump on both units HL&P 9 decided to keep Unit 2 shut down.n2 l

10 D. Corrective Actions Ii Hl.&P took many corrective actions to improve the design and performance of the systems

. 12 associated with starting and operating the TDAFWPs. These actions can be categorized as; 13 equipment improvement to avoid steam intrusion iato the turbine casing, to prevent accumulation 14 of condensate, and to gain more positive control of the govemor; revised maintenance and testing A 5 procedures; and clearer field labeling "3 The equipment changes included refurbishment of the trip I

(t-) 16 e d throttle valves and the govemor, modification of the drain system, and relocation of the steam 17 teak-off systems. Procedural changes included adding a number of verification tests to the testing 18 program, revising the system description to reflect the new design, revising training materials, and 19 incorporating new vendor information. A monitoring program to detect leakage in MOV-0514 was 20 established along with augmented surveillance testing, system line-up verification, and multiple 21 reviem of these changes by oversight committees and a contractor. These actions were 22 reconfirmed and supplemented in a letter to the NRC."' This letter addressed several generic items 23 that dealt with problem prevention, identification, and resolution. Rese additional actions included 24 the establishment of an Operations Work Control Group that had responsibilities to monitor and 25 control incoming service requests, a training program to enhance vertical and horizontal 26 communications skills, and a revised process for addressing industry operating experience.

"2 Station Problem Report 930431. DET response item #2065.

(A

'" LER 93-007, March 5,1993.

\

"' Letter, HL&P to NRC, ST-HL- AE-4477, June 25,1993.

Page lil '!3 The Liberty Comuhmg Group

1 1

Rt.5:

Chapter Three - \laintenance O\

l E. Evaluation 2 HL&P formed an Event Response Team on February 5.1993 to perform a root cause analysis, to 3 identify needed corrective actions, and to determine any generic implications of the TDAFWP 4 events.'" The team included representatives from Design and Plant Engineering, Quality 5 Assurance, and the Corrective Action Group at STP as well as vendor representatives. A contractor 6 (Sargent & Lundy) was also employed to perform an independent assessment.

7 As a matter of policy, the NRC carries out timely inspections of significant operational events at 8 nuclear power plants. The TDAFWP trips at STP were considered significant because they 9 represented a failure of safety-related equipment to perform its designed function, and the NRC 10 dispatched an Augmented Inspection Team (AIT) to STP on February 5,1993. The NRC's team il included four inspectors and covered the period from February 4 to February 24,1993."'

12 HL&P's team characterized the root cause of the Unit 1 TDAFWP failure as inadequate materiat 13 condition. Their findings included: the turbine casing exhaust dram was partially blocked, 14 govemor valve stems were pitted, the steam leak-off from MOV-0514 was piped to the same drain O 15 as the casing drain, valve seat leakage in MOV-0514 was increasing, and the design did not 16 provide sufficient margin to trip because of the MOV stroke time and the govemor valve 17 response."'

18 The NRC's AIT said that their fmdings conceming the causes of the problems were " consistent to with the licensee's findings." They reported:"'

20 "For the Unit 1 TDAFWP 14, no defmitive root cause was identified. However, the 21 AIT considered the misadjustment of the govemor valve linkage to be the most 22 probable cause of the overspeed trips, This misadjustment, which occurred during 23 the previous plant outage, reduced the govemors' ability to control turbine speed."

'" Letter, HL&P to NRC, ST HL-AE-4334, February 11,1993.

NRC Inspection Report No. 93-07, ST AE-HL-93340, March 24,1993.

, Station Problem Repon 930431, DET response item #2065.

d "'

NRC Inspection Repon No. 93-07, ST-AE-HL-93340, March 24,1993, p. 23. ,

Page 11174 The Liberty Consultmg Group

i RL5-:

Chapter Three - \taintenance R -

1 HL&P found that the root cause of the Unit 2 TDAFWP failure had been ineffective

implementation of a configuration change on the upstream drain system.'" The NRC agreed.

3 stating:"'

4 "For the Unit 2 TDAFWP 24 overspeed trip, the root cause was determined to be 5 a condensate build up upstream of MOV-514 caused by an incorrect valve lineup 6 ,

combined with an inoperable or degraded steam trap in the drain line for the steam 7 admission line. This caused a slug of water to enter the turbine and result in a 8 turbine overspeed."

9 A valve that bypassed the steam trap in question on Unit 2 had been shut sometime in April 1992.

10 Based on temperature data, it was determined that the steam trap had performed its function until ii late January 1993. The trap then degraded in performance and condensate became trapped in the 12 steam admission piping."'

13 In addition to examining the root causes of the problems, the NRC's AIT examined several other 14 matters. The team's conclusions included:a2 0

\, 15 o HL&P's " preventive maintenance program was being accomplished for the 16 TDAFWPs." HL&P had " performed the proper corrective maintenance on both 17 unit's TDAFWPs, when the need for maintenance was identified."

18 o The turbine " speed control systems did not operate as intended."

19 o HL&P's "PMT activities were found to be appropriate for the maintenance 20 conducted. In general, the PMTs reviewed by the AIT were technically adequate 21 and licensee personnel were technically knowledgeable in their performance."

During the month following the AIT's inspection, the NRC performed another inspection of the 23 TDAFWP issues. In this inspection the NRC documented apparent violations concerning

'" Station Problem Report 93-0431,3/11/93, DET response item #2065.

'" NRC inspection Report No. 93-07, ST-AE-HL-93340, March 24,1993, p. 23.

, NRC Inspection Report No. 93-07, ST AE-HL-93340, March 24,1993, p. 8.

"2 NRC Inspection Repon No. 93-07, ST AE HL-93340, March 24,1993, pp. 24 25.

Page 11175 The Liberry Comultmg Group I

l

- ? '.5..'

Chapter Three - \1amtenance m

I maintenance by unqualified personnel and inadequate procedures for assuring that the TDAFWPs 2 were in their normal ambient conditions prior to perfonnance of the surveillance test. Since the 3 sur eillance tests might not have been adequate, an additional apparent violation was noted. which a related to operating the plant without completely tested equipment. Additional apparent violations 5 related to inadequate detail in the procedure for adjusting the TDAFWP govemor valve and having 6 an incorrect valve line up.'" These findings were made despite the fact that the NRC's AIT, while 7 aware of all these conditions, had not identified these matters as violations. The NRC's apparent 8 violation regarding the inadequacy of the surveillance test is noteworthy for its use of hindsight.

9 Neither HL&P nor the NRC recognized during the long and concentrated review of overspeed trips to that the pre-test ambient conditions of the pump might affect the test's results. Once the situation

!! had finally been recognized, however, the NRC found HL&P in violation of NRC regulations by 12 having inadequate test procedures.

13 The DET inspection followed both the AIT inspection and the follow-up inspection on the 14 TDAFWPs. The DET report mentioned the overspeed trip problem as an example "of poor root 15 cause determination and poor maintenance efforts."'" The report stated:'"

16 " Repeatedly, the overspeed trip tappet of a turbine driven auxiliary feedwater pump 17 (TDAFWP) did not retum to its normal position after a manual or overspeed trip.

18 Re initial corrective action involved removing a sticky tar-like substance from the -

19 tappet and the upper turbine housing. Personnel did not determine the cause of the '

20 tar-like substance and took no action to preclude its recurrence. Approximately six 21 months later the tappet stuck again in its tripped position when the turbine was -

22 manually tripped."

23 None of these problems was found to have anything to do with the unwanted overspeed trips of 24 the TDAFWPs. Although they were associated with TDAF.WP overspeed trips, they were not 25 causes of the problems in early 1993. Also, the DET's characterization of the corrective 26 maintenance was quite different from that provided in the AIT report, which detailed all 27 maintenance that had been performed on the TDAFWPs. The AIT report indicated that corrective 28 maintenance had been performed on the overspeed trip mechanism for both Units 1 and 2. On at 29 least one occasion on each unit, the mechanism failed to reset after a trip. On one such  ;

I NRC Inspection Report No. 93-05, ST-AE HL-93365, April 8,1993.

DET Report, p.15.

t' '*' DET Report, p.16.

Page 111-76 The Liberty Consultmg Group l

RL.; .'

Chapter Three - \taintenance n

t maintenance, but not the " initial" one. a tarry substnce interfered with a trip plunger operation.

2 About four months later, according to the AIT. the overspeed trip plunger was found to be sticking.

The AIT noted that HL&P might not have been pursuing these matters to a sufficient extent.

4 The DET also stated;"6 5

^

"The mechanical maintenance staff was not trained to maintain the TDAFWP 6 govemor or the TDAFWP overspeed trip mechanism. This contributed to the 7 numerous unsuccessful attempts to resolve problems on TDAFWPs."

8 Neither the AIT nor the follow-up NRC inspection on the TDAFWP trips identified training of any 9 kind as a contributing cause of the problem. Both the AIT and the follow-up inspection made note to of the maintenance that had been performed by unauthorized personnel. But these personnel were 1I not part of the mechanical maintenance staff and did not cause any mechanical problems. The 12 follow-up inspection indicated that there was not enough detail in the work instructions for 13 adjusting the turbine govemor. However, even the vendor manual did not contain detailed" 14 instructions."'In any event, the adjustment of the turbine govemor was not a significant cause of 15 the overspeed trips; the overspeed trip on Unit 1 on February 1 occurred after the vendor had IO 16 17 corrected the govemor valve adjustment. It is not clear how the DET reached a conclusion that was different from those reached by inspectors who were concentrating in the area of TDAFu?

18 problems.

19 The DET report included the comment:'"

20 "In December 1992, during maintenance to repair an AFW turbine trip throttle 21 valve, a replacement disc and seat were not available in the warehouse. The valve 22 was reassembled and the system declared operable. This leaking valve contributed 23 to numerous overspeed turbine trips in January and February of 1993." ,

24 The failure to have replacement parts for a single particular valve may or may not be indicative 25 of the adequacy of the entire replacement parts program. HL&P's decision to disassemble the 26 valve, hoping that some repairs could be made, was reasonable despite the unavailability of spare DET Repon, p.19.

Letter, HL&P to NRC, Response to Notice of Violation, ST.HL AE-4477, June 25,1993, Attachment 4 A 'd DET Repon, p. 20.

Page 111-77 The Liberty Consulting Group

RL S .'

Chapter Three - \taintenance i parts. Furthermore. the trip and throttle valve leakage was determined only to be a possible J contributor to the Unit I trip.

2 3 In May 1993 the NRC formally issued a Notice of Violation and imposition of civil penalty related 4 to the TDAFWP issues. The facts underlying the violations were the following: (1) the Unit 1 5 TDAFWP had possibly not worked as required between December 26,1992 and February 4.1993.

6 (2) the surveillance test procedure did not specify that the pump should be in its normal standby 7 condition before testing, (3) the governor valve maintenance instructions did not contain sufficient 8 acceptance criteria, (4) there had been a valve misalignment in Unit 2, and (5) maintenance had 9 been performed by unauthorized personnel. The fme imposed for these violations was S175.000.'"

io F. Conclusions iI HL&P made the decision to shut down Unit I and to keep Unit 2 shut down until the situation with 12 the TDAFWPs could be resolved. HL&P recognized that there had been multiple failures and, 13 because it was not known whether the failures in both units were related, decided that it was 14 necessary to identify the root causes of the problems before further operation. The NRC's 15 confirmatory action letter simply sanctioned HL&P's decision.

16 HL&P acted promptly and reasonably by establishing an Event Response Team on February 5.

17 1993. HL&P's team was constituted in a reasonable way. The NRC also took immediate action 18 and dispatched an AIT to the STP site. The NRC found that its determination of the root causes 19 of the TDAFWP problems was essentially the same as that made by HL&P.

20 The complexity of the issue, as demonstrated by the many factors that had contributed to 21 overspeed tripping, made it difficult to recognize the trips as being indicative of a water drainage 22 problein. Each time there was a problem, HL&P had reasonable assurance that it had taken action 23 sufficient to correct the problem. Because problems with the TDAFWPs were of such a varied 24 nature and because previous corrective actions appeared to be satisfactory,"the system engineer 25 did not consider the various failures as indicators of a different and larger [or generic] problem.""

26 With the benefit of hindsight, a case can be constructed that STP should have recognized that the 27 surveillance test procedure might have been giving false indications of TDAFWP reliability.

Letter, NRC to HL&P, Milhoan to Cottle, ST-AE-HL-93428 May 28,1993.

\j "'

Letter HL&P to NRC, ST-HL-AE-4434, May 11,1993.

Page 11178 The Lsberry Consultmg Group

l RL S. .'

Chapter Three - Maintenance O

() 1 Howes er, the fact that the NRC issued an Information Notice to all operators of nuclear power plants about this problem indicated that the requirement that the pumps be in normal ambient 3 conditions before testing was a subtle one and one that the NRC and other plant operators might l

.: not have been fully aware of. "It is expected that recipients will review the information for 5 applicability to their facilities and consider actions, as appropriate, to avoid similar problems."

6 HL&P took comprehensive corrective actions to ensure that this problem and others like it would 7 not occur again. The NRC's AIT found some practices at STP that could have been improved but 8 agreed with HL&P's root cause analysis, maintenance, and testing associated with the TDAFWPs.

J 9 Libeny concluded that HL&P's handling of the overspeed trips of the TDAFWPs was prudent. A lo review of the entire history of the pumps led to the conclusion that STP had had reason to believe ii the pumps were reliable and prior deficiencies had been resolved. When overspeed trips occurred 12 in early 1993, HL&P acted in a thorough and safety-conservative manner.

('

n (j

NRC Infor nation Notice 93 51, Repetitive Overspeed Tripping of TD-AFWPs, July 9,1993.

Page 111-79 The Libery Comuhmg Group t

m A s.4 4 - emm. --n w_,.. .e Esmo.-.

- a._s 4 a_, .am. .s-s A,a.,__w__.2.a A._4,y,m2.mme aWaA.re.cy,g- e, mumm,,_ mA .gaur_a_r.m_.____.pw.o,,ymj 4 s , a wa_.

% y ,_,p_gg,,__

t e

iI e

l I

4 e

i J

q 2

i i

i i

)

e r

I l

- . _ . ______ _ __ _ ____~__ _ _ . _ _ _ _ _ __.-_ . . _ ... ___ _ _ _ _ _ _ _ ,._._....__.__ .._ .. _ ., ,, , _. _ _ . - . , . . . , _ . , . _ .

l RLi~.'

Chapter four - Engineering i Chapter Four - Engineering 1 l

2 1. Introduction and Summary of Findings 3 The DET said that it had found a dedicated engineering staff at STP that had produced a comprehensive major modification and good design basis programs. However, the DET also 5 believed there were weaknesses in the engineering support provided at STP. At the root of many 6 of the DET's concerns was the conclusion that the number of engineers at STP was insufficient.

7 Consequently, the DET said that engineering's actions took too long and the backlog of work was 8 large and growing. In addition, the DET concluded that system engineering was not sufficiently 9 effective.

10 Liberty reviewed the engineering activities criticized by the DET and, in the context of the iI prudence standard, determined that STP management had made reasonable decisions and taken 12 reasonable actions given the knowledge and circumstances that existed at the time. Moreover, is 13 many cases Libeny found that the information relied upon by the DET as the basis for its findings 14 was either incomplete or preliminary. Liberty found that the decisions made by STP's management 15 regarding the size of the engineering staff had been reasonable and had appropriately considered 16 expert advice, industry experience, and plant specific factors. With respect to the other criticisms 17 of engineering, Liberty determined that HL&P had identified the significant issues, and had taken is actions to resolve those issues. It is the use of hindsight (which is inappropriate in a PUC prudence 19 proceeding) that allowed the DET to conclude that engineering's actions had not been effective. l 20 Finally, Liberty concluded that some of the matters discussed in the DET report need to be 2i examined in light ofindustry experience, and the significance of the issues presented in the DET 22 report needs to be carefully considered.

23 Management recognized that STP's system engineers had a heavy work load and, prior to the 24 DET's evaluation, had made adjustments in the division of responsibilities. Liberty found that 25 engineering knew the amount of work it had to perform and prioritized that work reasonably. l 1

26 Similarly, Liberty concluded that engineering at STP was effective in its use of operational 27 experience. Contrary to the conclusion that might be drawn from the DET report, STP had 28 outstanding programs in probabilistic risk assessment and configuration management.

29 The DET's criticisms of engineering support at STP stood in stark contrast to prior NRC views. l 30 For example, the NRC conducted an extensive inspection of the engineering and technical support Page IV-1 The Luberry Consultmg Group

?LS-:

Chapter Four - Engineering V i functions at STP in February 1992.' That evaluation addressed all major aspects of these functions 2 and included two topics that involved significant interface relationships with other site ,

3 organizations. These two topics were station modifications and design change work packages.

.t Among the numerous strengths cited by the NRC were the fellowing:

5 "[M]odification packages. .were well written and complete. Considerable effort 6

had been incorporated into the modifications to identify and address all issues of 7 safety significance."

8 "[T]he technical engineering response to. . Conditional Release 9 Authorizations...were well documented and reflected conscientious and to conservative efforts to resolve the identified problems."

11 "The temporary modification program was found to be functioning properly."

12 "The inspectors found design engineering to be a hard working, dedicated group

!3 and that engineering was producing a quality product." -

1. "The new design basis documents were viewed as reliable and complete design 15 aids."

r

/

( 16 The inspection report also noted some positive, forward looking programs that were being pursued 17 by HL&P. "The licensee has developed a significant number ofinitiatives to enhance the plant and is its performance including comprehensive DBD and PRA programs."("DBD" and "PRA" referred t'

19 to design basis document and probabilistic risk assessment programs, respectively.)lhe report also 20 recognized significant steps being taken in the area of outage planning.'

21 The cover letter that transmitted the inspection report highlighted a few of the key conclusions 22 reached by the NRC. It stated:

NRC Inspection Repon No. 92-04, ST AE-HL-93023, March 30,1992.

I NRC Inspection Repon No. 92-04, ST AE-HL 93023, March 30,1992, p. 2.

O k NRC Inspection Repon No. 92-04, ST-AE-HL-93023, March 30.1992, p. 3.

Page IV 2 The Lsbery Consuhmg Group

PU l Chapter Four- Engineering 1 "Overall as a result of this inspection. we view that your engineering programs are 2 providing quality products and are providing good plant support.. We view the 3 assessment capabilities of your organization as a major strength. We also observed 4 that you have (a) signincant number ofinitiatives in progress to improve your 5 engineering activities."'

6 The self-asst ..:nt program at STP was especially highlighted in the inspection report as well.

7 "Thp inspector 3 found that Sve assessments were performed in 1991. ..The engineering 8 assessment activities at STP are considered a strength."5 9 The NRC was obviously impressed by the numerous initiatives that were in process during the 10 1992 inspection. The report listed 57 individual programs that the NRC believed were deserving 11 of note. These initiatives included work on design basis documents, configuration management.

12 diesel generator reliability, toxic gas monitoring system replacement, reliability centered 13 maintenance, and probabilistic risk assessment. Two specific initiatives were singled out for 14 special recognition:6

• 15 " Strategic Plan for Plant Modernization and the IRE 04 Outage Planning. .[have]

O 16 had a very significant impact on future engineering planning and workload. These 17 two initiatives are considered to be a strength for promotion of the effective 18 utilization of station resources."

19 Similar remarks were made in another NRC inspection report that described a detailed evaluation 20 of the engineering support provided for the electrical distribution system.' The cover letter that 21 transmitted this report said:

22 "The team noted a number of strengths and relatively few weaknesses related to 23 your engineering and technical support staff's efforts.... Further, the team 24 considered the activities of your engineering and technical support staff to be 25 superior "

Letter, NRC to HL&P, A. Beach tr D. Hall, ST AE-HL-93023. March 30,1992.

NRC Inspection Report No. 92-On, ST-AE-HL-93023, March 30,1992, pp. 21-22.

n NRC Inspection Report No. 92-04, ST- AE-HL-93023, March 30,1992, pp. 22 23.

NRC Inspection Report No. 91-05. ST-AE HL-92804, July 18,1991.

Page IV-3 The f.derry Comultmg Group

i 0 2.C .'

Chapter Four - Engineering ,

/m i Given this litany of praise from previous NRC inspections. it is easy to understand the surprise

with which HL&P received the harsh criticisms contained in the DET report. Even considering that 3 the DET's review may have had different purposes and methods, there remains a significant 4 discontinuity between the DET's assessment and prior NRC reviews of engineering. Moreover.

5 given the NRC's prior positive reviews of engineering, it was clearly reasonable for STP's 6 management to believe the course that it was following in 1992 and the pace at which it planned 7 to implement improvements was in keeping with the NRC's expectations.

8 Liberty's analysis of the many issues discussed in the DET report demonstrated that HL&P had 9 conducted its affairs in a sound, business-like manner. It managed engineering resources very to closely, making sure they were sufficient to support plant operations and maintenance activities.

It HL&P kept engineering resources at a size that was cost-effective and required a prioritization of 12 tasks, since everything was not going to be completed at once. Safety, quality, and operational 13 readiness were held paramount in the management of resources.

14 Major programs gamered senior management support. These programs included: training, the 15 corrective action group, reduced temporary modifications, replacement of troublesome systems

[ 16 (such as toxic gas monitors and fire protection equipment), risk assessment, and configuration 17 management.

18 Engineering at STP was very conscious of the importance of having independent evaluations made 19 to help enhance its performance and intemal processes and to provide technical assistance in 20 properly addressing equipment performance and reliability issues. In the two-year period prior to 21 the time of the DET's investigation,15 individual studies were performed for engineering, either 22 alone or in conjunction with other departments. HL&P carefully evaluated the results presented 23 in each report and implemented beneficial actions on the basis of the insights and technical advice 24 that were offered. The regular studies of engineering and the fact that actions were taken as a result was strong evidence of prudent management.

25 l

\

Page IV-4 The Libert>' Consulting Group I

_ _m . __

i 4

RLS .'

Chapter Four - Engineering

(

\ i II. Engineering's Resolution of Plant Problems 2 A. Root Cause Analyses and Corrective Actions L

3 The DET was critical of engineering's root cause analysis and corrective action programs.'

4 "Neither the plant nor the design engineering staff had sufficient resources to 5

- appropriately support the site. This caused engineering to be slow in identifying 6 deficient conditions and hasty in performing investigations or root cause 7 evaluations, resulting in many engineering solutions or products that corrected the . .

8 symptom, but not the root cause. Approved corrective actions generally took a long 9 time to implement because of schedular or financial considerations....

10 "The engineering departments gave weak support in resolving plant problems. The 11 root cause analyses (RCAs) and resulting corrective actions were often ineffective 12 in preventing repetitive equipment problems."

,, 13 STP established organizations, policies, and procedures to perform detailed root cause analyses.

14 Having both an ISEG (Independent Safety Engineering Group) and a separate Corrective Action 15 Group went beyond typical industry practice and provided an opportunity for conducting an i 16 outstanding root cause analysis program. The results developed by these two organizations 17 received strong management support, as evidenced by a formal tracking and commitment process 18 that required timely response by the assigned organization. Management also demonstrated its 19 commitment to the program by n2pporting the trammg of 119 individuals in root cause analyses.'  !

20 STP took a series ofinitiatives to continually strengthen its corrective action program. In a report 21 to the NRC in May 1992, HL&P described how it was reorganizing the corrective action function 22 at STP.' The impetus for this change was a perceived need to improve human performance. As 23 part of the overall plan to enhance the corrective action program, HL&P invited INPO to provide 24 training on corrective action techniques. This training was provided in July 1992. In addition, 25 HL&P placed a greater emphasis on performing self-verification. A Corrective Action Group was 26 established. This group reported directly to the Plant Manager, Process changes were made so that 27 the c6rrective action program was less burdensome for its users and provided for more explicit 28 accountability. A number of other specific changes were made, including the provision for Event  ;

I DET Report, pp. 25-26. l DET response item #1107.

Lener, HL&P to NRC, ST HL AE 4083, Att. #2, May 1,1992, pp.19-20.

Page IV 5 The Liberty Consulting Group l

RL5-l Chapter Four - Engmeering

/

(v l Response Teams. improved consistency in the investigations, integration of other reporting 2 processes, and establishment of priorities.

3 STP provided a summary ofimprovement programs being instituted within nuclear engineering a to the DET." This summary set forth plans and accomplishments in four broad areas: an 5 operational improvement plan (which included material condition matters, procedural compliance, 6 and similar activities), programmatic enhancements and procedural streamlining, responses to a 7 consultant's report on engineering, and engineering involvement in support of operations and 8 maintenance. The description of responses to the consultant's report contained numerous instances 9 in which notable progress had been made, including the areas of enhanced upward communication.

10 accountability, establishment of clear goals, avoidance of micro-management, and work force 11 management.

12 The consultant's report that was prepared for HL&P in early 1992 by Performance Data, Inc. (PDI) 13 focused on areas where significant improvements could be made." It contained a summary of 14 strengths in the engineering functions, in part relying on evaluations performed by others. It noted 15 that the latest SALP report had indicated that engineering was improving, that a recent INPO 16 evaluation was positive concerning engineering activities, that some important management 17 changes had been made, and that the EDSFI (electrical distribution system functional inspection) 18 repon had said that " technical support is superior." This same inspection report also took specific 19 note of HL&P's effective corrective action program: "He team determined that prompt corrective 20 actions had been implemented for identified problems and that critical self assessments of various 21 aspects of the facility design had been performed.""

22 The DET report provided several examples to support its conclusions about engineering's support 23 in resolving plant problems. Several of these examples are discussed in the subsections below. One 24 of the examples, which concemed the Qualified Display Processing System, is addressed in the 25 chapter of this report on maintenance.

Nuclear Engineering Improvement Programs, DET response item m3063.

" Analysis of Engineering Activities, Performance Data, Inc., February 14,1992, DET response item =3231.

O 5 "

NRC Inspection Repon No. 91-05, ST-AE-HL-92804, July 18,1991.

Page tv-6 The Laberty Consulting Group

RL.5 .'

Chapter Four - Engineering

('s i B. Temporary Modifications 2 Temporary modifications (TMs> are temporary changes made to plant equipment that do not 3 conform with permanent design documents. This often include temporary rigging or supports, 4 electrical jumpers or pulled fuses, disabled annunciator alarms, and temporary instrumentation.

5 This are necessary at all nuclear plants from time to time and are controlled by procedures to 6 enstire that appropriate engineering and safety consideration is given to each."

7 Even though the purpose of This is to improve the operation of systems or equipment on a 8 temporary basis, it is desirable to keep the number of TMs to a minimum. That is, to the extent 9 practical, everything in the plant should be part of the permanent plant design. Sometimes.

10 howes er, TMs are required to be kept in place because of such factors as the lead time required for Ii design and procurement of parts, or the need for an outage or vendor information to install a 12 permanent modification. ,

13 Regarding TMs at STP, the DET concluded:"

O "TMs were not thoroughly evaluated and were not aggressively pursued to closure.

V) t 14 15 as illustrated in the following:

16 " Sixteen TMs were installed for more than 2 years, including some that caused 17 problems for operators. Some TMs were originally assigned a long restoration is period (1 to 2 years) or given an extension without adequate justification. Some 19 were later converted to permanent modifications and remained open until the 20 permanent modifications were closed."

21 Liberty found that HL&P monitored the number and age of open TMs. Graphical and tabular data 22 were presented to management every month in the Station Report. Initially HL&P tracked open 23 TMs by unit and by whether the TM was more or less than six months old.'6 In July 1992 the 24 Station Report mentioned an increased effort to close TMs that were more than two years old. Two 25 months later, meetings were held that resulted in action assignments to reduce the number of TMs 26 installed for more than two years. Later that year the Station Reports staned to include the number

" STP Procedure OPGP03 ZO 0003, Temporary Modifications. DET response item #3011.

pg " DET Report, p. 28.

V " Station Reports, January 1991 through February 1993 DET response item #0018.

Pagei\ ?

The Liberty Coruultmg Group

n s.. i j Chapter four - Engineering l

l of This in the two-year and-older category. The number of open two-year-old This was reduced l 2 by over 40 percent, from 27 in October 1992 to 16 in February 1993."

3 Even prior to the actions taken by HL&P to significantly reduce the number of This. however. the 4

l NRC made some very favorable comments regarding the effective management exercised by 5 HL&P in its Thi program. In an inspection report issued in the spring of 1992, the NRC stated that 6 "[t]he temporary modification program was found to be functioning properly. Noteworthy was the 7 management attention that open temporary modifications received." The inspectors were 8 obviously impressed by the thoroughness of HL&P's Thi program and the extent to which it was 9 frequently monitored. ne NRC also noted the strict criteria to which the program had to adhere.

10 Inspections were made by the NRC to verify that:"

ii "the temporary modifications were installed in accordance with the description in 12 the packages. There were no discrepancies noted and. . tags were appropriately 13 attached [and] the control room drawings,..were sufficiently annotated. . The use .

14 of temporary modifications receive a high degree of management visibility. A is temporary modification coordinator has. . responsibility (for the program.] There 16 are audits performed in addition to those by the quality assurance group. . [A]

p 17 monthly review [is] performed by the temporary modification coordinator [to note]

V 18 19 those that are installed for more than 3 months. The applicable system engineer is required by procedure to either:

20 -

Initiate the restoration [if no longer needed).

21 -

Initiate. . measures requesting a permanent change. ., or 22 -

Develop an approved restoration action plan.

23 The DET requested information about the open This and about the number of open This over time.

24 From this the DET concluded that This "were not aggressively pursued to closure." While the total 25 number of open This and the number more than six months old stayed reasonably constant over 26 time, HL&P pursued closure. The following table shows the total number of open This (for both 27 units) at the end of each month, the total number of open This more than six months old. the 1

Station Repons, July 1992 through Febnaary 1993, DET response item #0018.

4 NRC Inspection Report No. 92-04, ST AE HL-93023, March 30,1992, pp.10-12. -

NRC Inspection Report No. 92-04, ST AE HL-93023, March 30,1992, pp.10-11.

L Page IV-8 The Liberty Coraulting Group l

L___--- - - - - ._. . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __

i RLS :

Chapter Four - Engineering s i number of Dis closed during the month, and the percentage of TMs open at the beginning of the 2 month that were closed during the month?

Total Open Percent 3 M a th Oz.a >6 mm. Closed Closed 4 January 1991 57 42 4 ..

l 5 . February 1991 58 40 5 8.77' b 6 March 1991 59 42 4 6.9094 7 April 1991 55 41 8 13.56 %

8 May 1991 55 43 3 5.45 %

9 June 1991 54 41 2 3.64?b 10 July 1991 55 41 1 1.85 %

11 August 1991 66 44 0 0.00' b 12 September 1991 65 43 6 9.099 b ,

13 October 1991 66 43 6 9.2 3* b 14 November 1991 63 44 6 9 09 %

,C 15 December 1991 70 42 1 1.5 9.

16 January 1992 66 49 5 7.14%

17 February 1992 69 50 2 3.03 %

18 March 1992 67 49 2 2.90 %

19 April 1992 70 49 4 5.979.

20 May 1992 72 51 4 5.719  ;

21 June 1992 71 55 2 2.78' b 22 July 1992 70 51 5 7.04 %

23 August 1992 68 50 4 5.71%

24 September 1992 69 49 7 10.29 %

25 October 1992 74 52 1 1.44' b 26 Nosember 1992 66 55 11 14 87 %

27 December 1992 59 47 10 15.15 %

28 January 1993 57 30 6 10.17 %

29 February 1993 55 28 7 12.28 %

I

/

(

'b

Station Repons, January 1991 through February 1993. DET response item s0018.

Page IV-9 The Lsberty Consulting Group l i

RU-:

Chapter Four - Engineering A

1 These data showed that HL&P had closed This in every month except one during this 26-month

period. The percentage of open TNis that were closed in each month was significant. In five of the 3 six months before the DET's evaluation, HL&P's progress toward closing TN!s improved a substantially. In each of those months HL&P closed more than 10 percent of the open TN!s. At 5 the end of February 1993, one month before the DET arrived at STP, the total number of open 6 TMs was at its lowest level since June 1991, and the total number of open This older than six 7 months was at a level that had not been experienced in the past 26 months. Finally, and as noted 8 above, the number of open two-year-old TMs was reduced by over 40 percent from October 1992 9 to February 1993. ' By the end of March 1993, at which time the DET's evaluation was pending, 10 the total number of open TMs had been reduced to 54, and the total over six months old had been 11 reduced to 25.2 During the year 1991, STP's goal for open temporary modifications was less than 12 80 total and less than 40 older than six months. In 1992 the goals were made more aggressive. 65 t3 total and 35 older than six months. In 1993, STP added the goal of having no TMs older than two 14 years.

15 The DET requested information about the kind of operator compensatory action required because 16 of the existing TMs. The response indicated that there were no such actions required in Unit 2.

17 However, Liberty leamed that there may have been some compensatory actions related to the Unit 18 chillers. In Unit 1,13 of the open TMs required some type of operator action. Operator actions 19 consisted of checking a valve position monthly, emptying a tank, pumping sumps, and verifying 20 a temperature reading once per shift.23 The operator actions required because of TMs were not 21 particularly burdensome or out ofline with normal watchstanding responsibilities.

2: HL&P closely monitored the number and age of temporary modifications. Every month HL&P 23 reported not only the number of open TMs but also the number closed during the month. HL&P 24 aggressively pursued closure of open TMs. HL&P's actions to close TMs, particularly during the )

25 six months before the DET evaluation, were particularly noteworthy. HL&P established goals for 26 the number and age of open TMs and then made those goals more ambitious. l l

2' Station Reports, January 1991 through February 1993, DET response item *0018.

1 s

- DET response item $3215.

" DET response item #1091.

~~

Page IV-10 The Liberty Coruulting Group

RU.. {

Chapter Four - Engineering {

f n I

'd i Liberty concluded that HL&P's management of TMs was reasonable. Particularly noteworthy was  ;

, 2 the management attention applied to, and the progress attained in. the reduction of TMs. neither l 3 of which was mentioned in the DET report. '

4 C. Injector Pump Hold-Down Studs l

5 One of the examples used by the DET to show that root cause analyses and corrective actions were

6 not effective in preventing recurring problems had to do with certain hold-down bolts. The DET 7 report said?

o 8 "The licensee did not determine the root cause of repetitive failures of the fuel 9 injector pump hold-down studs associated with the standby diesel generators 10 (SDGs). Nine separate failures occurred between 1987 and 1993, including five ii failures on SDG 22. The failure of these studs was a significant contributor to the

. 12 high unavailability of SDG 22."

• 13 While five of the nine failures observed were on SDG 22, each failure was in a different location c)

(Q 14 15 and each was associated with a different one of the twenty diesel cylinders.25 Because these studs had been changed on several occasions, it could not be logically concluded that there was a 16 common cause of the five failures, as the statement seems to imply.

17 A detailed root cause analysis was conducted for each of the nine failures.26 One of the failures was i

is clearly associated with a connecting rod failure that placed an unusually high loading on the fuel f 19 injection pump. Five failures were apparently due to improper torquing or installation techniques, 20 and the other three failures were induced by fatigue. An outside firm concluded that these failures 21 were due to inadequate or loss of preload on the studs. HL&P believed the situation to be 22 important enough that it had four different organizations perform root cause analyses at various 23 times. These analyses were conducted by three firms recognized in forensic analysis (SWRL 24 Bechtel, and MPR) and HL&P's own metallurgical laboratory.

" DET Report, p. 27.

" DET response item #3080.

" DET response item s3080.

Page IV Il The Liberry Consulung Group

RLS.'

Chapter four - Engineering 1 It is notable that the first six failures all " occurred during the first few hours of service following 2 installation" of the studs."In each case the corrective action was successful in preventing repeat 3 failures for substantial periods of time. There were three failures during a two-month period from a Nosember 1990 to January 1991 and two failures in March 1993. Otherwise the failures occurred 5 10 to 17 months apart. This substantial period between failures also masked any generic issue. if 6 indeed there was one.

1 7 Although any repeat failure ofimportant equipment is of concem, the effects of these failures need  ;

8 to be fully understood. In the first place,"[t]here has never been a forced outage caused solely by 9 an SDG failure."3 One of the failures extended an outage for a few days. Second, the failure of any to one or two fuel injection pumps, even on a single SDG, does not affect the safety function of the 1i diesel generator. This conclusion was confirmed by the original equipment manufacturer, Cooper-12 Bessemer." These diesel generators were designed for common industrial use, and the studs for 13 the fuel injector pumps were designed to fail when subjected to abnormally high loads. For this la particular application at STP, it would have been preferable to overdesign the studs for high loads 15 and not for failure.

O 16 HL&P ensured that the NRC was kept informed of these stud failures, even though this was not

(

17 strictly required. A letter was sent to the NRC on December 20,1990 regarding two stud failures 18 that had taken place in November 1990, and a supplemental letter was sent in April 1991 to 19 describe the January 1991 stud failure.30 These three failures were associated with a special 20 installation tool that proved to be inadequate. These incidents were also discussed with the NRC 21 by telephone on Februan 1,1991.

22 Upon the determination of the root cause of each failure, an action plan was prepared and 23 implemented. After the first two failures in 1987 and 1988, all bolts and nuts were carefully 24 retorqued on all 20 cylinders of all 6 SDGs. Following the third failure in 1990, the corrective 25 actions included a mandate to replace the studs any time a fuel injection pump was removed or 26 loosened. As a result of the series of three failures in late 1990 and early 1991, the installation

" Memorandum. M. Pacy to C.T. Bowman, SPR Operability Review, March 29,1993, included in DET response item #3080.

" DET response item #2193.

" DET response item #3080.

1

• DET response item #3080-001.

9 Page IV 12 The Liberty Consultmg Group

FL.N Chapter four - Engineering i procedures were rewritten, all studs that had been installed using the special tool mentioned earlier 2 were replaced.)' and other actions were taken."

3 HL&P participated in an industry group comprised of plants with similar diesel generators and was a

instrumental in getting the manufacturer to take several steps to investigate and resolve the stud 5

failure issue. This commitment was made as part of a Cooper-Bessemer diesel engine owners' 6

group discussion on August 27,1991.33 After the March 1993 events, HL&P once again replaced 7

all the studs.3d This was one element of a 13-step action plan that was undertaken in the spring of 8

1993. This study concluded that the root cause of the continuing failures was insufficient torque 9 to provide the proper level of prestress in the studs. The corrective action taken by HL&P was to 10 revise the installation procedures. Under the revised procedure, a lubricant is applied to the threads i1 of the nuts, and a torque is applied that is sufficient to overcome a locking device used under the 12 nut. This technique was jointly developed by STP engineering, Cooper-Bessemer, and MPR 13 Associates. The newly installed studs will be monitored closely for some period of time to confirm 14 that these actions are satisfactory. Although previous root cause analyses were partially valid ir$

15 concluding there was improper torquing of the bolts, these evaluations consistently overlooked the 16 potential for a lack of prestress and for overtorquing."

17 in summary, HL&P had a root cause analysis performed of each failure and prepared and 18 implemented an action plan in each situation. The substantial times between failures were some 19 evidence that these corrective actions had been successful until additional facts entered into the 20 situation, in each case having to do with various aspects of the installation process (inappropriate 21 special tool, too much retorquing, and inadequate preloading). The incidents did not cause any 22 forced outages and were not a significant threat to safety. The NRC was kept fully informed.

23 HL&P involved three contractors in the root cause analysis and obtained the assistance of the l 24 manufacturer by forming an owners group. The latest root cause analysis that identified the need 25 to obtain a proper level of prestress in the studs is expected to elimmate future failures of this type 26 on the basis of proper implementation of the corrective actions. This circumstance is a good DET response item #3080. l DET response item #3080-001, pp. 5 8.

DET response item #3080-001.

DET response item #3080-003.

'y/ "

DET response item 83242.

Page IV-13 The Liberty Consultmg Group

1 Ru-:

Ch.tprer Four - Engineering l

C i example of the difference between the NRC's standard and a prudence standard. In each facet of 2 the situation, HL&P acted in a reasonable fashion. However, because the true root cause of the 3

failures was not determined, the actions, while reasonable, were not completely effective. It is a

wonh noting that the most recent actions taken at STP were also reasonable. Whether they are. in 5 fact, effective will not be known for some time.

6 D. Toxic Gas Monitors 7

STP has had a history of the toxic gas analyzers initiating spurious Engineered Safety Features 8 /ESF) actuations." A series of corrective actions had not been effective in eliminating these false 9 indications by late 1992. The DET report also noted this continuing difficulty: "The RCAs (root to cause analyses] and accompanying corrective actions were ineffective in preventing repeated Ii failures of the toxic gas monitors and containment ventilation isolation system.""

12 HL&P recognized that minor modifications were unlikely to resolve continuing difficulties in the 13 toxic gas monitoring system and decided to take a different approach. In early 1992 HL&P 14 decided to replace the monitors with an improved design and add a third monitor to improve the

[\ 15 trip logic." The toxic gas monitors were replaced on Unit 1 by mass spectrometers. The Unit 2 16 replacement was planned for the next refueling outage. In addition, the trip logic was enhanced.

17 Instead of using a two-monitor system where 1 out of 2 indications caused a trip, three mass 18 spectrometers were installed where 2 indications out of 3 would be required to produce a trip 19 signal. To facilitate testing of the system, a three position switch was installed to clearly indicate 20 a tripped condition, and new calibration gases were to be used. Electrical power supplies were also 21 upgraded. The uninterruptible power supplies were to be refurbished, filtered power outlets (to 22 provide surge and noise protection) were to be used, and isolation transformers (to regulate 23 voltage) were to be installed. HL&P submitted a proposed Technical Specification change to the 24 NRC to accommodate the changed design.

I l

" DET response item #3052.

" l DET Repon, p. 27. '

O t, "

Lener, HL&P to NRC, ST-HL-AE-4083, Att. 82, May 1,1992, p. I1.

Page IV 14 The Liberty Consulting Group i

RL S..'

Chapter Four - Engineering O  !

V i Many of these corrective actions were identified and implemented as a result of an extensive report 2 developed by HL&P's ISEG, the independent safety engineering group." Many of these )

3 modifications were new or just being installed when the DET made its visit. Also, both units were 4 shut down. These factors made it impossible to demonstrate the effectiveness of the corrective 5 actions, but the basic principles of the redesign (more sophisticated equipment and enhanced logic, 6 for example) were reasonable and should significantly reduce the number of false actuations, if not 7 eliminate them, 8 The radiation monitoring system (used to actuate the primary containment ventilation isolation 9 system) also experienced relatively frequent spurious actuations, averaging three to four incidents 10 per year." HL&P's assessment of the radiation monitoring system conducted in late 1992 11 concluded that the system was functioning effectively but that its material condition was 12 declining." The ISEG report (noted above) also addressed these problems and developed a number 13 of recommended corrective actions, which were addressed in early 1993 by engineering.

14 It was determined that the Dose Assessment Computer System was obsolete in both hardware and 15 software,42 and it was replaced. A number of other corrective actions were taken, including:

\g 16 replacement of the algorithm used in the ventilation radiation monitor (to make it more accurate 17 at low count rates), redefinition of the function of the condenser vacuum pump radiation monitor 18 (from two functions to one), replacement of the sampling chamber in the liquid waste discharge 19 radiation monitor (to enhance its response time), installation of new radiation monitors on the 20 condensate return lines (from liquid waste and boron evaporators), installation of a new N-16 21 radiation monitor in the main steam line (for early detection ofleaks in the steam generators),

22 replacement of a switch in the condensate polishing liquid radiation monitor (to eliminate 23 sticking), possible replacement of the main steam and steam generator blowdown radiation 24 monitors, replacement of the process flow probes on ventilation systems (to avoid frequent 25 adjustments), and maintaimng calibration on spare detectors. In addition, power supplies for these 26 monitors were to receive enhancements similar to those previously made in the power supplies of

" ISEG Report 92 22,"Revicw of Station ESF Actuations for Trends and Common Causes," December 1992 DET response item #3105.

• DET response item #3052.

" Health Report: Radiation Monitoring System, October 30,1992, DET response item #3052.

b '

DET response item #3052.

Page IV-15 The Liberty Crwhmg Group ,

4 l

i

, 1 p:5.'.

Chapter four - Engmeering G

i the toxic gas monitors. Also, surveillance procedures were upgraded and electrical shielding was 2 added.

I 1

3 Liberty found that HL&P had taken comprehensive and reasonable actions in 1992 and early 1993 4 to fully investigate the root causes of spurious actuations in both the toxic gas and radiation 5 monitoring systems. Internal assessments were performed and numerous recommendations were 6 made for corrective actions that were specific and directly related to the causes of the false 7 actuations. Many corrective actions had already been completed by April 1993 but their s effectiveness could not be fully demonstrated during the outage that was coincident with the 9 DET's evaluation. Liberty also noted that the NRC had changed its reporting requirements so as 10 not to require LERs on HVAC (heating, ventilation, and air conditioning) actuations. This action i1 demonstrated not only that events like control room HVAC isolation lack safety significance, but 12 also that STP issued several LERs that would not have been required under the new reporting 13 requirements.

14 E. Solenoid-Operated Valves p

( 15 Nuclear power plants use a large number of solenoid-operated valves (SOD. At STP there are 130 16 SOVs supplied by Target Rock, and 122 of them are in safety related sersice, primarily to provide 17 a primary containment isolation function. Target Rock SOVs are common in the nuclear industry 18 and nearly all plants have had significant problems with these valves. The NRC has issued generic 19 letters on the matter, and EPRI has undertaken an initiative to help resolve difficulties experienced 20 with these valves." It would be difficult to investigate the maintenance history of solenoid-21 operated valves at any operating nuclear power plant and not identify recent failures. STP's 22 experience was similar to the rest of the industry.

23 The generic nature of SOV difficulties was widely recognized and was illustrated by the NRC's 24 Generic Letter 91-015 and the detailed backup information with that letter, NUREG 1275. A 25 problem caused by high temperature at STP "had been previously reported as a generic problem 26 by Palo Verde "45 HL&P also noted that " incorrect SOV indication has been an industry wide

" DET response items nos. 3153-001 and 3196.

DET response item #3153.

( Operational Readiness Closure Package, Engineering Backlog (CAL-5). November 13,1993. Tab 15 l (SOVs).

PageIWl6 The Liberty Cortsultmg Group

l 4

RLS .'

Chapter Four- Engineering D

k

~

d I concem." EPRI convened a workshop on SOV problems on May 13 15.1992 in Philadelphia. Two 2 representatives from STP's engineering department attended the workshop. The workshop 3 confirmed widespread failures due in large measure to improper application of the valve design 4 and numerous extemal factors (such as voltage, temperature, and moisture) that were not fully 5 recognized by the plant designers (architect-engineers) and could not be identified by the owner-6 operators until multiple failures had occurred.

7 EPRI developed a maintenance guide in mid-1992 to help its utility members deal more 3 successfully with the numerous SOV problems. This guide was distributed within the STP 9 organization?

10 The DET report noted ongoing difficulties with SOVs at STP and stated;"

1I " Widespread, longstanding problems with the application and performance of

. i2 Target Rock solenoid-operated valves (SOV) were not resolved. These valves were ,

., 13 used extensively in several safety-related systems. Multiple LERs involving wear, 14 aging, debris, contammation, and valve misapplication had occurred since 1990."

15 The repon also mentioned a particular issue related to a valve associated with a steam generator 16 sampling line.

17 HL&P spent considerable effort to resolve its problems with SOVs. A large number of station 18 problem reports were written starting in 1991, a long series of meetings was held. numerous 19 corrective action plans were prepared and implemented, and correspondence was exchanged with 20 both the manufacturer and EPRI. Procedures were revised and equipment modifications made. In 21 addition, HL&P established a task force in March 1992 to address the generic SOV issues included 22 in NRC Generic Letter 91-015." Both this Generic Letter and the accompanying NUREG-1275 23 addressed numerous industry problems that had to be evaluated by all nuclear utilities. The STP 24 task force developed a plan of action that was initially distributed on June 26,1992. The plan 25 called for a detailed three-phase program that was projected to take 3,720 person-hours to 26 complete. This action plan was revised several times through the remainder of 1992 and by

" Memorandum. D.W. Clark to R.R. Hemandez, May 22,1992.

Memorandum, R.R. Hernandez to Distribution, August 21,.1992.

DET Report. p. 27.

O- " Letter, HL&P to NRC, ST-HL-AE-4083, Att. #2, May 1,1992, pp.15-16.

Page IV 17 l The Lsberty Corauhmg Group j i

RLS.:

Chapter Four - Engineenng i November 30 had been expanded into a five-phase program that was scheduled to last into 1994 2 Action was taken to replace the SOVs in the steam generator blowdowTilines an application for 3 which this type of valve was not well suited.

4 The NRC recognized the value and breadth of these ongoing efforts by HL&P to forcefully address 5 the many issues associated with SOV operation. An NRC resident inspector's report in mid-1992 6 noted that planned actions on SOVs were extensive and included: a verification of(appropriate) 7 SOV application, compliance with equipment qualification requirements, a review of maintenance 8 and surveillance procedures, formation of dedicated teams to perform field work, and provision 9 of additional training for maintenance and engineering personnel.so 10 HL&P took reasonable and extensive actions to cesolve difficulties with the operation of its ii solenoid-operated valves. It kept abreast ofindustry experience and actions regarding SOVs. It 12 corresponded with the manufacturers and with EPRI to gain as much insight as possible into 13 corrective actions. A task force was established to address the problems, and a comprehensive plan

• 14 developed to resolve the difficulties, a plan that the NRC praised in an inspection report. STP's 15 experience with SOVs had been no different from that at other nuclear power plants and its 16 handling of the issue was reasonable and pmdent.

17 F. Startup Feedwater Pump -

18 Another example in the DET report regarding corrective actions dealt with the startup feedwater 19 pump. The report stated:si 20 "After a reactor trip, the startup feedwater pump (SUFP) failed to start upon 21 demand because oflow oil pressure. Repeated occurrences of moisture intrusion 22 had caused the oil filters to become clogged, reducing the lube oil pressure. A 23 previous SUFP trip on low lube oil pressure had not been properly evaluated, 24 resulting in the failure to recognize design deficiencies."

25 In November 1992, the Unit 2 startup feedwater pump experienced a trip. It was determined that 26 the lube oil filter, which is not designed to be moisture absorbing, had become saturated due to the 27 accumulation of water in the lube oil system. When the filter became saturated, the increased NRC Inspection Repon No. 92-21, ST AE-HL-93145, August 3,1992.

DET Repon, p. 27.

Page 1%I8 The L5erry Consuhmg Group

. - . . .- .- . .=, . _ - ._ . _ __ .. _. _ , -

RLS. .'

Chapter Four - Engineering i pressure differential across the filter caused the pressure of the bearing oil supply to drop. which 2 then tripped the pump motor." ,

3 This type of event had occurred previously and HL&P had established a task force in April 1992 4

to investigate and resolve the problem. As a result of this evaluation, actions were taken to provide 5 for better sealing of the system to help prevent water ingestion during heavy rains. These 6 imptovements were not successful, as demonstrated by the event in November 1992. The task 7 force was reactivated. Efforts to develop a more effective solution were initiated on December 16,

• 8 1992."

9 The same feedwater pump failed to start on December 27,1992, but this event was caused by an 10 incorrect setpoint on the pump start permissive pressure switch. This matter was readily corrected.

11 The pump tripped once again on February 3,1993 because of water intrusion caused by

^

12 " increasing seal failure."#

13 The startup feedwater pump is not a nuclear safety related component and therefore does not 14 receive the same high priority attention that a safety-related system is accorded, unless its is unavailability creates a plant availability problem. HL&P instituted a practice of frequently 16 replacing the lube oil filters associated with the startup feedwater pump motor to help correct the i 17 moisture problem. This approach was successful in that thereafter the SUFP never caused an 18 outage or a reactor trip, except in one case where the reactor was about to trip anyway because of 19 another failure. As the pump motor continued to experience problems, HL&P simultaneously made 20 a temporary fix and initiated action to establish a permanent solution."

21 As part of the renewed effort to resolve the moisture issue, water absorbing filters were installed 22 in the lube oil system to determine the rate of water pickup. An analysis was also conducted on the 23- operating conditions of the system, including its temperature cycles. It was found that under certain 24 high humidity conditions substantial condensation was taking place because the oil was at a ,

25 temperature below the dew point. It was also concluded that the pedestal seal associated with the ';

4 4

" DET response items nos. 2224 and 3148. I DET response items nos.1198 and 3266.

1

g. "

DET response item #3148.

{

" DET response item #3148.

Page IV-19 The Liberry Consultmg Group

Rl. S. .'

Chapter four - Engineering n

i feed pump was not well suited for this particular application. This seal was subject to leakage 2 because it was designed for dynamic use (where the pump is always in operation), instead of static 3 service, which was the usual situation in this application.

4 As a result of these findings, several changes were made. First, a three-way valve was installed to 5 direct the now oflube oil so that the feed pump seals were bypassed under certain temperature 6 conditions to prevent cooling the oil below 110 F. This modification was installed in late 1991 and 7 early 1992. Second, a water-absorbing filter was made a permanent part of the system to guard 8 against the unlikely event of further water accumulation.

9 Third, the pedestal seal was replaced by a labyrinth seal, which was more appropriate for this type 10 of service, where the feed pump is operated only occasionally. In parallel with this replacement, i1 a small pump was installed to circulate water through the new seal to maintain it within the proper 12 temperature range. Fourth, a small vent was placed in the pump casing to relieve any moisture 13 formation.

14 During the first month of operation after all these changes were made, all operating parameters 15 remained within specifications.

l [nV) 16 The documentation provided by HL&P showed a diligent attempt to make a sound engineering 17 evaluation of the moisture intrusion problem. It is not always possible to identify the most 18 effective corrective measures, and only by using hindsight can one determine if the measures to chosen were " effective." The actions taken by HL&P were reasonable and Liberty believes that 20 it is incorrect to state that the situation "had not been properly evaluated" simply because the initial 21 fix did not prove to be the ultimate fix.

22 G. Steam Generator Feed Pump Turbine 23 The DET related a concem about a series of trips experienced by the feed pump turbine and the 24 adequacy of the corrective actions taken. Specifically, the repon stated: 56 l

l i

O k DET Report, pp. 27-28.

Page IV-20 l The Liberty Consultmg Group

I l

1 R3 .

Chapter Four - Engineering i "During oil pump transfers, the steam generator feed pump turbine tripped 2 repeatedly because the oil pressure decreased rapidly. Engineering mistakenly 3 accepted the recommendation of a vendor to drill holes in the pump casing to 4 prevent air binding, which, w hen implemented, exacerbated the problem."

5 The feed pump ofinterest here is the pump that supplies feedwater to the steam generators during 6 normal operation. The feedwater is heated in the steam generators and converted to steam. The 7 powtr used to run this feed pump is provided by this steam, which drives a turbine connected to 8 the feed pump. This turbine has large seals that need to be lubricated and cooled by oil supplied 9 by a main oil pump. If this oil is not circulated continuously, the bearings might fail, creating a 10 situation in which that part of the unit would have to be shut down.

I1 Following a trip of a steam generator feed pump turbine (SGFPT) in Unit 2 on September 2,1990, 12 HL&P sought the advice of the vendor (Westinghouse) in trying to identify the cause of the event.

13 It was noted at the time that two previous trips had been experienced due to low oil pressure, and 14 HL&P believed it was important to resolve this issue. Westinghouse suggested two altemative is solutions, and HL&P was able to discount one of the altematives based on a particular system 16 response that had been consistently observed. The accepted alternative was based on the 17 conclusion reached by both Westinghouse and HL&P that air was being trapped in the column V

18 leading from the idle main oil pump. This accumulated air did not permit oil pressure to build up 19 quickly enough to prevent a trip on low oil pressure."

20 The trip event of concem can occur only when the oil supply to the turbine bearing is being 21 switched from one main oil pump to another. When one ptunp is turned off, the oil pressure begins 22 to decrease. The system was designed so that a second main oil pump would be immediately 23 energized and would start to pump oil in time and at a pressure that would prevent the low cil 24 pressure trip point from being reached. As the event in September 1990 demonstrated, this design 25 objective was not always fulfilled.

26 The accepted corrective action was to drill a small (%") vent hole near the top of the main oil 27 pump column to relieve any accumulated air pressure. This would permit the idle pump to reach 28 the necessary oil pressure in time to avoid a low pressure trip. HL&P observed that a nearly 29 identical fix had been successfully made on the discharge pump of the main turbine tube oil 30 system. He combination of a detailed analysis by HL&P of the system response during these trip

" DET response item #2210.

Page IV-21 ,

The bbeny Constdring Group 1

RLS.:

Chapter four - Engineering i events, the recommendation made by Westinghouse, and the satisfactory experience on a similar

sy stem led to a technically sound and reasonable decision to install the vent holes.

3 The reasonableness of this decision was further affirmed by another trip event on April 5,1992.

4 Although the vents had been installed on four of the six SGFPTs prior to April 1992, none had 5

been added to the system that experienced this trip on April 5. Nor had any other trips occurred 6 since the September 1990 event, except one, which had occurred just a few days earlier. At this 7 point the decision to add vents appeared to be confirmed as the proper one.

8 A trip event also took place on Unit 2 on March 30,1992. It was determined that the cause of this 9 event was two electrical leads that had bumed and severed. After repairing these leads, several start to tests were conducted on the main oil pumps, each resulting in a trip. No conclusions were reached i1 at this point, but it was postulated that the vent hole might have contributed to a faster decay of the 12 oil pressure without permitting the starting pump to reach an adequate oil pressure in time to avoid 13 a trip. At this point, Westinghouse was again requested to perform a more detailed investigatiori la of the problem."

15 Further investigation of these trip events in 1992 by HL&P revealed that this system was not Y 16 properly designed for this type of backup service. To resolve this issue, it was concluded that the 17 addition of a small (one quart) oil accumulator and a check valve to the automatic trip headers 18 would be adequate to prevent the oil pressure from dropping so quickly and thereby permit the 19 backup pump to reach sufficient oil pressure to prevent a trip." This modification was installed 20 on Unit 1 in 1992 and on Unit 2 in 1993. It appears to have been successful. It was also determined 21 that the vent installed previously was in fact necessary to relieve the air binding phenomenon.

22 The NRC recogmzed the effective efTorts being made by HL&P to resolve this situation in one of 23 its inspection reports in early 1992.60 The NRC resident inspector observed that:

21 "several initiatives were indicative of effective licensee management involvement 25 in operational activities. Licensee management initiated additional actions to DET response item #2213.

Memorandum, R.R. Hernandez to C.A. Ayala, September 4,1992.

j

-( "'

NRC Inspection Report No. 92-08, ST- AE-HL-93077. May 22.1992, p. 2.

Page IW22 The Liberty Consuhmg Group

1 i

P:S.:

Chapter four - Engineering s

t address recurring Steam Generator Feedwater Pump problems. The licensee

. 2 initiated a task force in response to main feedwater equipment problems."

3 in a subsequent inspection report the NRC noted the specific actio:.s taken by HL&P:

4 "The licensee has developed long-term corrective actions to resolve the problems 5 associated with MOP (main oil pump] trips. .. [A] decision was made. .to install 6 ,

a small hydraulic accumulator. .to prevent rapid depressurization of the header."

. 7 The NRC said that HL&P planned to make this modification in the fall of 1992. HL&P did so.

I 8 The NRC also recognized that the vent installed previously by HL&P was necessary but not 9 sufficient. An NRC inspection report said: 52 to "A review of industry experience revealed that the most probable cause for the 11 SGFP trips. .was air binding in the pump casing. [A] vent path has been 12 installed. .(and] the vent path appears to prevent air binding but now allows for a .

( 13 more rapid pressure decay."

14 HL&P performed root cause analyses for each trip event. The corrective actions taken to install 15 an air vent and an accumulator were both necessary (not " mistakenly accepted") and were already 16 in place on Unit I prior to the DET visit. HL&P took timely and appropriate action to perform root 17 cause analyses, contact the vendor, and take sequential corrective actions to ensure they were both 18 necessary (individually) and sufficient (taken together).

I

(

(

" NRC Inspection Repon No. 92 24, ST-AE-HL 93184, September 10,1992, pp.12-13.

" NRC Inspection Repon No. 92 24, ST AE-HL-93184, September 10,1992,p.13.

Page IV 23 l The Liberty Consultmg Group l

1 M.5 '

Chapter Four - Engineering i O l O i H. TSC Diesel Generator 2 Another of the DET's "[e]xamples ofineffective engineering support, investigations, root causes 3 analyses and corrective actions is the following:""

l 4 "The Technical Support Center diesel generator was not reliable, as evidenced by l 5 repeated failures to start and load during testing. Contributing to the poor reliability 6 was exposure to the environment, design weaknesses, and poor circuit breaker 7 reliability. The licensee only partially implemented proposed resolutions to these 8 problems.""

9 The diesel generator that supports the Technical Support Center (TSC) experienced a number of to component failures that significantly affected its reliability, In a March 1993 memo, HL&P 11 described three short-term and three long-term actions designed to fully diagnose and resolve the 12 cause of these failures." The short-tenn actions supported the need to gain better data on diesel 13 generator performance and included an increased testing frequency (from quarterly to monthly),

14 a consolidation of service requests and preventive maintenance activities, and maintenance of a is formal log on the engine. The long term actions were designed to resolve known problems and

,O 16 included changing the output breakers to a more reliable design, enclosing the diesel generator, k) 17 and installing load banks to enhance full-load testing.

18 Earlier, in 1992, in recognition that the plant emergency diesel generators also had a history of 19 component failures, HL&P took a series of three initiatives to maintain an adequate level of 20 reliability." First, a study was performed by a contractor and a report developed that described the 21 diesel generators and their history, recommended goals and objectives, and set up a three-phase 22 program for system improvement." Second, as a result of this report, HL&P established a task 23 force to evaluate and implement the recommendations made. Third, HL&P became actiw in an 24 owners group that was initiated to resolve generic equipment problems on this type of diesel 25 generator. ,

DET Report, p. 26.

DET Report, p. 28.

Memorandum, Leazer to Jordan, March 23,1993, BOP and TSC Diesel Generator Reliability and Availability, DET response item #2331.

" DET response item #4053.

Diesel Generator Long Range Improvement Program, August 13,1992, by Impell.

Page IV-24 The Liberty Coruulting Group

RL5-:

Chapter Four - Engineermg i The recommended goals that HL&P addressed included:

o Leak reduction 3 o Optimization of PM frequency and the number of engine starts 4 o Increasing starting times 5 o Reducing environmental effects 6

o Relay protection 7 o Decreasing surveillance loads o

8 Identification of additional actions that could improve reliability 9 o Reduction of parts obsolescence 10 o Optimization of the timing of surseillance testing ii The contractor report referred to earlier also recommended that these goals be achieved in three 12 phases having end dates in 1993,1994, and 1995. The first phase focused on surveillance 13 activities. The report also provided estimated costs, considerable detail on the bases for the la recommendations, and a sound framework for reaching decisions on future actions.

15 HL&P initiated three actions in 1992 to address the overall matter of diesel generator reliability.

N 16 A detailed evaluation report was prepared by an independent contractor that included specific 17 recommendations, a decision model, and a logical basis for the recommendations submitted.

18 HL&P reacted in a positive way to the report by setting up a task force to initiate activities to 19 improve DG reliability. In early 1993 these activities were extended to the TSC diesel generator.

20 Liberty found that HL&P's numerous and extensive actions, both taken and planned, to improve 21 the reliability of the TSC diesel represented reasonable actions in response to the information 22 available at that time.

I l

l l

3 Page IV-25 The Libero Comultmg Group l

$.b Chapter Four - Engmeermg f) i I. Torque Setting on Motor-Operated Valves 2 Many of the valves used in a power plant are operated by complex motors. In order to ensure that 3

these motors can operate the valves properly and in the prescribed time period, a preset torque is 4 applied to the valve actuator, which is driven by the motor. In the situation discussed below. the 5

vah es serve to isolate pumps in the residual heat removal (RHR) system. a safety-related system 6 used to remove decay heat from the reactor during shutdown conditions.

7 HL&P completed a station problem report on February 2,1993 to describe a situation in which the s torque on the actuators of five ou+. of six RHR pump suction motor-operated isolation valves had 9 exceeded the maximum torque rating." This condition was discovered by HL&P after the valves to had been declared operable. The valve actuators had been tested during the Unit I outage in late ii October and early November of 1992 and a successful post-maintenance test (PMT) performed.

12 The overtorque condition was found in early February 1993 during an engineering evaluation of 13 test results. A station problem report (SPR) (93-0365) was immediately prepared to further-14 investigate the matter. On the basis of the satisfactory PMT in November 1992, design engineering 15 believed the MOVs would " perform their design function," but only a detailed inspection of the A 16 valves could validate this conclusion." An investigation revealed no damage to the actuators. They k) 17 were reassembled and declared operable. HL&P also performed an evaluation to verify that the 18 design life of 2,000 cycles had not been exceeded.

19 The DET used this matter to support its conclusion that STP's engineering had not been effective 20 in resolving plant problems."

21 "The engineering staff did not always adequately evaluate equipment operability 22 as illustrated below: ..

23 " Torque measurements and computations associated with testing of motor operated 24 valves (MOVs) were not evaluated to verify valve operability. The licensee 25 discovered, upon evaluating previous test data, that several residual heat removal

6 valves had been torqued above design values because of a deficiency in the test 27 procedure and associated engineering documents to measure or compute torque."

DET response item #3249.

DET response item #3249.

(O t

DET Repon, p. 28.

I Page IV-26 The bberry Consultmg Group

l Ru-:

CHpter Four - Engineering i Because of the ovenorqued condition. the NRC prepared a Notice of Violation ofits regulations.

2 After careful analysis. HL&P concluded that the situation did not constitute a violation and 3 submitted its justification to the NRC in May 1993. Specifically, HL&P found that the suction a isolation MOVs in the RHR system had been properly determined to be operable and that the basis 5 for reaching this conclusion was adequate.' HL&P's conclusions were reached on the basis of 6 investigations as well as tests and evaluations performed by an independent consultant in March 7 1993.72 The NRC did not withdraw the violation, finding "that prompt action to correct or 8 adequatelyjustify an ovenorque condition. .did not occur."' However, the NRC did acknowledge 9 that accepuble corrective actions were subsequently taken. The NRC's technical evaluation found 10 that u hile STP should have declared the valves inoperable, ajustification for interim operation ii until the next outage could have been an acceptable action.

12 A number of NRC inspection reports had recently praised HL&P's MOV program and its 13 documentation. In an inspection of HL&P's activities to respond to NRC Generic Letter 89-10, 14 the NRC stated that dynamic testing of the valves was a panicular program strength." A 15 subsequent NRC report indicated that HL&P's " corrective actions were comprehensive and 16 timely" with respect to MOV problems." Late in 1992, the NRC noted that the MOV program at 17 STP had improved and had received strong management support.'6 18 There was clearly a problem in this instance because the valve actuators had been left in an 19 overtorqued condition. However, HL&P's evaluation process was sound. It was during this process  ;

20 that HL&P had identified the problem. Liberty concluded that HL&P's actions had been 21 reasonable. This conclusion was formed on the basis of(1) other evaluations of STP's MOV <

22 program, (2) industry-wide difficulties in assuring MOV operability, (3) HL&P's discovery of the 23 specific overtorqued conditions, and (4) HL&P's analysis and correction of both the problem itself 24 and the instructions used to set valve torque.

' Letter, HL&P to NRC, ST-HL-A -442L May I8,1993.

': Report,"SB-2-60 Limitorque Actuator Cycle Testing Results," March 20,1993.

' Letter, NRC to HL&P, ST-AE-HL-93466, July 8,1993.

• NRC Inspection Report No. 92-06, ST AE-HL-93038, April 10,1992.

l l

' NRC Inspection Report No. 92 21, ST-AE-HL-93145, August 3,1992, p. 6.

l NRC Inspection Report No. 92-30, ST-AE-HL-93270 December 31,1992.

l Page IV-27 The Liberty Consulting Group l

l

P L S. .'

Chapter Four - Engineering V i J. Flow Noise in the Auxiliary Feedwater System 2 STP experienced some unusual flow behavior in the auxiliary feedwater system during initial 3 startup activities in 1988. This situation was noted in the DET report, which stated:"

4 "The licensee started up with a significant design deficiency that resulted in 5 excessive water hammer in the auxiliary feedwater system (AFW). Engineering's 6 resolution to the water hammer issue was to install mechanical stops on the AFW 7 valves to prevent them from closing, which created additional operational concems.

8 Operators could no longer effectively throttle valves during certain plant conditions 9 to control flow to the steam generators. As a result, operators controlled flow by to cycling the stop check valves, resulting in an excessive number of thermal cycles it on steam generator nozzles."

12 The auxiliary feedwater system is used to supply feedwater to the steam generators during times 13 when feedwater is required and the normal feedwater systems cannot do so.

14 The flow of feedwater is regulated by control valves. If a control valve is suddenly closed, a is substantial low frequency pulse in the water may be initiated. The pulse travels through the water 16 in the pipe until it meets an obstruction or turn and results in a sharp repon called a waterhammer.

17 When a valve is not fully closed and high pressure water is being forced through a small opening 18 in the valve, a vibration pattem, in very unusual situations, can be set up in the flowing water, 19 creating a significant noise.

20 It was not clear why this situation was highlighted by the DET as an example ofineffective 21 engineering support. The subject received considerable attention by engineering and did not, in 22 fact, create operational difficulties. The initial event was not a waterhammer but a low cycle (about 23 22 Hz), high amplitude fluid vibration caused by the flow pattem created when the valve body was 24 close to (but not touching) the valve seat. This vibration was observed with two of the valves but 25 not with the other two. (There are two such valves in each unit.)

26 HL&P installed limit stops in the two problematic valves to prevent flow in this particular regime.

27 This allowed a continuous flow of feedwater of 50 to 70 gallons per minute during periods when 28 the auxiliary feedwater system was in operation. The modification also created a need to 29 occasionally deliver feedwater to the steam generators in a batch wise fashion. This type of

(,/ ) "

DET repon. p. 27.

Page IV-28 The Lsberty Consulting Group

d P:.5 .'

Chapter Four - Engineering i operation created some thermal cycling of the steam generator inlet nozzles but did not threaten 2 to exceed the allowable number of cycles. which is 20.000. Although there was no intention to 3 continue this mode of operation. doing so would not have caused STP to exceed the permissible a number of thermal cycles for the operating life of the plant.

5 in a continuing effort to improve the matter, a replacement valve has been installed in Unit 2 to 6 determine whether a different geometry within the valve will result in more acceptable flow 7 dynamics. This is a reasonable response to the situation but it will not be known if the response 8 is effective until a period of operation has passed. If successful, the corresponding valve will be j 9 replaced in Unit 1.

10 In reviewing HL&P's handling of this unusual flow behavior, Liberty found that an appropriate it investigation was made of a unique problem, that a workable temporary fix was made to allow 12 proper system operation, and that a reasonable modification was installed to determine how to 13 permanently resolve the situation. Engineering took timely action to maintain the plant in a' u satisfactory operational condition while seeking a more satisfactory solution. This was evidence 15 of prudent management.

'N r

(

Page l\-29 The Libern Comuhmg Group

Chapter Four - Engineering -

i III. System Engineering 2 A. Introduction -

3 . The DET praised the system engineering program, characterizing it as " comprehensive."'  ;

4 However, the DET expressed concern about the implementation of the program because it believed 5 that insufficient resources had been provided to the organization. STP's management recognized  !

6 that some responsibilities had been assigned to the system engineers that, while appropriate in 7

view of their experience and capabilities, detracted from their primary function of ensuring 8 systems operability. As a result of HL&P's observation, several responsibilities were reassigned 9 (prior to the DET visit) so that the system engineers could become more deeply involved in ,

10 system related matters " For example, procedure writing was transferred to the Programs Division.  ;

1i temporary modifications were assigned to Technical Support, and project engineers were selected 12 from other organizations.

13 PROTECTED MATERIAL FOLLOWS  !

14 INPO conducted two assistance visits at STP in January 1993. The purpose of the first visit was 1 15 to review the work management process and to suggest areas where improvements could be made. .;

16 The second visit, two weeks later, was focused on two areas, one of which was a review of system 17 engineering to develop suggestions for enhancing the program. In both instances INPO

. 18 complimented HL&P's programs in several specific areas.

19 INPO noted that HL&P had established a plan "to improve the process for monitoring and trending i 20 equipment performance to enhance overall system reliability."" To facilitate the pursuit of this 21 objective by system engineering, HL&P established a technical support engineering group in mid-22 1992 located in the area that served the Maintenance Department to provide attention to daily ,

23 maintenance activities and thereby permit system engineers more time and fiexibility to monitor r 24 systems. This arrangement was . lot fully effective in January 1993, but INPO recognized this 25 technical support group as a resource that would significantly strengthen the system engineer I

i DET Report, p. 26.

?

DET response item #3230.

Lener, INPO to HL&P, D.L. Gillispie to D.P. Hall, February 10.1993, which encloses 3 trip report of the 1

( assistance visit made on January 25-29,1993.

Page IV-30 The Liberry Consulting Group v -- - , ._. - - - - . - _ _ . . _ . ----.---- - ____.

RL.M Chapter Four - Engineenng D

(v i -croup's ability to fulf11 its responsibilities. INPO also cited the use of several multidisciplinary l

task forces to resolve long-standing equipment issues.

3 During the second INPO visit, the review team noted that HL&P had recently revised the work a process program to improve the management and coordination of work."" A performance 5 indicator was established to track actual work starts compared to scheduled starts. The program 6 also fequired the use of sen' ice requests for all station work. The trip report of the assistance visit 7 provided considerable detail on the strengths of the system engineer function that were obsen ed.

8 Again, the technical support engineering group was cited as a strength; it was noted that the group 9 was " staffed by very experienced personnel" and was " balanced to include both systems and 10 design engineering knowledge." This arrangement improved communications and the timeliness Il of response to emerging issues. INPO also discussed efficiencies in the warehouse and the fact that 12 productivity was being enhanced through the use of self-directed work teams. INPO made no 13 mention of system engineering failing to receive adequate resources.

14 INPO conducted its regular, overall site evaluation in May 1992. This evaluation addressed each

_ 15 major functional area, one of which was engineering support. The final report that is prepared 16 following this type of evaluation always includes the response made by the utility to any 17 significant findings. This response is often in the form of commitments to address INPO's 18 observations. In this case, INPO noted several areas where the system engineering function could 19 be improved. As a result of this finding, HL&P decided to: (1) revise the " system engineering 20 guidelines" to address work priorities and expectations from the monthly walkdowns routinely 21 performed by system engineers,(2) conduct system performance monitoring and trending. (3) 22 reassign tasks so that system engineers could fulfill their primary responsibilities, and (4) establish 23 a preventive maintenance team in plant engineering. The preliminary results of these actions were 24 observed during the assistance visits that were conducted eight months later.

25 In the report on its May 1992 evaluation, INPO also noted that improvements could be made in 26 equipment monitoring. HL&P responded to this observation by instituting additional system 27 performance monitoring on the diesel generators, the turbine-driven auxiliary feedwater pump, the O

Letter, INPO to HL&P, D L. Gillispie to D.P. Hall, February 8,1993, which encloses a trip repon of the assistance visit made on January Il 15,1993. .

l

( "

INPO Evaluation of South Texas Project Electric Generating Station, May 1992, p. 41.

PageIW31 The Lsberty Consulting Group

PLS.:

Chapter Four - Engineering 1

essential chilled water heat exchanger. pumps in the RHR and low pressure safety injection 2 systems, and other equipment.

3 END OF PROTECTED MATERIAL 4

HL&P conducted a performance assessment of activities it had conducted during the period from 5 May.1991 through March 1992. The results of this assessment were reported to the NRC in May 6 1992. A portion of this evaluation involved a review of system engineering." The assessment 7 report noted that the responsibilities of system engineering had been clarified by emphasizing the 8 " system advocate" role in the system engineer guidelines. Also during this period the concept of 9 preparing system health reports was initiated and better avenues of communication were 10 established with station management. The revised system engineer guidelines, which were 11 distributed in September 1992, addressed training and the conduct of walkdowns." Although the 12 system engineer organization was making a valuable contribution to reliable plant operation, it was 13 still establishing detailed working relationships and processes when the DET made its visit in early la 1993. It is helpful to understand this evolving process when considering each of the following 15 subsections, which discuss specific system engineering attributes.

b v

16 B. Resources 17 HL&P had determined, and INPO had confirmed, that actions were needed to help alleviate the 18 heavy work load that had been assigned to the system engineers. Starting in mid-1992, significant 19 actions were taken to permit the system engineers to concentrate on their primary functions. As 20 noted above, significant progress had been made by early 1993. The DET report, however, did not 21 reflect HL&P's initiatives or achievements in this area. The DET report indicated that:"

22 " Program expectations for the system engineers greatly exceeded the resources 23 provided. .. System engineers generally did not complete their monthly 24 walkdowns or did not sufficiently document them when performed. .. System 25 health reports lacked useful detail."

Letter. HL&P to NRC, ST-HL-AE-4083, Att. #2. May 1,1992, p. 9.

DET response item #3022.

r j "

DET Repon, pp. 28-29.

Page IV-32 The Liberry Consulting Group

i l

i l

i RLS.: l' Chapter Four - Engineering V l Prior to the DET's visit. the NRC conducted a substantial inspection of all of the engineering

functions. including activities of the system engineers. In its inspection report. the NRC was 3 complimentary of the system engineers and their reliance on the useful system health reports. The 4 report stated that the system engineers "were proactive in developing semi-annual system status 5 reports. .[which are] intended. .to initiate reviews and adjust priorities based on system " health" l 6 concerns."" The report also praised the individual contributions being made by the system 7 engineers:"

8 "Overall, the system engineers appeared to be a highly skilled and motivated group.

9 Although their workload was high, there was an attitude that they 10 would. . accomplish their assigned work within the existing resources."

11 After the DET evaluation, STP documented its plans regarding the system engineering 12 organization. Regarding the matter of resources, two programs were completed: a restructuring of 13 system engineer responsibilities and a funher reduction in the burden placed on system engineers."

14 Each of these programs was a continuation of the efforts begun in mid-1992 and reviewed and 15 commented on by INPO in early 1993. While the results may not have been apparent at the time

[ 16 of the DET's evaluation, Liberty found that HL&P had recognized and was dealing with this I

17 matter in a reasonable fashion.

is C. Inforrnation Systems 19 The DET report indicated that "[c]omputer assistance to aid the system engineer in documenting 20 and trending system performance and condition was not generally available.""

21 HL&P had taken several substantial actions to evaluate its information systems and to establish 22 a usable, integrated network for use by engineering and other site organizations. A discussion of 23 these actions is provided in the chapter on management and organization. A new software package 24 had been acquired in 1992 and major modifications were being made to it in early 1993. Because 25 a reasonable decision had been made to apply all available resources to establishing this new NRC Inspection Report No. 92-04, ST AE-HL-93023 March 30,1992, p.17.

" NRC Inspection Repon No. 92-04, ST-AE-HL-93023, March 30,1992, p.19.

( "

Operational Readiness Plan, Rev. O, pp. 26-30.

N)%

" DET Report, p. 48.

Page IV-33 t The Lnberty Consultmg Group  : I

- PL5 :

Chapter Four - Engineering

{

l information system so that it would be operational as quickly as practical, very little attention was -

2 being given to upgrading the numerous individual systems being used at that time.

3 ne design ofinformation systems that will meet regulatory and critical business needs as well as

.: optimize efficiency, data control and access, and cost-effectiveness for a nuclear power plant is not 5 a simple task. HL&P concluded that due to the complexities involved and the limitations on the 6 current computer processors, there were no short-term options that should be implemented.

7 Initially, a two-year implementation of an integrated data base was seen as necessary. The total 8 plant system included components of functional data bases for work management, action tracking, 9 purchasing and materials management, chemistry and radiation protection, personnel, financial, 10 records management and document control, and component data."

i 11 The DET was provided extensive information regarding the plans for and activities that were 12 taking place in 1993 and was given information from the independent consultant who was 13 conducting a quality oversight function." This information showed the progress that was being 14 made and the care that was being taken to institute an effective information system that could be 15 applied to the entire site.

t 16 By early 1993 computer systems were generally available to the system engineers. STP's computer 17 systems were in a transitional phase in which many information systems were being used is reasonably, although the applications were in individual functional areas and were not completely 19 integrated.

• i 20 D. Training ,

21 The DET report stated that "[s]everal engineers were deficient in training or equivalent work j 22 experience..*2 This observation differed from recent INPO findings, which were cited above and 23 indicated that the system engineering organization was " staffed by very experiencei personnel" 24 and included a balance of design and system engineering expertise. Moreover, the DE T report did 25 not discuss the significant initiative being pursued in early 1993 to provide further trainiag. System

• STPEGS Long Range Information Systems Plan, January 9,1992. DET response item #0029.

" DET response item #4025, CPL-154. i j

" DET Report, p. 29.

Page 1%34 The Liberty Consultmg Group

I l

l RL ' .'

Chapter Four - Engineering O

I engineers had been scheduled to attend licensed operator training on their primary and backup 2 systems. At the time of the DETs evaluation. 20 individuals 92 percent) had completed this 3 training." This subject was addressed in a report made to the NRC regarding performance 4 improvement initiatives."

5 Other comments by the DET regarding training for system engineers are addressed in the chapter 6 of this report on management and organization.

7 E. Conclusions 8 HL&P made a concentrated effon to establish an effective system engineering organization and 9 to provide it with the authority and expertise needed to fully accomplish its mission. The DET 10 characterized it as " comprehensive." As part of this commitment to the system engineering ii concept, a technical support engineering group was established in mid-1992. This organization s 12 received widespread recognition for its importance and effectiveness. On two occasions in early 13 1993, INPO cited this initiative as a significant strength within engineering.

b

( 14 The DET report included observations regarding the limited level of detail contained in 15 descriptions of the system walkdowns conducted by system engineers and in system health reports 16 they had prepared but did not indicate that this situation had had any adverse safety or operational 17 effects. Clearly this organization had a very full complement of work to be accomplished, and at 18 the time of the DET's evaluation, the size and scope of responsibility for system engineers were 19 being adjusted. The NRC recognized that the system engineers were " highly skilled" and 20 approached their responsibilities with "an attitude they would accomplish their assigned work 21 within the existing resources."

22 As was the case for all site organizations, system engineering was experiencing some predictable 23 difficulty in working through a transition period while computer systems were being upgraded.

24 The several individual automated systems available to them were sometimes difficult to properly 25 coordinate, but HL&P fully recognized the need to establish an integrated information system and 26 was in the process of doing so. ihis was a situation that was understood in mid-1992 when the 27 decision was made to make a najor transition, and HL&P should be commended for having made

" DET response iterns nos. 3230 and 3259.

/m kj\ "

Letter, HL&P to NRC, ST-HL- AE-4295, January 14,1993,p.3.

Page IV 35 The Liberty Consuhmg Group

1 n s.: l Chapter Four - Engineering

]

I such a major commitment to install a system that could be used in a coordinated way by all site

organizations.

.t The DET claimed that the level of training and experience of system engineers was less than a adequate. and yet they were also " highly skilled," according to an earlier NRC assessment. INPO 5 also recognized the skilllevel of these engineers in January 1993 and noted that the organization 6 was " staffed by very experienced personnel."

7 System engineers are not required by the NRC. Some nuclear plants, including STP. have found 8 that the system engineer organizational concept has proven to be valuable. Liberty concluded that 9 HL&P's efforts in establishing, staffing, and supporting the system engineer program were 10 reasonable.

I f

k, 1

l O

Page1%36 The Liberty Consulting Group

RLS.'

Chapter Four - Engineering

^

Q,O

~

i IV. Engineering Work Load 2 A. Backlog 3 The word backlog can often suggest an excess. However, in the case of an engineering a organization for an operating nuclear power plant, as in the case of many support organizations 5 aside from the Maytag repairman, there will always be a certain amount of work remaining to be 6 performed at any given time. To a large degree, engineering's function is to solve problems. These 7 problems, of matters that could be improved, are either presented to engineering by others or are 8 self-generated. At a nuclear plant, these problems are not solved instantaneously. In fact, even the 9 most straightforward matters take time simply because of the need to adhere to procedural controls.

10 Therefore, engineering will always have some backlog of problems to solve and must assign some ti measure of priority to their solutions.

12 A management evaluation of the amount of engineering work to be done is not a simple task. One 13 cannot simply look at the numbers of tasks on the books to be performed and conclude they are 14 too large. A good evaluation should consider the nature and importance of each type of work item

't is and the rates at which work is being originated and completed. Trends should be considered over O 16 a sufficient period of time to smooth short-term aberrations. Finally, a good evaluation should 17 present data accurately. Liberty found that the DET was able to perform only a cursory evaluation 18 of the engineering backlog at STP.

19 The DET said?

20 "The engineering work backlog was large, increasingly rapidly.... The backlog 21 consisted of approximately 10,800 work items on May 1,1993, including 253 22 modifications, 395 engineering change notices, 6674 preventive maintenance 23 feedback items,209 predictive maintenance items,200 Station Problem Report 24 (SPR) investigation items,690 plant change form items,204 design change notices, 25 381 request-for-action items, 54 TMs, 385 procedures, 33 vendor equipment 26 technical information program (VETIP) items, 51 vendor packages, 660 " closure" 27 items,44 operating experience review (OER) items, and other miscellaneous items.

28 The backlog did not include work assignments of administrative or contractor 29 personnel."

O" " DET Report, p. 30.

Pagei\ 37 The 1.iberty Comulting Group

?l 5.:

Chapter Four - Engineering O

1 The nature of each type of work should be considered before it is lumped together with others. For

eumple. the DET mentioned predictive maintenance items in its description of the backlog. These 3 engineering work items are tracked by quarter and by procedure, do not accumulate from one a quarter to the next. This engineering activity is used to determine conditions that may indicate 5 incipient degradation of equipment. The results of these observations are periodically reviewed by 6 engineering to determine whether additional inspections or preventive maintenance actions should 7 be performed. Typically these monitoring activities are not added to the backlog because if any 8 are not specifically addressed during one period, they will be evaluated in the subsequent period.

9 This is comparable to a person checking the engine oil level on his car whenever the gas tank is to filled. If the person does not perform the check at one fill-up, it does not mean he has to do so 11 twice at the next fill-up. The presence of a timely process at STP was conftrmed by obsersing that 12 only 209 predictive maintenance activities (out of a quarterly total of 1,233) remained to be 13 performed midway through the second quarter of 1993. As an additional example, the single la largest category in this total included 3,406 (of the reported 6,674 total) preventive maintenance 15 feedback items that were intentionally on hold because they pertained to preventative maintenance 16 tasks that were in an inactive status."

[d 17 18 ne incoming and completion rates for the various types ofengineering action items were provided to the DET. The following table provides a summary of those data."

DET response item 83315.

\, "

DET response items nos. 3315 and 3013-001.

Page IV-38 The Lberry Consultmg Group

l RLS-:

Chapter Four - Engineering l

\N  !

1 Work Tme inecmine Rate ComNetion Rate 1 2 Modincations 46 year 50 year 3 Enpneermg Change Notices 42 'y ear 77J> ear a PM Feedbacks 283 month 385 month 5 SPRs Cat.1-4 52/ quarter 45 ' quarter 6 SPRs Cat. 5 205' quarter 120 quarter 7 Plant Change Forms 85/ week 90/ week 8 Design Change Notices 65emonth 75/ month 9 Requests for Action 0 20' month 10 Temporary Modincations 6! month 8/ month I! Procedures 23 month 12/ month 12 VETIP 7. month 6/ month 13 Vendor Packages 22/ month 27/ month .

14 Design Closures 35/ month 20/ month f 15 Operating Experience Rep. 13/ month 12/ month l

, l l

16 In more than half of the various types of engineering work activities, the action items were being 17 worked off at a rate faster than that at which they were being generated. This was true in spite of 18 the fact that extensive support was required to respond to the DET evaluation, and both units were 19 in an outage, a period when (1) new work was being identified rapidly, and (2) engineering was 20 giving its primary attention to outage support. This positive work-off rate existed for PM 21 Feedbacks, ECNs, modifications, DCNs, PCFs, temporary modifications, and vendor packages.

22 The Request for-Action program was no longer in use, so the positive work off rate there was 23 expected. Further, the negative net work-off rate was slight for the higher priority station problem 24 reports (SPR) (Categories 1 through 4), the vendor equipment technical information program 25 (VET /P), and operating experience reports (OER).

> 26 The low priority SPRs were being generated faster than they were being resolved, but the 27 outstanding number of Category 5 SPRs represented only a 3%-month backlog at the current work-28 off rate. The only work items with a negative net work-off rate and a considerable backlog were 29 procedures and design impact assessment closures on installed modifuations.

O N

v/

Page IV-39 1 The Lnberty Consulting Group l

1

? L S. .'

Chapter Four - Engineering

\ l Es en though over half of the Phi Feedbacks were intentionally on hold, the remaining number was 2 significant enough to examine funher. Only 199. a backlog of about 16 days, were late, according 3 to HL&P's schedule. In addition, HL&P was taking aggressive action to address the remaining 4 active items. The 1993 Niaster Operating Plan, issued before the DET's review. included the 5 following operating plan:"

6 " Reduce overdue PM feedback to zero. Augment existing staff by 4 contractor 7 personnel to answer all overdue PM feedbacks by May 1,1993 and continue 8 support through July 1,1993 in order to answer all PM feedbacks by their due date.

9 Augment existing staff by 2 permanent employees starting July 1,1993 in order to 10 answer all PM feedbacks by their due date."

11 HL&P provided the DET with additional data that showed the month to-month issuance rate, 12 work-off rate, and backlog for the period from January 1992 through March 1993 for design 13 engineering." (Design engineering is one of the departments under the Vice President, 14 Engineering.) Examination of those data shows meaningful trends. The station problem report is backlog had grown over the entire period, but it correlated to plant outages. Since one or both units 16 of STP had been in an outage since the fall of 1992, 'he recent trend had been up. Design  :

17 engineering had worked down the number of outstanding OERs consistently from March 1992 18 through the end of the year. The number grew again early in 1993 due to other higher priority work 19 and an unusually high number issued in March 1993. Outstanding work generated by 20 correspondence or in action items (typically management commitments) remained relatively 21 constant, even though the number issued increased during the first three months of 1993. The data 22 showed a large increase in the number of outstanding service requests, but this was due to a new 23 program started in July 1992 that replaced NCRs and RFAs and greatly expanded the scope of 24 items being addressed. In general, the outstanding workload for design engineering increased in 25 February and March 1993 because of the attention being temporarily diverted to technical issues 26 like the TDAFWPs, the dual-unit outage, and support of the DET evaluation.

27 An examination of all the data provided to the DET showed that the engineering backlog was 28 clearly not " increasing rapidly." Rather, it was fair to characterize the backlog as essentially stable 29 for over a year with the only increases being associated with an unplanned dual-unit outage. A 30 persistent backlog of PM Feedbacks was being aggressively worked off.

Master Operating Plan - 1993, February 18,1993, PED-5.

't

\ "

DET response item m3013 001.

l Page IV-40 ,

The Liberty Consulting Group l

l- , . ,

1 d

RLS:

Chapter Four - Engineering

-l i B. Emergent Work

, 2 The DET expressed concern about how engineering work was being allocated. " Emergent work 3 consumed approximately 40 percent of engineering time and prevented engineering from reducing 4 the backlog accomplishing scheduled work, or helping to improve the plant."' 0 5 Two observations are necessary to place the DET's remark in perspective. First. HL&P had told

, 6 the DET that emergent work consumed 35 percent of engineering's effort (28 percent in design 7 engineering and 41 percent in plant engineering).' ' Second, from Liberty's experience, the i 8 dedication of 35 percent of engineering to emergent work was reasonable and, if anything.

9 surprisingly small, considering the investigation of problems with the TDAFWPs in both units and 10 the unanticipated dual. unit outage. HL&P had also reported to the DET that, in addition to l

1i emergent work,46 percent of engineering's effort was going to scheduled work and 19 percent was 12 aimed at reducing existing backlog.

13 Liberty found that the available information showed, not that emergent work had prevented work 14 on scheduled and backlogged activities, but rather that HL&P had effectively managed its is resources to address the full spectrum of engineering's obligations.

16 C. Work Control 17 The DET said that the engineering backlog was:ic2 18 " ineffectively managed. . The licensee did not have an effective method to 19 determine the size and composition of the engineering backlog. This conclusion is 20 based on the fact that data initially given to the team was grossly inaccurate and it 21 subsequently took more than 4 weeks to provide reasonably accurate data "

22 HL&P used tracking systems to monitor various aspects of the engineering work load. Among the 23 engineering work items being tracked were plant change forms, drawings, SPRs, OERs, PM 24 feedbacks, RFAs, modifications and ECN packages, predictive maintenance items, DCNs, 25 temporary modifications and procedures (which were both tracked manually), VETIP bulletins, DET Report, p. 30.

f}

DET response item #3315.

DET Report, p. 30.

Page IV-41 The Libern? Consultmg Group

PU .

Chapter four - Engineering

,/ i sendor packages. information SRs. selected closure items, and a composite of design engineering 2 work mhich was updated and distributed weekly). Although plant engineering did not have an 3 integrated tracking system similar to the one used by design engineering. HL&P believed. and the 4 NRC did not disagree. that the tracking systems being used were comprehensive and that no 5 outstanding work items were being missed. Engineering's support to maintenance and operations 6 was adequate, as discussed in section 11.

7 Many aspects of the work load data being used at STP and provided to the DET need to be 8 understood to further appreciate the nature of this issue. The initial data provided to the DET 9 included some gross estimates of the work load with no regard to the relative importance of much to of the work included. The initial data provided were based on an assumption that the DET wanted 1i an accounting of all possible work items regardless of their relevance to what was necessary for 12 engineering to address to provide adequate technical support for plant activities. Later submissions 13 to the DET were focused on a better recognition of the DET's intent but were misinterpreted as 14 being an indication that engineering did not appreciate or recognize its true work load at the time.

15 Contributing to the DET's perception, however, was the fact that many of the tracking systems n 16 used at STP did not separate out the priority work items. The initial data represented a worst-case

( 17 scenario and transmitted an incorrect message.

is in addition, many of the items tabulated in the several tracking systems remained open because 19 they had not been implemented by other departments. The engineering portion of the work had 20 been completed and did not represent a backlog item for engineering. There were, however, some 21 discrepancies in the data provided because of differences in time as to when certain data were 22 entered into the various tracking systems.

23 The statement by the DET that implies IE&P took four weeks to respond with accurate data was 24 not correct. Responses to the DET requests were consistently provided within a few days. After 25 the DET had had an opportunity to assess earlier responses, follow-on requests were submitted to 26 HL&P weeks later. This elapsed time was a result of the DET's evaluation process, not a lack of 27 pertinent information at STP. For example, the data quoted in the DET report were requested on 28 April 29,1993 and responded to later that same day.S3 This reply was supplemented in a minor 29 way on May 7 and again on May 11, but the added information was not used in the DET's report.

30 Similarly, the design engineering data, tabulated by month, were both requested and provided on

. ,~

Y

• DET req;est a3315.

Page IV 42 The Liberty Consulung Group

- .. . . - _ . . . . - ~ - _ . -.-. - .- _ . -

l l

l RLS. '

Chapter Four - Engineering i

April 5.1993.* This response time is remarkable in view of the fact that some analyses had to be

conducted to determine generation and work-off rates in each category.

l 3

A deliberate effort was made at STP during 1992 to focus attention on demonstrating that outages a could be managed effectively and on schedule. An effon was also made to address selected 5 equipment issues to ensure continued operability. These initiatives were not carried out to the 6

exclusion of completing imponant, outstanding engineering work but were accompanied by the 7

recognition that some engineering work could be appropriately deferred. Having successfully 8

managed its planned outages in 1992, HL&P planned to address the engineering work load in 1993 9 aggressively, 10 Liberty found that engineering knew what work had to be performed, knew the amount of effort ii required to perform that work, and prioritized its efforts appropriately. In a previous inspection 12 report, the NRC made a special point of noting that HL&P had understood its priorities and had 13 been able to respond to important issues in an effective and timely way. The repon stated that "the 14 technical engineering responses to the nonconforming conditions identified in the Requests for 15 Action (RFAs) which are issued as Conditional Release Authorizations were well documented and 16 reflected conscientious and conservative efforts to resolve the identified problems. Timeliness was 17 appropriate to the relative significance of each issue."'"

18 D. Work Load and Overtime 19 According to the DET report, the engineering backlog was not only increasing rapidly but also 20 created a need for substantial overtime.'"

21 "The number of work items in the backlog was increasing at a net rate of 428 each 22 calendar quaner (7 person years each quarter). To compensate for this workload.

23 numerous individuals worked more than 70 percent overtime and some worked 24 more than 100 percent overtime in a pay period."

DET request #3013 001.

NRC Inspection Report No. 92-04, ST AE HL-93023, March 30,1992, p. 2.

O '"

DET Report, p. 30.

Page IV-43 The Liberty Consultmg Group

?3.'

Chapter four - Engineermg

(/ i The DETi attempt to consolidate such disparate activities as modifications, which may require 2 a year or two to develop, with RFAs and PM Feedbacks, which can generally be handled in a few 3 hours, is fraught with problems. Nevertheless, according to the data HL&P provided to the DET.

4 a strict accounting of the net increase in the number of action items per quarter was 359, not 428, 5 as reported by the DET. A calculation of this type is not valid however, because for example. the 6 largest single contributor to this value is the set of predictive maintenance activities that, as 7 explained earlier, do not accumulate in a backlog. Eliminating just this one item results in a net 8 decrease it 'he number of backlog items to 319.

9 The other contributors to the increasing backlog were Category 1-4 SPRS (301 person-hours per to quarter), Category 5 SPRs (1,275 person-hours per quaner ), procedures (1,320 person-hours per 11 quarter), VETIP actions (39 person-hours per quaner), and design impact assessments (1,125 12 person-hours per quarter). Combined, these items represent an increased workload of 4,060 person-13 hours, or about 2% person years. More realistically, the low priority SPRs would not be addressed la during this period ofintense attention to the outages. Removing the low priority SPRs from this 15 list results in an increased load of 2,785 person-hours, or about 1% person-years. Both of these 16 values are significantly lower than the 7 person-years reported by the DET. Moreover, even though g 17 work in certain categories was increasing, other categories of work were being worked down at i8 a rate of 17,097 person-hours per quarter, or six times faster than the rate ofincrease in work being 19 experienced in other areas.

20 The DET report correctly noted "that the modifications, the engineering change packages, and the 21 preventive maintenance feedback items were the most significant items in the backlog."'" The first 22 two items were being worked off at a relatively significant rate, and the third was recognized and 23 being managed by HL&P. These results demonstrated that HL&P was not only giving reasonable 24 attention to the most important activities but was also reasonably reducing its person-hour 25 workload at a significant rate.

26 The data provided by HL&P regarding time expended in engineering showed a few specific 27 pockets of significant ovenime. In the Plant Engineering Department (PED) there was an average 28 of 7.5 people who had overtime in excess of 70 percent during February and March 1993.'" This 29 represented under 5 percent of the 155 employees in PED. Of those individuals who had overtime DET Report, p. 30.

N '"

DET response item #3258.

Page IV-44 The bberty Consultmg Group l

l

\

RL5-:

Chapter Four - Engineering b

\.j i in excess of 70 percent. there were two who exceeded 100 percent ovenime during the first pay 2 period in February and five during the second pay period. The high ovenime level was 3 concentrated in the PED staff, the Section XI group, performance technology, and project 4 engineering. These groups would have been concemed with the investigation of the TDAFWP 5 issues and preparation for the DET sisit. HL&P's ovenime policy contributed to the high reponed 6 overtime as well. In at least one case an individual's base 40 hours4.62963e-4 days <br />0.0111 hours <br />6.613757e-5 weeks <br />1.522e-5 months <br /> was charged as overtime 7 because he was directed to change his shift assignment with less than 7 days' notice '" There was 8 essentially no significant overtime (greater than 70 percent) in the months prior to February 1993.

9 The use of large amounts of overtime in the Design Engineering Department (DED) was much 10 more limited. On average only 1.0 individuals exceeded 70 percent overtime in the period from ii February 1 to April 15, 1993." Trie average number of personnel in DED during this period was 12 99. This use of overtime was solely on the MOV project, which was on the outage critical path for 13 a portion of this period.

14 This use of extensive overtime in engineering for a shon pe-iod of time (primarily in February and 15 to a lesser extent in March 1993) was not unreasonable, considering the activities going on at the 16 plant. Furthermore, this limited and concentrated effort could not reasonably be related to the 17 overall engineering workload (or backlog) assigned to engineering.

18 The final point in the DET report regarding engineering's workload was related to drawings. The 19 report stated:

20 "The licensee was not incorporating amendments into site vendor drawings in a 21 timely manner. On March 19, 1993, approximately 11,150 vendor drawings 22 (approximately 50 percent being safety-related) had one or more unincorporated 23 amendments. Drawings with many unincorporated amendments rendered the 24 associated vendor drawings cumbersome to use and impeded work planning and 25 execution. Previous initiatives to reduce this backlog were not effective."

26 In early 1993 HL&P was taking affirmative action to incorporate outstanding amendments into 27 its vendor drawings. The number of drawings that were completed in February and March 28 significantly exceeded the number of new drawings generated with outstanding amendments by DET response item #3271.

l',)

(f "'

DET response item #3283.

Page IV-45 The bberty Consulting Group

P!S. .'.

Chapter Four - Engineering n

I a margin of 399 to 202."' Aggressise action had been initiated to employ contractors to further 2 reduce the number ofdrawings with outstanding amendments." An intemal commitment had been 3 made as part of the Master Operating Plan for 1992 to " incorporate open amendments into 100 4 selected vendor' technical manuals and associated drawings." By March 1.1993 HL&P had 5 identified and prioritized "the vendor manuals that are the most often used and most cumbersome 6 to interpret." At the time of the DET's visit, HL&P was in the process of contracting to do this 7 work. This action would update about 200 of the most important drawings. A similar commitment 8 was undertaken to support maintenance activities in order to " reduce the backlog of vendor 9 documents with five or more amendments by updating at least 33 vendor manuals and 23 vendor 10 drawings by December 31,1993."")

Ii Nearly all of the 11,096 drawings that had unincorporated amendments at the end of March 1993 12 had live or fewer amendments and generally did not pose difficulties in their use. At the time, there 13 were 255 drawings that were potentially safety related and 154 drawings that were not safety-14 related that had six or more outstanding amendments. It was these drawings that were the primary 15 subject of engineering's efforts and the two goals contained in the Master Operating Plan. This set 16 of drawings with six or more amendments represented only 0.4 percent of the safety-related 17 drawings at the site and 0.3 percent of the non-safety-related drawings. Moreover, the existence is of unincorporated amendments has no direct safety significance because all of the required 19 information was available and visible as pan of the individual drawing packages."' HL&P's 20 concern (and presumably that of the DET) was that unincorporated amendments could make a 21 drawing harder to use and increase the likelihood of an oversight or some other mistake.

22 E. Conclusions 23 HL&P was making a concentrated effort to track its workload and either to make significant 24 reductions in or to keep up with all engineering work items that were important to adequately 25 support maintenance and operations and to maintain configuration control. The engineering 26 workload was being reasonably managed and contractor help was applied in those areas where it l

l DET response itern #3055. l "2

Master Operating Plan - 1993. February 18,1993. RMS-1.

g Master Operating Plan - 1993, February 18,1993, MNT-1.

I i

\ "*

DET response item #3055. i4 PageIW46 The Liberry Consultmg Group

.. . .-. =..

l

• l RLS.

Chapter Four- Engineering i would be most meaningful. No clear basis was provided in the DET report for its statements that

the " backlog was large" or that it was " increasing rapidly." The increase in the backlog that was -I

. 3 experienced in early 1993 was both expected and reasonable in view of the dual-unit outage.  !

4 HL&P was managing its allocation of engineering personnel in a reasonable and cost-effective i 5 manner. The percentage of time being devoted to emergent work was held to 35 percent despite 6 the existence of a dual-unit outage. Providing this level of attention to emergent work did not

. 7 detract unnecessarily from reducing the backlog and did not adversely affect the suppon  ;

8 engineering was providing to scheduled work or plant improvements generally. ,

i c

9 Contemporaneous documentation from early 1993 showed that engineering management had a ,

10 clear picture of the work that had to be accomplished and the level of effort that was required to 11 complete it. On the basis of this information. HL&P established reasonable and clearly 12 communicated priorities. This conclusion was also confirmed during an NRC inspection in 1992.

13 Although the DET was concerned about the amount of overtime being expended by engineering 14 in early 1993, an analysis of the time records demonstrated that overtime had never been an issue '

15 ' of concern. For a very short period of time, there were a few concentrated areas where high levels 16 of overtime were used, but these instances are readily understood and appear reasonable because 17 of their short duration and the importance of the tasks being addressed. Here was neither evidence 18 of widespread use oflarge amounts of overtime nor reason for large amounts of overtime due to 19 the backlog.

1 f

I l

Page IV-47 The Liberty Consulting Group y

%.5 .

Chapter Four - Engineering p) t V

l i V. Use of Operational Experience 1

l

The DET concluded that STP's use of industry and site operational experience had been j 3 inadequate."' This conclusion was in sharp contrast to the one reached by the NRC one year a earlier. As a result of an inspection specifically focused on the use of operational experience. the 5 NRC said:"*

6 "The program for handling and feedback of operating experience information. with 7 one exception, appears to be well defined and is being implemented. The inspector 8 did not identify any instances where information considered to be important for the 9 safe operation of STP was not provided in a timely fashion to the operating staff.

10 What appears to be a concern is that timeliness regarding final review and ii concurrence ofOperating Experience Reports (OERs) and Station Problem Reports 1: (SPRs) is not defined in the controlling procedures."

13 The DET used several examples to support its conclusion. The results of Liberty's review of the la DET's examples are provided below.

N l is A. Reactor Coolant Pump Bearing Failure 16 1. Introduction 17 Four reactor coolant pumps circulate water through the reactor and steam generators in the reactor 18 coolant system of each unit at STP. The pumps are huge, have a pumping capacity of over 100,000 19 gallons per minute, and are powered by 8,000 horsepower electric motors. The pumps are not 20 safety-related because, although the plant cannot operate without them, they are not required to 21 bring the plant to a safe shutdown condition."'

22 Regular outage preventive maintenance on the pumps includes sampling the oil in the upper and 23 lower bearings and mnnmg the motor uncoupled from the pump. During October 1992, Unit I was 24 in A scheduled refueling outage. Oil samples had been taken on all reactor coolant pumps, and 25 satisfactory uncoupled motor runs had been performed on pumps IB, IC, and 1D. On October 31.

DET Repon, p. 30.

/ NRC Inspection Report 92-03, ST-AE-HL-93030, March 2,1992, Appendix, p.1.

\2 "'

STP Updated Final Safety Analysis Repon, Section 5.4.

Page IV-48 The Liberty Comultmg Group

i RLS ' 1 Chapter Four - Engineering l A

i maintenance personnel were preparing to perform the uncoupled motor run on the last pump. I A.

2 A high oil level alarm had been activated for the lower bearing, and so. as part of the preparation 3- for the test, approximately three gallons of oil were drained from the lower bearing reservoir. The 4 reservoir has a sightglass indicator, and the technician drained oil until the sightglass indicated the 5 appropriate lesel. The reservoir holds about 23 gallons of oil. Later it was determined that the 6 reservoir had contained about 19 gallons of oil after the technician had performed the draining 7 operation."'

8 After an appropriate oil level was reported, the motor was started. After about two minutes, the 9 lower bearing temperature was noted to be high and the motor was stopped. On the basis of the 10 high temperatures, bearing damage was expected. Oil samples and a visual inspection confirmed i1 that the lower bearing had been damaged. The bearing was replaced and a satisfactory uncoupled 12 motor run was conducted on November 4,1992."'

13 The immediate cause of the damaged lower bearing was detennined to be a false indication in the 14 reservoir sightglass caused by a sludge-like material in the sensing line. This material had p 15 prevented an accurate indication in the sightglass.': The NRC reviewed this event and noted in

\ 16 its inspection report that HL&P's " sampling program implemented the guidance provided in the 17 technical manual" for the pump.i2' HL&P had not changed the oil, but as noted in the same NRC 18 inspection report,"[r]outine changeout of the oil was not required as long as the samples were 19 satisfactory. Additionally, routine changeout was not desired, in part, because of the difficulties 20 associated with the disposal of the waste oil." The NRC also noted that the vendor, Westinghouse, 21 had observed some sludge material in RCP motors they had refurbished but not to the extent that 22 it "could result in a sensing line obstruction."'22 Station Problem Repon 921037, DET response item #2191, and Reactor Coolant Pump I A Investigation.

January 26,1993, DET response item #3235.

Station Problem Report 921037, DET response item #2191, and Reactor Coolant Pump i A Investigation, January 26,1993, DET response item #3235.

Station Problem Report 921037, DET response item #2191.

NRC Inspection Report No. 92 32, ST-AE-HL-93292, January 19,1993,p.8.

! V "'

NRC Inspection Report No. 92 32, ST AE-HL-93292, January 19,1993,p.8.

Page IV-49 The Ltberry Comultmg Group l

PLS.

Chapter Four - Engineermg

/%

' i 2. Operational Experience

The DET used this event as an example of a failure to " properly implement the OER [ operational 3 experience review] program. . An investigation showed that reactor coolant pump motor bearing 4 oil levels had a history of erratic readings and that a lower reactor coolant pump motor bearing was

"")

5 damaged during a previous outage because ofinsufficient LO [ lube oil) in the lower bearing 6 A review of the extensive documentation associated with this event showed that it was not a failure 7 of the operational experience program. First, this documentation gave no indication that the oil 8 levels had a history of erratic readings. Second, the prior incident had a different set of 9 Circumstances and had a different immediate cause than the Unit 1 incident in 1992.

10 During the second refueling outage of Unit 2, a reactor coolant pump lower bearing was damaged 11 due to a lack of lubrication. This event, however, was caused by retained air pressure in a 12 contamment building penetration being applied to the reservoir. The earlier event in Unit 2 showed 13 HL&P that the lower bearing was sensitive to the reservoir oil level. As a result, HL&P issued a 14 standing order that required notification of maintenance planning and system engineering in the

( ,/ 15 event that the oil levels on large motors were found to be out of tolerance."' One of the issues 16 HL&P identified from the Unit 1 incident was that the personnel who had found the high oil level 17 had not complied with this standing order. However, it was not the case that HL&P did not try to 18 use the earlier event to prevent similar ones in the future.

19 Also with regard to operational experience, HL&P was participating in a Westinghouse Owners 20 Group Oil Evaluation Program. The industry program, which is scheduled to be completed in 21 1995, likely will produce an enhanced reactor coolant pump oil sampling program and oil 22 replacement schedule."' If there had been firm lessons from operational experience in this area, 23 the owners group probably would not have been required. As noted earlier, Westinghouse had 24 never experienced the kind of problem found at STP.

DET Report, pp. 30-31.

J

} n' Station Problern Report 921037, DET response item #2191.

NRC Inspection Report No. 92-32, ST AE-HL43292, January 19,1993,p.8.

Page IV-50 The Liberty Consulting Group

l

~

R!.S. ' \

Chapter Four - Engineering i 3. Communications 2 The DET also said that the reactor coolant pump 1A incident was a case in which "{ploor 3 communications resulted in several recurring safety-related equipment failures and damage to a safety-related equipment.""* While insufficient communications may have been associatedw' ith 3 this event. it was not indicated as a primary cause by any of the evaluations performed. Further, o the event could not reasonably be called recurring, and the failure did not involve safety-related 7 equipment.

8 In describing the event in this part ofits report, the DET said:ut 9 "The maintenance worker failed to notify the control room that the lube oil had to been drained. The maintenance worker's supervisor, stationed in the control room, 1I stated that he did not know of the suspect high lube oil level and would have 12 stopped the job if he had known that 3 gallons had been drained."

13 HL&P's investigation of this matter reported different facts. The maintenance worker's supenisor, C la who was stationed in the control room, directed the worker to determine the cause of the high level ,

k 15 alarm. The supervisor then ordered the worker to drain oil to the appropriate level. The worker 16 communicated with the control room that the oil level had indicated properly after he had drained '

• 17 the oil. The only part of the story that the DET reported correctly was that the supenisor had said 18 he did not know how much oil had been drained and that if he had known he would have stopped 19 the job before the motor was started."'

20 4. Conclusions 21 After reviewing this event, Liberty concluded that there were no unreasonable management actions 22 taken or decisions made in connection with it. HL&P took prompt action to investigate this event.

23 A comprehensive report was developed and specific root causes were identified. Corrective actions 24 were extensive and properly implemented. ,

DET Report, p. 23. j DET Report, p. 23.

~ '"

Reactor Coolant Pump I A Investigation, January 26,1993, DET response item #3235.

PageIW51 The Lsberty Comulting Group j l

PU.:

Chapter four - Engineering p

b i Even though the damaged bearing was easily replaced and the incident did not affect plant

operations. HL&P did not minimize the significance of the matter. Many corrective actions were l 3 taken. a special investigation was conducted, and the Group Vice President. Nuclear got directly 4 invok ed in assuring that personnel understood the significance and the implications of this matter.

5 HL&P's concern was not so much about the failed reactor coolant pump bearing as it was about 6 why it had occurred and what needed to be done to prevent similar, but more significant 7 occurrences.*

8 B. Diesel Rocker Arms 9 The DET also expressed concern about the effectiveness of the operational experience review to program at STP because of evidence that vendor notices had not been fully incorporated into work ii instructions. One example concerned replacement rocker arm assemblies in the standby diesel 12 generators /SDG). The report stated:"

13 "Whsn replacing SDG rocker arms with a modified design, the licensee failed to la include specific Cooper Bessemer ser ice bulletin requirements for torquing and g\} 15 installing modified parts. 'Diis could have prevented the replaced rocker arms from 16 functioning properly."

17 HL&P received a bulletin dated June 26,1985 from the manufacturer of the standby diesel 18 generators stating that new rocker arms and push rod assemblies were available to facilitate 19 maintenance activities. This notice was incorporated into the vendor manual for information 20 purposes. A second notice was sent to HL&P indicating that the old design for these assemblies 21 was no longer available. This second bulletin, dated October 9,1989, also provided instructions 22 for installing the new assemblies. This bulletin was made a part of the vendor manual at STP on 23 December 18,1989 along with the proper design change notice (DCN). It was not necessary to 24 make any replacements, and individual assemblies could be replaced without having to replace all 25 assemblies in a particular diesel generator."' l 1

l

• Memorandum D.P. Hall to Distribution, January 29,1993, Reactor Coolant Pump 1 A Event Analysis. ,

included in DET response item #3235.

)

i (A

V) "'

DET Report, p. 31.

DET response item 83137, pp.197199 and 211-237.

U Page IV-52 I

i The Liberry Consulting Group i

RU-:

Chapter Four - Engineering i On April 2.1993 a concem was identified by HL&P as to whether the retainers for the new rocker 2 arm assemblies had to be torqued and. if so. to what value. It was readily confirmed that torquing 3 was necessarv, and within three hours ofidentifying the concem. a station problem report (SPR) 4 was prepared and immediately taken to the Unit I control room, which in tum notified the Unit 5 2 control room. The SPR was written for SDG #21, which was out of service for maintenance at 6 the time.'M 7 The responsible engineer for diesel generators was also notified. After a briefinvestigation of the 8 situation, and recognizing that SDG #22 was already several hours into a 24-hour test run, the 9 engineer decided to secure SDG #22 to prevent any possible damage. The SDG was secured about to six hours into its 24-hour run and about five hours after the concern about torquing had been i1 identified."' By coincidence this was the only SDG that had new assemblies installed (three on 12 the inlet and three on the exhaust)."' It was later determined that no damage had occurred.

13 The issue, as identified by the SPR, was that the requirements for proper installation of the new 14 assemblies that were contained in the vendor bulletin and properly incorporated into the vendor 15 manual had not been transferred to the work instructions.us 16 in summary, the operational experience review program was implemented properly and in a timely 17 way regarding the vendor bulletin on new rocker arm assemblies. The difficulty occurTed because 18 not all of the pertinent information had been incorporated into the work packages. Although the 19 situation could have led to a problem, no equipment damage was inctured. When a concem was 20 identified regarding the torquing of the retainers for these new assemblies, HL&P took prompt 21 action to notify both control rooms, to prepare a problem report, to contact the vendor,"6 and to 22 secure the one diesel generator that was in operation (under test conditions). These actions were 23 accomplished in about five hours.

"2 DET response item #3149-001.

DET response item #3149-001.

DET response item #3310.

DET response item #3149.

DET response item #3149-001.

Page IV-53 The Lberty Consuhmg Group l

L

?l5-:

Chapter Four - Engineermg r

(

b i C. Vendor Equipment Technical Information Program 2 Equipment vendors typically provide a technical manual that describes the design and operation 3 of the equipment and provides detailed instructions on testing, maintenance, and repairs. As the a supplier upgrades the equipment or identifies operational or maintenance problems it will usually 5 inform the purchaser of the equipment about these matters. These notifications frequently contain 6 revised maintenance instructions,information on new replacement parts, and guidelines designed 7 to prevent problems. It is important to maintain the manuals in a current status so that maintenance 8 and modification activities are carried out accurately. At STP, the process for ensuring that vendor 9 manuals are kept current is called the Vendor Equiprnent Technical Information Program fi'ETIP).

!0 The DET report contained several observations about the VETIP at STP.*

11 "The licensee assigned limited personnel and hardware resources to the VETIP to 12 receive, distribute, and track vendor information. The licensee added staff 13 temporarily to correct problems, but did not take long term corrective actions, thus 14 permitting problems to recur. For example, the licensee noted a significant backlog

/_ 15 of unincorporated vendor information in 1988, temporarily reduced the backlog in 16 1991, but since then, allowed a significant backlog to accumulate. Many examples 17 of inadequate incorporation of vendor information were repeatedly noted by 18 Quality Assurance (QA), the Independent Safety Engineering Group (ISEG), and 19 other audit groups without substantive corrective action being taken."

20 Liberty found that the information provided to the DET indicated that STP made reasonable 21 decisions and took reasonable actions in connection with the VETIP.

L 22 Three NRC inspection repons contained observations on the vendor manual program at STP. Two 23 inspections were conducted in 1988. In both of the 1988 inspections the objective associated with 24 evaluating VETIP was to ensure compliance with an NRC document, Generic Letter 83-23. a 25 portion of which addressed the licensee's interface with its vendors. Both reports noted no 26 deviation from the NRC's requirements.* The third inspection was more heavily concentrated on 27 the VETIP and the reported findings were uniformly positive. 'Be objective of the inspection was 28 to perform "an in-depth review of the licensee's vendor equipment technical information program DET Report, p. 31.

' / "'

NRC Inspection Report No. 88-50. ST AE HL-91755, September 1,1988, p.14. and NRC Inspection Report No. 88-70, ST- AE-HL-91940, January 5,1989, pp.19-20.

Page IV-54 The Liberry Comulting Group

PLS. '

Chapter Four - Engineering V 1 (VETIP) by verifying that selected bulletins had been incorporated." As part of this process. the 2 NRC reviewed the pertinent procedures at STP and observed that "these procedures provided 3 adequate controls for. . receipt, review, statusing, and distribution of vendor supplied design, 4 technical. and QA documents applicable to this facility." The conclusion reached was that "[t]his 5 inspection determined that the licensee's programs were efTective and were being implemented in 6 conformance with requirements."'"

7 These findings by the NRC were consistent with investigations performed by HL&P's QA 8 organization. As a result of an audit of the document control function at STP, QA concluded that 9 "results indicate that the processing and control of vendor documents by PDC (Project Document 10 Control) appear to be etTectively implemented." This conclusion was reaffirmed in each of the 11 next two years. In 1989, QA stated that the " processing and control of vendor documents by PDC 12 appear to be effectively implemented" In an audit of the design and modification control 13 function at STP, the findings made by QA were equally positive and more specific: "The audit la assessed two areas under the VETIP program. Vendor bulletins / advisory reviews are effectively is performed including corrective actions and follow-up actions. He relatively small number of open 16 VETIP items (19) indicates effective implementation of the program."i42 U 17 A review of three ISEG reports, four QA audits reports, a QA surveillance report, and four NRC 18 inspection reports, all of which address VETIP, showed that in only one instance was the matter 19 of " inadequate incorporation of vendor information" mentioned. That exception was an ISEG l 20 report in 1992, where in two cases out of twelve examined there was a lack of proper incorporation 21 of vendor information.) Other findings were noted, however, in these evaluations of VETIP. In 22 three cases, observations were made about needed improvements in the process whereby the user 23 (usually maintenance) reviews and responds to newly incorporated material. These three 24 observations were associated with the user process and not the incorporation of new information.'"

NRC Inspection Repon 89-41, ST AE H1.-92305, November 21,1989, pp. 29-30.

STP QA Audit 88-41, October 11,1988,p.3.

I STP QA Audit 89-08, April 19,1989, p. 3. l "2

STP QA Audit 90-11, September 13,1990,p.3.

1 ISEG Report 92-05, March 1993, included in DET response item #3111.

(

i

\ '"

ISEG Report 90-29, June 1990, ISEG Report 9105, June 1991, and QA Audit 88-41, October 11,1988.  !

Page 1V-55 The Liberry Consulting Group

a a

RL5 .'

Chapter Four - Engineering

/3 ij i There were also three observations related to concerns about the database being used and about 2 the tracking system being unable "to track departmental commitments to completion."us 3 There were only three additional findings discussed in these documents, and they were associated a with three different matters: the lack of timeliness in incorporating new material, concern about 5 the size of the backlog (although this unfinished work was associated with non-safety related 6 equipment), and inadequate implementation of a vendor recommendation.u6 7 In addition to these reports, there was one event, documented in an SPR, where information e 8 required for troubleshooting and repair of certain MOVs was not in the vendor manual."7 9 Although there were areas where improvements could have been made in VETIP, especially in the 10 tracking system, Liberty did not find instances in which " inadequate incorporation. .were 11 repeatedly noted." HL&P had recognized the need for improvement in its tracking system and had 12 made an intemal commitment to enhance and update it. Improvements began in the fall of 1992, 13 a formal action plan was developed in May 1993, and the new system was made operational later 14 in 1993."'

n 15 Actions had been taken in early 1993 to reduce the backlog of amendments to vendor drawings.

16 The number of drawings with outstanding amendments was reduced by 28 in January and by 17 another 169 in February 1993. A more detailed discussion of those efforts is presented in section 18 IV of this chapter, Engineering Work Load.

"5 ISEG Report 90-29, June 1990, ISEG Report 91-05, June 1991, and QA Surveillance Report 88-033, January 29,1988.

STP QA Audit 92-18, conducted October 26 to November 13,1992, ISEG Report 91-05, June 1991, and STP QA Audit 88-41, October 11,1988.

Station Problem Repon 93-0782, issue identified February 26,1993. DET response item p3092.

Memorandum, M. Pacy to C.T. Bowman, April 2,1993, DET response item 83111, pp. 278 79.

PageIW56 ,

The Liberty Consulting Group i

i R!.S.: )

Chapter Four - Engineering i D. Application of Risk Assessment Technology

The DET report was complimentary of the initiatives taken by HL&P in establishing a risk 3 assessment organization and in developing useful applications of risk technology. The report 4 stated:*

5 "The licensee established a probabilistic risk assessment /PRA) group and a 6 detailed PRA for the site. The licensee used this PRA infonnation to enhance some 7 plant procedures and hardware, and tojustify 16 proposed changes to the technical 8 specifications (TS)."

9 However, the DET expressed mixed feelings about the status of the data used in PRA 10 applications.'"

Ii "However, the licensee had not updated the PRA database to reflect actual plant 12 equipment failure data. The PRA had been updated to reflect the frequency and 13 duration of maintenance for some plant equipment."

(

\

14 Finally, the DET repon presented a few specific observations about the application of PRA 15 technology "'

16 "The licensee was not using the unique capabilities of the PRA group to identify 17 plant equipment reliability or to help in rankmg modification or maintenance work.

18 During this evaluation, the licensee used PRA to address team concerns with the 19 reliability of the SDGs, in particular for SDG 22, but only in response to specific 20 and repeated team requests." ,

21 The highly positive statements regarding the PRA group and its usefulness were supported in a f 22 number of NRC inspection reports and HL&P documents. In response to a request from the DET, l 23 HL&P summarized its present and planned applications of PRA. HL&P noted that its "PRA efforts 1

24 were begun in 1983" and that PRA technology had already been applied in 14 broad areas, 25 including hardware and procedure improvements, LERr, justifications for continued operation, 26 corrective action programs, safety evaluations, design reviews, and " risk based evaluation of DET Repon, pp. 31-32.

DET Repon, p. 32. I i

DET Repon, p. 32.

Page IV-57 The Liberty Comultmg Group

?L.G.'

Chapter Four - Engineering O

i maintenance scheduling for shortening refueling outages." This last point was in direct contrast 2 to the DET report. which claimed PRA had not been used "in ranking. . maintenance work." This 3 same HL&P document also listed nine areas in which there were plans to use PRA. including a implementation of the maintenance rule and principles of reliability centered maintenance.

5 revisions to design basis documents, prioritization of work for system engineers. and system 6 performance monitoring."2 7 A second HL&P document reiterated much of this same material with some additional emphasis 8 on the use of PRA in the development of emergency procedures, which are based on scenarios 9 treated in detail in PRA documents such as individual plant evaluations (IPE).")

10 Recent NRC inspection reports were complimentary of the PRA program at STP as well. The NRC 1I issued an inspection report in the spring of 1992 regarding its detailed evaluation of engineering 12 activities. One of the areas investigated was the application of PRA technology at STP and 13 HL&P's progress in completing an individual plant evaluation, a plant-specific PRA mandated by 14 the NRC. The conclusion reached in the inspection report was that "[t]he licensee has initiated very 15 comprehensive DBD and PRA programs.""' In another example, a subsection of a report by the f

16 NRC issued in late 1992 was devoted to HL&P's performance of a shutdown risk assessment. This

(

17 work was done to improve specific outage activities or to confirm that they could be carried out 18 with confidence that a large safety margin was being maintained. The inspection report concluded 19 that "[s]hutdown risk assessment activities. .were considered positive licensee initiatives.""5

0 A second inspection report issued contemporaneously with the DET visit was equally enthusiastic 21 abcut STP's use of PRA techniques. Regarding the assurance of shutdown safety, the NRC 22 observed that " shutdown risk assessment will be a licensee priority during the outage (the third 23 refueling outage on Unit 2 scheduled for early 1993). A shutdown risk assessment team was 24 developed... [whose] chairman was a representative from plant operations who was a licensed 25 senior reactor operator." The report went on to describe the details of the team's activities and "2

DET response item #3110.

Procedure Changes as a Result of Probabilistic Risk Assessment (PRA), April 6,1993.

NRC Inspection Report No. 92-04. ST AE HL-93023, March 30,1992, pp. 24 25.

O NRC Inspection Report 92 26, ST-AE-HL 93218, October 16,1992,p.28.

Page IV-58 The Lsberty Comultmg Group

R L S. .'

Chapter Four - Engineering O

V 1 methods of operation. It concluded that "[s]hutdou risk assessment continue [s] to be licensee 1 2 strengths."*

3 The DET report's statement regarding the use of site specific failure data must be placed in a perspectis e. Because STP had been operating for only a short period of time (4% years for l' nit 5 1 and less than 4 years for Unit 2 at the time of the DET visit), and because STP had experienced 6 very few failures in its safety-related systems, the failure data available were very sparse. Although 7 there are techniques for incorporating sparse data into more generic industry data (using Bayesian 8 update methodology), such incorporation must be done with the utmost care and judgment. In 9 many cases it is neither practical nor prudent to do so because it would unduly bias the data. HL&P 10 was reasonable in not making wholesale updates. It was judicious, however, in accounting for ii maintenance activities, many of which can indirectly reflect on failure histories.

12 The statement in the DET report about identifying equipment reliability was also incorrect. As 13 stated earlier, HL&P had applied its PRA expertise to hardware improvements and to corrective la action programs aimed at solving equipment problems. In addition, the shutdown risk assessment s 15 program was based almost entirely on equipment reliability calculations. There is always room for 16 broader application of the technique, but it may not be cost-effective to do so.

17 The last statement regarding PRA in the DET report ("but only in response to specific and repeated 18 team requests") must also be placed in context. The DET made two requests regarding the history 19 of all diesel generators. These requests did not mention reliability analyses. HL&P provided 20 extensive replies (152 and 433 pages).'" Each of these requests was followed up by the DET with 21 one additional question. In the first case the DET requested a statistical analysis. In the second case 22 two of the nine-part question related to reliability."' There were several other related questions on 23 diesel generators, but none pertained to the issue of reliability, except in one case where the DET 24 questioned the level of involvement by the PRA group."' A second round of two questions 25 regarding diesel generator reliability, when the first round did not require the issue, should not 26 have been characterized as " repeated... requests."

NRC Inspection Report 93-04, ST-AE HL-93376, April 16,1993, p. 29.

DET response items nos. 3176 and 2193.

I DET respnse items nos. 3677 and 2193-001.

'h "'

DET resp < nse items nos. 3677-001,2193-002, and 2193-003.

Page IV 59 The Liberty Consuhmg Group l

I

? ' S .'

Chapter Four - Engineering s

i In summary. the PRA program at STP received several favorable appraisals from the NRC

throughout 1992 and early 1993. Even the DET observed the useful application of PRA technology 3 by HL&P. Although criticized by the DET. HL&P's use of site-specific data was reasonable and a was consistent with industry practice for newly operated plants. The use of PRA in maintenance 5 activities and in identifying equipment reliability was also reasonable and appropriate.

u)

O %

Page IV-60 The Liberty Consulting Group

.. - .-. - ~. . .-

RL5-:

Chapter Four - Engineering

\

d i VI. Support to Engineering 2 The DET concluded that STP's management had not provided sufficient support to engineering 3 so that it could fulfill its responsibilities in an effective and supportive fashion. Specifically, the a report stated:*

5 " Management did not support engineering by assigning an adequate number of 6 Staff; supplying resources to implement engineering modifications, corrective 7 actions, and improvement initiatives in a timely manner; an accurate material 8 database; and maintaining an accurate management infom1ation system, including 9 personal computer and software support. These factors reduced the effectiveness to of engineering performance."

11 The report attempted to illustrate these statements by citing six examples, each of which is 12 discussed in one of the following subsections. He DET's examples were: inadequate information 13 systems, sparse computer resources, increasing workload, weak and inequitable training, 14 unsuccessful improvement programs, and untimely modifications.

15 A. Management Information Systems (MIS) 16 he DET believed engineering had inadequate systems to evaluate system performance, to trend 17 maintenance history, to access industry and site experience, to perform root cause analyses, and 18 to make informed decisions.* The team also appeared to be concerned that "the equipment 19 maintenance history database was not accurate and current." The basis for this concem was not the 20 availability of an adequate automated system but an assessment that the quality of data was " poor" 21 and the backlog of unentered data was large, possibly consisting of six to eight months of data. As 22 a result, the DET said that engineering did not use the infom1ation in the database. The DET report 1

23 went on to say that "various databases showed conflicting and incomplete information" related to 24 CH chillers, standby diesel generators, temporary modifications, and MOVs. Also, the DET said .

l 25 that retrieval ofinformation by major category was often not possible, and that in one case HL&P 26 had had to " manually search service requests" to find where specific replacement parts had been 27 installed.

4 DET Repon, p. 32.

DET Repon, p. 32.

Page IV 61 The Liberry Consulting Group 1

- - _ . . _ . ~ ,

l

I

\

RL 5 .'

Chapter Four - Engineering

/O --

\

\w/

i The earlier discussion in section III about system engineering summarized the situation that existed 2

at STP in early 1993 regarding information systems. A management decision had been made in 3

1991 to institute a total change in STP's information systems. HL&P recognized that the sy stems a

in use in 1992 had a number of shortcomings relative to current computer system capabilities. In 5 early 1993. STP was in transition to a site integrated system.

6 The equipment maintenance history data base was not current in all respects, but the backlog of 7

data entry was overstated in the DET report. In contrast to the DET's observation, Liberty found 8

that the data base was being used by engineering. As described in the chapter of this report on i 9 management and organization, Liberty's review of the first 20 equipment histories requested by 10 the DET demonstrated that the histories were up to date rather than six to eight months behind.

1i The last two statements made in the DET report regarding information systems also deserve 12

' comment. It is correct that Plant Change Forms could not be tracked by system or type because 13 the input data did not carry such an index. However, these documents were accurately and 14 completely monitored. Similarly, in the case of the SDG rocker arms, the tracking system may not 15 have been the most efficient one possible, but the records were retrievable and accurate.

(w 16 B, Personal Computers 17 ne DET report stated that "[t]he effectiveness of engineering was hampered by sparse computer 18 resources and analytical tools to monitor and assess component or system performance. Until the 19 end of 1992, only five percent of the system engineers had a computer to aid in performing their 20 job function." The purchasing of personal computers for widespread use in plant engineering was 21 initiated in 1992. By the time of the DET visit,80 percent of the system engineers had personal 22 computers, and action had been taken to provide them to the remaining 20 percent during 1993.'*2 23 Moreover, prior to 1993, system engineers had access to a computer through terminals located in 24 their work area. Although this system was not as efficient or convenie - 95 a PC-brid system, it 25 was not unreasonable and any implication that the system engineers did not have any computer 26 support was not correct.

'(-)

DET response item #3178.

Pa ge IV-62 The Liberty Comultmg Group

1 l

l Rl.S. .'

Chapter Four - Engineerme j

i C, Increasing Backlog of Work l 2

The DET report stated that "[b]acklogged engineering work continued to increase.. . even though 3

most groups. . worked substantial amounts of overtime." " The subject of engineering workload 4

is discussed in some detail in section IV of this chapter. That discussion provides specific data on 5

the engineering workload in the various categories of engineering activities. It describes the 6

reasonableness of the manner in which this workload was being managed, and in most cases being 7

reduced, in early 1993. The presentation in section IV also indicates the very limited use of high 8

levels of overtime in cases where a few operational issues had to be resolved expeditiously. The 9

workload in engineering was not unusually high, it was being reduced in a reasonable fashion, and to the amount of overtime being expended was reasonable and was due almost entirely to the dual-ti unit outage and specific equipment issues, not a lack of management support for engineering.

12 D, Training 13 The DET report stated:'"

14

" Management support for trauung was weak and inequitable. PED was weaker than is DED in terms of background and experience, had more staff (170 vs.148), but 16 were assigned only ea seventh We training budget of DED. He licensee primarily 17 used OJT (particularly in PED), or previous work experience rather than training."

18 he statement in the DET report regarding reliance upon on-the-job training and previous work 19 experience implied that these attributes of an individual's background were somehow inferior to 20 formal training. He fact is that plant engineering personnel (PED) had a hands-on responsibility 21 for ensuring plant operability (much more so than design engineering [DED]), which requires 22 knowledge of actual equipment operation (how valves work, how pumps and motors operate 23 together, and how instruments sense important plant parameters), an understanding of how to 24 conduct an effective walkdown of systems (what to look for, typical sounds of equipment when 25 it is operating properly or not, and how to assess abnormal conditions such as small leaks), and a 26 sense of how to perform analyses of actual and potential equipment failures. Some of this }

27 knowledge can be gained in a classroom setting, but most ofit must be imparted by experiences 28 in the plant: seeing how equipment works, observing what it looks like when dinssembled, and i i

DET Report, p. 32.

(n)

(/ '"

DET Report, p. 32.

Page IV-63 J The Liberry Consultmg Group l

?LS .'

Chapter four - Engineering CN

() I appreciating how incipient failures can lead to major problems. The fact that HL&P was relying

heavily on OJT and prior plant experience for plant engineering demonstrated that management 3 understood the relatise importance of this hands-on exposure.

4 A discussion of the DET's comments on the funding provided for the training of engineering 5 personnel is presented in the chapter of this report on management and organization. That 6 discussion points out that the budget figures used by the DET as the basis for comparing DED and 7

PED training had very different components and, standing alone, should not support any 8 conclusion as to the relative emphasis being placed on trauung. The DED budget number was for 9 payroll-related expenses for training coordinators. The actual funding used for off-site training was 10 not a separate program element for DED. However, PED's budget reflected specific off-site 11 training. In neither case was on-site training shown as a separate program element, and yet this 12 constituted a major fraction of the training provided to all engineering personnel.

13 E. Application ofImprovement Programs 14 STP's engineering tracked several initiatives meant to improve both plant and engineering

/ 15 performance under a category called improvement programs. The DET report stated that 16 "(ejngineering performance was not substantially improved through the improvement program 17 process." The report went on to indicate that some programs were not completed on schedule.

18 some were cancelled, and some were not evaluated for effectiveness.'"

19 "The licensee fell behind its schedule in completing many improvement programs 20 designed to improve engineering performance and cancelled some after investing 21 substantial resources. Some corrective actions resulting from improvement 22 programs produced no improvement in performance and were later cancelled. The  ;

23 licensee appeared to classify improvement program action items as " closed"  ;

24 without evaluating thetr effectiveness." '

25 Before the specifics of these statements are addressed, it is helpful to recognize that much of what 26 is presented in the above quotation could be said about nearly any series of complex programs 27 being pursued in a reasonable fashion by conscientious management. For example, the fact that 28 selected programs were cancelled, during both their development and implementation phases, and 29 that others were actively maintained indicates that some management evaluation process had taken

/~'N DET Report, p. 33.

Page IV-64 The Liberty Consuhing Group

RLS.. '

Chapter Four - Engineering 1 place. Also. departure from an original schedule in every situation is common and does not l 2 necessarily imply unreasonable management; often, the opposite is true when long-range programs 3 have t- give way temporarily to unexpected issues requiring prompt management attention.

4- Certainly it is sound management practice to cancel a program when its expected benefits are not l

l 5 being realized. Taken as a whole, the description set forth here could be that of an exemplary  ;

f 6 situation in which sound and reasonable decisions were being reached on a continuing basis. To l 7

a large extent that is the conclusion reached by Liberty regarding STP's engineering support.

8 An appreciation of the overall budgeting process that an owner / operator of a power plant must go 9 through to establish a reasonable business plan is critical to understanding the difficult choices that to must be made regarding proposed plant modifications. The limited funds available to a utility must 1i be balanced among many competing needs, such as safety, operability, reliability, and regulatory 12 initiatives. For example, one of the key assumptions established for the development of the 1993 13 budget for STP was that "[e]ngineering backlogs will not be significantly reduced." 66 This i 14 guideline reflected a decision reached through careful deliberation that a reduction in the backlog 15 would be kept at a modest level because other priorities were judged to be relatively more 16 important, j 17 Despite this important assumption regarding the allocation of funds, the 1993 budget for STP 18 showed an increase over 1992.In addition, the budget for the five year modernization plan for 19 STP was increased substantially from the 1992 projection to that for 1993. "

20 STP had a successful history of developing and instituting improvement programs. An .

21 independent, detailed study of engineering activities was performed in late 1991 and reported on 22 in 1992.'" As a result of the numerous recommendations made in the independent report, many 23 actions were initiated to respond to those matters believed to be important. As many of these 24 actions were reaching completion, the status of each was briefly described in a memo from the DET response item #3138.

DET response items nos. 4028 and 4070.

DET response item #3138.

Analysis of Engineering Activities. Performance Data, Inc., February 14.1992.

Page1%65 The Liberty Comulting Group

PLS.' .

Chapter four - Engineering Vice President. Engineering." This memo indicated that seven actions had been completed.

2 another seven had been discussed with the proper super isory personnel with a view toward J 3

incorporating them into ongoing processes. one had been deferred one had been combined into a

another action item one action was still being formally tracked, and several other were not pursued 5

after they had been evaluated. Four were listed as not having been formally tracked, although 6 informal actions were being continued.

7 One of the documents provided to the DET was a summary of actions initiated under the 8

Operational Improvement Plan.'" This listing included 34 items for design engineering (DED) and 9

27 items for plant engineering (PED). In the case of DED, four items had been deleted: two had 10 been evaluated as not adding value to the engineering process, one had been incorporated into 1i another item, and one had clearly been not cost-effective. Six items were still open and 24 were 12 considered closed. The completed actions included a three-phase program of upgrading vendor 13 documents, a streamlining of the modification process and DCNs, incorporating amendments into la drawings, and developing lists related to relays, fuses, critical loads, and instrument setpoints. In 15 the case of PED,26 items had been closed and one had been combined with another action. The 16 completed items included streamlining procedures, the RFA process, and safety evaluations; j 17 clarifying the role of system engineers; and establishing an RCM program.

\

18 Another part of the package provided to the DET included the results of the engineering portion 19 of the 1992 Master Operating Plan. This document described seven goals: four were achieved. two 20 were nearly met, and one was deferred. In a June 1991 letter to the NRC, HL&P recounted some 21 of the discussion that had taken place in a meeting with the NRC held on May 10,1991." Two 22 reports that provided the basis for the NRC meeting were enclosed with the letter. One of the 23 subjects addressed in the meeting was improvement initiatives, a " majority [of which] were self-24 initiated by HL&P as pmdent." The basis for these improvement plans was a review and analysis 25 of a number of assessments, including those from nuclear assurance, the corrective action program, 26 special task forces, independent consultants, and INPO as well as the NRC These programs 27 covered nearly every major function at STP, including engineering.

Memorandum, S.L. Rosen to Distribution, Analysis of Engineenng Activities, June 16.1992, DET response item 83063.

DET response item #3063.

, f%

Letter, HL&P to NRC, ST.HL-AE 3783 June 3,1991.

Page IV-66 The Liberry Consulting Group

'l l

l

RLS..'

Chapter Four - Engineering b I In a section of the first report enclosed with the letter to the NRC on " General Oversiew of Station 2 Performance" that addresses " Management Perspective," HL&P stated:

3 "A major goal of HL&P's Operational Improvement Plan (OIP) is to make 4 improvements in the programs, equipment, personnel performance, and procedures 5 that make STPEGS a better place to work and which contribute to safe, reliable 6 operation."

7 in the area of engineering, the report noted that " existing mechanisms for requesting Engineering 8 assistance were consolidated." In the second report discussed with the NRC, which was on "HL&P 9 Initiatives to Improve Station Performance," it was observed that engineering had made major to contributions to resolving four key equipment problems by performing root cause analyses. The 1I report also presented a discussion of how four findings identified as a result of a Nuclear 12 Assurance assessment had been resolved by engineering. These findings were associated with 13 incomplete design packages, inadequate reviews of these packages, and insufficient acceptance la criteria and tests for modifications. Improvement programs were established for each issue and 15 were successfully concluded. Also, as part of the overall improvement process, significant 16 reductions were made in the engineering workload associated with modifications.

17 A subsequent letter, nearly a year later, reiterated the continuing success of some of the is improvement programs at STP.m This letter, which was transmitted in May 1992, described some 19 of the results of a performance assessment conducted at STP from May 1991 through March 1992 20 and enclosed two reports similar in format and content to the two reports submitted with the June 21 1991 letter. One of the achievements noted within engineering was a streamlining of the RFA 22 process, the success of which was demonstrated in major reductions in the time it was taking to 23 respond to RFAs, both nonconforming and informational. The second report included with the 24 letter described the assumption of responsibility for the preventive maintenance program by 25 engineering and the upgrading of the guidelines used by the system engineers it also described 26 engineering's critical involvement in addressing nine equipment issues, in addition, six 27 modifications were designed and implemented to help improve station availability.

28 A major initiative undertaken by design engineering was to perform a self assessment. This led 29 to a number ofimprovements, including commitments to pursue " modernization efforts in two key 30 areas: equipment obsolescence and long range improvement planning. These two areas represent b "'

Lener, HL&P to NRC, ST.HL-AE-4083, May 1,1992.

Page IV-67 The Labero Comulting Group l

ap l C hapter Four - Engineering i the scope of the Long Range Improvement Program." which was scheduled for completion in 2 1992. Other DED initiatives discussed in the report included improvements in instrumentation and

control, the addition of a vent path in the reactor coolant system, an upgraded vendor manual for a the turbine generator, enhanced fuel assembly movement sequences, and many other activities 5 associated with procedures, PRA. design basis documents, and safety evaluations. A tabulation 6 showed further reductions in the engineering workload associated with permanent modifications.

7 One of the accomplishments noted was the establishment of a shutdown risk assessment group.

8 The success of that group's activities are discussed in subsection V.D.

9 Early in 1993, two additional letters were sent to the NRC regarding improvement programs. The 10 first letter addressed performance improvement initiatives specifically and covered a wide variety 1i of programs, one of which was improvements made in training for system engineers and technical 12 support engineers. The second letter described initiatives developed by the NSRB to improve 13 performance at STP.'" In this case, STP had established four standing committees in November la 1992, which started operating in January 1993. One of the committees was specifically established is for engineering and was assigned responsibility for resolving identified problems associated with 16 control of the modification control process, the essential chilled water system, the locked valve C 17 situation, and issues related to solenoid operated valves. In addition, "[t]his standing committee 18 [on engineering) provides overview in the areas of configuration management, engineering 19 assurance, life cycle engineering, modification and modernization programs and engineering 20 support to operations including training associated with the engineering areas."

21 In response to a DET request, HL&P discussed actions taken to help enhance system engineering 22 in four particular areas, including training.'" At about the same time, an Engineering Program 23 Strategic Plan was issued, primarily to address long-term objectives and initiatives.'" These 24 initiatives addressed productivity and performance (such as vendor technical documents and 25 equipment setpoints), regulatory issues (such as MOVs and SOVs), and availability and reliability.

Letter, HL&P to NRC, ST HL AE-4295, January 14,1993.

"5 Letter HL&P to NRC, ST HL AE-4342, March 30,1993.

DET response item s3230, Actions to Enhance Systems Engineering, April 20,1993.

O

\ "'

Engineering Program Strategic Plan. March 30,1993, DET response item #4072.

Page IV-68 The Liberty Consulting Group

l RLS-:

Chapter Four - Engineering O

I The NRC had conducted a comprehensive inspection of the engineering and technical support 2 functions at STP in February 1992."' One of the key observations made in the report regarded j 3 some positive, forward looking programs that were being conducted by HL&P. "The licensee has a developed a significant number of initiatives to enhance the plant and its performance." The s inspection team was obviously impressed by the nurnerous performance improvement initiatives 6 that were in process. The report listed 57 individual improvement programs that were apparently 7 believed to be deserving of note, including programmatic and process functions as well as 8 equipment-related matters.

9 In summary, HL&P devoted a great deal of attention to the development and implementation of to improvement plans for a variety of engineering activities. HL&P was committed to evaluating the ti effectiveness of these plans and to self-assessments as they were carried out. The fact that some 12 of these plans had been cancelled in the past is a tribute to the reasonableness of HL&P's 13 management of such initiatives. It was apparent to Liberty that those plans that were eliminated la were evaluated as not being effective, and therefore HL&P decided not to expend additional funds 15 or manpower on tasks that were deemed not to be fruitful or cost-effective.

(/ 16 F. Timeliness of Modifications 17 The DET report claimed that "[s]ubstantial recurrent problems noted by maintenance, operations, 18 engineering or other groups often resulted in design modifications to resolve the problem.

19 However, the modifications were not installed in a timely manner." To support this proposition.

20 the report observed that in a modification scoping meeting held in April 1993 to determine work 21 to be included in the 1994 budget, only about 50 percent of the identified modifications were 22 expected to be installed that year. The DET also noted that "[m]any of the modifications 23 considered for 1994,1995 or 1996 were initiated between 1987 and 1990."* A similar situation 24 existed for ECNs.

25 The decision to carry forward a fraction of outstanding modifications was typical of industry 26 practice and reflected reasonable judgment by HL&P management. Nearly every utility has an 27 annual evaluation process in which all outstanding work items requiring capital expenditures or 28 substantial O&M costs are thoroughly reviewed to determine their cost-effectiveness and the NRC Inspection Report No. 92-04. ST- AE-HL.93023, March 30,1992.

O

(

DET Repon, p. 33.

Page IV-69 The Liberty Comultmg Group l

r 1 1

I l

1 P: 5.'

Chapter four - Engineering i potential effect of not pursuing them (often based on a risk evaluation) for a given period of time.

2 Usually these evaluations are done not only for the ensuing year but also for a three or five ,s ear 3 period to coincide with Snancial planning models.

4 HL&P used several criteria in evaluating whether to proceed with proposed modifications. These 5 criteria were: station goals and objectives, regulatory commitments, economic merit. Nuclear 6 Group priority, long range improvement, maintenance of a similar configuration between the two 7 units, outage installation requirements, department ranking of priorities, and continuation of multi-8 year projects. No numerical weighting was applied to these criteria but the first three were given 9 the highest priority. The evaluation was an integral part of the strategic planning process and the 10 development of the Master Operating P!an. A data base was established to integrate the 1i modification rankings and the selection criteria.'"

12 The identification of a potential modification does not automatically mean that it either (1) has to 13 be installed within the next year or two, or (2) is required to meet safety or reliability goals.

14 Although it may appear cumbersome to continue to carry modifications that were identified several is years earlier, or appear to constitute inevitable work that is simply being deferred, the continual 16 re evaluation of such proposed work can be beneficial. Often work that might otherwise be 17 permanently rejected can be readily incorporated into another modification deemed necessary at 18 little added cost. On occasion temporarily rejected work, when considered in conjunction with 19 more recent events, can help identify a trend that needs to be addressed. Although the deferral of 20 such work implicitly means total rejection, it can be of value when viewed as part of a larger 2i situation that evolves later on. It might be imprudent to completely lose track of a modification that 22 was at one time considered to have some value.

23 It should not be implied from the DET's statements that all proposed modifications should be ,

i 24 worked soon after they are identified. To follow such a prescription would be highly wasteful of  ;

25 resources and money. Modifications must be carefully managed and limited to those activities that 26 clearly foster an adequate level of safety and promote plant availability at a reasonable cost.

l l

b '"

. (j DET response item #3138.

Page IV-70 The Liberty Coruultmg Group

l l

l ats.: i Chapter Four - Engineering V

I Although it did state that the documents associated with majcr modifications were "well written 2

and comprehensive."'" the DET report did not set out the considerable strengths exhibited by 3 HL&P in the engineering support of STP.

4 G. Independent Studies of Engineering 5

HL&P was aware of the importance of occasional independent assessments ofits performance 6

from other than a regulatory viewpoint. Because of this interest in ensuring that the STP 7

organization continually met sound business objectives, HL&P decided to tap the resources of 8 outside experts to evaluate various aspects ofits engineering processes and activities.

9 HL&P contracted for a number of staffing studies that included detailed evaluations of the 10 engineering organization. Two studies were made in 1991 and one in 1992.a2 These studies were iI performed by two consulting fums that were employed by HL&P to make independent evaluations 12 ofits staff, including engineering, and to make detailed comparisons of existing staffing levels at 13 STP with the staffing required to meet functional requirements (in the 1991 investigations) or 14 discipline related needs (in the 1992 evaluation). The February 1991 report concluded that

(

) 15 engineering management and plant engineering were staffed appropriately to support a two unit 16 plant, but that design engineering was understaffed. Following this study HL&P increased its 17 engineering force from 276 to 318. The DET requested HL&P to compare its February 1,1993 is staffing to the level determined to be appropriate in the 1992 study performed by ASTA. That 19 comparison showed that plant engineering was at the level recommended and that design 20 engineering had four more people than the recommended number."'

21 Several independent studies were made in specific functional areas. For example, three long range 22 improvement programs were developed for engineering for the diesel generators, the essential 23 cooling water system, and the low pressure rotors of the turbine generator,'" Additional DET Report, p. 26.

"2 Technical Services Stafung Analysis. August 1991, and Staffing Analysis Report February 19,1991, reports by T. Martin & Associates; STP Staffing Level Study, September 1992, report by ASTA, Inc.

DET response item s4068.

Diesel Generator Long Range Improvement Program, August 1992, Essential Cooling Water System Long h Range Improvement Program, May 1992, and Long Range Plan for Resolution of Cracking in STP Turbine

/ Generator Low Pressure Rotors, May 1992,Irnpell'ABB.

Page IW71 The Liberry Consultmg Group l

l l

RL 5..'

Chapter Four - Engineering G I 1 engineering studies were done for the erosion / corrosion program and the fire protection system."- 1 2 Each of these evaluations was used by HL&P to make important enhancements to plant systems.

3 The studies of the diesel generators and the essential cooling water system became the bases for 4 making improvements in the reliability of the equipment. The other three evaluations led directly 5 to equipment replacements.

6 A study was also conducted on the procedures used by engineering. This was used to reduce some 7 of the complexity in the engineering processes. "

8 The engineering organization at STP made substantial efforts to ensure that its performance met 9 industry standards and that actions were taken to maintain reliable plant equipment. Despite all to these studies and their universal application in a useful and timely way, the DET report indicated iI that "[t]he licensee failed to make effective use of studies." Although this introductory statement 12 appears to be sweeping in nature, the DET's discussion focused on a single assessment made by 13 PD1 in 1992. The full quotation read as follows:'"

. 14 "The licensee failed to make effective use of studies critical of engineering 15 activities. A substantial contractor review of engineering activities completed in 16 February 1992 resulted in 42 major findings, 20 conclusions, and 11 17 recommendations. The licensee did not track the recommendations because "the 18 recommendations and associated actions were not viewed by the line organizations 19 as having sufficient specific value to warrant attention." Many of the findings and 20 conclusions made in the contractor report were similar to the team findings.

21 Examples include: Duties of system engineers not effectively communicated, 22 frequent priority and modification scope changes, crisis management atmosphere, 23 information database fragmentation and inconsistencies preventing efficient use of 24 resources for engineers, large backlogs, work process inefficiencies, poor to non-25 existent professional training, supervisors not aware of their responsibilities, PED 26 not proactive, system engineer group understaffed, trending not done, focus on 27 short term problems, transition from construction to operations not fully 28 accomplished, and budgeting and tracking of engineering manpower not specific."

Review of the ErosioWCorrosion Program, May 1992 Altran Corp., and Fire Detection System Replacement Assessment, May 1993, Hurst Engineering.

I q >

Policies and Procedures Assessment, February 1992, T. Martin & Associates.

\M '"

DET Report, p. 33.

Page1%72 The Liberty Consulting Group

RLS-:

Chapter Four- Engineering s 1 As indicated earlier in subsection VI.A. an independent consultant was performing a periodic 2

quality assurance oversight function in connection with the installation of a new information 3

sy stem software system. This firm had documented its findings and recommendations in a series a

of reports by the time the DET had completed its evaluation. As a supplement to this work. which 5

emphasized software development, a study was performed by another consulting firm regarding 6

the use of personal computers.'" This second study resulted in the purchase of personal computers 7

for system engineers later in 1992 and eventually for all technical support personnel in 1993.

8 The actions taken by engineering are summarized in subsection VI.E. Each of the 9

recommendations made in the PDI report had been assigned to individuals within engineering, a 10 management evaluation had been conducted, and appropriate actions had been taken. This follow-11 up activity was documented in a memorandum from the Vice President, Engineering, and was 12 shared with the DET.'" Although at the time that the memorandum was written only one action 13 item was still being formally tracked, it is apparent that the recommendations made by the la consulting firm had been followed closely. Moreover, substantial and visible actions had been 15 taken in response to the study. The various subjects listed in this DET statement are addressed 16 elsewhere in this report (e.g., system engineering, information systems, workload, training, and 17 staffing). In each case it was demonstrated that HL&P had indeed been effective not only in ON is managing these various activities but also in applying the results ofindependent evaluations.

19 H. Conclusions  :

20 HL&P gave considerable attention to managing the engineering workload. Emphasis was properly 21 placed on priority work items, most of which were being successfully completed. Training of 22 engineering personnel also received an appropriate level of support and attention from STP 23 management. Liberty found that training had been given substantial emphasis. .

24 HL&P has established a long and enviable history of establishing and implementing useful 25 improvement plans. Plans that have involved engineering include an Operational improvement 26 Plan initiated in 1990, Master Operating Plans developed annually, a Long Range Improvemmt 27 Program conducted in 1992, and an Engineering Program Strategic Plan prepared in March 1993.  ;

e Computer Replacement Long Range improvement Plan for STP, report by Impell/ABB, July 1992.

Analysis of Engineering Activities Performance Data Inc., February 14,1992, Memorandum. S.L. Rosen to Distnbution, June 16,1992.

Page IW73 The Liberty Consulting Group

o PL5 .'

Chapter Four - Engineering

\

i HL&P continually evaluated the benefits being derived from these programs and extended.

J  : revised, or terminated them as appropriate. These programs were initiated, assessed. and 3 implemented in a reasonable way.

.: Proposed modifications were also scrutinized in a reasonable and business-like fashion. Those 5 activities that directly supported safety and operational objectives received the proper amount of 6 management support and attention. Although the DET would have preferred a more swift 7 installation of more modifications, it did acknowledge that major modification packages were 8 "well written and comprehensive." In short, HL&P managed its modification program in a l 9 reasonable manner.

10 Engineering at STP was very conscious of the importance of having independent evaluations made 1i to help enhance its performance and internal processes and to provide technical assistance in 12 properly addressing equipment performance and reliability issues. In the two-year period prior to 13 the time of the DET's investigation,15 individual studies were performed for or including 14 engineering. HL&P carefully evaluated the results presented in each report and implemented is beneficial actions on the basis of the insights and technical advice that were offered.

f*^

(N]. 16 Liberty concluded that the management of STP had provided sound and consistent support for 17 engineering activities, including well rounded training and business based criteria for managing is workload, improvement programs, and modifications.

O Page1W74 The Liberty Consuhmg Group

i RLS.:

Chapter Four - Engineering O

W i VII. Configuration Control 2 A. Background and Perspective 3 The DET report included a discussion of configuration control practices at STP and noted four a specific instances in which practices "could adversely affect plant operations."'" The report stated:

5 " Configuration control weaknesses which adversely affected safety-related plant 6 equipment. Were noted in several instances, such as molded case circuit breakers, 7 SDGs, and environmental qualification of MOVs. In other instances, such as .

8 vendor drawings, the team observed weaknesses in configuration control that, if 9 left uncorrected, could adversely affect plant operations. Ineffective management to oversight and direction, including insufficient resources, were significant ii contributors to these weaknesses."

12 The four examples provided by the DET were:"'

13 o An electrical setpoint index for MCCBs (molded-case circuit breakers)"was not 14 properly understood or implemented in the field, resulting in operability concems."

\

15 o Replacement parts for the diesel generators were installed without the benefit of the 16 latest requirements provided by the vendor in its service bulletins.

17 o "The licensee did not maintain the environmental qualification of valve actuator is motors in containment by installing "T" drains as required by design."

19 o Many amendments to vendor drawings remained unincorporated during the DET 20 evaluation.

t 21 Configuration control is one of the four major elements of configuration management fCM).

22 Configuration management is a set of techniques used to ensure that the physical plant 23 configuration agrees precisely with the design documentation and that the design documents are 24 accurate and retrievable. Configuration management consists of configuration identification. .

'l DET Repon, p. 33. j d "'

DET Repon, pp. 33 34.

The Liberty Consuhing Group

?L %.'

Chapter four - Engineering O

i configuration control (often referred to as design control). document control. and contiguration

s critication.

3 Contiguration identification is defining the scope of the configuration management program and a establishing a system that uniquely identifies each configuration item. Typically. the scope of a 5 CM program includes all safety-related equipment plus numerous other systems that are imponant 6 to reliable plant operation. Configuration items include systems, structures, components, parts.

7 process software, design documents, drawings, simulators, training materials, and other equipment 8 and information required to completely define the plant and its design basis. One of the most 9 prominent configuration items is the set of design basis documents, which is discussed later in this 10 section.

Ii Configuration control is the process used to (1) determine whether to change any configuration 12 item, (2) formally approve any proposed changes, (3) ensure that physical plant changes are made 13 in accordance with approved change documents, and (4) ensure that design basis documents are 14 promptly updated to accurately reflect any physical changes.

15 Document control is the system used to store all design basis documents, to maintain these 16 documents in a legible status, and to retrieve these documents promptly and accurately.

17 Configuration verification is the process used to periodically confirm the precise correspondence 18 among all configuration items.

19 Design basis documents provide system descriptions, including all functional and physical 20 requirements of the system and its components and parts. These documents include all regulatory 21 and accepted, industry-standard requirements as well as site- and utility-specific requirements.

22 They typically include vendor requirements and include or refer to all necessary drawings. Each 23 drawing is considered a unique configuration item.

24 Design basis documents (DBD) are necessary to determine whether and how to make plant 25 modifications. Any change made to plant equipment must adhere to the requirements set forth in 26 these DBDs, must be compatible with any interfacing equipment and processes, and must fit within 27 the space available while meeting these requirements. It would be neither prudent nor safe to 28 proceed with a safety-related modification until all pertinent DBDs had been consulted and a T) 29 determination made that there was compliance with all necessary requirements.

PageIW76 The Liberty Consulting Group

RLS.:

Chapter Four - Engineering j

D l The conRguration management program at STP is comprehensive and well controlled. This CN1 2 program was established by a contractor in 1988 and was upgraded in 1990.": The program.

3 known as the Design Basis Document Program, was founded on four specific goals:(1) to address a regulatory requirements (set forth by the NRC) for quality and completeness. (2) to provide an 5 index for the DBDs. (3) to establish a Chi baseline, and (4) to provide convenient and reliable 6 references to support plant maintenance and operation. The program also incorporates a long list 7 of " musts." including system descriptions, design inputs, calculations and accident analyses.

8 operator actions. regulatory requirements (such as Technical Specification bases and licensing 9 commitments), modification histories, and system interaction considerations (such as seismic, EQ, 10 and fire protection requirements). The program also incorporates STP's Document Tracking and ii Distribution System, which is managed by Document Control. As of April 2,1993, all but 5 of the 12 95 planned DBDs had been prepared, approved, and issued."2 13 ne Chi program at STP incorporates the four basic elements of configuration management to the 14 fullest extent and has been operational since the two units went on line in 1988 and 1989.

15 The NRC has consistently recognized the strength of STP's Chi program. The NRC took special t 16 note of how effectively the design basis was being maintained during an inspection conducted in 17 the spring of 1991.* In an inspection conducted nearly one year later, the NRC made an especially 18 comprehensive evaluation of the Chi program and of the DBDs in particular."5 The NRC 19 concluded that "the licensee had established a very comprehensive STP Design Basis Document 20 (DBD) Program. .(that included] selected nonsafety-related systems."* The NRC made favorable 21 comments on various aspects of the program, including plant modifications, design documents, 22 and drawings. It stated:"7 Design Basis Document Program Plan, Rev. 2, March 26,1990, Impell Corp., DET response item u3104.

Design Basis Document Program (Table of Contents), April 2,1993 DET response item #3122.

NRC Inspection Report No. 91-05, ST-AE-HL-92804, July 18,1991. I NRC Inspection Report No. 92-04, ST AE HL-93023, March 30,1992, w NRC Inspection Report No. 92-04. ST-AE-HL-93023, March 30,1992, p. 24. I NRC Inspection Report No. 92 04, ST-AE HL-93023, March 30,1992, p. 2 and p.14.

Page IV-77 The Liberry Consulting Group

l 23 . ,,-

Chapter Four - Engineering

/-

(*) i 2

"The modi 6 cation packages resiewed were well wrinen and complete.

Walkdowns indicated that the hardware changes were consistent with the design 3 packages. The new design basis documents were viewed as reliable and a complete design aids. . . The inspectors found the description of the design basis 5 documents provided by the licensee indicated that they represent very 6 comprehensive documents."

7 Regarding the configuration verification function, the report stated: "

8 "The design engineering quality engineering group performs intemal self-9 assessments of such areas as configuration drawing control associated with 10 modification packages (MDP) and engineering change notice packages.. The 1i engineering assessment activities at STP are considered a strength."

12 Several months later the NRC conducted a special operational performance inspection of the 13 service water system and cited the application of the CM program as one of several strengths at la STP. The inspection repon said that "[t]he team noted several strengths, including. .the design is basis calculations."'" Specifically, the report also said that "[t]he team performed detailed 16 walkdowns in both units to verify that th'e facility drawings reflected the as built configuration of 17 the system. The P&lDs appeared to be accurate and complete."

18 The resident inspector's report that covered most of the period during which the DET was 19 conducting its evaluation also commented favorably on the CM program at STP. It stated:*

20 "Three systems were walked down during the inspection period....The correct 21 alignment of the systems (Unit I fuel handling building ventilation, Unit 2 essential 22 cooling water, and spent fuel pool cooling and cleanup) indicated that the plant 23 operations department was maintaining good configuration management control 24 of the safety related systems, even with both units shut down."

25 Drawings are an important part of the design documentation and require close management 26 control. STP maintained an especially comprehensive and strong program for maintaining control 27 over and the accuracy of its key drawings. Key drawings, which are formally classified as 28 con' trolled design basis drawings, have been designated by STP as including P&lDs and one-line NRC Inspection Repon No. 92-04, ST- AE-HL-93023, March 30,1992, p. 21.

• NRC Inspection Repon No.92-201, ST-AE-HL-93166, August 24,1992.

s

• NRC Inspection Repon No. 93-15, ST-AE-HL-93464, July 6,1993, p. 2.

PageIW78 The Liberry Comultmg Group

RLS.'

Chapter Four - Engineering V i drawings. which is typical of industry practice as well as control logie diagrams. elementary

drawings, and lighting drawings, for a total of about 4.700. These drawings must be updated and 3 approved within 15 working days of a posted amendment and must be distributed to each unit's 4 control room. each technical support center, and the emergency operations facility. As of mid-5 April 1993. only 20 key drawings were in the updating process.

6 B. DET Examples- Configuration Control 7

l. Molded Case Circuit Breakers fMCCB) 8 As mentioned earlier, the DET report indicated that a setpoint index for MCCBs had not been 9 properly implemented and that this had resulted in operability concerns. The report went on to say to that ten of these breakers were associated with safety-related MOVs. The entire quotation from the 11 DET report as well as a full discussion of the actual situation involving the MCCBs is provided 12 in the chapter on maintenance. Although the DET expressed its concerns in the context that.

13 " safety-related plant equipment [was] adversely affected," an investigation of the matter showed 14 there was no safety significance to the physical problem with the breakers. Also, no equipment was 0

( 15 rendered inoperable because of this matter.

16 The issue raised by the DET was not so much configuration control as it was the degree of 17 accuracy contained in the work procedures. HL&P discovered the weakness in the work 18 instructions, determined its significance, and performed timely corrective actions, which included 19 resetting the breakers, clarifying the procedures, and conducting training.

20 2. Installation of SDG Rocker Arms 21 The second example provided in the DET report of possible weaknesses in the CM program at STP 22 was a set of two situations related to the installation of replacement rocker arms in the standby 23 diesel generators. One item had to do with incorporating revised installation procedures from a 24 vendor bulletin into the pertinent work instructions. The second item was related to HL&P's 25 reliance on manual retrieval of service requests to determine where new rocker arms had been 26 installed. The DET report stated:*

DET Repon, p. 34.

Page1%79 j The Liberty Consultmg Group

p:3.;

Chapter Four - Engineering i "When installing SDG rocker arms with a modified design, the licensee failed to M 2 include specific Cooper-Bessemer service bulletin requirements for torquing and 3 installing the modified parts, which could have caused the replaced rocker arms to 4 function improperly. Once alerted to the bulletin requirements, installation of the 5 rocker arms was still not completed correctly, i.e.. the requirement to replace both 6 the intake and exhaust rocker arms as a set was not accomplished. The licensee also 7 had to resort to hand searches of service requests to locate where the modified 8 rocker arms were installed."

9 The actual history of the installation of these rocker arms is presented in detail in subsection IV.B.

10 This particular situation was not strictly a configuration control issue because the information i

11 received from the vendor in October 1989 regarding new installation procedures was properly 12 incorporated into the vendor manual on December 18,1989 along with the required design change 13 notice.2 The difficulty occurred because this new information was not properly transferred from 14 the vendor manual to the work instructions. No reasonable configuration control system will 15 prevent a mistake of this type.

I 16 Although it is true that not all inlet and exhaust rocker arms were installed as a set, this was not 17 a requirement imposed by the vendor (or by HL&P). As indicated to the DET in April 1993, "[t]he 18 CB [ Cooper Bessemer, the manufacturer] Engineering Manager indicated that having new style 19 rocker arms on intake valves and old style assemblies on exhaust valves of the same cylinder (or 20 vice versa) was not a problem."2" 2t HL&P identified a potential installation problem regarding torquing of the retainers on the rocker 22 arm assemblies on April 2,1993.2 ' It took immediate action to correct the situation on the six 23 assemblies that had been installed as of that time, and no operability problems (or damage) were 24 experienced.2" There was a document control weakness, however, in not being able to readily 25 retrieve the appropriate records regarding the replacement rocker arms.206 2*2 DET response item #3137, pp.197-199.

2'2 DET response item #3310.

2* DET response item #3149 001.

2" DET Response item #3310.

2*

DET Response item #3149-001.

Page IV-80 The Liberty Consulung Group

PLS .'

Chapter Four - Engineermg i 3. Environmental Qualification of Valve Actuator Motors

A third example cited in the DET report as indicating a possible weakness in the CM program was j 3 related to a failure on STP's part to install a "T" drain in the casing of two motor-operated valves I a in accor6,re with an EQ requirement. (This drain may be necessary to maintain operability of the 5 MOV dunng certain design basis accident conditions. Its presence or absence is not material 6 during normal plant activities.)

7 The DET report stated: 2" i

8 "The licensee did not maintain the environmental qualification of valve actuator l 9 motors in containment by installing "T" drains as required by design. A service to request submitted in November 1990 to install two "T" drains in Unit 2 train B ii residual heat removal suction isolation valve was still open during the evaluation.

1: The team requested the licensee to determine which MOVs did not have installed 13 "T" drains. The licensee found five actuator motors that did not have "T" drains.

14 The engineenng staff evaluated three of the five, concluded that no action was 15 required, and was evaluating corrective actions for the remaining two valve 16 actuator motors."

(

\ 17 The sequence of events that led to not installing the required drain in a timely way is fully 18 described in subsection V.B of the chapter on maintenance. The missing drain was clearly 19 documented. The mistake made was in not recognizing the EQ requirement when preparing the

o service request for installation of the drain. HL&P took comprehensive actions to ensure similar 21 problems did not exist elsewhere. The NRC performed a special inspection of MOVs. SOVs, and 22 transmitters in May and June of 1993 and reported that "[t]he inspector did not identify any 23 discrepancies with any of the components."2" 24 4. Vendor Drawings 25 The final example cited by the DET regarding the CM .mgram at STP was the fact that numerous 26 vendor drawings had amendments that had not been incogarated. The report stated: "

• DET Report, p. 34.

• NRC Inspection Report No. 93-19, ST AE-HL-93483, July 23.1993 p.15.

p

" DET Report, p. 34.

Page IV-81 The Liberry Consultmg Group

PLS .'

Chapter four - Engineering b

h i "The many unincorporated amendments to vendor drawings remained signiticant and cculd impede work planning and execution. As of March 19. 1993.

, approximately 11.150 vendor drawings (approximately 50 percent being safety-a related) had Sne or more outstanding unincorporated amendments."

5 Again. this was not a configuration management issue because the documentation was accurate 6 and readily retrievable. A full discussion of the actual situation with regard to vendor drawings is 7 presented in subsection IV.D. As noted in that discussion, HL&P had initiated several actions to 8 reduce the number of outstanding drawing amendments associated with the most important vendor 9 manuals.:' In addition, there were only 255 drawings that were potentially safety-related that had 10 six or more outstanding amendments. It was these drawings that were receiving priority attention 11 in early 1993.:"

12 HL&P fully understood the situation with regard to vendor drawings, had reasonably put the 13 situation into proper perspective, and had taken action to update the most important documents on 14 a reasonable time schedule. The situation was neither directly safety significant nor an indication is of any weakness in the CM program.

s 16 C. CODClusiODS 17 A comprehensive configuration management program had been established at STP when the first 18 unit entered commercial sersice. This program has been enhanced and maintained as a critical and i

19 useful tool for engineering and the modification process. By early 1993 a nearly complete library 20 of design basis documents and associated drawings had been established.

l 21 The success of this CM program received a consistently high level of praise from the NRC.

22 Numerous NRC inspection reports commented in favorable terms on various aspects of the CM 23 program at STP, including reports issued in July 1991, March 1992, August 1992, and July 1993. i In the last of these reports, the subject evaluation was conducted in parallel with the DET's  !

24 25 assessment.

l l

l

• Master Operating Plan - 1993 February 18,1993 RMS-1 and MNT-1. ,

\ 2" DET response item a3055-001.

Page IV-82 The Liberty Consulting Group

RLS .'

Chapter Four - Engineering f%

f \ t

\ / The DET report sets forth four examples of situations in which it found configuration control to

have been less than adequate. A careful analysis of these four situations revealed that none of them 3

showed unreasonable management decisions made or actions taken in the CM program at STP. In a

two cases. related to circuit breakers and replacement parts for standby diesel generators. the 5

proper information was not completely transferred from the design documents (in this case vendor 6

manuals) into detailed work instructions. Similarly, in a situation involving an appendage to some 7

valve bodies. the required instruction was not included in the service request. The fourth concem 8

was associated with outstanding amendments on numerous vendor drawings. HL&P had 9

previously initiated actions to reduce the number of unincorporated amendments. Most to importantly, howeve., this situation was not a configuration control problem because required i1 documentatim. a - .arate and readily retrievable.

12 The configuratici management program at STP was comprehensive and had proven itself to be 13 effective. In addi. n, it had been consistently recognized as successful.

U(

Page IV-83 The Lsberty Consultsng Group

_ _ _ - _ _ . _ _ , _ . _ _ _ - ~

RL%.'

Chapter Four - Engineering i VIII. Essential Chilled Water System 2 A. Introduction 3 The DET performed an in-depth assessment of the Essential Chilled Water System (CHsystem>

a to " gain insight into the licensee performance of activities such as maintenance, testing, operation 5 and design control."2'2 The DET concluded:

6 " Functional and programmatic weaknesses were observed in the design, testing, 7 modification and maintenance of the system, that if uncorrected. could adversely 8 affect the operability of the system. The ability of the CH system to function for 9 extended periods, during a design basis accident under low heat load conditions to was never demonstrated, either by testing the system at various design basis 1i accident heat loads, or by engineering analysis."2" 12 By way ofillustrating these general conclusions, the DET report presented seven examples, each 13 of which are discussed below. The examples were:

la o analysis of specific low heat load conditions is o testing of this same condition 16 o contamination problems 17 o cancelled installation of a vibration probe is o use of a temporary modification 19 o missed post-maintenance test <

20 o maintenance training.

21 B. DET Examples - Essential Chilled Water System 22 The first example cited in the DET report is associated with a type of design basis accident that i 23 is postulated to occur during extremely cold weather conditions if plant operators were to permit 24 continued operation of all three trains of the chilled water system, even though one train would be j 25 more than adequate to fulfill the intended cooling function. The report stated:

2'2 DET Report, p. 2.

2" DET Repon, pp. 34-35.

Page1%84 The Liberty Consuking Group

RLS-:

Chapter Four - Engineering i "The licensee did not complete an analysis for the CH system under low heat load

conditions. If an accident occurred during cold weather and all chillers operate, the 3 chillers would be under-loaded. causing surging and failure, resulting in loss of CH 4 cooling of safety-related equipment. A 1989 Safety System Functional Assessment 5 (SSFA) of the essential cooling water (ECW) sy stem conducted by the licensee 6 identified concems with chiller operation for accidents during cold weather. The 7 licensee performed calculations for some concerns of the SSFA, but failed to 8

~

address overcooling of the CH system during accidents when more than two trains 9 of the chilled water system operate, as in response to a safety injection or loss-of-10 offsite-power signal which start all three-trains, or when heat loads would be lower 11 than those assumed for worse case high heat loads accident conditions. The 12 findings in a March 26,1993, QA report of a CH system SSFA included continuing 13 concems about compressor low heat loads. Although the planned action to resolve 14 the findings was adequate, a number ofissues, such as verification of field setting is of compressor suction pressure trip setpoint, were not adequately addressed by the 16 licensee. The licensee made a commitment to the team to evaluate under-loading 17 of chillers during accident conditions."2" Is There are several important elements contained in the DET's statement that illustrate the initiatives 19 taken by HL&P to ensure the operability of the essential chilled water system. On two occasions.

( 20 about four years apart. HL&P conducted a safety system functional assessment (SSFA) of this

\ 2i system. Traditionally, this type of evaluation was performed by the NRC, but in recent years many 22 utilities, including HL&P, started performing these assessments themselves. The DET 23 acknowledged that "the planned action to resolve the [SSFA] findings was adequate." In fact, the 24 course of action had been developed prior to the DET's evaluation and had been distributed a few 25 days later in a memorandum from QA that stated: "On March 23,1993, representatives from QA, 26 DED, PED, ISEG, Maintenance and Plant Operations met to review the results (of the SSFA] and 27 devise action plans and duc dates."2n in a supplement to this memorandum issued ten days later, 28 QA noted that the overall conclusion reached during this SSFA had been omitted from the earher i 29 memorandum. That conclusion was:2a 2

DET Report, p. 35.

• Memorandum, R. J. Rehkugler to Distribution," Quality .*"nnce (QA) Report of Essential Chilled Water g (CH) System Safety System Functional Assessment (SSFA)," March 26,1993.

Memorandum, R. J. Rehkugler to Distribution. April 5,1993.

Page IV-85 The bberty Comultmg Group

i e

M S.:

Chapter Four - Engineenng D

i]

2 i "Some functional and programmatic weaknesses were observed in the design.

maintenance and operation of the Essential Chilled Water System that, if left 3 uncorrected. could affect the ability of the system to perform as expected on 4 demand. However, in spite of these weaknesses, the team has concluded that the 5 Essential Chilled Water System, in its present configuration and 6 condition... Mill perform its safety functions."(Emphasis added.)

7 The,DET essentially used the first sentence of HL&P's conclusion as its overall conclusion.

8 Interestingly, however, the DET did not include the second sentence, which recorded the ability 9 of the system to fulfill its safety functions without being modified.

i 10 The SSFA conducted by HL&P was particularly comprehensive. The SSFA included four goals:*

11 o Confirm that the " system can perform its. . design basis functions" 12 o Confirm the " adequacy of operations, maintenance, and testing procedures" 13 o " Confirm the adequacy of maintenance and testing" on the system

. 14 o " Confirm the adequacy of... corrective actions in response to station and industry I5 experience."

C i

( 16 HL&P's SSFA included numerous inteniews and an extensive review of pertinent documents.

17 including 86 design documents,14 procedures, 311 documents related to maintenance and is suneillance activities, and 142 other documents. The assessment resulted in six findings that 19 related to operational inconveniences but not equipment operability. These were: 1) reliable 20 operation under low load conditions during normal operation,2) lack of upgraded emergency 21 procedures, 3) unincorporated preventive measures, 4) personnel interfaces, 5) configuration 22 control, and 6) dependence on vendor assistarce. The action plan addressed these findings in 23 considerable detail, spelling out 23 individual action items.* i l

24 The first two findings were related in that the emergency procedures did not explicitly state the )

1 25 operator actions required during extremely cold weather conditions (i.e., the actions needed to 26 secure at least one train if all three are actuated), as described in the design basis document. By 27 expressing this matter the way it did, the DET was establishing unnecessarily high standards of 28 performance. While the need to upgrade the procedure was clear, the DET focused on

• Attachment to memorandum, R. J. Rehkugler to Distribution. March 26.1993.

[ *

( Attachment to memorandum, R. J. Rehkugler to Distnbution, March 26,1993.

Page IV 86 The Liberty Comultmg Group

l RLS.:

Chapter four - Engineering i demonstrating whether the system could operate during extreme low temperature conditions 2 without operator action.

3 The DET report said that the analysis for low heat load conditions was not complete. Although 4 HL&P commined to perform additional calculations to satisfy the DET's questions, the conditions 5 imposed by the DET including the use of a 100-year low temperature, were highly unusual. HL&P 6 told-the DET that "the postulation of a large break loss-of-coolant accident (LOCA) together with 7 a 100-year extreme temperature was beyond normal licensing requirements." Because of 8 successful experience in operating the system at low temperatures and calculations made with 9 somes hat less restrictive conditions, HL&P concluded:"We have no doubts that the chilled water 10 system can be successfully operated during a LOCA or other design basis accident during cold ii weather."2

1 The bases for HL&P's confidence in the system's capabilities included:

13 o Demonstration that the " chillers are capable of continuous operatie down to 10%

14 of rated load."

. 15 o "The essential chillers have operated successfully with... inlet temperatures as low

\

16 as 38 F, which occurred during a three-day period in late 1989."

17 o " Operation of the chiller with excessively low. . load will cause the compressor to 18 trip on one of'several different protective interlocks, "The compressor will be 19 restarted automatically when the condition clears in many of these cases and can 20 be manually... restarted after the condition clears in the remaining cases."

21 o " Operation with... inlet temperatures between 45 F and 54 F has been fairly 22 routine. Temperatures below 45" are very unusual."220 23 o The design calculation showed that "the chillers can operate with a condenser water 24 temperature as low as 35 F."22' 25 in view of these analyses and experierses, Liberty concluded that HL&P's consideration oflow 26 heat load conditions was both reasonable and " complete." Furthermore, the system has a number DET response item #3146.

• DET response item #3146.

• Attachment to memorandum, R. J. Rehkugler to Distribution, March 26,1993.

The f.iberry Consultmg Group I

l

l

?is.:

Chapter four - Engineering r\

(V ) I 2

of protective interlocks to prevent failure. The DET sets forth no basis for its conclusion that " loss of CH cooling" would occur if these extreme conditions were to exist.

i l

l l

3 It was not clear what the DET meant when it said that "[t]he licensee performed calculations for a some concems of the (1989) SSFA."(Emphasis added.) As far as HL&P could ascertain, all SSFA 5 findings were satisfactorily addressed and closed. In closing this finding from the SSFA. HL&P 6 used.a suitably conservative set of assumptions and never intended to address the more extreme 7 conditions postulated by the DET four years later. Calculations performed as a result of 1989 8 SSFA showed that post-design basis accident operation at temperatures below 54 F was 9 satisfactory with manual control and that the SSFA concem was resolved.::: The DET's statement to about " continuing concems about compressor low heat loads" reflected the 1993 SSFA finding ii that the emergency procedures did not delineate the expected operator action under these extreme 12 conditions. 22 13 The DET report's second example relating to essential chillers dealt with testing.

14 "Preoperational, surveillance, arid post-maintenance testing (PMT) performed on 15 the CH system did not demonstrate that the system would be operable for extended i 16 periods under design basis heat load conditions. The piping design configuration

() 17 did not allow the CH system to be tested with heat loads representative of those 18 anticipated during accident conditions."

19 This was certainly an accurate statement but could not be used by the licensee in making 20 enhancements to the system or its operation. The system was not designed to facilitate the 21 enactment of design basis loading. Very few, if any, systems are designed to be tested for 22 " extended periods" under full-load accident conditions. As is the case for all safety systems.

23 however, design basis calculations are made to demonstrate adequate system capability. In many 24 other cases the NRC has recognized the fact that systems and equipment are not tested under actual 25 design basis accident conditions. The design basis document stated that "[a]fter a DB A, such as 26 a Loss of Coolant Accident (LOCA), all three trains of CH pumps and essential chillers stan 27 automatically "::' The operation of three trains in the limiting accident condition was recognized 28 and addressed by HL&P.

22 Operational Readiness Closure Package #12, Essential Chiller Relblity, November 1,1993, Executiu Summary.

• Attachment to memorandum, R. J. Rehkugler to Distribution. March 26,1993, item A.I.

( - '

Attachment to memorandum, R. J. Rehkugler to Distribution, March 26.1993, item B.1.

Page IV 88 The Liberty Consuhmg Group  :

l l

RLS.:

Chapter Four - Engineering p

i The third example given in the DET report was:

2 " Compressor refrigerant and oil contamination was a long term problem that 3 significantly affected reliability. The vendor proposed installing a proven a refrigerant clean-up kit that would allow uninterrupted chiller operation. Although 5 the modification was approved in September 1991 for installation in 1992, its 6 installation date was deferred to October 1994 for Unit I and April 1995 for Unit 7

2."

8 The installation of a refrigerant clean-up kit on each chiller was deferred, as indicated in this 9 statement. The deferral was part of an ongoing process of reviewing all modifications to ensure 10 that a reasonable balance was achieved among budget requirements, safety requirements, and 11 operability needs. HL&P decided that an increased frequency of preventive and corrective 12 maintenance actions could temporarily substitute for installing the kit. Because of the three-train 13 redundancy in the system, this decision had no direct effect on safety o: aperability. This was just la 1 of 24 modifications that were deferred.225 In addition,11 other modifications associated with the 15 chillers were cancelled in June 1990 and 4 more at various other times. " Additional discussion 16 of the budgeting process is provided in the chapter on management and organization.

f3 (j)'

i 17 The fourth example put forth by the DET was also associated with a modification:

18 "In 1993, after further evaluation and repeated attempts at installation, the licensee 19 cancelled plans to install a proximate vibration probe assembly recommended by 20 a vendor in 1988 to detect high speed thrust bearing displacement and an automatic 21 compressor trip function for the 300-ton compressors to prevent catastrophic 22 failure."

23 Elis proposed modification was one of the 24 hems mentioned earlier.227 The statement in the DET 24 report regarding the prevention of failure overstated the facts, however. This type of probe may 25 prevent consequential failure that could result from whatever caused a high thrust bearing 26 displacement, but it cannot preclude the failure that caused the abnormal behavior itself nor major 27 failure modes due to other types of equipment failure. The decision by HL&P on whether (or

• Operational Readiness Closure Package #12. Tab 2 Att. #2, Essential Chiller Reliability, November 1, 1993, Report.

• DET response item #3155-001.

/

l

• Operational Readiness Closure Package #12, Tab 2 Art. #2, Essential Chiller Reliability, Nos ember I.

k 1993 Report.

Page IV-89 The Lsbery Consultmg Group

n s.:

Chapter Four - Engineering I when) to install such a monitoring device had to be reached on the basis of the relative risk of this particular type of failure compared to other failure modes that can be mitigated or prevented by 3 the installation of other protective or monitoring devices.

1 4 The fifth illustration used in the DET report concemed the use of a temporary modification:

l 5 "In 1989, the licensee implemented a temporary modification to remove an ECW

~

6 valve actuator which automatically controlled flow to the chiller condensers by 7 using an upstream manual valve rather than correcting automatic control system 8 design and material deficiencies."

I 9 This matter may have been an operational inconvenience but was not a safety concem. As with 10 other temporary modifications, the major objective is to make the plant operarianal within i1 established safety and quality requirements, which is the reasonable and prudent action to take.

12 This situation, along with numerous other temporary modifications, was being actively pursued j 13 by HL&P as pan of an overall program to muumize the number of temporary modifications. He' I 14 effectiveness of this program is discussed in subsection II.B.

i

,p 15 This temporary modification was a consideration only during low-temperature operation and was 16 only one of the many findings contained in the 1993 SSFA report. HL&P summarized the situation 17 by saying:

18 "In 1988 temporary modifications established manual control of essential cooling 19 water condensing flow through the essential chiller condensers. The existing I 20 automatic control function was eliminated because of continuing maintenance 21 difficulty with the electro-hydraulic actuators originally installed to regulate 6" and 22 8" butterfly valves. Essential cooling water flow control is necessary to maintain 23 refrigerant Pressure and Temperature within the required operating range when the 24 ECW supply Temperature drops below 54 F."

25 Because this manual control mode was called upon infrequently, it was decided that the situation 26 had a relatively low priority. As further confirmation of the validity of this approach, it was 27 decided in late 1993 to replace this temporary modification with a permanent installation that 28 would continue to rely on manual control.

l l

Page IV-90 The Liberty Consulting Group

RLS.:

Chapter Four - Engineermg r~N.

I The sixth essential chiller example cited by the DET referred to a missed post maintenance test:

2 "After maintenance work was performed on the feeder breaker for essential chiller 3 21C, the chiller was declared operable without PSIT. The following day the chiller a tripped during a routine start attempt because of breaker problems."

5 The Ph1T was overlooked because the inspection of the chiller breaker was completed as a separate 6 work item from the overall senice request, which called for inspection of a large number of 7 breakers provided by the same manufacturer. The senice request properly called for the Ph1T but 8 was not adequately reviewed upon completion of the work to ensure the Ph1T was actually 9 conducted. The oversight was simply a local mistake and not a programmatic weakness, since the 10 work package clearly called for the test. The situation was fully corrected by Afarch 1,1993, prior ii to the DET visit, and included the issuance of separate work packages plus the conduct of job-12 specific training. 2' 13 The last example used by the DET had to do with craft training:

la "The maintenance craft e .,onnel introduced air into the essential chillers and p

is flooded a control panel .ith oil because they did not understand how the chillers 16 function under vacuums. Inadequate training caused poor maintenance work and 17 contributed to degraded performance of the equipment and the lack of availability."

18 This incident was apparently due to inadequate understanding on the part of the craft personnel 19 assigned to the job. HL&P took immediate action to correct the situation.

20 HL&P initiated a series of actions to improve the performance of the essential chilled water 21 system. A meeting with the NRC was requested by HL&P in early 1992 "to discuss HL&P 22 initiatives on the STPEGS Essential Cooling Water (ECW) system."2:' This discussion was 23 subsequently confumed in a detailed letter to the NRC that addressed a large number of initiatives 24 designed to enhance plant performance at STP. Regarding chillers, HL&P stated:

• Lener, HL&P to NRC, ST-HL-AE-4250. November 13,1992.

(

• Lener, HL&P to NRC, ST HL-AE-4012, Febmary 14,1992.

Page IV-91 The Liberry Consultmg Group

e R'5:

Chapter four - Engmeering f}\

(

i 2

"HL&P has initiated action to improve the reliability of the Essential Chillers.

Design requirements are being more accurately determined to allow greater 3 flexibility in operating the chillers with the aim being to minimize chiller cycling.

4 The increased flexibility is expected to allow operation of the chillers closer to 5 optimum capacity and to consequently help reduce the amount of required 6 Corrective maintenance."*

7 HLB;P's Nuclear Safety Review Board (NSRB) also acted in late 1992 to identify areas where 8 chiller performance could be improved. This NSRB initiative was described in a letter to the NRC 9 in early 1993 and noted an effort to raise the awareness of operators of general design criteria and j 10 the safety analysis basis.*

11 C. Conclusions 12 HL&P recognized that the essential chilled water system had a history of nuisance problems and 13 reasonably sought to overcome this situation by increased attention to preventive maintenance and

~

14 carefully selected modifications. The DET asserted that design and maintenance activities were 15 still not sufficient to preclude potential operational problems and proffered seven examples in 16 support of this proposition. While these circumstances may have been of concem to the NRC. they (G) 17 did not reflect unreasonable decisions or actions by STP management.

18 The first two examples were associated with the analysis and testing of the system under extreme 19 cold weather conditions. Although HL&P believed it had adequately analyzed system operation 20 with cold inlet water in an extensive evaluation conducted in 1989, the DET wanted to see 2i calculated results for a more extreme set of assumptions. HL&P agreed to perform the analysis but 22 noted that the postulated conditions went beyond normal licensing requirements. In addition, 23 HL&P pointed out a number of protective interlocks that would p;eclude system damage under 24 extreme cold weather conditions.

25 The DET insisted upon a design basis test that could not be performed because the chiller system 26 was not designed for this type of performance test. Few, if any, systems are designed to 27 accommodate a design basis condition for " extended periods," as suggested in the DET report. The 28 analysis corresponding to the test had been satisfactorily performed.

• Letter, HL&P to NRC, ST HL-AE-4083, May 1,1992.

(Oj

• Letter, HL&P to NRC, ST-HL AE-4342. March 30,1993.

Page IV-92 The Liberty Consulting Group

RLS-:

Chapter Four - Engineering i The third and fourth examples were related to proposed modifications that had not been installed V  : at the time of the DET assessment. In one case a modification had been deferred and the frequency 3 of pres entive and corrective maintenance activities was increased to adequately compensate for

.: its deferral. In the other case the installation of a vibration probe had been cancelled. This sensor 5 would have been useful in helping prevent a catastrophic failure in the event of a low probability l 6 event. In both cases HL&P took deliberate action to evaluate the risks associated with not 7 installing the modifications and the potential benefits of completing them. These results were then 8 incorporated into an overall evaluation of which modifications would be pursued in the next year 9 as part of the overall budgeting process. This approach ws both reasonable and necessary. l 10 The fifth example was a concem about the use of a temporary modification. This interim fix had iI limited use only during cold weather and so was a minor issue. Part of the DET's concem was that i 12 the fix required manual control. The permanent modification, since installed, retained this manual

( 13 feature because it was deemed to be sdtisfactory.

i l

1.s The last two examples had to do with an overlooked test and an improper maintenance procedure.

15 Both were properly corrected by HL&P prior to the DET evaluation.

A kd 16 Liberty found that HL&P had acted reasonably in addressing each of the matters identified by the 17 DET. In fact, in each case the required actions and decisions had been completed prior to the 18 DET's visit to STP. Although one temporary modification was still in place, the package for the 19 permanent modification had already been prepared.

(3 i I b

i

\,j l PageIW93 The Liberty Consulting Group

i RLS-Chapter Four - Engmeering O

d i IX. Fire Protection Systems 2 The fire protection system at STP consists of four parts: fire detection equipment, a data 3 acquisition system. a fire suppression system, and fire barriers. The DET report stated that "[t]he 4 licensee did not resolve several chronic fire protection issues in a timely manner." There were 5 four areas about which the DET expressed concern: penetration seals that had shrunk and cracked.

6 a large backlog of service requests, a fire alarm system that emitted frequent spurious signals. and 7 the control of transient combustibles. The DET concluded that "[m]anagement did not adequately 8 oversee and direct the efforts to resolve these issues in a timely manner."233 9 Penetration seals are used to ensure that the fire barriers are complete in places where, for example.

10 cables penetrate a wall. The DET gave the following account of the issue in this area:*

11 " Excessive shrinkage and resultant cracks of Hydrosil type penetration seals .

12 allowed free air to pass between fire areas and raised questions of structural 13 integrity, making the seals ineffective fire barriers. The problem was previously la identified in 1990 and was thought to have been corrected after a 100 percent

p. 15 survey in 1991-92 and subsequent repairs / rework. The cracking was again

() 16 17 identified in March 1993. The investigation of the problem was scheduled to be completed by May 31,1993."

18 Investigation of this matter showed that HL&P had taken appropriate action when the deficiency 19 was first identified. The corrective measures and 100 percent survey gave reasonable assurance 20 that the matter had been resolved. HL&P discovered a recurrence of the problem in March 1993 21 and took immediate and appropriate action to correct it.

22 As the material used fqr penetration seals cured, it also shrank, leaving some hairlin.e cracks and 23 a few small clearances around the edges. The vendor had never tested for the long term condition 24 of these seals. HL&P performed detailed inspections of these seals every 18 months and repeatedly 25 identified deficiencies in early 1990, late 1991, and early 1993. At the time of the first discovery 26 of shrinkage in 1990, no approved repair method had been developed, even though it was soon 27 recognized as an industry-wide problem. By May 1990 a repair method had been devised and 1

" DET Reporz, p. ix.

g DET Report. p. 36.

• DET Repon, p. 36.

Page IV 94 The Lsberty Consuhmg Group i

1 RL5-Chapter Four - Engineenng i repairs were initiated in July 1990. In most cases the repair consisted ofinjecting an epoxy into 2 the cracks using a hypodermic needle. In other cases a long-lasting caulk was injected.

3 As a result of the latest findings in March 1993, even more detailed inspections were conducted.

4 These supplemental obsen ations were completed in June 1993. and needed repairs were installed 5 by October 1993, except two that were done in December 1993.

6 Concerning the seemingly large number of senice requests on the fire protection system, the DET 7 report stated:233 8 "At the time of the evaluation, the licensee had a large backlog of 361 open SRs for 9 fire pro!0ction systems (164 for Unit I,122 for Unit 2, and 75 common). Included to are 249 SRs associated with fire suppression system problems, the majority being ii valve packing leaks; and 112 SRs associated with fire detection systems,30 percent 12 caused by trouble alarms because of dirty fire detectors. In addition to the backlog 13 of 361 open SRs for fire protection systems, the licensee had another backlog of 68 14 SRs for Unit 1 and 163 for Unit 2, pertaining to fire barrier breaches including is Thermo-Lag installations and Hydrosil penetration seals. The large backlog g 16 indicated that the reliability of fire protection systems was questionable."

17 HL&P formed a special review team in early 1993 to assess whether these open senice requests 18 posed a threat to continued plant operation.236 The review included all open Unit I and common 19 service requests, system health reports, and system walkdowns. The evaluation included all aspects of the fire protection system, including the water and halon fire suppression systems, the fire 20 21 detection system, and the fire barriers. Three criteria were established to assess the potential effect 22 of these individual conditions on pir.nt operation. These criteria were applied to each senice 23 request and posed the questions of whether each request had to be resolved in order to (1) maintain 24 reliable unit operation for at least 18 months, (2) address an open SPR, LER, or other corrective 25 action, or (3) maintain the ability of the fire protection system to perform during unit operation.

26 It was concluded that none of the senice requests had to be satisfied to meet these c'riteria for the 27 Unit I and common fire suppression systems, fire detection systems, or fire barriers. Unit 2 was 28 not evaluated. These results were given to the DET.

2" DET Report, p. 36.

2

k DET response item #2289-001.

Page IV-95 The Liberty Consulting cira;

PLS.'

Chapter Four - Engineering i The DET expressed concem about the alarm handling system and fire detectors. The report said: "

2 "Be Pyrotronics fire protection computer system, which monitors fires in various 3 plant areas and alarms in the control room, was unreliable with numerous chronic a problems, including defective detectors and electronic transmitter boards.

5 Numerous false alarms frequently annunciated (20-30 each day) and control room 6 operators could not quickly ascertain which detector was in alarm status.

7 Replacement parts were not available because the system was obsolete. Although 8 a modification was proposed to replace the system, the modification received low 9 priority, and was not scheduled for installation until 1996. The team raised 10 concerns about the system reliability and the ability of operators to determine if and Ii where a fire existed."

12 Detailed information maintained by HL&P showed that the alarm rate was actually in the range 13 of 5 to 10 per day, not the 20 to 30 range, as estimated by operations personnel and stated by the 14 DET.

15 There are about 1,600 thermal fire detectors in each unit at STP. These detectors experienced a 16 high rate of failure in that they were alarming well below the established temperature setpoints.

17 Initially, HL&P decided to request that the vendor of the detectors assess the cause of the

( 18 failures.* However, due to the delays this assessment would nave caused, HL&P quickly 19 determined that a complete replacement of the detectors was the most reasonable alternative. The 20 problem detectors were all replaced by January 29,1993.

21 The primary causes of unintended alarms initiated by the fire detectors vcere high humidity and the 22 sensitivity of the detectors to moisture. The replacement detectors are photoelectric cells and are 23 water tight. The use of the new detectors has resolved the deficiency.

24 HL&P recognized the reliability problems in the fire protection alarm handling system. It had been 25 studied extensively by both engineering and operations. Modifications were prepared in 1991 to 26 replace the fire protection " system with a more reliable, current technology system."5 HL&P's 27 original plan was to include this replacement work in the 1994 budget. Engineering change notice I

2" DET Repon, p. 36.

2" DET response item #2289-001.

O *

\] DET response item #3147. l Page IW96 The Liberty Consultmg Group

RLS:

Chapter Four- Engineering  !

I packages had been issued to resolve the alann handling problems in the computer system.* The 2 initial phase of this work was under way in early 1993 and has since been completed. The task 3 included work in the control room to better present information to the operators.

4 All discrepancies noted during fire protection system surveys were typically summarized in a 5 memorandum, unless a major problem was observed or timely corrective action was not taken.

6 Since September 1992, STP has followed the practice of writing individual SPRs for each such 7 discrepancy. This change in practice resulted in what appeared to be a substantial backlog of SPRs 8 for the fire protection system by early 1993. The number of new SPRs dropped rapidly from 49 9 and 40 in September and October 1992 to 19 and 16 in January and February 1993. Most of these 10 SPRs were related to oily pads located at leaks and improper storage of liquids and other il combustibles.

12 The DET commented on the existence of transient combustibles.: .

13 "In April 1993, the licensee located significant quantities of transient combustibles la such as wooden tables, waste oil, oil-soaked rags, and miscellaneous combustible g 15 items located throughout the plant. The presence of such large amounts of transient i 16 combustibles was indicative of an inadequate control program."

17 The DET's statement implied a long-standing problem, but the situation was actually a temporary is one. An apparent high number of SPRs was caused by a lowering of the threshold for preparing 19 SPRs. Previously, minor discrepancies noted in fire protection surveys were identified by

, 20 memoranda. In March 1993 a heightened awareness by fire protection surveyors caused SPRs to 21 be written on numerous wooden tables and chairs be'ng i used to support the outage.:':

22 HL&P took a number of initiatives to assess its fire protection systems. The results were 23 documented periodically in system health reports. A report on the fire detection and data 24 acquisition systems issued in mid-1992 indicated generally good performance and concluded that 25 the systems' health was good.2o The outstanding deficiencies were believed not to affect system

DET response item #3147, 2"

DET Report, pp. 36-37.

2

DET response item #3234.

U 2" FA System Health Report, June 18,1992, DET response item #3320. l I

Page IV-97 j The bberty Consultmg Group

l l

• l PLS: l Chapter four - Engineering 1

I I operation. On a rating system of 0 to 10. where zero represents the optimal condition. the system 2 was assessed as 0.04. In a subsequent report six months later. the fire detection (FA) system was 3 obsers ed to be in good shape and to have a "very low failure rate."* The report also noted "no 4 adverse trends." A contractor (Hurst Engineering) had been hired to develop "a design package for 5 replacing the FA system." False alarms due to dirty smoke detectors were being addressed by 6 having a contractor clean and calibrate the equipment.

7 HL&P al.so recognized a potential weakness with spare parts in the fire detection system. In the 8 system health report of June 1992, it was noted that "[t]he stock of spare pans should last for about

9 2 years."* In addition, it was observed that "[t]he computer has no spare parts and its failure 10 would result in the need to post fire watches."

11 The health report on the FA system of December 1992 said:

. 12 "It is expected that this system will remain relatively trouble free over the next .

13 several months. Maintenance has addressed most of the problems during the past 14 outage. Most failures and false alarms are during seasonal changes when the 15 ambient temperature takes a major change."

b)

V 16 Similarly, the fire suppression system was assessed and a health report issued near the end of 17 1992.5 The report identified numerous valve issues and said that the relief valves would be 18 replaced by March 1993 and the isolation valves within six months.

19 HL&P took positive steps to conduct detailed investigations ofits fire detection, data acquisition, 20 and fire suppression systems. When problems were observed, a number of timely corrective 21 actions were instituted, including valve replacements, developing a design for system 22 replacements, and a program for cleaning and calibrating detection equipment. When problems 23 with penetration seals were found in 1990, they were repaired, and when a similar situation 24 recurred in 1993, additional repairs were made. The existence of certain combustibles was duly 25 noted in early 1993, and the situation was promptly corrected.

FA System Health Repon, December 18,1992, DET response item #3320.

FA System Health Report, June 18,1992, DET response item 83320.

/

O t (j  :"

FP System Health Report. December 16,1992, DET response item #3320.

Page IV-98 The Liberty Consultmg Group

RLS .'

Chapter Fise - \tanagement and Organuation i Chapter Five - Management and Organization i

2 1. Introduction and Summary of Findings 3 The DET's evaluation of management and organization resulted in conclusions in the areas of a direction and oversight, support and resource utilization, communications and teamwork.

5 corrective actions, self assessment and quality oversight, and information systems. Except for its 6 statements that HL&P had identified "most of the problems" included in the DET report and that 7 recent management and organizational changes had been positive, the DET discussed very few of 8 the positive aspects of STP's management.

9 Despite the critical nature of the DET report, Liberty's analysis showed that HL&P had made to reasonable decisions and taken reasonable actions in the areas examined by the DET. Two points i1 are worthy of note in this regard. First, the NRC in its role as safety regulator is primarily 12 concerned with outcomes. Thus the NRC is not content that a licensee choose from among*

13 reasonable options. The NRC is concemed with whether the option chosen achieved the desired 14 result. This is a fundamental distinction between the NRC's diagnostic evaluation and a PUC h 15 diagnostic. Second, like all diagnostic evaluations, the DET took a snapshot of the project and identified areas of possible concern and areas in which performance appeared to be deficient or 16 17 could be improved. Here again an NRC diagnostic differs significantly from a PUC diagnostic. In 18 management audits that Liberty has performed for utility commissions, a diagnostic study is 19 sometimes used to identify issues that require further study. In the DET process, however, once 20 issues are identified and preliminary conclusions reached, it becomes necessary for the licensee 21 to either prove those conclusions wrong or proactively address the issues raised. Most licensees 22 choose this second course of action because it tends to resolve matters more quickly and in the 23 long run perhaps more comprehensively. As a result, many of the conclusions reached in the 24 DET's snapshot are never tested.

25 Liberty examined the DET's findings, and the facts that were used as support for them, as it would 26 in the second phase of a management audit. Moreover, Liberty did so to see if the issues raised 27 showed that STP's management had made unreasonable choices or taken unreasonable actions and 28 not to determine if the choices or actions had been " effective," i.e., achieved the desired result. In 29 many instances Liberty found that a more in depth analysis of the facts did not substantiate the 30 DET's original conclusions. For example, the DET said that the STP team reflected a lack of 31 experience at other nuclear facilities; yet the objective evidence showed that experience at other 32 nuclear facilities had been ample.

Page V-1 The Liberty Consultmg Group

RLS .'

Chapter Fne - \tanagement and Organuation i Liberty found that STP's budgeting process and practices were reasonable and realistic. The fact '

2 that there was a wrinen justification for a budget line item was not evidence that management was 3 unreasonable for rejecting thatjustification. In fact, management could be viewed as unreasonable 4 ifit approved every budget request just because it was accompanied by a justification. Nor is it 5 unreasonable to delete a previously approved budget item in the face of changed circumstances.

6 The. overwhelming evidence indicated that staffing decisions had been reasonably made on the 7 basis of adequate study and expen advice. The DET was critical of the use of task forces at STP.

8 Liberty's review of the task forces that were active at the time of the DET's review showed that 9 their use had been a reasonable supplement to, not a substitute for, an adequate permanent to organization. In the area of quality oversight, the great weight of the evidence received by STP's 11 management indicated it had made reasonable decisions and that this function was performing 12 well.

13 The DET used the exact words that were written by HL&P over a year before the DET's visit to 14 describe the status ofinformation systems at STP, However, the DET's snapshot did not capture is what had been done during that year to bring about improvement. Liberty found STP's efforts in 16 that area to have been substantial and reasonable.

D s

\

\

Page V-2 The Liberty Consultsng Group 1

RL5.:

Chapter Fne - stanagement and Organuation f 'N (v) i II. Direction and Oversight 2

The DET concluded that the management direction and oversight at STP had been ineffective. The 3

DET report said that "(s]everal factors significantly decreased management's effectiveness 4

throughout the organization."' In summary form, these " factors" were: (1) insufficient face-to-face 5

communications, (2) over-involvement by senior management (or lack of middle management 6

authority), and (3) lack of commercial nuclear power experience at plants other than STP.

7 Typical of the DET's discussion in this section ofits report was the following: 2 8

" Middle managers often failed to obtain feedback on problems and give consistent 9

direction because they did not interact frequently enough with people in the plant.

10 The reactive mode of the organization, a result of the poor material condition of ll equipment, contributed to managers not spending enough time for face to face 12 communications with people in the plant. Although the licensee initiated the 13 management surveillance program in 1990 in an attempt to increase management's ,

la presence in the plant, the plant staff had not fully accepted this program. The 15 perception by plant personnel was that the managers focused on minor 16 housekeeping items rather than effectively interfacing with personnel and 17 providing one-on-one direction and feedback."

O 18 Other than the management surveillance program, there is very little contained in these remarks 19 by the DET that can be assessed objectively. The DET did not say how it had determined that 20 middle managers had often failed to obtain feedback or how it had determined that interaction with 21 people in the plant had not been frequent enough. It did not say how it had concluded the 22 organization had been " reactive" or how it had determined that the alleged reactivity had been 23 caused by the plant's material condition. The DET reported its perceptions, or reported what it had 24 been told by STP personnel in interviews, without providing the framework or criteria to give these 25 perceptions or reports context or credibility.

26 Management Surveillance Program 27 As to the management surveillance program, the DET portrayed a positive management action as 28 a problem. The document that implements the Senior Management Surveillance Program explains 29 that the program not only provides for senior management observation of work routines and DET Report, p. 38.

8 DET Report, p. 38.

[

i Page b3 '

The Ltberty Consuhmg Group '

PL5 .'

Chapter Fne - $1anagement and Organization r

/ T material condition but also provides visibility for senior management and promotes direct O

I i 2 communications with station personnel. It also directs that "[s]urveillances should also include 3 discussions with plant personnel."3 4 A considerable number of reports were generated by the Senior Management Surveillance 5 Program.' A review of these reports showed that the program addressed much more than minor 6 housekeeping issues. While the reports did address material conditions, including housekeeping 7 matters, it was clear that there had been consistent face-to-face discussions during the plant tours.

8 For example, the first item in the " scope" of the very first Senior Management Surveillance 9 Program report given to the DET was:

to " Spent time in both Control Rooms observing operations, which included the ramp ii up of power in Unit 1. Talked to the operators concerning their perception of how 12 the operations were going and current or potential difficulties."

13 Other examples from the surveillance reports showed that the managers had dealt with important la issues and gotten feedback from plant personnel.

i 15 "An operator in Unit 2 Control Room spent some time explaining the 16 " clearance / surveillance" program he has spent much personal time developing (at 17 work and at home) on a 286 Personal Computer."

18 " Scope ... observations in the Unit 1 Control Room for approximately one hour to 19 monitor operator actions, plant operations conditions and Status of Control Room 20 equipment conditions."

21 "They informed me that there was also a display out in the remote shutdown rooms 22 of both Unit I and Unit 2. Rey indicated that the displays were not being repaired 23 due to a lack of parts."

24 "There was a good discussion and interaction between PORC (Plant Operating 25 Review Committee] members on the presented material. The PORC chairman did 26 a good job in controlling the discussions, giving each member an opportunity to 27 express his views, and ensuring items of concern were appropriately discussed."

Administrative Practice AP03-10, DET response item #4017.

(~\

s DET response item #4006.

Page V-4 The Liberty Consultmg Group

• RLS :

Chapter Fise - \tanagement and Organization i " Scope . . In both units. evaluated personnel understanding of the relationship of U 2 theirjob functions to station and company goals. ...Many of them expressed that 3 they could not easily understand how we came up with many of the numbers nor a w hat the numbers represent. Approximately half that I talked with had not yet been 5 to the employee meetings on goal performance."

6 " Scope . . Ascertain the general attitude of Operations and Maintenance personnel 7

~

regarding the two major objectives of the 1991 Master Operating Plan. All 8 personnel responded that the objectives / goals were to make STP a better place to 9 work and to enhance plant reliability and/or availability. Most personnel expressed to that the goals were good for STP and they could see improvement at STP."

11 "Later, when the Unit 2 Supervisor was asked if the TSC Diesel was operable he 12 stated that it was. "It had an alarm on the strainer to filter D/P but aat it only lasted 13 a short time during start up.""

14 "These items were discussed with Vic Simonis of the Emergency Preparedness 15 Division." -

16 "I toured the Unit 1 Turbine Generator building with Mr. John Collins, Reactor 17 Plant Operator, for the purpose of viewing first-hand the accessibility of equipment 18 that is operated by the RPO. . Mr. Collins mentioned that recently more operator 19 input into the design of permanent scaffolding is taking place. We looked at a 20 couple of areas in which more operator feedback at an early state of the design 21 would have made the scaffolding much more operator-friendly."

22 "There may be a problem with information from Mr. Hall's meetings getting to the 23 control rooms. I spoke to several of the personnel, including one shift supervisor 24 who did not know the information that Mr. Hall put out in today's meeting. They 25 were also not aware ofinformation put out in previous meetings."

26 "The Unit 2 outage meetings were orderly and well run. Problems were discussed 27 openly and clearly defined responsibility for timely resolution was assigned. The 28 meetings were well attended by line management personnel."

29 "The consensus of that crew was that the twelve hour shift was much better; it 30 allowed them to better enter a test and complete the test without having to conduct j 31 a relief part way through, and it was much easier to keep track of the outage on a i 32 day to day basis because there was only one crew and not two involved in plant 33 conditions since they were last on watch."

34 "I found the QC inspector who was involved with confirmatory inspections and ,

35 together we inspected the defect area."

l

\

Page V-5 The Liberry Cortsultmg Group

s l l

l PLS.:

Chapter Fise - \tanagernent and Organization n

) "Several personnel questioned whether cafeteria service would be (V .

2 1

expanded / modified to provide such service inside the plant."

t l

3 " Discussed the recent Unit 2 stuck open spray valve and associated plant trip with 4 Control Room Operators."

5 "I talked to two people - one in each control room about their knowledge of the 6 SALEM 2 turbine overspeed event."

7 "I got into a discussion about nuclear instrumentation and power distribution 8 control with the Unit 2 shift supervisor, Mr. Keen."

I 9 "Almost all the people I talked to during these tours were impressed with this 10 outage's preplanning and the suppon from Operations. They also mentioned 11 emergent activities have delayed critical path, added unanticipated scope to the 12 outage, and created scheduling conflicts with the scheduled activities."

t3 "During the visit I got the impression that operator awareness of the role of NSRB .

, 14 has improved. Personnel seemed pleased that an NSRB member would come out 15 to look at one of their previous problem areas."

p 16 Liberty concluded that STP's management surveillance program, while not required, was a

'(/' 17 reasonable action by management that resulted in a valuable prograr.1 that had the potential for 18 significant benefits. Liberty also found that the reports generated from this program provided 19 considerable evidence of face-to-face discussions between management and plant personnel.

20 Management Experience 21 The DET was critical of the operating experience of STP's managers: 5 22 "Most managers at STP lacked commercial nuclear operating experience outside 23 of STP. Some managers had Navy nuclear experience, but had very limited 24 experience at STP Weaknesses in commercial operational experience resulted in 25 a lack ofinsight into some operational problems and failure to fully recognize the 26 safety significance of some operational issues and also contributed to 27 management's failure to fully appreciate and compensate for the unique challenges 28 of the plant's design."

m

' DET Repon, p. 39.

Page V-o The Liberty Coruultmg Group

I l

1 RLS.:

Chapter Fise - Management and Organization O

i After the accident at Three Mile Island in 1979, the NRC began to examine the experience of 2 nuclear plants' personnel more closely than before. As part ofits review of the operating license l 3 application for STP, the NRC reviewed the experience of HL&P's personnel. The NRC concluded a that its " review of the resumes included in the FSAR and the summary data on key personnel 5 supporting South Texas indicates that the corporate support elements reporting to the Group Vice 6 President - Nuclear are well qualified technically and by previous experience in nuclear plant 7 design and construction."* After the Safety Evaluation Report (SER) was issued and before 8 licensing, STP added new personnel and made organizational changes. The NRC reviewed these 9 matters as well. In Supplement No. 3 to the SER, the NRC staff said:'

to "The staff reviewed the qualifications of key management personnel and iI interviewed the following individuals to better understand the working 12 relationships between the operating and support groups. .. [14 individuals listed]

13 The Staff finds that there is a clear understanding of the organization. interactions, 14 and schedule for bringing the unit into service, and that the mechanism for ,

15 coordination between the operating staff and the support personnel is adequate. The 16 staff finds that the management and technical support organization of the applicant 17 is adequate."

O 18 If the NRC applied consistent criteria in its evaluation of experience and capability to support the 19 operation of STP, then the only logical explanation for the difference between the NRC's initial 20 review and that of the DET is that STP's experience level had declined. But in fact. just the 21 opposite occurred. During the period that separated the two reviews, STP had a relatively high 22 retention rate and also added to its staff. The commercial operating experience level thus rose 23 during that period, both with the increase in experience at STP and with the addition of new 24 personnel.

25 It should not be surprising that most managers at STP " lacked commercial nuclear operating 26 experience outside of STP." Given the age of both STP and the nuclear industry, combined with 27 the stability in the industry (i.e., no new plants in recent years), it is likely that this statement could 28 be made about any nuclear plant. Nevertheless, experience from other facilities can be valuable.

29 and, in evaluating the reasonableness of staffing decisions, the DET's comment should not be 30 taken lightly. The basis for its finding had to be the response to the DET's request for the rdsumds 1

• l NRC Safety Evaluation Report for STP, NUREG-0781, April 1986, p.13-3.

NRC SER Supplement No. 3, NUREG-0781, May 1987, pp.13-1, -2.

1 I

Page V-7 The Liberry Consultmg Group

]

l l

\

RLS '

Chapter Fise - Management and Organization r '

(x 1 of management and supersisory personnel at STP.8 That response provided the job descriptions. '

2 organizational structure diagrams, and rdsumds for many personnel at STP.

3 An analysis of the rdsumes of the personnel holding the top 79 positions in STP's organization

. showed the following:

5 o The average total professional experience was over 21 years.

6 o The average total nuclear experience was over 18 years.

7 o The average total non STP nuclear experience was 9.5 years.

8 The considerable nuclear experience at facilities other than STP included both Navy operational 9 and commercial construction experience. The DET specifically noted a lack of non STP 10 commercial operating experience. However, of the personnel holding the 79 top management 11 positions,46 percent had actually had experience at other operating commercial nuclear plants.

12 Even if the 6 positions (of these 79) in the operations department were not considered,43 percent 13 of the top management personnel had experience at other commercial operating plants. These 14 personnel included:

s

'4 d 15 Group Vice President, Nuclear Vice President. Nuclear Generation 16 Vice President, Nuclear Engineering Assistant to Group Vice President 17 Manager, Human Resources General Manager, Information Resources 18 Plant Manager Director, Nuclear Security 19 Director, ISEG General Manager, Licensing 20 Department Manager, Technical Services Division Manager, Chemical Operations 21 Division Manager, Health Physics Division Manager, Work Control Center 22 General Manager, Nuclear Assurance Manager, Nuclear Quality Control 23 Department Manager, Maintenance Division Manager, Electrical Maintenance 24 Division Manager, !&C Maintenance Director, Nucley Generation Projects 25 Department Manager, Design Engineering Division Manager, Mechamcal Engineering 26 Division Manager Engineering Support Division Manager, Electrical Engineering 27 Division Manager, Mechanical Systems Division Manager, Reactor Engineering 28 On the basis of a review of the r6 sum 6s provided to the DET, Liberty concluded that STP's 29 management had considerable outside commercial nuclear experience.

30 The DET stated that some managers had had Navy nuclear experience but limited experience at 31 STP. Actually, quite a large number of personnel at STP have had Navy nuclear experience. The 32 DET probably was referring to those managers who had had twenty or more years of Nasy n

DET request #0020.

Page V-8 The Liberty Consultmg Group

t RLi.:

Chapter Fis e - Management and Organization

( i 2

experience and had recentlyjoined the staff at STP. The rdsumds provided to the DET showed that 6 of the personnel holding the top 79 management positions fell into this category. These were:

3 Department Nianager - Planning & Assessment. Division Nianager - Quality Performance.  ;

a Division Nianager - Niaintenance Planning, Division hianager - Niechanical Niaintenance.

5 Division hianager - Emergency Response, and the Deputy Plant Nianager. The qualifications.

6 which included command of nuclear submarines, of these personnel were impressive. Liberty 4

7 concluded that STP's organization was not dominated by personnel with over twenty years of 8

Navy experience, and that, on the basis of their rdsumds, these panicular people were extremely ,

9 well qualified for their positions.

to Other DET Comments  !

il The paragraph of the DET report that started with the comment about commercial operating i 12 experience continued as follows:'

13 "In addition, many managers had recently been rotated and into positions for which 14 they had little background. The majority of the department level managers had '

is been rotated one or more times during the past year, The large number of recent .

16 management changes caused sufficient concem for top management to issue a one 17 year moratorium in early 1993 on management rotations in an attempt to stabilize 18 the management team."

19 The DET provided no basis for its conclusion that managers had been rotated into positions for 20 which they had little background. As to the moratorium, the memorandum that instituted it sheds 21 a different light on STP's management team from that suggested by the DET. The complete text 22 of that memorandum from the Group Vice President, Nuclear is provided below.'

l 23 "Over the past two (2) years we have been involved in changing STP and shaping 24 its future. Such changes have been in the form of overhauling our systems and 25 procedures, relocating to new facilities, installing new technologies, changing our j 26 culture, and forming a management team for the future.

DET Report, p.39.

i O

Memorandum, Hall to Distribution, Management Stability and Experience, March 23.1993, DET response j item a4013.

l Page V-9 i The Liberry Consu!rmg Group l

i

R:5 :

Chapter Fhe - Slanagement and Organization "With regard to our management, we now have in place a strong team. We are '

2 confident each manager has the capability and knowledge to perform their 3 respective assignments successfully.

4 "To enhance success, STP should maintain a stable managernent team of the 5 highest possible quality. A team based on capabilities, knowledge and experience.

6 Of these three qualities, our managers' experience base in their current assignments 7 is the least strong.

8 "nerefore, in order to increase our experience base, each Manager should remain 9 in their current assignments for at least one (1) year from the date of this memo.

10 Should Station demands change, exceptions to the one (1) year requirement shall ii be approved by the Group Vice President. Nuclear "

12 The DET report continued with the paragraph below."

13 "The lack of clear and consistent direction and oversight often contributed to a ,

14 failure to successfully implement a key program or improvement initiative 15 Although the licensee recently revised the Master Operating Plan (MOP) to better 16 reflect management's stated strategic goals, many of these new goals r:mained to 17 be implemented. The discontinuity between the strategic goals and the daily 18 activities had undercut the credibility of senior management's plans and the MOP."

19 What the DET had in mind when it used the phrase "these new goals remained to be implemented" 20 is not clear. Similarly, the DET did not explain the " discontinuity" that it apparently saw.

21 Apparently the DET was referring to the plans and programs that were designed to help STP meet 22 its stated goals. In this regard, the DET requested information from HL&P to "show how the 23 Master Operating Plan Goals are translated into individual accountability."':

24 The response to that request explained to the DET that each of the MOP goals had a goal manager 25 and that the goal manager was responsible for providing a monthly status report on his action 26 plans. The DET was told that action plans with scheduled completion dates had been distributed 27 to responsible individuals and that the Planning & Assessment department had increased its own 28 resources to provide timely updates and the status of actions aimed at the MOP goals.

DET Repon, p. 39.

DET request #4031.

Page Wlo The Liberty Consultmg Group

tTL5 :

Chapter Fise - \tanagernent and Organization i Liberty concluded that the DET report did not present information or raise issues that indicated

STP's management was unreasonable relative to direction and oversight. The management

) 3 suneillance program was a voluntary initiative that clearly enhanced communications with

.: management. The nuclear experience of STP's management had been judged acceptable by the 5 SRC when an operating license was granted and had been increased considerably in its first few 6 years of operation. STP had a well-defined and -structured process for assuring that there was both 7 accountability for and Visibility ofits goals and objectives.

1 i

l l

l O

The Liberry Consuhmg Group i

?:.' .'

Chapter Fn e - Wnagement and Organization I

i III. Support and Resource Utilization 2 Section 2 A.2 of the DET report presented the finding that management had not adequately funded 3 and staffed STP. To support its conclusion, the DET used some specific examples. However. tha a DET also made unsupported com.ments about management's practices that were difficult to either 5 confirm or refute. The following discussion groups the DET's comments into three areas: boad 6 state,ments about management practices, funding, and staffing.

9 7 A. Management Practices 8 The DET said that management had failed to provide sufficient resources to maintain performance 9 levels and that "[s]ignificant station activities were not adequately funded despite th- clearly stated 10 objections of the responsible middle level managers.""

, 1I STP's budget-development process required justifications for items not included in a base or target 12 budget as well as statements of the impact that would be incurred if an item were excluded from 13 the budget. This was a reasonable and thorough process. The DET characterized the impact la statements as " objections," and its implication was that nothing should be excluded from the is budget if the responsible manager objected. Clearly, in a world oflimited resources, choices have 16 to be made. Liberty found that STP developed the information to allow reasoned choices.

17 The DET continued this opening paragraph with the following statement:"

18 "Although top management clearly stated to the team that resource limitations had 19 never prevented the accomplishment of necessary maintenance or the resolution of 20 safety-related problems, middle level managers perceived that resources could not 21 be approved if the proposed line item caused department budgets to exceed the 22 target budget levels established by senior management."

23 This statement was not only true but was reasonable and healthy. Top management had the 24 challenging task of trying to hold expenditures within budgets approved by STP's owners. while 25 not excluding necessary work. Perceptions by middle-level managers of the importance of budget 26 limitations could only be a problem if they failed to report necessary maintenance or problems.

27 Unquestionably, there was no reluctance by station employees to report problems at STP.

DET Report, p. 39.

e "

DET Repon. p. 39 (s

Page V 12 The Libery Consuhing Group i

I 1

l l

RLS . l Chapter Fhe - Management and Organization '

p '

i 'he DET made the following statement about the " management systems" at STP:"

2 "STP management had not established management systems that would effectively 3 and efficiently accomplish the strategic goals listed in the MOP by implementing 4 those goals into the daily work schedule. The planning, scheduling and work 5 process controls did not support the timely and reliable completion of work by 6 maintenance, operations and engineering. Although station management nad 7 ,

recognized this problem, they had failed, until recently, to focus the necessary 8 resources to correct this situation."

9 STP's Master Operating Plan (MOP) in effect at the time of the DET's review included strategic to objectives related to enhancing the plant availability and reliability. It also spelled out management ii priorities: (1) safety, (2) people management, (3) efficiency and cost effectiveness, (4) reliability 12 of sersice, and (5) community and industry support." To STP's credit, areas needing improvement 13 in planning and scheduling tools, work control processes, and information systems had been 14 identified by HL&P and were recognized to be significant relative to STP's objectives and ,

15 priorities. Liberty found that management had established systems that were consistent with 16 objectives and management priorities. Management also identified measures that could make its 17 planning and control systems more effective and was in the process of implementing those 18 measures.

19 The DET's discussion of management's practices reflected the NRC's emphasis on safety rather 20 than budget limits."

21 " Senior management's reaction to unforeseen, emergent work was to defer or 22 cancel other previously budgeted line items to maintain the target budget 23 expenditure goals. This approach resulted in deferral or cancellation of budget line 24 items that had previously been judged to have merit. STP routinely experienced a 25 significant end-of year deficit in the accomplishment of planned, prioriy work 26 because of the failure to adequately anticipate and budget for emergent work. The 27 increasing backlogs of deferred work in maintenance, engineering and operations 28 were clear indicators of this management approach."

29 When senior management was notified of emergent work that could not be accommodated within 30 the current budget and required a budget change, it first determined whether deferring, changing,

" DET Repon p. 39.

1993 Master Operating Plar. NGP-115, pp.1-3.

I

(/ DET Repon, p. 40.

Page V-13 The Lsberry Consulang Group i

I

l l

RL S.. '

Chapter Fo e - Management and Organization i or canceling budget items could compensate for the new work. This was good fiscal management.

2 If changes could not accommodate the new item, the STP Management Committee was asked to 3 approve a budget increase. Deferrals or cancellation of budget line items would create a real 4 problem only if they had an impact on safety or a cost-deficient impact on reliable plant operation.

5 The fact that a budget item was deemed to have enough merit to be included in the budget does 6 not mean that funds had to be spent on the item, regardless of changed circumstances. The DET

! 7 said-the " increasing backlogs" were indicators of this management approach. However, as 8 discussed in the chapters of this report on operations, maintenance, and engineering, the backlogs 9 were not actually increasing.

to B. Funding ii The DET discussed three specific examples as support for its views on the adequacy of funding.

12 Each of these is discussed below. The first of the DET's examples concemed maintenance, 13 training "

14 "One example that illustrates the senior management response to high priority 15 budget requests was the previous maintenance and training managers' request for 0,1 16 maintenance training. Both managers had established the need and requested the

{

17 funds to provide additional maintenance craft training in response to recognized is deficiencies. The request was not adequately funded despite a clearly wTitten  !

19 budget justification highlighting the significant consequences of not funding this  !

20 program. Subsequently, licensee's maintenance staff knowledge was found to be 21 below industry standards and the licensee was forced to initiate an accelerated 22 remedial training program."

23 Re DET did not provide enough informatieu in the description of this example to make it possible 24 to understand with certainty to which budget requests and actual funding the DET referred. One 25 of the DET's questions stated that training had requested $1.! million, and maintenance $1.3 26 million, in the 1993 budget for craft skills upgrade and requested information on where these 27 requests were reflected in the approved 1993 budget." He response to that request showed the 28 DET that a total of $2.0 million had been included in the 1993 budget for this program element.

29 Apparently, the DET decided to call the element "not adequately funded" because of the reduction 30 from a total requested amount of $2.4 million. However, as detailed in the chapter of this report DET Report, p. 40.

I " DET request a4035.

Page V-14 The Lsberry Coruultirig Group

I R!.5 .'

Chapter Fhe - \tanagement and Organization q

on maintenance, these requests for funding were made after, not before. the National Nuclear (O 1 2

1 Accreditation Board had told HL&P that performance-based training forjoumeymen was required.

3 By the time the DET performed its evaluation. it was apparent that the training accreditation issues 4 had been satisfactorily resolved.

5 The DET also requested all 1992 budget proposals submitted by maintenance in 1991. including 6 impact statements. Although HL&P explained that once a budget was finalized it purposely did 7 not maintain all of the documentation that may have been generated in order to develop the budget.

8 it nevertheless provided the DET with over 400 pages that had been retrieved from files pertaining 9 to the 1992 and 1993 budgets. STP's budget actions with respect to requests for maintenance 10 training budget undenvent a comprehensive review process, as did all other departmental budget ii requests. The components of the training budget request for 1992 that were not funded at the 12 requested levels primarily involved electrical, mechanical and I&C Level 1 apprentice training.

13 Some component-specific training for electricians was reduced, and some funding for unfilled 14 electrical joumeyman positions was eliminated. However, about $800,000 was included in the' 15 budget for electrical, mechanical, !&C, and otherjourneyman training.

/

16 The consequences, which were identified in budget analysis documentation for budget cuts 17 provided to the DET, had little to do with the accreditation issues STP had experienced in late 18 1992, which precipitated the need for additionaljoumeyman trainmg.2' If all funding requests had 19 been approved STP would still have faced the same accreditation issue. That issue involved the 20 method used to exempt journeymen (not Level 1 apprentices) from general training (not specific 21 task or skills training) on the basis of their stated prior work experience.

22 The DET's conclusions implied a cause-and-effect relationship that did not exist. STP's 23 maintenance staff knowledge was not found to be "below industry standards." Rather, journeyman 24 work qualification status could be granted on the ba sis of experience rather than on a performance-25 based assessment of knowledge and skill, and requalification using new assessment methods and 26 training was required to achieve a desired level of maintenance craft cenifications.

l 27 The DET's second example dealt with operator training.22

DET request #4036.

" STPEGS 1992 Budget Analysis Cost Center 857 - Nuclear Training. Rev. 6 September 1991.

() " DET Report, p. 40.

(

Page V-15 The Liberry Comultmg Group

l PLS. '

Chapter Fisc - \tanagement and Organization l O2 i 3

"A second example was the elimination of funding for the operations training pipeline for replacement of plant operations personnel due to attrition. Operations staffing had been reduced to such a low level as to become the critical resource for i

4 implementation of some station activities. Attempts tojustify funding for training 5 of additional non licensed operators were rejected by senior management based on 6 inaccurate staffing projections that failed to account for the actual scope of work 7 and responsibilities assigned to operators. Failure to provide additional non-8 licensed operators had prevented upward progression of non-licensed operators into 9 the ranks of licensed operators and precluded utilization of licensed operator 10 experience in other functional areas at STP."

ii The subject of operator training is addressed in the chapter of this report on operations. In 12 summary of that information, STP's management made a reasonable assessment of the need to i t3 provide additional funding for the operator pipeline. The decision not to provide such funding was 14 due, in part, to the low turnover of personnel at STP Every year of STP's operation there were 15 more licensed operators than the prior year. Liberty found that there was a former manager in 16 operations at STP who believed that additional operator pipeline funding was needed. He did not' 17 convince his management of that need, however, and independent staffing studies supported the is adequacy of current staffing. The DET interviewed this person before it began its on-site review 19 and apparently agreed with his position. The DET statement regarding inaccurate staffing 20 projections did not specify what the inaccuracies were. As pointed out in the chapter on operations, 2i STP routinely employed outside expertise to help assess the appropriate size ofits operation staff.

22 Moreover, STP did not inhibit the progression of non licensed operators into licensed operators 23 and there was substantial evidence that licensed operator experience had been used in other parts 24 of STP's organization.

25 The third example used by the DET concemed the budget for engineering. The DET said? l 26 "A third example of the impact ofinsufficient funding was the budget exclusion

]

27 noted in the proposal submitted by engineering highlighting the fact that l 28 engineering backlogs of modifications and corrective actions would not be reduced 29 in 1993 due to lack of funding. In fact, engineering backlogs had continued to 30 4 increase in 1993."

31 The engineering budget proposal for 1993 actually noted that "[e]ngineering backlogs will not be 32 significantly reduced."2' This was a considered and reasonable management decision. Furthermore.

22 DET Report, p. 40.

DET response item #4028, p 407.

Page W16 The Liberty Consultmg Group

? U .'

Chapter Fise - Management and Organization h I as discussed in the chapter on engineering, overall engineering bacidogs had seen reduction trends.

2 and this was true even during a period (i c., two unit outage) when short-term increases might be 3

expected. Finally. the DET did not mention that the approved 1993 O&M budget for engineering a was 5 percent larger than the 1992 budget.

5 The DET criticized the funding for training?

6 " Support of training, including funding, was weak. For example, the team 7

identified several instances where lack of training contributed to system engineer 8 ineffectiveness, weak problem resolution, and poor decision making. However, the 9 funds budgeted for training ofdesign engineering department (DED) engineers was to seven times greater than the funding budgeted for approximately the same number ii of engineers in the plant engineering department (PED) which included the system 12 engineers. This appeared to be a historical discrepancy in that engineering training 13 funds were based on previous year's budgets and not on actual need."

14 The DET compared the total amounts contained in two program elements in the 1993 budget 15 Program element H95439 - DED Training Participation had a total budget of $280,698. Program 16 element H95440 - PED Training Participation had a total budget of $41,001, about one-seventh

, 17 of DED's program element. Liberty's examination of the information that supported the total 18 budget numbers showed that these two figures were not comparable?

19 The DED program element budget consisted of two items. The first was $279,198 for payroll and

0 payroll adders for 4.3 equivalent people. These people were serving as training coordinators for

DED. The second item was $1,500 for expenses related to attendance at off-site training on eddy 22 current testing? However, the PED program element consisted entirely of costs associated with 23 off-site training, conferences, and semmars. Rese costs included tuition, travel expenses, and meal 24 expenses. The PED program element did not contain any payroll costs. He 1993 budget detail for 25 PED oft-site training reflected the extent and variety of the off-site training planned for PED, as 26 shown below?

" DET Report. pp. 41-42.

DET response item nos. 4024,4028,4028-001.

DET response item 84028-001.

" DET response item 84028 001.

Page V-17 The Liberty Comultmg Group

Rt.5-:

Chapter Fise - 41anagement and Organization f}

.(s l 2

EPRJ conference turbine performance monitonng Westinghouse seminar RCP seals 5150. I

' 1.630.

3 Diesel engine operation & maintenance training 1.500.

4 Woodard diesel gosernor control course 2.325.

5 EPRI seminar on instrument air systems 1.900.

6 Off site training - breaker conference 2.300.

7 Training seminar heat rate analysis 2,050.

8 Vibration seminar 1,105.

9 Thermography training 1,050.

10 -

ILRT training 2,500.

I1 Advanced check valve training 1,500.

12 Power-operated relicf valve training 1.375.

13 Westinghouse owners group 1.225.

, 14 Post-Accident sampling system 1,145.

' 15 INPO training - safety system performance 200.

16 Stuan & Stevenson diesel training 450.

17 Industnal fire seminar 200.

18 Fire brigade training 4,876.

19 HVAC testing school 1,780.

20 Reliability-centered maintenance 3.240.

21 Radiation monitoring users group 5,000.

22 Diesel. MOV SOV users groups 4,125.

23 Other meal costs 22jL

, 24 Toul S42,001.

25 DED also included travel costs and off site trauung expenses in the 1993 budget but included such

(

26 items in various program elements like H95473 - DED Engineering and H95474 - DED Research 27 and Development.

28 Even if the DED and PED budgets for off-site training were comparable, the result of such a 29 comparison would provide little information relevant to total traimng effons because most training 30 is performed on the STP site, On-site training provided by STP's training department would not 31 be reflected in the engineering depanment budgets.

32 The DET said tha t "[s]tation improvements were adversely impacted due to budget pressures.""

33 To support this statement, the DET cited two examples. He first concerned funding provided for 34 plant labeling. HLdP in fact provided $1 million for the relabeling effon. This matter is addressed 35 in the chapter of thic report on operations.

O

\

" DET Report, p. 4?

Page V-18 The Liberty Consultmg Group

1 RLS .'

Chapter Fis e - %1anagement and Organization I

i The second example that the DET used to attempt to show that improvement was adversely 2 affected by budget pressures was the statement that funding for "the Long Range Engineering 3 Improvement Plan. .was eliminated in the 1993 budget."3 4 Departmental proposals for the 1993 budget compared proposed amounts with the 1992 budget.

5 The engineering proposal showed a $1 million program element for "Long Range Improvement 6 Program"in the 1992 budget that was not included in the 1993 proposal. The program element had 7 been included in the 1992 budget to obtain consulting assistance in the development oflong-range 8 plans. By July 3,1992, the consultant (Impell) had provided instructions for long-range plan 9 development, an equipment obsolescence determination guide, and specific long-range 10 improvement plans for computer replacement, turbine generator low pressure rotors, diesel 1i generators, and steam generators. Some funds from this program element had also been used in 12 the analyses of the ECW system. HL&P decided that with the information provided by Impell, it 13 could continue the long-range planning on its own. In August 1992, a change notice to the 1992 14 budget was therefore approved that returned $323,000 of the initial $1 million.32 Since the initial 15 consulting effort had been completed, there was no reason to have this same program element in 16 the 1993 budget. This did not mean that there was no long-range improvement planning, or that v} 17 " budget pressures" had caused some important program to be eliminated.

18 hone of the funding issues identified by the DET demonstrated unreasonable decisions or actions 19 by STP management. These issues were concems captured by the DET's snapshot that, upon 20 detailed examination, did not show either an unreasonable process or an unreasonable result.

21 C. Staffing 22 The management and organization section of the DET report discussed STP's staffmg and repeated 23 conclusions presented in other parts of the report. The following is a listing of DET comments and 24 the sections of this report where those comments are addressed.

DET Report, p. 42.

D DET response item 54028.

'3 g 1992 Budget Change Notice 92-089 August II,1992, and Memorandum, R.R. Hernandez to S L Rosen.

) Long Range Improvement Planning, July 6,1992.

Page bl9 The Lsbero Consultmg Group

Rl.S .'

Chapter Fne - \lanagement and Organization 1 DET Comment This Reoort '

b 2 Operators exceeding overtime limits Operations chapter 3 Number of operator crews Operations chapter 4 Maintenance overtime Maintenance chapter 5 Ei ineering overtime Engineering chapter 6 ISEG and OER backlog This chapter, self assessment and quality oversight 7

The DET commented on independent staffing studies commissioned by STP:))

8 "The decision to have several station staffmg studies conducted by outside consultants 9

indicated senior managements' concem over appropriate staffing levels. However, the 10 recommended staffing levels in the most recent study was based on incorrect ii assumptions on productivity. The average time required to complete a service request 12 (SR) was estimated in this study as less than 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />. Information supplied by the 13 licensee at the request of the team indicated that the actual time required to complete 14 an SR ranged between 42 and 50 hours5.787037e-4 days <br />0.0139 hours <br />8.267196e-5 weeks <br />1.9025e-5 months <br />. Additional information found by the team 15 indicated that the latter figures may also be low because the licensee's management 16 information system did not account for lost time due to lack of coordination and parts .

17 availability, which were significant problems at STP, and time expended by one craft 18 assisting another."

19 The staffing study to which the DET referred was performed by ASTA, Inc. during 1992.34 Only 20 one part of the staffing study, that pertaining to the maintenance department, depended on the 21 average number of hours needed to complete a service request. ASTA used 29.2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> per senice 22 request " based on past experience which includes lost time such as waiting for parts, tool pick-up, 23 etc."" The DET requested information on the number of maintenance service requests and actual 24 hours required to perform them. The data provided by STP showed the average hours per sen ice 25 request to be 29.2.)* HL&P also admitted that some of the data could contain some errors. But even 26 aRer correcting for errors, the data showed the average hours per service request to range from 25.6 27 to 26.0." It is possible that the range of figures (42- to 50-hour averages) provided by the DET in 28 its report was given to the DET by persons in interviews. However, Liberty could find no data to 29 support these higher amounts in any of the records ofinformation given to the DET.

DET Report, p. 41.

DET response item #0066.

ASTA Stafung Level Study, DET response item =0066, p.16.

DET response item #2315.

s ,

' i DET response item nos. 2315,231$-001, and 2347. <

l Me V 20  :!

The Lsberty Consultmg Group

f RLS-:

Chapter Fise - Management and Organization G

b i IV. Communications and Teamwork 2 A. Task Forces 3 The DET said that there "was a weak coordination and accountability between disciplines during 4 routine work. As a result, an excessive number of task forces, outside the normal organization.

5 seemed to be required to accomplish routine work."" Later in its report, the DET said: "Another 6 result of this lack of coordination was an excessive dependence on task forces, outside the normal 7 organization, to accomplish many tasks at STP. While the team was onsite, there were more than 8 40 task forces in place ""

9 The DET argued that the task forces had resulted from poor communications, but it could just as to easily be argued that the task forces promoted good cross-disciplinary communications.

ii Organizational theorists have said that the need for task forces may demonstrate a weakness in the 12 organizational structure or behavior. But when making such statements, the theorists could not 13 have been thinking about the organization of a nuclear power plant, which has to deal with a wide la variety of complex technical, operational, and regulatory matters overlaid on the financial, human, 15 and production concerns typical of any organization its size. Whatever the organizational (Q) 16 principles that guided the DET to its statements, it is peculiar to find the argument in a report 17 prepared by the DET, which was, after all, itself a task force.

18 The DET called the number of task forces as well as the dependence on the task forces 19 " excessive." There was, however, no frame of reference for determining whether the number 20 should be considered either moderate or excessive. Within the context of an organization 21 employing in excess of 2,000 people, more than 40 task forces is not so large. More significant 22 than number is the nature and purpose of the task forces. Liberty reviewed the information 23 provided to the DET about task forces at STP and found the following:'

" DET Report, p. 37, and repeated p. 43.

" DET Report, p. 44.

I "

IQ DET response item nos. 3023,3023-002, :nd 3023-004.

Page V 21 The Liberty Consulting Group

I i

l l

Pl.%l l Chapter Fne - \lanagement and Organuation i o HL&P identified 41 task forces that were in effect at the time of the DET's review.

2 Three of these were HL&P corporate task forces that each had a single representative from STP. These were the Budget Development Steering Committee and the 4 Business Planning for HL&P and the Work Family task forces. The existence of 3 these task forces reflects very little on the STP organization or the communications 6 within the STP organization.

7 o At least two of the active task forces were nothing more than a mechanism to 8 interface with equipment vendors. For example, the membership of the Turbine 9 Generator task force was all STP engineering personnel and a representative from to Westinghouse. Similarly, the Plant Computer task force was made up of engineering 11 personnel and a contractor from Hurst Engineering.

12 o At least two task forces were made up ofindividuals drawn entirely from within one 13 organizational element at STP. He Training Accreditation Retention task force was' la composed of only training department personnel. The Simulator Upgrade task force is was totally comprised ofindividuals from one division except for one person from 16 another division in the same department. In fact, the simulator upgrade group was 17 shown on organization charts given to the DET as a regular part of the training 18 department!'

19 o One task force (STP Minority Business Council) had a chairperson named but no 20 other m:mbers appointed at the time of the DET's review.

21 o One task force (Source Term Reduction) had been shifted to a project management 22 approach with no standing committee remaining.

23 Of the remaining 32 task forces, some were formed in order to provide the necessary cross.

24 discipline response to issues raised by the NRC. For example, the Solenoid-Operated Valve task 25 force was a response to the NRC's Generic Letter 91-05 and NUREG 1275. Another task force.

26 Maintenance Rule, was even named after the new NRC regulation on maintenance. Some task 27 forces were in place to help assure good input from various parts of the organization that might use 28 or be affected by the results of the task force's work. For example, in other parts ofits report the fD Q "

DET response item *0020.

Page V.22 The Liberty Comulting Group

l l

PL5-:

Chapter Fn e - \tanagement and Organuation O I DET was entical of STP's information systems and the failure to adequately support operations 2 and others. Yet the Long-Range Information Systems task force was designed specifically to 3 obtain a good understanding of the needs of the users ofinformation systems.

4 Ty pical ut the remaining task forces was the Water Hammer task force. Waterhammers occur 5 sometimes in mechanical fluid systems and are associated with irregular and mixed-phase fluid 6 flow.Waterhammers have the potential to damage plant equipment and harm personnel. The 7 causes of waterhammers are related to system design and layout and to operating conditions and 8 procedures. Sometimes circumstances involving design, layout, and operation come together to 9 cause waterhammers. In April 1992, STP formed a Water Hammer Identification and Correction 10 task force. The group included representation from operations (who operated the systems and ii identified waterhammers) and engineering (who designed the systems). The task force was charged 12 with identifying and documenting known circumstances that led to waterhammers, prioritizing a those conditions for correction, and proposing operational changes, moditications, or other means la to prevent hydraulic transients that have led to waterhammers.'2 This was hardly routine wcrk.'

15 Furthermore, given the unique nature of both causes and cures, a cooperative, ongoing effort 16 between engineering and operations was very appropriate.

('

l7 Liberty's conclusions were different from those of the NRC's task force, the DET. An examination 18 of the 41 active task forces showed that only 32 were multi-disciplinary STP groups. These task 19 forces were not relied upon for routine work and appeared to have been necessary to address 20 complex and multi-disciplinary issues. These task forces may have actually promoted good 21 horizontal communications.

22 B. Speakout Program 23 As part ofits evaluation of communications at STP, the DET presei:ted an assessment of STP's 24 Speakout program. He DET said:')

25 "The licensee had established several formal methods for employee > to communicate 26 upward to management. The Speakout Program was intended for safety concems and 27 the Employee Assistance Program (EAP) was intended for nonsafety concems.

" Memorandum, Rosen to Distnbution, April 23,1992, DET response item #3023.

l l

( " DET Report, pp. 43-44.

Page V-23 The Liberty Consulting Group l

I

-4 RU.:

Chapter Fis e - \tanagement and Organization l

1 i Although both programs were supposed to be anonymous there was a perception 2 among many employees that these programs were not, which limited their 1

effectiveness. There was also the perception that management was not interested in 4 hearing about problems as demonstrated by the lack of results when issues were 5 brought fonvard. Some employees had used the Speakout Program to express concerns 6 for the lack of maintenance training, but their expressions of concern proved to be 7 ineffective. Some employees expressed concem that some issues, such as excessive 8 overtime, were processed as nonsafety concems in the EAP. Although personnel 9 ' expressed reluctance to report some problems. the team did not detect any reluctance

, 10 for employees to report issues perceived as immediate safety concems."

1I One year before the DET, the NRC evaluated STP's Speakout program as part of a special 12 inspection designed to resolve concems raised in a petition submitted to the NRC. Because of the 13 nature of the allegations contained in the petition and the fact that the petitioner had raised 14 concems in the Speakout program, the evaluation of the Speakout program was a key element of 15 the NRC's inspection. He 1992 NRC inspection appears to be have been thorough. The report 16 contained a description of the program and a detailed summary of the inspection's review of 17 Speakout cases, practices, and investigations. The inspection included interviews with STP staff 18 and contractors. The summary conclusion of this review of the Speakout program was that "the 19 Speakout program was viewed as an effective factor in addressing employee concerns.""

20 As to the specific failings alleged by the DET, the earlier NRC inspection reached the following 21 conclusions:"

22 "Most of the licensee's staff and contractors who were inteniewed during the 23 inspection expressed a general confidence with taking concems to the Speakout 24 program. Many of the inteniewees had taken concems to Speakout or had known of 25 individuals who had taken concems to Speakout. .Some inteniewees stated that they 26 suspected, in a couple of instances, that the identity of Speakout concernees had 27 become known to the concemees' managers. The conjectural evidence expressed by 28 these inteniewees was exclusively based upon the belief that only a limited number 29 of people had known of the specifics of the concems. He team [the NRC inspectors]

30 considered these opinions, but were unable to substantiate them."

31 Thus the two inspections by different parts of the NRC yielded distinctly different results. The 32 DET did not identify any recent changes that would help to reconcile its findings with the earlier NRC Inspection Report 92-07, ST-AE-HL 93083, June 1,1992, p. 29.

NRC Inspection Report 92 07, ST-AE-HL-93083, June 1.1992, p. 28.

Page W24 The Liberty Consultmg Group

Rl.L:

Chapter Fise - \tanagement and Organuation I esaluation. N!oreover, as late as November 1992, the SALP report noted that the " licensee's

program for handling employee concerns (SPEAKOUT) was evaluated by the NRC during this 3 assessment period and was found to be generally effective ""

4 During the period from December 1992 through April 1993, a behavioral consulting firm 5 performed an organizational assessment at STP. Among the findings of that study was the fact that 6 STB?s "[e]mployees reported a willingness to communicate nuclear safety- or quality-related 7 concerns."

8 The DET did not explain the basis for its conclusions. The nature of the DET's statements implies 9 that they were based upon information that the DET received in interviews. Unlike the earlier NRC 10 inspection (and the consultant's study), the DET did not state whether or how it had attempted to Ii substantiate what it had been told during interviews. Given that the earlier inspection had focused 12 on the Speakout program and had described how it had reached its conclusions, and that HL&P's 13 consultant had corroborated those conclusions, Liberty concluded that management would have' 14 been reasonable to rely upon the earlier NRC assessment of Speakout in making decisions and 15 taking actions in this area.

U

'l NRC SALP Repon, ST AE41L-93239, November 18,1992,p.22.

(j

Organizational Interface Assessment, Behavioral Consultant Services, Inc.. April 1993, p. 4-3.

Page V 25 The Libery Consultmg Group i

l

. - , . . _ . - --.-= .~ -. -- _, ., . - -

PU.:

Chapter foe \tanagement and Organization I i V. Corrective Action Process .

2 A. Background and Perspective 3 The DET concluded that the ineffectiveness of STP's corrective action process was a " major 4 obstacle" to improved performance." The DET repeated this same finding as one of the four root 5 causes to which the DET attributed STP's " declining performance.""

6 The section of the DET report that supported the finding on the corrective action process was, for 7 the most part, a summary collection of difficulties and incidents that had been discussed elsewhere -

s in the report. This organization was logical because the corrective action process affects, and is 9 participated in by, all parts of the organization, and any problem that is viewed in other than to unavoidable terms can be related to that ubiquitous process. This report also addresses these iI problems and incidents in other sections. The topics and their locations in this report are:

I2 DET Comment This Recort 13 Root cause analyses and corrective actions Engineering chapter la QA effectiveness This chapter, next section 15 Root cause analysis training Engineering chapter 16 Fuel injector hold-down studs Engineering chapter 17 Qualified display processing system Maintenance chapter is Relabeling program Operations chapter 19 ISEG effectiveness This chapter, next section 20 Use of operational' experience Engineering chapter 21 One topic included in the DET's assessment of the corrective action process that is not addressed 22 elsewhere in this report concerns STP's outage planning. That topic is covered in the next 23 subsection below.

i t

24 Except for its statement that management had "recently" begun to respond to problems identified 25 internally and externally, the DET did not discuss the positive aspects of STP's corrective action 26 process. As a result, the DET's snapshot of STP did not reflect with the conclusions previously 27 drawn by the NRC. For example, the NRC's SALP report issued in September 1991 said:"

" DET Report, p. 44 DET Report, pp,49-50.

t NRC SALP Report, ST AE HL-92831, September 6,1991, p. 23.

Page W26 The Lsberty Consultmg Group

R!.5 .'

Chapter Fn e - 41anagement and Organization

/~~N i "The corrective action process was found to be generally effective. with recent 2 enhancements resulting in a signi6 cant improvement in the quality of problem reports.

3 .The response to, analysis, reporting, and corrective actions for most events were a generally good."

5 The next SALP report from the NRC in November 1992 continued along the same lines saying 6 that "[t]he licensee's self assessment and corrective action programs were evaluated as good.""

7 Thus, while the corrective action process is one that. by definition, can and must be constantly 8 impoved, STP's management cannot be faulted for unreasonable decision making or action in this 9 area given the contemporaneous feedback it was receiving from the NRC.

to B. Outage Planning i1 The DET's assessment of the corrective action process included the assertion that the 1992 Unit 12 1 refueling outage had started with an " unrealistic" 63-day schedule. The DET's implication was 13 that management had not planned for sufficient time to support necessary corrective action. The la DET said:"

Q is 16 "Although senior management expressed the desire to become more responsive on corrective actions, it appeared from documentation and interviews that little progress 17 had been made and that budgetary pressures had an adverse impact on corrective 18 actions. This situation was clearly exemplified by the 1992 Unit I refueling outage 19 where management established an unrealistic goal of a 63 day schedule which led to 20 starting up the unit with an excessive number of deferred work items and an unreliable 21 AFW train (even though the outage actually lasted for 105 days)."

22 The DET did not say how it had concluded that the outage schedule was unrealistic. Liberty 23 examined outage schedule planning to determine whether establishment of that schedule 24 represented an unreasonable management action.

25 The 1992 refueling outage was the fourth such outage for Unit 1. The outage schedule durations 26 planned for the prior three outages were 55 days,65 days, and 75 days." Several matters were

" NRC SALP Report, ST AE HL-93239, November 18,1992.p.22.

" DET Report, p. 45.

" Outage Critiques for 1RE01, IRE 02, and 1RE03.

Page V-27 The Liberry Coruulting Group

ULb. l Chapter foe - \lanagement and Organization l

1 apparent when these prior outage plans were considered. First. the outage plan for the founh i V  : refueling outage was consistent in duration with the plan for the prior outages. Second. as 3 discussed in the chapter of this report on maintenance. the backlog of work did not indicate that 4 the fourth refueling outage should have been planned to be longer than the others. And finally the 5 NRC nes er criticized the prior outage duration plans.

6 The actual duration for the prior three refueling outages exceeded the planned duration.

7 Management consistently established demanding sqhedules for refueling outages. The first 8 refueling outage on Unit I lasted 75 days, or 20 days longer than originally planned and 5 days 9 longer than anticipated in a revised plan issued about a month into the outage. During turbine 10 inspections conducted in this outage, linear crack indications were discovered on the low pressure 11 turbine #13 stationary blades. HL&P decided to conduct additional inspections, which extended 12 the outage " The second refueling outage on Unit I lasted 82 days, or 17 days longer than 13 originally planned. The primary contributors to We outage extension were three unanticipated la problems. These problems were associated with the reactor building polar crane, the manual' 15 switches for the engineered safety features actuation, and a reactor coolant pump seal leak." The 16 third refueling outage on Unit I had been planned to begin in March 1991. However, the main

(~N 17 generator was damaged because of a ground fault, and the damage was significant enough for the E) 18 owners of STP to decide to start the refueling outage on January 15,1991. He 75-day plan, which 19 included the generator repairs, was exceeded by 4 days, and the outage was completed on April 20 4,1991." In each of the prior three refueling outages, unanticipated problems extended the outage 21 beyond the plan.

HL&P planned the fourth refueling outage on Unit I to last 63 days, but it actually lasted 104 days.

23 (Technically, the refueling outage lasted 63 days and a subsequent forced outage lasted 41 days.)

24 Breaker open occurred at 2 a.m. on September 19,1992, and breaker closure occurred at 9 p.m.

25 on December 31,1992. HL&P's retrospective analysis of the outage schedule duration identified 26 18 schedule impacts that totaled 49 days. It also identified eight items that had resulted in schedule 27 gains totaling 8 days. The major negative schedule duration impacts were: a weld repair required 28 on the control rod drive mechanism housing, diesel generator trouble shooting and testing, motor-29 operated valve testing and gear replacement, feedwater and main s$ isolation valve work, and

" Outage Critique for IRE 01.

Outage Critique for IRE 02.

O *

(L.j'i Outage Critique for 1RE03.

Page V-28 The Leerry Coruultmg Group J

RLS.:

Chapter fis e - Management and Organization I refueling machine repairs. HL&P's analysis of the outage determined effects (in terms of hours) 2 caused by critical path activities. Liberty did not attempt to examine the detailed reasons for each 3 impact. However, the very nature of the problems causing these impacts showed that they were a reasonably unanticipated." The objective evidence clearly indicates that the plan was reasonable 5 and that, esen using a retrospective view of the items that extended the outage duration, the plan 6 was not unrealistic, 7 The DET asserted that the outage had concluded with an " excessive" number of deferred work 8 items and an " unreliable" auxiliary feedwater train. The DET did not specify its grounds for 9 reaching these conclusions. The chapter of this report on maintenance addresses both the issue of to the work backlog and the issues related to the auxiliary feedwater system.

V O

V " Outage Critique for IRE 04.

Page \-29 The Liberty Consultmg Group

i

\

pts. '

\

Chapter Fis e - Management and Organization

/

( i VI. Self-Assessment and Quality Oversight '

I

A. Introduction 3 With regard to the effectiveness of STP's self-assessment and quality oversight functions. the DET 4 found that:"

s ' Managers did not respond etTectively to the findings, concerns, and recommendations

. 6 of their principal self assessment and quality oversight functions, including the Nuclear 7

Safety Review Board (NSRB) and QA. In addition, management had not fully

8 supported the ISEG review for lessons learned."

9 In fact, the DET felt so strongly about this matter that it used almost the same language to describe to one of the four root causes of the performance at STP."

t " Management had not responded effectively to findings, concerns and -

> t2 recommendations of its principal self assessment and quality oversight functions, 13 including the Nuclear Safety Review Board (NSRB), and Quality Assurance (QA).

14 These and other STP oversight functions had clearly identified significant (N 15 programmatic weaknesses associated with operations, maintenance, engineering, and 16 the corrective action process. In addition, manageme.nt had not fully supported the 17 Independent Safety Engineering Group's (ISEG) review for lessons learned.

18 "Since 1991, the NSRB had advised STP management of specific problems which 19 could affect safety and needed management attention, but little, if any improvement 20 was noted in many of these areas. Similar information was available from QA audits 21 and reviews."

22 The bases for the DET's findings are discussed below in sections that focus on the three principal 23 quality oversight groups, NSRB, QA, and ISEG.

DET Report, p. 46.

DET Report, p. 49.

Fage v-30 l

The Liberry Consuhing Group l

i RL5 .'

Chapter Fne - \tanagement and Organization O

V i B. Nuclear Safety Review Board (NSRB) 2 The NSRB is an independent nuclear safety oversight organization at STP. The DET's speci6c 3 Ondings relative to the NSRB were" 4 "Since 1991, NSRB had advised STP management of specinc problems which could 5 affect safety and needed management attention. An NSRB " Watch List" was 6 established in 1992 to emphasize the importance to safety of the 10 top issues of 7 ' concern to the NSRB. In February 1992, the material condition of the plant and the 8 corrective maintenance backlog concerns were near the top of the NSRB Watch List 9 of concerns. In the last 14 months, the NSRB had repeatedly advised senior 10 management to take proper remedial measures, but little, if any, improvement was 1I noted in these areas. As of April 1993, these same concerns were near the top of the 12 NSRB Watch List. Personnel concems were also included on the 1993 NSRB Watch 13 List, such as supervisor effectiveness, the effectiveness of communicating the station la philosophy of getting things done right, rather than just getting them done, and 15 personnel integrity."

16 In the November 1991 NSRB meeting, the group discussed ways in which the NSRB could 17 enhance its ability to make NSRB concerns more visible. A draft list of 17 concerns was is considered for the purpose of providing "a compilation of the active signi6 cant NSRB concerns.

19 to be reviewed and updated on a monthly basis."6: In the December meeting, an initial list of ten 20 items was established as the NSRB's Watch List. That Grst list contained items relating to the 21 material condition of the plant and the maintenance backlog.62 22 An analysis of the issues contained in the NSRB's Watch List over the period from December 1991 3 through April 1993 showed that only four items had ever disappeared from the list.63 Two of these 24 were removed after the first month's listing. The DET chose to characterize the fact that items had 25 remained on the list as " repeatedly advised senior management to take proper remedial measures."

26 but there is another, more detached explanation. First, the nature of the items on the list was such 27 that they were not likely to be disposed of quickly. In other words, if the issue could have been DET Report, p. 46.

" Memorandum, Chakravorty to NSRB, October 31,1991, attached to minutes of November 13,1991 meeting.

NSRB Meeting 91 14 minutes, d't. #7.

" NSRB Meeting minutes art. 85 of 92-02, att. 85 of 92-03, att. #4 of 92-04, att. 45 of 92-05, art. *I of 92-07, y att. #1 of 92-08, att. #1 of 92-09, att. *l of 9210, att. #1 of 9211, att. #1 of 93 01, and att. si of 93-02.

Page V-31 The Lsberty Consulung Group

RLS.'

Chapter Fisc - \lanagement and Organization Q 1 easily resobed. it probably would not have made the list in the first place. Second. the NSRB did 2 not remose an item from the list even if recent performance in the area was good. For example.

3 the number of reactor trips was an NSRB Watch List item from the time the list was started. In 4 earlier years. STP's number of trips was higher than desired. In 1992. STP had only one trip while 5 critical in each unit and met the station's goal. Yet the NSRB kept that item on the list for the 6 entire year. In other words, even though management had implemented a successful trip reduction 7 prog _ ram, it is clear that the NSRB still considered the issue to be one that should be listed. Thus.

8 the fact that an item was on the list did not mean that the NSRB had not been effective, that 9 managers had not responded effectively, or that there had been no improvement.

10 In late 1992 the NRC conducted a special Operational Safety Team Inspection (OSTI). The report ti from that inspection contained the following conclusion: "The nuclear review board oversight 12 function was noted to function very effectively.""

13 HL&P specifically addressed the NSRB's initiatives in a letter to the NRC." That letter described 14 the NSRB's standing committees that permitted the NSRB "to perform effective high level 15 oversight of safety issues and to conduct its business efficiently." The letter also described the i 16 membership of the NSRB and the qualifications of those members. Three of the eight members V 17 were personnel from outside HL&P, each having over 30 years of experience. One served on the 18 equivalent of the NSRB for 14 different nuclear units. The other two had major areas of expertise i 19 in operations, radiological safety, training, emergency preparedness, and engineering. Drawing I 20 upon their experience with other nuclear plants, these individuals brought a number ofissues to j 21 the attention of the NSRB. Finally, the letter provided a summary of 20 issues that were examples  !

22 of matters that had come to the attention of the NSRB. For each issue, the letter summarized the 23 manner in which it had been identified and its current status. l 24 The information given to the DET about the NSRB's stmeture, charter, membership and 25 performance, combined with earlier NRC assessments, indicated that the NSRB had been effective.

26 The NSRB's Watch List initiative was evidence ofefforts to improve its effectiveness not a vehicle 27 for repeatedly advising of the need to take remedial measures. Liberty concluded that 28 management's decisions and actions related to the NSRB were reasonable.

I i

1

)

NRC Inspection Repon No. 92-35, ST AE HL-93325, March 3,1993. l 4 Lener, HL&P to NRC, Hall to Milhoan, ST-HL-AE-4342, March 30.1993.

Page V-32 The Liberty Consubmg Group

RL5 .

Chapter Fue - Managernent and Organization r

I

( i C. Quality Assurance (QA)

Relative to QA's effectiveness in STP's self-assessment function the DET concluded "

3 "Similar findings from QA audits and reviews had been equally ineffective in bringing 4

{

about corrective actions and improvement even though these Gndings and  !

5 recommendations were routed to top management and other managers who had j 6 , responsibility for the issues addressed. The team found numerous records which 7 documented QA's persistence in attempting to gain management's attention to their 8 safety Gndings. Examples included continuing weaknesses in:

9 "* the corrective action process to "-

material condition of safety-related systems 1i "-

management's use ofinformal documents (e.g., night orders) to amend 12 or revise procedural requirements before changing the procedures 13 themselves la "- con 6guration management -

15 "- maintenance training, qualifications, and work activities"

] 16 The effectiveness of QA had been evaluated many times before the DET's investigation. It is 17 useful to consider these prior evaluations to determine whether they reached similar findings or 18 discovered the precursors to a lack of effectiveness that was to become more evident by the time 19 of the DET's evaluation. Liberty found, however, that prior evaluations of QA, including those of 20 the NRC. were very positive.

21 Industry Assessments 22 Nuclear utilities cooperate in a periodic effort to evaluate each other's quality assurance program.

23 In September 1991, a Joint Utility Management Audit ffUEf) was performed on STP's Nuclear 24 Assurance Depanment (quality assurance is part of the Nuclear Assurance Depanment). The audit 25 was performed by personnel from Entergy Operations, Inc., New Hampshire Yankee, and 26 Commanwealth Edison."

DET Report, pp. 46-47.

(

1991 JUMA audit, September 13,1991, DET response item a0016.

Page V-33 The Liberty Consulting Group

Rib.'

Chapter Fis e - %1anagement and Organization i Relative to STP's QA surveillance prograr the 1991 JUMA concluded:" i 2 "Overall, the surveillance program has been developed and is being implemented in an 3 effective and productive manner. . . Review of surveillance packages indicate some 4 excellent field activity oveniew. This information has also proven useful to station 5 management."

6 The-1991 JUMA also commented on QA's audit program."

7 *

"The audit program and its implementation is effectively providing third level quality 8 oveniew at STP. This program appears to be satisfying technical requirements and is 9 supplying a useful product to quality and plant management."

10 Another nuclear utility initiative, the Cooperative Management Audit Program (C3fAP), conducted i1 an audit of STP in April 1992. This audit was conducted by personnel from Public Senice Electne 12 and Gas, Duquesne Light, and the Tennessee Valley Authority. "The purpose of this audit was to 13 verify the effective implementation of the Quality Assurance Program for Nuclear Assurance 14 Activities."' Overall, that audit concluded:"

b 15 " Based upon the audit results, the CMAP auditors have concluded thai Houston Light V 16 and Power personnel have effectively implemented their QA program. Further, Nuclear 17 Assurance personnel are providing "added value" to both program implementation and 18 station operation."

19 In September 1992 there was another JUMA audit at STP. This time the audit team consisted of 20 personnel from North Atlantic Energy Senices Corp., Duke Power Company, and Florida Power 21 and Light. The audit summary contained the following statement:t2 22 "The JUMA team has concluded tha: there are no significant flaws in the areas audited 23 and that positive change has been recently observed within Nuclear Assurance. This 24 change is noteworthy because it has been detected by quality personnel and by 25 individuals who interface with your quality organization."

1991 JUMA audit, September 13,1991, DET response item #0016, p. 2.

1991 JUMA audit, September 13,1991, DET response item 80016, p. 3.

1992 CMAP Assessment Report, May 26,1992, DET response item s0016, p.1.

i 1992 CMAP Assessment Report, May 26,1992, DET response item #0016 p. 2. '

O (j '

1992 JUMA audit, October 7,1992 DET response item #0016.

Page V 34 t The bberty Consultmg Group l

_~ - . _ - -_ - . - _ - -.. ..

RL5-:

Chapter Fhe - \tanagement and Organization i

The " positive change" and the positive direction referred to by the JUMA report referred to 2

personnel changes within the Nuclear Assurance organization. As to the specific efforts of QA and 3 its effectit eness. the JL'MA concluded:"

1 4

"The review of the audit and surveillance reports and checklists showed that the  !

evaluators are thorough in their work and demonstrate a good understanding of the 6

processes and programs being evaluated. In general the reports convey problems to the 7

'affected organization in a fashion that is easily understood and, therefore, easily 8

addressed. Deficiencies were often corrected during the surveillance or audit and 9

required no followup actions. In most cases where a Deficiency Report was issued, '

10 responses had been developed before the DR was sent out. This demonstrates an 1i attitude of cooperation among departments that promotes a thorough understanding of 12 deficiencies and what is necessary to adequately address the problems. Timely 13 completion of corrective action and verification of the activities is enhanced by these la endeavors."

15 NRC Assessments 16 In addition to outside evaluations from industry, the NRC routinely evalur.ted STP's quality 17 programs, including the effectiveness of QA. The NRC's SALP reports provide a summary and 18 overall assessment of these routine evaluations. The SALP that covered the period from February 4 19 1990 through May 1991 said:"

20 "The licensee's programs to assure quality, including the self assessment process, were '

21 . generally well implemented. QA audits were performance based. Contract auditors 22 were well utilized to supplement the licensee's QA staff."

23 The S ALP that covered the period from June 1991 through July 1992 reaffinned the same general 24 assessment of QA."

25 "The licensee's QA program relating to audits appeared to be well structured, with 26 organizational responsibilities and functions clearly defined. Audits were scheduled 27 and performed by independent and qualified personnel, including technical specialists.

28 The scope of audits was found to be comprehensive and audit findings reflected 1992 JUMA audit, October 7.1992, Attachment D, p. I, DET response item #0016.

NRC SALP report, ST AE-HL 92831, September 6,1991, p. 22.

h "

NRC SALP report, ST AE HL 93239, November 18,1992,p.23.

Page 4 35 The Liberry Coruultmg Group

h O

RU.:

Chapter Fis e - 31anagement and Organization i supportise and meaningful findings. Written responses to findings appeared to be i

' 2 timely.

3 DET Assessment 4 The DET said that its team had "found numerous records which documented QA's persistence in 5 attempting to gain management's attention to their safety findings." Of course, this is exactly 6 what QA should have been doing. The DET requested " examples ofissues identified by QA that 7 were not adequately fixed."" The DET also asked for "[m]emos or other documents showing that

8 QA or Nuclear Assurance elevated persistent, uncorrected findings to the attention of top 9 management. Replies, also, please." Liberty reviewed the responses to those requests.

10 in response to the first of these questions, the DET was provided with a package ofinformation Ii that identified QA audit findings that had not been completely resolved by the time a follow-up 12 audit was performed. However, the nature of these findings was far different from the impression I 13 given by the DET in its generalized list of " continuing weaknesses." For example, the la configuration management issue had to do with the program requirements, (specifically, the 15 responsibilities of an NSRB subcommittee on configuration management not temg clearly N 16 defined) rather than any particular examples of a lack of configuration control. Another issue 17 related to configuration management was that an audit had discovered some valves that were 18 required to be simply closed by plant drawings but were actually locked closed. These specific 19 matters were corrected during the audit. In another audit a year later, out of a sample of I 85 valves, 20 3 valves were not locked that shouM have been, and 4 others were not adequately locked.' These 21 findings were not unusual or particularly significant. The section of this report that discusses the 22 identification of problems by operations describes HL&P's response to deficiencies in the locked 23 valve program.

24 Concerning the use ofinformal documents like night orders to revise procedures, the follow-up 25 QA audit in 1992 actually stated that actions taken (in response to a 1991 finding) had been DET Report. pp. 46-47.

DET request #3651.

' DET request 83693.

i f

(

DET response item #3651.

Page W36 The Liberty Consultmg Group  ;

RLS.:

Chapter Fi$e - Management and Organization i effectis e. (Refer to the section in the operations chapter of this report that discusses control room 2 written guidance for more information on this issue.) Thus the DET raised an issue that had been 3 dealt with and resolved before the DET arrived.

4 With regard to " maintenance training, qualifications, and work activities," the repeated QA audit 5 fmdings dealt with the qualification of the fire brigade leader, the instructors used for respiratory 6 protection classes, and operator qualification to use self-contained breathing apparatus.

7 in a 1991 audit, QA raised a concem that focused on the number of reactor operators who had 8 maintained their qualifications as Site Fire Brigade members. In 1992, a similar concern focused 9 on the administrative controls necessary to ensure that the reactor operator who would fill the fire 10 brigade leader position would not also have critical watchstation duties.

I1 QA raised a concern in 1991 about the qualifications of the instructors used for general employee ,

12 training (GET) on the use of respiratory protective devices. QA had discovered an NRC document 13 that indicated training on such equipment should be given by health physicists, industrial 14 hygienists, or safety engineers. The people used for GET training had to cover a wide variety of topics and generally did not fit these classifications. The next year this matter had not been D 15 16 resolved, apparently because Health Physics and Training did not agree on the training the 17 iratructors should receive.

18 In 1991 QA noted that not all reactor operators had current qualification on the use of self-19 contained breathing apparatus. In 1992 QA again found that some reactor operators had either let 20 their respiratory gear fit test time exptre or had not demonstrated their qualifications on time.

21 These qualification concems appean d to be legitimate and appropriate for QA to identify. In each 22 case the responsible organization would readily admit that the matter should have been corrected 23 before it was noted in a follow-up audit. However, the significance of these matters on a relative 24 basis is not high, and none of them suggest unsuccessful persistent attempts to raise important 25 issues with top management.

26 The DET said that these repeated QA findings had dealt with the material condition of safety-27 related equipment. However, the information provided to the DET did not dentify any such 28 matters. QA noted that one of the washing machines used to launder protective clothing had not O

b Page V-37 The Liberty Consulung Group

l l

l K5.

)

Chapter Fne - %1anagement and Organization t I A been working for a long period of time. The washing machine for clothing should not have been I 2 equated with safety-related equipment.

3 l The second DET request asked for memos or other documents showing that QA had elevated 3

4 1 persistent and uncorrected findings to top management. When QA identifies a concern in one of i 5

its audits or other activities, it issues a deficiency report (DR). The initial issue is called revision 6 0. If-the response to the DR is not received on time, or if QA is not satisfied with the initial 7

response. QA issues revision I to the DR. Additional revisions (e.g., Rev. 2, Rev. 3) are issued 8

until the deficiency has been resolved. Revisions are issued to higher levels of management.

9 IIL&P's response to the DET consisted of 775 pages and included deficiency reports that had gone 10 past revision 0. The size of the response was impressive. However, the impression that is created 1I by the contents of the response is very different from the one that is created by its weight The 12 response included only 24 deficiency reports,17 of which were issued in 1990 or earlier. Only 4 13 of the 24 DRs had gone past revision 1 and only I of those 4 had been issued after 1990. The 14 following list summarizes the DRs provided to the DET

• 15 RfL Ktv. Subiect 16 87-081 5 Corrective actions 17 88 003 i Equipment qualification preventative maintenance 18 88-047 1 Verification of computer code 19 88-065 i Training of materials management personnel 20 88-104 1 Classification of procedure 21 88 140 1 Review of vendor technical manuals 22 88-161 2 Prompt notification system 23 88 162 1 Prompt notification system 24 88 167 1 Documenting deficient items 25 88-192 1 Evaluations for preventative maintenance 26 89-021 1 Nonconformance reports 27 89-071 i Engineering procedures 28 89-107 2 Preventative maintenance revisions 29 90-012 1 Instrument serpoint index 30 90-030 1 Security barners 31 90-035 1 Emergency Plan 32 90-038 1 Escorted visitor rules during diving operations 33 91-004 1 Self-contained breathing apparatus qualifications 34 91-010 2 Processing Technical Specification changes 35 91-013 I Timing of procedure change implementation 36 91-014 i Operator medical examinations 37 91-051 1 Cubicle cooler verification in surveillance procedures 38 92-007 1 Administrative control requirements 39 92-011 1 Speakout concern #12204 fm Q

\ "

DET response item a3693.

i Page W38 The Liberry Consuhmg Group I

i

n s-;

Chapter Fise - \1anagement and Organization i Liberty concluded that the subject and nature of the DRs were not an indication of unreasonable 2 management but indicated thorough, although rarely needed, follow-up by a QA organization.

3 D. Independent Safety Engineering Group (ISEG) 4 In the section ofits repon on safety assessment and quality oversight, the DET made the following 5 comment on the ISEG."

6 "The ISEG review for lessons teamed was not fully effective due to weak management  !

7 support as discussed earlier. Less than 50 percent of the OER documents that required 8 review for applicability to STP systems had received any review by ISEG. l 9 Approximately '00 OERs were open in April 1993. Many reviews had been 10 incomplete or did not address the industry identified problems or recommendations.

11 Additionally, there had been 5 ISEG directors in the previous five years and, 12 frequently, as was the case during the evaluation period, this position was filled in an 13 " acting" capacity." ,

14 Liberty found that literally hundreds of OER (operating experience report) documents were 15 received by STP, and that none of the status reports given to the DET indicated that ISEG had not b

(

16 completed its applicability review of any OERs. In fact, it is obvious from these status reports that 17 ISEG had to have completed such a review for these many documents." While ISEG admitted that 18 it had not completed this initial review for some OERs within its own procedural guideline of 19 seven days, the screening had otherwise been accomplished in accordance with procedure IP-20 2.2Q."

21 Liberty examined the DET's comments about 300 OERs being open in April 1993. The DET J

22 received a status report on open OERs and apparently counted a total of 305 open OERs." What 23 was apparently overlooked was the fact that many of the OERs in the status report were repeats.

24 After ISEG's initial screening, there are often several actions required with the same or different 25 due dates and d;fferent departments for a single OER. For example, a plan of action may have been 26 required by both Design Engineemg and Plet fngineering in response to an NRC IE Notice. l i

l I

DET Report, p. 47.

" DET response items nos. 1000,3097, and 3681.

" DET response item #3600.

" DET response item 83681.

) u, m, c_% c,

RLS.'.

Chapter Fise - Management and Organization i

Each of these was tracked separately in HL&P's system. But in this example there was only one I 2

open OER not the two apparently counted by the DET. So instead of 305 open OERs there were 3

actually only 139. Other data provided to the DET connrmed that the 300 figure could not have 4

been correct." In any case, the total number of open OERs is only an indication of the volume of 5

work required. not the effectiveness of the ISEG or STP as a whole in handling OERs. A better 6

measure is the number of OERs that were not completed by the scheduled date. Of the 139 open 7

OEP.s, only 22 were behind HL&P's own schedule for completing the actions or reviews. This 8

number had grown from 5 overdue as of February 1,1993, and the growth was not surprising.

9 considering the fact that a two-unit outage and the DET's evaluation were occurring in the to February through April period.

1I These OERS were assigned by ISEG to various departments within the STP organization, which 12 were to develop or review the plans of action to deal with the various reports received from the 13 NRC, INPO, and other sources." In addition to the open OERs discussed above, there were la another 124 OERs that had completed departmental review, but for which ISEG had not prepared 15 and sent to records management the OER completion package. This latter category of OERs was 16 where ISEG's backlog existed, but items only went into this category after initial screening by

(

\

17 ISEG and preparation of the plan of action by the responsible department. Bus while the records 18 may not have been in the vault, people on the site were aware of these operational experiences and 19 had taken actions as appropriate. ISEG's fmal review could determine that additional actions were 20 necessary, and the number requiring fmal processing was large, but these reports had been 21 reviewed.

22 Finally, the very nature of the ISEG, an NRC-mandated organization, is such that the entire STP 23 organization is best served if personnel are rotated through the ISEG and then sent back to other 24 parts of the organization. This rotation plan is explicitly stated in STP's procedures for the ISEG."

25 The cunent ISEG director was in an acting capacity because the regular director had temporarily 26 taken charge of the group that was investigating the auxiliary feedwater pump overspeed trips. This 27 temporary assignment was a logical extension of the type of work performed by the ISEG for a 2s focused investigation of an important matter.

" DET response items nos.1000 and 3097.

• DET response item 83681.

" DET response item *0006.

Page b40 The Libery Consultmg Group

. _ _ _ _.- _ ___ -- - __ _m . . - __ _ _ .. _-

i l

RLS: 1 Chapter Fise - \tanagement and Organization i VII. Information Systems

A. Information Systems Planning and Progress i 3 The DET was critical of STP's information systems. However, some of the DET's remarks were 4 simply copied from STP's Long Range Information Systems Plan fLRISP). For example, the DET 5 said;"

I 6 "There was no interactive (sharing of data) interface between the different computers 7 which meant that similar data was duplicated on different computers. This method of  ;

8 managing data was inefficient and increased the probability for error due to multiple '!

9 entry at different time intervals." i 10 Over a year before the DET even started its assessment, STP's LRISP included the following;"

I 1i "There is currently no interactive (sharing of data) interface between the different , ,

t2 computers which means similar data is duplicated on different computers. This method  !

1.1 of managing data is inefficient and increases the probability for error due to multiple 14 data entry at different time intervals." i i

15 The DET report continued;"

N ,

16 " Additionally, STP was experiencing significant delays in accessing data from its main 17 Computer system due to hardware and processing limitations."

is STP's LRISP had already reported;"

19 "STPEGS cunently is experiencing delays in accessing data from Prime due to 20 hardware (access channels) and processing limitations."  :

i 21 The DET also said that "(i]nformation systems, including Management Information Syste-is, did  ;

22 not adequately support and in some cases were an impediment to plant operations, work control l

l i

l

" DET Report, p. 47.

STPEGS Long Range Information Systems Plan, January 9,1992, p. 2, DET response item #0029.

" DET Repon, pp. 47-48.

( "

STPEGS Long Range Information Systems Plan, January 9,1992, p. 3, DET response item =0029.

Page W41 ',

The Liberty Consuhing Group

'1 l

PL.L:

Chapter Fne \lanagement and Organizatmn A

(O) I and other management functions. Although this had been presiously identified by the licensee. this  !'

2 situation detracted from station performance over several years."";

1 3 Liberty found that HL&P's 1991 review of the current information systems had detemiined there a

was a need for improvement in the areas of: (1) multiple computer platforms, (2) computer 5

} response time, (3) user access, (, redundant data, and (5) documentation on computer I 6 applications." While these areas were substantial, HL&P did not conclude that the existing 7

, systems "did not adequately support operations, work control, and management functions."

3 l l 1

8 In recognition of both the evolving information needs at STP and the capabilities of computer  !

9 sy stems, STP established a steering committee to oversee the development of the LRISP in April 10 1991. The membership of the steering committee indicated STP's recognition of the importance ii of the task as we'l as the fact that the decisions made would affect many different groups at STP.

12 The committee included the Group Vice President, Nuclear; the Vice President, Nuclear 13 Generation; the Vice President, Nuclear Engineering; the Vice President, Nuclear Suppon; and the' la General Manager, Information Resources."

(n) 15 11e design ofinformation systems that will meet regulatory and critical business needs as well as 16 optimize characteristics in efficiency, data conaol and access, and cost-effectiveness for a nuclear 17 power plant is not a simple task. HL&P concluded that due to the complexities involved and the is limitations on the current computerprocessors, there were no short-term options that should be 19 implemented. Initially, a two-year implementation of an integrated data base was seen as 20 necessary. The total plant system included components of functional data bases for work 21 management, action tracking, purchasing and materials management, chemistry and radiation 22 protection, personnel, financial, records management and document control, and component data."

23 During 1992, STP canied out the initial phases of the LRISP. These efforts were well documented.

24 STP's Master Operating Plan for 1992 specifically called for the implementation of the LRISP.*

'2 DET Remrt, p. 47.

STPEGS Long Range Information Systems Plan, Ja:cary 9,1992, DET response item a0029.

STPEGS Long Range Information Systems Plan, Ja aary 9,1992, DET response item =0029.

STPEGS Long Range information Systems Plan, January 9,1992, DET response item =0029.

tg'\

( ,/ Master Operating Plan - 1992, MOP il, Rev. 0, p.16, December 26,1991.

Page V-42 The Liberty Consulang Group

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - . . _ _ _ _u

RLS.'

Chapter Five - Management and Organization i Because of the importance of the work and the care that must be taken to get the desired results.

HL&P hired Andersen Consulting to provide periodic quality assurance oversight of the 3

implementation. These reports from Andersen Consulting showed the progress that was being 4 achieved and the care being taken in the implementation."

5 The 1993 Master Operating Plan included more specific actions to be taken as part of the LPJSP.

6 These included upgrading of the mainframe computer, replacing terminals with work stations 7 (PCs), performing initial system integration testing, reviewing work processes for functional data 8 bases, conducting training, and completing data transfers." STP's 1993 budget included multi-9 million dollar specific O&M and capital expenditures for the LPJSP. O&M included software and to consulting costs and the capital expenditures were for hardware costs." The LRISP was included ii as "special scope" items in STP's long-range expenditure plan with $1 million per year or greater 12 planned for years 1994 through 1997.'*

13 The DET acknowledged some actions by HL&P.' '

14 "The licensee was in the process of purchasing a new computer program directed at p 15 improving information systems. However, management's errors in establishing the

(' 16 current system were being repeated in the information system improvement program 17 in that input and feedback from the end users was not being adequately incorporated."

is HL&P was in the process of implementation of the INDUS Passport software, an integrated 19 package that was aimed at several of the areas identified in the LRISP study as needing 20 improvement, including response time, user access, redundant data, and documentation? The 21 process of putting the INDUS system in place was not as simple as " purchasing a new computer 22 program." In fact, Andersen Consulting had been hired just for periodic oversight of the Andersen Consulting to HL&P, Miers to Hall, August 7,1992; Andersen Ccasultmg to HL&P Miers to Hall, October 20,1992; contained in DET response item #4025 - CPL-154.

Master Operatmg Plan - 1993, MOP-!!!B Rev. 3, p.15-1.

DET response item =4025, CPL-25.

DET response item =0013.

DET Report, p. 48. I d '"

DET response item *0063.

Page W43 l The Laerty Consultmg Group l

RLS .'

Chapter Fhe - %1anagement and Organization r

1 implementation effort.'" Andersen discussed the need to " regain and maintain the confidence of 2 on site personnel"in the new system and the effective way in which that was done at STP. The 3 DET requested and was provided with inputs from end-users of the new systems as to how the data a bases should be customized to meet particular users' needs.'

• The DET also requested copies of 5 memos that documented potential problems or inadequacies of the new system. The DET 6 concluded that those memos indicated end-user needs were "not being adequately incorporated."

7 Libetty found, however, that those memos were evidence that end-users' needs were being 8 solicited and factored into the new system.

9 B. DET Examples to The DET report listed nine " weaknesses in information systems."'" The first of these items was 11 "[e]quipment history records were incomplete and approximately eight months behind in being 12 updated." It could not be determined what information the DET had used to conclude that-13 equipment history records were eight months behind in being updated. However, the DET 14 requested the equipment histories for many components and systems during the course ofits O 15 review. A review was conducted of the equipment histories provided to the DET. The review was 16 terminated after the first 20 items because there were entries in the histories very close to the date 17 of the request. This ilid not prove that there were no matters more than eight months old that is needed to be added to the histories. However, Liberty wanted to determine whether equipment 19 histories failed to contain information from the last eight months, as alleged by the DET. This 20 proved not to be the case. The results of the review of the first 20 DET items are shown in the 2i following table.

1

'" Andersen Consulting to H1,&P. Miers to Hall, August 7,1992; contained in DET response item a4025 -

CPL-154 C ""

DET request #0063-001.

k '"

DET Repor:,, p. 48.

Page V 44 The Liberty Consultmg Group j i

l

RLS.' .

Chapter Fise - Management and Organuation

)

T Equipment Last Histor) 1 DET Reauest = Ecuirment Histon Date Entn Date 2 2103 Cire. water discharge sahes 4593 2'18 93 3 2117 Essential Chiller 004 4'5 93 32293 4 2130 CCW pump IB 4 6.93 2 3.93 5 2135 FW system 4693 32593 6 . 2137 Sequencers 4/6/93 3'24'93 7 2139 inverter 46 ~93 32793 8 2145 Essential Chiller 001 4 6c93 4 2/93 9 2145 Essential Chiller 002 4'6.93 4293 10 2145 Essential Chiller 003 4 6/93 42/93 11 2145 Essential Chiller 005 4/6/93 4'l c93 12 2145 Essential Chiller 006 4693 42/93 13 2145 Chill w ater pump 005 4i6/93 3/26,93 14 2145 Chill water pump 006 4 6/93 1/16 93 15 2145 Panel ZLP625 4 6/93 3 21/93 16 2145 Pressure indicator 4 6'93 1/16/93 CN 17 2145 Temperature indicator 4'6'93 20293 18 2177 AF system 4/8'93 2.27/93 19 2178 CH system 4/8'93 3.21'93 20 2179 DG system 4/8/93 4'5 93 21 2193 DGs 4/26'93 3/31'93 22 The other eight items either were not explained completely enough to allow confirmation or had 23 more to do with management's choice of allocating resources than information systems. In one 24 case (" acquisition of parts information was cumbersome, slowing down maintenance work 25 package preparation"), the DET's statement was consistent with previously identified findings.

26 Another ("[c]omputer assistance to aid the system engineer in documenting and trending system 27 performance and condition was not generally available") was at odds with the fact that 80 percent 28 of the system engineers had personal computers."

29 Liberty concluded that STP's management made reasonable decisions and took reasonable actions 30 related to STP's information systems. STP developed and was implementing a plan for upgrading 31 computer systems. This effort required considerable resources, was being carefully planned and A

DET response item #3178.

PageV 45 The Liberty Comultmg Group

l RLS.:

Chapter Fise - \tanagernent and Organization

, i monitored, and required a significant amount of time for implementation. None of the examples '

2 in the DET report demonstrated unreasonable management decisions or actions.

I I

O l

l 6

Page v.4e The Liberty Comuhang Group

-- . .__ . _ .