ML20206G019

Forwards Diagnostic Evaluation Team Rept for Plant on 880822-0902 & 0912-16.Weaknesses in Performance Noted.Root Causes of Licensee Continuing Poor Performance Listed. Integrated Plans for Improvement Requested within 60 Days

ML20206G019
Person / Time
Site:	Fermi
Issue date:	11/16/1988
From:	Stello V NRC OFFICE OF THE EXECUTIVE DIRECTOR FOR OPERATIONS (EDO)
To:	Mccarthy W DETROIT EDISON CO.
Shared Package
ML20090K093	List: ML20206G019
References
NUDOCS 8811210466
Download: ML20206G019 (3)

Text

.y 4 a

& MGr UNITED STATES

[0,j

[n E

NUCLEAR REGULATORY COMMISSION

• WASHINGTON. D. C. 20555 o

,,,,, November 16, 1988 Docket No. 50-341 Detroit Edison Company ATTN: Mr. Walter J. McCarthy, Jr.

Chairman and Chief Executive Officer 2000 Second Avenue Detroit, Michigan 48226

SUBJECT:

DIAGNOSTIJ EVALUATION TEAM REPORT FOR FERMI ATOMIC POWER PLANT This letter forwards the Diagnostic Evaluation Team Report for the Fermi Atomic Power Plant. The onsite evaluation was conducted over the period August 22 to September 2, 1988 and September 12 to September 16, 1988, by a taam of NRC headquarters and regional evaluators with team leadership and support provided by the Office for Analysis and Evaluation of Operational Data. As you are aware, this is a new NRC evaluation tool that is intended to provide an independent assessment of licensee safety performance. Following the conclusion of the onsite evaluation, the findings were discussed at an exit meeting with you and other company executives and managers on November 1, 1988.

The NRC effort irvolved a broad-based evaluation of overall plant operatio1al perfonnance and tna capability of Detroit Eriison company management, policies, and prog"ams to improva plant operations at Fermi. P&rticular attention was directed in the areas of management and organization, operations ard training, maintenance, testing, quality programs, and erigineering surport.

Based upon the entensive evaluation, both onsite and through suosequenL analysis, the team observed rec 1nt imprnuments in Fermi's perferrance and capabilitics. yot identified a number of weaknesses that, require additionel attention and involvement by DECO management. The team concluded that tl1e root causes of Fermi's continuing poor parformance and appareni, inability to sustain improvements were: (1) a protracted design and construction period, (2) the failure of management to adequately and offectively plan for the transition from a design and construction project to an operating plant, (3) lack of BWR operating experient:e throughout the organization, and (4) management slowness in determining and implementing effective solutions.

Further, the team concluded that, although the essential elements needed to achieve future improvements are now largely in place, some areas need additional management attention to increase the rate of progress and assure continued success. These included the need to: (1) achieve organizational stability as soon as possible, (2) improve effectiveness of first and second line supervisors, (3) improve organizational climate, (4) fix fragmented and overlapping engineering support responsibilities, (5) fix known equipment problems. (6) set priorities according to plant needs, (7) allocate resources to selected areas and better utilize existing resources, and (8) improve effectiveness of operator training programs. Section 2 of the enclosed report G311210466 831116 PDR ADOCK 05000341 il(i P PDC

e.

Mr. Walter J. McCarthy, Jr. provides the team evaluation results which include findings and conclusions and root cause analysis. Section 3 of the report provides the detailed eva'uation results. Some of these items may be potentiei enforcement findings. Any enforcement actions will be identified by our Region III office.

To underscore my concern and interest in Fermi's safety performanco, Mr. J. M. Taylor, Deputy Executive Director for Regional Opera *. ions toured the Fermi plant and attended the exit meeting, as my personal representative.

Mr. Taylor reported that your responses to the findings were positive and the meeting discussions were very constructive. In addition to the findings, conclusions and root causes discussed above, a question was raised regarding the status of planning and preparations for the 1989 refueling. It was suggested that planning should already be underway. Further, it was suggested that experience from utilities that had highly successful refuelings should be actively sought and utilized.

It is my view that the key elements of Fermi's new management team are essentially in place and that this team has a good understanding of the underlying causes of Fermi's performance problems. I believe that it is important for you and the Fermi management team to carefully review the enclosed report, with special emphasis on the eight areas identified above as requiring additional management attention. Following your review, I request that the Fermi management team determine the actions needed to address each of these areas giving due consideration to the team's evaluation results identified in Section 2 of the report. As part of your evaluation, I believe that it is essential that CECO define its own priorities and balance its resources in an integrated manner which will provide the most rapid and sustained imorovement in Fermi's safety perform 1nce. It is furtner reques, tad that you provide my office with your integrated plans for improvement within 60 days of the date of this letter. Following our evaluation of your response, the appropriate NRC senior manacers will meet with your management team to discuss DECO's plans for improvement so that we may fully understand and support your plans and actions.

In accordance with 10 CFR 2.790(a), a copy of this intter and tha enclosure will be placed in the NRC Public Document Room. Shauld you have any questions l concerning this evaluation, we wuuld be pleased to discuss them with you.

1 Sincerely, f . _ -. 0~

%4 # ..

6torStbo Executive Director for Operations 1

Enclosure:

l Diagnostic Evaluation Team Report  !

for Fermi Atomic Pc'.ar Plant cc w/eoci: See Page 3

. e

< '=

Mr. Walter J. McCarthy, Jr. cc w/ enc 1:

Ronald C. Callen Michigan Public Service Commission John Flynn, 0"N Nuclear Faci 11cies and Environmental Monitoring Section Office Division of Radiological Health Walt Rogers, USNRC Monroe County Office of Civil Preparedness A. Bert Davis, USNRC/RIII Lynn Goodman, DECO Distribution w/encls:

<-DCS- PDR LPDR EDO R/F JClifford AE00 RF 00A R/F DEIIB H/F ELJordan, AE00 C.JHeltemes, AE0D RLSpessard, AE00 SDRubin, AE00 DA111 son, AE00 HABai?ey, AE00 CWBurger, RII MECashatt, TTC RGFreeman, AE00 LGarner, RII SCGuthrie, NRR EJLeeds, AE00 ATHowell, RIV WDLanning, NRR RLLloyd, AE00

} TEMurley, NRR RLPerch, AE00 2PStewart, RIV ABSutthof f, NRC Contr actor DJSulliven, AEOD MTaylor, EDO (w/o enc 1)

PThurmond, NRC Contractor JFMert, N'lC Contractor KPWolley, AE00 FIYoung, RI MCalahan, CA JFouchard, PA WTRussell, Regional Administrator, RI MLErnst, Acting Regional Administrator, RII ABDavis, Regional Administrator, RIII RDMartin, Regional Administrator, RIV

'3 Martin, Regional Administrator, RV M V '/ ' N 8f 0FC :00 TIDE IB,

AE00 D0A :AE00 :AE0D ~

p /

l': ~D0  :  :

DATE :11///88 :11/[/88 :11/g/88 :11/7/88 :11/h88  :  :

0FFICIAL RECORD COPY

,..s DIAGNOSTIC EVALUATION TEAM REPORT FOR THE l

l FERMI ATOMIC POWER PLANT )

. I November 1988 l 1  :

I i  :

l l l

l l

l i l

l i

i U.S. Nuclear Regulatory Comission +

Office for Analysis and Evaluation of Operational Data I Division of Operational Assessment  :

Olagnostic Evaluation and Incident investigation Brcnch l l

G311210467 881116 I PDR ADOCK 05000341 PDC P

i

o .* .

. s.

OFFICE FOR ANALYSIS AND EVALUATION OF OPERATIONAL CATA DIVISION OF OPERATIONAL ASSESSMENT Licensee: Detroit Edison 4

Facility: Fenni Atomic Power Plant Location: On Lake Erie in Monroe County, Michigan, 8 miles East-Northeast of Monroe, Michigan Docket No.: 50-341 Evaluation Period: August 21, 1988 through September IS, 1988

'eam T Manager:

R. Lee Spessard, AEOD Deputy Team Manager:

Wayne D. 1.anning, NRR Team Leaders:

Dennis P. Allison, AE00 Henry A. Bailey, AE00 Team Members:

Charles W. Burger, RII

,Nevlin E. Cashatt, TTC Robert G. Freeman, AEOD Larry W. Garner, RII Steven C Guthrie, NRR (Hrst Week)

Ac9cr T. Heuell, RIY Eric J. Leeds, AEOD

, Ronald L. Lloyd, AEOD Pobert L. Perch, AE00

. J. Perry Steriart, R V Kevin P. Wolley, AE00 Francis I. Young, F,1 Consultarts:

Ann B. Sutthoff, NRC Contractor Paul Thurmond. NRC Contractor Jonathan M. Wert, Jr., NRC Contractor O

Submitted by: ' .  ?-*" h P/

R. Lee 5pessa4?"tam Manager Date Approved by:

Eoward L v2sk ordan,lirector,AEOD ~

///7, Date L +

O o

} .

Q EXECUTIVE

SUMMARY

During the NRC senior management meeting in June 1988, NRC executives recommended that a diagnostic evaluation be conducted at the Fermi Atomic Power Plant (Fermi). This recommendation was based primarily on the continued poor plant performance as reflected by the number of Category 3 ratings given by the last two Systematic Asses: rent of Li':ensee Performance (SALP) Boards. Tha functional areas of Operations and Surveillance were rated Category 3 by both Boards. Performance problems continued at Fermi after the last SALP period that ended on March 31, 1 18. Although Detroit Edison had completed a number of major organizational changes and initiated several programs to improve performance, there was an apparent inability to sustain improvements. Questions also remained concerning the plant's material condition as well as management and orgar :"*ional effectiveness. ,

Based on there concerns, and the recommendation of the NRC senior managers, the Executive Di fctor for Operations (ED0) directed the Office for Analysis and Evaluation or'd,.erational Data (AE00) to conduct a diagnostic evaluation at Fermi to identify the underlying causes for the licensee's continued poor performance and apparent inability to sustain improvements. The EDO directed ths; a Diagnostic Evaluation Team conduct a broad-based evaluation of overall pida operatic 6a1 performance and the capability of Detroit Edison management, policies and pegrams to improve plant operations at Fermi.

A 19 member team spent a total of three weeks at the Fermi site during August and September 1988, evaluating the functional areas of management and ciganization', operations and training, maintenance, testing, quality programs and engineering support. Based upon the team assessment,' it was concluded that 1"e root causes of Fermi's continued poor performance and apparent inability to nstain improvements were: (1) a protracted design and ennstruction period, (2) the failure of management to adequately and effectively pian for the trans1 tion from a design and construction project to an operating plant, (3) lack of BWR operating experience throughout the organization, and (4) management slowness in determining and implementing effective solutions.

~

Overall, the team observed recent improv wents in Fermi's performance and capabilities, yet identified a number of weaknesus that require additional attention and involvement by Deco management. Initially, Deco management was slow in taking aggressive and effectiu action to fill key positions with professionals having extensive nuclear plant operations experience, and to implement site specific policies and programs to improve performance and accountability. However, actions taken in this regard during the last two years represented significant accomplishments and provided the essential elefnents needed to achieve future improvements. Further, the team concluded that the actions being implemented at Fermi generally address the causes for performance problems while maintaining an acceptable level of opert.tional safety. Notwithstanding these actions, the team determined that some areas needed additional management attention to increase the rate of progress and assure continued success. These included the need to: (1) achieve organizational stability as soon as possible, (2) improve effectiveness of first and second line supervisors, (3) improve organizational climate, (4) fix fragmented and overlapping engineering support responsibilities, (5) fix known equipment problems, (6) set priorities according

~

i

i, to plant needs, (7) allocate resources to selected areas and better utilize existir.g resources, and (8) improve effectiveness of operator training programs.

The team's major findings and conclusions for each of the functional areas evaluated are summarized below.

(1) Personnel changes had strengthened the management team and new programs had shown positive results in some areas. However, organizational instability, p3rticularly in the engineering support and maintenance areas, was found to be a continuing problem. The large number of new managers and change in management philosophy, resulted in the coexistence of two organizational cultures (old and new), causing some adverse impacts on morale, productivity, and employee relations. A lack of schedule integration contributed to manpower planning and forecasting problems, led to reactive management and reduced availability of safety systems. Resources were also being strained to support safe plant operations, while.

attempting to implement the improvement programs on schedule. In setting priorities, management was often found to be more sensitive to external influences than to plant neads and generally lacked a self-confident, take-charge attitude with regard to these influences.

(2) In Operations, a sense of ownership, accountability, and profes-sionalism was evident. Operator performance had improved, and.thair morale was good. There were also a number of weaknesses. For example, the ability of the control rocm operators to achieve a high standard of perfor .ance w6s adversely affected by the continued lack of experience at a well-run BWR. Fermi management also had not effectively utilized the Shift Operations Advisors to help develop

• this higher st6ndard. Further, Operations management did not reflect a broad safety perspective and their oversignt of routine daily plant activities was weak. Excessive challenges to the operating crews resulted from unreliable and unavailable equipment. The operator training program was weak.

(3) in Maintenance, improvements were noted in communications with operations and decreasing the corrective maintenance (CM) backloo.

However, significant efforts remained before maintenance could be fully effective in improving equipment availability and reliability.

The preventive maintenance program had not yet been optimized in this regard. Further, the trend and analysis program for equipment failures had not been effectively implemented. Planning and scheduling were fcund to be ineffective, and a lack of spare. parts continued to be a problem. Inadequately trained contractor craf t personnel, insufficient System Engineer staffing and the continued Maintenance Department restructuring were factors also slowing the rate of improvement.

(4) Tescing was found to have a number of strengths including surveillance scheduling and tracking, administrative controls for inservice testing and surseillance test procedure reviews.

Weaknesses included the failure of management to require implamentation of procedure changes resulting from the Technical Specification Improvement Program, the failure to track check valve 11

l'o .  :

O test failures, and the lack of a process to systematically identify l UFSAR commitments for testing and assure their implementation. l (5) In Quality' Programs, QA audits were well-planned and showed concerted  !

effort by management to shift toward performance-based reviews. .

Weaknesses in the QA organization included a lack of BWR operating l experience below the supervisory level and surveillance findings that I were limited to administrative or compliance issues. The Deviation ,

Event Report System, although adequate, had weaknesses in the tracking and trending systems. Weaknesses were also identified in  !

root cause .inalyses. The Board Nuclear Review Committee and the  :

Nuclear Safety Review Group were proactive and provided a strength to  :

the Fermi organization. However, the Independent Safety Engineering  !

Group was not performing some technical specification required functions, r (6) In Engineering Support, the Nuclear Engineering staff appeared well L qualified technically and had positive attitudes abcut improving Fermi performance. However, weak work interfaces and communication problems contributed to fragmented and overlapping engineering <

support for the plant. Systems Engineers were overcommitted and  !

sometimes designed tempo try modifications which should have been L designed by Nuclear Engineering. The engineering groups were often i slow to resolve plant deficiencies and occasionally provided inadeouate resolutions. A specific example was the motor-operated  :

valve torque switch problem.  !

i k

I

i j .

4 1

i n

I a

1 iii

O TABLE 0.e CONTENTS Page EXECUTIVE

SUMMARY

........................................................ i ALR0NYMS................................................................. v

1.0 INTRODUCTION

........................................................ I 1.1 Background................................... ................ I 1.2 Scope and Objectives...................................... .... 4 1.3 Methodology............................................... .... 4 1.4 Plant Description.............................................. 5 1.5 Organization................................................... 5 2.0 EVALUATION RESULTS.................................................. 9 2.1 Findings and Conclusions....................................... 9 2.2 R o o t C a u s e A n a ly s i s . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.0 DETAILED EVALUATION RESULTS......................................... 21 3.1 Management and Organization.................................... 21 3.2, Operations..................................................... 37 3.3 Maintenance and Modification................................... 51 3.4 Testing........................................................ 67 3.5 Quality Programs and Administrative Controls Affecting Quality. 77 3.6 Engineering Support............................................ 89 4.0 EXIT MEETING........................................................ 106 A P P E N D I X A - Ex i t M e e t i n g S u mma ry . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 APPENDIX 8 - EDO Direction to Diagnostic Eva luation Teara. . . . . . . . . . . . . . . . . 124 iv

, o , .)

'O

's

.( ACRONYMS AEOD Office for Analysis and Evaluation of Operational Data ALARA As Low As 9easonably Achievable ALS Abnormal Lineup Sheet ASME American Society of Mechanical Engineers ATWS Anticipated Transient Without Scram AMA American Management Association AWP Annual Work Plan BNRC Board Nuclear Review Committee l BOP Balance-of-Plant

BWR Boiling Water Reactor CAC Control Air Co'epressor CECO '

Central Component CEO Chief Executive Officer l CM Corrective Maintenance .-

l CRD Control Rod Drive l CST Condensate Storage Tank l CTP Core Thermal Power l' de Direct Current dp Differential Pressure ,

Deco Detroit Edison Company .

DER Deviation Event Report DET Diagnostic Evaluation Team- .

ECR Engineering Change Request EDG Emergency Diesel Generstor i '

EDO Executive Director for Operations

(, EDP Engineering Design Package l EDP Engineering Design Package EECH Emergency Equipment Cooling Water E0P Emergency Operating Procedures EQ Equipment Qualification EPRI Electric Power Rer,earca Institute ESF Engineered Safety Feature ESFAS ESF Actuation Signal ,

FBP Fermi Business Plan FIP Fermi Interfacing Procedure, FMD Fermi Management Directives .

FW Feedwater GE General Electric '

GETARS General Electric Transient Analysis Recording System v

p

-~ - .

.~ .

p

g, k_. . t

ACRONYMS (Continued)

HP Health Physics .

HPCI .High Pressure Coolant Injection HPES Human Performance Evaluation System HVAC Heating, Ventilation and Air Conditioning I&C Instrumentation and' Control IAS- Interruptible Air System -

3" r

IE Inspection and Enforcement  !

INP0 Institute of Nuclear Power Operations -

IOC Independent Overview Comittee ,

. ISEG Independent Safety Engineering Group , ,

ISI ~ Inservice Inspection IST Inservice Testing  :

LC0 Limiting Condition.for Operation LDP Leadership Development Program .

LER Licensee Event Report l LLRT Local Leak Rate Testing LOCA Loss-of-Coolant Accident LPCI Low Pressure Coolant Injection Low Power Set Point LPSP LS Limit Switch MART Maintenance Assistance Review Team M&M Maintenance and Modification M&TE Maasuring and Test Equipment  ;

~

MMS Material Management System MOV PoMr-0pmted (alve MRB  !.4anagement f.aylew 50ard MST Mcintcaance suppor.i Technician MVL '1 aster Valve List  ;

NASS Nuclear Assistant Shift Supervisor l NE Nuclear Engineering  ;

N!AS Non-Interruptible Air System Nt'.M Nuclear Material Management L NOIP Nuclear Operations Improvement Program NPPO Nuclear Power Plant Operator -

NPRDS Nuclear Plant Reliability Data System NQA/PS Nuclear Quality Assurance and Plant Safety .

NRC i Nuclear Regulatory Comission NRR Nuclear Reactor Regulation NSRG Nuclear Safety Review Group

- NSO Nuclear Supervising Operator NSS Nuclear Shift Supervisor O&ti Operations and Maintenance OSR0 Onsite Review Organization vi

ACRONYMS (Continued)

PASS Post-Accident Sampling System PDC Potential Design Change PE Preliminary Evaluation PEP Performance Evaluation Program PI Position Indicator PM Preventive Maintenance PRIDE People Really Involved to Develop Excellence PST Performance Scheduling and Tracking QA Quaiity Assurance QA/PS Quality Assurance and Plant Safety QC Quality Control

• RACTS Regulatory Action Commitment Tracking System RBCCW Reactor Building Closed Cooling Water RCIC Reactor Core Isolation Cooling RHR Residual Heat Removal R0 Reactor Operator R0!P Reactor Operations Improvement Program RPM Revolutions Per Minute RSCS Rod Sequence Control System RWCV Reactor Water Cleanup RWM Rod Worth Minimizer SALP Systematic Assessment of Licensee Performance SE Safety Evaluation SER Safety Evaluation Report SIL Service Information Letter SLC Standby Liquid Control 50ER Significant Operating Event Report SOA Shift Operations Advisor SPF Surveillance Performance Form SPRS Spare Parts Reference System SR0 Senior Reactor Operator SSFI Safety System functional Inspection SST Surveillance Scheduling and Tracking STA Snift Technical Assistant SWHI Stone and Webster Michigan TE Technical Engineering .

TS Technical Specification TSIP Technical Specification Improvement Program UFSAR Updated Final Safety Analysis Report yde Volts Direct Current WR Work Request vii

.. l

1.0 INTRODUCTION

1.1 Backaround -

As construction of Fermi 2 neared completion in early 1985, the NRC had a high  !

degree of confidence that Detroit Edison Company (Deco) would operate the Fermi 2 plant safely. This confidence was based on a number of factors, .

" including the licensee's cooperation and attention to NRC concerns, its responsiveness to NRC recommendations to undertake and accomplish changes,  ;

placement of top management at the site, and evidence that operators were ,

properly trained by the high success rates on their exec.instions.

i On fiarch 20, 1985, NRC issued a low-power license that permitted Fermi 2 to

operate at power levels up to and including'5 percent of rated thermal power.

2 During low-power operation, a few scrams and engineered safety feature (ESF) j actur.tions occurred, but none was considered significant. When compared to j other newly licensed plants, the experience at Fermi was viewed as typical and not indicative of future potential troubles beyond those typically encountered at new plants. At the full-power licensing meeting on July 10, 1985, the Comission and the staff considered Fermi 2 to be a "model" facility and cited

many areas deserving recognition. On July 15, 1985, NRC issued Fermi 2 a license permitting full-power operations. As a license condition, the licensee

was required to implement a Shift Operations Advisor (50A) program to compensate for a recognized lack of operating experience.

1 After issuance of the full-power license, NRC learned of an event concerning an out-of-sequence control rod pull resulting in an inadvertent criticality that p occurred on July 2, 1985. Beginning with this event and over the next three i months, NRC's confidence i'n the licensr$'s ability to operate the plant V

decreased. The frequency of reportable events increased during this period of t the low-power operation and the basis for these avents indicatej major problems

! in the same areas that were previously cited at .1oteworthy. During a special

! operational readiness assessment team inspection held in August-September 1985.

J identified weaknesses included poor comunication between management and snif t i operating personnel, lack of teamwork among the control room operations staff, and difficulty integrating the Shif t Operations Advisor and Reactor Engineering 1

)i roles with those of shift operating personnel. As an approach to resolving these problems, the licensee implemented corrective actions in a plan i identifiedastheReactorOperationsImprovementPlan(ROIP). The ROIP addressed these weaknesses and was expected to lead to fewer operational l occurrences and Technical Specification violations. Despite these efforts of i the licensee to improve performance, events involvang operational errors and degraded equipment continued to occur. Additionally, programatic weaknesses were identified in the areas of engineering and security, l As NRC concerns continued, the NRC Regional Administrator for Region !!! issued a letter pursuant to 10 CFR 50.54(f) in December 1985 and requested that the

licensee evaluate and address management weaknesses, develop a comprehensive 4 plan to ensure the readiness of the facility to operate , and identify the actions necessary to improve regulatory and operational performance. As part of the response to the NRC letter, Deco formed an Independent Overview

< Comittee (IOC) to evaluate Fermi 2 management, organization, and improvement programs. The IOC identified management problems including the operating staff's lack of comercial nuclear experience, inadequate leadership, lack of 1

1 1

accountability, management ineffectiveness, and organization and management systems problems.

DECO then initiated a major improvement program in April 1986, identified as the Nuclear Operations Improvement Plan (NOIP), in response to NRC actions and IOC findings. The NOIP incorporated the goals and outstanding issues of the ROIP, as well as identifying additional corrective actions. Some of these initiatives included: ,

o Appointment of an experienced individual from outside the company as Group Vice President, Nuclear Operations.

o Assignment of technical advisors with previous nuclear experience to key management and plant functional areas.

~

o Organizational and management changes to provide more effective support of Nuclear Production.

To provide a more comprehensive and long-term set of standards for Fermi, Deco issued the Fermi Business Plan (FBP) in January 1987. The FBP is presently the central controlling document that collectively contains the mission, goals, strategies, ard action items to be carried out by the Fermi organizations.

NOIP action items that remained open were incorporated into the Fermi Business Plan.

Although many of the licensee's performance indicators had shown positive

, trends, there was little performance improvement during the two and one-half years after establishnent of the NOIP and the implementation of other management enanges. This lack of improvehnt was during the Systematic Assessment of Iicense! Performance (SALP) 8 period, where the Detroit Edison Ccapeny was assigned three Category 3 ratings, and in the subsequent SALP 9 period ending March 31, 1988, where a total of five Category 3 ratings were assigned, as well as poor performance during the Local Leak Rate Test (LLRT) outage in April '.Aay 1988.

P<irformance problems continued following the SALP 9 period. Operational events occurred involving personnel errors, procedural errors, equipment malfunctions, and design dt+1ciencies. Table 1.1-1 lists operational events which have occurred in April-June 1988. DECO completed significant organizational changes and initiated a number of major programs to improve perforlance; however, the benefit of these initiatives, particularly in terms of a sustained improvement has yet to be demonstrated.

Based primarily on the continued poor performance of the Detroit Edison Company, a recommendation was made during the June 1988 NRC Senior Management Meeting that a diagnostic evaluation should be conducted at the Enrico Fermi Atomic Power Plant, Unit 2. Subsequently, the Executive Director for Operations (EDO) directed the Office for Analysis and Evaluation of Operational Data (AE00) to conduct an independent diagnostic evaluation of the Fermi 2 operational performance to identify the underlying causes for the licensee's continued poor performance and lack of ability to demonstrate sustained improvements.

2

0 3

s Table 1.1-1

/

OpERATI0tiAL EVEtiTS AT FERMI 2 DURit1G APRIL-JUf4E 1988 o On April 8,1983, due to operator error, approximately 130 gallons of sodium pentaborate solution was inadvertently pumped from tne standby liquid control (SLC) tank into the reactor vessel, o On April 9,1988, the RHR system inboard injecti: . valve inadvertently isolated with no apparent actuation signal d.sch caused a loss of shutdown cooling. In addition, one RHR pump rL. cead headed for 33 minutes.

On May 9, 1988, feedwater (FW) was diverted to the condensate sto' rage tank o

because of a procedural error. This caused the r1 actor level to decrease resulting in the further opening of the FW valve to compenscte for the level decrease. The subsequent cold water addition resulted in the reactor trip on high power,

~

o On May 10, 1983, while the unit was at 20 percent power, the licensee was conducting a turbine overspeed trip test when a pressure regulator failed.

This caused,the two bypass valves to suddenly close, which caused a reactor scram when the reactor pressure increased to 10e0 psig, o Subsequent to the completion of the LLRT Outage in May 1988 until the end of June.1988, the plant was pcwer limited due to problems with secondary chemistry control for sulfates, turbine backpressure as a result of cooling tower damage, and drywell air temperature limitations.

o On May 28, 1988, an instrument line failed on the reactor water cleanup system resulting in personnel contamination and high contamination levels in most of the reactor building. The event was attributed to procedural and design deficiencies. A repetitive event occurred or, July 13, 1988.

3

. s'. '

e

t 1.2 Scope and Objectives  !

The ED0 directed the Diagnostic Evaluation Team (CET) to conduct a broad:y structured evaluation to assess overall plant ope aions and the strength of

Detroit Edison's major programs for improving plant operations at Fermi 2.

To provide the assessment of plant performance requested by the EDO memorandum,  ;

the DET evaluated several functional areas with the following specific goals: -

o Functional area effectiveness: assess the effectiveness (including strengths, weaknesses, and issues) of the operations, maintenance, surveillance, quality programs, testing, and engineering areas in  !

ensuring safe plant operation; assess the adequacy of procedures, l programs, and compliance by the licensee to codes, standards, e commitments, and regulatory requirements, j o Technical support: assess the effectiveness (including strengths, l

, weaknesses, and issues) of the technical support provided to the station in the areas of operations, surveillance testing, {

maintenance, and operator training and quality verification.  !

L o Engineering support: assess the quality and timeliness of }

engineering support provided by the engiceering departments.

• including analysis, design modifications, equipment operability (

determinations, technical program development, and technical advice. [

i*

i o Management controls: assess the effectjveness (including strengths, l, weaknesses, and issues) of management leadership, direction,  ;

i oversight and involvement, and the organizational climate at Fermi.

• l The assessment of plant performance, particularly with regard to corporate and

plant management, was written in the DET scope due, in part, to the numerous

j management and organizattunal changes in the Detroit Ediscn Company. In i sddition, significant performance problems ark of ten directly attributable to e

! weak and poor management. Thus, the scope of the diagnostic evaluation j 4

included a critical assessment of licensee management ability to develop and i

! implement programs to sustain safe operational performance.  :

1

] 1.3 Methodolog l I The diagnostic evaluation at Fermi combined several methods of assessment, with  !

i special emphasis on the interfaces and relationships between operations and i various corporate and plant support groups. In the course of the evaluation, l

. the team observed plant operations, reviewed Jertine;.s documents, conducted  !

)

interviews with plant and corporate personnel at all levels, and assessed the l functional areas of operations, surveillance, maintenance, testing,  ;

engineering, quality programs, station and corporate management controls, and -

4 organizational climate. The team used contractors to assist in the evaluation *

] of management controls and organizational climate. l 2 i The team devoted several weeks t? in-office document reviews and preparations  !

, that included team meetings and briefings by NRC regional and headquarters l staff knowledgeable about the Detroit Edison Company and Fermi 2. On

! August 22, 1988, the team began an initial 2-week evaluation at the station and l corporate offices. A majority of the team returned to the Fermi site on i i I i 4 I l

September 12, 1988 for an additional week to complete the evaluation. Through-out the onsite evaluation, team representatives met periodically with the plant ,

manager.and corporate officers to discuss team activities, observations, and  !

preliminary findings. The team also met at the end of each day to discuss  !

observations and findings in each functional area. The Fermi Resident  !

Inspectors frequently attended these meetings and functioned as technical advisors to the team during the onsite evaluation. The exit meeting with corporate officials and canagers was held on November 1, 1988 at the Fermi site

.(see Section 4.0 for details).

1.4 Plant Description f The Fermi site, located on Lake Erie in Monroe County, Michigan, about 8 miles East-Northeast.of Monroe, Michigan, contains Fermi Units 1 and 2. Fermi 1 was l a demonstration breeder reactur which is no longer operational. Fermi 2 is a '

General Electric-designed boiling water react'or (BWR-4) with a Mark-I  ;

containment. The licensed thermal power is 3292 MWt with an electrical rating of 1093 MWe.

Construction of Unit 2 was authorized by the AEC/NRC by issuance of a construction permit on September 26, 1972. An operating license was issued to -

the Detroit Edison Company on March 20, 1985, and Unit 2 achieved initial ,

criticality on June 21, 1985. A full-power license was issued on July 15, 1985 i and the unit was declared in conmercial operation in February 1988. As of  ;

Octoter 1,1988, the unit had not yat completed its startup program.

1.5 Organization .

The Cetroit Edison Company (DECO) corporate and Fermi 2 organization as of August 1988 is illustrated in Figure 1.5-1. The corporate officers who have primary responsibilities for Fermi are the Senior Vice President, Nuclear ,

Generation, who reports directly to the Chairman of the Board and Chief [

Executive Officer; the Vice President, Nuclear Operations; and the Vice l President, Nuclear Engineering and Services. All three of these corporate officers are located at the Fermi site.

The Fermi organization, which has f.volved into a somewhat autonomous h organization, includes the line organization below the Senior Vice President I and his two Vice Presidents. The Fermi organization includes about 325  !

managers and supervisors and nearly 1000 staff.

[

The nuclear enginee:'ing organization (Figure 1.5-2) is located onsite. It  !

provides engineering support for major projects at the station, licensing l support, and other services. The nuclear operations organization is shown in i Figure 1.5-3. l

[

[

[

\

i

)

5

[

t

I DETROIT EDISON CIUIT EXECITTIVE OITICER DETROIT EDISON E E g PRESIDENT l

M OR VICE PRESIDENT NUCLEAR REMAINING DETROIT EDISON ORGANIZ.'. TION r OMBUDSMAN t.UCLEAR SAFETY REVIEW GROUP - BUSINESS CIIAIRMAN PLANNER

=

I I 1 I RE I % I E E. E.NT VICE PRESIDENT 0 NUCL ALITY 1R EDURF g ENG ERING OE ONS INEERING E F F COORDINATOR GROUP n,. . ,

y - .

Figure 1.5.L Fermi Organization -

9 1

VICE PRESIDENT .

NUCIEAR ENGINEERING

& SERVICES GENERAL DIRECTOR .

NUCLEAR ENGINEERING ARCIIITECT/ E ENGINEER a E E GENERAL GENERAL SUPERVISOR SUPERVISOR ENGINEERING g D N g ENGINEERING '

_ C_ . . _

~ I i _

SUPERVISOR SUPERVISOR ENGINEERING ENGINEERING DESIGN SUPPOIUr I I I I SUPERVISOR . SUPERVISOR SUPERVISOR SUPERVISOR EII.CTRICAL IAC ARCIIITECTURAI) MECIIANICAL &

ENGINEERING EtiGINEERING CIVIL FLUID SYSTEMS Figure 1.5-2. Nuclear Engineering .

l PLANT ,

i MANAGER '

i i

SUPEJUift. -

l OPERATIONS i

I

] _ - -

I I I I I I j GENERAL START UP GE SITE - f OPERATIONS ENGLISII j SUPERVISOR ENG'NEER OPERATIONS OPERATIONS ENGINEER ADMINISTR. ELECTRIC RADWASTE TEST PIIASE MANAGER SITE MGR.

. __  % . . . . . . -. - - mr .. - - . . - - , - - . _ -

1 1

GENERAL OPERATIONS

, om SUPERVISOR -

. SUPN)RT j llEMISTRY SlifI.; NUCLEAR SIIIIT ENGINEER l OPERATIONS

. SIIll'T TECIINICAL ~'

I ADVISORS SUPERVISORS ADVISORS l M j  ! NUCLEAR ASST TECHNICAL SILDT " ENGINEERING j SUPERVISORS NUCLEAR i SUPERVISING j OPERATOPE f

j -

1 NUCIEAR 1

POWER PLANT 1

OPERATORS 1

5'igure II>-3. Operations Organization l.

6

,._,n ------ v- - --- - , - - , - - - - - - - - - - - - - m w- ,----,---,_--w r_, r-------~ -- -w- - - -- - - - - - - - - - - -- - --

O

2.0 EVALUATION RES'JLTS 4

2.1 Findings and Conclusions J

Overall, the team observed recent improvements in Fermi's performance and

! capabilities, yet identified a number of woo nesses that require additional attention and involvement by DECO management. Initially, DECO management was s!ow in taking aggressive and effective action to fill key positions with professionals having extensive nuclear plant operations experience, and to implement site specific poi lcies and programs to improve performance arid accountability. However, acticns taken in this regard during the last two years represented significant accomp!!shments and provided the essential elements needed to achieve future improvements. Further, the team concluded that the actions being implemented at Fermi generally address the causes for performance problems while maintaining an acceptable level of operational safety. Notwithstanding these action's, tfie team determined that some areas 4 needed additional management attention to increase the rate of progress and i assure continued success. These included the need to: (1) achieve i organizational stabilii.y as soon as possible, (2) improve effectiveness to I first and second line supervisors, (3) improve organizational climate, (4) fix

< fragmented and overlapping engineering support responsibilities, (5) fix known i equipment problems, (6) set priorities according to plant needs, (7) allocate 1

resources to selected areas and better utilize existing resources, and 4

(8) improve effectiveness of operator training programs.

i j The findings and conclusions for each evaluated area are suamarized below A reference is made to the appropriate report section for additional details.

I J 2.1.1 Management and Organization

1. Management was not effective in accomplishing a smooth aiid prompt

! transition from the "old" organizational culture to the new one. The i

coexistence of two cultures had an adverse impact on personnel attitudes, morale, and management / employee relations. The changes in management philosophy and practices had not always been understood or accept 9d by 4 subordinates resulting in some conflict and resistance. (Section 3.1.2)

2. The top down manner in which Fermi management communicated was not always i understnod or accepted. Some individuals indicated a reluctance to J communicate upward for fear that expressing opposing or differing views l would result in reprisal. Horizontal communications were irrproving, j primarily through daily and weekly meetings. (Section 3.1.2.2) 4

{ 3. The organizational and personnel development expertise of the Corporate

! Employee Relations group had not been used extensively in the design, j develooment, and implementation of Fermi management programs and personnel j , polic;es. (Section 3.1.3.1)

]

! 4. Although there have been some improvement in interdepartmental interfaces, 2 there was still a general lack of clarity in the interfaces among Nuclear i Engineering. Technical Engineering, and Maintenance and Modifications. No l written guidance existed to define the interfaces. (Section 3.1.3.2) t I

! 9

' - c L ,

5. Organizational instability was a significant management problem which has  !

not yet been resolved, particularly in Nuclear Engineering. The staffing  :

and organization have been changing substantially for over two years.  !

Although the personnel changes have strengthened the management team, the i overall environment of frequent change adversely impacted performance, '

, productivity, and morale. (3.1.3.3) [

6. The Fermi Business Plan provided a system and' process for planning, i i staffing, scheduling, controlling, renitoring, and evaluating performance.  :,

However, its objectives, actions, and schedules were not realistic for ,

some organizational units. Additionally, employt a feedback concerning i their performance against Annuai Work Plans had not been received on 6 "

regular basis in a large number of cases. (Section 3.1.4) l

7. The implementation of new management programs and personnel policies such I 1 as the incentive pay program, overtime policy, hiring policy, and bid-out  !

. policy had weaknesses in communication of program content and purpose consideration of employee relations; implementation preparation; and .

organizational development planning. (Section 3.1.5) -

j 8. The implementation of an accountability program combined with an

+

improvement in the disciplinary policy has helped focus personnel j attention on improving individual performance. The Operations Department  :

personnel error rate has exhibited an improving trend over the past year. l i (Sections 3.1.5.3 and 3.1.5.4) l

i j -

9. Although the need to upgrace administrative procedures waJ ioentified by  !

the independent Overview Committee in 1986, and reemphasized by the NRC in [

l 1987, much work remained to be done in order for this activity to be f j completed by the end of 1988. (Section 3.1.5.5)

! 10. Fermi resources were strained in an effort to both support the safe j 'peration of the plant and implement the improvement programs on schedule. ,

Resource problems were compteunded by deficiencies in *.1anagement and  !

! supervisory skills in some organizations, and delays 'y Fermi c management l j in recruiting outside personnel with BWR operating expeaience and in i expanding Fermi plant operating krowledge in its organizational units  !

outside of Operations. (Section 3.1.6)  ;

1 i

11. Only recently has Fermi senior management placed high priority on training i of first ano second line supervisors. Further, the team did not find any j i training plans for intermediate (middle) and executive management, nor was  !

i there a systematic assessment of training needs for station personnel [

j other than control familiarization room for new employees. personnel, and no training)

(Section 3.1.7 requirement for plant l i

i

) 12. There was a lack of schedule integration which contributed to manpower [

j planning and forecasting problems, led to reactive management, and reduced i j availability of safety systems. preventh, maintenance scheduling was not  ;

i fully coordina<,ed with surveillance and inservice inspection activities. l

} A similar isck of coordination was observed among Fermi Business Plan

schedules, operations and maintenance schedules, deficiency event report t

! schedules, and Nuclear Engineering schedules. In addition, the L

organizational units had their own schedules to address systems and i 4

equipment problems. The net effect was that more actions were scheduled l l l i

i 10 t

O than could be properly implemented, and there was confusion about tha priorities of competing schedules. (Section 3.1.8)

13. Although the new managers and supervisors had good technical capability, these individuals did not effectively emphasize good human relations or personnel skills. Fermi management needed to pay more attention to human relations matters such as maintaining group cohesiveness and morale in order to achieve real and sustained improvements. (Section 3.1.9) 14 Upper managers were micromanaging the day-to-day business matters that could have Leen delegated to lower level managers and supervisors. I Consequently, little emphasis was placed on del 7ating more authority and '

responsibility. There were a number of problems attributed to the lack of delegation: too much dependence on upper management; subordinates unsure

, of their accountability and responsibility; explanations not always clearly commu'.icated; selective work overload; and inadequate followup by i i management. (Section 3.1.9.1)

15. Fermi management had systems in place to comprise an adequate management information system.' However, there ser led to be an inability to integrate phnning and scheduling and act on thi- information due, in part, to the utility's reaction to outside influences, e.g., the NRC and INp0. This reflected management's lack of self-confidence and a strong take charge attitudo in dealing with these organizations. (Section 3.1.9.2)

. 2.1.2 Operations

1. Within the operations organizstion, a sense of ownership, accountability, and professionalism was evident. Further, operator morale was good.

(Sections 3.2.1 and 3.2.2) .

2. Although operator performance had improved, their ability to per4rm at a high standard was adversely affected by lack of corrriercial BWR experience l at a well run plant. In addition, varying levels of expertise between  ;

i shift crews in overall plant operations were noted. The varying  ;

operational experience level between shif ts was comoounded ::y management's ,

tendency to schedule complex evolutions when the best performing shifts '

were on duty. (Section 3.2.2.1)

3. The top down style of on-shift management stifled initiative at the 4 Nuclear Supervising Operator level and did not effectively utilize the l collective judgement of the Nuclear Shift Supervisor and the Nuclear

Assistant Shift Supervisor. This practice also limited the review and

oversight function of the Nuclear Shift Supervisor. (Section 3.2,2.2) l 4 Operations management did not provide proper oversight of operator l performance. Oversight of routinu daily plant activit les was weak, and operator performance outside the control room was not routinely monitored.

In addition, a lack of direct supervision was noted during activities involving manipulation of in-plant equipment. (Sections 3.2.2.2 and 3.2.5.2.1)

5. A lack of attention to detail still persisted in the Operations Department despite management emphasis on pe~sonal accountability. This resulted 1

l 11 t

from a lack of supervision of activities and weak work practices.

(Section 3.2.2.3)

6. The licensee had not effectively utilized the Shift Operations Advisors (SOAs). Severai Nuclear Shift Supervisors indicated that they had little confidence in the expertise of de SOAs. This was attributed, in part, to the removal of the SOAs from shift requalification training activities.

Ineffective use of the SOAs had been previously identified by the NRC (1985). (Section3.2.2.4)

7. Although communications between the operating shifts and maintenance were impreving, shift communications with other support organizations were marginal. A prior loss of creditability of the engineering and licensing departments with operations, coupled with the operators own sense of

, accountability, had resulted in the reluctance of the shift crews to request assistance from outside the Opert.tions Department. (Section 3.2.2.6)

8. There were an excessivo number of challenges to the relatively inexperienced shift crews due to poor equipment reliability and availability. In addition, poor maintenance of control room instrumentation involving balance of plant equipment detracted from the operators' ability to operate the plant. Equipment issues were not resolved because of maintenance and engineering resource problems.  ;

(Section 3.2.3)

9. The operator evaluation program was no longer effective at increasing the level of operator performance. The same evolutions were performed repeatedly and became redundant. In addition, Operations Department management had not raised their expectations of acceptable operator performance as operator performance improved with experience.

(Section 3.2.5.2.1)

10. The Operations Department management did not possess a sufficiently broad recognize the si safety equipment various parspective and to promptly system and problem <. fully (Section 3.2.6.2)gnificance of

11. Marginal training had resulted in operations personnel possess' g random weaknesses in plant knowledge. This resulted from low instructor morale, out-of-date training materials, resources which had bee" 'everely strained in order to support normal and special taske, and lar ooJ lesson plans. The marginal operator training program reflr idequate management attention. (Section 3.2.7) i 1

12. There was a substantial difference in the ability of the shift crews to l effectively utilize the emergency operating procedures (EOPs) uring simulator scenarios. Weaknesses existed in some shift crews cspability to properly utilize the procedures. Additional training for the weaker shift crews on the imp!ementation of the E0P5 is warranted. (Section 3.2.7.2) i

13. The licensee initiated additional training to resolve noted weaknesses in the operator's understanding and implementation of Technical Specifica-tions. The implementation of the Technical Specification case history portion of this training program was weak and ineffective. The majority 12 1

of case histories were simplistic and the explanations of the solutions were incomplete. (Section3.2.7.3) 2.1.3 Maintenance

1. ,The Maihtenance and Modification (M&M) Department has been in a continuous state of transition because of management issues and plant problems. The most recent change was the hiring of a new M&!! Superintendent with extensive industry experience in plant operations and maintenance.

(Section 3.3.1)

2. The recent M&M Deoartment organizational changes hava provided a good foundation for irnprovement, but improvement efforts were being slowed due .

to strained resources and the continued restructuring of the M&M

.. Department. (Section3.3.1) ,

3. Inadequately trained contractor craft personnel were used to compensate for strained resources. This had an adverse impact on th. reliability of safety-related equipment. (Section 3.3.1)

4. The I&C Training Department developed an excellent training program for I&t. personnel. This program is expecteu to help eliminate the type of personnel errors experienced in the past. (Section3.3.2)

5. Ineffective M&M Department planning and scheduling adversely affected the imolementation of the pH program, resulting in numerous missed PM tasH; contributed to increaseo time for cctrpletion of CM work; and adversely dffactud the availab'lity and reliability.of plant systems. ($4Ction 3.3.3.1 and 3.'3.5)

6. MOV fsilures and problems due to incorrect torque switrh and limit switch settings were recurring. These incorrect switch settings were caused by inadequate procedures, inadequate maintenance practices, and a lack of control of switch settings. Although many of these probleus were identified by licensee personnel, as evidenced by deviation event report , '

trending reports and engineering consultant findin Fermi mancgem nt had not taken sufficient steps to resolve the problem,gsfSection33,4j

7. Lack of needed spare parts resulted ;n reduced reliacility and availcbility of plant systems. Soare part problems included:

(1) deficiencies in the spare parts reference system; (2) poor warehouse practices; (3) lack of preplanning of mettrial requirements; al.d (4) lack of a program for the identification ano persible r'.placemant cf equipment  ;

for which spare part: are not commercially available. (Section 3. 3.6) '

8. The program to trene and analy:e equi,) ment failures was not effectively implemented because of a lack of maintenance and plant sysum engineers, insufficient management oversight, and a lack of perscnnel training in failure and root cause analy;es. (Section 3.').7) 2.1.4 Testing i

1. The reviews of surveillance test procedures perfo:med through the licensee's Technical Spetification Improvement Progrsm (TSIP) were ,

comprehensive.

However, the extent n which the results of this ru iN i

13 i

g _ - , , . - _ . - . - ,_. - , - . _ - , - . . _ . . -_

will be reflected in surveillance test procedures is unclear since management did not require the proced'.re modifications recorama1ded by program personnel to be implemented. Further, few plans had bien N de for mainttining completed. (the knowledge Section 3.4.1.1)base developed by the TSIP af ter the program was

2. Tne Surveillance Scheduling and Tracking program and its im' lema.ntation were viewed as a strength. The program had resulted in the use of a relatively small amount of grace pericd. The staffing levels of botn the Surveillance Scheduling erd Tracking and Inservice inspection Groups were margina', for per/orming cay-to-duy activities, and had centriouted to the missing of a required increased frequency vahe stroke test. (Sect'ons 3.4.1.P. aad 3.a.1.3.2)

3. On the who!a, administrative control of the Inservice Testing program was

.imed as a strength, but '.'le documenti governing the p ogram were ricedlessly curtersone. Licentee plans to consolidate the goserning decaments into a sirgle Nrmi interfacing Procedure would result in the deletion of important prcgrain descrittion information.

(Section 3.4.1 3.2) 4 Deficiencies in ASPE Section XI velve usting included the failure to rate direct, local observations of valve performance, the failure to 9erify newly calculate.i Inse vice Test program maxirnuia ealve stroke times against matter valve list maximums, and the failure to track check valve test failures. (section 3.4.1.3.'.)

5. Emergency 0;esei Generator local control psal treters reccivea no periodic calibration or prevent 1"e main'.enanca, including the kilcwa'.t rateii used to verify corrpliance wiS Technical Specifications. (Section 3.4.1.4)

6. The Perforriance. Schedcling and Tracking nrogram nad the .eatures necessary to ccmtrol, ;chedule, and trget psriodic testing not required by Technical Specif: Cations, but it did not identify the reviewer of coroletM tests.

Its capability to identify and track cast due items was not of ten used.

(Sectica 3.4.2.'.)

7. The licensee appeared to have no cechanism to systevatically idcn'ify UFSAR cer:nitu.ents that warranted periodic r.entino or to ensure the tirely development and implementation of such testing 0?ter the need was

'dentifice. For example, there was a lack of kcactor Core Iso 1Ltion (coling (RCICI testittg to verify 'JfSAR commitments. (Se: tion 3.4.2.2)

3. Yhe licensm Lt.d v progran f or tes+.ing rtiW valves that were not included in the Jection XI : nserv ce Testir', program. (Section 3.4.2.2) .

9. 0;erators typicilly verified the operability of ecuipnent folicwing raintenance by performing Technical Spec'.fication surve.1: r".e te'.t orocedu et. Se;h testing, howe er, will not always verily the adec, acy o' the rv aterance or satisfy HF3AP cortnitments. / S e c

• i or. 3 4. 2. ' ',

10. The Performance E oluation Program (PEP) was $ntended t" iuprove plant reliability, avaliability, and e.'iciency, but provam development das proc 1 ding slovly due to a lid of support 0 cm upper nrnagencat. The tcan 14

)

a viewed the PEP as an example of lack of management followup in progran implementation. (Section 3.4 3) a 2.1.5 Quality Programs and Administrative Controls Affecting Quality  ;

1. The Deviation Tvent Peport (DER) tracking system proyided an adequate

, means to record, rnonitor and closecut individual DERs; however, failures

.' to follow proctdures and procedur61 weaknesses were identified. (Section 3.5.2)

2. Tne current DEP trending prgeam was not very efficient in that data acquisition was difficult and the trending program had limited capatilitier since only the major root cause could be assigned for each tread code. This iripscted and litritad the effectiveness of the trending h program. (Section 3.5.2) . .

3. Root cause cualysis by the liccnsee was weak. For example, root causes of the MOV toraue switch prot ' ems were not addressed prior to the 'econd failure of the Reactor Recirc Pump Discharge Vaha. This wea'; ness was .

also indicated by an inability to consistently meet licensee-specified evaluation rejection -stes on initial root cause Jeterminations. This resulted frNi insufficient root caure training across the orginization.

(Sections.3.4and3.5.3) 4 Overall, the QA Aur'it Program was a strength which was effectively used by '

. the feitai organization. liowever, some issues raised in audits were improperly evaluated and taking an excessige amount of time to correct.

(Section 3.5.4)

S. Nor;-Tecnnical Specificatier surveillances were considered weak. Overall Ouality Program staffs lacked operating exoerience and surveillance flidings aene* ally involved adninistrative or compliance type issues rather thar safety issues. (Seccion 3.5.5)

6. Activities performed by the Nuclear Safety Review Group (NSRG) were furward-looking and provided a stren% h to the Fermi organization.

Weakresses identified included Technical Spot fication inconsistencier regarding canduct of meetings by telephone or wal:through.

j (Section 4.5.6.2) f i 1 The Independent Safety F,ngineering Group ( KEG) had not oeen performing a 'i l functions specified in the Technical Specification, spent minimal time on surveillance of p unt operations and mcintenance activities and  ;

contributed little toward the reduction of hurnan errors. .

(Section 3.5.6.3) 3 Inconsisteaties were noted between lechnical Specification requirements regarding the Onsite Review Organization (OSRO, r;tivities and actual practice (i.e., meetings by phe ne or walkth ough, unrevietrad safety

que.tions determination not doctrcitad in writing, alternate OeH0 me

nbors ,

.iot properly appc tnted, and dccuments may not receive adequate revhw

prior N vot,ag for appioval/ disapproval). (Section 3.5.6.4) r 15

_, 9 - -y . .

m, - - - . . , . . , - - , - . . , n_,, .,% ,%e..-- s-1, ,+ _ , --- ,e ,, --- r v-

2.1.6 Engineering Support

1. At least 10 organizational structure changes have been made to the Nuclear Engineering organization since 1984, including three different Vice Presidents of Nuclear Engineering and Services since 1036. This, coupled with all supervisors being in an "acting" capacity, has resulted in uncertainty and anxiety within the DE staff. (Section 3.6.1.1)

2. The Nuclear Enginee-ing staff appeared to be well qualified technically, and exhibited positive attitudes about making Fermi succeed.

(Section 3.6.1.1)

3. Poor relationships existed between Nuclear Engineering and the plant organizations, mainly Operations, while in the transition phase from der,ign/ construction te operations. Although substantial improvements had been made, additional improvements were needed. (Section 3.6.1.1) 4 The role or the on-site A/E with respect to modifications and the design change process has not stabilized and continues to be evaluated by the licensee. (Section 3.6.1.2) i

5. The lack of established detailed procedures or other form of formal wo.*k interface control resulted in a communication weakness between engineering support groups, operations and maintenance. The. licensee had recognized the weak interfaces and hired a consultant to study organizational interfacts and recommend improvements. (Section3.6.1.3)

6. System Engineers were over-committec and understaffed to. adequately perform their duties. Additional training was also needed. The staffing concern had been addressed by the recent approval of 13 additional staff positions. (Section 3.5.1.3) i 7. Nuclear Engineering did not maintain adequate control of torque switch i settings for POVs. The lictnsee was au re that problems existed and did not take timely corrective action to resolve the issue. (Section3.6.2.1)

S. Based on a limited review of rect.nt Deviation Event Reports, a number of l design deviation'; were identified between the as-built condition of the plant and cesign documents or vericus data bases used to keep track of design-related inform 6 tion. (Section 3.6.2.2)

9. A r:odification to remove the delay volume in the RWCU system subsequently resulted in conditions which produced two water hammer events. Nuclear l Engineering had not adaquately considered this possibility during the design phase of the modificaticn, (Section 3.6.2.4)

10. Pressure and teeperature qualification by testing was necessary for a s9ction of core spray piping to be in accordance with UFSAR comitments.

It took aoproximately or.e year fror, the date of di.ccovery of this cotidition by St.ne and Webster before it was documented on r. DER, and an additional year before the test.,ig was c.mplete. These delays were indicators of inadequate ettention toward safety issues and the need for timely r.id effective corrective actions. (Section 3.6.2.4) 16

11. After the failure of a surveillance test, Nuclear Engineering did not evaluate possible damage to the EECW Div. Il pump or consider the question of system operability. The system fill and vent procedure which preceded the surveillance was alto found to be inadequate. (Section3.6.2.5)

12. Design requirements of the Post Accident Sampling System (PASS) were not consistent with actual installed system capabilities with respect to the sm. % 9 process and the availability of the nonsafety-related RBCCW.

(L r, an3.6.2.6)

13. Nu6 r F# pang response to INP0 SOER 86-03 check valve issues did not adec.s s ' Wrat n 7.essary desi requ a (San in *,.6.2.7) gn reviews concerning check valve testing 14 Numero- Pocedures existed,related to design control, the design change proces m ei "<' Jesign packages, engineering change requests, as-bui ' ;ueh ementation of modifications, urgent modifications, mincr ms ' c&ttort d temporary modifications which were overly detailed, vifusine .o follow, and not well integrated. (Section 3.6.5.1)

I 15. System Engineers were designing temporary modifications that should have  ;

been designed by Nuclear Engineering as indicated bj the nature of those that were subsequently tejected by Nuclear Engineering. The System Engineers also failed to confer with Nuclear Engineering when required to do so by procedure regarding various design constraints. (Section 3.6.5.2)

16. The licensee did not previde consistent engineering support throughout the modification process (i.e., design, constructicn, testing and closecut),

ann rather were technicians than used.

have eng(ineers follow the implementation of modifications, Section 3.6.5.3)

) 17. Excessive numbers of changes were made to modification work packages 1 indicating poor preparation and implementation on the part of Nuclear i Engineering. For example, potential design changes (PDC) had been revised as many as five times; engineering design packages (EDP) had t,een revisod J as many as seven times; and as many as 40 engineering change requests ,

(ECR) had been written against a single modification package.  ;

(Section 3.6.5.3) i l

, 18. The modification work prioritization system (Management Review Board) was I viewed as a strength and is expected to help improve communication between the plant staff and nuclear engineering. (Section 3.6.5.4)

19. Inadequate preliminary evaluations have resulted in the licensee not  !

performing safety evaluations for modifications when such evaluations were appropriate. A sample of safety-related design packages will be reviewed by the licensee to determine if problems with preliminary evaluations are widespread and to evaluate the potential for potential unreviewed safety questions because detailed safety evaluations were not performed.

(Section 3.6.6) l 17

2.2 Root Ca ' Analysis Based on the team analysis, as provided below, it was concludei that there were four root causes for toe licensee's poor performance and apparent inability to sustain improvements. These causes, which were all iraerdependent to varying degrees were: (1) a protracted design and contruction period, (2) the failure of management to adequately and effectively plan for the transition from a (3) lack of BWR design operating and construction experience projectthe throughuJt to an operating and organization, plant,(4) management slowness in determining and implementing effective solutions.

The design, construction, and preopere.ional testing period was protracted. A total of 16 years were recuired from the beginning of the project in 1969 until the plant was licensed for operation in 1985. During this period, the plant underwent a number of major design and construction transitions, including two different prime construction contractors, and several design engineering contractors even though DECO served as its own AE. At one point, the entire projact was halted for approximately 1 year. -

During this period, a design-engineering dominated organization evolved at Femi. There was a strong emphasis on solving the technical and engineering problems that are associated with a project of this tragnitude, and with getting a "vintage" plant ready for operation in a post-THI environment, hany significant plant upgrades were made and were needed to be made in order to meet the new and higher standards that existed for plants entering operation in the mid-1980's. These factors all demanded a high level of technical and engineering expertise, rather than operations knowledge and experience. Thus, in the latter stages of the project, the principal focus of Femi management was on the prompt resolution of construction, engineering, and startup issues that needed to be completed for both the low-power and full-power operating licenses. -

As a result, the organization (from the top down) at the beginning cf plant operation was still strongly engineering-oriented, and possessed a strong sense I

of plant cwnership. A lack of prior BWR or other commercial nuclear power i plant operating experience also existed throughout the organization; instead DECO relied heavily on contractors who possessed this expertise. Accordingly, the management and staff lacked the operations experience and perspective that were necessary to know what was needed and provide for a smooth and successful transition to a commercial operations phase. Thus, corporate and Fermi managennt had failed to adequately and effectively plan for this transition.

During the first year of operation, a premature criticality event on July 1, 1985, resulted in the initial identification of serious management def.iciencies i

at Fermi. The event led to a substantial loss of confidence by NRC and DECO in the ability of the Femi staff to proficiently operate the plant. The lots of confidence and trust led DECO to also becorre highly reactive and overly responsive to the influences of cutside organizatior.s including the NRC. This compounded DECc's problems in addressing management and organization weaknesses.

Following this event, management moved very slowly in determining and completing effective tolations. Early performance problems were mistakenly attributed to a learning experience of a new plant and through early 1986, DECc 18

e still did not recognize the irportance of having management and staff operating experience in the organization. This slowness to act was attributed to the difficulty of the engineering-oriented organiz) tion to fully understand, appreciate, and act on operations-oriented problems and the strong sense of -

pride and ownership within the organization to solve their own problems.

Following the premature criticality event, Detroit Edfson studies and NRC inspections identified more fully the broad range of management and organizational problems that were adversely effecting Fermi operating performance. In addition to the lack of commercial nuclear plant operating experience and the dominance of the engineering organization, it was found that significant leadership deficiencies, micromanagement by some senior managers and ranagement ineffectiveness in problem identification and resolution were adversely affecting teamwork and personal accountability within the Fermi crganization. .oor p

Communications within the Fermi organization and management ,

system problems related to planning, scheduling and setting priorities also sr.riously weakened management effectiveness and contributed to negative employee attitudes toward work performance. The personnel policies in place at the time, which emphasized human relations and employee security and well-being instead of work performance, productivity and work quality also centributed to counterproductive worker attitudes. Detroit Edison management pra.tices and personnel polie.ies related to hiring, discipline, overtime and promotion effectively created a Fermi organizational culture which exhibited a strong sense of loyalty between Detroit Edison and its employees but did not promote asenseofvalueforjobperformance,personalaccountabilityandtearwork within the organ m tion.

Once the full extent of.the weaknesses became more evident, corporate anJ Fermi management started to act. Senior management with the requisite knowledge and experience were recruited. This, in #, urn, led to: additional recruitments; pv sonnel shifts; organizational restructuring; and initiation of new policies, programs and procedures. Actions taken curing the last 2 years have been so numerous, in fact, that this period seems to be characterized by constant change. The efects on the transition to a new culture were slow, however, and its very nature, although necessary for long-term improvement, was disruotive and unsettling to personnel and, thus, to ongoing activities. The "right" personnel, effective organization and revised practices slowly evolved over a relatively long-term. This delay in achieving a stable organizational structure and in completing needed improvement programs had a continuing adverse impact on the level of performance and capabilities at the plant.

Further, this long and unsettling transition to a new organizational culture that piaced more emphasis on operations, accountability and discipline, resulted in unanticipated staff morale and attitude problems. The two.

"cultures" coexirted. creating friction, and this situation still continues to some degree. This has tended to slow the emergence of a strong plant operations and operational support organization.

Overall, the essential elements for achievement of the needed improvements are now largely in place. The organizational structure is staffed with a number of experienced and talented individuals as recruitment continues; the emphasis has been placed on the policies, practices and procedures inherent for successful plant operations; and the transition to the "new" culture is just beginning to reach down in the organization to the first and second line supervisors.

9 19

Provided proper management attention and an emphasis on the human side of personnel management occurs, it is expected that the problems and weaknesses noted during the diagnostic can be overcome and that progress will continue to be made toward achieving a higher level of performance and capability.

t 9 0 b

G l'

O t

20 .

3.0 DETA! LEO EVALUATION RESULTS 3.1 Management and Organization  !

The management and organization evaluation was based on approximately 80 l interviews, document reviews that included recent history of the plant, and i direct observation by the diagnostic evaluation team. The data that was l collected by the team was analyzed to identify management and organization ,

related strengths and weaknesses which were evaluated in terms of the degree of their impact on Fermi performance and safety. In addition, the strengths and weaknesses were compared with the results of previous assessments of management  !

and organization effectiveness at Fermi that had been conducted by the NRC and [

theIndependentOverviewCommittee(IOC).  !

l This Section begins with a review of management background and hist.ory. .A  !

discussion of the organizational culture and climate follows. The climate i assessment provided the team with a current picture / status of attitudes and  !

morale across the Fermi organization. Subsequently, management and l organi:.3 tion strengths and weaknesses are discussad in the following areas: l (1) organization, (2) Fermi Business Plan, (3) personnel programs and  ;

administrative policies, (4) staffing and personnel qualification, (5) training t and personnel development, (6) planning and scheduling, and (7) management  !

effectiveness. j 3.1.1 Management Background and History f

A prerequisite for understanding and evaluating current Fermi management and organization is an understanding of a number of marAgement issues that have ,

existed in the past. ,

i When construction began in 1969, Detroit Edison acted as its own engineer of l record, however, it has also used Sargent and Lund l Webster (1977-Present) as the archite.t engineerA/E). (y as Numerous well as Stone and engineering j services had been contracted out to companies such as Bechtel, AUS, Nutech,  ;

Teledyne, General Electric, Giffels and Associates, and Multiple Dynamics. In  !

conjunction with efforts provided by several engineering companies, two major  !

constructors were also used onsite: Ralph M. Parsons Company between 1969 and I 1974, and following a construction shetdown between 1975 and 1976 Daniel  !

Construction Company from 1977 until cumpletion in 1985. This organizational l lack of consistency and continuity in engineering and construction contributed  ;

to: the exceptionally long construction period (approximately 16 years); many  ;

of the engineering problems subsequently identified; and the prolonged startup  !

program, it was clear from both Detroit Edison and NRC studies that a number of management and organization problems existed at Fermi in late 1985 and early 1986. These problems were defined as follows:

o At the time the license was issued, there was a lack of comercial operating experience from the Chairman of the Board down to the craft personnel. The licensee had been required to implement a Shift Operations Advisor (SOA) Program, as a license condition, to support the inexperienced control room operations staff. The plant manager and other key menagers had technical assistants with comercial operating experience in order to augment their qualifications.

21

operating experience in order to augment their qualifications.

However, there was ineffective use of these individuals, particularly the SOAs, which contributed to the continued inability of the plant staff to effectively manage normal plant evolutions, o Tendency of some upper level managers to micromanage.

o Lack of accountability of subordinates and follow-up on assigned work.

o Management ineffectiveness in the areas of problem identification and resolution, developing teamwork, providing support to operations, planning, resolving interdepartmental conflicts, implementing management directives and important agreements on a timely basis, and

. line management decisionmaking and implementing decisions. - -

o Organizational problems related to the division of responsibilities within Nuclear Engineering and the authority of the Quality Assurance organization, o Management system problems associated with rigid and cumbersome administrative procedures and controls and lack of an integrated system for planning, scheduling, and establishing priorities.

o Poor communications and attitudes at all levels.

A major improvement program, the Nuclear Operations Improvement Plan (NOIP),

was initiated in April 1986 in response to the NRC actions and IOC findings.

The NOIP incorporated the goals'and issues of the Reactor Operations Improvement Plan (R0!P), which had been established by Fermi as an initial response to the start up problems, along with 22 corrective actions. In addition to the development of the NOIP, a number of other improvements were l 1 nplemented between January 29, 1986 and May 9, 1986. Some of these included:

o Appointment of an experienced individual from outside the company as Group Vice President of Nuclear Operations.

o Assignment of a nuclear industry-experienced individual to advise the Vice President - Nuclear Operations, o Organizational and management changes within Nuclear Engineering to provide more effective support of Nuclear Production, o Initiation of a search for nuclear industry-experienced individuals to fill management positions in Engineering and Security, o Implementation of a series of 2-day workshops for approximately 90 Fermi 2 managers and supervisors to increase their sensitivity and responsiveness to safety and regulatory issues.

Since the recruitment of a new Senior Vice President in May 1986, there had been significant recruitment of additional experienced indh idnis from cutside 22

O the Company for key management positions. These included:

Vice President, Nuclear Engineering and Services (May 1088)

Vice President, Nuclear Operations (June 1987)

Nuclear Safety Review Group Chairman (October 1986)

PlantManager(September 198i)

Director, Nuclear Quality Assurance and Plant Safety (Januai; 1988)

Director, Nuclear Engineering (September 1988)

Superintendent, Ooerations (November 1986)

Superintendent,TechnicalEngineering(June 1988)

Superintendent, Maintenance and Modifications (May 1988)

Director, Nuclear Licensing (June 1988)

There was, in essence, a new management team at Fermi which was still being completed. Fermi continued to recruit outside personnel ar.d to identify long-term employees with the technical and management ski'Is required to support an operating plant. With the formation of this new management team, an additional number of new problems developed, including a change of culture.

3.1.2 Organizational Culture and Climate The team found that there were two cultures that influenced Fermi performance.

The first was the traditional Fermi culture which existed within the environment of nuclear power plant design and construction. The second culture represents the collective experience of the new management team and is the culture to which employees were adapting. The transition was taking place, but it had gont slowly. Management ineffectiveness in mergina the two organizational cultures had an adverse impact on personnel attitudes, morale and management / employee relations.

Under the old culture (prior to mid-1986) employeer were developed in an environment with an emphasis on human relations. Some came from fossil plant backgrounds, while others were hired at the entry level and grew up with the company which offered a strong commitment to employee welfare and employment security. Management had very limited nuclear plant operating experience.

Employees were loyal and had pride in Deco and its way of doing business.

Deco was a strong engineering company with Engineering in charge of the Fermi project. When operations began, Engineering was relegated to a support role.

This was difficult to accept and created conflicts among organizational units.

A new culture had its inception in mid-1986 after the present Senior Vice dresident, Nuclear Generation was hired. This culture can be distinguished from the old culture as follows.. .

o Greater emphasis on technical capability and the hiring of professionals with commercial nuclear operations experience into the technical and managerial ranks, o More focus on technical problem solvirg and less focus or attention to organizational and human relations issues, o More direct management supervision and follow-up.

23

4 0

o More accountability; management by objectives; performance indicators.

o More planning, organizing, monitoring and evaluation of work, o Job security only to the degree senior management is satisfied with performance, o Limited use of overtime, o Gnater individual responsibility and risk.

o Less on engineering influence; more operations influence. ,

o Increased discip,11 nary action, particularly for serious offenses, o Recognition based upon progress or results.

The changes in management philosophy and practices had not always been understood or accepted by subordinates, resulting in some conflict and resistance. In some cases, the transition between the two cultures had created new management problems and exacerbated existing problems. The team concluded

, that most of the changes at Fermi were necessary. Some of the most significant i problems arising from the transition are discussed below and in subsection 3.1.9.

~3.1.2.1 Human Relations  ;

, At'the time of the evaluation, the top down manner in which Fermi management i established and implemented changes had adversely affected the security, pride, '

motivation, morale, attitudes and performance of some old culture employees in that they either did not understand the new philosophy or did not support it.

These individuals reported that the new plan for the future of Fermi, and the  ;

related changes, had not been adequately explah i or justified. It was i evident to the team that a number of old culture Individuals were accustomed to i a set of totally different management practices, and they were having r

! difficulty adjusting to the newer management style and the additional stress  !

associated with change.

Because of the need to improve the management performance at Fermi, changes were frequent and substantial. During this process, comunications were ,

primarily directed from the top down. Some individuals from the old culture i

did not feel that they were sufficiently involved in the decisionmaking process. In addition to dead-ended career paths, old culture emplo ,

reported that they were being locked in by the new bid out policy (yee,s see Section 3.1.5.3), and that the incentive pay program (see Section 3.1.5.2) was based  !

j substantially on the performance of others over whom they had no control. -

t Still others indicated that the new management had not taken the time to  !

provide feedback on performance. Some employees reported inconsistencies in i management style and communications, and were uncertain as to their job t

secu ri ty.

l l 24

3.1.2.2 Communications The team conducted an evaluation of Fermi communications and found that some of its results were similar to those of a communications study conducted by the licensee in 1987. Prior to 1986, employees indit.ated that they learned more about the company and its regulatory problems through the news media than from ranagement. Fermi senior management was comitted to improving communications, and many individuals believed that Fermi communications had improved during the past two years. However, the team found that there were still communication problems:

o Management had not always moved swif tly to resolve conflict between i individuals and improve interfaces between groups, o Some employees believed that progress on the business plan or work i plans was not always effectively communicated.

4 o It appeared that some managers did not listen to all the facts before

, making decisions. They sometimes became emotional and did not always i i reprimand in private. '

o There was considerable negativism or negative reporting with little  ;

positive reinforcement or recognition. [The team noted that a new '

Recognition Program had been implemented in September 1988.]

o Information abou.t 'he t new management's vision of where Fermi was

, headed was either not co municated or was not understood and accepted by the old culture employees.

, o People did not appear to understand the Fermi decisionmaking and priority setting processes.

l Most communication appeared to still be top down. Some individuals indicated a reluctance to communicate upward for fear that expressing opposing or differing

] views would result in reprisal.

Other individuals indicated that Fermi had made progress in solving horizontal communication problems. Several daily and weekly meetings were held to deal

- with this issue, and some people physically located together to improve i effectiveness of their comunications. Fermi management was aware of specific interface problems and their causes. Actions, although delayed, were being taken to correct these problems.

I

! 3.1.3 Organization (1) the relationship The team reviewed the Fermi organization including:(2) between the OEco corporate organization and Fermi, interdepartmental i interfaces at Fermi, and (3) organizational stability. The following subsections describe the cbservations that the team made in these areas.

l l

25

3.1.3.1 Fermi 2 - Corporate Relations The organizational and personnel development expertise of Corporate Employee Relations group hao not been used extensively in designing, developing and firplemerting new Femi management programs and personnel policies. In view of the people problems discussed in Section 3.1.2, it appeared hat using the expertise available at Corporate Employee' Relations might be helpful in implementing new programs.

Overall, DECO corporate management supported the Fermi plans and programs and recognized that the success of Detroit Edison depended largely on successful operation of Fermi. The Board of Directors was actively involved in reviewing l plant progress and performance through the Detroit Edison Board Nuclear Review l Comittee. The Femi reporting chain had been modified so that the Senior Vice President, Nuclear Generati,on reported directly to the Chairman of the Board and Chief Executive Officer In, stead of the DECO President, i Corporate services available to the Fermi organization were extensive and I

included public affairs, personnel management and ur. ion relations. The Corporate Employee Relations office provided a number of internal management consultative services covering policy formulation, organization design, organi;.ation development, and training. Fermi had made limited use of these services from time to time. Recently, Fermi had requested corporate assistance in evaluating the effectiveness of the Annual Work Plans and developing a leadership training program for first and second line supervisors, l

Detroit Edison recently approved policy changes intended to treat Fermi as a distinct organization. These changes, which involved hiring from outside the company to fill vacancies, establishing a new personnel' transfer policy and implementing an incentive plan, are discussed in Section 3.1.5. These changes were intended to provide Fermi management additional tools to deal with Fermi problems, l 3.1.3.2 Fermi 2 Interdepartt.: ental Interf aces Although there have been some improvements in interdepartmental interfaces, there was still a general lack of clarity and precision in the interfaces among Nuclear Engineering, Technical Engineering, and Maintenance and Modifications (Section 3.6.1.3). Specific interface issues are discussed in the respective functional areas of the report. These interface problems had been known for years and represented another example where Fermi management had failed to take appropriate corrective actions in a timely manner, e.g., written guidance to clarify responsibility. Recently Fermi management had hired a consultant to study these interfaces and recommend improvements.

The team also identified several strengths with regard to organizational interfaces, o personnel from Operations stated that the responsiveness and levels of cooperation with Nuclear Engineering. Technical Engineering and Maintenance and liodification had improved significantly over the past year.

26

o Quality Assurance and Health Physics had established good working relationships with Operations and Maintenance and Modification.

These findings reflected efforts to improve teamwork and to provide plant operations with quality support services.

3.1.3.3 Organizational Stability Organizational instability was a significant Fermi managemen* problem which has not yet been solved, particularly in Nuclear Engineering. The staffing and t organization had been changing substantially for over two years. Personnel changes had strengthened the management team. However, the overall enviroament of frequent change was a significant issue that advercely impacted performance,

'. productivity, and morale. Many individuals were uncertain about thel. roles, the expectations.of management, organizational r,esponsibilities and organizational interfaces.

Upper management had taken the initial step of defining the overall organizational structure, rotes and respontibilities. Each organizational unit was defining its own organization functions. However, the definition of organizational units, their responsibilities and their interfaces had act been completed. it did not appear that this prccess would be completed quickly since there were a number of temporary (acting) assignments in middle level management positions and supervisory positions, particularly in Nuclear Engineering. Upper management had not developed a master plan and schedule for , l achieving organizational stability. .

3.1.4 Fermi Business Plan (FBP)

The FBP provided a system and process for planning, staffing, scheduling,  !

controlling, monitoring, and evaluating performance. It was based on a broad set of corporate goals and performance requirements. The Fermi goals and performance requirements had been translated into organizational unit level goals, strategies and action items. These action items were further broken down into specific requirements for individuals and set forth in the individual I employee's annual work plan (AWP). The FBP was designed to provide a comon i reference for the entire Fermi organization to follow, including implementation j of the improvement programs. j l The of FBP objectives, actions, and schedules were not realistic for some l

! organizational units. AWPs did not match up wiQ the actual jobs being '

I performed or did not exist in some cases. For example, Technical Engineering  ;

I was in the process of redefining its functions and responsibilities and '

l expanding its staff. Maintenance and Modifications was reorganizing. .There l were a number of positions to be filled in Nuclear Engineering. The i effectiveness of the FBP/AWP process will not be proven until there is greater stability in the overall organization, and staffing is completed.

There was a noticeable lack of feedback regarding FBP states and performance at j the supervisory and working levels. Distribution of performance feedback and  !

status data at the lower levels was limited primarily to monthly updates of [

performance charts that were posted at numerous locations at the site along ,

with occasional articles in company newsletters. Some interviewees indicated i that these charts were difficult to understand. Employee feedback concerning ,

27  !

performance against their AWP had not been received on a regular basis in a large number of cases.

3.1.E Personnel Programs and Administrative Policies i

The team found that Fermi management had initiated a number of new management programs, personnel policies, administrative policies, and personnel performance improvement programs. Management programs and personnel policies were based on sound concepts and principles. They were directed toward major problems of accountability and performance that had been experienced by Fermi in the past. The Fermi Business Plan provided clear and precise statements of goals, objectives, and action requirements for most organizations. However, there was a negative response to some management and personnel policies, including overtime policy, the new hiring policy and the bid-out policy because l they were hastily implemented and not presented to the staff in a manner to win. ,

their support. There were weaknesses in communication of program content and purpose, consideration of employee relations issues, organizational development planning and implementation preparation. These weaknesses resulted in some misunderstandings, lack of management credibility, and unexpectedly negative responses.

3 Specific strengths and weaknesses of key management programs and personnel policies that had been implemented are discussed in the following subsections.

I 3.1.5.1 Incentive Pay program Fermi management had recently introduced an incentive pay program and expanded the policy for recognizing exceptional performance. Both programt were consistent with management's focus on improving individual performance. In parti:ular, the incentive program provided the opportunity for each participating employee to earn an increase of up to 10 percent of base pay, Award of the increase was based on achievement of performance goals for the overall Fermi organization, department level, and at the individual level. The l program stressed individual achievement and team work. This program was unique 1 to the Fermi organization in that it was not implemented elsewhere in the 4 Detroit Edison Company, f The incentive pay program was an example of a management concept which had not

been implemented well. This can be attribLted to the rush to implement the  ;

program af ter the licensee received a Category 3 in 5 functional areas during r the SALP-9 assessment. Senior management believed that the incentive pay i program would provide the impetus for employees to improve their performances '

and show progress for the Fermi organization. There was considerable misunderstanding among personnel in regard to the relationship of the. incentive ply program and incremental raises to base pay. Many interviewees believed that Fermi employees would not receive the same base pay annusi raises as other  !

Detroit Edison personnel because of the incentive pay program. They viewed [

this as unfair. Many believed that the incentive program had been established  :

to save money - not to reward deserving employees. Several employees indicated L that the program could not have objectivity bacause of: (1) unclear indivicual (

responsibilities, (2) unrealistic 90415 and actions specified in the FBP within  ;

the proposed schedules, or (3) supervisor tendencies to reward their buddie:, '

In summary, a progran that should have beer strongly supported by plant ,

managers and staff had, at best, been received with mixed feelings, l 28 '

3.1.5.2 Personnel Policies Managerrert had recently established three personnel policies that were also unique to the fermi organization. They covered overtime, hiring practices, and bidding for other Detroit Edison jobs. The team concluded these new policies were overall a strength, although there were negative factors involved in the implementation.

The new overtime policy applied to nonunion employees. In essence, the new policy eliminated premium pay for all management and supervisory personnel.

Straight time pay was received for all overtime for personnel up to middle managers. The change in overtime pay policy placed Fermi closer in line with industry . practices.

The new overtime policy had its greatest impact on the first line supervisors in Operations and Maintenance and Modifications. In both groups, there were individuals making substantial amounts in overtime pay and there was a negative j reaction to the policy.

The new hiring policy allowed Fermi to hire personnel from outside the company, J This policy provided Fermi management with additional flexibility to recruit personnel with commercial nuclear power experience. The policy resulted in some negative reaction because the hiring of outside personnel was viewed as cutting off the career paths of some employees.

The revisW hid-out policy basically prevented Fermi personnel from bidding on Detroit Edison jobs outside of the Fermi organization without management approval. Turnover had been low at Fermi, and there appeared to be few people biddir.g on outside jobs. Announcement of this policy, however, was met with some resentment directed toward management. Management explained that the policy was impleoented :c that poor performers could not just transfer to another Detroit Echon unit. They would have to either perform or be subject to adverse action. Any good employee who had the opportunity to advance his/her career would be allowed to transfer to another organization.

3.1.5.3 Accountability Programs During the past year, the licensee implemented an accountability program in order to emphasize personal responsibility and accountability throughout the organization. The program was initia+ed in response to the high nurrter of personnel errors occurring at the site which could not be attributed to procedural or programmatic deficiencies. The core of the program involves accountability meetings in which the individual (s) involved, accompanied by their supervisors (through the superintendent level), review the actions taken during the event with a senior management team. Accountability meetings could be initiated as a result of a Deviation Event Report (if the report indicates that human error was a major factor in the event), by the Plant Safety group, or through an individual work group. Accountability meetings were conducted separate from any disciplinary actions taken against the individual (s) by management, allowing emphasis of the meeting to focus on lessons learned and potential corrective actions which could reduce the probability of recurrence.

Approximately half of the individuals found responsible in accountability meetings eventually received disciplinary action.

29

Corrmercing in February 1988, 27 accountability meetings had been conducted through mid-September. The meetings addressed human errors ranging from a missing keycard ard violation of radiation work permit requirements to events involving plant safety, such as the isolation of RHR shutdown cooling and a reactor scram caused when feedwater was inadvertently diverted to the conden.

sate storage tank. A review of the accountability meetings revealed that in addition to the operators, management also held maintenance workers and instrument, technicians equally accountable for performance errors. Individuals

found accountable through the process were often required to prepare lessons

, learned from the event and review the event with their coworkers in group meetings or through mamoranda.

The licensee had successfully instilled the concept of personal respor.c:Lility and accountability in the operating crews. The majority of personnel

interviewed on this matter indicated that the accountability process .had been

~

beneficial in focusing attention on improving personal performance. For example, the operations department personnel error rate over the past year indicated an improving trend. The team observed one accountability meeting and i

had a positive impression. The team considered this type of program as useful and effective.

3.1.5.4 Disciplinary policy A shift in the site disciplinary policy had accompanied the licensee's

, heightened emphasis on personal accountability. The disciplinary policy l philosophy had been altered to prescribe disciplinary consequences appropriate

! with the safety fignificance of the event. The past policy assigned a pre-i scribed. sequence of disciplinary consequences based on the number of offenses

, rather than the significance of the event incurred by the inappropriate action (s). Under the past policy, disciplinary action started with a written record of an oral warning, was escalated with the next offense to a written record of a written warning, and with additional offenses, progressed to j suspension and eventual discharge. Management currently adheres to a policy J that relates the discipline to the significance of the event as well as the performance record of the individual. Thus, the consequences of a first offense could be as severe as suspension or discharge, if warranted.

, The new disciplinary policy had been favorably received by the rajority of operators interviewed. The operators regarded the policy as an improvement l over the old policy which was not viewed as fair or particularly rational, i Although the policy was an improvenent over old policy, the team found implementation of the new policy was perceived to be non-uniform between the operating shifts. Discussions with operators indicated that disciplinary actions taken varied with the supervisor involved. The operators perceived the appeal process just to be a formality and not a true adjudicatory review of the case.

The team found that disciplinary action was generally taking several weeks after the incidents as opposed to minimizing the time between the incident and the discipline in order to prevent the possibility of a prolonged demotivating effect on the individual's performance.

3.1.5.5 Administrative Policies 30

Stemming from a recommendation made by the IOC, corrective actions to revise and upgrade administrative procedures were initiated. More than 600 .

administrative procedures were being reduced to approximately 200, thereby  !

eliminating many redundancies and confusion that existed because of lack of  :

standardization among the administrative procedures of organizational units. ,

The administrative procedures upgrade program included procedures defining  :

crganizational unit responsibilities and interface requirements. l Even though the corrective action had been initiated, most of the work remained i to be done although the 1986 IOC report pointed out that rigid and cumbersome '

administrative procedures contributed to almost all fermi problems. The ne'i for more effective administrative procedures also had been pointed out to Fermi .

management more recently (August 1987} by an NRC Operational Safety Team Inspection (OSTI). Completion of the program was scheduled fo'r the end of 1988 t and management was emphasi:ing timely. completion. l l

3.1.6 Staffing and personnel Qualifications The team found that resources were strained in an effort to both support the  !

safe operation of the plant and implement the improvement programs on schedule.

Deficiencies in manpower planning and forecasting were a major cause of the ,

manpower shortfall. Resource problems were compounded by deficiencies in management and supervisory skills in some organizations, the existence of people problems as discussed in Section 3.1.2, and the lack of speed with l which Fermi management had recruited outside personnel with BWR operating i experience and expanded Fermi operating knowledge to its organizational units outside of Operations.  :

1 l 3.1.6.1 Adequacy of Staff to Meet F8p Requirements The team concluded that Ferni resources were strained to effectively implement i i  ;

its the 1nprovement operational support. programs and commitments The team conclusion was based on: as well

1) as thep(rovide quality large number i of cocnitments and action items that were currently underway and had to be l completed within the next three to five months; (2) the number of positions that were currently unfilled; (3) the current contracting for additional support services from outside sources; and (4) the current status of many of the improvement programs which were marginally on schedule or behind schedule.

Indications of staff limitations included the "flattening out" of the reduction in corrective and preytntive maintenance backlogs. In addition, it was recently necessary to redirect contractor construction craf t personnel to augment the maintenance mechanical and electrical staff because 10 of the 60  !

available Fermi craft were reassigned to support the procedures upgrade program and approximately 12 maintenance personnel had been assigned to training i programs. ,

Compounding the manpower deficiency was a weakness in management and supervisory skills in several organizational units. For example, middle management of Nuclear Engineering appeared to have problems coordinating work, resources, and priorities. This conclusion was based upon the failure of Nuclear Engineering to efficiently and effectively respond to industry problems and recuirements such as SOER 86-3 and the MOV torque switch problem, and failure to assign responsibility for development of a test program regarding valver that were not included in the IST program, but had FSAR test commitments.

l 31 1

l

t In another example, Fermi management indicated that first line supervisors in Maintenance and Modification lacked supervisory and leadership capabilities.

The team evaluated that situation and determined that a part of the problem was a lack of leadership and supervisory skills to effectively carry out job requirements. Some supervisors indicated that their productivity was adversely impacted by a nurN r of elements external to the job including completeness of work packages, schedules, availability of spare parts and materials, and tagging constraints. Further, the design deficiencies and obsolete equipment continue to exist in the plant as a result of the engineering problems and associated delays encountered during the 16-year construction of Fermi. These problems, which produced operational inconveniences and little flexibility, provided an additional burden on the work force beyond those normally expected during the startup of a facility.

3.1.6.2 Manpower Planning and Forecasting -

The team found that manpower planning and forecasting was weak and contributed to limited resources. Scheduled activities in Engineering and Maintenance frequently were slipped due to lack of manpower resources. For example, activities in the Maintenance and Modifications work schedules (72-hour schedule) were completed on schedule less than 50 percent of the time. Lack of manpower planning contributed to this poor performance.

On two occasior.c during the last year, management initiated a policy to reduce the number of contract personnel. First, contract personnel reductions occurred approximately four months prior to the local leak rate test (LLRT) outage. This was a period of intense maintenance activity when uaintenance and' engineering planning for the outage should have occurred. Later, it was recognized the outage planning and scoping were inadequate, and additional contractor personnel were brought in to support the outage. The LLRT Outage had a $24M cost overrun.

A second reduction in contractor forces occurred imediately af ter the LLRT outage in June and July of 1988. Again, the reductions preceded a period of intense activity associated with scheduled implementation and completion of a number of improvement programs and there were also high commitments of Operations and Maintenance personnel to training. Fermi management stated that Fermi receives whatever resource support it needs from corporate and that the reductions were a Fermi decision based on their analysis of manpower needs.

Fermi has experienced additional problems as a result of using untrained contractor personnel to augment their permanent staff. These problems are discussed in detail in Section 3.3.

3.1.6.3 Staff Operations Experience The level of comercial NPP operating experience was not increased in a tirely manner. There was a gap of almost 18 months from the time the current Senior Vice President, Nuclear Generation was hired to the time sore key management positions were filled. Other key positions (e.g., Superintendent. Technical Engineering; Superintendent, Maintenance and Modifications; and Director, Engineering) had been filled in the last three months. Several key positions in Nuclear Engineering and Planning and Scheduling had not been filled.

32

Although there was a delay in filling positions, Fermi management had done a good job in identifying and hiring capable managers with pertinent operating experience for key staff positions. It was clear to the team that these new managers had the expertise and experience to effectively organize their work units and to address Fermi problems. Based on interviews the new management personnel, for the most part, were respected by Form 1 emp}oyees and recognized for their technical and managerial capabilities.

In addition to bringing aboard a core of experienced personnel, Fermi had expanded its operational experience on staff by management personnel from other organizational units attending operator training, although this was not done on a programmatic basis. Operational experience was further expanded to other organizational units by the reassignment on a temporary or permanent basis of SRO qualified personnel to key management and organizational interface positions. .

3.1.7 Training and Personnel Development Only recently has Fermi senior management had placed high priority on training first and second line supervisors. Some supervisors had come out of union ranks and had never perceived themselves as part of the management team.

Consequently, they possessed the a* tributes of the old culture and were reluctant to support the new manags ant philosophy and style. Management had had limited success in penetrating.this barrier and recognized that the ultimate success of the numerous improvement programs and personnel and policy changes was jeopardized without getting first and second line supervisors motivated to carry out the progran.

The t.radership Development Program was designed specifically to adoress team building and improve supervisory skills among first and second line supervisors. The team attended the introductory session for Fermi managers and considered the involvement of Fermi management in the program to be excellent.

The LDP impact on the performance of the supervisors would not be evident in the short-term bec:use it was designed to be implemented over a 2-year period.

The team did not find any training plans for intermediate (middle) and executive management. In addition. that there was ,ot a systematic assessment of technical training needs for noranion station personnel other than control room operating personnel. The team did not find any training plans for engineers or other staff personnel in any of the organizational units evaluated. There did not appear to be a requirement for plant familiarization (i.e., plant systems training) for new employees in technical positions or for employees who during the course of the many organizational changes assumeo new jobs or added new responsibilities to their existing jebs. .

3.1.8 Planning and Scheduling Fermi management was committed to improving planning and scheduling as reflected by the emphasis placed on this area in the Fermi Business Plan and in a number uf organizational and personnel changes designed to eliminate planning and scheduling problems:

o The previous planning and scheduling organization was changed in April 1988 as a result of peor performance in planning and scheduling the LLRT outage, and inefficient and ineffective scheduling of 33

r I

maintenance work flow. Outage planning and scheduling a., now one organizational unit. Operations and Maintenance (0&M) planning and scheduling was relocated within the Maintenance and Modifications 1 group. '

t o The responsibility for scheduling and tracking TS-required I surveillance testing was relocated from Technical Engineering to l Operttions in September 1988. .

o Within Nuclear Engineering, a system for prioritizing and scheduling  !

engineering design packag-es (EDP) had been established. The system l which included the establishment of a "top ten" list, a "must" list, l and a "want" list for all proposed EDPs. Priorities for EDPs were '

established by the Management Review Board (see Engineering Section),

o An experienced outage planner was being recruited to increase ,

capabilities of the outage planning and scheduling group. '

o Meetings were regularly scheduled to address planning and work l priorities.  ;

The team found that planning ano scheduling was improving -- particularly in the areas of O&M planning and EDP planning. Fermi management placed considerable emphasis on meeting schedules, and beginning with the R0!P and continuing through the FBP, had established numerous performance reasures and methods for monitoring schedule commitments. Typical improvements observed by the team included an increase in the number of Technical Specification surveillances being completed befoce the grace period, and reduction in LERs i associated with missed surveillance /ISI requirements.

On the other hand, the team noted that planning and scheduling had a number of weaknesses and continued to be a major problem. While the FBP provided a f l framework for scheduling programs an6 improvement action, there was a lack of I schedule integration. PM scheduling was not fully coordinated with schedulinq l surveillance and ISI activities. A similar lack of coordination was observed I among FBP schedules, O&M schedules, DER schedules, and Nuclear Engineering schedules. The team observed that the 'ack of planning and scheduling integration contributed to manpower planning and forecasting problems, led to I reactive management, and reduced availability of safety systems.

An integral part of planning and scheduling was establishing a framework for '

l setting priorities on required actions and tasks. Management had attempted to prioritize items in the Fermi Business Plan and had established general prior-  ;

ities for the various organizational units. For example, maintenance . work flow was geared to (1) emergency repair requirements (2) surveillances, (3) PM and 1 (4) CN. Another example of tystenatically establishing priorities was the EDP ,

planning process. i I

Notwithstanding the emphasis management had placed on defining priorities in the planning process, the team found problems with the priorities established at Fermi. Each system for planning and scheduling 1cng-term and snort-term j activities (e.g., FBP, O&M scheduling, the DER system, RACTS) had its own  ;

priority system. In addition, different organizational units such as Nuclear [

Engineering, Operations, and Maintenance and Modifications had their own "top  :

ten" fix lists for systems and equipment. The net effect was that more actions  !

34 r

]

were scheduled than could be properly implemented, and there was confusion about the priorities of competing schedules. Examples of priority conflic'.s included:

o Regulatory and Industry (e.g., INPO) comitments and improvement program initiatives competed with day-to-day support of the plant for manpower resources.

o Where priority guid'elines had been established, they were difficult l

to follow due to the requirements to address non-scheduled items and staffing limitations. This was particularly true in Nuclear Engineering and to a degree in Maintenance, o Problems that had existed for a long period of time anc had l tignific_ ant negative consequences had not been promptly addressed. ,

l For example, material control had been a problem for several years I

but only within the last few months had there been a program initiated to resolve the problem (see Maintenance Section).

3.1.9 Management Effectiveness l Although some progress had been made toward solving management problems and l Fermi appeared headed in a positive direction, the team found that problems i

still existed related to delegation and ability to act on known information.

Corporate and Fermi senior managers were actively involved in helping to assure safe and efficient plant operation. Specific examples of senior managements involvement included

(1) i Vice President's requiremen,the plant t that he bemanager review of CERs; notified imediately a' o"* (2) the Senior c problems l which required communication with NRC, (3) the cooperation between the Vice President, Nuclear Engineering and Services and the Vice President, Nuclear Operations to establish priorities for modifications, and (4) the establishment of accountabiitty meeting >. Overall, senior management involvement had improved during the past two years, but in some cases, senior management had not followed through sufficiently. For example, the Plant Managcr s review of l OERs should have led him to initiate additional corrective actions on the MOV I

torque switch problem (see Section 3.3). In other cases senior raanagement was too involved in order to compensate for the lack of maturity and stabilization of the new organization and management team. This practice resulted in working continuous long hours and diverted senior management's focus such that important things, which might otherwise have been identified, went unnoticed.

The organization had not yet come together as a team and been stabilized.

l Below senior nanagement, the level of manager / supervisor involvement in work varied conside ably throughout the organization. Some inoividuals stated that they were too encumbered and overloaded to supervise ' heir subordinates as much is they would have liked.

Because of the multituce of management problems, regulatory requirements, workload and daadlines Fermi was forced into a reactive mode. It did not appear that there could be a change-over from reactive to proactive noce in the short-term. It appeared to the team that there were areas requiring additional attentier. These areas included management development, organizational development, human relations, manpower planning, and staffing.

35

+

Management expressed its firm commitment to nuclear safety. Examples provided included: (1) an ombudsman for reporting safety concerns, (2) the Safety Review Group's part of the Fermi Business Plan; (3) publicity / awareness information, accountability meetings, and seminars, and (4) the incentive pay progran. However, the team concluded, as did the NRC in a letter to Detroit Edison of Decembe* 1985, that morale and attitude pr6bleu , if not corrected by  ;

the improvement programs, would continue to adversely affect performance, t However, a large majority of those interviewed were positive about the future, ,

only a minority of employees felt trapped at Fermi, were distrustful and had ,

negative attitudes about their future.

i Fermi's Senior Vice president had hired new managers and supervisors with good l technical capability. The team found that these same individuals had not effectively emphasized good human relations or people skills to be effective, particularly when so many people concerns had been expressed by old culture employees. These employees felt that there was not enough balance between technical 2nd human relations skills and people were no longer rated as the i number one resource at Fermi. The team concluded that fermi management needed  !

to pay more attention to human relations matters such as maintaining group cohesivness and morale in order to effectively implement real and lasting l improvement.

3.1.9.1 Delegation The team did not find any uniform emphasis on delegating more authority and respontibility. As indicated earlier, seme old culture employees appeared to be resisting thJ acceptance of new authority and responsibility. In other ,

cases, managers or supervisors were practicing micromanagement possibly because -

they lacked cont 1denca or trust in their subordinates. Senior management

• appeared to be too involved (i.e., micromanaging) ih day-to-day business L matters that superintendents and lower level managers should have been doing.

Senior managers lacked confidence in certain lower-level managers and also attempted to compensate for organizational instability and weaknesses. This practice contributed to workload problems and inability to perform normal management functions. i l'

At some lower levels the team found too much upward dependence. Some pe6ple wao did not know management's philosophy, or what to expect, adopted an f attitude of taking everything to the boss before doing anything. The I observation made was "to be safe and avoid mistakes, take it upward or wait t until you are told exactly what to do and when." Some people also said they  !

did not trust management enough to take problems upward. If they expressed an j opposing view or different philosophy, they risked punishment. In some cases.  :

the team found high levels of stress. -

The team found the following kinds of problems related to delegaticn: f (1) subordinates were not always sure of their accountability, limits of their r responsibility, nor felt they had the authority to match their I responsibilities, (2) expectations were not always clearly explained at the f beginning of an assignment (3) work overload, and (4) inadequate follow-up by f the manager or supervisor to make sure the job was done on time and correctly.  !

In several cases, individuals indicated that if they did what they were told l and it was done right, they received little, if any, recognition. If they did j it wrong, even according to specific instructions, their supervisor did not {

I

[

36 I

support them. Because of the fear of naking a mistake, there appeared to be excessive upward delegation, s

3.1.9.2 Performance Monitoring and Reporting the team found that Fermi management had systems in place to comprise an

, adequate management information system, performance did not appear to be significantly hampered by a lack of information. Management used more than 16 ,

kinds of reports, plans, and numerous meetings to track activities. Rather, there seemed to be an inability to integrate planning and scheduling and act on this information. Part of this problem was due to the utility's reaction to outside influences, e.g., the NRC and INPO.

The operating problems experienced at Fermi since 1985 hcve had an adverse effect on Fermi's relationship with the NRC staff. Some of the events involved questions about full reporting which reduced Femi's credibility. The slow pace of improvement has also affected Femi's credibility with the NRC staff.

As a result, FernI was in a position of responding more to NRC pressure and less to internally generated priorities based on plant needs. This lack of credibility had also adversely affected the organizational climate, e.g., the Femi organization was in a reactive mode and communications with the NRC were strained. With respect to NR; comunications, Fermi management lacked a "take charge" attitude and a showing of confidence necessary to demonstrate their ability to exceed regulatory requirements and achieve a noticeable improvement in performance.

3.2 Operations In assessing

• plant operation, the team observed control room and in-plant activities of both licensed and non licensed operators, conducted tours of all aress of the facility, examined the interface between operations department and other departments, observed managerial involvement and effectiveness, examined operations improvement programs, and conducted reviews of logs ar.6 records.

The team also interviewed both licensed and non-licensed operators as well as operations department management. The team evaluated the Operations Department and the effectiveness of the operator training program. The team interviewed training departrant parsonrel, observed operator training sessions and reviewed initial and requalification training programs, training and simulator facilities, training staff qualifications, and management oversight and support for the pregram.

The team's rajer findings and conclusions included: (1) the operations staff exhibited good rcrale and a sense of ownership accountability and professionalism; (2) although their performance had improved, the operations staff lacked comercial EWR experience (other than experience at Femi 2);

(3) random and repetitive equipment failures caused an excessive nueber of challenges to the operators; (4) the operator's level of knowledge of plant syr.tems and Technical Specifications, and their demonstrated ability to implement the emergency operating procedures, varied considerably arong operators due to a gene? ally weak training program; (5) operations departrent management did not provide proper oversight of operator performance; (6) the operaticns departrent management at Fermi did not possess a sufficiently broad safety perspective to orceptly and full equiprent and system prcblems; and (7)the y recognize Shif t Operations the significance of various Advisors (SOAs) 37

were not effectively utilized or integrated into the operating shifts. These

.najor team findi.lg "id conclusions are discussed in the following sections.

3.2.1 0,9eratie ns Department Organization 3.2.1.1 Management The Operations Department was led by the Operations Superintendent, who ,

reported directly to the Plant Manager, as shown in Figure 1.3. The Operations  :

Superintendent was hired from outside the company in December 1986 to bring corrnercial BWR experience and perspective to the operations department staff.

Eight supervisors reported directly to the Operations Superintendent including the Operations Engineer. The shift crews report to the Operations Engineer, i

, 3.2.1.3 Shift Staffing - -

Operations has the ability to staff six 8 hour9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> shifts in accordance with the Technical Specification requirements. Each shift crew consists of a Nuclear Shift Supervisor (NSS), a Licensed Senior Reactor Operator (SRO), a Nuclear '

Assistant Shif t Supervisor (NASS), an SRO, 3 Nuclear Supervising Operators >

(NS0s), who are licensed reactor operators (R0s), and typically 4 to 6 non licensed Nuclear power Plant Operators (NPP0s). Each shift complement also includes a Shift Technical Assistant (STA) and an 50A. The licensee plans to upgrade 4 R0s to Sr.Os and 6 NPP0s to R0s in 1989. The licensee has also  !

indicated that in the future they would like to increase the shift corplement from 3 to 4 NS0s per shift.

3.2.2 Conduct of Operations The control room staff conducted themselves in a professional manner and were I uniforms which clearly identified their organizational positions. Control roon i access and noise level were well controlled by tha various NASSs observed.

Rope boundaries were used to deter unnecessary personnel from entering the control room during evolutions. The morale in the Operations Department was i

..')od, although the majority of the operators were aware that their perforrance i

! needs to improve. The stren ths and weaknesses of the staff in the conduct of operations are discussed as ollows. j 3.2.2.1 Level of Experienet i Although the majority of operators have a Naval nuclear background, SW1  !

experience among the shif t crews (other than femi 2) was minimal. During l l initial licensing, the recognition of this lack of BWR experiehce led to the '

requirement that personnel with experienc, at other nuclear power plarits (SCAs) i l be placed on shift to act as consultants to the shift crews. The operators l have gained experience during the s'artup and operations of the plant and this  :

has resulttd in an irprovement in op retor performance. However, the team I concluded that the operations departments ability to perform at a high standard '

is currently adversely affected by the operators lack of comercial EWR experience at a well-run plant.

The team also noted that there was varying levels of expertise between shif t l crews in overall plant operations. Discussions with operations managerent  !

indicated their awareness of a range in perforrance levels among the shifts. i The licensee attributed this, in part, to the amount of time some shifts have  ;

i l 38 t

4 been together, differences in abilities of the individuals, and the amount of operating experience. The team concluded that this latter item, level of expe-ience among shifts, had been compounded by management's tendency to schdu4 major startup testing and preplanned cor.. plex plant naneuvers when the b* d orming shifts were on duty. The team noted that this variance in v would continue to widen if operations management continued this

,hift Leadership i ., has the overall responsibility for all plant activities during his sh1ft. The licensee moved the NSS from the control room proper to the shift supervisor's office at the back of the contro' room complex. There the NSS makes all major decisions concerning activities that will occLr <iuring his shift; these decisions range from authorizing surveillances to ensuring that necessary logs and paperwork h6ve been adequately completed. Instead of initiating work at the NSO level and passing it through the NASS to the NSS with a recommer.dation for concurrence, the NSS initiated all shift activities.

Thus, all activities are prescriised by the NSS to the appropriate individual, i.e., top down. The team' concluded thac this top down style stifies initiative at the NSO level and does not effectively utilize the cellective judgement of the NSr and the NASS. This practice also limits the review and oversight funct10n of the ivS5.

For activities that were performed in the control room, ample supervision and guidance were provided by the NASS. Ir. contrast to the control room, the team observed a lack of direct supervision during activities involving manipulation of in-plant equipment. Discussions with NPP0s indicated that shift supervision seldom observed their work in the field and rarely provided feedback to the NP90s en the performance of their duties.

The team found that operations management oversight of operator performance of routine daily , Cant activities was weak. Operaticas management evaluation of operator performance outside he control room is discussed in Section 3.P.5.2.1.

? 2.2.3 Shift Work Practices in spite of management's emphasis on personal accountability, the team observed that weaknesses still exist in various work practices, as in the following examples of inattention to detail:

(1) The failure of an c;erator to initialize the abnormal valve lineup sheet after hanging tags (ced tag r2 cord / abnormal lineup No. 88-1 W7, tags 13 and 14) to Prform a quarterly preventive maintenance (PM) on an emstgency diesel generator (EDG), resulted in a second set of tags being prepared and a subsequent attempt to hang them.

(2) A lack of indication for a standby liquid control (SLC) system heat tracing circuii, was identified by the team to a control room opera tor. Two days later, the team checked to find that the operators had not yet investigated the problem.

Additionally, this condition remained undetected by the NPP0s despite required logkeeping on associated equipment. Subsequent

'19

a investigation revealed that the affected portion of SLC heat tracing was indeed operable; the lack of indication was due to a burned out light bulb.

(3) A review of an active abnormal lineup sheet (AL5) for condet. sate '

, system instrumentation dated April 17, 1988, indicated that the pressure transmitters for the suction of the north and south condensate pumps were valved out of service. The operators indicated ~  ;

that the control room indication from these transmitters had Leen functional during operational periods since April 17. Investigation revealed that the transmitters had been returned to service sometime near the end of the local leak rate testing (LLRT) outage (late

, April-early May) and that the ALS had not been appropriately updated.

The control room operator and NPPO logs were ge.nerally adequate, except for the following weaknesses:

(1) Centrol. room logs did not provide sufficient information in all cases to reconstruct events. For example, the sequence of events associated with a residual heat removal (RHR) systen, wa~terhammer that occurred during the evaluation could not be adequately reconstructed from log entries.

(2) The log sheets used by the NPP0s were not sequenced so that instrument readings could be collected in a logical panel-by-panel

fashion. This resulted in the NPP0s having to return to equipment previously inspected for additional reaf igs.

In addition, several weaknesses were noted in NPPO tog keeping practices.

The team observed NPP0s enter contaminated areas and memorize instrument readings, which were entered on log sheets after they exited the contaminated area. Common industry practices " svoid the need to memorize instrument readings include using communications devices to transmit the readings to other operators. In another instance, an NPP0 noted gauge values in an uncontaminated area but did not record the values until after exiting the area.

3.2.2.4 Role of the SOA and STA The SOAs were placed on shift (as a license condition) to provide tne shift crews with the benefit of personnel with previous commercial BWR experience.

There are six SOAs, one per shift. The SOAs are not being effectively utilized, however. For example, the SOAs could be used to ensure that adequate performance standards are maintained during day-to-day operations. The SOA on shift was not tipically consulted at the beginning of an evolution, but becan,e involved only when the crew ran into difficulties. Interviews with the NSSs indicated that they had little confidence in the expertise of the SOAs. The team attributed this, in part, to the removal of the SOAs from shift requalificatie ' dining activities. This lack of integration of the SOAs with the shift crews was previously identified during an NRC special operational readiness assessment team inspection in September 1985 and there was no indication that the licensee had corrected this deficiency.

The team observed that the STAS were appropriately and uniformly used for the resolution of technical issues. However, inconsistent utilization of the STAS in other capacities were noted from one shift to another. This inconsistency 40

ranged from delegating STAS solely to log-keeping activities on the one hand te STA involvement in routine shift activities on the other. Ciscussions with tht STAS indicated that the STA position was corsidered permanent and had no designated career path. The licensee 1:. no+. usirig the Slh position te develop personr.el with operations experience which then :ould be transferred to other departments.

3.2.2.5 Shift Relief and Turnover Shift relief and turnover were tho*ough and complete. The oncoming crew reviewed logs and turnover sheets, and performed panal walkdowns with thr ccunterparts. The NSS briefed the entire oncoming shift, including the health  !

physics (HP) and chemistry personnel, on activities and evolution: which were expected to be conducted during the shift. Short-term relief turnovers

, conducted to allow operators interim brer.ks were also thorough.

3.2.2.6 Communications Communications among the operators on shift were generally adequate. In the control rocm, the Operator: verbally inform.Gd each c"ier of each ection that could affect the plant. Communicatinns with operators in the field useslly '

included a "readback" by the operator receiving the transmission to ensare complete and accurate understanding of the me: sage.

Communications between tne oper.ating shifts and maintenance personnel were observed to be improving (see.Section 3.3.5). Thic .vas der to the licensee initiating three daily meetings (at 8:30 a.m., noon, and 4 p.m.) between the organizations to discuss the status of work. In addition, the licensee had placed SRO-qualified personnel in the maintenance department to assist in the scheduling of work and the preparation of work packe.ges.

The team found that communications between the operating shifts and other support organizations were marginal. The operators displayed a reluctance to approach the engineering or licensing departments for assi:tance. Several operators indicated that less than satisfactory responses from engineering and licensing in the past had resulted in a loss of credibility. This loss of credibility combined with the sense of personal responsibility and accountability for all aspects of plant operations exhibited by the operators had resulted in their reluctance to request assistance from outside the operations department.

Examples include the shift crew utilizing onshift personnei to: -

(1) Troubleshoot a relay card problem, .

(2) Walkdown the RHR system to check hangers and sr,ubbers following a water hammer (although engineering expertise was subsequently requested by the NSS and received), and (3) Resolve a Technical Specification interpretation concernino actions

• to be taken following a loss of rod position iridication. (Weaknesses in the operators' understanding and implementation of technical specifications are discussed in Section 3.2.7.3.)

c 41

g.. g, -

+: -- - -

7 7 7 y ( , .

r .

m ,y i.2.3 Materhl Ccndition of the Nant

'9 3.2.3.1 Control'R nm and Associated Instrumentat%n a .,

I' The team f auWi that, in generai, control room instructstation was's.intained in '

3'3dtisf actory condition, except tnat the mai,ntenGnce of Control room j

- int,trumenta tioa fnyc1eing balance cf plant equipment.was per. Approximately 90 items involving contro'l roem instrumentation required maintenance or e engineerir.g attention. These inciudad the following examples:

,' (1) The ininimum flow a lve in the soutn heater drain pump recircula' ion '

line had a body-co-bonnrt leak, causing the operators to ute the  :

} redundant pump as the preferrad mode of ap9rr. tion. '

(2) ihe operators were reqpired to turn the north turbine building closed ,

couting~ water pump controi .#ftch to RUN and then wiggle it in ordar to rtart the puma. ,

3., Battery 2PB clarm annunciator felsoly alarmed interm ttently and was considered a rui ance alarm by' the licensee. *

j. '

ine cb rt recorders for bala:.r.e-of-plant equipment were unreliable. In i contrast, the pri.aary p' ant r. hart reccrdct. have receb ed adequate attention .

. and are in significantly better conditi;..i.

! i "ite tean concluced that tne poor maintenance or contr21 t 'i.m instrumentatien t 4

involv.ing b'alence of plant equipment detracted from the the operainr ' ability tc operate the plant. t

$ 3 J.3.2 Equipment ur.d System Operability .

The team oberved many weaknesses in the reliability and availability of plant equipment. The team witnessed numerous challenges to the ootraters caused by j inadequately peWorming equipment. Exagies included: '

! (1) Repetitive pro'alems with exWa: tion steam valvet r-ssulted in significant delayn in plant operations.

I 1 (2) lia repetitive failure of the B recirculation IJop discharge valve to  ;

strcke resulten in 2 plant Outdowns withir. 10 dayt -

(3) Problems with the heatur dra<ns resulted in constent cycling of the  !

valves controlled by the feedwater hsater level control system.

, (A) The feedwater system controls caused the reat. tor vessel level to

• cycle from the low to high alarm setpoints during plant shutdown. i Most operators readily a nnowledged a gen wal frustration with the poor l l performance of'vacious pieces of equipment. In addition to those alread,

! mentioned, the operators also cited the turbine generator cc..trols and the j j reactor water cleanup system.  ;

1 The team found several valves that are routinely owrated vere difficult to i

~

access from the floor. For example, the operator iad to climb on enduit and pipcs to reach the diesel generator expansion tank f411 valves. ,

1

)

j 42 L -, . - . - . - , - . .

w 7 q.7  : .

lI' x

- (. s' The teait founc revisions were baing m:de to procedure: 'c. compansate fo:

degraded equipment. For ext..iple, th; applicable y ocenure fov P amergency 7 equi; ment cooU.ig water lEE(M tystem hao been reviseo ', allow ccm op2raticn despite valves G33-F001, G23-F101 and N L 3-F60. Sr.tig .a a degradeJ condition. In addition, plant. operations centinued despitt wi.rped south

< feedwater pump ro'.or, Tnis caus d the south Nactt.r feedwater pump to require a 1?-hour warm 9n period, thus limit'ig plant flexibility.

The teatM Overall concl tion'was that poor eq0ipment reliaoility and availouili., c hcid an excessive burden on the plant operators. Furthermore, the team concluded that plant management was unaoh..to retolve equipment issues

( ac to saaintenar.ce and engineerinq rcsource problems (sem Sections 3.3.1 and

}, 3.6.1.3).

l 3.2.3.3 E usekeeping Areas traversed.frequentiy by m.ragers W senior supervisors were clean, painted, 'ni clear of ut r. ructions. Outbuildings and area of the plant nrt ustly accessible are characterized by oiscarded trach, scattered tools, 4

steined floors, and loose nuts, bolts, and light bulbs. Operations management float the NSS to the Plent. Manager sehiom tour out-of-thsway areas of the plant. For examp'.a M The RHR building was infrvuently toured by management. From June 1 through August 24, 1988, tt.e Plant Matieger had bJen in the building {

tw'ce, the Operations Superint0ndent had not been in the building,

and Oe Opi ations E.gineer tad been in t'e building onco.

• c (2) In add:ticn, elagament infra uently +oured tne reactor building.

! From June 1 through August 24, 19P8, the Plant M w nger had been in

,the'buildir.g twice, the Operations Snerintendent once and the

. Oparations Engineer three tines (twice on tH sam

• day). Most of the

NSSs also did not routinely tour :he rcactor butluing. Although one ISS had be3n in the buildinq sever, timas, one NSS had not entered the builsing during chte pt.rfod surveych although it thould be noted that

the reactor building was contambated during that period and tt.et

. c.ess-out was rJquired for entry into most of the building.

l

[ The equipment labeling system was excellent. Valves. pumps and runs of piping ;

were well identif'ied. Maps at the en'.'r4 r.as to each room within the various

, buildfngs showed the lo:ations of major pieces of equipment and entry and exit ,

doors sad were judged ".c be a cJetul perscnnel aid. However, several maps were worn to the extent of Ming illegible, as fer examp h, the saaps located at the ,

( en*rance to the reactor building, d

The tean also noted the following deficiencies:

l r

l (1) Sev : al deactivated card readers, such as those at the exits from the l battery rooms, no longer had tags iridicating they had been j

, deactivated. ,

i

! (2) Severa' fire door closhg mechanisms were innperable and thus did nnt close the doors completely.

I L

I -

43 l

L

y . - ~ > _ _ .

1 .

p ,' .

~

u 3.2.4 System Cnnfiguratior. Cor.trols l D

3.2.4.1 Tagouts for Work Requests-The PM work packages prepared by mainterance lacked sufficient work scope explanation for the operators to adequately determir.e the needed protection

, (taggingreqr'rements). This was not a problem with corrective maintenance -

(CM) work packages. The lar.L' of sufficient work scope in the PM work packages created tr.e need for operators and maintenance personnel to jointly spend time cetermining the required protection. This extra requirement reduced personnel efficiency, slowed the work' process, and increased the probability for error c

, which could, in turn, lead to plunt challenges to the operators.

l 3.2.4.? Controlled Drawings j

~

Thre of ten control roors drawings which were mdited had red-lining deficien- l cies. Red-lined drawings served as an interim r.otification to the operators of 4~

a ccqfiguration che.nge and remained '.s. effect un;.il the drawings are revised to incorporate the oitstanding changes. Controlled drawing 650721.2530-12 was not red-lined to reflect the changes made by enginee. ring design package (ROP) 3793, r b Controlled drawings 61721-2679-1 and 6M721-5741 had incorrect red-line changes to reflect EDF 4800, even'though the latest revision of the drawings had

^

already incorporated the correct EDP 4800 changes. The licensee determined that an STA han incorrectly red-lined both drawings. Although these items had ,

minor safety significance, the team determined that this represented another exuple of intsttention to detail (f ee Section 3.2.2.3 for additional examples).

3.2.5 Improvement Programs 3.2.5.1 Procedure Improvement Program -

The licensee was rewriting and upgrading selected plant procedures as part of a procedure improvement program during the evaluation. The licensee first  :

informed the tGC of the need tu improve ddministrative procedures in the l

, Reactor Operations Improvement Plan (ROIP) on October 10, 1985. The scope of t 2 the procedure upgrade was later increased to include operating and surveillance  ;

, procedures as well as administrative ,

! Operations Improvement Program (NOIP) procedures submitted to the NRC in the on licensee's May 9, 1986. Nuclear The  !

procedure improvement program was scheduled to be completed by December 1988 l and had been incorporated into the Fermi Business r.an, i 3.2.5.1.1 Operating Procedures The team was limited in their review of the operating procedures because the licensee had completed and issued only a few procedures at the time of the >

evaluation. Therefore, the assessment of the procedure improvement program was based on the process for improving the procedure 3.  :

The improvement program emphasized the reformatting and structuring of the procedures for consistency so that they are in accordance with human f,"tors guidance provided by the licensee. This process included ensuring that all i notes, cautions, and special instructions were properly placed before the applicable procedural step. Also, all notes which contain a prescribed action ,

were being eliminated and the action incorporated as a numbered step in the  !

procedure.

i L

$4 >

,mer,-

m y ee r ,,em, n v- , ,e-v4, -.-,w-,----,v,-- ,-w+,m ,----,,,-m, .-em,-~,,--m--n- w,- ~, -,e ,~,--ea,,,n

9 The process lacked program review by an outside organization. The procedures were being prepared by an autonomous group within the operations department and decisions for requesting input from other departments (e.g, training, engineering or technical support) were generally made by the group leader. The action flowpathing of the procedures was performed by one irsiividual without formal peer review for accuracy.

At the time of the evaluation, the licensee had completed approximately 15 of 120 system operation procedures and none of the 11 involving integrated plant operation (plant startup or shutdown). Although a great many of these procedures were in various stages of the improvement pipeline, there was no detailed schedule for completion of the procedures in order to meet the program deadline.

3.2.5.1.2 Administrative Control Procedures At the time of this evaluation, only one of approximately 20 administrative control procedures had been approved and issued. Therefore, as was the case with the operating prucedures, the team evaluated the process for improving the administrative control procedures rather than evaluating the procedures themselves.

The process for improving the administrative control procedures included several feedback mechanisms and a great deal of operations management attention. The procedure upgrade consisted of condensing approximately 100 duplicative and redundant administrative control procedures into about 20 procedures. A systematic approach was being used to ensure vital information was not lost in the condensation of the procedures. The draft procedures were circulated among the NSSs, the operations management, and support departments for review and comment. A system was in place to ensure that these comments were addressed.

The licensee had no programmatic controls for implementing the revised administratise control procedures. The operations department management implemented the necessary training for each new procedure on a case-by-case basis. For example, the first approved procedure was implemented prior to training personnel. The training was later performed by an NPP0 for each shift crew immedictely af ter shif t turnover. In contrast, a complete training program had been prepared and was scheduled to be completed prior to issuance the next administrative control procedure to be implemented. In light of the significant problems associated with the implementation of administrative control procedures in the past, management controls for implementing the revised administrative control procedures was weak.

3.2.5.2 Personnel Programs 3.2.5.2.1 Evolution Evaluation Program The licensee committed to performing evaluations of operator performance of various evolutions twice per week from November 1987 through the completion of the plant warranty run. The purpose of the program is to provide shift operators with on-shift traink.g in the conduct of normal plant evolutions.

Each shift performs evolv* Rns which are monitored and evaluated primarily by operations department n. onagers against specific performance standards. At the 45

time of its inception, the licensee anticipated that the program w uld run for approximately 3 months.

Initially, the program was effective and successful, providing valuable feedback to the operators on the performance of their duties. As the program  !

progressed beyond its anticipated lifecycle, the same evolutions were performed repeatedly, and became redundant. *he evaluation of operato s was conducted primarily by operations department ,pervision. As operator performance improved, operations department c M gement did not raise their expe:tations of acceptable operator performance, this is evidenced by a March 17, 1988, status report from the Operations Engineer to the Plcnt Manager stating that operator performance was judged to be "Better than Average" in 34.7 percent of the l categories rated. However, in discussions, senior management indicated that the operators were performing at a minimally acceptable level and that '

. continued improvement was necessary.

The team determined that the Evolution Evaluation Program needed to be revitalized. The team considered the practice of having operations department supervision perfor.n the majority of operator evaluations, rather than having a support organization such as quality assurance (QA) perform the evaluation to be a weakness.

a The team found that the managers evaluating operator performance did not typically monitor operator performance outside the control room. The audit of

! past evolution evaluations indicated that only 1 of approximately 50 evolutions evaluated NPP0 performance in the plant. In light of the career path of NPP0s (to futum licensed operators), the team concluded that the program did not adequately establish performance expectations or evaluate in-plant operator performance.

3.2.5.2.2 PRIDE Program The PRIDE (People Really Involved to Develop Excellence) program was initiated at the request of the Chairman of the Board to provide the operators with a vehicle to resolve plant deficiencies. PRIDE meetings are held on Monday i mornings for each shift during its training week. They are attended by an

, entire operations shift crew and a PRIDE facilitator. Their purpose is for the

, shift crow, led by the NSS, to identify an operational problem, define its dimensions, select and develop a viable solution, and establish a plan of action utilizing the shift crew to resolve the problem. For example, one current problem being addressed by a shift crew is how to best identify items in the control room that are Technical Specification related, such as recorders and valves.

Although the program has the potential to provide a number of benefits (e.g.,

increased teamwork), the scope of the program was inadequately conveyed to the operators upon its inception. Initially, operators prceived the program as a l universal method for resolving all equipment, proew ure, personnel and any i other problems experienced at Fermi. When the operators flooded management "

with perceived problems, the program became overloaled and many ,

operator-identified problems were inadequately handled. Problems and issues were dropped without explanatiot or satisf actory resolution. By the time the actual scope of the progrem became apparent to the operators, the credibility  :

of the program had already been seriously damaged. However, despite its poor I

46 l

. _ , . ~ _ _ ,_ _ __ _ ,. _ . _ _ _ __.

start, the operators indicated that the PRIDE program had improved and become more effective as a problem-solving and communication tool.

3.2.5.2.3 Operating Practice Standards The operating practice standards were prepared by operations department management to help convey management's performance expectations to the control room operators. The operating practice standards provide the operators with One guidance as theyexists set of standards conduct for routt'1e each of plant evolutions the job and categories onday-to-day shift e.g.,(business.

the NSS, NASS), with the exception of-the STA and the 50A. An 4ditional set of standards provides general guidance appropriate to al. members of the shift crew. For example, Operating Practice Standard 105, Equipment Operation, instructs an operator to point to the label on the component and state both the name and number of the component to ensure the correct action is taken prior to actual component operation.

The team concluded that the operations practice standards are a useful management tool for conveying performance expectations and operating philosophy. The operating philosophy and performance expectations described in the standards were not incorporated in the current administrative control procedures and were not an adequate substitute for formal prescriptive adminis-trative control guidance. The administrative control procedures necessary to provide formal administrative guidance were currently being improved and rewritten as part of the procedure improvement program.

3.2.6 Management Performance 3.2.6.1 Ove rti'me A review of the operations department personnel overtime records from March 1 through August 31, 1988, indicated that overtime guidelines were regularly exceeded in April and May (during the LLRT outage). For example, one NSS vorked 12-hour shifts for 11 days consecutively without a day off. Although authorized, the overtime worked by the NSS did not meet the intent of Technical Specifications and represents a significant management weakness. However, records for June through August indicated that management had since taken positive action to properly administer and control operator overtime.

3.2.6.2 Safety Perspective The team concluded that operations department management did not possess a sufficiently broad safety perspective to promptly and fully recognize the significance of various equipment and system problems. This concern went beyond operator compliance with Technical Specification requirements and is directed to the operations management. Management was slow to recognize the significance of equipment problems which warrant immediate attention and the exercise of conservative judgement. Examples include the various problems experienced with safety-related motor operated valves (MOVs). The magnitude of the MOV problem was readily discernable in the deviation event reporting process and known to engineering and maintenance personnel (see Section 3.3.4),

but the operations department was not aggressive in pursuing a course of corrective action. Another example was an event in which the EECV system required approximately 5 hours5.787037e-5 days <br />0.00139 hours <br />8.267196e-6 weeks <br />1.9025e-6 months <br /> of venting because of air intrusion. Management was aware of the air intrusion into the safety system but was slow in 47

~

c .-

recognizing its safety implications. The system was not declared inoperable despite the need for an inordinate amount of venting. A third example was the failure of the operations department to recognhe that the oper:bility of the Division II non-interruptible air supply (NI.\S) control air compressor (CAC) '

was required to support the operability of a standby gas treatment subsystem, control room emergency filtration system damper, and main steam isolacion valve leakage control subsystem.

3.2.7 Training 3.2.7.1 Training Staff The operations training department consisted of 10 instructors and 2 supervisors. This staffing level was sufficient to meet the normal training

., needs of the operations department. However, de.partment resources were sevorely strained during the past 2 years in order to support special projects, such as development and implementation of the new Emergency Operating Procedures (EOPs), additional Technical Specifications training, and rewriting of course materials. These special projects sometimes involved reassignment of some personnel from one task to another with little time allowed for personnel to become familiar with one task before being reassigned to one of higher priority. This process contributed to the low morale anong some members of the staff. Morale had also been adversely affected by the impersonal management style of the operations training department supervisor. At the time of the evaluation, the training department was in the process of reorganization, and this individual was stepping aside to allow for enhancement of supervisory leadership.-

The licensee had increased the plant operations experience level in the training staff in response to weaknesses previously identified by the licensee t

and the NRC. This had been accomplished by exchanging three personnel between ,

the operations department and the training department, and the licensee was in the process of transferring one additional operator to training. However, several operators perceived that operations department management was taking

. this opportunity to resolve several internal personnel problems. This perception has the potential for undermining the gains in creditability associated with the transfer program. Another weakness previously identified involved the knowledge level and qualifications of the operations training department staff. This issue had been addressed by the establishment of a formal instructor qualification procedure in March 1988. Implementation of this procedure was anticipated to be a positive step in improving the performance snd credibility of the department. However, i.he team could not assess the effectiveness of this change because only one instructor had been broaght into the department since its implementation. .

The operations training department was deficient in auditing instructor performance. The Fermi Nuclear Training Business Plan specifies that four different types of evaluations are to be conducted for each instructor i annually, in addition to a periodic audit by supervision or independent consultant. Although the department consists of 10 instructors, only 8 out of a possible 40 evaluations were performed between February 9,1987 and March 23, 1988, i

l 48

/

3.2.7.2 Operator Training The quality and effectiveness of the initial and requalification training programs were marginal due to below average instructor performance caused by low morale (as discussed in Section 3.2.7.1), and less than adequate program materials and facilities, as exemplified by an out-of-date student text, lack of good lesson plans, and a simulator with limited capability to provide challenging scenarios. The licensee indicated that plans were in progress ta upgrade the capability and fidelity of the simulator. P,ased on these deficiencies, the team concluded that the marginal operator training program was the result of inadequate management attention.

The team also concluded Inat these training problem contributed to tne numerous random waknesses in operator knowledge that the team observed during tneir performanc These weaknesses are o,e on the sirial*

xemplified by thetor, board walkdowns, following and observations interviews.

made by the team, o While attempting to lower power on the simulator, an NSO continued to attempt to reduce recirculation flow by the master controller even though it was on its limiter.

o W511e an NSC was attempting to rapidly drive rods in on the simulator, another crew member had to point out to him how the emergency-in switch could be used to accomplish this action.

o Several operators could not explain why" a second control rod drive (CRD) pump is required to be started by the E0Ps.

o Several operators did not know that isolation of the RHR valves, E11-F015 A or B, on a shutdown cooling isolation signal, could be detected by a lit indicator light.

o One NSO did not know that RHR valve E11-F028 would isolate on a loss-of-coolant accident (LOCA) signal. He mistakenly indicated that RHR valve E11-F608 would close on a shutdown cooling isclation signal.

o One operator could not explain the purpose of the recirculation system.

o On shift, an operator demonstrated lack of understanding of the potential impact of having the RHR system in an off-normal configuration which resulted in a water hammer event.

o Four out of five operators did not have an in-depth knowledge of the English Electric Turbine Control system.

The team also found a substantial difference in the ability of the shift crews to efficiently utilize the E0Ps during simulator scenarios. One shift demonstrated the ability to utilize two procedures simultaneously while exhibiting a sense of control of the plant and knowing the direction the transient was progressing so as to anticipate the next step. This shift crew anticipated the direction of the transient ind was so well prepared for the next E0P step that the team questioned if che shift had been recently trained on this scenario. The instructor indicated that this was not the case.

49

In contrast, two weeks later, cnother shift's performance, though successful in controlling the three scenarios, exhibited few if any of the above attributes.

On one scenario involving a contairment challenge and anticipated transient witnout scram (ATWS), the team concluded that the NASS implementing the E0Ps was unfamiliar with the procedures. The NASS gave no indication that he was anticipating the direction of the transient or the necessary mitigating steps.

He took a long time to read and then reread the procedures prior to issuing instructions and he took a long time to acknowledge some information supplied by the panel operators. Finally, the NASS executed only one procedure at a time for approximately 15 minutes after the transient had developed into two well defined paths, containment and ATWS. Although the team determined that the transient was controlled in accordance with the E0Ps, it noted that the scenario was straight forward and one that the operators had previcusly experienced during training. Therefore, the team did not gain confidence that this crew would respond well to a more complex transient.

The team was unable t) determine the cause for the apparent difference between shifts. The training department supervisor indicated that the individual NASS had successfully demonstrated his knowledge and familiarity with the E0Ps in other simulator training sessions and during classroom training on the procedures. The licensee attributed the team's perception to the personality of the individual involved and the recent emphasis on careful reading of the E0Ps. This emphasis on careful reading of the E0Ps was due to a previous shift having initially missed two procedural steps of an E0P during simulator training. Though the team considered the licenste's position plausible, it concluded that weaknesses exist in the shif t crews' capability to properly utilize the E0Ps. In light of this information, the team determined that additional training for the weaker shift crews on the implementation of the E0Ps is warranted.

In order to further assess the shift crews' capability to effectively implement the E0Ps, the team also evaluated information concerning previous simulator audits, one conducted by the NPC and one conducted by INP0. The team concluded that although these previous audits reported adequate results, they may not have been representative of the licensee's overall capability to implement the E0Ps. These audits observed two different shift crews, who were also different from the two crews observed by the team. The shift observed by the NRC E0P inspection had just completed their normal training cycle, which included the use of the E0Ps. In addition, the NASS, the individual directing shift activities in accordance with the E0Ps, had been involved in the development of the procedures. The shift observed by INPO had requested and received special, additional training time on the simulator the week before the audit to address self-perceived weaknesses.

The simulator was not being used effectively as a tool to discover, and thereby address, differences in performance of the shifts as well as the random weaknesses previously noted by the team. The team observed training on the simulator which involved the use of only one instructor, who served as both machine operator and evaluator. This p.*actice can result in minor maloperation of instrument controls and subtle procedural execution problems going undetected. The team noted, however, that the final simulator evaluation gh .n at the end of the training week had involved more that one instructor and was observed uy one or more managers, including senior management.

50

The licensee had not maintained classroom training materials, such as student texts, classroom tests, and viewgraphs current with the plant configuration.

The student text, the primary systems training reference for initial license and requalification training, had not been updated since 1984 to reflect changes in plant design. In 1985, during an INPO training accreditation audit, the need to updato the student text was identified. In 1987, the NRC pointed out that the student text, as well as various instructor materials, required revision to reflect actual plant status. In February 1988, the licensee began a pilot program to revise the student text. At the time of this evaluation, the licensee had just begun to upgrade the training materials, such as classroom tests. Although the licensee performs a certain amount of on-shift operator training (through required reading and brief after-shift. meetings),

the team considers this an inadequate sub titute for maintaining trainirg materials current with the actual plant status. The team also found tMt the

, lesson plans in many instances were superficial, e.g., the lesson plans were so general that the material to be included in most lectures is left to the discretion of the instructors.

3.2.7.3 Technical specification Training The team reviewed the licensee's action to resolve noted weaknesses in the operators understanding and implementation of Technical Specifications. The licensee had developed and was in the process of implementing a Technical Specification Improvement Program to address this issue. As part of the program, the licensee had ,imr amented a 2-week training course in Technical Specifications, which had been 'eveloped in conjunction with General Electric.

This course provided a good foiadation for the safety and technical basis of the Technical Spacifications. However, the team concluded that operator training had not provided the necessary insight into the expected safety perspective and conservatism which i'. incorporated into the specifications over and above the design technical bases.

The licensee also had developed a library of Technical Specification case histories to train the operators on possible operating scenarios involving Technical Specification interpretations. The case histories were being used in class during requalification training week. Although this had been a concerted effort by the licensee, the program had several weaknesses: (1) the majority of case histories were simple, straightforward lookup exercises that required little knowledge of the specification, (2) most of the specification usage was per'ormed by the NSSs and NASSs, leaving the NS0s as observers and not Nrticipants, and (3) during classroom training the team observed that solutions had been given to the operators without sufficient explanation of the reasoning behind the solution. The team concluded that this trairing method had the potential of improving the operators knowledge of Technical .

Specifications; however, the program implementation was weak and ineffective.

3.1 Maintenance and Modification The evaluation of the Fermi maintenance program ine.luded a review of the preventive maintenance (PM) and corrective maintenance (CM) programs for the emergency component cooling water (ECCW), high pressure coolant injection (HPCI), low pressure coolar.t injection (LPCI), reactor core isolation cooling (RCIC) systems; a review of maintenance and testing of safety-related motor-operated valves (MOVs); interviews with station maintenance managers, staff engineers, and technicians; and a review of the Maintenance and 51

Modification (M&M) Department staff qualifications and organization. To a lesser degree, the team evaluated maintenance procedure adequacy, maintenance technician training, and the use of feedback of industry operating experience '

in the maintenance program.

Despite noted improvements in maint9 nance, the team concluded that significant work was still required by the licensee'for maintenance to become a good performer and effectively support plant operstions (i.e., improve reliability of equipment). This conclusion indicates that management changes and new programs had not been fully effective either because of poor identification of ,

root causes of problems or a need for additional time before the changes manifest themselves as a more discernable improvement.

The team found that problems persisted in the areas of planning, scheduling, and spare parts availabiljty due to: poor comunication and coo,'dination ,

between departments and groups; lack of preplanning of material requirements; i

and poor preplanning of work packages due to inefficiency within the mechanical  ;

and electrical maintenance group. These deficiencies resulted in deferral of

PM and CM, delayed completion of work activities, and contributed to increased l unavailability of plant systems.

Despite an extensive program for periodically inspecting and lubricating MOVs, 4

the licensee's overall program for ensuring reliable MOV operation through good  !

engineering and maintenance practices was inadequate. Numerous MOV failures  !

and problems due to incorrect torque switch and limit switch settirgs were r

! recurring. Although many of these problems were recognized by some licensee (

personnel, as evidenced by DERs, trending reports, and engineering consultant a findings, the licensee had not taken sufficient steps to resolve the problems t j effectively. The team concluded triat due to poor departmental comunications, L a the licensee failed to recognize the magnitude and significance of the MOV i problems and, therefore, failed to devote the necessary resources and

priorities to implement the required cor ective actions. Poor maintenance practices, particularly with MOVs, had been a contributing cause of plant scrams, challenges to the operators, and plant unavailability.

I Although it appeared that the licensee had provided a good foundation to  !

j improve the performance of the M&M Department and some improvements had been j observed, the team concluded that further improvement efforts could be slowed 4

by several factors. The team found that there were limited technical resources i to support maintenance programs. The team found the Technical Engineering j group and MAM Department were not staffed with a sufficient number of t i

experienced engineers. Consequently, the areas of PM, spare parts, and r equipment failure trending were not receiving sufficient attention. Other i 4

factors such as the adverse impact on existing resources caused by tra.ining and  ;

implamenting high quality improvement programs in conjunction with providing  !

adequate plant support, the use of inadequately trained contractor craf t  !

personnel to perform maintenance on plant equipment, and the continuing ,

restructurir.g of the M&M Department may also slow further improvement efforts.

Improvement programs to resolve PM and spare part problems have not been implemented and may require several years to complete.

l 3.3.1 Haintenance and Modification Department Staff l The team found the staffing and organization structure of the M&M Department

had been in a continuous state of major transition. The restructuring came i

i j S2

_ - _ . - - - - _ = - - _ _ - - . --- -- _-

about in response to resolving management issues and addressing plant problems.

Some of the more significant changes included: (1) relocating maintenance planning and scheduling from the Planning and Scheduling Group to within the M&M Department, (2) hiring a new M&M Superintendent, and (3) transferring the previous M&M Superintendent and approximately 40 personnel from the M&M Department to the Nuclear Materials Management (NMM) Group. The team found that of the recent changes in M&M Department management, the most significant was the new M&M Superintendent. This superintendent had extensive industry experience in plant operations and maintenance. On the basis of discussions with the new M&M Superintendent, he appeared qualified and an asset to the management team. At the time of the diagnostic evaluation, the team learned that another major departmental reorganization was imminent and that an additional employee from outside OECo was being hired to strengthen the M&M Department management team.

There were approximately six technicians for e'very foreman in the department.

The team considered the span-of-control uf the maintenance foreman and the overall staffing level of the M&M Department to be adequate, with resources strained as discussed below.

(1) Discussions with the M&M Superintendent indicated that the instrumentation and control (!&C) group maintenance backlog had been slowly increasing due to the diversion of existing resources to support the procedure upgrade effort and due to the frequent occurrence of having less than a full complement of I&C technicians ,

because of attrition and th) long lead time to train new I&C personnel. Other interviews indicated the procedure upgrade effort has put the I&C group behind on technical surveillance reviews by approximately 2 months.

(2) The mechanical and electrical maintenance group had approximately 61 craft personnel, of whom 12 were committed to training through 1989.

In addition, the group had approximately 10 additional personnel supporting the procedure upgrade effort. Although the licensee had redirected construction craft personnel to replace the personnel assigned to write procedures, the capability of this group had been diminished because the replacements did not have a significant amount of equipment repair experience, and due to the training commitments.

The licensee had initiated a number of actions to improve maintenance perform-ance which included: (1) hiring three additional I&C technicians, which had been effective in stopping the increasing !&C backlog trend, (2) reassigning of responsibilities ar.J organizational changes to improve the efficiency of the mechanical and electrical maintenance group, and (3) placing SRO-quali.fied personnel in the planning and scheduling group to improve work package preparation. The team observed improvements in the areas of planning and scheduling, communications between operations and maintenance, and a continuing decrease in the CM backlog.

The recent maintenance organizational changes had provided a good foundation for improvement, but improvements were being slowed due to strained resources and the continuing restructing of the M&M Department. The team observed that progress had slowed in reducing the Cd backlog since the April 1988 local leak rate testing (LLRT) outage and the number of maintenance-related control room instrumentation deficiencies appeared to be trending upward. At the time of 53

] q the diagnostic evaluation, approximately 90 items involving control room instrumentation required maintenan:e and/or engineering attention.

The practice of using inadequately trained contractor craf t personnel to

, compensate for strained maintenance resources had an adverse impact on the reliability of safety-related equipment. This is because the licensee had no program for qualifying contractor personnel on specific equipment and/or systems, and contractor personnel were used by the licensee to perform maintenance tasks on safety-related as well as balance-of-plant (BOP) equipment. Contractor craft personnel were used by the licensee during the spring outage to disassemble and check it0V springpacks for grease intrusion ,

which required the removal and reinstallation of the torque switches. Although l the craft personnel received some general training, they received no specific ,

training in MOV maintenance. Five valves, including the reactor recirculation i pump B discharg conducted durine valve, which had failed to close during a special startup foundtestt, g the diagnostic evaluation (see Section 3.3.4.1) were '

with their torque switch improperly installed in a preloaded condition.

3.3.2 Instrumentation and Control Training 4

The I&C Training Department had just implemented a continuing training course for upgrading and requalification of I&C personnel in identified areas. In '

addition to covering systems and infrequently used fundamentals, training ,

covered lessons learned. The lessons learned portion of the course reviewed applicable current industry and site events. Specialized technical training

was also to be included when required. This relatively new course was to be scheduled such that a full cycle of the training for each shif t would be completed in a 3-year period. The team concluded that the I&C Training Department had an excellent training program for I&C personnel, which should help eliminate the type of personnel errors experienced in the past. For this first quarter of 1988, a licensee event reports (LERs) involved the I&C group, of which 2 LERs involved personal error.

3.3.3 Preventive Maintenance and Corrective Maintenance Programs 3.3.3.1 Preventive Maintenance Program Although the scope of the PM program appeared to be broader than the industry average, the team found that the program did not fully support equipment reliability and had contributed to safety system unavailability due to problems in program development and implementation. The two most significant problems identified were: (1) the licensee had initially identified an unrealistically high nutber of PM activities to be performed for both safety-related and BOP equipment; and (2) there was a failure to properly prioritize the PM activities and to integrate them with surveillances and CM after they had been identified.

As previously noted by NRC Region !!!, these problems had resulted in a substantial number of PMs that had either never been performed and/or were past due to be performed; and that as many as 25 percent of the delinquent items identified were safety related.

The PM program was initially developed to comply with vendor manual recommenda-tions for the initial equipment list, but the program apparently did not consider that a disparity could exist between vendor manual recomendations and the requirements of actual operating conditions, nor did it differentiate equipment on the basis of safety significance or potential impact on plant 54

O 9

reliability. Consequently, the PM tasks totaled over 7000 items covering most equipment in the plant. During the last 9 months, the licensee conducted a review of PM tasks against vendor manuals, prioritized PM tasks, and significantly reduced the backlog of PM during the April 1938 LLRT outage.

Despite improvements in this area, many problems still existed and were in the process of being addressed by the licensee during the evaluation.

The current PM program did not appear to be adequately balanced to provide effective program coverage between safety-related and B0P equipment. "A" priority tasks covered safety-related equipment and critical BOP equipment important to plant reliability and availability, and "B" priority tasks covered all other plant equipment, including B0P equipment not covered under "A" priority tasks. There were very few "B" priority tasks being performed.

Written justifications were required in order to perform "B" priority tasks.

In addition, current PM program requirements had resulted in rigid.PM schedules. Based on discussions with licensee personnel, the team found that the rigid PM schedule requirements had caused problems in the coordination of PM with surveillance schedules and CM activities, and resulted in safety systems being taken out of service to perform only PM. For example, based on a review of safety system unavailability for the month of June 1988, the team found that on the average, one division of a safety-related system was out of service to perform PM for approximately 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> each day.

To improve the methods for changes to PM requirements, primary responsibility for monitoring the effectiveness of the PM program was transferred from the M&M Department to the plant system engineers in the Technical Engineering group.

Transferring responsibility for monitoring the PM program to the Technical Engineering group appeared functionally correct and should result in an improvement in the implementation and performance of the program. However, the near-term effectiveness of this change could be limited because the system engineers were already overburdened (see Section 3.6.1.3).

The licensee was developing an improvement program to address many of the identified weaknesses in PM program. The improvement program would require approximately 9 to 12 full-time personnel for 2 years, appeared comprehensive, l and adequately addressed most of the identified weaknesses within the PM i program. Due to insufficient maintenance resources, program development and implementation were contingent upon receiving contractor technical support.

Thus, substantive improvement would not be expected for several months if not longer.

3.3.3.2 Corrective Maintenance Program The team reviewed the licensee's effor'.s to manage the CM backlog and Jnade the following observations:

(1) The M&M Superintendent had a thorough knowledge of the status of the l maintenance backlog and plant problem areas that required resolution.

! The CM backlog since the April 1988 LLRT outage was slightly trending downward, indicating that licensee effort s to reduce the backlog were l beginning to be effective.

l (2) Although licensee efforts to reduce the CM backlog were beginning to

! be effective, discussions with M&M Department management revealed that little progress had been made in substantially reducing the 55

4 number of open CM work requests that were greater than 3 months old.

The team attributed this to the licensee's persistent problems in effectively planning and scheduling maintenance activities and problems in obtaining necessary spare parts. Licensee actions to address these problems are. discussed in Sections 3.3.5 and 3.3.6 of this report.

3.3.4 Motor-0perated Valve Program Despite an extensive program for periodically inspecting 6nd lubricating MOVs, the licensee's overall program for ensuring reliable M0V operation through good engineering and maintenance practices was inadequate. Numerous MOV failures and problems due to incorrect torque switch and limit switch (LS) settings were recurring. Although many of these problems were identified by licensee

, personnel, as evidenced by DERs, trending reports, engineering consultant findings, Fermi 2 management had not taken sufficient steps to resolve the problem. As a result, many forms of disparate MOV corrective actions were occurring concurrently. The team concluded that that the root causes for recurring MOV problems were inadequate communication between sufficiently high levels of management within the Nuclear Engineering and Nuclear Production Department, and lack of a centralized program responsibility for MOVs. The team also determined that there were several contributing causes. These were weaknesses in establishing engineering priorities for resolving MOV switch-setting problems in a timely manner; training technicians and engineers in M0V testing techniques; and consistently incorporating industry operating experience. The team referred the fo.llowup and resolution of MOV problems to Region III. -

3.3.4.i Control of MOV Torque Switches There was no central controlled document that specified safety-related M0V torque switch and LS settings. Additionally, there was no procedural guidance that required the documentation of the safety evaluation for a change of switch setting except for the 32 MOVs that were within the scope of IE Bulletin 85-03, "Motor Operated Valve Common Mode Failures During Plant Transients Due to improper Switch Settings." MOV failures were recurring because of personnel, procedural, ano Section 3.6.5.5) programmatic

. The followingweaknesses weaknesses in to setting torque switch control torque switches (see also settings were observed by the team.

(1) The M&M Department had recently established a data base of MOVs that contained, in part, a listing of the required torque switch settings for both the open and close directions of MOV travel. The data base, however, was an uncontrolled document and was inconsistent, in some cases, with torque switch settings found on some controlled documents. The table below documents three examplas in which the torque switch settings for three IE Bulletin 85-03-tested valves (as recorded in the licensee's July 25, 1988 response) were not in agreement with M&M Department records.

56

O

• q O

July 25 85-03 Response Maintenance Data Base Torque Switch Setting Torque Switch Setting Valve No. Open/ Closed Open/ Closed Actual E4150F011 2.25/2.00 2.75/2.75 2.25/2.0 E5150F013 2.25/2.25 2.0/2.0 2.25/2.25 E5150F019 Less Than 1.50/1.50 1.50/2.0 Less Than 1.50/1.50 (2) On August 20, 1988, and again on August 28, 1988, recirculation pump B discharge valve B3100F0318 failed ts close upon demand during dynamic testing of this MOV. Several :ontributing causes were associated with these two failures. One of the most significant causes, however, was that the torque switch settings were improperly set following a change out of the MOV operator for equipment qualification (EQ) several years before.

As a result of the improper torque switch settings, the replacement EQ MOV operator failed to deliver the required thrust to close the valve under dynamic flow conditions. Although the investigation was still ongoing during the diagnostic evaluation, it appeared that the lack of an approved data base of MOV torque switch settings contributed to the improper setting of the torque switch.

(3) During June 1986, the closed torque switch was increased on HPCI Turbine Exhaust Line Vacuum Breaker Isolation Valve E415F079 during a LLRT to a setting beyond the vendor's maximum recommendation. The closed torque switch setting was increased so that the valve would pass the LLRT, even though the valve had passed several earlier LLRTs at its previous setting. The MOV motor current was measured to be as much as 18 percent greater than the name plant full-load rating.

Apparently, no formal engineering review or safety evaluation was conducted prior to increasing the torque switch setting.

This practice of increasing torque settings to pass LLRTs is unacceptable because it only treats the symptoms of a problem and not the actual problem. Such symptomatic repairs of MOVs was the subject of IE Notice 82-10. "Following Up of Symptomatic Repairs to Assure Resolution of the Problem." A review of licensee documents revealed that M&M Department personnel had made a similar observation in that they felt the practice was occurring because Fermi 2 did not.have the required spare parts (i.e., valves, seats, rofurbishment tools) to refurbish valves that failed LLRTs.

In view of the recent April 1988 LLRT outage, the team determined if i this practice still existed. A review of licensee records revealed that in almost all instances, a valve was dissembled and refurbished if it failed its LLRT. The team did find, however, that the closed torque switch setting for EECW Division 2 Supply to Drywell Equipment Outboard Isolation Valve P4400F606B was 1, creased so that it would pass its LLRT.

57

(4) The HPCI SSFI in January 1988 identified that the Reactor Building Closed Cooling Water (RBCCW) Division 1 Return Isolation Valves P4400F601B and P4400F606B (discussed in Paragraph (3) above) had torque switch settings that were not consistent with the minimum and maximum values prescribed by the vendor. P4400F601B had a torque switch setting below the minimum recomended value (1 versus 1-1/4).

This MOV is an isolation valve between RBCCW and EECW and closes upon EECW initiation. When this condition was discovere.d, this valve was evaluated by Engineering to be operable in the above condition on the basis of previously successful completion of surveillance tests, the most pertinent one being 24.201.07, "Dynamic Auto Initiation of EECW, ,

Yalve Position Verification Test." In this test, the RBCCW pumps  !

continue to run while this valve starts to shut and the EECW pumps  !

' start. The above-minimum torque switch setting of 1-1/4 was >

supposedly recomended by the vendor on.the basis of the maximum  :

differential pressure the valve would be expected to experience under '

design basis conditions. These conditions should include piping -

! failures in the nonessential RBCCW system, although the team did not

review the basis for this recomendation. Therefore, while

{

P4400F6018 successfully closed during a surveillancs test, there was no evidence it experienced design-basis differential pressure (dp)  ;

during the test. It is not a requirement that this valve be in-service tested under design basis conditions, but that would be  ;

the only alternative for demonstrating that the valve was operable '

other than actually setting the torque switch to within the ,

vendor-recomended values. The abcVe torque switch setting was not changed until sometime later in early 1988 (apparently for the valve-to pass its LLRT (as was the case for P4400F606B).) to a closed torque

switch setting greater than 1-1/4 Therefore, in the absence of further justification, P4400F6018 should have been declared ,

inoperable during this interim period. '

f (5) Maintenance Procedure HI-M0002, Revision 9, "Limitorque Motor I Operator - Periodic Inspection," provides, in part, for the  ;

inspection of the torque switch compartment and switch inspection.  !

This procedure did not require the documentat. ion of the as-found 1 torque switch settings. In view of the problems the licensee hat experienced in controlling torque switch settings, the team viewed this procedural inadequacy as an additional factor contributing to this problem. The licensee indicated that they planned to revise i

, this procedure to include documentation of torque switch settings. l (6) Maintenance Procedure MIE0043, Revision 4. Motor Operator valve (MOV) j Electrical Testing," provides, in part, for the adjustment of M0V [

torque switch settings. The team reviewed Section 7.4 and found that

the torque switch setting guidance was inadequate in that the procedure did not require

(1) that any evaluation be performed or i

i documented (2) only verbalfor changing by concurrence theatorque switchSupport Maintenance settingsTechnician for B0P MOVs;(MST) '

) was required for changing the torque switch settings of ,

i safety-related (non IE Bulletin 85-03) MOVs; and (3) only those MOVs ,

! tested under the scope of IE Bulletins 85-03 required written  ;

j concurrence by an MST before changing the torque switch settings.  !

f l 58 i l _ _ . _ _ _ . - - - . -_-- _

.. }

l Discussion with licensee maintenance personnel revealed that in the past, even safety-related M0V switch settings were changed, in some i cases, solely on the basis of the decision of the electrician performing the. work. The team concluded that this practice had '

significantly contributed to the licensee's lack of confidence in MOV  !

torque switch settings. Prior to the completion of the onsite ,

period, the licensee had revised MI-E0043 to require that MST approval must be obtained before changing torque switch settings for {

all MOVs.  ;

3.3.4.2 Control of MOV Limit Switches  !

The team observed several continuing weaknesset associated with the control of '

MOV limit switches. For the most part, these weaknesses were identified by NRC  ;

Region III inspectors during previous inspectiors and during the HPCI SSFI. .

The team found, however, that licensee corrective actions had not always been '

fully implemented or, in specific cases where they had been fully implemented, these actions had not been adequate. The following examples pertain: .

(1) The MPCI SSFI in January 1988 observed that procedure MI-E0043, Revision 2, "Motor Operator Valve (MOV) Electrical Testing," did not (

t provide precise instructions on setting MOV limit switches, but rather placed too much reliance on "skill of the craft." The team agreed with this observ.ation. DER 88-0168 was specifically written to address this SSFI concern. Although this procedure was ,

subsequently revised and DER 88-0168 was closed on May 23, 1988, 3 Revision 5 of M!-E0043 still provided inadequate control of LS '

settings. For example, Step 7.5.2 stated, "Manually close valve, l then back off slightly to allow for coast of moving parts." l Revision 5 also increased the amount of torque switch bypass to a  !

value from 2 to 5 percent of disc / gate or stem travel. Although the l minimum bypass of the torque switch was increased, the team  !

questioned if this was sufficient to ensure that MOVs operate t properly. Setting the torque switch bypass too close to the fully '

closed position may result in a valve tripping on high torque before overcoming unseating forces. The industry norm for torque switch '

bypass is typically between 10 and 20 percent of valve stem travel {

and the licensee had insufficient data to justify using less. A ,

review of licensee records (DER 87-449 and work request 669468)and i discussions with licensee personnel revealed that both safety-related l and 80p MOVs had failed to open because of improperly set limit  !

switches. Region III personnel were reviewing the licensee actions .

in this regard.  !

t (2) Discussions with licensee personnel revealed that they were flesitant to increase the torque switch bypass setting because the valve position indicator switch and the torque bypass switch were wired .

anto the same position indicator rotor in many cases. Thus, the f position indication light for a valve could indicate that the valve [

was closed when it was actually open by the amount of total valve  !

stem travel that corresponds to the amount of torque switch bypass. l t

This problem was documented in NRC Information Notice 86-29, "Effects of Changing Valve Motor-Operator Switch Settings." Increasing the f.

59 L

amount of torque bypass would accentuate this problem. Since the licensee normally performs in-service testing and valve stroke timing surveillance on the basis of remote valve position indication (no one actually observes the valve locally), increasing the amount of torque ,

switch bypass to a significant percentage of valve stem travel would invalidate this method of stroke time testing. In addition, the open/close signals from some valves provide permissive signals and interlocks for other safety functions. The team found, however, that for most P4Vs there was a separate position indicator rotor available, but it was not used to prevent this problem. The team .

found no evidence that the effect of this arrangement on plant operation had been analyzed. The licensee's review of Information .

Notice 86-29 was closed by noting that it would be included in '

response to IE Bulletin 85-03. There was no evidence in the Bulletin ,

response that this effect had been analysed, t

(3) NRC Region III inspection Report 87-022 documented a concern that the licensee adopted a valve control logic that deliberately backseats globe and gate valves using the torque switch. In an October 3,  !

1907, letter to the NRC, the licensee committed to rewire i safety-related MOVs to use LS on open stroke to prevent backseating. [

Many B0P MOVs were still configured to trip on torque at the completion of the open stroke. These BOP MOVs may fail as a result  ;

of this practice because the valve may become stuck on its backseat. '

In addition, the procedure for LS setting Mi-E0043 step 7.5.3.5 does not provide precise enough irstructions to ensure that the open LS is ,

set with enough stem travel left to prevent backseating during L coasting after limit switch actuation. The procedure directs the technician to "manually open the valve, then back off slightly (less i than 5 percent) to allow for coast of moving parts." The amount of coast required is a function of several parameters, including valve i stem speed and the amount to back off should be precisely set.

3.3.4.3 MOV Maintenance Procedure Weaknesses In addition to the procedural weaknesses associated with the setting and control of MOV torque and limit switches, that are discussed respectively in Sections 3.3.4.1 and 3.3.4.2, the team found two other MOV procedural i weaknesses.

(1) Maintenance Procedure MI-M0002, Revision 9, "Limitorque Motor i Operator - Periodic Inspection," provides in part, for the periodic inspection of MOV motors. The procedure, however, provides no l requirement to periodically inspect, clean, and replace as necessary  !

the brushes for the motors of direct current (de) powered MOVs, even though such guidance is reconnended by Limitorque. The licensee has i approximately 36 de powered safety-related MOVs. A subsequent review of licensee records revealed that the HPCI SSFI documented the same  ;

weakness in January 1988. Discussions with M&M Department personnel  !

revealed that the licensee pisnned to revise M1-M0002 to include this '

guidance before the end of 1988.

(2) M!-E0043, Revision 4 "Motor Operator Valve (MOV) Electrical Testing," provides, in part, for the post-maintenance tecting of MOVs 60 )

I

I b

following MOV repairs or switch adjustments. The team reviewed the post-maintenance testing requirements (Sections 7.7 and 7.8) and found them inadequate for providing assurance that an MOV will operate upon demand following MOV maintenance. The procedures provides additional assurance of post-maintenance operability for valves tested under IE Bulletin 85-03. The procedure requires that for those 32 MOVs that were tested under IE Bulletin 85-03, a Maintenance Support Technician (MST) must review the work performed and the post-maintenance testing completed prior to closecut of the work package. The team found two examples of inadequate post-maintenance testing:

a. The team oWerved that inadequate post-maintenance testing requirements contributed to the initial failure of the licensee to identify the root cause of the August 20, 1988, failure of Recirculation Pump B Discharge Valve B3100F031B to close upon demand. Following repair > to 83100F031B, the MOV was only tested under static flow conditions without the use of diagnostic electronic signature analyses which would have revealed that the MOV torque switch was set too Icw to allow the valve to stroke under dynamic flow conditions,

b. A review of licensee records (DER 88-0102) revealed an example where the valve packing was adjusted to the RCIC Turbine Steam Stop Valve E5150F045, but not all the post-maintenance testing ,

requirements of MI-E0043 were performed because plant operators '

desired to restore RCIC to operable status as soon as possible

-following completion of the packing adjustment.

3.3.4.4 Response to IE Bulletin 85-03 The licensee's July 25, 1988, response to IE Bulletin 85-03 had incorrectly reported some information and had failed to followup on the generic aspects of problems found. The team had the following two observations.

(1) The testing in response to IE Bulletin 85-03 included electronic signature analyses for MOV parameters, such as stem thrust, motor current, and stem travel while loaded with simulated differential pressure (load cell) or with a differential pressure across the valve ,

disc. Thirty-two valves were tested in the HPCI and RCIC systems.

Four valves were found to be overthrusting; three of these were more than 120 percent over design values. This overthrusting condition 1 indicated potentially serious operational problems with approximately 10 percent of the safety-related valves tested. This type of testing was not promptly extended to the other safety-related MOVs to determine if similar problems may exist. A review of licensee records indicated one instance in which an MOV operator overthrusting condition resulted in damage to residual heat removal (RHR) MOV E1150F0078.

(2) The July 25, 1988, response implies that for RCIC Pump Suction from .

Suppression Pool First Isolation Valve E5150F031, there is a limiter  !

plate that prevents the torque switch setting to be increased above a value of 4.00. Discussions with licensee personnel revealed, 61  !

. l however, that there was no limiter plate installed for E5150F031, and that the maximum setting of 4.00 was an administrative limit. The i team found that there were five other safety-related MOVs that did not have torque switch limiter plates, including HPCI system MOV E4150F079, that is discussed in Section 3.3.4.1. These MOVs could be

, damaged if the torque switch settings were set beyond the value that the limiter plate is intended to restrict. Discussions with licensee personnel revealed that a potential design change (PDC) had been issued in order to install the torque switch limiter plates for the affected MOVs.

3.3.5 Planning and Scheduling Ineffective M&M Department planning and scheduling adversely affected the implementation of the PM program, resulting in numerous missed PM tasks; contributed to increased time for completion of CM work; and adversely affected the availability and reliability of plant systems. Problems that contributed to poor performance of maintenance planning and scheduling are detailed below:

(1) Poor performance of the planning and scheduling group contributed to a number of scheduling oversights and errors resulting in PM tasks not being completed within their specified frequencies.

DERs 88-0242, 88-0243, 88-0247, and 88-0248 document examples involving scheduling oversights / errors.

(2) PM-scheduled tasks did not make allowances for potential rescheduling of work due to plant / equipment conditions that may affect completion of the work. For example, DER 88-0245 documents a case where PM tasks could not be completed because the reactor building crane was inocerable. The repair of the crane took longer than planned and resulted in a schedule slippage.

(3) Poor planning resulted in work packages sent to the field without prior verification that all necessary job prerequisites were satisfied.

(a) DER 88-0260 documented a case where an instrument could not be calibrated because it was not installed. The engineering design package EDP-3644-2 that specified installation of the instrument had not been implemented.

Verification of completion of the EDP was not performed prior to scheduling the work.

(b) DER 88-0040 documented a case where replacement of-underwater lights for refueling could not be performed because replocement spare parts were not a stock item. The work package was sent to the field without prior verification that the spare parts were available.

(4) Poor communication and coordination between operations and maintenance resulted at times in the delay and/or refusal of work packag.s to be released for implementation. For example, DER 88-0132 documents a delay of 14 Agastat relay changeouts because of plant conditions, conflicting surveillances, and apprehension of 62

1 4

operations. Poor coordination and communication among the work

groups involved contributed to a misunderstanding of the work scope.

Discussions with licensee personnel indicated that due to

insufficient lead time for operations to review work packages, lack  :

1 of feedback from maintenance addressing Operations questions and  ;

i concerns, and poor coordination of PM, CM and surveillance  :

activities, Operau-s was apprehensive, in some cases, about l dpproving work packages for implementation. i i (5) The supervisors in the mechanical and electrical group did not have i

adequate time to plan work because work packages were often provided l 4

to the first line supervisors 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> before they were scheduled for l

implementation.

l

- (6) Lack of involvement of first line supervision during work contributed [

to delays in completion of routine maintenance activities. Foremen, j

! who were responsible for outaining necessary spare parts, were l

scending an excessive amount of their time tracking down spare parts '

because of availability problems. i i 1

]' In April 1988, the maintenance planning and scheduling function was relocated  !

from the Planning and Scheduling Group to the M&M Department. The maintenance l

planning and scheduling group had a good mix of technical qualifications  !

i including planning and scheduling expertise, two SR0s, and maintenance  ;

experience. This group maintains a 48-week, long-range schedule of surveillances and PMs; a 30-day "look ahead" schedule, which includes PMs, i surveillances, and modifications; a 7-day schedule, which was a refinement of  ;

i the 30-day schedule and incorporates scheduled CM actions, and the 72-hour work  ;

j schedule. Forced outage planning and scheduling also was conducted by the t j genup. *

, j t

i Licensee efforts toeffect.

having a positive improveThese scheduling effortsand planning p(erformance included: 1) conductingappeared daily to be 1 meetings between maintenance groups and purchasing, and between maintenance and l

cperations to review issues with work upcoming) packages, maintenance (2 providing activities operations and resolve work packages delays and in advance i and holdino personnel accountable to be knowledgeable of the work scope in work packages, (3) reassigning of the functions of work package preparation from a single group to each individual maintenance group, and (4) reassigning MSTs to

! the maintenance groups to support this function and assist the foreman.

9

! Discussions with licensee personnel indicated that an improvement in the

{ coordination of maintenance and surveillance activities was noted by Operations j and that efforts made to improve coordination of maintenance activities with Operations had produced improved working relationships. There were no priority "A" PM tasks that were overdue, which indicated an apparent improvement in the j scheduling of PM activities. Licensee changes to improve the work package i preparation process should result in foremen receiving better quality work j packages and far enough in advance of schedule due dates for review and j preparation of the work activity. In addition, the thirty-day look-ahead

schedule was a significant imprnvement that allowed foremen, supervisors, and work package preparers time to cetter plan their time and resources. These

improvements, however, may be slowed due to the following additiona! weaknesses observed by the tean

i 1

i

63

l (1) Although approximately 35 percent of the available M&M Department manpower was designated for contingencies, non-M&M Department schedule requirements, such as DER review and resolution, emergency  !

i plant support and forced outage activity inordinately disrupted the schedule of activities and, in particular, precluded the implementation of scheduled CM.

(2) Several staff of the planning and scheduling group did not have an adequate knowledge of plant systems, thus limiting their ability to  !

estinate job manpower and schedule requirements.  !

3 (3) The supervisor of planning and scheduling lacked knowledge of how [

planning and scheduling is done at other plants, thus limiting the {

l ability of Fermi to emulate proven practices without a  ;

trial-and-error period.  !

f (4) T.1e PM work packages lacked sufficient work scope, thus requiring I

that operators and maintenance personnel jointly spend time determining the needed tagging requirerents (see Section 3.2.4.1).

I >

i  :

1 Improvements were still needed ir W coordination of PM with CM and i surveillance schedules. The plaunis.? and scheduling group had not fully

optimized the PM schedules with 1.e su/veillance schedules. A significant 6

number of PM tasks were due by '.ne end of the month of August and most of the  ;

surveillances due at the end of August on the corresponding equipment had  ;

already been performed.

l I

3.3.6 Spare part Availability [

4 l i Lack of availability of required spare parts had resulted in reduced [

i reliability and availa'sility of systems. The availability of spare parts had i j become an increasing problem at Fermi and had a significant impact on the "

performance of the M&M Department. Due to the protracted construction period, i j a lot of the mechanical and !&C equipment at Fermi is of an older vintage than r

, normal for a plant in a startup test program. Some of the vendors are now out i of the nuclear business or out of business altogether. Problems that  ;

j contributed to the lack of availability of spare parts included:  ;

l (1) Deficiencies within the Spare Parts Reference System (SPRS), which is f I

used to determine required spare parts for components, had resulted

in work packages sent out into the field without the necessary parts

! required to perform the job. Fer sample, OER 88-0028 documents a  !

j case where maintenance on the ACIC and core spray north equipment cooling unit enuld not be fully completed. {

! It was found tha.t the [

] SPRS indicated that the component required two matched belts, when it t

required three. DER 88 0246 also documents a case where PM could not [

be completed because the filters listed in the vendor manual were l

) different from the filter specified for the component in the SPRS. l l

i (2) Poor warehouse handling of safety-related material resulted in materials identified as available in the warehouse, when in reality

they were not. Warehouse practice at the time of receipt of j safety-related material was to enter into the Material Management ,

i <

I 64 [

.

• i i

System (MMS) the material received as stock quantity available. This information was automatically transferred aver to the SpRS. However,

, in reality, the meterial was not available for use until it was  ;

accepted by quality control (QC) and released for stocking. This i p*actice had resulted in cases where work packages were sent into the  ;

field after verification of spare part availability through the SPRS even through the material had not been accepted by QC and was put on QC hold.

L (3) Lack of planning of material requirements which resulted in delays of  ;

CM activities and of PM tasks not being completed within their  ;

respective frequencies. For example, DER 88-0411 documented a case where changeout of Agastat relays for the radwaste sump could not be completed as scheduled because parts were not available. It was  !

found that although parts availability was checked and replacement  !

parts were ordered, the PM schedule duc date was missed because the  ;

replacement parts could not be obtained in time. DER 88-1269 '

documents another case where known needed replacement parts should [

have been available for repairs. Filters nieded for the turbine . i building Jd radwaste heating, ventilation, and air conditioning ,

(HVAC) systems which were found to have a high differential pressure  !

and needed replacement; however, no replacement filters were  !

available. J (4) A working program did not exist for the identification and possible '

replacement of equipment for which spare parts are not comercially i available. l The team found that numerous work packages in the 7-day and 3-day work schedule  !

were deferred due to lack of spare parts. For example, on January 11, 1988, *

('

the licensee discovered that 2 of 28 electrical box cover holts en the division B RHR pump had sheared off. However, because of problems in obtaining the necessary spare parts from the equipment vendor, work was put on hold. At the [

time of the diagnostic evalution, replacement parts still Nd not been obtained by the licensee. As of August 29, 1988, there were approximately 44  !

work package deferrals due to lack of spare parts. In addition, the team found [

that because of insufficient spare parts due to poor outage planning, the i licensee, in some cases, increased torque switch settings for POVs in the 1986 and 1988 LLRT outages to pass local leak rate tests. l Durin the June 1988 reor anization, a new group was created, consolidating i Mater $alEngineering,NucearProcurementandNuclearMaterials,calledthe  !

NuclearMaterialManagement(NMM) group. The NMM group will t.ssume the i responsibility of developing and implementing an improvement program to address i the obsolete equipment problems. The NMM group required an additional 15 engineers / technicians to implement the program, which at the time of the  ;

diagnostic evaluation had not been approved. Once the program is established, '

staffed, and implemented, the licensee estimated that it would take i approximately two years to complete. '

The licensee had initiatcd a number of other corrective actions to improve  !

spare parts availability, such as preplanting the necessary material  !

requirements needed to perform PP scheduled for the coming months to ensure {

material availability, improving the SpRS to i:nsure an accurate list. of all l

I 65 ,

parts for each component exists, and reviewing warehouse inventories. The team

, cbserved that many of the corrective actions were either just recently implemented or were still ongoing and; therefore, the effectiveness of these actions could not be fully evaluated. In addition, the licensee had engaged a consulting firm to evaluate the broad area of material control prior to the start of the diagnostic evaluation.

3.3.7 Equipment Failure Trending and Root Cause Analysis Although the M&M Department had a program in place to track equipment failures, the team found that the effectiveness of the program was limited primarily due to a lack of technical resources to support tie program. The team reviewed the

, methods for tracking of equipment, as specified in Management Directive FHD MA1, Revision 0, entitled "Maintenance." Trending and failure analysis of equipment were being performed both in the Technical Engineering group (see .

Section 3.4.4) and in the M&M Department. Within the M&M Department, the Nuclear Plant Reliability Data System (NPRDS) Coordinator was responsole for compiling f ailure data, analyzing failure data for repetitive equipment failures or negative trends, and providing periodic reports on the results of analyses. On the basis of a review of various documents and licensee interviews, the team had the following observations:

(1) A review of trend reports indicated that the results of analyses of equipment failure data appeared to be based primarily on a statistical compilation of the data rather than on a detailed analysis of the failure data. The team found that because only one individual was assigned this function and because of monthly reporting periods, insufficient time was available to allow a more rigorous analysis of the data. Also, no maintenance personnel attended the recent Human Performano Evaluation System (HPES) training course which provided trainii., in root cause analysis.

(2) A review of data contained in the March snd June 1988 trend reports indicated that the licensee was experiencing failures of MOVs. The team found that ,nany of the MOV failures that had occurred appeared to be due to a lack of PM and problems in setting proper torque and limit switches. Weaknesses in the root c,.use evaluation and trending of equipment failures for MOVs contributed to a lack of effective corrective acticns to prevent recurrence.

(?) Maintenanc6 personntl and system engineers did not always make effective use of the equipment failure history. Administrative Procedure POM 12.000.015, Revision 31 entitled "Work Request,"

specified as part of the work package preparation that maintenance personnel perform a review of available equipment history to determine if a recurring failure is indicated, and if so, consider the need for a PDC or DER prior to formulating required rework or repair. It was found that the work request tracking system was not an effective diagnostir tool, and as a result, personnel were forced to use alternative methods, such as St manual equipment history cards or NPRDS, which were not easily accessed by personnel. In addition, a sample review of work packages indicated a weakness in documenting the cause of equipment failure. Lack of documentation of root cause of equipment failures would diminish the effectiveness of 66

[. -

5 ,'

equipment history files as a diagnostic tool. To improve performance in identifying repetitive equipment failures, increased management attention had been placed on emphasizing the use of equipment history files. The licensee had also revised the DER process to lower the threshold of reportability to promote identification of problems adverse to quality.

The team concir hd that because of a lack of maintenance engineers and plant system enginee.4 in the Technical Engineering grc q , insufficient management oversight, and a lack of personnel training in failure and root cause analyses, the equipment failure trending prograr was not effectively implemented.

3.4 Testing The team reviewed the 1.icensee's testing program; and their implementation, with emphasis on testing required by Technicei Specifications (TS). Testing done in accordance with Section XI of the ASME Boiler and Pressure Vessel Code was required by TS 4.0.5, and was also evaluated by the team. Testing procedures were reviewed for adequacy and completed tests were reH ewed, including follow-up corrective action. The team witness the cc : auct of surveillances on an EECW pump and on an EDG. Testing of cafet elued and BOP systems were included in the evaluatier., with special empnas . isacr.d on testing for the EECW and RCIC systems.

The team observed a number of strengths and good practices in TS-required testing, as well as some programmatic and technical weaknesses. Testing needed to satisfy requirements other than those found in TS and Sectirn XI wac determined to have significant deficiencies. e 3.4.1 Technical Specification Surveillance 1: sting 3.4.1.1 Administrative Control of Surveillance Testing Procedures Surveillance tests intended to meet TS requirements were performed by a wide variety of groups on site, including Operations, Maintenance (including I&C),

Radiation Protection, Reactor Engineering, and Chemistry. Reeponsibility for the technical accuracy, completeness, and clarity of surveillance test procedures was similarly distributed widely across the various Nuclear Production work groups. A "Responsible Section Head" was designateri for each surveillance test procedure to ensure each procedure *eceived a comprehensive review at least every two yetrs. The Procedures Coonaination Group (under the Business Planner) tracked the review process and issued a notice to alert the  !

Responsible Section Head of the need to perform the biannual revitw. If the review was not completed by the two yer oint, the procedure was addad to a monthly overdue list that was distributed to each Responsible Section Head and the Plant Manager. Programmatically, this approach to controlling surveillance test procedures appeared to be adequate. A review of the last six monthly overdue lists showed that an average of about 150 procedures were carried in an overdue status each month. The licensee planned to reduce the number of overdue procedures to zero by the end of 1988 by conducting iannual reviews on all procedures concurrent with u human factors procedure upgrade effort that was expected to be completed by that time.

! Additional comprehensive reviews of all TS surveillance test procedure:. were

being performed through the licensee's Technical Specification Improvement t

67

. , t l

. 1 Program (TSIP), and this was viewed by the team as a strength provided the l results were fully implemented. The TSIP was begun in October 1987 to address i a long history of incorrect TS interpretations, incomplete or missed  :

surveillances, poorly written tests that did not test the necessary components, and failure of personnel to follow procedures. The results of the team's -

evaluation of the TSIP surveillance procedure review process are discussed i below. Additional evaluation of the provisions of the TSIP is found in Section  :

3.2.7.3. l 1

Under the TSIP, an extensive review of each TS requirement, the system and r equipment operating characteristics, and the design basta was performed.

4 Procedure language was assessed for clarity, and a determination was made ,

whether the surveillance test adequately demonstrated operability. A file containing related design calculations, UFSAR excerpts, drawings, and other <

pertinent information was assembled to document how each TS surveillance  ;

requirement was satisfied. Where appropriate, recommendations for TS changes  ;

l or procedure modifications w6re made. In addition, TSIP personnel were '

developing a computerized cross reference between each TS requirement,  !

associated surveillance procedures, and components tested,  !

t Some weaknesses in the TSIP were identified, however. Program effectiveness  ;

may be diminished because the section heads responsible for procedure  !

maintenance were not required to implement the procedure modifications i recomended by TSIP personnel. The program relied heavily on the persuasive '

skills of program mansgers to implement those modifications. Therefore, the i extent to which the results of this review will be reflected in revised l procedures is unclear. I The review phase of TSIP was expected to be completed on schedule in December 1988, after which the TSIP Group will be disbanded. Discussions with the i licensee revealed that few plans had been made for maintaining the knowledge base developed by TSIP and documented in the TS requirement f les after disbanding of the TSIP Group. It had not yet been decidei which group or -

groups would be gihn custody of the files developed by T,;P, nor what their [

responsibilities for maintaining the files would be. The licensee agreed that maintaining this knowledge base on an ongoing basis in response to plant t modifications, revisions to the TS or UFSAR, and regulatory changes would be  ;

beneficial in assuring the ability to oemonstrate TS compliance, but was f reluctant to commit to this course of action due to resource constraints. The i licensee did plan to upgrade the procedure review guidelines used for the  !

biannual procedure review to bring these reviews more into line with the  ;

standards of TSIP. ,

3.4.1.2 Surveillance Schedulino and Tracking Program ,

J Responsibility for day-to-day administration of the TS surveillance test  !

program lay with the Surveillance Scheduling and Tracking (SST) Group. The SS: t Group used a computerized program with manual backup to ensure that  !

surveillance tests were performed within the time limits specified by TS, F Because their efforts most directly supported the Operations Department, the  !

SST Group was transferred from the Technical Engireering Group to the [

Operations Department during the period the team was on site, t I

68 l

The SST program and its implementation sere viewed as a strength by the team.

Conservatism was built into the SST program to aid in assuring the timely completion of TS-required surveillance tests, and effective methods were used to notify the appropriate personnel of upcoming surveillances. A mode change list was preptred when necessary to identify all surveillances needed prior to changing modes, and SST personnel bricfed the NSS each working day to ensure he was aware of surveillances nearing the end of their grace period.

The effectiveness of the program can be seen in the relatively call amount of grace period used. The maximum grace period permitted by TS 4.0.2 was a 25 percent extension beyond the required surveillence interval. SST was used

~

to schedule and track 1083 individual surveillance items, and in the period of June 1, 1988 to September (some items came dut more 30, than1988,)a once . total of 1621 Of these, items a total required of only performance 64 items used 24 hour2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> s or more grace time, and only two were not completed. before their grace time expired.

The SST Group staffing level of three full time personnel was marginally adequate for performing day-to-day SST activities and had sometimes resulted in delays in routing completed tests for review (see Section 3.4.1.3.2).

One improvement planned for the SST progran was the development of a computerized related sork items cross reference. Each surveillance task in the database was to be flagged to identify related work items as an aid in

~

scheduling related surveillance, PM, and other work items together to minimize equipment cown time. The licensee was us'ng a manual method to schedule 4

related work items together until the p' .ess could be automated. Under the manual method, the SST Group brought.f( vard surveillances where possible to correspond to scheduled PM. Related corrective maintenance items were folded into the schedule at meetings between the SST Group and other groues responsible for performance of surveillance, PM, and corrective maintenance.

Although this method of attempting to schedule related work together appeared to be programmatically sound, licensee personnel stated that in practice related work items were still not being well coordinated, primarily due to weaknesses in pM scheduling (see Section 3.3.5).

3.4.1.3 ASME Section XI Testing 3.4.1.3.1 ASME Section XI Testing Commitments i

Fermi was committed to perform inservice testing (IST) of ASME Code categories A, B, C, and 0 valves, ASME Code class 1, 2, and 3 pumps (and some non-ASME Code classed pumps) in accordance with Section XI, Subsections IWV and IWP, of the ASME Boiler and Pressure Vessel Code, 1980 edition, through the Winter 1980 Addenda. The Fermi IST program had received NRC approval. Where it was impractical to meet Code requirements, relief was granted by NRC.

3.4.1.3.2 Administrative f.ontrol of ASME Section XI Inservice Testing Responsibility for the IST program lay with the Inservice Inspection (!SI)

Group in the .echnical Engineering Group. Requirerants for program administration were well defined and in general well executed. Formal procedures we"e in place to specify the purpose and scope of the progrcn; to designate the personnel responsible for program development, maintenance, and 69

implementatier.; and to specify program requirements with regard to test results analysis, corrective action, and record keeping. Although not formally called for in program procedures, the team found that self m dits of the program were done y the ISI Group, resulting in improvements to the program and correction I of program deficiencies. The provisions of the program generally represented a '

conservative approach to fulfilling ASME Section XI requirements. Scheduling and tracking of tests required by the ISY program were done through the SST program discussed in Section 3.4.1.2. On the whole,. administrative contrcl of the IST program was viewed as a strenilth by the team, but come weaknesses were found. ,

The team found the documents governing the I$T program to be needlessly l cumberscoe. IST program administrative control requirements and guidelines i were dir.tributed among three different administrative procedures: 4E 5.6, t "Inservice Inspection Programs;" POM 12.000.61, "Inservice Testing Program for l Pumps and Valves;" and POM 41.000.22, "Inservice Testing Program for Pumps and .

Valves--Implementation and Control." Many of the provisions found in one procedure were duplicated in one or both of the others.

I Nonconservative approaches permitted by Section XI, but that were no longer  :

being followed by the licensee, were found in the administrative procedures, t For example, all three administrative procedures permitted placing a pump in  ;

service (i.e., declaring it operable) under circumstances that call for i establishing new. testing reference values for the pmp, provided the new l reference values were obtained within 96 hours0.00111 days <br />0.0267 hours <br />1.587302e-4 weeks <br />3.6528e-5 months <br />. All three administrative procedures also permitted a valve to exceed its maximum stroke time and not be I declared inoperable until up to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> later. l The licensee indicated to the team that the IST administrative be consolidated into a single Fermi Interfacing Procedure (FIP) procedures and the would t conconservative provisions would be removed by the end of 1988. However, some ,

i licensee personnel expressed concern that the guidance provided by upper  !

management for carrying out this consolidation effort was overly restrictive. ,

! In particular, program description information such as methodologies to be used i for ISI testing, calculating ISI test acceptance criteria, and establishing .

pump reference values was not to be included in the FIPs. but could be included t

in an "overview document" if the group responsible for the program chose to do so. It was the team's view that a detailed program description is essential to maintaining program integrity and ensuring that all requirements associated *

, with a program are fulfilled. As such, the team concluded that it would be j advantageous for the licensee to require that formal program descriptions be maintained in a visible location, whether in the FIPs or elsewhere, rather than leaving this as a discretionary item.

The process by which completed Section XI tests were reviewed was found to have problems.Section XI requires that testing frequency be increased when testing  ;

indicates degradation of component performance. For valves this means that

quarterly testing must be changed to monthly testing when stroke time increases .I l by 25 percent or more on successive tests. As reported in LER 88-031, the i stroke time for HPCI system valve F067, while less than the maximum allowed. [

experienced an increase of 36 percent between the tests of May 8, 1988 and June i

10, 1988. This was not discovered until August 15, 1988, well after the valve .

! should have received its first increased frequency stroke time test. The valve -

{ was subsequently tested on August 16, 1988 and its stroke time had returned to l a value close to that determined in the May 8 test. The valve was also placed

}

70

i on the monthly "IST Pump and Valve Out-of-Spec Status Report" dated August 18,  !

1988 to request that the SST Group increase its scheduling frequency. The repnet erroneously specified an increased frequency of 46 days for the valve,

~

which would have been appropriate' for a pump with degraded performance. SST '

e' personnel recognized the error, however, and put valve F067 on the proper monthly test frecuency. .

t Three factors contributed to the licensee not discovering problems like the  !

'F067 valve oroblem described above in a timely manner. First, changes in valve i stroke time were not calculated at the time testing tvas completed. When the

'NSS ceviewed a completed valve stroke testi he verified only that a maximun  !

stroks time was not exceeded. Second, the marginal staffing level of the SST  :

Group sometimes caused delays in routing the completed test from Operations to '

- the 15] Group for review. Third, the marginal staffing level of the ISI Group l' . caused deleys in analyzing the completed test data. The corrective action  !

taken by the licensee was to publish the IST Pump and Valve Out-of-Spec Status Report biweekly instead of monthly, and to change IST administrative procedures i to require ISI Group review of tests within seven working o'ays of their i completion. In addition, during the team's onsite period, the Technical Engineering Group made a request to the Plant Manager for another full time  ;

employee to support the IST program. The licensee acknowledged the marginal staffing in the SST Group but had no immediate plans to increase their numbers.

• 3.4.1.3.3 ASME Section XI Pump Testing ASME Section XI pump testing at Fermi performed unoer the IST program met or -

exceeded the requirements of the ASME Code. Particular strengths observed by  :

the team are discussed below.

A formal procedure was in place to govern the establishment of new pump

. reference values. This procedure employed a linear regression technique to develop a best fit curve for reference value data points. Lines were then i drawn to designate acceptable pump performance, and the alert and required action ranges. The alert and required action lines were drawn somewhat more [

conservatively than Section XI would require, to ensure that marginal performance was properly evaluated. ,

r i The number of pump vibration readings taken exceeded the requirements of Section XI. Personnel were required to undergo a formal qualification process

! cefore being placed on the licensee's list of qualified vibration technicians, '

and only these personnel were permitted to do vibration testing. Logs were j maintained to identify the tests a given vibration analyzer was used fcr, and l to identify the technician who performed each test. Although not formally -

required by IST program procedures, pump performance data, ine.luding vibration results, were trended by the til Group.  ;

Conduct of pump testing was evaluated and is discussed in Section 3.4.1.4.

l 3.4.1.3.4 ASME Section XI Valve Testing i 1  :

The team reviewed the most recent revision of the Fermi 2 valve IST program. -

program records and completed valve tests were included in thi', review, as well i as interviews with operators responsible for performing valve testing. [

Notwithstanding the failure to perform timely reviews of tes*. results discussed in Section 3.4.1.3.2, licensee records for trending motor operated valve stroke l

[

71 l

____ _ i

times were found to be well kept, and were considered a strength. Valve stroke times were reviewed to detect slowly increasing trends, although this was not formally required by program procedures. Along with these strengths, the team identified several notable deficiencies.

Procedure p0M 24.207.04, "RBCCW/EECW Miscellaneous Valves Test," stated in its precautions and limitations section "The surveillance test should be terminated promptly upon indication of excessive valve vibration, jerky valve motion, binding, unusual noise, overheating or other erratic operation." Discussions with licensee personnel revealed, however, that valve testing consisted of timing valve strokes from the control room, and no one was routinely stationed at valves under test to make the kinds of observations referred to in the above statement. 2S1 personnel indicated to the team that the valve testing program rould be revised to require direct observation of valve performance at least once a year for accessible valves. Valves in. areas not readily accessible during operation were to be observed when plant conditions permitted.

The ISI Group used a formal methodology for determining or changing IST program maximum allowed valve stroke times. This method derived maximum valve stroke times based on 135 percent of a historical average of the measured stroke times for each valve, and was intended to identify degradation of valve performance that warranted corrective action. The team found that this methodology did not include comparison of newly determined IST program maximum stroke times with the master valve list (MVL) maximums determined by Nuclear Engineering design calculations. For example, the maximum allowed stroke time for EECW valve F601A was unilaterally revised in August 1988 by the ISI Group from 50 seconds to 66 seconds because its measured stroke times were consistently very close to 50 seconds. Nuclear Engineering design calculation DC-2712, rev. D gave the maximum stroke time for this valve as 50 seconds, and gave the general guidance that deviations between the MVL and field tested values were to be addressed to Nucl6ar Engineering for resolution. The licensee planned to resolve these problems by requesting a revision of the MVL maximum value for valve F601A and revising the work instruction governing IST maximum valve stroke time determinations to reouire verification of new values against the MVL.

The Fermi valve IST program had no provisions for tracking check valve test failures. For example, EECW check valves F182 and F111A were tested on March 19, 1987 and July 27, 1988, respectively. In each case, the check valve failed its seating test on the first attempt, but passed on a second attempt with no documented corrective action between attempts. These failures were documented on the cover sheet for each test, and therefore could have been noted by ISI personnel during their review of the completed tests. However, there war no mechanism in the valve IST program to keep track of such failures. Tracking of check valve failures would bring to light repetitive problems and serve as a basis for further investigation a,d/or corrective action. The licensee indicated that work was underway to computerize many aspects of the IST program, and this would be one of the items that would be followed with the computer.

3.4.1.4 conduct of Testing The team observed the performance of two surveillance tests. The first was the quarterly pump test for EECW Division I, and the second was the monthly EDG *11 start and load test. In each case, a thorough prebrief was conducted in the control room to discuss test objectives, prerequisites, and precautions. An 72

._j

4 overview of how the tests were expected to proceed was discussed, including which specific persons would be responsible for carrying out key parts of the procedure. The operators performing the tests were generally knowledgeable and followed the procedures properly. There was good coordination between personnel in the control room and the operators running the test at the equipment location. For the pump test, health physics coverage was necessary, and the team observed good coordination between testing and health physics personnel. Test conduct was judged to be adequate overall, but some weaknesses were identified by the team.

Deficiencies were noted in the verification of instrument calibration data.

The EDG test procedure, POM 24.307.14, did not call for verification of calibration data. Further investigation revealed that the EDG local control panel meters received no periodic calibration or PM, and were last calibrated in 1984 This included the EDG kilowatt meters used to verify compliance with TS. The licensee indicated that one product of TSIP was to be a list of instruments used in verifying TS compliance, and that this list would be used by 1&C personnel to ensure all necessary instruments were placed in a periodic calibration program. For the near term, the licensee planned to seek assistance from an offsite corporate calibration team to calibrate the deficient EDG kilowatt meters, and to create 'A' priority PM events to calibrate all of the meters periodically.

An example of inattention to detail was observed during the EDG test. Step 10.11.5 of Attachment I to the procedure called for adjustment of the EDG output voltage to between 4160 and 4300 volts. Test personnel mistakenly thought 'that each voltmeter division represented 200 volts, when in fact each represented 100 volts. and the voltage was inadvertently set at about 4120 volts. After prompting by a diagnostic team member, the voltage was adjusted 4 to within the requircd range.

Procedure POM 24.207.08, which governed the EECW pump test, called for verification that the system gauges to be used in the test were within calibration. The methoo used by test personnel to obtain this data was inefficient. The use of calibration stickers on plant process instrumer.tation was discontinued in the spring of 1988, so it was necessary to verify .

calibration data with the !&C shop. This was not done before entering the plant, and a 10 to 15 minute delay was experienced while the control room obtained the data from the !&C shop and relayed it to the test lead person at the pump, 3.4.2 Non-Technical Specification Testing 3.4.2.1 Performance Scheduling and Tracking Program The licensee maintained a computerized Performance Scheduling and Tracking (PST) program to schedule and track the performance of periodic testing not required by TS. The PST program was run by the SST Group within the Operations Department. The testing managed by this p-ogram satisfied items such as 80P fire insurance requirerrents, UFSAR commitments, NRC commitments, special requests from Nuclear Engineering or other work groups, or as a result of the licensee's Performance Evaluation Program (see Section 3.4.3). The structure of the PST program was very similar to that of the SST program (see Section 3.4.1.2), and in general it appeared to have the features necessary to adequately control, schedule, and track testing 'that had been determined to be 73

important to safety and/or was intended to fulfill Fermi commitments, but not required by TS.

One programmatic weakness was noted. The PST program contained no provision for identifying the person or group that should review the completed test. The memory of SST personnel was relied on for successful routing of the completed test to the correct reviewer. For example, a post modification test used to check RCIC turbine peak RPM during the system starting transient (see Section 3.4.2.2) was scheduled through PST. After completion, the test was routed to a Technical Engineering Group Systems Engineer, who subsequently determined the correct reviewer to be an engineer in Nuclear Engineering.

One weakness in FST program implementation was noted by the team. Although the program was capable of proddCing a print out Of past due PST items, such a print out was not often made nor disseminated. This left open the possibility that NRC commitments could go unmet and/or UFSAR requirements could go unverified for extended periods. ,

3.4.2.2 RCIC System Testino Testing activities pertaining to the Reactor Core Isolation Cooling (RCIC)

System were evaluated by the team. Although the implementation of RCIC testing required by TS appeared to be adequate, it was found that specific comitments of the UFSAR were not being verified by testing, and that testing dictated by good engineering practice was not being done in many cases.

No credit was taken for RCIC system operation in the Fermi accident analyses.

Therefore, the system, with NRC concurrence, was not designsted or maintained as safety-grade. The system was nevertheless important to safety, as reflected by its treatment in TS. In the event that RCIC becomes inoperable, TS 3.7.4 requires a unit shutdown be commenced after 14 days, and'if the HPCI system is also inoperable, TS 3.0.3 requires a unit shutdown be commenced within one hour.

Many licensees thoroughly test the RCIC system in their ASME Section XI IST program. The only RCIC system components included in the Fermi IST program, however, were one relief valve, pressure boundary valves, and valves with a specific safety function (i.e., primary containment isolation valves). While the requirements of Section XI are limited to components with a specific safety function, it was the team's view that additional testing was needed to fulfill UFSAR comitments and to assure system reliability by detecting and correcting gradual degradation of system components.

Fermi UFSAR Section 5.5.6.2.2.4, "Valve Operation Requirements," specifies the performance requirements for several RCIC system valves. Of particular interest were turbine steam supply valve F045, pump discharge valves F012 and F013, and pump minimum flow bypass valve F019. The UFSAR commits to each of these valves meeting specified stroke times in both the open and close directions, against specified differential pressures. The differential pressure requirements were not tested for any of these valves, F019 received no stroke time testing in the open direction, and F045 received no close direction stroke time testing. Procedure POM 24.206.05 called for open stroke time testing of F012, F013, and F045 at the time of the diagnostic evaluation, but the licensee planned to delete these tests at the next procedure revision because the open stroke time testing was not called for by the IST program, 74

whose provisions the procedure was intended to fulfill. Valve F045, which must stroke open to admit steam to the RCIC turbine, would not be rectiving open stroke time testing after this revision. While this may not violate IST program requirements since RCIC was not a safety-related system per se, the team considered it a poor practice in view of the importance of RCIC.

Other RCIC valves had UFSAR commitments that did not appear to be adequately verified by licensee testing programs. Differentiai pressure requirements were nut verified for steam supply isolation valves F007 and F008, and differential pressure requirements were not verified and no testing, in the open direction was dore for turbine exhaust to torus vacuum breaker isolation valves F062 and F084 No provisions of any kind were found in licensee testing programs to verify the performance of the following valves: turbine exhaust to torus vacuum breaker check valves F063 and F064, cooling water line relief valve F018, and barometric condenser relief valve F033. The licensee had no program for testing relief valves that were not included in the Section XI IST program.

For a discussion of the licensee's effort to develop a check valve testing program, see Section 3.6.2.7.

Several valves essential to successful operation of the RCIC system but without UFSAR or other specific requirements appeared to be inadequately tested. The pump suction and discharge check valves, and the suppression pool suction check valve were not tested by any licensee testing program. The following motor operated valves must reposition to support RCIC operation, but were receiving no stroke time testing to detect degradations in performance: condensate storage tank suction isolation valve F010, suopression pool suction iso.lation valve F029, and cooling water line isolation valve F046.

No testing cf the RCIC turbine mechanica'l or electrical overspeed trip functions was apparently being done. DER 88-0279 documented deficiencies in RCIC turbine maintenance procedures anc indicated that no procedures existed for overspeed testing. These deficiencies were discovered in February 1988 and had not been resolved at the time of the team's review. ,

During a procedure revision in January 1988, the licensee discovered that the 50 second maximum RCIC system response time specified in the UFSAR (e.g.,

Section 7.4.1.1.3.5) was not being periodically verified. While this discrepancy was under review, the licensee installed a modification that was expected to affect RCIC system response time (a blank flange was placed in the line that bypasses turbine steam supply valve F045). To meet the UFSAR commitment and verify the effect of the modification, the licensee decided to monitor RCIC system performance using the General Electric Transient Analysis Recording System (GETARS) computer. A testing job was created in the PST program (see Section 3.4.2.1) to administer the GETARS test'ing, and surveillance procedures that run the RCIC turbine were revised to call for  ;

taking a GETARS trace.

Several deficiencies in this process were found by the team. The PST database adequately)specified parameters which GETARS channels (and thus which RCIC systemwere to to verify system response time and turbine peak RPM). Although the database was capable of flagging the test as fulfilling UFSAR commitments, this had not '

been done, and as previously noted in Section 3.4.2.1, the PST database did not identify the end user (i.e., the reviewer) of the completed test, which could 75

potentially lead to significant delays between test performance and analysis of results.

There was no period committed to for performance of the GETARS testing.

Although the PST event was scheduled in conjunction with a RCIC 12-week surveillance, the applicable step in the surveillance procedure stated "If available, start GETARS to record the RCIC start," leaving open the possibility that the testing would not be done for an extended period if GETARS was not available for some reason.

Evidence was found by the team thet the failure to properly verify UFSAR comitments by periodic testing was not limited to the RCIC system. In May 1987 the licensee documented in DER 87-0158 that no periodic test procedures existed to test reactor building floor and equipment drain system motor operated flood control valves, contrary 'to the comitments of UFSAR Section 9.3.3.2. The licensee appeared to have no mechanism to systematically identify UFSAR commitments that warranted periodic testing, or to ensure the timely development and implementation of such testing after tne need for it was identified.

Following maintenance that may affect the operability of a TS-related system or component, operators comonly verified the operability of affected equipment by performing TS surveillance test procedures. Such testing may be insufficient to verify the adequacy of the maintenance or to satisfy UFSAR comitments. For example, the packing for RCIC turbine steam supply valve F045 was tightened in January 1988 to eliminate steam leakage. The work package called for a post-maintenance valve stroke time test, but for reasons that were not well documented or understood by the licensee (refer to DER 88-0102).. the work package was closed out without the valve stroke timing being performed.

Surveillance test procedure 24.206.01, "RCIC System Pump Operability and Valve Test" was performed successfully four days af ter work was completed. While this test verified that the system delivered rated flow at the required pressure in accordance with TS 4.7.4.b, it did not time the stroke of valve F045 or test the overall response time of the system, both of which could have been adversely affected by the maintenance that had been performed, and both of which had maximum values specified in the UFSAR.

3.4.3 Performance Evaluation Program The Performance Evaluation Program (PEP) was begun in the fall of 1987 to improve plant reliability, availability, and efficiency. Responsibility for the program lay with the ISI Group within the Technical Engineering Gru p.

After completion of its development, the program was to provide performaace monitoring, testing, er.d evaluation methods for both BOP and safety-related systems and components. Examples of monitoring techniques included direct observation during plant rounds, and predictive monitoring such as lubricating oil and vibration monitoring. Results of TS required testkg would be used, as well as performance tests devised specifically for PEP. Energy balance computations, vibration spectrum analysis, lubricating oil analysis, and trending of valve stroke times are examples of predictive evaluation techniques that would be used under PEP.

There were several potential benefits of completing and implementing PEP.

Plant efficiency, reliability, and availability, could be improved by identifying optimal equipment operating ranges, and identifying and correcting

)

76

plant problems before failures occurred. The data gathered under PEP could i also be used to better prioritize corrective maintenance items, and to optimize the types and frequency of PM done on plant equipment. '

Despite the potential advantages of PEP, the program had received little support from upper management. The manpower allotted for PEP activities was such that the program was only about 10 to 15 percent developed by the  ;

licensee's estimate one year after its inception. The Technical Engineering Group submitted a request to the Plant Manager for two more staff members to -

support PEP while the diagnostic team was on site. The team viewed the PEP as  :

an example of lack of followup in the implementation of a program.

3.5 Ouality Programs and Administrative Controls Affecting Quality  ;

The team reviewed the Quality Assurance (QA) organization staffing, scope of .

responsibilities, programs, and program implementation to evaluate QA effectiveness with respect to specific activities associated with plant opera-tions. This review included the licensee's organization and programs for QA i auditing and surveillance activities. The team found the QA organization was j performing its safety function n an acceptable level with weaknesses noted in -

the near-term contributions of the surveillance groups. The tcam also reviewed the licensee's administrative controls affecting quality and the Deviation [

Event Report (DER) process, including root cause and corrective action deter- i minations. The team found the DER process to be effective in identifying .

problems, but that current tracking practices and trending processes tended to (

limit their effectiveness in identifying generic problems; also, there were  !

weaknesses in root cause analysis.

3.5.1 Quality Assurance Functional Organization [

The QA organization at fermi 2 was headed by the Director, Nuclear Quality I Assurance and Plant Safety, who reports to the Senior Vice President Nuclear  !

Generation. Reporting to the Director, Nuclear Quality Assurance and Plant [

Safety, were the Supervisors of Quality Systems, Production Quality Assurance,  !

and Quality Engineerir.g and the Director, Plant Safety. The entire Nuclear Quality Assurance and Plant Safety (NQA/PS) organization, consisting of .

approximately 70 personnel, was located onsite at Fermi 2. ,

The NQA/PS organization underwent a significant reorganization of positions and I personnel since the appointment of the current Director, NQA/PS in January  ;

1988. The current supervisory-level personnel overall had good experience in j plant operations. The Supervisor, Quality Systems, and Director, Plant Safety, had active SRO licenses on Fermi 2, and the Supervisor, Quality Engineering, was previously licensed as an SR0 on a BWR plant. Additionally, the Supervisor i of Corrective Action Evaluations within Plant Safety had operations experience i as an STA at Fermi 2. The remainder of the NQA/PS staff did not have i significant experience in plant systems or operations.  !

Two additional SRO-qualified individuals were scheduled to be added to the QA  !

staff by the end of 1988. Another QA staff member was scheduled to start a l 7-month SR0 certification training program in January 1989. j t

The Quality Systens group performed all QA audits of the nuclear organization units to implement the QA program. This group also reviewed procurement  ;

documents for QA requirements, conducted audits and surveillances of vendor j 77

activities, and maintained approved vendor listr. Since the beginning of 1988, audit; conducted by Quality Systems have shifted more towards performance-based reviews.

The Quality Engineering group surveyed Nuclear Engineering design activities, Nuclear Production Technical Staff activities, and design changes and installations at the facility. These surveillances did not fulfill the requirements of any Technical Specification (TS), but were used by Fermi management as an additional tool to measure performance. In addition, Quality Engineering supported Plant Safety on the SCRAM /ESF Actuation Investigation Team for investigations of an engineering nature.

Production Quality Assurance consisted of a Surveillance Group responsible for non-TS day-to-day operational surveillances of health physics, chemistry,

- radiation protection operations, maintenance, and warehouse activities; and 4 Quality Control (QC), group responsible for all QC inspection activities, work request reviews, and site contractor procedure reviews.

The Quality Control group. consisted of 12 inspection specialirts, 10 of whom performed shift work. Around-the-clock QC coverage was provided by two QC inspectors assigned to work during each evening shift and remain on call if maintenance activities require QC support during the night shift. During major plant outages, contractors supplemented the small QC staff.

Plant Safety was responsible for conducting or monitoring various plant performance activities, including Human Performance Evaluation System (h.'ES) reviews to investigate personnel errors, SCRAM and ESF actuation investiga-tions, LER reviews, coordination of operating experience reports from INPO, Deviation Event Report (DER) tracking and trending, fire protection, and industrial safety activities. DER reviews, DER trend reports, and SCRAM-reduction efforts were recently added to the responsibilities of Plant Safety. Plant Safety also coordinated the accountability meetings discussed in Section 3.1.

2.5.2 Deviation Event Report System The Deviation Event Report (DER) System, initiated in January 1988, was the primary discrepancy / problem-reporting, root cause, and corrective action determination system used station wide. In addition to the discrepancies / problems reported by the Nuclear Generation organization, the DER System tracked and trended findings from QA audits and surveillances and Corrective Action Requests. The OER System also tracked actions initiated by the licensee in response to NRC and industry reviews and operating experience information.

In reviewing DERs, the Plant Manager assigned the r.esponsible department for the initial investigation, as well as the priority for completion of the initial investigation. The Plant Manager also reviewed each DER for adequacy and final closecut after completion of the proposed corrective action. Through the DER tracking system, the Plant Manager closely monitored overdue DERs on both initial investigations and corrective action completions, and discussed on a weekly basis the current status of efforts to complete those DERs with the assigned department.

78

o .

Overall, the DER tracking system provided an adequate means to record, ronitor, and closecut individual DERs; however, the team found the tracking system to be less effective due to a failure to follow procedures as well as containing procedural weaknesses, c While the guidance provided in procedure FIP-CA1-01, "Deviation and Corrective Action Reporting," appeared clear as to what situations require the initiation of a DER, this procedure was not always followed in

practice. In discussions with personnel from Plant Safety, the team found that problems discovered in the course of performing corrective action under an existing DER may or may not generate additional DERs. If the additional DERs were not generated, the new problems and their corrective actions were added to the existing DER. This practice is undesirable.

For example, one or more problems may be identified by the corrective l action program that do not have the same root cause as the problem for 1 which the original DER was issued and would not be captured in the trending program if they were not documented saparately in their own DERs.

o The team also observed that a number of similar deficiencias may be

]

grouped into a single DER even though the de'iciencies occurred several months apart. For example, a deficiency noted during Surveillance 88-0054, conducted in May 1988, was added to an existing DER (88-0468) generated in March 1988. This practice reduced the likelihood of identifying trends which could indicate a generit problem.

1 4 o Conversely, insufficient guidance was provided frr detemining under what circumstances individual DERs could be closed by consolidating them into a l

single. DER. Such a situation would be appropriate when a single event or

> problem was reported through multiple channels. For example, the team

! found that several industry advisories as a result of a power oscillation

event at another facility were appropriately combined into DER 88-0056. A separate DER (88-1204) was issueo in tracking the response to NRC Bulletin 88-07 concerning the same event. A situation where combining items did i

not appear appropriate to the team was when a DER was closed by reference i to a previous DER issued on a similar problem with the same or similar i corrective action. An example was the closeout of DER 88-0174 based on j the proposed corrective action of DER 87-0565. The practice of 1

consolidating the corrective actions of several DERs inte one DER and closing out all DERs by reference to a single DER was inc1nsistent with l General Requirement 5.2 of FIP-cal-01 which stated that DLRs shall be

tracked to completion and shall not be considered complete until all d

related corrective er,tions have been completed. In this cise, while the i original DER forms may reference another DER for closecut (f corrective

! action, the tracking system problem description was not always updated to include traceability to a referenced DER. Therefore, DERs listed as

! closed in the tracking system did not implicitly mean or provide assurance that associated corrective actions were in fact completed.

l l The DER Tracking Program and the DER Trending Program were separate entities.

I Obtaining reports from each required separate computer runs. Additionally, global searches of a particular type of component for a specific problem or trend code could not be made without prior research to deterinine the systens where the component was instN ied. Thus, multiple searches had to be made by system and the specific prG l.. or trend code to collect all relevant data.

The trending program had limited capability in that only the major root cause 79

~

. , , l could be assigned for each trend code. Other specific root causes could not be listed. The trending program was not very efficient as discussed above and the '

effectiveness of the program appeared to be limited. The licensee planned to combine the tracking and trending programs.

DER trend reporting had undergone significant revision and reformatting by the licensee. Analysis of trend information under the new DER system was complete only for the first quarter 1988 (covering January 11, 1988 to March 31,1988).

With the new DER trend report, four major root cause groups (General, Hardware, Procedures, and Personnel) provided general indication of areas requiring a more intensified review. Each of the four major groups was subdivided into specific root cause descriptions. Evaluation of root causes is conducted by the subcategories within each major root cause group and by the significant contributing organizations to each major root cause group. Due to their high visibility, LERs, each of which was initially described in a DER, were reviewed separately by monitoring the contributing organizations to the major root cause groups. Because of the revisions in the DER process that affected the number of trended root causes, the licensee did not attempt to compare the first quarter 1988 results with previous trend reports. .

3.5.3 Root Cause Analysis / Corrective Actions Root cause analysis by the licensee was weak. Senior Fermi management personnel indicated that HpES was the program used to evaluate root cause whether the event involved personnel errors or technical issues.

HPES wa's designed by INPO primarily to provide analytical methods for evaluating events involving human performance problems. HPES methods aided in determining causal and behavioral factors, including the physical environment, procedures, man-machine interfaces, and managerial oversights. Based upon a review of the Fermi Business Plan and on discussions with other Fermi personnel, the team learned that other methods were being developed for root cause determinations of technical issues.

The organization most closely associated with the problem was normally responsible for the root cause evaluation and determination of corrective actions. The inability of these organizations to consistently meet licensee-specified evaluation rejection rates was attributed to insufficient root cause training.

Plant Safety had the responsibility of reviewing root cause determinations and evaluating the corrective actions on DERs. The team observed that only a limited number of personnel had been trained in the HPES and that this lack of training, in effect, resulted in Plant Safety becoming th "safety net," and placed a significant reliance upon their review to ensure that proper root cause determinations were made. The QA/PS Business Plan (! tem 4.A.2) had a goal to maintain the quality of DER corrective action responses such that the number rejected for inadequate responses was less than 5 percent of the DERs eva lt.a te d . As reported by Plant SafetyI a rejection rate of less than 5 percent had been achieved for only 11 ci 33 weeks. While the rejection rates had generally declined during 1988, the less than 5 percent rate was achieved for only 3 weeks during the last d weeks reported prior to the week of September 7, 1988. Overall, the re jection rate of DER evaluations for 1982 was 7.7 percent.

80

r At the beginning of the diagnostic evaluation, six individuals had completed HPES training. The licensee indicated that approximately 20 additional individuals completed HPES training by the completion of the DET onsite evaluation. Subsequent team review indicated that 17 individuals had been trained; however, five of these were non-Detroit Edison personnel and not associated with Fermi. Additionally, no one in the Maintenance Department completed HPES training, even though a significant nurter of DERs were assigned

to Maintenance for corrective action.

Additionally, the team considered QA/PS Business Plan item 4.A.2 to be misplaced. All Fermi departments that were nomally responsible for initial root cause evaluations and determinations of corrective action: should have had this goal; rather than just Plant Safety, which functions as the runitor and evaluator of these initial deteminations.

3.5.4 Audits The team reviewed audit schedules and selected audit reports for 1987 and 1988.

Overall, the team evaluated the'QA Audit Program as a strength becauser it was used effectively at Femi. Audit programs were modified to be consistent with the recently developed Fermi Management Directives (FMD). Matrix tables were developed to cross reference where the various requirements of Title 10, Code of Federal Regulations, Regulatory Guides, and other Standards were covered by specific FMD policy numbers. Audits were scheduled to review the activities within a given FFD policy number. Strengths were noted in the audit program as follows: -

o The Nuclear Safety Review Group (NSRG) took an active role .in audits conducted by Quality Systems by participating in the ' planning, perfomance, and documentation of selected audits performed under the cognizance of the NSRG, as specified in the Fermi TS, as well as reviewing all other QA audits which were not under the purview of the NSRG.

o A review of selected QA audit reports by the team indicated that both supervisors and lead auditors properly planned the scope of the audits. A shift toward performance based audits was evident. Generally, the audit teams included technical experts in the subject area being audited, using cortracted technical experts to assist in audits of the fire protection program, maintenance, and vendor QA programs. Audit reports appeared to be thorough in identifying attributes of the subject area evaluated, and in most cases adequately identified and described deficiencies in terms of technical issues and prioritized their concerns. Recurring audit findings received higher-level management attention by use of Corrective Action Requests and accountability meetings.

Weaknesses were also noted in the audit program as follows:

o The QA audit of the Safety Review and ! valuation Program (A8-0037) appeared to improperly evaluate the program. Specifically the audit report stated: "0SRO was also verified to have reviewed the Security Plan yet revisions thereto have not been reviewed by OSRO. It is noted that Tech Spec Section 6.5.1.6(j) requires OSR0 review of the Security Plan, but does not rention revisions. Therefore, no deficiencies are noted in this area." The team viewed the Security Plan as a living document such that changes to the Security Plan became part of the Security Plan.

81

, Therefore, the team considered such changes to be part of the required

! OSR0 reviews under TS 6.5.1.6(j).

l

! o QA audit findings in general appeared to receive adequate visibility and

, management attention since audit findings were documented as DERs. (

)

However, some issues took an excessive amount of time to correct. An  :

example was the measuring and test equipment (M&TE) audit findings in (

Audit 88-0034, conducted May 23 through June 3, 1988, where 6 of the 7 i DERs issued as a result of the audit involved deficiencies similar to i those documented in 1987 audit findings a.id in DERs which were closed  !

within the past year.' A subsequent Quality Surveillance Report for surveillances performed July 31 to August 10, 1988, documented that at  ;

that time, corrective actions initiated for two OERs associated with the [

4 above audit were determined to be less than effective in preventing '

recurrence. This condition had persisted, even though the audit findings L were the subject of an accountability meeting. This issue was thoroughly [

4 discussed at the NSRG meeting conducted during the third week of the i onsite DET review. The team understood that this issue was being actively j pursued by management to assure resolution.

1 3.5.5 Non-Technical Specification Surveillances l Non-TS surveillances were performed by the Quality Engineering and Production  ;

Quality Assurance groups. Although these surveillances were mainly a i

management tool, the team evaluated the program to be weak overall. i i Surve111ances performed during 1988 resulted in few findings or observations.  !

P When findings or observation were made, they generally involved administrative  ;

or compliance issues. Occasionally, a few findings resulted in the generation  !

4 of a DER. As a result, the effectiveness of the surveillances was questionable and the attention given to the findings appeared limited. Some surveillances 4 conducted appeared to have questionable appropriateness. For an example, Surveillance S-0A-88-030, was conducted by a nonlicensed individual to measure [

the effectiveness of the Centrol Room Evaluation Team monitoring evaluations of t j trainen on the simulator.

Occasionally, the individuals perferming surveillances did identify significant  !

indicators or precursors of technical problems. Due to the current nature of [

i surveillance activities, the precursors of problems were not given sufficient  !

follow up attention for identification and correction of potentially generic or  !

chronic problems. An example was Surveillance 88-0054, which was performed to  ;

, verify effective implenentation of MOV actuation testing activities. No  !

( findings, DERs, or observations were issued as a result of the surveillance;  !

i however, two J ciencies were identified during the surveillance that included [

j incorrect maxim 6.A thrust acceptance criteria and actual measured thrust for an j j MOV that exceeded the maximum allowable thrust. The surveillance report stated that these deficiencies were resolved during the surveillance. These issues i

] were part of a much larger generic problem for which the licensee had other l

iv cators, including previous DERs. The torque switch setting issue is j i V ussed in Sect'ons 3.3.4.1 and 3.6.2.1. The second deficiency noted was  ;

resolved by DER (88-0468) which identified thrust anomalies. Combining a  ;

current deficiency with previously identified deficiencies appeared to have t

masked the above generic problem and was a questionable practice, as discussed l i

4 in Section 3.5.2. ,

a I  !

82 l

Schedules for planned surveillances were generated each month for the following month. Generally, planned surveillar.ces were used to verity adequate implementation of corrective actions on previously identified audit deficiencies or deficiencies identified by the NRC or INPO. QA management recognized the limited operating plant systems knowledge and experience at the working level within the groups performing surveillances. They were attempting to improve the technical capability of the staff. QA management eventually intends to use surveillance activities to identify and resolve issues prior to their discovery as part of a QA audit.

3.5.6 Safety Review Committees The team reviewed the activities performed by offsite and onsite safety review groups and found their overall level of performance to be adequate. Strengths were noted in the offsite review groups; however, the team identified some weaknesses in activities performed by the onsite safety review groups.

3.5.6.1 Board Nuclear Review Committee (BNRC)

The BNRC, a non nanagerial comittee made up of members of the Board of Directors, monitor overall safety policy, plant operations, and review activities. The activities of the BNRC were not covered by regulation. The team viewed their activities as a strength oy providing members of the Detroit Edison Board of Directors with a broad overview of Fermi and industry-wide activities.

The tean reviewed BNRC activities by reviewing minutes of meetings conducted during 1987 and 1988, and observing BNRC member interactions at Nuclear Safety Review Group (NSRG) meetings. In general, BNRC reviews covered a broad spectrum of design, operational, and personnel issues, as well as performance and safety reviews. The BNRC was also involved in the review of generic and industry-wide issues which required forward-looking and long-term planning and implenienta tion. Attendance at BNRC meetings, in addition to the comittee, normally included the Chairman of the Board and Chief Executive Officer, the President and Chief Operating Officer, Vice Chairman of the Board and Chief Financial Officer, Senior Vice President Nuclear Generation, Vice President Nuclear Operations, and Vice President Nuclear Engineering and Services.

Members of the BNRC also regularly attended scheduled meetings of the NSRG.

While not specifically on the agenda, the attending BNRC members were observed by the team to actively participate in the NSRG meetings by listening attentively to the issues discussed and asking appropriate, pointed, and inquiring questions to company management for the issues identified.

3.5.6.2 Nuclear Safety Review Group (NSRG)

The team review of the NSRG included document reviews of audits performed under the cognizance of the NSRG in 1987 and 1988; minutes of NSRG meetings conducted during 1987 and 1988; discussions with the Director, NSRG; members of the NSRG staff, and observation of an NSRG meeting that was attended by two members of the BNRC, Overall the tean concluded that the activities performed by the NSRG were forward looking, appropriate, and provided a strength to the Fermi organization.

83

The NSRG, as specified in the Fermi TS, provided independent reviews and audits of designated activities in nuclear power plant operations, nuclear engineering, chemistry and radiochemistry, metallurgy, instrumentation and control, radiological controls, mechanical and electrical engineering, and quality assurance practices. The NSRG membership was composed of both Detroit Edison and non-Detroit Edison personnel. In mid-1987, changes were made in the membership that shifted the focus of the NSRG to the operational phase of the nuclear plant and away from the construction phase. The non-Detroit Edison personnel provided the NSRG with a broad range of outside nuclear experience and knowledge, enhancing their ability to consider generic and industry-wide issues, in addition to the required reviews of current plant activities.

The NSRG meeting minutes provided the team with sufficient details about the issues discussed to indicate NSRG attitudes and concerns. The NSRG placed a strong emphasis on quality safety ev.aluation reviews, performance-based QA audits, and chemistry issues. The NSRG's three active subcommittees monitored, reviewed, and presented to the full NSRG informatiun about 0A audits radiologicalcontrols/ chemistry,andtheOnsiteReviewOrganization(OSRO).

Members of the NSRG were actively involved at the site by scheduled participation on selected QA audits and by tours through plant work areas.

The team noted several instances where NSRG meetings were conducted by telephone conference call or where individual members were tied into meetings by speaker telephone. Examples include NSRG Meetings87-05B, 88-01A,88-02A, 88-02B,88-02C, 88-03A and 88-038. For some of these meetings, the members on the telephone were counted to meet quorum requirements. The team views the use of the telephone by individual NSRG members or teleconferences among members to be an inappropriate way to establish a quorun to meet requirements specified in TS 6.5.2.6 for the conduct of NSRG meetings. Further discussion concerning the use of telephones and walk-throughs for quorum meetings is contained in Section 3.5.6.4 3.5.6.3 Independent Safety Engineering Group (ISEG) as specified in the Fermi TS, The shallIndependent be composedSafety Engineering of at least five dedicated Group fu (ISEG)ll time engineers who function to examine unit operating characteristics, NRC issuances, industry advisories, LERs, and other sources of plant design and operating experience information, including information from plants of similar design, which may indicate areas for improving unit safety (TS 6.2.3.1). The ISEG was responsible for maintaining surveillance of unit activities to provide independent verification that these activities were performed correctly and that human errors were reduced as much as practical (TS 6.2.3.3). In addition, the ISEG had the authority to make detailed recomendations for revised procedures, equipment modifications, maintenance activities, operations activities, or other means of improving unit safety (TS 6.2.3.4). Overall, the team determined that the ISEG was ineffective in meeting its required responsibilities.

The team found that: (1) the ISEG had not been performing all functions specified in Ferni TS 6.2.3.1, (2) the focus of current ISEG activities has resulted in the majority of their time being spent on the review of safety evaluations (a responsibility of the NSRG delegated to the ISEG), with minimal time spent on surveillance of plant operations and maintenance activities, (3) the utilization of current ISEG members did not appear to meet the intent of Fermi TS 6.2.3.2, and (4) the ISEG contributed little toward the reduction Sa

1 1 ,

I r of human errors. The following paragraphs provide a discussion of each of these l findings:

(1) The team found that not all of the functions specified in Fermi TS 6.2.3.1 were being performed by the ISEG. Specifically, the ISEG l was not reviewing NRC issuances or industry advisories. The review of NRC issuances was performed by Licensing. Plant Safety reviewed l

INPO information and Nuclear Engineering reviewed vendor advisories, I but no single organization at fermi retained overall responsibility l for NRC issuances and operating experience information.

As a result of a reportable event involving inadequate implementation of a General Electric Services Information Letter (SIL) into procedures, the ISEG was involved in a programmatic, not a technical, review of GE SILs. Thirty-eight of the 453 SILs on file were I

evaluated by ISEG. Fifteen of the 38 SILs evaluated were found to be  ;

inadequately implemented. Followup activities were planned to review '

the implementation of all Sils received, but ISEG was not assigned to '

l perform this task.

1 (2) The focus of current ISEG activities resulted in much of their time being spent on the review of safety evaluations. Technical Specification 6.5.2.7 specified that the NSRG shall be responsible for the review of safety evaluations. The NSRG delegated this l activity to the ISEG. Through August 1988, the ISEG completed the I

review on 157 of the 164 safety evaluations received. The Supervisor, ISEG, indicated that the review of the safety evaluations I was the top priority within ISEG and it would continue to be a major effort through the end of 1988. The ISEG completed a total of seven task evaluation reports related to their responsibility for independent verification of unit activities, covering review of GE S!Ls, Technical Specification interpretations, observation of operator rounds, evaluation of an Engineering Design Package (EDP),

l observation of a local leak rate test activi:y, and observation of a I valve lineup. However, six of the task evaluations were completed by or prior to April 1988. The seventh report was completed in July 1988 as a follow-up to a previous evaluation. Only one other evaluation task concerning electrical busses was in progress during the team's onsite evaluation. Since that time two more task evaluations were generated as a result of DET concerns involving the review of DERs for precursors to generic or chronic problems and review of DERs for adequate root cause analysis.

(3) The composition of the ISEG, as specified in Technical Specification 6.2.3.2, shall be composed of at least five dedicated full-time engineers located onsite. The Fermi ISEG has a total of five members; however, since February 1988, one member of the ISEG was detailed full-time to the Technical Specification Review Team. While assigned to the Technical Specification Review Team, the detailed ISEG member reported to personnel within the line organization and, therefore, did not retain independency. This assignment of an ISEG member full-tine to Technical Specification Review Tean was not consistent with the requirements of Technical Specification 6.2.3.2.

I 85 l

(4) The ISEG, as specified by Technical Specification 6.2.3.3, shall be responsible for maintaining surveillance of unit activities to provide independent verification that these activities are performed correctly and that human errors are reduced as much as practical.

The Human Performance Evaluation System (HPES) was the primary program used by Fermi to evaluate human error and to identify root causes. The ISEG, as specified in Safety Review Business Plan (Item 1.C.5), had the responsibility to evaluate HPES reviews of events and follow-up actions. The team found that none of the ISEG members had been trained in HPES and none were scheduled to attend the September 1988 HPES training course. After the team finding was made, an ISEG member was subsequently assigned and completed the HPES training; however, this member was currently detailed to the Technical Specification Review Team, as described above.

3.5.6.4 Onsite Review Organization The team review of the Onsite Review Organization (0SRO) included document reviews of OSRO meetings minutes for 1988, observation of two OSR0 meetings conducted during the DET onsite evaluation period, and discussions with the OSRO Chairman. The OSRO, as specified in Technical Specification 6.5.1.1, functioned to advise the Plant Manager on all matters related to nuclear safety as described in TS 6.5.1.6. At Fermi, the OSR0 Chairman was the Plant Manager, an assignment which appeared to create an awkward organizational situation in that the Plant Manager headed a comittee whose role was to advise the Plant Manager. .

During its review of the 1988 OSR0 meeting minutes, the team found that:

(1) the OSR0 conducted several meetings by use of walk-throughs or via the telephore, (2) determinations in writing of whether or not an unreviewed safety question existed for each item considered was not evident as required by TS 6.5.1.7b, and (3) OSR0 alternate members were used for a period of over two months without being appointed in writing as required by TS 6.5.1.3. The following paragraphs provide a discussion of each of these findings:

(1) The team found numerous 1988 OSRO meetings that were conducted by the use of walk-throughs or via the telephone. (A walk-through is a process where a change is hand-carried from OSR0 member to OSR0 member and reviewed by each on an individual basis.) Examples of walk-through meetings include OSR0 Meetings 2621, 262H, 263B, 264A, 282C, and 284A. Examples of telephone meetings include 262A and 265A. The team views the use of walk-throughs or the telephone as inappropriate to conduct meetings when TS 6.5.1.5 clearly stated that a quorum be present before the OSRO can carry out its responsibilities. In addition to not providing the forum for a personal exchange of views among comittee members with different tecnnical disciplines, the adequacy of this type of review is questionable because members did not have access to drawings or procedures when making determinations by telephone.

For example, the OSRO "conducted a meeting" by telephone on August 25, 1988, regarding p~ mtial Design Change 9418 concerning pipe replacement as a rs S tigue failure of the EECW return line at the RHR purp seal - is discussion was followed by an OSRO meeting (284J) the . , to formally document the telephone m ,

meeting. While the determination that the change did not involve an unreviewed safety question remained the same, additional action items concerning vibration tests were initiated by the OSR0 that were not considered during the telephone meeting. The use of walk-throughs and telephone reviews to substitute for OSR0 meetings was previously identified by the NRC staff during the Operational Safety Team Inspection conducted July 27 to August 7,1987, as a questionable practice. Additionally, the team noted that OSR0 Meeting 272, held on April 19, 1988, did not meet the quorum requirements of TS 6.5.1.5 in that at least one-half of the OSR0 members, including alternates, were not present.

(2) Fermi TS 6.5.1.7b specified that the OSRO shall submit its determinations in writing to the Nuclear Review Safety Group with regard to whether or not each item considered under TS 6.5.1.6 (a)

J through (f) constituted an unreviewed safety question. While the written approval / disapproval of items considered by the OSR0 as specified by TS 6.5.1.6a was clearly stated, it was not evident that the unreviewed safety question determination was being made in writing. When the team pointed out this requirement, the licensee incorporated a statement in OSRO meeting minutes that approval of an item also constituted a determination that an unreviewed safety question did not exist. Long-term corrective action by the licensee included incorporation of a similar statement in FIO-FMP-01, Safety Review Group Organizations.

(3) . Fermi TS 6.5.1.3 specified that all alternate members shall be appointed in writing by the OSR0 Chairman to serve on a temporary basis. Alternates to OSR0 had been designated through issuance of plant Order EFP-1083, Special Personnel Assignments. With the issuance of F10-FMP-01 Safe *y Review Group Organizations, on June 3, 1988, Plant Order EFP-1083 was cancelled. Contrary to TS 6.5.1.3,

alternate members continued to be used at OSR0 meetings af ter June 3, 1988, without any appointment in writing by the OSR0 Chairman. This issue was brought to the attention of the OSR0 Chairman by the team and was corrected by issuance of a memorandum on August 25, 1988, designating in writing the alternate members of the OSRO.

The team observed OSR0 meetings on August 25 and August 26, 1988. Discussions at these meetings included reviews of startup test procedure changes, administrative procedures changes, and potential design changes. The ti, observed that the only copy of the changes in evidence at the meetings was the original held by the OSR0 Chairman. The general substance of each change was read by the OSR0 Chairman to the rest of the OSRO members and then voted upon.

Approval of changes tended to be silent votes of approval with no dissensions expressed rather that a positive statement or action by individual OSR0 members

signifying approval of each change discussed and cor.sidered at the meeting.

The OSRO Chairman told the team that all changes brought up for a vote at these

meetings had been out for review (i.e., previously distributed to all OSR0 i members). Any comments received from OSR0 members were addressed / resolved by i

the group responsible for the change, but these comments / resolutions were not specifically identified or discussed as part of the OSRO meeting. Negative responses were not required from OSR0 members. This procedure provided no assurance that OSR0 members voting approval of changes had, in fact, reviewed 87

. i the specific changes prior to voting. This same concern was also documented in QA Audit 88-0037 as Observation 2, which in part states:

. . . DER 88-0525 was classified as a reportable event and was presented for OSR0 review on April 19, 1988 and was subsequently approved af ter a short discussion as to the root cause of the event. The DER was not distributed to the OSRO members and details of the event documented on Security Report 88-0508 were not discussed. -

Personnel in Plant Safety and Quality Assurance wno were also responsible for reviewing the OER reported that these details were essential to their review. The OSR0 Chairman and four cembers who were present for the OSR0 review were interviewed and reported that they had not read the details of the event. Therefore, it was determined that the review was inadequate . . . "

3.5.7 Sumary and Evaluation of Findings -

Overall, the QA organization is performing its safety function at an acceptable ,

level. QA audits are well planned and show a concerted effort by nanagement to '

shift more towards performance based reviews. Audit findings receive the same level of attention as other deficiencies since they are an integral part of the "

DER system. 0A audits are effectively used by Fermi's organization. Recurring audit findings receive higher level management attention by use of corrective action requests and accountability meetings. All QA audits are reviewed by the NSRG.

Other factors within the current QA organization and procedures reduce the effectiveness of QA. Weaknesses that exist include the lack of BWR operating experience below the supervisory level. This in turn appeared to impact the -

cuality of surveillances conducted by the Quality Engineering and Production Quality Assurance groups. These surveillances have not achieved the level of '

performance based reviews shown in audits. Management was aware of these weaknesses and had action items in the Business Plan to address them.

l The D3 systens for tracking and trending are not fully effective. Guidance for initiating DERs is not always followed in practice and insufficient guidance exists for combining DERs in the tracking system. These factors

influence the effectiveness of the DER trending program. Additionally, inefficiencies exist in the trending program due to limitations on the J assignment of root causes and the manner in which data must be manipulated to '

conduct global searches. l Root cause training has been insufficient at the organizational levels performing initial root cause analysis and corrective action determinations.

This is evident by the inability of the licensee to consistently achieve the goal concerning rejection rates for DERs. This goal incidently is not properly assigned. It is presently assigned to the group that evaluates the DER responses instead of being a: signed to the organizations that initiate the DER (

root cause and corrective action responses. [

i ,

Correction of M&TE audit findings which were repetitive also appeared to be l taking an excessive amount of time to correct even after being the subject of

followup surveillances and an accountability reeting, ,

1 88 ,

i

The performances of the Board Nuclear Review Committee and the Nuclear Safety Review Groups are a definite strength to the Fermi organization. Both of these safety review comittees are forward looking and involved in long-term planning through the review of generic and industry-wide issues. Active involvement by the NSRG in QA audits and emphasis on quality of safety evaluations strengthens the position of the QA organization to carry out their responsibilities.

While the overall level of performance at Fermi 2 for the functions assigned to the safety review groups is evaluated as adequate, improvements in safety performance can be achieved by increased attention to the administrative controls specified by TS for the safety review groups.

Weaknesses exist within the safety review committees such as the use of walkthrough and telephone meetings to conduct TS required reviews. Walkthrough and the telephone by their very nature are noncondusive to providing an adequate forum for exchange of viqws and raise questions concerning adequacy of r,ev i ews . Additionally, the ISEG is not performing all TS required functions including review of NRC issuances, industry advisories, and reducing human errors as much as practical. The functions specified by TS as ISEG responsibilities are for the most part being carried out by other parts of the Fermi organization, but this does not relieve the ISEG of its responsibilities.

Based on observations of OSR0 meetings, the quality of reviews at these meetings are questionable due to no assurance that members actually reviewed changes upon which they voted and votes were silent votes of approval.

Additionally, based on audit reviews, the OSRO does not review Security Plan revisions which is not consistent with TS requirements.

3.6 Engineering Support The tean conducted an evaluation of engineering support provided to the Fermi plant by the Nuclear Engineering and Technical Engineering groups. The evaluation included reviews of the design change control process, corrective actions, engineering interf aces, failure analysis, event response, work prioritization, management changes, material management, staffing and

resources and related administrative procedures. Interviews were held with individualengineersandengineeringmanagementwithintheNuclearEngineering (NE), Stone and Webster Michigan (SWMI), and Technical Engineering (TE) groups to obtain information related to the evolution of the current engineering

organizations, work experience, educational background and the work environment at Fermi.

In the past few years improvements had been made to engineering support.

Several new managers had been selected. Interactions between Nuclear .

Engineering and other groups had been itaproved. Good systems for prioritizing proposed modifications and for performing 10 CFR 50.59 reviews had been instituted. However, there were a number of weaknesses remaining. Significant portions of the management team were not yet in place, staffing shortfalls were noted and there was a lack of clear cefinition for roles and interfaces.

Design change procedures were confusing and not well integrated. Technical inadequacies included: delays in dealing with potentially significant issues; loss of control of motor-operated valve torque switch settings; inappropriate temporary modifications and discrepancies between design documents and the as-built plant.

89

)

i 3.6.1 Staffing, h .6 a id Organization Resources in the em *F1  ; support organization appeared to be adequate to i accomplish requh N *v . until the Spring of 1988 when budgetary restrictions ,

calling for a subuoni,1a1 cutback in overtime went into effect. As a result of i this cutback in overtime, tighter review and approval criteria were applied for  ;

engineering design packages (EDPs) and potential design changes (PDCs), which j reduced the workload on NE (including SWMI) and some sections of the Technical i Engineering Group, j Specific results of the team evaluation as they relate to Nuclear Engineering, SWMI and Technical Engineering are included below, t 3.6.1.1 Nuclear Engineering lf Nuclear Engineering had undergone at least 10 organizational structure changes since 1984, including three different Vice Presidents of Nuclear Engineering i ano Services since 1986. With the exception of the Vice President of Nuclear i Engineering, all supervisors were in "acting" positions. The temporary status i of engineering supervisors has caused uncertainty and anxiety within the NE staff. l The Nuclear Engineering staff appeared to be well qualified technically, with a i good experience base across all the plant engineering disciplines. Some were  !'

registered Prefessional Engineers. However, of those who were interviewed, none of the Nuclear Engineering Staff had previous nuclear power plant experience at other operational sites. All of the Nuclear Engineering Staff l interviewed had positive attitudes and were dedicated to making Fermi succeed. ,

There had been a poor relationship between NE and the plant organizations,  !

Operations Department in particular, which began during the design / construction i phase and continued through the issuance of the low power license. The tean {

determined the root cause to be a reluctance within NE to relinquish the  ;

influence and attendant prestige it had enjoyed during plant design, constructfon, and preoperational testing to the Operations Department as the {

transition was made to the operational phase. After occupying the premier i I

position within the Fermi organization for many years, it was difficult for NE to assume a support role, and this resulted in poor comunication and teamwork between NE and Operations. Subsequently, NE management was replaced, and although additional improvement was still needed, interviews conducted by the team indicated that substantial improvement in the interaction between NE and  !

the plant organizations had a! ready taken plac2. [

t 3.6.1.2 Stone an' Webster Michigan' -

[

SWM1 which was located onsite, performed the majority of engineering design f modificationworkandreporteddirectlytotheGeneralDirectorofNE. As with <

NE, there had been changes in the organizational structure and supervisory  ;

personnel of SWMI since SWM1 became a part of the Fermi organization in Septer4er 1985. The staffing level of SWMI had fluctuated significatly during I this tire as well. The team found that these frequent changes in SWM1 were due i to the failure of NE, with its organizational and leadership instability, to i establish and maintain a consistent policy for the employment of SWHI, At the (

time of the diagnostic evaluation, Fermi management had still not established ,

i 90 l

_ _ . _ _ - ~ - - _ _ . - . - .

l

. I f

such a policy or dtcided to what extent NE personnel should be involved in  !

design modification work.

3.6.1.3 Technical Engineering

! . Technical Engineering, as part of Nuclear Production, is located onsite and j

! currently reports directly to the Plant Manager. Two key management changes i l recently occurred to bring in personnel having BWR operations experience.  ;

These were the hiring of a new Superintendent of Technical Engineering and a  !

new Supervisor for Plant Systems Engineering. The team views these additions as positive. Technical Engineering was made up of four main sections reporting to the Superintendent of Technical Engineering as shown in Fig. 1.5-2.

The team mainly focused on the Systems Engineer function of Technical

^

Engineering since Systems Engineers were assigned responsibility for temporary j modifications, provided technical expertise to resolve plant systems and i equipment problems, and acted as the primary engineering interface with operations and maintenance personnel. Nuclear En

, system and equipment expertise responsibilities from a design(gineering basis controlas well had p i

standpoint) and interfaced with plant operations and maintenance personnel.

l The interface between System Engineers and Nuclear Engineering was not defined.

The team found no procedures or other documents that clearly defined the Systems Engineers' role, function or responsibilities. The lack of established

detailed procedures or other form of formal work interface control, constituted a definite weakness regarding corrunication between engineering support groups,

' operations and maintenance. The licensee had recognized the weak interfaces i

and had hired a consultant to study interfaces and recomend improvements.

]

During the team evaluation of the System Engineers group, it appeared that

]

these engineers were over-committed and that the group was-understaffed.

Morale and performance seemed to be affected by the constant pressure of j retsponding to plant system problems or operational demands. There were a

approximately 250 systems in the plant assigned to only 9 systems engineers.

System Engineers were assigned systems on a primary and secondary (or backup) basis. When considering both primary and secondary n'esponsibilities, a Systems 4

Engineer may have 90 or more systems to follow. System engineering

organizations at similar EWRs typically have more than 20 dedicated engineers to do the equivalent job that Fermi was attempting to perform. Due to workload demands, the System Engineers have historically worked overtime on a regular basis. For example, with the exception of a brief period from May to June 1988 when overtime was not authorized, the overtime rate had varied between 25 and 40 percent. The highest number of overtime hours worked by any one individual in one week was 50 hours5.787037e-4 days <br />0.0139 hours <br />8.267196e-5 weeks <br />1.9025e-5 months <br />. ,

Due to the current staff shortage and work constraints the Systems Engineers wereadmittedlyunabletobecomeasknowledgeableofalloftheirassigned systems as they should. Some Systems Engineers also appeared to lack general knowledge in various areas such as systems interactions Technical Specifications, and pump and valve operation. Additional training in these areas is needed. These engineers did have adequate knowledge for some of their assigned systems because of experience gained during the startup program.

DECO management approval had been granted to hire 13 additional system engineers and to upgrade the pay scale. The team perceived this as a positive 91

e step which should increase the effectiveness and performance of the Plant Systems Engineering group by reducing the workload as well as improving morale.

However, as the group becomes fully staffed with the addition of 13 more engineers, function of aappropriate training (will Systems Engineer be required resolution to problems, of plant adequatelyplant perform the modifications, handling of Temporary Modifications, and root cause l determination). ,

3.6.2 Engineering Involvement 1

When given an adequate amount of time to thoroughly examine various deficiencies (licensee event reports or high visability deficiencies),

Engineering appeared to do an acceptable job at root cause analysis and of establishing appropriate corrective actions. However, the evaluation team discovered many areas involving Engineering support to the plant which could be i greatly improved. For example engineering: (1) failed to control motor operatedvalve(M0Y)torqueswltchandlimitswitchsettingseventhoughthe problems were known to exist for a number of years; (2) took an excessive amount cf time to resolve and closecut deficiencies; (3) did not perform adequate reviews concerning the consequences of performing or not performing modifications; and (4) did not adequately address the design significance of safety-ralated or balance-of-plant (BOP) check valves.

l 3.6.2.1 Motor Operated Valve Torque Switch Setting Control l

It was apparent that Nuclear Engineering had not controlled toroue switch or limit switch settings in the past and failed to resolve the finding once it was discovered. The licensee was aware that torque switch settings were not adequately controlled as early as October 1984 as reported in letter NE-84 1351. The Fermi $5FI identified the fact that controlled documents did not exist to ensure proper and consistent settings for torque switches on December 17, 1987 The licensee had identified many instances where valve operators failed to operate because torque or limit switch settings were too low and also discovered instances where torque switches were set considerably beyond the maximum allowable.

Prior to the failure of valve B31-F031B to operate in August 1988. Engineering had not yet generated an effective program, including controlling procedures, to maintain torque switch settings in accordance with design requirements.

Subsequent to the valve failure, the licensee examined records and perfortned actual inspections of valve operators resulting in 76 of 176 sa aty-related valves exhibiting some deviation between actual, maintenance or engineering lists regarding torque suitch settings. Torque switch settings for safety related valves must be maintained in design documents and verified such that the valve will perform its intended design function under varicus postulated conditions. Fermi's practice of using uncontrolled data sheets and skill of the craft to set and maintain torque switches or limit switches without engineering evaluation did not ensure valve operability. (See Section 3.3.4 regarding MOV issues.)

3.6.2.2 Configuration Control Based on a limited review of recent DERs, it appeared that numerous design deviations existed when comparing the as-built condition of the plant with i

! 92 l

design docurents or various data bases used by the licensee to keep track of design-related information.

As outlined in Fermi Management Directive CM1 Rev. O, "Design Control," Nuclear

> Engineering has the responsibility to ensure that structures, systems, and components are maintained in the configuration specified in approved design documents. The team's review indicated that in a number of cases critical drawings did not reflect the as-built condition of the plant because of the existence of unauthorized modifications or inadeouate or inaccurate drawings.

l A partial listing of recent DERs which relate to inadequate configuration control is included in Table 3.1.

i I

1 I

i 93

.o .

TABLE 3.1 Examples of Inadequate Configuration Control DER No. Subject 88-0493 Crawings 6M721-5710-16 and 6M721-5122 do not agree with as-built conditdon.

88-04g4 Unauthorized installed modificatiori to RHR pump roon ventilation.

88-0716 Drawing 6M721-5717 4 does not agree with as-built condition. ,

88-0720 Valve G51-F044 was missing (never installed).

88-0722 Unauthorized installed modification.

88-0723 Unauthorized modification. ,

88 0725 Hissing fuse.

88-0752 Inconsistency between control room drawings and as-built condition regarding valve identifications.

88-0763 Drawing 61721-2185-3 not in agreement with as-built cendition. .

88-0835 There are no administrative controls to ensure that plant procedures, PM programs, spare parts quantities, etc., are updated as a result of engineering char;es.

88-0874 bychem tubing not installed.

88-0875 Conflicting information regarding EQ equipment in CECO data base.

88 0910 Raychem heat shrink not installed.

88-0911 Incorrect unistrut clamps installed for main steam line .

I radiation detectors. l 8J-0912 CECO data base contained incorrect information regarding I thermocouples.

88 0934 Deficiencies in the master salve list (discovered a 1984) not  !

yet corrected.  !

l l 08-1322 CECO data base does not include all information contained on the master valve list. ,

88-1326 Labeling errors -- as-butit versus diagram.

88-1327 Unauthorized tempcrary modification installed, f 88-1328 Incorrect fittings installed.

I i

! i 94 i

4 +

3.6.2.3 Reactor Water Cleanup lodification Modification EDP-6671 was performed to remove the delay volume (24" diameter ipe) and relocate a high point vent installed in the reactor water cleanup p(RWCU) system. The purpose of the delay volume was to reduce the Nitrogen-16 l radiation and was an ALARA concern. Fermi was reported to be the only boiling water reactor that had a delay volt.me. According to Fermi, the RWCU was unable to support plant operations during normal shutdom depressurization due to flashing at the inlet of the RWCU pumps, resulting in pump trips. Void formation within the delay volJme was increased due to cooldown, system leakage and sampling evolutions. Preparation for the modification was started in November 1986 and was "as-built" specified in May 1988.

It was apparent to the team that Nuclear Engineering did not adequately examine the consequences of removing the delay volume and the effects of future void- -

formation on the smaller dit. meter replacement pipe. Prior to the modification, the length of the void within the 24-inch dianeter pipe was much shorter than the supposed length of the void with the 6-inch diameter pipe assuming that other related condittons remained somewhat constant.

Subsequent to the modification, two instances occurred which RWCU compression fittings separatee ahen the RWCU pump was started. The reactor building was contaminated from the incidents. Licensee analysis of the first event (May 28, ,

1988) indicated that the individual fitting (assembly) was the cause of the event. The root cause analysis lacked sufficient depth to discover that a water hamrer event had taken place. Wher the second event occurred (July 13, 1988) Nuclear Engineering determined that the failures were due tv water hammer caused by starting the RWCU pump after a void had been established in the piping. The void formation process (cooldown, leakage and sampling) was not remedied by EDP-6671, and the probability of producing a situation conducive to water hammer was greatly increased when the modification was installed. The ,

licensee issued a vent and fill procedure to minimize void formatior prior to starting the RWCU pump and intends to issue a PDC to instali a minimum flow bypass line around the pump as a permanent fix by September 1989.

3.6.2.4 Pressure and Temperature Upgrading of Core Spray Piping Modification package EDP-6349 (hydrostatic test of core spray piping) was accomplished to obtain objective evidence that the piping components between core spray valves E2150F004A/B and E2150-F005A/B were capable of withstanding "

the design pressure of 1250 psig of the reactor pressure vessel as stated in the UFSAR. This particular section of piping had been qualified co only a t pressure of 500 psig and a temperature of 212'F. Valve F005 is the outboard containment isolation ;alve while F004 is the outboard shutoff valve.

Prior to the suc escful accomplishmeat of the hydrostatic test, a portion of the core spray system was in an una'nalyzed condition. This condition was initially ciscovered by Stone and Webster during March 1986. DER 87-106 documented the conditian on March 20,1987 (one year af ter the discovery of the  ;

deficiency) and the hydrostatic test was not performod until March 16, 1988 f (another year later). As can be seen from the above dates, an excessive amount of time elapred from discovery to resolution. A safety-significant condition ,

could have existed if the section of piping had been improperly fabricated or had material flaws, and could have resulted in a pipe burst if pressurized b3 yond the existing materiai qualification of 500 psig at 212'F. Fermi senior 95

management made a aecision to postpone the qualification testing because they felt that a safety concern did not exist based on engineering judgement, and that other work items were more important. Time delays associated with E0P-6349 were indicators of inadequate attention toward potential safety issues and the requirements for timely and effective correc'ive actions.

3.6.2.5 Air Entrapment in EECW System During the performance of surveillance test 24.207.09 (EECW pump and valve operability test) on March 27, 1988, flow could not be establishad using flow indicator P44-FI-R405B. DER 88-0595 reported that "lots and lots" of air had to be vented from the system, and that the presence of air in the EECW system had caused problems with the flow indicator. The DER stated "A review of the control room NSO log book showed that fill and vent of EECW Div. II was completed at 0355 on 26 March 88, so there should not have been much air in the sy stem. " The team questioned the licensee regarding operability of the EECW pump and possible origins of the air. The licensee indicated that the air had come from opening the system up during the LLRT ottage. The corrective action required by the DER was to perform additional filling and venting as specified on April 27, 1988 because the filling and venting evolution performed on March 26, 1988 had not been effective. Engineering did not ini ally question whether damage had been done to the system while the pump was running with air in the system, or whether any operability cone. erns existed (air binding of the pump and/or in the cooler).

This event indicated two weaknesses cone;.ning engineering support; (1) the original vent and fill procedure was inadequata and was not well prepared and reviewed, and (2) when the flow instrument f ailed tu work because of air in the EECW lines while the pump was running, engineering did not evaluate possible damage to the pump or consider the question of system operability. The fill and vent procedure was revised on August 16, 1988 to add more vent valves since some true high point vents were not included in the original procedure.

Follow-up testing did, however, indicate that damage had not occurred.

3.6.2.6 Post Accident Sampling System The team evaluated the procedures for operation of the post accident sampling system (PASS) to determine if sufficient guidance existed to operate the PASS cooler to meet design requirements. There were some apparent conflicts between design and actual operation.

For example, the orginal basis for the period of three hours to perform a sample following an accident contained in NUREG 0737, "Clarification of TMI Action Flan Requirements" was one hour O atain a sample and two hours.for analysis. The sampling process requiru th.t the liquid samples be adequately cooled using the PASS cooler which ir c. & by the reactor building closed cooling water (RBCCW) system which is non-safety related system and is designed to be isolated from the EECW system upon an engineered safety features actuation signal (ESFAS).

Section 11.4.4.4 of the UFSAR states that "The PASS isolation valves and sampling panel are supplied with Class 1E power and automatically restorable power, respectfully. Both can be operated within 30 r.linutes of an accident in which there is a loss of offsite power. The post accident sampling system (PASS) provides the capability of obtaining recitor coolant and containment 96

atmosphere samples under accident conditions to enable analysis to be completed within three hours of the decision to sample."

The licensee indicated that they intended to restore offsite power prior to putting the PASS cooler in operation. The team concluded that this action did not meet the intent of the safety evaluation report (SER) or the UFSAR and the licensee procedures (POM 78.000.15, Rev. 3, "Determination of Extent of Core Damage" and POM 78.000.14, Rev. 9, "Post Accident Sampling Analysis") provided inadequate guidance on post accident sampling in the event that there was also a loss of offsite electrical power.

3.6.2.7 Engineering Response to INP0 SOER 86-3 Following the multiple failures of safety-related check valves at San Onofre Unit 1 in November 1985, the NRC requested the industry to develop and implement a comprehensive program to provide assurance that safety-related check valves would function properly and reliably under all design conditions.

As a result of this request, INPO provided guidance to each plant on the scope and content of such a program in SOER 86-3, "Check Valve Failures or ,

Degradations," dated October 15, 1986. The INPO SOER referen'ced ASME Section XI testing requirements and stated that..."the code requires that applicable valves be tested to verify that they will open or close to perform their safety function." INP0 also stated that "valve reliability could be improved by expanding the scope of inservice testing programs beyond the minimums required by the Code. In particular, the reliability of some important check valves not

.now included in inservice testing programs could be improved by a combinatier of periodic testing and preventive maintenance activities. Tests should be designed to demonstrate that check valves will fully open and close under actual or simulated operational conditions."

The licensee's response to 50ER 864 was formally isstad on August 18, 1988 (letter NE-PJ-88-0443). The licensec planned to review various check valve problems (e.g., sizing, leak tightness, material requirements, piping layout vs valve performance, maintenance, and drawing adequacy). The three systems initially identified for review were the HPCI, RHR and main steam /feedweter. '

Based upon this abbreviated study, the effort may be scaled up or down for the l ren,aining systems *. The overall effort appeared to be reasonable with the f exception of testing. Application Guideline 2.1.4 (attached to NE-PS-88-0443) was the only action item which referred to testing. Action items included (1) identify check valves that have specified seat leakage limits; (2) for those "

valves having seat leakage limits, deteruine which valves have a history of not passing seat leakage tests; (3) determine if seat leakage limits can be increased; and (4) review the design details of each problem check valve,  !

including manufacturer's test data, to determine whether it is practical to  !

achieve the specified leakage limit. Detennine whether a different type of  !

valve should be considered. l From the assigned action items, the testing evaluation consisted mainly of a ,

material history check on a narrowly defined set of check valves. The scope of Engineering's design reviews leakage) to determine or studies if safety-related (orrelating balancetooftesting ()othervalves plant check than sest were being properly tested to verify that the design function was obtainable appeared to be inadequate. A common proolens in the industry is that some check valves which must close to perform their safety functions are not verified to close but are only tested to pass flow in the open direction. NRC Information  ;

97

Notice No. 88-70, "Check Valve Inservice Testing Program Deficiencies," dated August 29, 1988 was issued to address some of the above issues.

The licensee forwarded a copy of letter NE-PJ-88-0619 dated October 6, 1988 to the team for review, which indicated that some engineering efforts were planned in addition to the material history reviews discussed within the licensee's response to SOER 86-03. Time did not permit a complete evaluation of these  :

additional actions.

As a general rule, check valves installed in safety-related (and nonsafety related) systems need to be reviewed to determine the safety function and operability status, for possible inclusion in, or revision to the licensee's testing programs. For example, DER 88-1419 was written to docume Region III inspection finding regarding the lack of testing of ro,nt an NRC d control system check valves C11-F111, C11-F1618 and C11-F161A. These check valves were ,

designed (assuming a failure of a solenoid valve) to permit depressurization of the air header past the failed valve, and to ensure the insertion of the control rods, it was the team's view that testing should be required to verify operability. An initial review performed by the lir.ensee on August 15, 1988  ;

indicated that valves C11-F111, C11-F161B and C11-F160A/B must function as t i

intended, i.e., that a full scram will occur on receipt of a full scram or automatic rod instrtion signal. The Region III inspection finding was reported ,

to the licensee during the September 8 to October 23, 1987 inspection. DER  ;

88-1419 was not written until July 29, 1988 (10 months after discovery) and was 1 still open at the close of the evaluation. .

3.6.2.8 RHR Pump Deadheaded for 33 Minutes The inadvertent closure of the RHR Loop B Recirculation Inboard Isolation Valve went undetect4d by the operator causing the RHR pump to run deadheaded for '

epproximately 33 minutes.

While making normal inspections of the control room panels, the control room 450 noticed that reactor water temperature had decreased approximately 10*F '

below where it had been all night. While verifying proper valve alignment of

, the RHR loop B flow path, which was being used for shutdown cooling, the l E11-F015B (RHR loop B isolation) valve was found closed and the isolation

'; seal-in light was lit. Review of the RHR loop B flow recorder determined that E11-F015B had been closed for approximately 33 minutes. The operator should ,

have known that the isolation valve was closed by the presence of the white seal-in light on the control board, and that the RHR pump was deadheaded. The r operator also had access to the control board RHR flowmeters which read zero l during this event. l A review of the closing logic for E11-F015B revealed that the only way for this valve to isolate and the isolation seal-in light to light was by energizing .

relay E11-K638, which would require all four of the following conditions to t occur: (1) the reactor water level had to be less than Level 3, (2) reactor ,

pressure had to be low, (3) valve E11-F008 was open, and (4) either valve l E11-F009 or E11-F608 was open. When the event occurred, the reactor was  !

shutdown and the RHR loop B flow path was being used for shutdoyn cooling, therefore, the only condition which was not met was reactor water level less than Level 3. The possible causes were evaluated in detail by Engineering. It was determined that either wiring in E11-K638 was momentarily shorted to  ;

voltage potential, or relay A71BK18 (which is energized at reactor water level i 98

_ _ . - - - . - - - - -Q

s' ,

C greater than Level 3) was momentarily deenergized causing E11-K638 to energize.

If either of these conditions existed longer than momentary then E11-F015B could not have been reopened.

Engineering's approach to this problem was' adequate in that the investigation ,

of the root cause appeared to be thorough.. Although the root cause could not be verified, it was determined that the most probable root cause of the valve closure was a defective A71BK18 relay. The relay was replaced and a functional test performed to reverify the logic. The replaced relay was analyzed for possible fault conditions, however, none could be found. A pump and valve operability test was also performed with satisfactory results.

3.6.2.9 L'w o Pressure Coolant Injection Swing Bus Design Deficiency The event occurred on September 8, 1987 when a licensed operator identified the ,

wrong fuse to be removed during preparation of a maintenance protection tag-out package. An operator in the field removed the wrong fuse, resulting in the loss of DC control power to bus 72C position 3C and that deenergized the AC Motor Control Center (NCC) 72CF Low Pressure Coolant Injection (LPCI) swing ,

bus. The normal feed to the LPCI swing bus was from bus 72C and upon loss of  ;

DC control power the power to the swing bus was lost which provided AC motive pewer and control to sesen LPCI loop selection, injection and isolation valves.

Nuclear Engineering subsequently reviewed the design and determined that the DC control circuitry for Bus 72C equipment was inadequate. Assuming a loss of coolant accident along with a loss of offsite power,.a single failure of DC control power would preclude proper autoaatic injection of all four LPCI pumps.

This condition, had it not been accidently discovered, could alone have prevented LPCI from performing its intended safety function to provide low pressure makeup water to the reactor vessel for core cooling under LOCA conditions, j Two design changes were developed to correct the situation. A plant design i review was also conducted to determine if similar conditions existed elsewhere; none were identified.

3.6.3 Fermi SSFI Corrective Action ,

The licensee contracted with WESTEC Services to conduct a Safety Systems l

Functional Inspection (SSFI) of the High Pressure Coolant injection (HPCI)

I system and certain safety-related systems which support HPCI operations. The I inspection began during the week of November 16, 1987 and concluded with an .

! exit meeting on December 18, 1987. As inspection observations occurred, they  ;

t were documented by WESTEC and discussed with the licensee, in total, 50 "inspection observations" were documented. The 50 findings resulted in the licensee writing 30 DERs and one Regulatory Action Commitment Tracking System (RACTS) item. Of the 31 documented findings, 10 were still listed as open ,

while others listed as closed still had outstanding or incomplete actions (i.e., corrective action included as part of the procedure improvement program l and the torque switch issue) which were required to be completed to adequately l resolve the findings. Four of the 10 listed remaining outstanding items were the responsibility of the Nuclear Engineering group. l l

99

i 9 It was also apparent that the evaluation and resolution process of identified findings did not receive adequate management attention since nearly all of the DERs were not writ'en until February or March 1988 (three months after the licensee first knew of the concerns), and that corrective actions had either been slow or failed to fix known problems. For example:

(1) DER 88-0155 was written to document SSFI inspection observation EP-4 dated December 15,1987, "Inadequacy of battery surveillance to ensure minimum battery cell electrolyte temperature." The SSFI report stated that "As stated in letter NE-PJ-87-0769, DC-0213 shall be revised to support the minimum temperature criteria no later than 1-31-88." Documentation provided to the team indicated that Rev. O to 0C-0213 was not issued until April 4, 1988.

(2) DER 88-0172 was written to document SSFI inspection observation TR-01 dated December 4, 1987, "Technical information concerning plant design changes and modifications is not being uniformly inccrporated into the nuclear training program on a timely basis." The SSFI report stated that l the same issue was raised by a recent INPO evaluation and that the l training material maintenance system would be modified. Management  :

directive TQ-01, Training and Qualification, identified the responsibilities to ensure training materials were complete and accurate and up to date before they were used for training. Implementing procedures for the management directive were to be included in the Fermi Training and Qualification Manual. Both of these corrective actions were to be complete by June 30, 1988. Documentation provided to the team indicated that at least portions of the corrective actions were not scheduled for completion until April 1989.'

(3) DER 88-0174 was written to document SSFI inspection observatidn MT-6 dated December 18, 1987, "There is no verifiable, controlled document to ensure proper and consistent settings of motor operated valve torque switches."

This particular OER was closed out on March 22, 1988 since it was a "duplication of the same problem identified in OER 87-565." DER 87-565 concerned torque switch settings that were not set in accordance with manufacturer's recomended settings. DER 87-565 was still "open" when the team concluded its evaluation. (See Sections 3.6.2.1 and 3.3.4 for additional information).

3.6.4 Equipment Challenges Based on interviews with senior Operations Department Management the team I developed a list stating what was most needed to be fixed in the plant to preclude equipment challenges. This list from the plant operations perspective was then compared with the Engineering modification list (Top 10, Nust and Want list). Approximately half of those items on the Operations generated list could not be found on the current Engineering modification lists. Some of these items may have been under active study but not yet characterized as ,

modifications on the modification lists. However, comunications between  !

Operations and Nuclear Engineering may still need improvement.

If material design deficiencies exist in the plant, Engineering needs to be made aware using appropriate communication channels. If deficiencies exist because of poor operations or maintenance practices, then these issues must also be resolved. Good basic teamwork will ultimately be required to fix the 100

3 plant equipment problems and to continue to improve the overall plant material condition, Consideration might be given to producing a combined physical problem list developed by a team consisting of one key individual from each department.

3.6.5 Engineering Modifications The team reviewed the modification implementation process to determine the quality and extent of engineering support provided to the plant. Engineering support can be provided from either Nuclear Engineering, Technical Engineering or Stone and Webster depending on the type of modification being performed. In general, Nuclear Engineering provided the front-end design work for minor modifications, Technical Engineering did the design work for temporary modifications and SWHI provided the front-end design work for major modifications. Results of the review are included below. ..

3.6.5.1 Design Change Procedures It was apparent that numerous procedures existed which were overly detailed, confusing to follow, and not well integrated. These procedures related to design control, the design change process, engineering design packages, engineering change requests, as-built notices, implementation of modifications, urgent modifications, minor modifications and temporary modifications. As an example, the two main procedures (recently revised in March 1988) used to perform modifications were POM 12.000.064, "Implementation of Modift:ation" and and inconsistent. P0M NOIP 11.000.004, 12.000.064 was used"Design Change Process,"

by the Maintenance and Modification were confusing (MdM) group while N0!P 11.000.004 was used mainly by Nuclear Engineering. NOIP 11.000.004 did not refer to P0M 12.000.064, and the cover sheet reference to NOIP 11.000.004 by P0M 12.000.064 appeared to be incorrect since it required work to be performed to original revision of NOIP 11.000.004. POM 12.000.64 required the Maintenance Support Technician (MST) of M&M to "initiate the necessary documentation per NOIP 11.000.004" whenever changes were required to a modification document during the implementation phase of the modificatian process. It was the understanding of the team that M&M did not use the formal process described in NOIP 11.000.004, but rather contacted Nuclear Engineering l

or SWMIDesign Potential directly to initiate Change Engineering (PDC) revisions or Design Engineering Package Change (EDP) revisions (ECRs).

Requests To be effective, working procedures need to be coordinated, explicit in their language, and must be followed and revised as policy or improvements are made.

Discussions held with the licensee indicated that the ongoing procedure rewrite effort was designed to significantly upgrade procedures and that the concerns noted would be corrected.

3.6.5.2 Temporary Modifications l The System Engineers had primary responsibility for temporary modifications. A l temporary modification is defined as a temporary minor alteration made to plant equipment that does not conform with approved drawings and design documents.

j These alterations are expectec to be installed for a short duration. Within seven days from the date that a modification is installed, a copy of the Temporary Hodification Request (TMR) is required to be sent to Nuclear Engineering for review. The results of the Nuclear Engineering review is to be returned to the System Engineer within 14 days.

l 101 i

If a temporary modification is to remain longer than 30 days, a renewal must be performed within 30 days of the initial installation. After the 30 day initial review is performed, subsequent April 1, July 1, and Octoberare 1) required.

periodicThe renewals once Team was per quarter concerned that the (January 1, Temporary Modification Procedure appeared to allow temporary modifications to remain installed for an indeterminate length of time. At some point, the licensee should convert the temporary modification into a permanent modification or remove the temporary modificat' ion. Such guidance was not provided in tP rocedure.

It appeared from *he number of temporary modifications designed by the Systems Engineers and installed in the plant and then subsequently revised or cancelled by Nuclear Engineering, that the System Engineers were performing modifications )

that would be better performed by Nuclear Engineering. The Systems Engineer I also failed to confer with Nuclear Engineering regarding various design constraints when required to do so by the temporary modification procedure.

Examples of installed temporary modifications which were rejected by Nuclear Engineering include the following:

(1) TMR 87-024 The Rod Worth Minimizer (RWM) was required by Technical Specifications to be operational at all reactor power levels below 20% core thermal power (CTP), and used signals from the feedwater control system to define the power level at which it is operational. During power ascension the RWM was observed to stay in operation (i.e., restricting rod movement by program) up'to 38% CTP. The plant personnel, via the temporary .

modification, simply lowered the setpoints on feedwater temperature causing the RWM to automatically bypass.itself. Upon review of the temporary modification, Nuclear Engineering rejected the modification since it left the flow setpoint instrumentation *.echnically out of calibration.

(2) TMR 87-026 l The Rod Sequence Control System (RSCS) was required by Technical Specifications to enforce its rod movement controls at reactor power i levels below 20% CTP, and served as a redundant functional back-up to the l RWM. The low power set point (LFSP) used to define the CTP at which RSCS I

was functional was taken from the Main Turbine First Stage Pressure.

On a particular power ascension with less than the normal number of feedwater heaters in service, RSCS remained enabled at about 35% CTP. The temporary modification picked a new setpoint based on prediction of turbine first stage pressure that would result at 23% CTP using the existing cold feedwater conditions, rather than normal plant operating conditions. Upon review of the temporary modification, Nuclear Engineering required that the setpoint be based on normal plant conditions. The optimum LPSP was selected by Nuclear Engineering based on calculations confirmed by start-up tests.

102

e (3) TMR 85-196 Shortly after fuel load the annunciator system was becoming overwhelmed by invalid alarms. These alarm inputs, due to process noise, were overloading the annunciator system and prevented real alarms from being processed. To fix this problem, a software scheme was developed that tested the validity of an alarm. To develop this software modification, it was.necessary to establish a communication (cable) link between the real time annunciator system and a temporary set-up in the engineering center.

The cable was installed in seismically qualified cable trays in the relay room. The seismic loading analysis that provided the basis for tray qualification was not modified and cable fill and separation controls were not enforced. Upon review of the modification, Nuclear Engineering rejected the temporary modification since the cable design proces's had not been used. It was noted that the temporary modification procedure required Nuclear Engineering concurrence prior to installation of temporary modifications, which address possible seismic or electrical s2paration criteria. This procedural requirement was not adhered to by the System Er.gineers.

A communication link was subsequently installed by Nuclear Engineering as part of a permanent design change.

3.6.5.3 Oversight of Modifications The licensee modification did noti.e.,

process p(rovide design, consistent engineering construction, support throughout testing and closecut). Once the the engineering design work had been completed for an EDP or PDC by either Nuclear Enr.ineering or Stone and Webster, the actual implementation, including necessary $ ork procedures to implement a modification, was performed by the M&M group, and was monitored by an Maintenance Support Technician (MST) who was part of the M&M group. The M&M group did not have a staff of engineers to follow modification work and Nuclear Engineering was rarely involved with field work of modifications unless an Engineering Change Request (ECR) or a revision to a modification package was requested. It would be a better practice to have engineers, either from Nuclear Engineering or some other group, following the implementation of modifications rather than using technicians.

The M&M group had some complaints concerning the quality of design work and adequacy of modification packages supplied to them. Many EDPs had to be revised or changed an excessive number of times which was indicative of the modification implementation inadequate preparation and/or followup phise by Engineering. PDCs had been revised during(as many as five times), EDPs had been revised (as many as seven times), and as many as 40 ECRs had been written against a single modification package. The format and information contained on an ECR also did not readily lend itself to auditing by Quality Assurance, or allow engineering to determine why an ECR had to be written such that future modification packages could be improved, thereby minimizie r engineering and M&M work or rework.

103

3.6.5,4 Modification Work Prioritization The modification work prioritization system was viewed as a strength by the team and should help to improve communication between the plant and engineering.

Fermi recently established a Nuclear Engineering / Nuclear Production priority work list for Engineering Modifications which was divided into four categories.

(1) Top 10; (2) Must modification thru first refueling; (3) Want modification thru first refueling; and (4) Engineering complete - work between now and end of first refueling. The list was a part of the engineering tracking system and was reviewed once a week by the Management Revie., Board (MRB). The MRB was made up of the Vice President Nuclear Operations; Vice President Nuclear Engineering and Services; Plant Manager; General Director, Nuclear Engineering; Superintdnden't, Technical Engineering; Superintendent, Maintenance and Modifications; and Supervisor, Planning and Scheduling. The function of the MRB was to approve / disapprove proposed modifications and to assign them to one of the above categories. The "Top 10" (actually consisted of 22 modifications), "Want" and "Engineering Complete" lists were categories of modification packages with no particular priorities established, while the Must list was prioritized. The prioritization of modification packages using the MRB concept allows the plant to communicate with engineering in the presence of appropriate plant management to assure that engineering efforts are focused in the proper areas to support the plant, not only for modifications that are currently "on the board," but also for additional needed plant modifications.

3.6.6 Preliminary Evaluations and 10 CFR 50.59 Safety Evaluations Fermi procedure FIP-SRI-01, Revis' ion 0 "Preliminary Evaluations and ,

10CFR50.59SafetyEvaluations"*estabIishestheprocessforaccomplishinga safety evaluation. This procedure became effective June 29, 1988, replaced two previous procedures and consists of a Preliminary Evaluation (PE) and a Safety Evaluation (SE) in one procedure. The PE was used as a screening device to determine if an SE would be required. The latest version of the PE required that the preparer provide justification if a decision was made to not perform an SE. The licensee had made recent improvements to their safety evaluation process, including both procedural changes, and extensive training for those who perform pes and SEs.

In the past, justificaticn for a decision not to perform a SE was not required.

Consequently, if a PE improperly concluded that an SE was not required, the SE would not be accomplished and a potential unreviewed safety question may have existed. This condition was further exacerbsted because QA audits were not done on pes for adequacy, as was the case for SEs. To review this area, the team chose four safety-related EDPs for review that had been "as-built" (field complete and affected documents revised) within the last 2-3 years: (1)

EDP-3289 - Revised the EDG fuel oil transfer system to include flow measurement capabilities; (2) EDP-3623 - Added a vent on i.he HPCI steam sup)1y line; (3)

EDP-4790 - Revised RHR systems P&lDs and valve information to slow miniflow valves in the normally open condition; and (4) EDP-6349 - Qualified a section of core spray piping to a higher pressure rating to comply with postulated (See Section 3.6.2.4 accident regarding conditions modificationand existing) wording in the UFSAR.

EDP-6349.

104

O A safety evaluation was not performed for any of the audited modification packages even through it appeared that they should nova. This finding was brought to the attention of the licensee persoanel, who upon review agreed '

that three of the modifications should have htad safety evaluations fi.e., EDPs 3289, 3623 and 4790). It was clear that safety evaluations should have been performed for EDPs 3289, 3623, and 4790 because they all required revisions to the UFSAR. 'Until recently, the licensee had not updated their 10 CFR 50.59 procedure (as other licensees have done beginring in 1984) to b,e consistent with NRC guidance. Licensee personnel agreed ?.o review a sample of safety-related EDPs to determine if a more widaspread problem existed with preliminary evaluations and to evaluate the potential for missed safety evaluations and potential unreviewed safety qu;:+ ions.

l' f

b e

105

4.0 EXIT MEETING The Deputy Executive Directo' for Regional Operations, Director, AEOD, Region III Administrator, Assistant Director NRR Projects (Acting), DET Manager, DET Deputy Manager, and other NRC personnel met with DECO managem9nt officials at the Fermi site on November 1, 1988, to brief them on the reselts of the Fermi diagnostic evaluation. The list of attendees is given at the end of this section.

The briefing which consisted of the team's preliminary findings and conclusions was led by L. Spessard, DET Manager. A copy of the briefing notes used during the meeting is included as Appendix A to the report. These notes were used for illustrative purposes and did not include all of the team's preliminary findings and conclusions in order to keep the briefing focused on the most important issues. Also, E. Jordan, Director, AE0D, discussed the NRC Diagnostic Evaluation Prot am and the NRC's basis for conducting a diagnostic at the Fermi plant.

The Deco response at the exit meeting, which was very receptive and positive, reinforced the team's preliminary findings and conclusions. According to R. Sylvia, Senior Vice President, many of the team's preliminary findings and conclusions paralleled those which INP0 had identified to Deco at their exit meeting the day before (October 31,1988).

l l

106

r-

~

y . ,

L ATTENDEES Fermi Diagnostic Evaluation Meeting - November 1, 1988 Name Organization j NRC J. A. Clifford Office of the Deputy Director for Regional Operations R. W. Cooper, III Region III, Division of Reactor Projects, Section Chief A. B. Davis Region III, Regional Administrator J. A. Gavula Region III, Division of Reactor Safety, Reactor Inspector E. L. Jordan Office for Analysis and Evaluation of Operational Data (AE00), Director W. D. Lanning Office of Nuclear Reactor Regulation (NRR), Deputy Diagnostic Team Manager T. R. Quay NRR, Project Manager for Fermi W. G. Rogers Region III, Fermi Senior Resident Inspector R. L. Spessard AE00, Diagnostic Team Manager J. M. Taylor Deputy Executive Director for Regional Operations M. J. Virgilio NRR, Acting Assistant Director for NRR Projects Deco S. G. Catola V,1ce President, Nuclear Engineering and Services G. Cranston General Director, Nuclear Engineering '

D. R. Gipson Plant Manager L. Goodman Director, Nuclear Licensing J. Lobbia President W. McCarthy Chairman and CEO W. S. Orser Vice President, Nuclear Operations R. B. Stafford Director, NQA and Plant Safety B. R. Sylvia Senior Vice President G. M. Trahey Director, Special Projects l

.. l 1

t 107

~

a G

APPENDIX A 0

e DETROIT EDISON /tiRC HEETING .

Ofl THE RESULTS OF THE FERMI DIAGNOSTIC EVALVATION NOVEMBER 1, 1988 I

A' HISTORICAL PERSPECTIVE ON THE NEED FOR A DIAGNOSTIC EVALUATION o FERMI WAS VIEWED AS READY AND FULLY CAPABLE TO. PERFORM WELL WHEN THE LICENSE WAS ISSUED o SIGNIFICANT OPERATIONAL EVENTS AND PROBLEMS AT FERMI RESULTED IN A QUICX AND SIGNIFICANT LESSENING OF NRC CONFIDENCE AND TRUST o NRC INSPECTIONS AND INDEPENDENT PEVIEWS IDENTIFIED SIGNIFICANY ORGANIZATIONAL PROBLEMS AND MANAGEMENT WEAKNESSES o PROGRAMS WERE DEVELOPED BY DECO FOR PERFORMANCE IMPROVEMENTS o FERMI PERFORMANCE HAS IMPROVED VERY SLOWLY ADDITIONAL INFORMATION WAS PEQUIRED TO FULLY UNDERSTAND AND EVALUATE CURRENT PERFORMANCE PICTURE, THE REASONS FOR CONTINUED BELOW AVERAGE PERFORMANCE AND,THE ADEQUACY OF CURRENT DECO ACTIONS AND PLANS 2

e

SUMMARY

OF FERMI DIAGNOSTIC FINDINGS o CURRENT LEVEL OF PERFORMANCE AND CAPABILITIES ARE BELOW AVERAGE AND SLOWLY IMPROVING o THE SUM OF Deco's ACTIONS GENERALLY ADDRESS CAUSES FOR PERFORMANCE PROBLEMS o ADDITIONAL MANAGEMENT ATTENTION NEEDED IN SOME AREAS TO INCREASE RATE OF IMPROVEMENT AND ASSURE CONTINUED SUCCESS ,

6 3

ROOT CAUSES o FAILURE OF MANAGEMENT TO ADEQUATELY AND EFFECTIVELY PLAN FOR

. THE TRANSITION FROM A DESIGN AND CONSTRUCTION PROJECT TO AN OPERATING PLANT o PROTRACTED DESIGN AND CONSTRUCTION PERIOD o MANAGEMENT MOVED VERY SLOWLY IN DETERMINING AND IMPLEMENTING EFFECTIVE SOLUTIONS o LACK 0F BWR OPERATING EXPERIENCE THROUGHOUT THE ORGANIZATION 1

l i

i 1

h I

4  :

CONTRIBUTING CAUSES r

PREVIOUSLY IDENTIFIED o LEADERSHIP WEAKNESSES o MANAGEMENT INEFFECTIVENESS o LACK OF INDIVIDUAL ACCOUNTABILITY o ENGINEERING ORGANIZATION DOMINANCE o LACK OF BWR OPERATING EXPERIENCE o FERMI ORGANIZATIONAL CULTURE ,

DIAGNOSTIC IDENTIFIED o ORGANIZATIONAL INSTABILITY o DIFFICULTY IN TRANSITIONING TO NEW CULTURE o LACK OF ATTENTION T.0 HUMAN RELATIONS MATTERS o UNRELIABLE EQUIPMENT

-- DESIGN

-- MAINTENANCE o INEFFECTIVE OPERATOR TRAINING PROGRAMS .

o FRAGMENTED AND OVERLAPPING ENGINEERING SUPPORT o COMMUNICATIONS ISSUES o INEFFECTIVE PLANNING AND SCHEDULING r

r I

l l

I f

1 5 4

-___________.I

DETROIT EDISON WAS SLOW TO FULLY APPRECIATE THE BREADTH AND IMPORTANCE OF THE PROBLEMS:

o EARLY PERFORMANCE PROBLEMS WERE MISTAKENLY ATTRIBUTED TO A LEARNING EXPERIENCE OF A NEW PLANT o Deco'S PREVIOUS DESIGN / CONSTRUCTION MANAGEMENT TEAM HAD DIFFICULTY IN APPRECIATING AND ACTING ON THE OPERATIONS-ORIENTED PROBLEMS o PREVIOUS Deco MANAGEMENT POLICIES AND PRACTICES MADE THE NEEDED ORGANIZATIONAL CULTURE AND CLIMATE CHANGES DIFFICULT TO CORRECT e

0 6

DETROIT EDISON, IN THE PAST FEW YEARS, HAS BEGUN TO MAKE PROGRESS Ill ADDRESSING THE CAUSES FDR FERMI'S PERFORMANCE PROBLEMS:

o NEW SENIOR MANAGERS WITH OPERATING POWER PLANT EXPERIENCE HAVE BEEN HIRED INTO KEY MANAGEMENT POSITIONS o NEW MANAGEMENT CONTROLS , PROGRAMS AND POLICIES HAVE BEEN IMPLEMENTED FOR IMPROVED ORGANIZATIONAL PERFORMANCE o ORGANIZATIONAL CHANGES HAVE BEEN MADE TO ENHANCE DEPARTHENTAL EFFECTIVENESS AND COMMUNICATIONS I

l 7

l NEW MANAGERS HIRED WITM COMMERCIAL NUCLEAR PLANT OPERATING EXPERIENCE:

SENIOR VICE PRESIDENT, NUCLEAR GENERATION (MAY 1986)

VICEPRESIDENTOFNUCLEAROPERATIONS(JUNE 1987)

VICE PRESIDENT NUCLEAR ENGINEERING AND SERVICES (MAY 1988) ,

FERMI PLANT MANAGER (SEPTEMBER 1987)

I DIRECTOR,NUCLEARQUALITYASSURANCEANDPLANTSAFETY(JANUARY 1988)

GENERAL DIRECTOR NUCLEAR ENGINEERING (SEPTEMBER 1988) ,

SUPERINTENDENTOFOPERATIONS(NOVEMBER 1986)

SUPERINTENDENT OF TECHNICAL ENGINEERING (JUNE 1988)

SUPERINTENDENT OF MAINTENANCE AND MODIFICATIONS (MAY 1988)

DIRECTOR, NUCLEAR LICENSING (JUNE 1988) k L

t t

P i.

8 i

. .,e n'

MANAGEMENT CONTROLS, PROGRAMS AND POLICIES FOR IMPROVED ORGANIZATIONAL PERFORMANCE ARE BEING IMPLEMENTED

!. o FERMI BUSINESS PLAN

-- MISSION GOALS, STRATEGIES AND ACTIONS

-- PERFORMANCE MONITORING, CONTROL AND EVALUATION ,

-- STAFFING AND SCHEDULING

-- ANNUAL liORK PLANS o ACCOUNTABILITY PROGRAM o INCENTIVE PAY PROGRAM o REVISED PERSONNEL POLICIES

- -- OVERTIME RESTRICTIONS '

-- . HIRING OUTSIDE EXPERIENCE

- -- BID-0UT RESTRICTIONS

-- DISCIPLINE STRENGTHENED o ADMINISTRATIVE PROCEDURES

-- REDUCTION IN NUMBER

-- STANDARDIZATION 9

., a THE NEW WORK ENVIRONMENT BROUGHT ABOUT BY THE PROF 0VND CHANGES IN 1

MANAGEMENT, ORGANIZATION, PERSONNEL POLICIES AND MANAGEMENT CONTROLS AND PRACTICES HAS HAD MIXED EFFECTS ON THE FERMI STAFF.

H MANY SUPPORT THE CHANGES:

o MORE PLANNING, ORGANIZING, MONITORING AND EVALUATION OF WORK o GREATER EMPHASIS ON TECHNICAL COMPETENCE o GREATER FOCUS ON REACTIVE PROBLEM SOLVING o CLOSER MANAGEMENT ATTENTION AND FOLLOWUP o INCREAF.EC INDIVIDUAL RESPONSIBILITY AND ACCOUNTABILITY (E.G.,

OPERATIONS) l l o GREATER' FOCUS ON OPERATIONAL NEEDS

! o IMPROVING INTERDEPARTMENTAL COMMUNICATION AND COOPERATION I

SOME HAVE REACTED NEGATIVELY:

o INCREASED FEAR ABOUT LOSS OF JOB OR DEAD ENDED CAREER PATHS o LOWERED MORALE o NEGATIVE ATTITUDES TOWARD MANAGEMENT o REDUCED IMITIATIVE 10

rrc ..

.o ,- .. .

'e Lo- t SLOW PROGRESS IS BEING MADE OPERATIONS

. , . , ' IMPROVEMENTS ,

o BETTER OPERATOR AWARENESS OF MANAGEMENT EXPECTATIONS

- - OPERATING PRACTICE STANDARDS

-- ACCOUNTABILITY PROGRAM ,

WEAKNESSES

,o LACK OF COMMERCIAL BWR OPERATING EXPERIENCE AT.A WELL RUN PLANT o LACK A SUFFICIENTLY BROAD SAFETY PERSPECTFVE o EQUIPMENT UNRELIABILITY CHALLENGES OPERATORS o INEFFECTIVE OPERATOR TRAINING PROGRAM 0

11 t

! ^*r s

i.

r

! SLOW PROGRESS IS BEING MADE I

MAINTENANCE

' IMPR0"O'i.':T5 o DECREASED CM BACKLOG o IMPROVED' MAINTENANCE /0PERATIONS COMMUNICATIONS WEAKNESSES l o STRAINED RESOURCES / STAFF

, o INEFFECTIVE PLANNING AND SCHEDULING .

o INEFFECTIVE UTILIZATION OF CONTRACTOR SUPPORT ,

l o LALK OF SPARE PARTS o PM PROGRAM DOES NOT PROMOTE EQUIPMENT RELIABILITY AND f CONTRIBUTES TO SAFETY SYSTEM UNAVAILABILITY o INEFFECTIVE TRENDING OF EQUIPMENT FAILURES o INADEQUATE MOV TORQUE AND LIMIT SWITCH SETTING CONTROLS 12

e g ',

' SLOW PROGRESS IS BEING.MADE SURVEILLA!!CE AND TESTING .

IMPROVEMENTS o TECHNICAL SPECIFICATION IMPROVEMENT PROGRAM o SURVEILLANCE SCHEDULING AND TRACKING PROGRAM

WEAKNESSES o TESTING PROGRANS LIMITED TO SECTION XI AND TECHNICAL SPECIFICATION REQUIREMENTS

, 6 e O

I 4

e 13 ,

SLOW PROGRESS IS BEING MADE k'

ENGINEERING SUPPORT IMPROVEMENTS

\'

. o COMMUNICATIONS AND COOPERATION WITH PLANT

? l o MODIFICATION PACKAGE PRIORITIZATION

. l WEAKNESSES l

o INEFFECTIVE MANAGEMENT LEADERSHIP AND DIRECTION e.

o ORGANIZATIONAL INSTABILITY o FRAGMENTED AND OVERLAPPING ENGINEERING SUPPORT c STRAINED RESOURCES / STAFF o INADEQUATE MOV TORQUE AND LIMIT SWITCH SETT!hG CONTROLS o TIMELINESS OF FIXING KNOWN PROBLEMS REQUIRING DESIGN ATTENTION 14

o SLOW PROGRESS IS BEING MADE OUALITY PROGRAMS IMPROVEMENTS o BECOMING PERFORMANCE ORIENTED o F'INDINGS ARE USED BY THE FERMI LINE OTGANi?ATION o SAFETY COMMITTEES (8NRC & NSRG) ARE PRGACTIVE WEAKNESSES o INEFFICIENCIES IN DER TRENDING PROGRAM o PROCEDURAL WEAKNESSES IN DER TRACKING SYSTEM o ROOT CAUSE ANALYSIS 4

4 I

15

c .

b

.i ISSUES REQUIRING ADDITIONAL MANAGEMENT ATTENTION 4

- o. NEED TO ACHIEVE ORGANIZATIONAL STABILITY ASAP 3

o 1 1 ROVE EFFECTIVENESS OF FIRST AND SECOND LINE SUPERVISORS o .. ^ ~0VE ORGANIZATIONAL CLIMATE o'

FIX FRAGMENTED AND OVERLAPPING ENGINEERING SUPPORT RESPONSIBILITIES o FIX KNOWN EQUIPMENT PROBLEMS o SET PRIORITIES ACCORDING TO PLANT NEEDS o ALLOCATE RESOURCES TO SELECTED AREAS AND BETTER UTILIZE EXISTING RESOURCES' o IMPPOVE EFFECTIVENESS OF OPERATOR TRAINING PROGRAMS 16

o AFFEN01X b J'f%q% UNITED 5TATES

[ ~% NUCLEAR REGULATORY COMMISSION B WASHINGTON. O, C. 2oSSS

• ., .f

%, * . . . . /

JUL 2 61988 MEM0'RANDUM FOR: Edward L. Jordan, Director Office for Analysis and Evaluai. ion of Operational Data FROM: Victor Stello, Jr.

Executive Director for Operations AUBJECT: DIAGNOSTIC EVALUATION OF THE ENRICO FERMI

., ATOMIC POWER PLANT , .

By this memorandum you are directed to conduct a diagncstic evaluation at the Enrico Fermi Atomic Power Plant. I have reviewed and approved your plans, as summarized below: -

Schedule of Principal Activities

. Team Freparation -

August 4 August 19, 1988

. Onsite Activities -

August 22 - September 2, 1988 i September 12 - September 16, 1988

. NRC Management Briefing -

October 4, 1988

. Exit Meeting with Licensee -

October 7, 1988

. Issue Evaluation Report -

October 28, 1988 t Team Organization

• Na.ne Oraanization Team Manager R. Lee Spessard Director, DOA/AEOD Deputy Team Manager To Be Determined Team Leaders Henry Bailey DE!!B/00A/AE00 9ennis Allison DEIIB/00A/AE00 Team Members James Bongarra LHFB/NRR Charles Burger DRS/RII Arthur Howell DRP/RIV Robert Freeman DEIIB/DOA/AE00 Eric Leeds DEIIB/00A/AE00 Ronald Lloyd DEIIB/00A/AE0D Robert Perch DEIIB/DOA/AEOD Kevin Wolley DE!!B/00A/AE00 Franc.is Young DRP/RI Paul Thurmond NRC Contractor Henry Tufts NRC Contractor Donald Tepper NRC Contractor "Additional members r.ay be added in the near fu*ure at your discretion as the evaluation proceeds.

Edward L. Jordan o ,

Evaluation Methodology The 01 agnostic Evaluation Team (OET) will ascertain the current status of plant performance in the functional areas of engineering and technical support, operations, maintenance, testing, quality assurance, and management controls i and involvement through the performance of observations, interviews and ,

document reviews. The evaluation will consider activities conducted at the corporate headq' carters as well as at the plant site. If significant problems are noted, emphasis will be placed on determining the ,

• cause(s). As .

necessary, the evaluation process will progress from tt identification of  :

problems, proxin. ate causes and related programmatic issues to the considera- ,

tion of management and organizational strengths and weaknesses. t Following the onsite evaluation activities, the DET will prepare an evaluation report for submittal to me in accordance Nith NRC Manual Chapter 0520, "NRC Diagnostic Evaluation Program."

/

' /[T2/

ictor 'Stel'lo,/h'r. i i Executive Direttor for Operations  ;

cc: A. B Davis, RIII T. E. Murley, NRR

J. M. Taylor, E00 .

, i d

1

)

l l

l  :

i i i I

i l

e V

l I

i i' i

l l

.__ _ _ _ _ _ _ _ ----______- _ _ -, _ ____ _ ____---~- _ . _ _ _ . . _ _ _ _ _ _ - - _ . _ _ _f