Text

Summary of 880406 Meeting W/Numarc,Utils,Science Applications,Inc,Epri,Sandia,Trident Engineering & Devonrue in Rockville,Md Re Emergency Diesel Generator Reliability Programs
	ML20148S481
Person / Time
Issue date:	04/12/1988
From:	Serkiz A; NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
To:	Kniel K; NRC OFFICE OF NUCLEAR REGULATORY RESEARCH (RES)
References
	REF-GTECI-A-44, REF-GTECI-B-56, REF-GTECI-EL, RTR-NUREG-CR-5078, TASK-A-44, TASK-B-56, TASK-OR NUDOCS 8804180463
	Download: ML20148S481 (157)
	v • d • e

. S p2 te eg'o

+ UNITED STATES b f " g' 'r ~ ~ ,,g NUCLEAR REGULATORY COMMISSION

i

, l WA52GTON, D C. 20M6

% ,,',,, /

sa==

MEMORANDUM FOR: Karl Kniel Chief Reactor and Plant Safety Issues Branch Division of Reactor and Plant Systems, RES FROM: Aleck W. Serkiz, Senior Task Manager Reactor and Plant Safety Issues Branch Division of Reactor and Plant Systems, RES

SUBJECT:

SUMMARY

OF MEETING WITH NUNARC STAFF ON DIESEL GENERATOR RELIABILITY PROGRAMS Meeting Date: April 6,1988 Location: U.S. NRC 5650 Nicholson Lane, Rm. NL/S-013 Rockville, Maryland 20852 .'

Purpose of Meeting: The purpose of this meeting was to provide NUMARCl an opportunity to discuss their views on reliabili~ty programs for emergency diesel generators (EDGs).

Attendees: See attached attendees list.

A. Marion (NUMARC) discussed the formation of a B-56 working group to interact with the NRC staff during the resolution of GSI B-56, "Diesel Generator Reliability," in a manner similar to the resolution of USI A-44. .Marion utilized Enclosure 1 to describe NUMARC's views and activities prior to this meeting. Enclosure 2 was utilized in discussions related to the importance of assignment of responsibilities and management controls in the implemention of an effective reliability program.

NUMARC staff noted that many plants appear to have an effective EDG reliability program since high levels of EDG reliability (i.e. 98%) are being reported through INP0's tracking program.

The staff informed NUMARC of the goal to resolve B-56 in FY 1988, described near tenn activities related to revising regulatory guides. SRPs and the development of an inspection module, and noted to NUMARC ; hat their activities need to coincide with our schedule. In response to NUMARC's request for

%~ 'p h

gugg $3Tgc \

Nhpgy 4

is m as !

K. Kniel 2- '

I additional meetings in April 1988, the staff recrevnended they review the !

e j

contents of NUREC/CR-5078 prior to future meetings so that substantive discussions could be held. A advance release copy of NUREG/CR-5078 was given '

to NUMARC at the conclusion of this meeting. This report has been sent to publications recently for printing and will be issued soon. ,

A. Marion will contact me to establish when another meeting is warrented.

Aleck W. Serkiz, enior Task Manager ;

J Reactor and Plant Safety Issues Branch Division of Reactor and Plant Systems Office of Nuclear Regulatory Research

Enclosures:

As stated cc: E. Beckjord

T. Speis .- l B. Sheron *

Attendees ,

, t PDR(w/copyofEnclosure3 report)P '

j -

i i

4 i

1 f

9 i

)

J l

i i

I

. 0-i i

l Attendees !

April 6, 1988 Meeting l NUMARC Briefing on EDG Reliability Programs .

l k

_NAME ORGANIZATION TELEPHONE NOS.

i A. Marion NUMARC (202)-872-1280 l S. Floyd CP&L/NUGSB0 (919)-836-6901 M. McGarry BCP&R/NUGSB0 (202)-371-5733 i M. L. Childers NU/NUGSB0 (203)-665-5949 l E. V. Lofgren SAIC (703)-821-4492

0. M. Chopra NRR/SELB (301)-492-0835 4 H. L. Wyckoff EPRI (415)-855-2393 S. N. Saba NRR/SELB (301)-492-1052 {

D. Tondi NRR/SELB (301)-492-0804 A. Serkiz RES/RPSIB (301)-4923555 2 l

A. C. Payne, Jr. SNL (505),-846-3568

'A. Notafrancesco NRR/PSB (361)'-492-1062 R. Colmar NRR/ILRB/PMAS (301)-492-3076 ,{

Harry Krug NRR/ILRB/PMAS (301)-492-3073 .,

W. B Henderson Trident Engineering (301)-267-8128 -

i C. S. Ondash Devonrue (617)-426-4556 Ca'ri Johnson NRR/RES (301)-492-8311 W. Minners NRR/RES ('301)-492-3151 I i

^

t i

?

I

m !

n

)

s .

ENCLOSURE 1 USED AT THE 4-6-89 .

NUMARC/NRC STAFF MTG >

EDG RELIABILITY REF. B-56 !

l I. TARGET RELIABILITY

[

A. ESTABLISH TARGET l B. ASSESS ACTUAL RELIABILITY I f C. MAINTAIN TARGET RELIABILITY

!!. GUIDELINE ELEMENTS A. RELIABILITY TARGETS l

B. SURVEILLANCE TESTING AND RELIABILITY MONITORING C. MAINTENANCE PROGRAM i

D. DOCUMENTATION i

E. MANAGEMENT OVERSIGHT AND ORGANIZATIONAL RESPONSIBILITY .

~

III. GUIDANCE DOCUMENTS A. REGULATORY GUIDES .

' ^

B. NUMARC 87-00 l

t .

l n..

h 9

l

~

ENCLOSURE 2 USED CT THE 4-6-C]

NUMARC/NRC STAFF MTG REF. B-56 12-15-87 A.W. Serkiz SERKIZ DVERV!EW OF PR1NCIPAL ELEMENTS EDG Reliability Program i

, l__ _ ___________g : ________ _ _t i ! License ! A EDG Target !,

! Requi r ement s : _

r t Reliability .,

l___ l g_-=___---- l t 0.95.0.975 !

____ t i f I __ _____

Responsibility I

l & Management W Controls I i_________._____ g

! : _1_______i i N______

veillance '

I Maintenance i Program i D Program I

.. ____3 : ;.

i E

_ __ _ _ _ _ _ y .

, 1- ----- -1 1 Data 1 1 - -- ^ -1 Per f ormance ! ! Collection : 1 Failure & !

O Monttoring r ! & Utilization i 1 Root Cause !

W - -------- - t t - - - --- ----- ! El Evaluation !

g ___________l .

l f -

I Problem I p Close-out i

. _________g i _____ .

I Operating

! Experaance

=1 from other

{ '!IP1 same ants w/Mfg EDG j .

9

-4 I

r NUREG/CR 5072 SAND 87-7176 Vol.1 A Reliability Program "or Emergency Jiese Generators at 3 ants

\ uc ear 3ower i

Program Structure :

I l

Manuscr et Comotetea: February 1988 Cate Published: J.iuen 1988 Preparea by DRAFT E.V. Lofgren, G.M. DeMoss J.R. Fragoia.

P L. Aopignani, G. Delarcne', J, Boccio" 4

- Science Applications international Corporation .

1710 Goocridge Drive McLean, VA 22102 ADVANCE Uncer Contract to: R E L.E A S E COPY Sancia National Laboratories -

Albucuercue, NM 87185 "ricent Eag:neenng l "Orconne.en National Laboratcry FOR USE BY B-56 PROJECT STAFF Prepared for Division of Reactor and Plant Systems Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555 i NRC FIN A1806 ENCLOSURE 3 PROVIDED To Nt4 MARC AT THE 4-6-86 MTG REF. B-56

NUREG/CR 5078 SAND 87-7176 Vol.1 A 9eliaaiity 3rogram "or Emergency Diese Generators at

\ uc ear 3 0wer 3iants Program Structure

ianuscnet Ccrnpintec: February 1988 Cate Publishec: h 1988 Preparea by DRAFT E.V. Lofgren, G.M. DeMoss, J.R. Fragola.

P L. Aopignani. G. Delarene'. J. Boccio" Science Apphcatiers International Corporation 1710 Goocndge Drive McLean, VA 22102 ADVANCE Under Contract to: RELEASE Sanc.a National Laboratones COPY Albuqueraue. NM 87185

'Tncent Eng:ntenng "Brecanasen National Laboratory FOR LJ S E BY s-56 PROJECT STAFF Prepared for Division of Reactor and Plant Systems Office of Nuclear Regulatory Research U.S. Nuclear Regulatory Commission Washington, DC 20555 NRC FIN A1806

i t

! 'l l r j ,

1 !

FORDORh i l

)

]

a PREPARED BY THE NRC STAFF t

i j This report provides insights into the principal elements of an emergency diesel generator (EDG) reliability program which have been derived from applications where high levels of reliability must be achieved and ;

i i

maintained. This report was prepared by Science Applications International i Corporation for use by NRC staff in connection with the resolution of I

! Generic Safety Issue B 56, "Diesel Reliability". This report therefore provides technical guidelines to NRC staff for use in evaluating emergency I

diesel reliability programs that may have to be reviewed in the future. It !

I should be clearly noted that the findings and recommendations provided in !

l this report are those of the contractor and contributing authors and do not l constitute regulatory positions or requirements. '

i :

j ABSTRACT l i !

The purpose of this report is to provide technical guidelines for NRC staff use in the development of positions j generator (EDG) reliability programs. Such for evaluating emergency diesel !

reviews will itkely result 1

i following resolution of US! A 44 and GSI 8-56. The diesel generator ;

2 reliability program is a management system for achieving and maintaining a selected (or target) level of reliability. This can be achieved by: !

(1) -

understanding the factors that control the EDG reliability and then '

4 applying reliability and maintenance techniques in the proper propor(2) tion to

)

achieve selected performance goals. The concepts and guidelines discussed i

in this report are concepts and approaches that have been successful in ;

applications where high levels of reliability must be maintained. i Both an EDG reliability program process and a set of review items for NRC i

use are provided.

reliability t*ograms. The review items represent a checklist for reviewing EDG .

program. Rather, They do r.ot, in themselves, constitute a reliability !'

the review items are those distinctive features of a reliability program that must be present for the program to be effective.

! i

! l l

)

1 i

l

)

1

4 l

l CONTENTS PELE FOREWARD ....................................................... iii ABSTRACT........................................................ iii C0NTENTS........................................................ v FICURES........................................................ vi i TABLES.......................................................... vi !

ACKNOWLEDGMENTS................................................. vii I

1.0 INTRODUCTION

.............................................. 1-1 1.1 Purpose of Document ....... ...............

1 1-1 1.2 EDG Mission ..........................................

........... 1-1 1.3 Diesel Generator Reliability Target .................. 1-1 1.4 EDG Reliability Program 0verview...................... 1-2 2.0 DIESEL GENERATOR RELIABILITY PROGRAM TEMPLATE ............. 2-1 I

\

3.0 REVIEW ITEMS DISCUSSION................................... 31 3.1 Review item A: EDG Reliability Target...............

3.2 Review Item B: 31 3.3 Review Item C: EDG Surveillance Needs .............. 3-1 EDG Performance Monitoring .........................................

3.4 Review Item D: 36 EDG Ma intenance Program. . . . . . . . . . . . . . . 3-6 3.5 Review Item E: EDG Failure Analysis and Root Cause Investigation .......,...................

3.6 Review item F: 37 Problem Closecut................... 39 3.7 Review Item G: Data System........................... ... 3 10 3.8 Review Item H: Responsibilities and Management Controls................................. 3-11 REFERENCES...... ............................................... R-1 l GL0SSARY........................................................ G-1 APPENDIX A - EMERGENCY DIESEL GENERATOR RELIABILITY TARGET A1 APPENDIX B - EDG SURVEILLANCE NEEDS ............................ .... B-1

)

APPENDIX C - EDG PERFORMANCE MONITORING ........................ C-1 APPENDIX D - EDG MAINTENANCE PROGRAM ......................... . . D1 APPENDIX E - FAILURE ANALYSIS AND IDENTIFICATION OF CORRECTABLE CAUSES.......................................... E1 l APPENDIX F - PROBLEM APPENDIX G - DATA CLOSE0VT.................... .............. F-1 SYSTEM......................... ............., G-1 l

APPENDIX H - RESPONSIBILITIES AND MANAGEMENT CONTROLS H-1..........

APPENDIX I - ENERGENCY DIESEL GENERATOR DATA REVIEW ............ 11 v

i l

j

LIST OF FIGURES Figure 2 1. Reliability Program Process .....................

Figure 3-1. 22 Top level Work Breakdown Structure Showing Tasks Necessary to Provide & Reliability Focus for Figure 3 2. Surveillance ................................

Systematic Root Cause Approach ................. 32 38 LIST OF TABLES i Table 1-1. Diesel Generator Reliability Program l

i Table 3-1.

Review Items..................................

Definition of Diesel Subsystems. . . . . . . . . . . . . . . . . .

13 3-3 l

l l

I l

l i

1 I

l I i

l l l l

l vi 1 \

s.

ACKNOWLEDGHENTS The authors thank Arthur Payne of Sandia National Laboratories, Lawrence Kripps of El International, Ron Battle of Oak Ridge National Laboratory, and Carl Johnson of the U.S. Nuclear " gulatory Commission for their many constructive comments cnd valuable insights in review of this documentt Wallace Norris of the U.S. Nuclear Repolatory Commission expedited the acquisition of the Nuclear Plant Reliabi1 Ety Data System data. The authors also thank Aleck Serkiz of the U ', Nuclear Regulatory Commission for his leadership document, and foresight in initiating and guiding the development of this 1

i I

l vii

1. INTRODUCTION This B-56, report "Diesel was prepared to support the resolution of Generic Safety Issue Reliability." B-56 is a generic safety issue (GSI) related to the Unresolved Safety Issue (USI) A-44, "Station Blackout." The resolution of USI A-44 establishes a need for an emergency diesel generator (EDG) reliability program that has the capability to achieve and maintain EDG reliability levels in the range of 0.95, or better. Regulatory Guide 1.155, "Station Blackout," provides guidance for assessing EDG reliability levels and implementing a reliability program to meet the requirements of 10 CFR Part 50, Section 50.63, "Loss of All Alternating Current Power."

This report describes in Regtlatory Guide 1.155. an EDG reliability program that is consistent with guidance 1.1 Purpose of Document The purpose of this report is to develop the major elements of an EDG relia-bility program and to maintain EDG reliability levels at 0.95, or better.

Therefore, a reliability progri.ni becotes a structured approach to concepts integrate and approaches that have been shown successful in applications where high levels of reliability should be maintained. This can be achieved by:

(1) understanding the factors that lead to EDG failures, (2) applying reliability monitoring and maintenance techniques in proper proportion to achieve reliability targets, and (3) providing a structured approach for closing out problems encountered and for their avoidance in the future.

1.2 EDG Mission The primary mission of EDGs at nuclear power plants is to provide highly reliable ac power to safety-related systems in the event offsite ac power sources are not available. To accomplish this, EDGs (normally in the stand-by mode) of time, should start and carry electrical loads within a specified period depending on the accident or event, and continue to provide ac power until offsite power is restored.

1.3 Diesel Generator Reliability Taroet The minimum EDG reliability level should be targeted at 0.95 per demand for each EDG for plants in emergency ac (EAC) Groups A, B, and C, and at 0.975 per demand for each EDG for plants in EAC Group D, as defined in Regulatory Guide 1.155 (Ref. 1). These reliability levels should be coasidered minimum target reliabi'ities and each plant should have an EDG reliability program containing the principal elements, or their equivalent, Section 1.2 of Regulatory Guide 1.155. The principal elements as outlined in reliability of an EDG Regulatory Guide 1.155. program should be consistent with guidance provided in 1-1

_ _ _ - - - - - _ - _ - - - - - - _ i

1.4 EDG Reliability Procram Overview l i

A diesel generator reliability program is a management system for managing diesel generator reliability. The rules and procedures that flow from the management system are all based on a consistent philosophy, which states that a specified reliability target can be achieved by understanding the factors that drive a diesel generator's reliability, and then applying reliability and engineering techniques in sufficient depth to ensure that the target is reached.

Table 1-1 provides an overview of the review items of an EDG reliability program that are consistent with Regulatory Guide 1.155. Items A through H represent program elements that should be developed in the preparation of an EDG reliability program. These items represent the necessary considerations ;

in the development of a reliability program designed to sustain the relia- l bility levels needed for EDGs at nuclear power plants. l Therefore, these items provide a checklist for assessing EDG reliability programs. However, these items, in themselves, do not constitute a reliability program process. The reliability program process that is .

t developed in Section 2 of this report and detailed in the appendices can be used in assessments of reliability programs.

Further, this report is not intended to establish unconditional and specific requirements. Rather, this report provides a reliability program approach consistent with current l experience and findings from other reliability program applications where !

sustaining a high level of reliability is essential.

i Section 2 also presents the relationships between these items

\

to l as review items) and the logic of a reliability program process (referred

. Section 3 provides summary definitions of each of the review itams. Appendices A i through H, corresponding to the items identified in Table 1-1, i provide i further insights into considerations, examples, and guidance for development of such elements. Appendix 1 was included for identification of EDG failure modes encountered for EDG subsystems and for the various manufacturers.

This appendix can be used as a supplemental aid in developing an EDG relia-bility program. However, Appendix I is not to be used to draw broad conclu-sions. The variability of failures, underlying causes, and historic trends cannot be properly extracted from Appendix I without reviewing the reported details for each event (which in many instances was sparse or lacking).

l l

1-2

l 1

l TABLE 1-1 DIESEL GENERATOR RELIABILITY PROGRAM REVIEW ITEMS A. EDG Reliability Target Ensure that the reliability target for the diesel generator has been established and that calculational measures have been defined that can be eval'uated and compared to the target.

B. EDG Surveillance Needs Ensure that the diesel generator equipment boundary has been defined and that the diesel generator reliability program has specified a task for analyzing the surveillance needs of this equipment I C. EDG Performance Monitoring Ensure that the reliability program specifies a task to monitor diesel ;

generator performance, using both statistical trending and engineering data, to spot dearadations in performance.

D. EDG Maintenance Program Ensure that the diesel generator maintenance program has a reliability focus that includes preventive maintenance, prioritization of maintenance actions and spare parts considerations.

E. EDG Failure Analysis and Root Cause Investigation Ensure that there is a task to systematically reduce identified diesel generator problems to correctable causes.

F. Problem Closeout Ensure that the diesel generator reliability program requires a formal problem closeout procedure and that this procedure involves both ,

1) '

establishing criteria for problem closecut when a reliability problem (is

detected, and (2) providing for any special monitoring activity that the criteria have been satisfied by the corrective action. to ensure G. Data System l l

Ensure that a data gathering, storage, and retrieval system with sufficient capabilities to support all features of the reliability program is in place or will be implemented as part of the diesel generator reliability program.

H. Responsibilities and Management Controls Ensure that there are clear line responsibilities and management controls in place that identify respouible individuals for implementing and operating the diesel generator reliability program, and ensure that these individuals are qualified to perform the functions for which they are responsible.

1-3

a

2. DIESEL GENERATOR RELIABILITY PROGRAM TEMPLATE The important elements necessary for successful operation of a diesel generator reliability program are depicted in Figure 2-1, although these elements could be combined in ways alternative to Figure 2-1. However, the reliability program should be a closed loop process with the following characteristics:

e A structured approached to problem detection. .A direct means of problem detection is monitoring EDG performance and comparing it to a reliability target. Other important prob-lem detection means come from diesel generator condition monitoring and reliability engineering (e.g., performance monitoring, operating experience) techniques.

e A aeans for aroblem orioritization and correction. The prioritization should take into account problem severity and impact on EDG reliability; the problem correction should take into account the priority and include, when appro-priate, failure and root cause analysis, e A formal problem closecut. The corrective action should be verified and the problem closed out by monitoring the EDG in order to ensure that the problem has been effectively corrected.

A brief discussion of each of the essential elements (as identified in Figure 2-1) of a diesel generator reliability program and how they relate to !

the review items identified in Section 1 is provided next. Section 3 follows with a detailed discussion of each review item. Reference 2 provides a more detailed discussion of the reliability program process.

Monitor Diesel Generator Reliability Performance This element encompasses both diesel reliability monitoring and condition monitoring that are required by Review Item C (EDG Performance Mcnitoring). Reliability monitoring refers to the direct tracking of diesel generator failure frequency and down-time and to tracking of characteristics that are related to failure frequency and downtime such as severity and cause of failure. Condition monitoring refers to tracking predictive conditions that are associated with diesel generator failure modes, e.g., moisture in the air start system or excessive vibra-tion, temperature, or pressure, i

Performance monitoring is accomplished by using information ;

obtained from diesel generator surveillance. For the purpose of this document, "surveillance" refers to any purposeful act to l obtain information concerning the operational readiness o', or deterioration of, the diesel generators. It includes demand testing, partial demand testing, walkaround inspection, teardown inspection, and condition monitoring. visual and The I

2-1 !

relia bility data pe fo ance

~ ~ ~ ~ ~ ~ ~ ~ - - ~

re abiiy rel a I ty

,m hMp aM pe rf orma n ce to targets ope ra tion s offectiveness k or corrective l l l

a ction

[____________________________________

verified y' problems Verify Assess problem corrective priority and A, a ction schedule pro blem ef f ective ness analysis A N impodant co rrect_ve i

problems action implemented corrective '8880" action IO' found problem s 3 f Implement Determine Identify corrective corrective problem a ctio n action cause FIGURE 2-1. RELIABILITY PROGRAM PROCESS

determination of appropriate surveillance needs is addressed by Review Item B (EDG Surveillance Needs).

Compare Diesel Generator Performance to Tarcets This element provides for the periodic assessment of diesel generator performance by comparing the. actual performance, as estimated in the first-element described above, to alert levels related to the diesel generator reliability target. Included in this element is comparison of observed degraded or incipient conditions to alert levels for these conditions. The alert levels should be set to ensure reliability levels of Regulatory Guide 1.155. They need to be quantitatively related directly to the diesel generator reliability target but should be consistent with this target level. Alert levels for NRC use and suggested actions are addressed in Review Item A (EDG Reliability Targets).

Evaluate Diesel Generator Reliability Related to Desian and Operation This element consists of the evaluation of the design and opera-tion of the diesel generator to determine if conditions exist that may result in unreliable operation or deterioration of the diesel generator. This element could uncover potential relia-bility concerns before they manifest themselves in deterioration of the diesel generators. Specific reliability techniques for accomplishing this element include design review to identify failure modes using techniques such as failure modes and effects analysis (FMEA) or fault trees, analysis of data collected through performance monitoring such as condition monitoring data, and analysis of other data sources such as maintenance records, Nuclear Plant Reliability Data System (NPRDS) reports, licensee events reports (LERs), inservice inspection testing results, etc.

The major thrust of this element is the evaluation of EDG design and operations in order to (1) identi fy design problems so they can be corrected and (2) identify specific surveillance issues (Review Item B) related to design (e.g.,

common cause, system interaction). The work in this element also drives aspects of Review Item C (Performance Monitoring).

Assess the Priority of Diesel Generator Investications and Corrective Actions This element provides for prioritizing maintenance actions on repair of noncatastrophic diesel generator failures and condi-tions (catastrophic diesel generator failures should be repaired when they are detected). The prioritization should account for the mean time to catastrophic failure given the observed condi-tion, the cutage time required for repair, and collateral damage that could result if the observed condition progressed to a catastrophic diesel generator failure. This prioritization 2-3

creates a reliability center to the maintenance actions addressed by Review Item D (EDG Maintenance Program).

Diesel Generator Failure and Root Cause Analysis This element provides for a failure investigation that can ultimately lead to a root cause analysis of diesel generator reliability problems, including assessment of when to apply root cause investigations. Root cause of failure or maintenance un-availability for diesel generator components can usually be attributed to one of the following broad areas:

e Design, manufacturing / construction inadequacy e Operating procedures inadequacy e Maintenance activities (scheduled, forced) e Environmental stress.

Determination of appropriate corrective actions can only be accomplished when the problem root cause has been identified.

Determine Corrective Action: Implement Corrective Action These two diesel generator reliability program elements are largely engineering-related activities needed to complete the reliability program process. They are currently performed at all ,

plants. l Verify Diesel Generator Corrective Action Effectiveness This element should show that the corrective action implemented was effective in correcting the diesel generator reliability problem. Two steps are necessary. First, identify criteria that would success.

be satisfied if the corrective action is to be counted as a This is done before the decision regarding corrective action effectiveness is made. Second, monitor the diesel genera- ,

tor performance to ensure that the actual performance meets the criteria. This element corresponds to Review Item F (Problem Closecut).

Implicit in the EDG reliability program depicted by Figure 2-1 are elements that should be present to support the process. These supporting elements I include (1) the existence of an EDG data base and data management system (Review Item G, "Data System") and (2) assignment of responsibilities and existence of management controls to ensure that the reliability program l process is adequately managed (Review Item H, "Responsibilities and Manage-ment Controls").

2-4

3. REVIEW ITEMS DISCUSSION The review items summarize the features that should be present for a successful diesel generator reliability program, that is, one that will provide assurance that the diesel generator reliability target will continue to be met over the plant lifetime. The review items were listed in Table 1-

1. A brief discussion of each of the review items is presented in the following subsections.

3.1 Review Item A: EDG Reliability Tarcet The reliability target for individual diesel generators has been estab-lished, ing on as part of the resolution of USI A-44, to be 0.95 or 0.975, depend-the plant-specific emergency ac power system (see Ref. 1). This target reliaoility is to be interpreted in the following way:

o The target is to be interpreted as an average value over a specified base-time or number of demands.

o The number of demands are to include actual demands for the diesel generator systems' function and demand tests of the system that involve an attempted start and run.

o Both failures included in to start and failures to run are to be the calculation of diesel generator reliability, o Diesel generator failures that are recovered with a success-ful start and load within 5 minutes are not to be counted as failures.

Appendix A presents a more detailed discussion of the diesel generator reliability target and how that target is to be estimated. Appendix A also i contains an EDG failure evaluation criterion for judging the acceptability l of EDG failure histories. !

3.2 Review Item B: EOG Surveillance Needs Surveillance is defined to include all failure detection and in-plant relia-bility information-gathering activities. The surveillance strategy for the '

diesel generators should be a result of an analysis of diesel generator surveillance needs. This analysis should be systematically performed and the resultant surveillance needs periodically evaluated. The dynamic nature of the surveillance plan, with respect to the EDG's performance, helps to ensure a reliability focus to the surveillance activities. The tasks neces-sary to provide a reliability focus to diesel generator surveillance are shown in Figure 3-1.

A diesel generator is defined as the diesel generator subsystems and equip-ment exclusively employed to produce emergency ac power, appropriately par-titioned among the generating units at the plant. Table 3-1 defines a diesel generator in terms of its subsystems. The pieceparts to be asso-ciated with the diesel generator are those whose sole function is related to 3-1

l TASKS NECESSARY TO PROVIDE A RELIABILITY FOCL'S TO DIESEL GEN ERATOR l

SURVEILLANCE

' DEFINE SET PREPARE EQUIPMENT SET REllABILITY DEVISE EVALUATE SURVE!LLANCE COMPONENT TO INCLUDE IN TARGETS FOR SURVEILLANCE PLAN TO

w RELIABILITY EQUIPMENT IN PERFORMANCE NEEDS MEET THE MONITORING k PROGRAM PROGRAM NEEDS PROGRAM i

I l FIGURE 3-1.

TOP-LEVEL WORK BREAKDOWN GTRUCTURE SHOWING TASKS l

1 NECESSARY TO PROVIDE A RELIABILITY FOCUS FOR SURVEILLANCE k

TABLE 3-1 DEFINITION OF DIESEL SUBSYSTEMS Inside the Boundary Speed Contro) - (Includes governor, speed sensing, frequency sensing, and fuel racks positioning)

Fue) Supp1y - (Includes equipment from the day tank through injectors)

Fue1 Storage -

Lube oli - (Includes prelube, preheating if appitcable)

Engine Coo 11ng - (Diesel-specific cooling water)

Heat Sink - (Radiator or site service water system up to and including inlet and outlet valves of heat exchangers)

Exhaust -

Environment Control - (Room temperature and humidity control)

Intake Air Supply -

Turbocharger -

Diese] Mechanical - (The casing and all components within, up to, but not including, attached pumps or other piping systems)

Air Start - (Includes starting air supply)

Generator Electro-Mechanica] -

(Including up to output breaker)

Voitage Regulation / Field F1 ash -

Start Control - (Autostart sensors, logic, remote manual start capability)

Other l&C - (Including trips, control room indica-tions) 3-3

TABLE 3-1 (Continued)

DEFINITION OF DIESEL SUBSYSTEMS Outside the Boundary Load Sequencer -

DC Power Supply -

AC Power Supply - (For auxiliaries, I&C)

Synchronization Circuitry -

Service Water Supply - 1 AC Power Distribution System 3-4

t diesel generator operability. For instance, a diesel generator may require service piecepartswater for operability, but only those service water components and whose function is solely to support the diesel generator should be included in the diesel generator boundary.

Analysis is required to ensure that surveillance of diesel generators addresses a minimum set of criteria for acceptable surveillance. The analysis should result in a documented surveillance plan. The surveillance plan should specify the diesel generator surveillance and the rationale for the specified surveillance.

provide acceptable diesel generator The considerations surveillance that are: should be addressed to

1. All critical failure modes of the diesel are covered by the surveillance. Critical failure modes are likely failure modes that would fail the diesel generator function of providing emergency ac power.

2. The analysis should identify engineering conditions that are precursors to critical failure modes and suggest surveil-lance methods (e.g., condition monitoring) to detect those conditions in a timely fashion.

3. The analysis should identify likely standby diesel generator aging mechanisms and identify surveillance to detect these.

4. The analysis should emphasize consideration of common cause failure mechanisms that could fail more than one diesel generator at a site and identify surveillance to protect against these failures.

5. Diesel generator repair outages can result from off-normal conditions or failures that are caused by stress on the diesel from starting and running. Failures can also result from mechanisms that operate on the diesel generator while it is in standby. Diesel generator demand test periods should be set by balancing the effects of these two failure causes related (failure to standby modes related to demand stress and those stress). The analysis should contain these considerations.

6. A surveillance plan should be prepared that defines the types of surveillance to be employed, the surveillance intervals for each type, and other considerations such as test staggering. Justification based on engineering, human, or reliability considerations should be given as to why the surveillance types and intervals were chosen and why they are sufficient to achieve the reliability target.

Appendix B presents a more detailed discussion of the assessment of diesel generator surveillance needs.

3-5

3.3 Review Item C: EDG Performance Monitorina e

Performance monitoring of a diesel generator includes monitoring physical conditions that are precursors to failure or correlated to degradations in performance. Examples include lube oil temperature, manifold temperature and starting air moisture. Performance monitoring also includes statistical trending of failures and outages that may show detectable degradations in performance. While surveillance provides a "snapshot" of diesel generator operability, performance monitoring provides the "memory" portion of the problem detection task of a diesel generator reliability program.

The criteria for evaluating a diesel generator performance monitoring approach are:

1. The reliability information necessary to track diesel per-formance should be identified and correlated to the proposed surveillance. This is to ensure that the proposed sur-veillance will provide all the reliability information necessary to track diesel generator performance.

2. All performance monitoring computations required to be per-formed on both the diesel generator engineering information (i.e., physical condition data) and repair outages / failures should be explicitly defined.

3. Alert levels that signal possible diesel generator degrada-tion should be defined for each engineering and statistical parameter used for the diesel generator per formance moni- '

toring program. The alert levels should be choosen to min-imize false alarm but be sufficiently sensitive to detect problems.

Appendix C presents a more detailed discussion of diesel generator performance monitoring.

3.4 Review Item D: EDG Maintenance Procram The maintenanca policy for the diesel generators should be documented and clearly exhibit a reliability focus. The maintenance policy should include procedures for preventive maintenance, triggered by observed conditions and/or regularly scheduled, and a description of the spare parts policy.

The maintenance policy should also establish the basis for maintenance actions and tneir priority. This involves the identification of those conditions or precursors to catastrophic failure that are (1) detectable, (2) potentially severe in terms of diesel failure, i.e.,

lead to catastrophic diesel generator functional failure, (3) require long out-of-service times for repair, if the condition proceeds to catastrophic failure, and (4) are relatively likely to occur. Thus, the maintenance policy should have the following characteristics:

1. A distinction in the treatment of failures or conditions that result in, or could proceed to, catastrophic failure of the diesel generator versus those that do not.

3-6

2. A distinction in treatment of those repair or maintenance actions that result in disabling the diesel generator versus those that do not.

3. A recognition that preventive maintenance actions can be triggered on either time, using failure mode mean time between failures as a guide, or on conditions observed during surveillance.

4. A recognition that disabling repair times for noncatas-trophic diesel generator failures or conditions, compared to the repair times and outage times for the catastrophic failures that could result from these conditions if the non-catastrophic conditions are not repaired, are an important element in the maintenance policy.

5. A recognition that the maintenance policy is driven by the target reliability of the diesel generator.

1

6. A recognition that the spare parts policy should include a consideration of both the frequency with which the spare part is needed and the downtime necessary to complete the repair with and without the spare part on hand.

A more detailed presentation of the issues to address in this review item is given in Appendix D.

3.5 Review item E: EDG Failure Analysis and Root Cause Investication The diesel generator reliability program should contain a structured approach for systematically reducing identified diesel generator problems to correctable causes. An example top level structured approach is shown in Figure 3.2. This structured approach involves the following steps:

1. Use a failure cause analysis to determine the proximate cause of the failure. The proximate cause is expressed as a description of the piecepart failure cause, e.g., "relay xx i

failed to transfer due to corroded contacts."

! 2. Compare the proximate cause to past failures or conditions on the same and other EDGs to determine if the problem appears to have a systematic root cause, e.g., corroded contacts could be caused by an environmental mechanism.

3. If no systematic root cause is indicated, continue EDG operations as usual, including EDG performance monitoring.

If a systematic root cause is indicated, begin a structured root cause investigation.

4. Determine if the problem is generic or plant specific by reviewing NPRDS and other data and analyses for similar 3-7

Monitor EDG J Performance 7 1r Failure or off normal condition observed 1r Determine proximate cause Problem Closecut (foilure cause onelysis) Assess if surveillance or performance monitoring 1r should be altered Compare to post foilures/ conditions j(

to indicate possible systematic cause Systematic cause 37 No systematic cause v

Perform root cause onelysis

+

Review other plant , Generic or Generic cause r

records (NPRDS), plant specific m A generic industry groups, etc. cause? cure exists?

Plant Specific UO l

Cause ir Y Yes 1r Review cperational Determine if procedures, install m operation- or design related imp ment specio! monitoring if required cause Design Operational Related Related

?

Redesign to Change operotions correct problem to correct problem ir ir FIGURE 3-2. SYSTEMATIC ROOT CAUSE APPROACH 3-8

l i

l problem symptoms, or through contact with other utilities or I industry groups, i

5. If the detected reliability problem is generic, contact other plants that have had the problem to determine what corrective actions, if any, have proved effective. If an effective corrective action has been devised, implement it and proceed to the problem closeout portion of the EDG l

reliability program. If not, proceed to the next step. ;

6. If the detected reliability problem is plant specific, '

determine if the cause is related to the systems unique i design or to operational aspects such as test or main- t tenance. This can be done by special monitoring during test, review of operational procedures, or engineering design review.

7. If the reliability problem is determined to be design related, determine the particular design deficiency (through special condition monitoring,perhaps), and redesign or specify other corrective action.

8. If the reliability problem is related to faulty operations, identify and correct the specific procedure (s) that are the root cause of the problem.

9. When the root cause has been identified and corrective action implemented, proceed to the problem closeout item of the EDG reliability program.

Appendix E describes the process in more detail. An EDG reliability program should be able to verify that the above or similar steps are included in the systematic problem investigation procedures.

3.6 Review Item F: Problem Closeout The reliability program plan should specify the procedure that will be used for closing out diesel generator reliability problems. The closecut procedure occurs as the last of the following steps in the reliability program process:

1. Problem detection. 1

2. Problem cause determination.

3. Corrective action implementation.

4. Problem closecut.

The problem closecut procedures should be verified to contain two elements:

1. Establish criteria for problem closecut that are based on the nature of the reliability problem detected.

3-9 m7w , - _,.,-,_w - -,- -- -

we e. -+-i.a-

--9

l 1

l

2. Provide for any monitoring activity, and speci fy closeout procedures to ensure that the criteria have been satisfied.

The problem closecut criteria should be numerically based and be capable of measurement. The diesel generator reliability program submittal should I specify any special problem closeout procedures that will be employed to provide assurance that corrective actions will be effective.

l Appendix F presents a more detailed discussion of this review item. '

3.7 Review Item G: Data System The reliability program should include a _ description of the data gathering, storage, and retrieval system that will support the diesel generator reliability program tasks. The supporting system should contain the following operational and maintenance data:

1. Store both catastrophic diesel generator failures and diesel repair outages from noncatastrophic failures.

2. Store the time of detection, times when repair was initiated and completed, and restoration time of the equipment for each diesel generator repair action.

3. Store a description of the root cause or condition that led to repair and the method by which it was dctected.

4. Store each attempted start and run, runtime, and any failure ;

rate or failure probability denominator information as '

described Appendix A.

5. Store in a retrievable way all the information identified in the licensees' response to all of the above stated review l items.

In addition to the above identified operational and failure information, the

data gathering, storage, and retrieval system should c: ntain operating i

experience information on similar EDGs as provided through NPRDS, Part 21 reports, 50.55(c) reports, l.E Rs , consultants, and especially EDG manufacturers and their suppliers (e.g., governor vendors). This information would be used to supplement data on plant experiences and as a basis for corrective actions to preclude problemr. experienced by other EDG owners. EDG vendor correspondence and recommendations and updated operation test and maintenance procedures should also be stored in support of the reliability program.

Appendix G presents a more detailed discussion of th.is review item.

3-10

_ _ ~ . _ _ _ _

l l

i 3.8 Review Item H: ResDonsibilitites and ManaQement Controls The reliability program should have clearly defined responsibilities and management check points to ensure that all items are interacting effectively to maintain the EDG reliability at, or above, target values. This item should provide a means for plant management to review the operation and effectiveness of the reliability program and for altering the program if it becomes necessary. In addition, a means for independent audit of the effectiveness of the EDG reliability program should be incorporated into this item.

The following considerations are important:

1

1. A procedure and schedule for verifying that the EDG reliability targets are being met should be established.

2. There should be an identified mechanism for altering the EDG reliability program .thould it become necessary. i l

3. Identification of qualified personnel who will implement and 1

maintain the reliability program. Personnel qualifications should include diesel operation, maintenance, diesel design, reliability methodology, and implementation of reliability programs. ;

4. An unconditional commitment on the part of plant management to implement and maintain an EDG reliability program.

Appendix H presents a more detailed discussion of this item.

l l

l 3-11

t REFERENCES

1. Regula Guide 1.155, "Station Blackout," (Task SI 501-4), USNRC, Draft, tory November 1987.

l

2. M.A. Azarm, E.V. Lofgren et al ., . "Effectiveness of Reliability !

Technology Applicable to LWR Operational Safety," NUREG/CR-4618, Draft, April 1986.

3. Nuclear Management and Resources Council, "Guidelines and Technical Bases for NUHARC Initiatives Addressing Station Blackout at Light Water Reactors," NUMARC-8700, November 20, 1987.

Available in the NRC Public Document Room, 1717 H Street NW., Washington, !

D.C.

)

i R-1 l

E Gl.05SARY Availability -

The probability that a component is ready to perform its i

mission, thus it is not out of service for maintenance or repair or in a failed state.

Catastrophic Failure * - A failure that is both sudden and complete. It causes cessation of one or more component functions. i Degraded Failure * - A failure that is gradual, partial, or both. Such a failure does not cease all compon nt functions, but compromises a function.

The function may be compromised by any combination of reduced, increased, or erratic outputs.

Disabling Repair Time - The time for which a component is unavailable due to '

being removed from service for a maintenance act (preventive or corrective).

The time is measured from the time a component or system is taken out of !

service until the time fully operational at which that component or system is restored to a condition.

l Failure * - The its required termination of the ability of an item or equipment to perform function. i Incipient Failure * - An imperfection in the state or condition of an item or l equipment so that a degraded or catastrophic failure can eventually be expected to result if corrective action is not taken, i

Reliability - The probability that a component or system will carry out mission. its Unavailability its mission, thus - The probability that a component is not ready to perform failed state. Theitopposite is out of service for maintenance or repair, or in a of availability, and numerically equal to one minus the availability.

Unreliability - The probability that a component or system will nqt carry out its mission. The opposite of reliability, and numerically equal to one

)

minus the reliability.

Consistent with IEEE Standard-500-1984.

G-1

i l

l l

APPENDIX A ENERGENCY DIESEL GENERATOR RELIABILITY TARGET (Review Item A) l l

l l

l 4

A-1

^

l l

l TABLE OF CONTENTS Paae A.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . A-5 A.2 ISSUES TO CONSIDER WHEN MEASURING EDG RELIABILITY ....... A-5 A.3 TEST VALIDITY ......................... A-7 A.4 TEST FAILURE . . . . . . . . . . . . . . . . . . . . . . . .. A-8 A.4.1 Failure to Start .................... A-8 A.4.2 Failure to Load . . . . . . . . . . . . . . . . . . . . . A-8

. A.4.3 Failure to Satisfactorily Function for Specified Test Runtime . . . . . . . . ............ A8 A.4.4 Other Diesel Generator Failure Conditions . . . . . . . . A-9 A.5 EDG FAILURE EVALUATION CRITERIA. . . . . . . . . . . . . . . . . A-9 A.6 MEASUREMENT OF EDG RELIABILITY , . . . . . . . . . . . . . . . . A-17 A.7 INTERFACES WITH OTHER EDG RELIABILITY PROGRAM REVIEW ITEMS . . . . . . . . . . . . . . . . . . . . . . . . . A-18 REFERENCES FOR APPENDIX A . . . . . . . . . . . . . . . . . . . . . . A 20 Table LIST OF TABLES A-1 '.

EDG Failure Evaluation Criteria for EDGs with

, Reliability Target of 95% . . . . . . . . . . . . . . . . . . A-11 A-2 EDG Failure Evaluation Criteria for EDGs with ;

I Reliability Target of 97.5% . . . . . . . . . . . . . . . . . A-12 A-3 An Alternate Presentation of EDG Failure Evaluation i Criteria for EDGs with a Reliability Target of 97.5%. . . . . A-13 Fiaure LIST OF FIGURES A-1 Representation of Successful Diesel Generator Test ...... A-10 A-3 i

A.1 INTRODUCTION l

The objectives of this appendix are to (1) define the reliability target for emergency diesel generators (EDGs) and (2) to clarify the measures necessary to evaluate the achievement ~of this target.

The EDG reliability target will be derived from the guidelines provided in Regulatory Guide 1.155, "Station Blackout." These guidelines establish EDG i reliability levels of 0.95 or 0.975. The EDG reliability program will key 1

on minimum reliability targets; all EDG failures should be acted upon '

without dependence on either achieved reliability levels or target reliability levels. The EDG reliability can serve as an indicator of how well a plant's diesels are prepared to combat a loss of offsite power. In order to achieve consistency and realism in reporting the EDG reliability, the following elements are herein defined for the Diesel Generator Reliability Program: (1) diesel test runtime, (2) test validity, (3) test i failure, (4) calculation / estimation of EDG reliability, and (5) EDG failure '

, evaluation.

1 A.2 ISSUES TO CONSIDER WHEN MEASURING EDG RELIABILITY The following issues should be addressed in order to define the EDG reliability target and to measure the achieved EDG reliability. The purpose of this section is to lay out the issues. Sections A.3 through A.7 provide general solutions to the issues that should be addressed by this program. '

What are the EDG test runtime reouirements?

Each EDG should be started and run for a sufficiently long time to demonstrate its continued operation under the same stresses that would be present at the random occurrence for an actual demand. Reference A-1 recommends that, to demonstrate continued reliability, an EDG should be run at or near full load for at least 2 hours2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months . A test time less than 2 hours2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months (e.g., I hour) j could be used if any of the following conditions are met:

1. The lesser test runtime is shown to cover all the dominant failure modes of the EDG.

2. Adequate condition monitoring is performed while under test, and such conditions being monitored exhibit a stable, steady-state characteristic.

3. The shorter EDG test runtime should be shown to be preferable, from an operational standpoint, to cover all operational modes and likely failure modes of the EDG, and not to subject the engine to undesirable thermal transients.

What constitutes a valid test?

A valid test requires the diesel to be started in the same i condition and subjected to the same stresses that would be present at the random occurrence of an actual demand, with the A-5 i

exception that mission requirements for fast start and fast load need not be met during all routine surveillance tests provided that the fast start / load tests are performed at some appropriate interval (e.g.,6 months). A more detailed discussion of test validity is provided in Section A.3.

What_ constitutes a countable test failure?

Failure of the diesel generator to perform its required mission during an actual demand, or failure to start and run successfully for .he test runtime on a valid test should be accounted for in the calculation of diesel reliability. Section A.4 and NSAC-108 contain the specific definitions of EDG failures.

How should EDG reliability be measured and compared to a sDecified taraet?

EDG reliability calculations should accurately take into account the number of actual demands and valid tests, EDG runtime, failures to start, run, and load. The reliability measure should be a point estimate of the average reliability of the EDG.

Comparisons to the target reliability should take into account both this point estimate and the corresponding uncertainties. A more d6 tailed explanation of the approach is provided in Section A.S. >

How can the NRC determine if a plant's EDGs are Derformina satisfactorily?

Evaluation criteria for determining satisfactory performance and various stages of alert are proposed in Section A.S. These evaluation are based criteria (which are to be considered interim criteria) on the number of catastrophic failures in a specified number of demands. They are consistent with the criteria in Regulatory

rate, Guide 1.155, but also account for the false alarm and they weight recent history more heavily than history. The proposed action statements (see Section A.5) dopast ngi include more frequent testing when alerts are violated, but rather require a reassessment of the EDG reliability program to ensure that the EDG reliability target will be met in the future.

When can eartial demand tests be substituted for full tests?

Plant operating conditions will often prohibit exact simulations of the conditions under which an EDG should operate during an actual diesel demand. Each plant should determine what condi-tions are not exactly simulated in EDG tests and determine through ventive other reliability activities (condition monitoring, pre-of partial maintenance, shorter test intervals, etc.) the adequacy demand tests and the need for supplemental surveillance. It has been determined that slow starts and gradual loading can be used to obtain a valid measure of the EDG reliability level.

A-6

A.3 TEST VALIDITY A valid test requires that the emergency diesel generatpr be tested in a condition in which it would normally exist when asked to respond to a ran-dom loss-of-offsite-power event, and other plant transients requiring the EDGs.

This requirement is specifically intended to preclude actions solely for the purpose of guaranteeing a successful test. This requirement is NOT intended to preclude required or necessary maintenance to the emergency diesel generator or its support systems.

adverse condition to go uncorrected. Nor is it intended to allow a known Good engineering practices and condition monitoring can actually mask reliability problems when these tasks are performed merely to meet a re_quirement or the purpose of performing either task is not properly understood. For instance, any action taken just prior to test, and not performed on a routine basis, or any action scheduled to be performed only just prior to test would tend to mask problems that '

would occur during a random test. l An example is that blowing down the diesel starting air receivers just prior to a test to aliminate the failure modes resulting from accumulated moisture l in the air start system would bias the failure to start probability and present an inaccurate and misleading picture of diesel reliability. Since 1 actual demands for the diesels occur at unpredictable times, there would be i no opportunity to blow down the air receivers just prior to an actual !

demand. Therefore, the impact of a possibly important failure mode would be !

masked from the reliability performance measures, unless the test procedure j required checking for, and recording, the condition of excess water in the i system. In this case, blowing down the air receiver just prior to test I could be in the category of a good engineering practice and would not mask reliability problems associated with this condition as long as diesel failure to start was correlated to finding the condition. Blowing down the air receivers on a daily, or on some other periodic schedule not tied to demand tests, or per shift basis (as part of an operators normal routine duties) would be an acceptable form of condition monitoring. l i

Another example of invalidating a test is when a preventive maintenance check is always scheduled to be performed just prior to the diesel generator test. Maintenance actions designed to operate, check, or perform a special l

inspection of components directly affecting the satisfactory operation of l the diesel generator fall into this category. This includes checking valve operations, relay operations, circuit breaker operations, etc. However, regularly scheduled maintenance actions not scheduled only prior to test, but occasionally occurring prior to test, are expected and acceptable.

Also, if these maintenance actions are performed as part of the test or as an addendum to the test and are performed to correspond with a relationship to a specific failure mode or modes of the diesel generator, and the test allows for the evaluation of that relaticnship, then this would also be acceptable.

A7

l l

A.4 TEST FAILURE j

l A failure of an emergency diesel generator test is indicated by any one of the following: .

1. A failure to start (manually or automatically, remotely l or locally). I

2. A successful start, followed by a failure to load. l

3. A successful start and load but the diesel generator does not satisfactorily function for the specified test runtime. <

Failure is defined as either a catastrophic failure in accordance with IEEE l i

Standard-500 or immediate failure in accordance with the NPRDS Reporting ;

Procedures Manual, Rev. 10.

l A.4.1 Failure to Start A failure to start is defir.ed as a failure of the diesel generator to respond to a start signal either manually or automatically. The diesel generator should be started from ambient condition and accelerate to the required speed (RPM) in the time specified by the plant's technical specifi-cations. However, an automatic start failure by itself, if immediately (in l l

less than 5 minutes) recovered mar.ually from the control room or from the '

EDG area, will not count as a failure (for station blackout). In all cases, the cause of the failure should be ascertained and corrected; however, the l investigation should not interfere with the current operation of the diesel generator.

A.4.2 Failure to load i 1

A failure to load is defined as a failure of the generator to produce l adequate electrical power or fail to provide that power to the appropriate i emergency bus. This includes inadequate voltage output, either too high or too low; inadequate frequency regulation, either too low or too high; the failure of the electrical current path from the generator to the bus l including cables, output breakers, etc. All generator load ratings will be considered satisfactory if the load ratings meet the requirement set forth in the plant's technical specifications.

A.4.3 Failure to Satisfactorily Function for Soecified Test Runtime A failure of the diesel generator to satisfactorily function for the speci-fied test runtime occurs only if the diesel generator does not function properly and should either be manually tripped, or is automatically tripped, prior to the completio- of the runtime. Tripping the diesel for an incipient condition that would not prevent mission success in an actual demand is not counted as a valid run test or failure to run. A test runtime l less than the suggested 2 hour2.314815e-5 days 5.555556e-4 hours 3.306878e-6 weeks 7.61e-7 months minimum runtime suggested in Reference A1 (e.g., I hour) could be used if any of the conditions given in Section A.2 3 are met.

A8

A.4.4 Other Diesel Generator Failure Conditions Abnormalities of the diesel generator are considered actual failures only if the diesel generator is incapable of functioning to support recovery of a station blackout event or other plant transients requiring EDGs. That is, only catastrophi: or immediate failures of the diesel generator are to be considered actual failures for the purpose of this document. Note that while noncatastrophic conditions should not be considered as failures from the standpoint of calculating the EDG unreliability measures, they do constitute failures from the point of view of the reliability program in that they should be addressed within the context of the program. For example, during a diesel generator test, an operator notices that one of the cylinders is operating at a slightly elevated temperature, still within the safe operating limits of the diesel, and although the temperature is stable, he decides to shut down the diesel to investigate the cause. By shutting down the diesel, the specified test runtime requirement has not been met.

Although this would imply a failure, it does not constitute a failure since, if required, the diesel generator would still be able to function satisfactorily.

While the above example denotes a degraded or incipient class of failure that would not be counted as an actual failure, the following example illus-trates an actual catastrophic or imediate failure that would be counted as a failure.

If the operator, in the above example, actually noticed a high cylinder temperature in one cylinder and he also realized that the cylinder temperature was increasing so that in a short period of time the temperature would be above the safe operating limit, he would have no choice but to shut down the diesel to protect it from further possible damage. Since the diesel would not be available to supply emergency ac power, this would constitute an actual failure of the diesel generator and therefore would be l counted as a failure.

Figure A-1 illustrates a diesel generator test success, with the associated failure parameters indicated.

A.5 EDG FAILURE EVALUATION CRITERIA For NRC use, the progression of failures as well as the overall failure history should be used to judge the acceptability of diesel generator performance. The EDG failure evaluation criteria are presented in Table A-1 (for EDGs having a reliability target of 0.95), and in Table A-2, (for EDGs having a reliability of 0.975). These criteria are based on the number of )

catastrophic failures recorded in a succession of three operating histories.

For Table A-1, the operating histories are the last 20 demands, the last 50 demands, and the last 100 demands. For Table A-2, the operating histories are the lar,t 40 demands, the last 80 demands, and the last 120 demands.

Table A-3 is provided in order to assist the NRC in evaluating EDGs having a target reliability level of 0.975 by showing the failure progressions for the last 20, 50 and 100 demands.

A-9

j MAX S **

O O

' < o e.a ' S . DESICN O NIN !

f 7- S f

i

/ S:

=

I I:t l

? !!

" l a b i

, & ~

l 8 ^

L

.= c g u= n

-7 [

l e ' $2 E~

N

' d 3 2 U~ C 2l

?

c'o I J l o J #

1 I I I l

V gi 4 0 10 13 5 min

- - "'C PfT (hrs)

TENE IO cec - Requirement to achieve 900 RPM l] set - Ircq u i r c=cn t to acleIcvc electrical ra t i nr.s NOTE A - denotes success area - the 5 min - 3.ccovery t.criod fur auto-start f a l liere D/C test is successful if PfT - Req.elevacut t o me e t re-gui rcel et wsloes t ime all parameters stay within tiec success area for the required mission time duration.

1 FIGURE A-1 REPRESENTATION OF SUCCESSFUL DIESEL GENERATOR TEST (SOURCE, G.E. STANDARD TECHNICAL SPECIFICATIONS)

TABLE A-1 EDG FAILURE EVALUATION CRITERIA FOR EDGs VITH RELIABILITY TARGET OF 95%

Combinations of Evaluation Criteria Failure Evaluation Time Period False Alern

(# Failures /d Dejg1Q Criteria fl Demand /2 Ws) Rate t 2/20 YYYNNNNY - 10 Months 262

S/50 YYNNNYYN = 2 Years 111 3 10/100 Y N N N Y Y lt Y - 4 Years 33 Failure Progression 12345678 Legendr Y = Yes M = No Interrretations of the Failure Prearessieris Failure Preaten sten Int erer et at ien

1. 3 2 failures in 20 demarsda Tais is an macceptable conditico rotatring inanedtete action

, 3 S failures in 50 demands to declare the EDG inoperable. There is strong evidence 3 10 failures in 100 danands that the long-tern EDG unreliability is larger than the target value and no evidence that it is improving. The EDG reliability progran should be improved or enhanced before the EDO can be declared operable again.

2. 3 2 failures in 20 demands This is an alert condition where action is recnernended to 3 S failures in 50 desanda declare the EDO inoperable. There is evidence that the ED3

< 10 failures in 100 demands is deteriorating over time and that the current reliability is unacceptable. The action taken may depend on other circumstances and information from the plant.

3. 3 2 failures in 20 demands This is a mild alert condition where no action by the NRC

< S failures in 50 desands is recosusended miess there are other recent indications

10 failures in 100 danands of DO deterioration. IDGs with acceptable unreliabilities will display this condition about 26 percent of the time, Although some concern is justified, a single failure, rith no evidence of degraded performance, should not lead to excessive concern,

s. < 2 failures in 20 demands This is an acceptable condition. No concrete evidence of

S failures in So demands unacceptable performance. *

10 f ailures in 100 demands S. < 2 failures in 20 datands This is an acceptable condition. There is an indicotton of a S failures in 50 demands a past problem that has probably been corrected. Low-level 3 10 failures in 100 danands vigilance is prudent to ensure continued acceptable operation.

6. < 2 failures in 20 danands This is an acceptable condition h t one that needs continued 3 S failures in 50 demands vigilance. The*e is indication that e continuar4 past problem 310 f ailures in 100 danands is being corrected, but the evidence is not convincing enough to warrant a decrease in vigilance.

7

2 failures in 20 demands This is an acceptable condition but one that needs continued 3 5 failures in S0 demanda vigilance. no interpretation of this condition is sinalar S 10 failures in 100 demands to the interpretation of condition 6 above, except that the history of unacceptable performance is less ertensive.

8. 3 2 failures in 20 danands The interpretation of this condition la scunowhat similar to ,

S failures in 50 daeands the interpretation of condition 3, except that there is e 3 10 failures in 100 danands history of a performance problem that may have been corrected, or partially alleviated. This situation is an ambisuous one, ,

regairing a more detailed evaluation, h e assessment would be different if there were 2 f ailures in the last 50 demands and 2 l f ailures in the last 20 desands than if there were $ f ailures I in the last 50 and 2 in the last 20. An alert condition is indicated by this condition, i

A-11

TABLE A 2 1

EDG FAILURE EVALUATION CRITERIA FOR EDGs WITH RELIABILITY TARGET OF 97.5%

Combinations of i Evaluation Criteria Failure Evelvation Time Period Felse Alarm (d Failures /d Demandai Criteria (1 Demand /2 Wlts) Rate ,

g 2/40 YYYNNNNY - 6 Monthe 263 I

g 4/80 YYNNNYYN - 3 1/2 Years 143 3 6/120 YNNNYYMY - S Years 8t '

l Failure Progression 12345678 letends Y = Yes i N = No

Interrretatier's of the Failure Preatessions Failure Preare nten Intereretation

1. I 2 failures in 40 demands This is an macceptable condition requiring isstediate action I 3 4 failures in 40 demands to declare the EG inoperable. There is strong evidence g 6 f ailures in 120 dessands that the long tors CG unreliability is larger than the target value and no evidence that it is improving. The j EDG reliability protras should be improved or enhanced before the CG can be declared operable again.

2. 3 2 failures in 40 demands This is an alett condition where action is recommended to l 3 4 failures in 80 derr. ands declare the EDG inoperable. There is evidence that the EtG

6 failures in 120 demands is deteriorating over time and that the current reliability is m acceptable. The action taken may depend on other i circumetances and information from the plant.

3. ! 2 failures th 40 demands This is a mild alert condition utere no action by the NRO

4 failures in 80 demands is roccumeended miese there are other recent indications a 6 fattures in 120 demands of CG deterioration. IDGe with acceptable unre11 abilities will display than condition about 26 percent of the time.

Although some concern te justi!!ed, e single feiture, with no I

i evidence of degraded performance, should not lead to excessive i concern. ,

4 1

4

2 failures in 40 danands This is an acceptable condition. No concrete evidence of

4 failures in 80 demands unacceptable performance. {

6 failures in 120 demands J I

S.

2 failures in 40 demands This is an ecceptable condition. There is an indication of *

4 fattures in 60 danands e past problem that has probably been corrected, low-level

2 6 f ailures in 120 danands vigilance is prudent to ensure continued acceptable opration.

, 6.

2 failures in 40 danands This is an acceptable condition but one that needs continued 3 4 tallures in 30 desands vigilance. There is indication that a continuing past problee l 1 6 f ailures in 120 demands is being corrected, but the evidence is not convincing enough to warrant e decrease in vigliance.

I l 7.

2 failures in 40 danands This is an ecceptable condition but one that needs continued j 4 failures in 80 de ar.ds vigilance, The interpretation of this condition is stallar

6 failures in 120 demands to the interpretation of condition 6 above, except that the history of unacceptable performance is less ertensive.

, 8. 3 2 failures in 40 danands The interpretation of this condition is scenewhat similar to

4 failures in 80 demands the interpretation of condition 3, except that there is a 2 6 f ailures in 120 demands history of a performance probles that may have been corrected 4

j or partially elleviated. This situation is an ambiguous one, requiring a more detailed evaluation. The assessment would be

] different if there were 2 f ailures in the last 80 demands and 2

! failures in the last 40 demands 'han if there were 4 fattures j

in the last to and 2 in the le t0. An alert condition is irdicated by the latter.

l l

1 l

I A-12 1

l

- --~ ~ U

l l

)

TABLE A-3 AN ALTERNATE PRESENTATION OF EDG FAILURE EVALUATION CRITERIA FOR EDGs WITH RELIABILITY TARGET OF 97.5%

i l

l Combinations of Evaluation Criteria Failure Evaluation Time Period Falso Alarm id Failures /d Demands) Criterie ft Demand /2 IQ,,Q Rata g 1/20 YYYNNNNY - 10 Months 393 3 3/50 YYNNNYYN - 2 Years 131 1 6/100 YNNNYYNY - 4 Years et l

Failure Progression 1234 5676 '

i Legend Y = Yes l N = No l

Interrretatims cf the Fellure Proaressions ;

Failure Prearessten Interrretation

]

1. 3 1 failure in 20 dernands This is an unacceptable condition requiring imediate action

, 3 3 failures in 50 demands to declare the EDG inoperable. There is strong evidence 3 6 failures in 100 dernands that the long-tern EDG unreliability is larger than the target value and no evidence that it is improvir,g. The i EDG reliability program should be improved or enhanced before I the EDG can be declared operable again.

2. 3 1 failure in 20 dernands This is an alert condition where action is recomended to 3 3 f ailures in 50 demands declare the EDG inoperable. There is evidence that the EDG j

< 6 failures in 100 demands is deteriorating orer time and that the current reliability <

is unacceptable. The ac'. ion taken may depend on other l circumstances and information from the plant. '

3. 3 1 failure in 20 dernands This is a mild alert condition where no action by the h%

3 failures in $0 dernands is rococumended miese there are other recent indications

< 6 f ailures in 100 demands of EDO deterioration. D Js with acceptable unreliabilities will display this condition about 39 p ecent of the time.

Although some concern is justified, a single f ailure, with no evidence of degraded performance, should not lead to excessive J

concern.

4. < 1 failure in 20 denands This is an acceptable c m dation. No concrete evidence of

= 3 failures in 50 den. ands unacceptable performance. *

6 f ailures in 100 demands

$. < 1 failure in 20 dernands This is an acceptable condition, There is an indication of

3 failures in 50 demands a past problem that has probably been corrected. Low-level 3 6 failures in 100 demands vigilance is prudent to ensure continued acceptable operation.

6. < 1 failure in 20 dernands This is an acceptable condition but one that needs continued 3 3 failures in 50 demands vigilance. There is indication that a continuing past problem 3 6 failures in 100 demar.3s is being corrected, but the evidence is not convincing enough to warrant a decrease in vigilance.

7. < 1 failure in 20 demands This is an acceptable condition but one that needs centinued 3 3 failures in 50 denands vigilance. The interpretation of this condition is similar

< 6 f ailures in 100 desands to the interpretation of condition 6 above, except that the history of unacceptable performance is less ertensive.

8. 3 1 failure in 20 dernands The interpretation of this condition is somewhat similar to

3 fallares an 50 demands the interpretation of condition 3 except that there is a 3 6 f ailures in 100 dernands history of a performance problem that may have been corrected or partially alleviated. This situation is an ambiguous one, requiring a sore detailed evaluation.

A-13 J

l

, . _ . _ ~ - . _

In interpreting the failure data, the oroaression of failures is very important. The interpretations consider the fact that failures that occur early in a series of demands are not as important as failures that occur in more recent history. The interpretations also consider the false alarm rates. For instance, in Table A-1, the false alarm rates are defined here as the percentage of time that a diesel whose .trug EDG reliability was satisfactory, i.e., .95 or less per demand, would generate 2 or more failures in 20 demands, 5 or more failures in 50 demands, and 10 or more failures in 100 demands. These false alarm rates are shown on Table A-1 in the last column. The EDG on-line time required to generate 20, 50, and 100 demands, assuming demands occur on the average of once every 2 weeks, also given on this table. Table A-2 presents similar information for EDGs is having a reliability target of 0.975.

All combinations of the evaluation criteria are shown in Table A-1 and for EDGs with a reliability target of 0.975, in Table A-2). There are e(ight a

such combinations, each with a somewhat different interpretation. The progression of failures represented by the combinat the evaluation criteria, and their interpretations, are presented on.icns of table.

each ,

The EDG failure evaluation criteria address several of the objections raised in Reference A-2 concerning using an average unreliability value as a measure of EDG performance. Even though the criteria are indicators of long-term performance, recent history is weighted more heavily than less recent history in the interpretations of the failure progressions. Also, the likelihood of false alarms (incorrectly concluding that there is an EDG performance problem) is accounted for in the interpretations. The objection '

in Reference A-2 that the evaluation criteria are slow acting remains a drawback to the scheme presented in Tables A-1 and A-2. This is an inevitable consequence of using a (catastrophic) failure count as a perfor-mance measure. However, it is deemed satisfactory as an interim measure until more sophisticated performance measures are developed and validated for NRC and industry use. -

A more detailed description of the interpretation of each failure progression is given below.

Failure Proaression 81 (Immediate Action Required)

For EDGs with Reliability For EDGs.with Reliability Taraet of 95% Tarcet of 97.5%

2 2 failures in 20 demanos 2 1 failure in 20 demands 1 5 failures in 50 demands 2 3 failures in 50 demands 2 10 failures in 100 demands 2 6 failures in 100 demands ,

l This condition is unacceptable and requires immediate action by the licensee '

to declare the diesel generator inoperable, and enter the corresponding '

limiting condition for operation (LCO) action statement appropriate to the l end of an allowed outage time with the EDG inoperable. There is less than a l 3 percent chance that the long-term EDG unreliability is acceptable, and there is no indication that it is improving. This strongly suggests that A-14

there are serious deficiencies in the EDG reliability program that should be corrected before the diesel generator can be declared operable with the targeted reliability. ;

Failure Proaression #2 ($trong Alert)

For EDGs with Reliability For EDGs with Reliability Taraet of 95% Taraet of 97.5%

i 2 2 failures in 20 demands 2 1 failure in 20 demands 1 5 failures in 50 demands 2 3 failures in 50 demands

< 10 failures in 100 demands < 6 failures in 100 demands This is a strong alert condition where action by the NRC is recommended to determine if conditions warrant putting the subject diesel generator into an inoperable status. There is strong evidence that the long-term EDG

, unreliability is deteriorating over time or is unacceptable. The action taken by the NRC and licensee may depend on circumstances other than the number of catastrophic failures. For instance, if the EDG repair outage time was also large and increasing, this would substantiate the fact that the EDG performance in recent history was deteriorating and below the target performance. Several failures in succession due to the same failure cause, or several repair actions in succession due to the same cause, would also indicate that the EDG reliability program was inadequate to achieve the target reliability. The reliability program should be improved before the diesel generator can be declared operable and capable of achieving the '

targeted EDG reliability.

Failure Proaression #3 (Mild Alert) l For EDGs with Reliability For EDGs with Reliability Taraet of 95% Taraet of 97.5,

)

2 2 failures in 20 demands 2 1 failure in 20 demands !

< 5 failures in 50 demands < 3 failures in 50 demands

< 10 failures in 100 demands < 6 failures in 100 demands This is a condition that provides a mild alert to the NRC and licensee that a diesel generator may be experiencing performance problems. EDGs with an acceptable reliability of .95 per demand will experience two or more failures in 20 demands about 26 percent of the time. Similarly, EDGs with an acceptable reliability of .975 will experience 2 failures in 40 demandt about 26 percent of the time. No immediate action by the NRC is recommended unless there are other recent indications that the EDG performance is deteriorating. Other indications of EDG performance deterioration are increasing or large repair outage times in recent history compared to past performance, or several failures or repairs due to the same failure cause.

l Also, Eart than 2 failures in the last 20 (or last 40) demands should be l cause for heightened concern. If these conditions are noted, they may indicate a failure of the EDG reliability program, and steps should be taken l to improvc the program. If they are agl noted, no action should be taken by i the NRC.

A-15

m. _ _. . _ _

Failure Proaression #4 (Acceptable)

For EDGs with Reliability For EDGs with Reliability i Taraet of 95% Taraet of 97.5%

i

< 2 failures in 20 demands < 1 failure in 20 demands

< 5 failures in 50 demands < 3 failures in 50 demands i < 10 failures in 100 demands < 6 failures in 100 demands This is an acceptable condition. There is no concrete evidence of unacceptable EDG performance. No action by the NRC is recommended. ,

Failure Proaression #5 (Acceptable)

For EDGs with Reliability For EDGs with Reliability Taraet of 95% Taraet of 97.5%

< 2 failures in 20 demands < 1 failure in 20 demands

< 5 failures in 50 demands < 3 failures in 50 demands 2 10 failures in 100 demands 1 6 failures in 100 demands i This is an acceptable condition. There is an indication of a past problem i

that has probably been corrected. However, low-level vigilance by the NRC is recommended to ensure that the EDG performance remains acceptable. No other action by the NRC is recommended.

Failure Proaression 86 (Needs Continued Vigilance)

For EDGs with Reliability For EDGs with Reliability Tarcet of 95% Taraet of 97.5%

< 2 failures in 20 demands < 1 failure in 20 demands 1 5 failures in 50 demands 2 3 failures in 50 demands -

i 1 10 failures in 100 demands 2 6 failures in 100 demands -

This is an acceptable condition but one that needs continued vigilance by the NRC. There is evidence that a long-term EDG performance problem is being corrected, but the evidence is not strong enough to warrant a decrease in vigilance. No other action by the NRC is recommended unless an j additional catastrophic failure occurs to change the failure progression.

! Failure Proaression 87 (Needs Continued Vigilance)

For EDGs with Reliability For EDGs with Reliability Taraet of 95% Tarcet of 97.5%

< 2 failures in 20 demands < 1 failure in 20 demands 1 5 failures in 50 demands 2 3 failures in 50 demands

< 10 failures in 100 demands > 6 failures in 100 demands This condition has an interpretation similar to that of failure progression i #6 above. The condition is acceptable, but requires continued vigilance to ensure that an apparent long-term EDG performance problem has been

]

1 A-16 i

f corrected. Additional catastrophic failures could change this progression 3 into one where NRC action is recommended.

.J ,

Failure Proaression 88 (Needs Investigative Action)

For EDGs with Reliability For EDGs with Reliability Taraet of 95% Taraet of 97.5%

J

> 2 failures in 20 demands 2 1 failure in 20 demands

< 5 failures in 50 demands < 3 failures in 50 demands-i l

2 10 failures in 100 demands 2 6 failures in 100 demands The interpretation of this failure progression is snmewhat similar to the interpretation cf progression #3, but the condition is an ambiguous one.

There is an indication that a past EDG performance problem may have been i corrected or partihily alleviated. However, current EDG performance- indi- 1 cates a possible partial recurrence of the problem, or another problem. A more detailed evaluation should be performed before action by the NRC is specified. However, this is an alert condition where investigative action 1 by the NRC is necessary.

1 A.6 MEASUREMENT OF EDG RELIABILITY i

The EDG reliability calculation method described herein differs from the !

NSAC-108 method by explicitly including diesel generator outage in the '

reliability calculations. Although the NRC has determined, in Regulatory Guide 1.155 and other publications, that the probability of failure on demand provides an adequate indication of EDG performance, the inclusion of EDG outage time in the reliability calculations will ensure high diesel ;

generator reliability (by encoura when it is absolutely necessary). ging licensees to take EDG out ofservice I i

The reliability (Q) of a diesel generator should be calculated as

]

follows:

I l

Q=1-(ad+9r)

I qd is estimated by the number of start and load run failures in i the last 20, 50, and 100 demands accumulated over no greater than

! a 3 year period, in accordance with NSAC-108, and is the value i

reported to the NRC for comparison to reliability target levels.

The uncertainty of the calculation should also be considered, h gr is estimated by summing the actual outage times of the diesel generator and dividing the sum by the length of time it took to accumulate the total number of demands (20, 50, and 100). Each j

j outage time measured as the total time the diesel generator is 1 declared out of service. Outage times that accrue when the diesel generator is required operable, whether due to a

corrective maintenance action or to a scheduled preventive maintenance action, and all outage times that accrue as a result of a corrective maintenance action regardless of the requirement

) A-17

for the diesel generator should be included in the unavailability calculation.

Preventive maintenance actions that are scheduled when the diesel generator is not. required are not to be included in the reliability calculations. For example, a preventive maintenance action that disables the diesel generator during a refueling period would not be accrued in the calculation of gr.

An example calculation is provided below; Examole Reliability Calculation Suppose the number of catastrophic failures for the specified diesel genera-tor was determined failures to start. Theto be ang in its last 20 load-run tests and there were no total time to accumulate 20 tests was one year.

hence, qq =

_1 =

.05 20 The above outage time failure, wasassociated 72 hours8.333333e-4 days 0.02 hours 1.190476e-4 weeks 2.7396e-5 months with the diesel, resulting from the repair of the allowable by Technical Specification). The total time period is entire time window. Additional 365 -

days,(and the diesel generator was require incipient or degraded failures, outage time may come from repair of preventive maintenance, or possibly unavailability during surveillance tests, hence, qr "

-- 72 hrs =

72 hrs 365 days 82 x 10'4 and _ 8760 hrs Q =

1 - (.05 + 82 x 10~4) =

0.942 -

94.27.

These realistic calculations show how EDG outage time can affect overall reliability and can cause an EDG not to meet the reliability target.

i A.7 INTERFACES WITH OTHER EDG RELIABILITY PROGRAM REVIEW ITEM The achievement of a target reliability provides an easy to use, indication top level of themeasure single sufficient EDG's performance.

of the EDG's However, it is not necessarily a performance. Review Item B (EDG Surveillance satisfactory or unsatisfactory Needs and Item C (EDG Performance Monitoring) will use the reliability measure)s and be required to respond to any degradations in this measure. As shown in the example of Section A.6.

reliability measure.one failure can result in a significant degradation of this The Review (demands, Item G (Data System) is necessary to provide the raw data failures, run hours, and outages) required to calculate the reliability measure and the EDG failure evaluation criteria. The change in this measure and the evaluation criteria should be monitored throughout the A-18

diesel reliability program and, if the reliability program is working, the measure and criteria should improve. In this sense the use of the reliability measure and criteria over the long-term is a measure of the long-term success of the reliability program.

A-19

9 1

REFERENCES FOR APPENDIX A A-1. K. Hoopingarner et al., "Aging of Nuclear Station Diesel Evaluation of Operating and Expert Experience (PhaseGenerators: I Study),"

Pacific Northwest Laboratories, NUREG/CR-4590, Vol. 1 PNL-5832, August 1987.

A-2. Letter from Joseph R. Fragola to Aleck Serkiz,

Subject:

"Comments to Station Blackout Regulatory Guide (Task SI 501-4)," dated June 6, 1987.

li 4

ir 1

O d

)

I 1

j A-20 I

APPENDIX B ENERGENCY DIESEL GENERATOR SURVEILLANCE NEEDS (Review Item 8) i l

l 1

l l

B-1

h i

TABLE OF CONTENTS E.411 B.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . B5 1 B.2 ISSUES TO CONSIDER WHEN ADDRESSING EDG EQUIPMENT 1

B0UNDARY AND SURVEILLANCE NEEDS . . . . . . . . . . . . B-5 i B.3 INTERFACES WITH OTHER EDG RELIABILITY PROGRAM i REVIEW ITEMS . . . . . . . . . . . . . . . . . . . . . B-16 REFERENCES FOR APPENDIX B . . . . . . . . . . . . . . . . . . . B-1B LIST OF TABLES Table B-1 Definition of Diesel Subsystems ............. B-8 B2 EPRI/NRC Cause Codes . . . . . . . . . . . . . . . . . . . B-13 LIST OF FIGURES Fiaure B-1 Issues for Defining EDG Surveillance Needs (Review Item B). . . . . ............... B-6 i B-2 Hypothetical EDG Stress Level During Standby '.

l Period and During Demand Test . . . . . . . . . . . . . B-14 l

)

s 1

1 N

)

i l B-3

. -_ _ ~ ..

B.1 INTRODUCTION !

t The objectives of this appendix are to 1) define the equipment boundary !

4 that comprises the emergency diesel gener(ator (EDG) system and (2) define ;

the important elements that should be included in submitted assessments that define the surveillance needs of the EDG system.

EDG surveillance should provide a measure of assurance that the EDG reliability target is being achieved. The EDG equipment boundary should be explicitly defined ~so that all pieceparts considered as part of the EDG system will be assessed as to '

their surveillance needs. The surveillance needs of the EDG system should be assessed so that, in the long term, the diesel generator reliability goal will should be met. Therefore, review of a diesel generator reliability program the EDG include a review of the equipment considered within the boundary of '

and a review of the process and rationale by which the surveillance needs of this equipment were determined.

Analysis of equipment boundaries and surveillance needs are two tasks of a !

Reliability Centered Surveillance (RCS) concept. This concept was developed to implement the problem detection portion of a reliability program. ,

description of RCS is given in Reference B-1. This reference providesA !

additional detail that could be useful for reviewing submittals to ,

ensure that they reflect an acceptable assessment of EDG surveillance needs.

i Section B.2 of this appendix identifies the technical issues that should be considered when addressing EDG surveillance needs. This section provides review checklist of issues to match against the surveillance needs a 3 that may be contained in a diesel generator reliability program under issues l review. i Section B.3 lists the interfaces between the assessment of EDG surveillance needs and other review items.

B.2 ISSUES TO CONSIDER WHEN ADDRESSING EDG EQUIPNENT B0UNDARY AND SURVEILLANCE NEEDS

Figure B-1 identifies the issues that should be addressed to provide Review Item B.

l (Also shown on Figure B-1 (dotted lines) are the other tasks that are required to implement an RCS program to provide the problem detection l

1 portion of a diesel generator reliability program.) Each of the issues are I discussed, under separate heading, below. I i i Has the EDG eouiDment boundar_v been established?

i The EDG equipment is defined to be comprised of those subsystems and piecep"ts that are exclusively employed to produce emergency ac

power. A single diesel generator is defined to include the i diesel engine, generator, and the subset of supporting e'luipment that is associated exclusively with the generation of emergency

ac power using that diesel generator. The pieceparts to be asso-ciated with the diesel generator are those whose sole function is i related to diesel generato' operability and production of 4

emergency ac power. For instance, a diesel generator may require I service water for operability, but only those service water components and pieceparts that are solely there for the diesel generator are to be included in the diesel generator boundary, l

B-5 6

-~, _ . , _ - _ - . , _ _ -,_-,r-- -__.,.,- ,_ ,- . _. --. - _

_ . . ._= _. -

r Issues for Defining EDG ,

1 Surveillance l Needs i

________ i Set EDG Issues for Issues for Issues for Devise EDG reliability defining EDG evaluating EDG preparing EDG performance targets equipment surveillance surveillance monitoring

! set needs plan ;

i

., -Define EDG -Ensure surveillance -Ensure surveillance subsystems coverage of all EDG types specified j pieceparts and t

-Define EDG critical failure -Ensure there are '

[ boundary by modes test intervals 1 specifying specified for -

pieceparts at -Ensure test each surveillance interfaces with efficiency type other systems

-Ensure aging -Ensure testing l is considered schedules are .

! considered

-Ensure EDG demand tests consider failure

mode frequency, cause, and severity

-Ensure common cause j is considered

! FIGURE B-I. ISSUES FOR DEFINING EDG SURVEILLANCE NEEDS (REVIEW ITEM B) r 1 _ _ _ _ _ . _ . . . . . _ . _ . . _ _ . _ , _ . _ . . . _ _ _ . _ _ _ . _ _ . ___ _ . . . .

l l

Table B-1 generically suggests a diesel generator definition in terms of its subsystems.

Has the assessment of diesel cenerator surveillance needs identified the critical failure modes and provided surveillance coverace for them?

Critical failure modes are defined as those diesel generator piecepart or subsystem failure modes that would fail the diesel generator mission of successful start, load, and runtime as specified in the surveillance plan. Critical failure modes can be identified using reliability techniques such as Failure Modes and Effects Analysis (FMEA), as described in Reference B-2. All critical failure modes should be identified and considered when devising the diesel generator surveillance plan. Between test and between-surveillance intervals should be established based on

the expected frequencies of the critical failure modes. The concept of adequate surveillance coverage of critical failure modes is referred to in the reliability literature as "test adequacy." An "adequate" test is one that is capable of detect-ing any of the critical failure modes with sufficient likelihood that the diesel generator reliability target is met.

Has an assessment been oerformed of noncatastrophic failures and conditions that will. if lef t unattended. Droceed to CatastroDhic

, diesel cenerator failure?

Some diesel generator critical failure modes are preceded by non-catastrophic failures or conditions that are detectable through test, inspection, or condition monitoring. The diesel generator surveillance assessment should identify these and assess surveillance needs by examining tradeoffs among:

e The diesel generator reliability target, o

The repair outage time for repair of the noncatas-trophic failure or condition.

1 i e The repair outage time if the noncatastrophic failure or condition is allowed to progress to a catastrophic failure. l 1

e Any diesel generator surveillance outage time required i to detect the noncatastrophic failure or condition.

l e

j The expected lag-time between when the noncatastrophic precursor condition becomes detectable, and when the catastrophic failure occurs, versus the interval between scheduled EDG outages.

It is cautioned thet not all critical failure modes have non-catastrophic precursors that are readily detectable. The ones with nondetectable precursors are referred to as residual failures. The only maintenance strategy for "residual" failures B-7

TABLE B-1 DEFINITION OF DIESEL SUBSYSTEMS Inside the Boundary Speed Controi - (includes governor, speed sensing, i l frequency sensing, and fuel racks j positioning)

Fue1 Supp1y - (includes equipment from the day tank through injectors)

~

Fue1 Storage -

Lube oli - (includes prelube, preheating if applicable)

Engine Cooiing - (Diesel-specific cooling water)

Heat Sink - (Radiator or site service water system up :

to and including inlet and outlet valves of heat exchangers)

Exhaust -

Environment Contro1 - (Room temperature and humidity control)

Intake Air Supply - [

Turbocharger -

Diesei Mechanica] - (The casing and all components within, dp i to, but not including, attached pumps or

) cther piping systems)

., Air Start - (!ncludes starting air supply)

Generator E1ectro-Hechanica] - (Including up to output breaker) l

! Voitage Regu1ation/Fie1d F1 ash -

1

, Start Contro) - (Autostart sensors, logic, remote manual start capability)

.i Other ISC - (including trips, control room indica-

)j tions) 4 B-8

TABLE B-1 (Continued) _

l DEFINITION OF DIESEL SUBSYtTEMS l Outside the Boundary Load Sequencer - l 1

DC Power Supply -

AC Power Supply - (For auxfilaries, l&C)

Synchronization Circuitry -

Service Water Supply -

AC Power Distribution System -

i 9

i i

B-9

is to repair them as they occur. The only surveillance strategy for residual failures is to ensure that the surveillance mix will result in their detection when they occur.

t The ability of a test or inspection to detect precursors to catastrophic failures is referred to in References 8-1 and B2 l as the efficiency of the test. An efficient test is one that has l a high likelihood of detecting precursors to nonresidual critical 1

failure modes.

Achieving efficient surveillance is important when the dominant diesel generator failure modes are of the nonresidual type, i.e.,

I they have detectable precursors. An example of this type of condition is water in the EDG airstart system, which could result in a failure of the diesel generators to start (and, depending on the commonalty of starting subsystems, could result in a common

, cause failure of more than one diesel generator at a plant).

References B-1 and B-2 provide further discussion of the issue of test efficiency. The determination of which critical failure modes are residual and which have detectable precursors requires an engineering evaluation. Once that determination has been made, statistical and reliability techniques could be useful for characterizing the detectability of the precursor, frequency of occurrence, or relative repair times to use in the tradeoffs suggested above. Use of these techniques is discussed in Reference B-2.

Has the assessment of EDG surveillance needs considered detection of acina conditions?

The definition of aging used herein is defined to include both the time-related deterioration of EDG pieceparts or subsystems and the equipment-use-related wearout of pieceparts or subsystems due to diesel generator cycling. Assessment of the EDG i

surveillance needs should include an assessment of any special surveillance needed to detect deterioration of the diesel generators due to aging phenomena.

Aging of EDG subsystems or pieceparts could be manifested by increased frequency and/or duration of repairs or changes in measurable physical conditions such as crankcase temperature or pressure, lube oil pressure, starting air pressure, or cycling frequency, etc. Since piecepart or subsystem aging will eventually show up as degraded diesel generator performance enless the aged parts are repaired or replaced, the surveillance assessment should include assessing the need to provide surveillance or condition monitoring to detect these conditions.

The first step in this process is to identify which portions of the EDG are expected to age quickly. The second step is to identi fy measurable physical conditions that can be used to detect the aging process. Finally, surveillence can be specified I to detect changes in the measurable physical conditions.

4 B-10

Reference B-3 provides a detailed discussion of aging phenomena and the detectable physical conditions that accompany aging for various types of components and pieceparts.

i Has the assessment of the EDG surveillance needs considered the need to detect common cause failures?

The assessment of surveillance needs should consider two types of t hardware common cause failures: (1) "hard-wired" comon cause failures due to the dependency of more than one diesel generator on a single supporting subsystem and (2) comon cause failures due to a single comon cause shock. An example of the hard wired type of common cause failure is the dependence of more than one diesel generator on a common diesel fuel tank. An example of a single comon cause shock resulting in failure of more than one diesel generator could be a common moisture problem in the air i

start accumulators that provide starting air to the diesel '

generators. Both types of common cause failure mechanisms should be identified and the needs for surveillance to detect potential common cause faults assessed.

The surveillance needs to protect against common cause failures should be assessed based on a reliability target for the EDG system, as opposed to the target for an individual generator. diesel The reliability target for the EDG system will depend ,

on the system configuration at an individual plant and therefore is plant specific. In general, the target for the EDG system will be more constraining than for an individual diesel i

generator, which will result in more intensive surveillance needs for the system than for an individual diesel generator. As i always, the frequency of occurrence of the common cause shock, which results in the camon cause failure, repair times, and whether or not precurst.rs to the common cause failure can be identified will al so impact the type and frequency of surveillance.

Hard wired common cause failures can be identified by performing a fault tree analysis of the EDG system to the piecepart level of detail or through use of an FMEA. In fact, the analysis to identify the diesel generator critical failure modes, if accomplished with identification of hard wired common cause failures in mind or if done at the EDG system level, will result in identifying hard wired common cause failure modes.

Single piecepart or subsystem failure modes that result in failure of more than one diesel generator represent hard wired common cause failures modes, and the above techniques are all capable of identifying these. Reference B-2 discusses identifi-cation of critical failure modes and therefore, by extension,

hard-wired common cause failure modes, using FMEA and fault tree analysis.

A common cause system is defined as a set of components that are all subjected to a particular comon cause shock or a particular type of comon cause shock. Identification of common cause B 11

systems that are not hard wired can hypothetically be accomplished through application of a four-step process:

1. Develop a fault tree for the EDG system to the piecepart level of detail (useful for hard wired common cause failure mode identification, also).

2. Extend the EDG piecepart fault tree to the failure cause level. Failure causes with potential common cause significance are shown in Table B-2, which are the EPRI/NRC cause codes from Reference B 4.

3. Obtain the reduced Boolean equation corresponding to the

! failure cause tree developed in step 2 above. A Boolean l reducticn code such as SETS, FTAP, or CAFTA can be used for this step.

4. Identify and collect all the single-term cut sets from the Boolean reduction process. These represent ootential common cause systems. Additional engineering assessment is required to prioritize these and devise surveillance types and intervals to protect against them.

The above is a formalized process for identifying potential and nonobvious common cause systems. The process is described in more detail in Reference B-5. It has been developed in theory i but has not, at the time of publication of these guidelines, been demonstrated in practice. An alternative method is to assess common cause systems using ergineering judgment.

Has the assessment of EDG demand test intervals considered the causes and severities of dominant failure modes?

Figure B-2 represents (conceptually) the stress on a standby '.

diesel generator through a cycle that includes: (1) in standby, (2) start-up for demand testing, (3) run during demand testing, (4) shutdown, and (5) return to standby status. The stresses during standby are due to environmental factors that produce oxidation, corrosion, thickening of lubricants, stratification of fuel, accumulation of moisture, etc. The stresses during the test cycle are due principally to factors that occur during diesel operation such as vibration, wear, mechanical stresses, l

and electrical contact burning due to arcing. Both types of i

stresses, those that result in failure while the diesel generator is in standby (standby stresses) and those that result in failure when the diesel generator is started and operated for a demand test (demand stresses), are capable of producing catas-trophic diesel generator failure, or alternatively, the need to repair a detected noncatastrophic failure or condition.

The partition of EDG failures by cause (standby stress-caused and demand stress-caused) and by severity (catastrophic failure or noncatastrophic repair) failure requiring diesel generator outage for leads to a situation where there is an optimum demand B 12

TABLE B-2 EPRI/NRC CAUSE CODES

D -

Design / Manufacturing / Construction Inadequacy DR -

Plant Definition Requirements Inadequacy DE -

Design Error or Inadequacy DM -

Manufacturing Error or Inadequacy .

DC -

Construction Error or Inadequacy '

DX -

Other(explain)

P -

Procedures Inadequacy (ambiguous, incomplete, erroneous)

P0 - Defective Operational Procedure PM -

Defective Maintenance Procedure PC -

Defective Calibration / Test Procedure l PX -

Other (explain) ,

)

H -

Human Actions, Plant Staff l HP -

Failure to follow Procedures HM -

Hisdiagnosis (followed wrong procedure)

HA -

Accidental Action HX -

Other (explain)

M -

Maintenance MS -

Scheduled Preventive Maintenance (including surveillance tests and calibration) 1 MF -

Forced Maintenance (repair of a known failura)

E -

Abnormal Environmental Stress EE -

Electromagnetic Interference EM -

Motsture (spray, flood, etc.) -

EF -

Fire ET -

Temperature (abnormally high or low)

ER -

Radioactive Radiation (irradiation)

EC -

Chemical Reactions EV -

Vibration Loads j EI -

Impact Loads !

EH -

Human-Caused External Event EN -

Acts of Nature I -

Internal (internal to component, piecepart ambient environmental stress)

IC -

Internal to Component, Piecepart IE -

Ambient Environmental Stress U - Unknown i

From Reference B 4, l B-13

, ; l' ' Lkll ! ,: I ' l ; '

Y B

D N .

_ A - D T N A

_ S D .

O

- I R

= P E _

+. 1 l I l I 4 Y

. B 1

D _

._ 18 I N _

I I I I I ,I I l g I 3 A _

T 1

D S

NT G -

. AS N I

ME R ET D U _

D _

= 2

. u I 1 g I I I l I i i I T L E

V

_. l l I I l g E L

I S

S E

R

T

_ ST _

S t GE _

s E DT e o _

t Li _

AM E d M n CM _

I E I

a TD T " m E Y

e HG d TN

_ B D

N A i g

n b

y OI PR Y U HD

__ T r d S u n

_ d a .

g n t 2 w s -

n o B w y b

d n

S i

n n

u r

d t

h u

s t

o n

r u

E R

U G

o a G G t I

F T t s

D D e

_ E r S

SL I 2 3 4 EE T T T T -

. RV TE o t

o t

o o o t t t SL

_. T o

T g

T 2

T 3

T 4

m'.# .

i J}1i1 >

- l

' ~ ' r i, li: i iSji

test interval. This can be seen by considering the four categories of failures resulting from the above double partition, namely: ,

1. Catastrophic failures due to standby stress causes. I

2. Catastrophic failures due to demand stress causes.

3. Noncatastrophic failures resulting in a need to repair due to standby stress causes.

4. Noncatastrophic failures resulting in a need to repair due to demand stress causes.

To provide timely detection of catastrophic diesel generator failures of the category 1 type, frequent demand tests should be conducted. However, frequent demand tests could result in frequent repair outages of the category 2 and category 4 types.

Therefore, the optimum demand test interval is obtained by l balancing the expected diesel generator unreliability due to standby stress caused catastrophic failures with the expected repair outage unavailability due to demand stress-caused require-ments to repair. The actual mix of the four categories of failure types is thought to be diesel generator dependent and therefore plant specific.

Reference B6 provides a more detailed description of the dependence of diesel generator reliability on test interval for various assumed mixes of the above four failure categories.

Reference B-7 also discusses this influence on demand test inter-vals.

Surveillance needs using the above model can be assessed using a data analysis to partition failure modes into the four

categories. Thus, this can only be accomplished for diesel generators with an operating history. A conservative estimate of upper limits on demand test intervals can be obtained by assuming all failure modes are standby stress caused. A data analysis to partition the failure modes will generally result in signifi-cantly larger acceptable demand test intervals than the conserva-tive assumptions would indicate. Engineering considerations, such as corrosion, wear, or fluid stratification, may indicate the need for longer or for shorter test intervals.

Has a surveillance olan for the EDG system been orecared?

The assessment of surveillance needs should be codified in l an EDG surveillance plan. Since this plan is one of the docu-ments that should be reviewed as part of the EDG reliability program review, it should contain evidence that all the above "surveillance needs" issues were considered in the assessment of EDG surveillance. This plan should list the types of surveil-lance to be employed, the intervals between surveillance for each type, and any schedules that will be used to synchronize surveil-l B-15

I lance among diesel generators in the EDG system, or between the diesel generators and equipment in other systems, e.g., with emergency core cooling system (ECCS) pump testing. Each of the above decisions concerning surveillance type, surveillance inter-vals, and surveillance schedules should be accompanied by a description of the rationale and assessments that led to the decisions. Also included in the surveillance plan should be discussion of any considerations for surveillance that result from performance monitoring remirements for the EDG system.

These considerations could include surveillance types or inter-vals that are set solely to support the performance monitoring techniques to be employed. Proposed surveillance should also be reconciled with technical specification requirements and with i any other regulatory requirements or guices for surveillance of r the EDG system.

The EDG surveillanco plan should not be written as a static document. Operational experience could indicate that either more i

or less intensive surveillance than originally assessed is needed. Thus, the surveillance plan should allow for possible changes in surveillance as these changes become necessary or desirable. To accomplish this, the surveillance plan should call '

for periodic reassessments of the surveillance needs of the EDG system. Changes could be based on operational experience with the system or based on research or other plant findings.

B.3 INTERFACES WITH OTHER EDG RELIABILITY PROGRAM REVIEW ITEMS !

Review of the assessment of surveillance needs for the EDG system should be coordinated with the review of the EDG reliability target (Review item A) and review of the performance indicators (Review Item C) that will be used.

The surveillance of the EDG system should support the calculations used to estimate whether or not the relisbility target is being met. The surveil, lance types and schedules should also provide the information used to track EDG performance in a broader context to spot possible degradations or recurring failures.

The following process can be used to ensure that the specification of EDG surveillance as described in the surveillance plan will provide the needed information for estimating whether or not the EDGs are meeting the i j

reliability target and for detecting possible performance degradations that may be occurring:

1. The following data are required to estimate the EDG reliability (sea Appendix A):

e Base operational period and base number of trials involving a diesel generator mission simulation, e Number of catastrophic failures in the base period, e Number of diesel generator attempted starts in the base period and the number of failures to start.

B 16

- \

[

o Total diesel generator operating time (i.e., running time) during the base period and total number of t failures experienced while running.

2. Compose a list of the data necessary for operation of the '

diesel generator performance tracking system (Review ;

Item C), including 1X21 of data and freauency with which it I

should be taken. !

3. Reconcile the lists in 1 and 2 above to obtain the list l l

of data necessary to estimate the EDG reliability relative '

l to the target and the list necessary for detecting possible t performance degradations.

L

4. Match the list in 3 to the information flow that would be generated by application of the EDG surveillance plan. 3 As a further review check, it should ba verified that the review of the data system capabilities (Review Item G) verifies that the reliability program data storage and retrieval system specifies explicitly that all j the data types in the list developed in step 3 above will be stored and are retrievable.

l i

t B 17

i I

l l i I i REFERENCES FOR APPENDIX B !

r i

B-1. E.V. Lofgren, "A Reliability Centered Surveillance Concept for Nuclear [

l Power Plant Standby Safety Equipment: Definitions, Risk Considera- !

tions, and Issues," BNL Technical Report A 3282, December 9, 1986.

B 2. S.M. Wong et al., "Trial Application of Reliability Technology to Emergency Diesel Generators at Trojan," BNL Technical Report A-3282, ;

April 1986.

B 3. K. Hoopingarner et al., "Aging of Nuclear Station Diesel Generators:

4 Evaluation of Operating and Expert Experience (Phase ! Study)," Pacific l Northwest Laboratories, NUREG/CR 4590, Vol. 1. PNL-5832, August 1987.

B 4. K.N. Fleming et al., "Cla:sification and Analysis of Reactor Operating l

Experience involving Dependent Events," :PRI NP-3967, June 1985.

B 5. E.V. Lofgren, "A Potential Method for Identifying Nonobvious Common-i Cause Failure Systems in Nuclear Power Plant Safety Equipment," SAIC

] Report prepared for BNL OSRR Project. 22 October 1986.

i B-6. W.E. Vesely, G.M. DeMoss, and E.V. Lofgren, i

Unavailability and Risk Effective Surveillance Test "Evaluation of Diesel j

Intervals," BNL Technical Report A 3230, May 1986.

B-7. G.M. DeMoss and E.V. Lofgren, "Availability Analysis of Trojan Diesel Generators," SAIC Report prepared for BNL OSRR Project, November 1985.

s

)

l I

)

1 l

j i

j B-18

d APPENDIX C EDG PERFORMANCE MONITORING

. (ReviewItemC) i t

l l

1 j

l i

l l

l r

l C1 - _ - - _ __

l i

l TABLE OF CONTENTS l

em !

i C.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . C 5 t

C.2 CONSIDERATIONS FOR EVALUATING EDG PERFORMANCE i MONITORING. . . . . . . . . . . . . . . . . . . . . . . . C 6 j l C.2.1 EDG Condition Monitoring. . . . . . . . . . . . . . . C 6 l l

C.2.1.1 Example EDG Conditions to Monitor . . . . . . C 6 C.2.1.2 Technical Issues for EDG Condition !

Monitoring . . . . . . . . . . . . . . . . C 12 !

C.2.2 EDG Reliability Monitoring. . . . . . . . . . . . . . C 15 C.2.2.1 Exampic EDG Reliability Monitoring !

Techniques . . . . . . . . . . . . . . . . C 16 i C.2.2.2 Technical Review Issues for EDG :

Reliability Monitoring . . . . . . . . . . C 28 C.3 INTERFACES WITH OTHER RELIABILITY PROGRAM REVIEW ITEMS . . . C 29 ;

t REFERENCES FOR APPENDIX C . . . . . . . . . . . . . . . . . . . . C 30 l i

LIST OF TABLES l

Table ;

C1 Per Shift Cher.ks . . . . . . . . . . . . . . . . . . . . . C-8 '

C-2 Daily Checks . . . . . . . . . . . . . . . . . . . . . . . C 9 l

C-3 Weekly Checks. . . . . . . . . . . . . . . . . . . . . . . C 9 C-4 Emergency Diesel Generator Monthly Test Data Sheet. . . . . . . . . . . . . . . . . . . . . . . C-10 C5 Failure Cause Type Cross Classified with Recent and Earlier Operating History . . . . . . . . . . . . . C 23 1

C3 -

LIST OF FIGURES Fiaure Page C-1 Work Breakdown Structure for Section C.2 "Considerations for Evaluating EDG Performance Monitoring" . C-7 C-2(a) Timeline Showing Hypothetical Repair Occurrence Times. . . . . . . . . . . . . . . . . . . . . . . . . . . C-17 C-2(b) Timeline Showing Ranked Intervals Between Repairs. . . . . . . . . . . . . . . . . . . . . . . . . . C-17

. C-3 Scatterplot and Hypothetical Regression Line For Repair Interval Ranks Versus Repair Occurrence Times. . . . . . . C-21

]

l I

C-4

C.1 INTRODUCTION Emergency diesel generator reliability programs should not only specify EDG surveillance, which provides a "snapshot" view of EDG reliability and operability at periodic intervals, but should also contain provisions for trackina EDG performance, using the results of successive surveillances. In this way trends in reliability and operability, and the engineering conditions related to these, can be observed. This performance tracking aspect of an acceptable EDG reliability program will provide the basis for detecting deteriorating EDG performance and instituting corrective actions before the performance becomes unacceptable (i.e., the EDG reliability falls below the target value).

The performance monitoring aspect of the reliability program provides the necessary information on EDG performance to trigger preventive maintenance actions. An acceptable EDG reliability program will contain adequate

provisions for performance monitoring. These "adequate provisions" are addressed in Section C.2 in terms of the characteristics that the performance monitoring portion of an EDG reliability program should have.

As used herein, performance monitoring is defined to include two types of monitoring activities: condition monitoring and reliability monitoring.

Condition monitoring refers to means by which the state of a component, subsystem, piecepart, or engineering condition is tracked over time or use and includes the criteria for alerting when abnormal conditions or trends are observed. Examples of condition monitoring for EDGs are: tracking lube oil pressure or crank case pressure and temperature; measurement of moisture content in starting air systems; tracking water jacket outlet temperature while the EDG is running; and periodic measurements of electrical contacts to detect and track corrosion or burning. Reliability monitoring for EDGs refers to the tracking of component, subsystem, or piecepart failures or repairs with the objective of providing an alert when the failure / repair frequency, or trends in frequency, indicate a deteriorating condition. Examples of EDG reliability monitoring include *:

direct tracking of repair frequency; tracking of repair frequency for failures of specific types, e.g., by distinguishing among failure severities or failure causes; and tracking repair or failure frequencie.= of EDG subsystems such as the governor or automatic actuation system. Whereas condition monitoring is primarily an engineering activity, reliability monitoring is primarily a statistical activity.

The performance monitoring techniques and issues discussed in this appendix are primarily for tl..e use of plant personnel who are operating the EDG reliability program. They provide plant personnel a means of recognizing and anticipating EDG performance problems. They are included in this dis-cussion of ELG reliability programs to provide the NRC with a basis for assessing the adequacy of this aspect of EDG reliability programs. These monitoring techniques may eventually result in useful information that NRC personnel may choose to use directly to evaluate industry EDG performance.

However, because of the large amount of information that they generate, considerable research effort is required before these component-specific techniques can be adapted to NRC direct use. An interim EDG failure tracking scheme, consistent with the EDG reliability goal as presented in the approach to resolving USI A-44 (Station Blackout), is presented in C-5

l i

l Appendix A. This failure tracking scheme is appropriate for NRC direct use l to monitor EDG reliability performance. ;

1 Section C.2 of this appendix identifies the issues that should be addressed '

in an EDG performance monitoring program in order that the program be acceptable in this regard. Thus, Section C.2 identifies the EDG performance

)

monitoring review items. Section C.2 is partitioned according to the struc- '

ture presented in Figure C-1. Section C.3 identifies those performance l monitoring review items that should be coordinated with review of other review items. l C.2 CONSIDERATIONS FOR EVALUATING EDG PERFORMANCE MONITORING l i

This section identifies the detailed review items for reviewiag EDG I performance monitoring approaches that are proposed as part of an EDG !

I reliability program.

1 C.2.1 EDG Condition Monitor 1na i Condition monitoring of emergency diesel generators refers to the process of I obtaining information about the state of engineering conditions that impact '

EDG reliability. The information may be obtained either directly (for example, direct measurement of the moisture content of the air start system or direct observation of corrosion or burning of electrical contacts) or indirectly (for example, measurement of metallic particles in the lubrica-tion system as an indicator of bearing or cylinder wear or measurement of acoustic vibrations as an indicator of crankshaft alignment problems or bearing wear). To be effective, the condition monitoring program should be applied to engineering conditions that are: (1) characterized by a measurable, precursor condition that is known to be related to an important EDG failure mode, (2) conveniently and practically measured without incurring an inappropriately large EDG outage time, and (3) accurate an;d give a minimum number of false indications.

C.2.1.1 Example EDG Conditions to Monitor Reference C-1 lists examples of conditions to monitor to provide a high degree of assurance of EDG operability. These condition monitoring parameters have been abstracted from this reference and are shown in Tables C-1 through C-4. They represent examples of parameters to monitor at 1 periods of per shift, daily, weekly, monthly, and start /run/ load tests, and yearly or refueling cycle.

The condition monitoring parameters measured and recorded should be subjected to trend analysis so that problem areas can be identified.

Graphical techniques are the preferred method of trending EDG parameters; however, a simple listing of the parameter can sometimes provide insight into a trend. For example, if cylinder temperatures started to rise, it would be readily noticed on a graph; however, it would also be noted if the temperatures were listed in chronological order. '

Ot' ar EDG test requirements should not be overlooked. Some examples include but are not limited to fuel oil, lube oil, and jacket water chemical )

l C-6 )

sO Section C.2 Considerotione for Evoluoting EOC Perf orma nce Monitoring Section C.2.1 Section C.2.2 EDG Condition EDG Rollobility Monitoring Monitoring 9

N Define S##t " C*2*1*1 Section C.2.1.2 Section C.2.2.1 Section C.2.2.2 Exomple EDG Review lesues Define Exompte EDG Review issues Condition for EDG condition Monitoring Conditions to Monitoring Reliability Reliability for EDG Monitor Monitoring Monitoring Reliability Schemes Techniques Monitoring FIGURE C- 1.

WORK BREAKDOWN STRUCTURE FOR SECTION C.2

'CO N SI D E R ATIO N S FOR EVALUATING EDG PERFORMANCE MONITORING,,

l l

I TABLE C-1. PER SHIFT CHECKS i To be performed by an auxiliary operator as part of routine shift walk-through:

I

1. The remote / local start switch in the remote position !

2. The auto / manual start switch in the auto position i

3. The fuel oil level (day tank) l

4. The lube oil level {

5. The jacket water / cooling water expansion tank level '

6. Diesel generator keep warm system Lube oil temperature, pressure Jacket water temperature, pressure l

Soak back pump pressure .

7. Governor setting, automatic or manual

8. Starting air receiver pressure

9. Any fluid leakage should be noted

10. Barring device disengaged

11. Cleanliness of the area

12. Starting air compressor should be checked for overheating

13. L.0. filter D/P

14. F.0. filter D/P

15. Duplex strainer / filters handle should not be in mid position, flow through one filter only

16. Annunciator circuit 1

i C-8

i TABLE C-2. DAILY CHECKS 1

(includes all of the per shift checks plus) l

1. Blowdown air receiver to check for moisture / water accumulated

2. Fuel oil storage tank level ,

3. EDG fire suppression system check )

l

~

TABLE C-3. WEEKLY CHECKS (includes all of the daily and per shift checks plus)

1. Associated circuit breakers / motor controllers Racked in Remote / manual in remote Control fuses installed Power to break verified i Auto / manual in auto Aligned to appropriate power source Fault indicators (flags)

W C-9

TABLE C-4. EMERGENCY DIESEL GENERATOR MONTHLY TEST DATA SHEET (includes all of the per shift, daily, and weekly checks plus the following)

Pre-operational Check List EDG ID #:

Date: Time:

EDG Integrator Reading:

Starting Air Pressure, Receiver A: psig Receiver 8: psig Governor Setting: Automatic / Manual Fuel oil level, Day Tank gallons Storage Tank gallons Engine Cooling Water Expansion Tank level: inches Lube oil filter D/P:

Barring e vice disengaged:

Lube oil :emperature:

Lube oil pressure: psig Jacket water temperature:

Jacket water pressure: psig Duplex strainer in use:

Annunciator circuit check:

l Operational Checklist Time EDG Started:

Method of Starting:

I Postoperational Checklist Time EDG Secured:

l Fuel oil level at end of run: (or amount of F.0. oil used) l Lube oil sump level at end of run: gallons EDG Integrator reading (time):

Other tests as required:

l - starting air compressor operational checks l - alternate power suppl 3 operational checks l - F.O. transfer pump operability checks

- etc.

Filling of the day tank and fuel oil storage tank (if required) should be accomplished immediately after the EDG is secured.

Any special tests associated with the fuel oil system should be l accomplished at this time.

l l

l C-10

l l

f. C-4 (cont.). EMERGENCY DIESEL GENERATOR MONTHLY TEST DATA SHEET Hourly Readings (to be recorded every hour during the test)

Jacket water pressure Fuel rack settings

Jacket water temperature (in) #1 cylinder (for each cylinder)

Jacket water temperature (out) #2 cylinder Jacket water cooler D/T #3 cylinder ,

Jacket water cooler D/P #4 cylinder '

Water pressure to turbocharger #5 cylinder Water pressure from turbocharger #6 cylinder Water temperature to turbocharger #7 cylinder Water temperature from turbocharger #8 cylinder Engine oil pressure #9 cylinder L.0. cooler outlet temperature #10 cylinder ;

, L.0. cooler inlet temperature Turbocharger RPM l L.0. filter inlet pressure Engine RPM L.0. filter outlet pressure KW ;

011 press to turbocharger V91ts Oil temperature from turbocharger Frequency (Hz)

Turbocharger inlet air temperature Amps After cooler air temperature Kilovars After cooler air pressure Alternator winding temperature Alternator bearing temperature l EDG vibration (mils) l Crankcase pressure

1 Cylinder exhaust temperature (for each cylinder)

2 Cylinder exhaust temperature

3 Cylinder exhaust temperature

4 Cylinder exhaust temperature

5 Cylinder exhaust temperature :

-#6 Cylinder exhaust temperature '

I

7 Cylinder exhaust temperature {

8 Cylinder exhaust temperature i

9 Cylinder exhaust temperature

10 Cylinder exhaust temperature Turbine exhaust pressure Turbine exhaust temperature Turbine exhaust backpressure F.0. filter inlet pressure F.0. filter outlet pressure F.0. temperature l

l C-Il

analysis. Fuel oil should be tested when brought on site and analyzed for !

conformance to the appropriate American Society of Testing and Materials l (ASTM) standard. Standard technical specifications currently require this i test every 92 days. Engine cooling water should be analyzed as recommended 1 by the manufacturer (i.e., chromate and antifreeze concentrations). Lube i oil analysis should also be performed in accordance with the manufacturer's instructions and the appropriate ASTM standard. Trending of these ,

parameters should also be accomplished, as they too can provide some i valuable insight. For example, if lube oil analysis shows additional water l accumulation each month, it may be an indication of a leaking lube oil cooler, or if the chromate concentration in the jacket water decreases rapidly, it may be an indication of a jacket water leak. Other component tests may also be required, such as the governor or relay. Those components should be tested as required by the manufacturer, keeping in mind the operability requirements of the EDG. l i

There have also been several recommendations regarding additional testing. l When viewed from the function of an EDG at a nuclear power facility, this i testing may or may not be justified. One such recommendation is circuit l diagnostic testing. Because of the automatic starting circuitry, this may l be an insurmountable task and introduce new and unwanted failure modes to the function of the EDG. Any new testing should be evaluated to ensure that new and unwanted failure modes and mechanisms are not introduced.

Although this appendix has mainly addressed mechanical components, it is not intended to place a lower priority on electrical, instrumentation, or I control systems. In fact, the instrumentation and control systems are considered by some to have the highest incidence of failure among the EDG support systems. For those items, the appropriate tests and checks should be performed as required by technical specifications, manufacturer's I recommendations, etc.

C.2.1.2 Technical Issues for EDG Condition Monitoring The following items should be reviewed to provide assurance that an EDG condition monitoring scheme, established as part of- an EDG reliability program, has the features necessary to be successful.

Have the enaineerina conditions to be monitored been explicitly identified? i The first and most obvious feature that is necessary for a successful EDG condition monitoring program is that the engineer-ing conditions that are to be monitored as part of the program should be explicitly identified. Examples of engineering condi-tions to be monitored were shown in Tables C-1 through C-4.

Although this list represents a good "start" at identifying an EDG condition monitoring program, each plant may wish to insti- l tute its own scheme, in order to treat the particular problems ;

experienced by each diesel. Because there appear to be differences in the reliability problems experienced by different plants, even among those using the same types of diesels, each plant should provide at least a nominal justification for the particular choice of a set of engineering conditions that it will C 12

l monitor. It is not necessary for any plant to monitor all .

engineering conditions identified - only those important ones I that could prevent the EDG from achieving the reliability target.

Listed below are specific questions that should be answered by the diesel generator user:

e Are all key parameters such as temperatures (cooling water, lube oil, bearings, exhaust gases), pressures (cylinders, fuel, lube oil, air), speed, torque, load or vibration levels nonitored?

l e Are there sufficient test points for each parameter? l l

e Is the monitoring equipment properly calibrated and accurate I over time?

I e Is the response of the monitoring equipment rapid enough for adequate correlation of operating changes and parameter variations particularly under test conditions?

e Are the data recorded with a satisfactory frequency and accuracy?

e Are all aclitions of fuel, lube oil, cooling water treatment !

chemicals, etc., recorded accurately (time, type, quantity)?

e Are all fluids (fuel, lube oil, cooling water) sampled at a l sufficient frequency? )

e Are the fluid samples representative (sampling point, I volume, time at which the sample is taken in relation to l other ever ts) and the analyses properly specified? .

e Are all operations of drains, blowdowns, and vents recorded accurately (timc, duration) along with the reasons for these operations?

For a new unit, care should be taken to specify instrumentation and procedurts that meet the requirements of the condition moni-toring pregram.

Have alert levels. or criteria for corrective action. been identified for each enaineerina condition to be monitored?

Alert levels, or criteria for corrective action, should be identified for each of the engineering conditions contained in I

the set to be monitored as part of the EDG condition monitoring program. Alert levels are normally as simple as a minimum and/or maximum value for a parameter or a trend in a parameter. They also include combinations of condition levels (e.g., high crankcase pressure coupled with high temperature). A single engineering condition may have a multiplicity of alert levels, I some of which merely alert the operator that a long-term C-13 j

phenomenon is continuing to progress at some rate toward eventual degradation. An example is the continuous change in acoustic vibration level at a given set of frequencies that may be tied to some wear-out phenomena. The actual "alert" may be a spectrum frequency level whereby the decision may be made, for the sake of prudence, to overhaul a portion of the EDG at the next scheduled reactor shutdown. Thus, the alert may require immediate action, or simply result in a preventive maintenance action at some specified time in the future. Both the alert level value and a simple statement of the probable action to be taken should be presented as part of the condition monitoring plan.

Are there orocedures for conductina the condition monitorina?

The EDG condition monitoring program should be formalized in a o set of procedures that contain checklists for the conditions monitored, monitoring frequencies, alert levels, and action statements for plant use. Examples of condition monitoring checklists were presented in Tables C-1 through C-4. These checklists also implicitly contain the condition monitoring frequency, since there are separate checklists for checks per shift, daily, weekly, etc. Alert levels and action statements would be condition specific and are highly dependent on the expected lag-time between observation of the engineering condition and the EDG failure mode related to the condition; severity of EDG failure by the failure mode related to the observed condition; and EDG repair outage time to correct the observed condition, compared to the repair outage time required if the condition were allowed to proceed to failure. These considerations should be implicit in the condition monitoring procedures.

Has justification been oiven for the monitorina freauencies for the EDG conditions to be monitored? ,

l As previously discussed, the frequencies with which the various l EDG engineering conditions are to be sampled, or monitored, depend on the nature of the conditions and how they are related to the EDG failure mode that is being protected against. These frequen-cies should be set based on the expected lag-time from observing I the failure precursor condition to the subsequent failure mode; whether the observed condition is a direct observation of a condi-tion that will impact reliability, or an indirect observation of a condition that will eventually result in deteriora*.:d reliability; and the severity of the failure if the failure moiu were to occur.

These considerations should be explicitly discussed in the condi-tion monitoring frequency justification.

Has consideration been oiven to the EDG outaae time reauired to perform the condition monitorina?

It is almost always beneficial from the standpoint of EDG availability to incur EDG outage time for the purpose of condition monitoring, which leads to preventive maintenance, in order to C-14 _

~

l avoid the subsequent EDG failures that would be experienced had the preventive maintenance not been performed. However, it is still incumbent upon the licensee to ensure that EDG outages for condition monitoring and preventive maintenance do not become i excessive. That is, the licensee's condition monitoring program should reflect the tradeoff on EDG reliability (as calculated in Appendix A) between preventive maintenance and EDG failure (and subsequent corrective maintenance).

Are there provisions in the EDG reliability orocram to alter the condition monitorina performed in response to updated information or monitorina techniaues becomina available?

It is inevitable that the appropriate set of monitored parameters and frequency of monitoring will change over time. This is true for two reasons: (1) because of wearout and aging mechanisms, I the important EDG failure causes are expected to change with time, and (2) additional failure information, and improved techniques for condition monitoring, will almost certainly result in a changed perception of the appropriate condition monitoring for an individual EDG. Therefore, it is important that the EDG reliability program have provisions for periodically reviewing and updating the condition monitoring performed on the diesel generators.

Is the croposed condition monitorina plan supported by the proDosed EDG surveillance?

Since condition monitoring is included as a type of surveillance, the review of EDG condition monitoring should be coordinated with review of EDG surveillance to ensure that there is absolute con-sistency between the surveillance planned and the condition moni-toring planned. '

C.2.2 EDG Reliability Monitorina The purpose of reliability monitoring is to provide an overall, summary-type l reliability assessment of the EDG or of individual EDG subsystems. Whereas I condition monitoring is primarily an engineering activity, reliability ,

monitoring is primarily a statistical activity. Reliability monitoring is '

not intended to be a replacement for condition monitoring. It is intended to augment a condition monitoring program. It is necessary to consider using reliability monitoring in conjunction with condition monitoring because, while condition monitoring provides a defense against individual, identified failure modes that have detectable precursor conditions associated with them, reliability monitoring provides an overall summary measure of the total impact of All failure modes, including those associated with the concurrent effects of several off-normal conditions operating together to produce EDG failure, which might not be detected if only condition monitoring were used.

Reliability monitoring can be applied to the entire EDG system or to individual subsystems such as the air start subsystem or the governor subsystem. It can also be applied to classes of pieceparts, e.g., the C-15

l l

overall reliability of all small valves, or all electrical contacts used in the emergency electric power system can be monitored. An application such as this could have value as a monitoring system designed to cetect aging l mechanisms that could have common cause implications. At the very least, !

the licensee should use the sumnary failure tracking scheme described in .

Appendix A for NRC overview use. This failure tracking scheme is a type of i reliability monitoring. 1 The following subsections present some examples of reliability monturing techniques to further define their use (Section C.2.2.1) and presen' the issues that should be addressed to review the adequacy of reliability T,cai-toring schemes proposed as part of an EDG reliability program (Section C.2.2.2).

C.2.2.1 Example EDG Reliability Monitoring Techniques The following presents a sample of some statistically based techniques that could be used to track EDG reliability or to indicate degradation or improvement in performance over time. A more detailed discussion of these !

techniques is provided in Reference C-2. Techniques of a similar nature are discussed in Reference C-3. It is emphasized that these tracking techniques are intended only as examples to indicate the nature of such techniques and how they could be used and are not intended to be a recommended set of I approaches to reliability monitoring. Indeed, since reliability monitoring is a developing art, there may be approaches that are superior to the ones outlined here; the techniques presented herein are not meant to be review standards.

Tests to indicate if the failure or repair frecuency has chanced over time Repair frequency has been identified as a reasonable measure of equipment aging or degradation (Ref. C-3) in some cases, especially the frequency of unscheduled repairs. Therefore, an indication of either a gradual or an abrupt change in the EDG repair frequency may be associated with 'a reliability problem that should be corrected. Change in repair frequency can be detected by plotting the occurrence dates of the repair actions on a timeline, as indicated in Figure C-2(a). Several methods can be used to provide a numerical basis for evaluating whether or not any trends noted in the repair occurrence-time plot are likely to represent actual changes in the repair frequency.

If the repair occurrence-time plot seems to indicate a somewhat abrupt change in the repair frequency, a Mann-Whitney test can be used to test the statistical significance of the indicated change. This procedure is described in detail in Reference C-2. An abbreviated outline of the process is given below.

1. Rank the times-betveen-repairs, as shown in Figure C-2(b),

from smallest (rank 1) to largest. (Ref. C-2 gives the procedure for tied ri.nks.)

C-16

6 i

t i I I I I I I I i 1 0 6 9 13 14 16 19 20.5 22.5 24 26 Relative Occurrence Time FICURE C-2(a). TIMEll NE SHOWING HYPOTHETICAL REPAIR OCCURRENCE TIMES

?

O

, t Ranks 9 7.5 8 1 5 7.5 2.5 5 2.5 5 l l l 1 1 I I I I I I O 6 9 13 14 16 19 20.5 22.5 24 26 Occurrence" Times FIGURE C-2(b). TIMELINE SHOWING RANKED INTERVALS BETWEEN REPAIRS I

.~

2. Partition the segment of repair history under analysis into two parts, representing "recent operating history" and "earlier operating history."

3. Obtain the Mann-Whitney statistics for the means and vari-ance of the means of the ranks in each partitioned part.

4. Since the mean ranks are approximately normally distributed, the unit normal distribution is used to test if the mean of the ranks in "recent operating history" is significantly different (in a ctatistical sense) from the mean of ranks in "earlier operating history."

A statistically significant difference in the rank means, as indicated in l the test in step 4 above, would lend quantitative support to the l observation that the repair frequency has changed between early and recent operating history, i

A numerical example is now given to illustrate the Mann-Whitney test for differences in EDG repair frequency between recent operating history and earlier operating history.

1 Examole Mann-Whitney Test to Compare Current Repair Frecuency to Past Freauency

1. Assume that the repair and failure events fall on the timeline with relative event occurrence dates as shown below. The intervals between events are ranked, also as shown below, from smallest to largest.

rwis 14 5 11 13 9.5 12 3 '5 7 8 2 9.5 1 5 eet 0 3.24.2 6.3 8.8 10.3 12.5 13.2 14.2 15.3 16.7 17.3 18.8 19.2 20.2

_ a dates partitim 3

(x10 ) ,

2. The intervals between events are ranked, as shown above. For ties, the average rank is used. In the above, there are two sets of ties.

The first occurs at ranks 4, 5, and 6, all of which are tied. The average is (4 + 5 + 6)/3 = 5. Therefore, rank 5 is used for all of these, and the ranking continues with rank 7. The next tie occurs at ranks 9 and 10. The average is (9 + 10)/2 = 9 1/2, and rank 9 1/2 is used for both of these.

3. The correction to the variance for tied ranks is computed from the above as:

2 E Tj = (ni-1)ni(ni+1) + (n2-1)n2(n2+1) j=1

= (3-!)3(3+1) + (2-1)2(2+1)

= 30 C-18

4. The analysis segment is partitioned at the arrow in step 1 above. The times between events to the right of the arrow are to be considered "recent operating history." This partitioning is entirely at the user's discretion but should be done so that roughly half the event intervals occur on either side of the partition. In this case, there are 8 intervals in "recent operating history," and 6 intervals in "past ;

operating history." Thus: 1 N=8+6

= 14 :

n=8

5. The Z statistic (unit standardized normal) is estimated using:

1 z= R - (N + 1)/2 y#R' I To estimate z, the terms R and oR must be estimated. !

6. R is estimated by summing all the ranks contained in the "recent l operating history" segment, and divioing by the number of segments: ,

R = (3 + 5 + 7 + 8 + 2 + 9 1/2 + 1 + 5)/8

- 40.5/8

- 5.0625

7. a 2 is estimated by:

g2 ,N-n R = N(N2 - 1) - E3 T3 -

12Nn N-1 where Ej Tj was calculated in step 3 to be 30 a2 p ,14(142 - 1) - 30 . 14 - 8 12(14)(7) 14 - 1

= 1.05965

8. Calculate z using the expression in step 5 above:

z - 5.0625 - 7.5

,/1.059655

= -2700 - (.461538) 1.176

- -2.3679 C-19 I

9. Obtain fzfor z - - 2.3679 from a standard normal distribution function.

fz .009 Since fz < .02, the Kruskal-Wallis correction should not be applied.

10. The interpretation is that there is only a 0.9% chance of getting as many small ranks as observed in "recent operating history," as compared to past operating history, if the process had nat changed. Therefore, it is concluded that the process has changed and that the repair (or failure) frequency has increased.

If the repair occurrence-time plot seems to indicate a aradual change with time (e.g., a gradual degradation process), a somewhat different trending approach may be appropriate (although the Mann-Whitney test could still be ,

used for this). Figure C-3 shows the between-repair interval ranks in i Figure C-2(b) plotted versus their occurrence . times. A statistical ;

regression analysis could be performed on the data shown in this plot, and i the slope of the regression line tested for significance. A statistically significant negative slope would indicate an increase in repair frequency over time. It is not necessary to partition the analysis segment into "recent" and "earlier" operating history to perform this test. Reference C- l 2 also discusses this trending technique in detail. Caution should be used l in accepting the significance level of the slope test since the normality assumptions usually made for regression analysis do not exactly apply in this case. However, the test can be used as an indicator of a trend in the repair frequency.

A test to indicate chanaes in EDG failure cause This reliability tracking test provides a way of detecting if the type of failure cause experienced by the EDG has changed over time. The test is fo'r a change in the orocortion of repair actions in response to demand stres's causes, compared to those due to standby stress causes; the change does not necessarily result in a change in the freauency of EDG repair actions as a whole, although it could. Such a change could come about due to wearout or aging mechanisms, or as a result of instituting a change in preventive maintenance, which predominantly affects one failure cause type.

This tracking scheme uses a normal approximation to the binomial to test if the relative proportion of repairs due to standby stress or demand stress failures has changed between "recent operating history" and "earlier operating history." Thus, the history of EDG repair actions should be par-titioned into these two parts, as in the Mann-Whitney test. Also, each repair action should be tagged as being standby stres. caused, or demand stress caused.

Reference C-2 contains a detailed description of this tracking scheme. The procedure is briefly summarized in the steps below:

1. The EDG repair history to be analyzed is partitioned into "recent" and "earlier" operating history.

C-20

_ =

t i

1 Hypothetical Regression Line 10 8_ ,

6_

1 .

1 4.

- a n 2, b ,.

1 I I t i i i g g g ;

O 6 9 13 14 16 19 20.5 22.5 24 26 l

Occurrence Time 3

I i

FIGURE C-3. SCATTERPLOT AND HYPOTHETICAL REGRESSION LINE FOR REPAIR INTERVAL RANKS VERSUS REPAIR OCCURRENCE TIMES i

i

2. The number of standby-stress-caused and demand-stress- caused repair events are recorded in each segment of operating history. A cross-classified table is developed, as shown in Table C-5.

3. The proportion of demand-stress and standby-stress-caused repair actions in "recent" and "earlier" operating history are statistically j tested using the normal approximation to the binomial (Ref. C-2) to !

indicate if these proportions have changed between earlier and recent !

operating history. ,

A numerical example of using the normal approximation to the binomial to I test for a change in failure cause type between operating history and j

earlier operating history is given below

Examole Acoroximation to Binomial Test for Chances in Failure Cause !

o 1. Partition the analysis segment into "recent operating history" and i

"earlier operating history.'

l 2. Count standby-stress-caused failures and demand-stress-caused failures in each partitioned part of the analysis segment. For example:

)

l Segment 1 Segment 2 )

(Earlier (Recent l Operating Operating History) History) Totals ;

Standby Stress 10 - nit 13 - n,i '

3 - n21 ,

Demand Stress 6 - n12 15 - n22 21 - n.2 TOTALS 16 - ni, 18 = n2. 34 - n 1

1 3. Estimate proportion of standby-stress failures to total failures as 13/34 .382353 - G

4. Get smallest product of four combinations nj, n,j/n - P If P > 4, normal approximation is acceptable. For example, smallest nj n j/n - (16)(13)/34 - 6.12, which implies that the approximation is acceptable.

l l

C-22 .

I I l

-,2 , .-me . -J A + - -

TABLE C-5. FAILURE CAUSE TYPE CROSS-CLASSIFIED WITH RECENT AhT EARLIER OPERATING HISTORY Earlier Operating Recent Operating History History Totals o

Repairs of n = Nu ber of n 21

= Number of n = Total number 11 *3 standby stress standby stress standby stress of standby caused failures repairs in repairs in stress or conditions earlier recent repairs operating operating history history Repairs of n 12 = Number of n = Number of n demand stress demand stress 22 demand stress *2 = Total number of demand caused failures repairs in repairs in stress or conditions earlier recent repairs operating operating history history n = Total number n = Total number. n = Total number I* 2.

of repairs of repairs of repairs Totals in earlier in recent of all types operating operating history history l

l, i

1 l

C-23 J

5. In general ps = 1 - 4[(nil /ni, - 1/2ni,) - (n21 n2.

/ + 1/2n2.)

e

( M8 (1-6) (1/ng, + 1/n2.)

pd = 4 (n12/DI. + 1/2ni,) - (n22 /n2. - 1/2n2.).

1

( MS (1-8) (1/ni, + 1/n2.)

6. Calculate the statistics:

ps=1-4[(10/16-1/2(2x16)) - (3/18 + 1/2(2x18))

( M8 (1-6) (1/16 + 1/18) ps = 1 - 4(3.416) pd=1-4[(6/16+1/(2x161) - (15/18 - 1/(2x181)

( M6 (1-8) (1/16 + 1/18) pd = 4(-2.341)

7. and p indicate significant changes in standby and demand stress psfailures. dPerform table lookup for significance.

ps = .00032 (is significant at the 0.032% level) pd = .0084 (is significant at the 0.84% level)

Both indicate a definite change. Thus, standby-stress-caused failure frequency has decreased, and demand stress-caused failure frequency has increased.

A trackino scheme for EDG repair outace time unavailability l

Tracking the proportion of time that an EDG spends out-of-service for !

repairs can indicate that EDG performance is degrading with age or wear (increasing outage time unavailability) or that repair and maintenance practices are becoming more effective (decreasing outage time unavailability). A technique for tracking EDG repair outage time unavaila-bility is presented in Reference C-2. This technique is summarized in the steps below:

1. Display the segment of repair history on a timeline that shows when each repair action was initiated.

2. Tag each repair or maintenance act by the EDG outage time required to

, complete the repair and restore the EDG to operable status.

C-24

l

3. For the jCgpute a running estimate of repair outage unavailability. .

repair action, the estimate of repair unavailability is: I gj - r /Tj where gj Estimate time l of the jtgf EDG repair.repair unavailability at the I rj = Outage time required to complete the jth repair.

Tj = Interval between the j-I th and jth repair events. I

4. Plot the repair outage time unavailabilities (qj's) versus the repair occurrence times.

5. Regress the repair outage times on repair occurrence times for the S above plot and test slope for statistical significance.

l A statistically significant slope (either positive or negative) would i provide quantitative substantiation to the argument that EDG repair outage time unavailability is changing with time. Of course, the interpretation of this change is not inherent in the statistics. Engineering judgment and other information should be used to interpret the change in terms of relia-bility or maintainability changes.

A numerical example illustrating use of scatterplots and regression analysis l l to indicate if there is a change in EDG repair outage unavailability over I time is given below.

Example Repair Unavailability Trackina 00 tion

1. Assume the following set of repair actions constituting the analysi's segment. The relative outage occurrence times, and the outage time's are displayed on the timeline below.

N 18 12 24 8

30 16 12 1

.s 4 1 2 8 .5 3 emit 0 3.2 4.2 6.3 8.8 10.3 12.5 13.2 14.2 15.3 16.7 17.3 18.8 19.2 20.2 omma tinus 3

(x10 )

2. The first step is to estimate the unavailability at each of the outage l l occurrence times. These are shown in the table below: )

l C-25 ,

i

Interval Unavailability Unavailability Occurrence Interval Lenath Calculation (x.001) (x.001) Time 0 - 3.2 3.2 12/3.2 3.75 3.2 3.2 - 4.2 1.0 24/1.0 24.0 4.2 4.2 - 6.3 2.1 8/2.1 3.8 6.3 6.3 - 8.8 2.5 30/2,5 12.0 8.8 8.8 - 10.3 1.5 16/1.5 10.67 10.3 10.3 - 12.5 2.2 12/2.2 5.45 12.5 12.5 - 13.2 0.7 1/0.7 1.43 13.2 13.2 - 14.2 1.0 0.5/1.0 0.5 14.2 l 14.2 - 15.3 1.1 4/1.1 3.64 15.3 l 15.3 - 16.7 1.4 1/1.4 0.71 16.7 l 16.7 - 17.3 0.6 2/0.0 3.33 17.3 1 17.3 - 18.8 1.5 8/1.5 5.33 18.8 I 18.8 - 19.2 0.4 0.5/0.4 1.25 19.2 l 19.2 - 20.2 1.0 3/1.0 3.0 20.2 i

3. The next step is to plot the repair unavailabilities (column 4 of the 1 above table) versus the repair occurrence times (column 5 of the above l tabl e) . The plot is shown below. !

30- l 28-26-24- *

{

22- '

20-18-16- .

14-12- .

10- .

8 - I 2 .

0 3.2 4.2 6.3 8.8 10.3 12.5 13.2 14.2 15.3 16.7 17.3 18.8 19.2 20.2

4. The next step is to use the regression module to estimate the best linear fit through the scatterplot. A hypothetical regression line is also shown in the figure above.

5. Significance tests would be performed as part of the regression analysis.

The possibilities are indicated in the following table.

C 26

l l

l Positiw sicpe plagstiw sicse

>13 leo trwd too t wd iso trwd (1 3 , >ft trtHcotim of a trwd Irdicaticri of ircrease in !rtHcaticri of chcrease in eunir treellaNtity remic timeitaMtity

<11 Strwg irdicatie of Strug irdicstlan of rupsir Strcrg irdication of repair rumir t% ierriesten timellaMtity ferrimsten treef tWtity amergesire

6. If desired, smooth over several repair actions, and repeat steps 2 through 5.

Smg.othina over 2 ReDair Actions In = 2) 1 36 38 2B 1.5 5 10 3.5 0 4.2 8.8 12.5 14.2 16.7 18.8 20.2

7. Table of intervals, unavailabilities, and occurrence dates for the I smoothed data.

Interval Unavailability Unavailability Occurrence Interval Lenath Calculation (x.001) (x.001) Time 0 -

4.2 4.2 36/4.2 8.6 4.2 4.2 - 8.8 4.6 38/4.6 8.3 8.8 8.8 - 12.5 3.7 28/3.7 7.6 12.5 12.5 - 14.2 1.7 1.5/1.7 0.9 14.2 14.2 - 16.7 2.5 5/2.5 2.0 16.7 16.7 - 18.8 2.1 10/2.1 4.8 18.8 18.8 - 20.2 1.4 3.5/1.4 2.5 20.2

8. Plot repair unavailability (column 4) versus occurrence time (column 5).

8- ,

o 6-4-

2- .

0 4.2 8.a 12.5 14.2 16.7 18.8 20.2

9. The regression line and significance tests are performed on the smoothed data scatterplot as they are on the unsmoothed data.

C-27

The above tracking techniques illustrate that methods sLq exist to perform reliability monitoring. Each licensee should evaluate what reliability monitoring applications and techniques are appropriate for their EDGs.

There are a set of issues that should be addressed to accomplish this deter-mination. These are discussed in the next section.

C.2.2.2 Technical Review Issues for EDG Reliability Monitoring The following presents items to review to provide a degree of assurance that EDG reliability monitoring established as part of an EDG reliability program and in conjunction with a condition monitoring program has the features necessary to be successful.

Has the reliability monitorina been directed toward EDG subsystems that have historically been the major contributors to EDG unreliability?

~

Appendix I lists diesel generator subsystems, by diesel !

manufacturer, that have historically been major contributors to EDG unreliability. Reference C-4 also contains an analysis of l EDG subsystem reliability by manufacturer. Also listed are ;

those subsystems (by manufactu.er) that have proved reliable in I standby operation. Unless plant experience indicates otherwise, I reliability monitoring should be directed toward the major '

contributors to EDG unreliability. Reliability monitoring should i also be directed toward EDG subsystem failures or problems that '

the plant has experienced in the past unless these problems have been corrected. Review of the EDG reliability monitoring approach should verify that the monitoring effort is being directed toward those areas where it is most likely to be needed.

Does the EDG surveillance plan suoDort the monitorina plan?

Tracking information that will be used to monitor EDG relia- -

bility should come from surveillance of the EDGs. Therefore, the planned surveillance of the EDGs should support the require- l ments of the reliability monitoring plan. Information necessary I to monitor the reliability of the EDGs should be obtained as l part of the surveillance of the EDGs. Review of the reliability monitoring approach should be coordinated with review of surveil-lance of the EDGs to ensure that there is consistency between the information required for monitoring and information likely to be generated by surveillance.

Are there clear orocedures and assioned responsibilities for implementina, conductino, and chanaina the proposed EDG relia-bility monitorina scheme?

The procedures to be used to monitor EDG reliability, including the s'.atistical procedures that will be used as alert levels, should be defined in the reliability program documentation.

Responsibilities for implementation and operation of the reliability monitoring activity should be clearly spelled out.

Periodic review of EDG reliability monitoring is suggested as a C 23

way of introducing changes to the program to provide more, less, or different monitoring. Review of a licensee's reliability monitoring program should verify that there are provisions for adapting the monitoring to the (possibly) changing characteristics of EDG performance.

C.3 INTERFACES WITH OTHER RELIABILITY PROGRAM REVIEW ITEMS Review of the planned parformance monitoring to be conducted as part of an EDG reliability program should be coordinated with review of the proposed surveillance of the EDGs. Both condition monitoring and reliability monitoring involve trending data obtained during EDG surveillance. There should be a correspondence between the information obtained from EDG surveillance and the information needs of the condition and reliability monitoring schemes.

. Both condition monitoring and reliability monitoring require comparing current performance to past performance. Therefore, sufficient information concerning past performance should be stored to allow these comparisons to be made. This will impact the needs of the data storage and retrieval system that will support the EDG reliability program. Thus, review of the performance monitoring proposed for an EDG reliability program should be coordinated with review of the data storage and retrieval capabilities that are to support the program to ensure that these capabilities are adequate.

Since the primary function of condition monitoring is to support the EDG preventive maintenance program (i.e., to trigger preventive maintenance) review of EDG condition monitoring should be coordinated with review of the preventive maintenance policy (Review Item D) to ensure that there are no disjointed aspects of either of these features of the reliability program.

The EDG reliability monitoring and alert levels should be consistent with the EDG reliability target (Appendix A). Therefore review of EDG relia-bility monitoring should be coordinated with review of the Appendix A issues.

Finally, since condition and reliability monitoring require a coordinated, planned effort, review of performance monitoring of EDGs should be coordinated with review of the management of the reliability program to ensure that EDG performance monitoring will be adequately managed.

C 29

I REFERENCES FOR APf DIX C l C-1. K. Hoopingarner et al., "Aging of Nuclear Station Diesel Generators:

Evaluation of Operating and Expert Experience (Phase I Study)," Pacific l Northwest Laboratories, NUREG/CR-4590, Vol. 1, PNL-5832, August 1987. l C-2. G.M. DeMoss et al., "Component Reliability Parameter Studies (CRPS);

System Architecture Document for Computer Assisted Reliability Data System (CARDS)," Report prepared for EPRI by SAIC, December 30, 1987.

C-3. W. Vesely and A. Azarm, "System Unavailability Indicators," BNL l Technical Report A-3295, 30 September 1987 (to be published). l 2

C-4. R. Battle and D. Campbell, "Reliability of Emergency AC Power Systems

, at Nuclear Power Plants," Oak Ridge National Laboratory, NUREG/CR-2989, ORNL/TM-8545, July 1983. ;

4 l

l l

C-30

\

i l

l l

1 I

l l

APPENDIX D EMERGENCY DIESEL GENERATOR MAINTENANCE PROGRAM (Review Item D) 3 i

l l

i i

l I

D-1

TABLE OF CONTENTS Elat D.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . D5 0.2 ISSUES TO CONSIDER WHEN ADDRESSING AN EDG MAINTENANCE PROGRAM . . . . . . . . . . . . . . . . . . . . . . . D-5 D.3 EDG MAINTENANCE PROGRAM INTERFACES WITH OTHER REVIEW ITEMS. . . . . . . . . . . . . . . . . . . . . D-8 l

l l

D3 l

1

I D.1 INTRODUCTION The objective of this appendix is to define the elements of reliability focused maintenance that are important for an EDG reliability program. EDG maintenance plays an important part in the achievement of a reliability i

target but is not a self-cceained activity. Maintenance actions should be driven by surveillance and performance monitoring results. All required elements of a maintenance program should exist to some extent in an existing plant maintenance program. However, to be an effective part of an EDG reliability program, EDG maintenance should be based on reliability considerations and actively interface with the other elements of the relia- ,

bility program. The maintenance policy is needed for a satisfactory preven-tive maintenance program, an acceptable spare parts inventory, correctly prioritized responses to problems, and the input of needed data for other review items such as failure analysis and root cause investigations.

A great deal of published material refers to or describes reliability centered maintenance programs. The descriptions can be very complex and somewhat prescriptive approaches to a program that really has a relatively straightforward function. In order for a maintenance program to function as an effective part of a nuclear power plant EDG reliability program, it should include:

Prioritization of maintenance action based on failure cause, 4

e root cause, fault severity, detectability, anticipated repair time, and likelihood of occurrences repeating or the severity increasing. ;

e Planning of preventive maintenance based on reliability characteristics of EDG subsystems and components, 1

e A proactive spare parts administration policy that includes a prioritization based on the conditions mentioned in the first bullet. ',

Section 0.2 of this appendix discusses the technical issues that should be considered in reviewing an EDG maintenance policy. Section 0.3 describes the interfaces between the EDG maintenance program and other items of the !

reliability program.

l 0.2 ISSUES TO CONSIDER WHEN ADDRESSING AN EDG MAINTENANCE PROGRAM The purpose of this section is to identify the issues that should be addressed when reviewing the planning and implemention of EDG maintenance.

The issues are not specific to EDGs, and probably are addressed to some extent in existing maintenance programs. The identified issues are designed to ensure that maint"ance is focused toward improving EDG reliability. The

major thrust of tF- ecti"e maintenance and unscheduled preventive main-tenance policy sh w ._ ue prioritization of action in dealing with problems.

l D-5

Has a distinction been made in the response.io problems based on the problems severity?

The major consideration in determining the level of response to a failure or other off-normal condition should be the severity of the problem. Among the data recorded upon occurrence of a failure or abnormal condition (see Appendix G) is the severity of that condition, which can be categorized as catastrophic, degraded, or incipient. All catastrophic failures will get immediate maintenance attention, directed toward correcting the failed condition, so that the EDG can be retested and restored to its standby condition. Failure cause analysis, root cause analysis, and other reliability program activities should not be allowed to significantly delay returning a failed EDG to service but are still required to understand the failure cause and how to prevent its recurrence.

The response to degraded and incipient failures is not as clear as the response to catastrophic failures. As failures occur, they should subjectively be evaluated for the potential of lead-ing to a catastrophic failure and for the potential long-term adverse effects of operating a diesel with the failure or abnormal condition present. A condition classified as incipient, such as a drop per minute lube oil leak at a flange (during EDG operation), may have essentially no chance of leading to a catas-trophic failure. The only adverse effect may be correctable by occasionally wiping up an oily area. A problem like this may be better off left alone until the next reactor or diesel outage.

Other incipient or degraded coaditions may have long term impli-catior.s (e.g., high vibration levels) or have a high potential for leading to a catastrophic failure (e.g., governor oil leak).

Engineering judgment and experience indicate that these condi-tions should be repaired promptly. ,

Has a distinction been made in the response to oroblems based on the expected repair outaae time?

This distinction is closely related to the first issue of this section. If the condition severity does not indicate a problem requiring immediate action, the expected outage time to repair should be considered. A slightly reduced risk of catastrophic failure is sometimes not worth an extensive EDG outage for maintenance, especially during reactor operation. Repair outage time should be considered when planning maintenance and should be reduced when possible by staging tools and spare parts and by repairing several conditions during a single diesel repair outage.

D6

Is preventive maintenance adeouately focused?

The purpose of preventive maintenance should be to reduce the number of catastrophic failures and to reduce the long-term degradations due to aging and wearout. In order to accomplish this, preventive maintenance can be keyed by:

o Calendar time o EDG runtime o Number of EDG starts e Response to condition monitoring or e A combination of 2 or more of the above. l The preventive maintenance tasks themselves should be determined based on systematic consideration of subsystem and component functions, the way functions can fail, and priority-based consideration of safety, reliability, and economics to identify applicable and effective preventive maintenance.

l Preventive maintenance should be scheduled to minimize the EDG :

downtime during reactor operations. The reliability program should allow some flexibility in scheduling preventive maintenance so that a preventive maintenance that is required by one of the keys can be evaluated and if possible postponed until the next reactor outage.

Does the maintenance orocram suooort the failure cause and root :

cause analysis?

The elements of failure cause and root cause investigations are ',

described in detail in Appendix E. The maintenance policy should be supportive of these investigations by directing activity to look for indications of failure causes and potential failure causes. During corrective maintenance and preventive maintenance, all abnormal conditions should be reported and documented into the data storage system (Appendix G) for future use. Any failed or degraded pieceparts that may be important in a detailed inves-tigation should be saved until the analysis is closed out.

Does the soare parts system suDDort DreventiVe and Corrective maintenances?

The utilities should periodically evaluate their spare parts requirements to ensure that they adequately support all main-tenance activities. Two types of evaluations should be performed on the spare parts support. The first is to measure the responsiveness in supplying the necessary parts for unscheduled and scheduled preventive and corrective maintenances. This is a relatively simple evaluation to perform regularly. The second is a more detailed evaluation of the spare parts inventory. The D-7

evaluation should take into account potential. component failures that will lead to catastrophic engine failures, the likelihood of the components failing, and the difficulty of repair. The empha-sis should be on stocking critical instrumentation and control (I&C) components because they cause many catastrophic failures, but the repairs are usually relatively simple. Major mechanical parts, such as the casing or head, probably cannot be replaced within the allowed outage time by onsite personnel, so there is little gain to having these parts as ready spares.

D.3 EDG MAINTENANCE PROGRAM INTERFACES WITH OTHER REVIEW ITEMS The key to the success of the maintenance program lies in the successful interfaces with the other reliability program elements. One of the key functions of a reliability program is to deal with problems, failures, and other off-normal conditions so that they do not recur or lead to catastrophic engine failures. The maintenance program is central to this function, j

In carrying out actions in the maintenance program, one should recognize that the policy is driven by the target reliability defined in Review Item A. However, specific maintenances (as opposed to the policy) should not be l driven by these targets but by the performance indicators identified in ,

Review Item C. Analysis of the performance indicators should indicate a i preventive maintenance that will reduce the likelihood of adverse perfor- '

mance and indicate the frequency with which maintenance of various types should be performed.

i The interface with the failure analysis work described under Review Item E l is vital for successful failure analysis. The maintenance people who actually tear down machines for the repairs should be involved with and aware of the failure analysis and root cause investigation. , j Another major interface is with the data collection system. It is extremely' important that all insights, including suspicions, of the maintenance personnel are entered into the data system. The information may seem minor at the time, especially if the condition being repaired is seemingly minor or routine, but the performance monitoring work or a failure investigation may need this information to spot trends or focus on the root cause of a problem. Recording detailed information also provides added assurance of meeting the criteria for problem closecut (Review Item F). Conversely, the experts using the information from the data system for failure analysis, unreliability reporting, performance monitoring, and problem closecut should ensure that maintenance personnel are trained to include the pertinent data in the collection system. In addition to being able to enter data into the system, maintenance personnel should be able to retrieve historical repair information from the data system, l i

D-8

APPENDIX E FAILURE ANALYSIS AND IDENTIFICATION OF CORRECTABLE CAUSES (Review Item E)

E-I

, t I

l TABLE OF CONTENTS I

4 P_ng :i t

E.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . E-5 [

E.2 ISSUES TO CONSIDER WHEN REVIEWING FAILURE ANALYSIS AND l ROOT CAUSE INVESTIGATION . . . . . . . . . . . . . . . . E-5 !

E.3 EXAMPLES OF FAILURE ANALYSIS . . . . . . . . . . . . . . . E-13 E.3.1 E x ampl e 1 . . . . . . . . . . . . . . . . . . . . . I E-13 E.3.2 E x ampl e 2 . . . . . . . . . . . . . . . . . . . . . E-13

) E.3.3. Example 3 . . . . . . . . . . . . . . . . . . . . . E-20 ,

E.4 INTERFACES WITH OTHER REVIEW ITEMS . . . . . . . . . ... E-20 LIST OF FIGURES l Fioure -

E.1 Systematic Root Cause Approach . . . . . . . . . . . . . . E-9 l

}< E.2 Example 1 ........................ E-14 l E.3 Example 2 ........................ E-19 .

j E.4 Example 3 ........................ E-21 l

k E-3 d !

1 I

E.1 INTRODUCTION The purpose of this critical review item is to ensure that the licensee aggressively and systematically reduces EDG problems to correctable causes.

Substantial long term benefits can be derived from the identification of the root causes of problems and the development of solutions that either eliminate these causes or minimize their impact. Diesel generator failures cannot be entirely avoided due to the complexity of these units and the type i of service experienced in nuclear power plants. However, systematically ;

eliminating the root causes of problems will improve diesel generator reliability.

Diesel generator problems requiring investigation and correction can be of 1

several types. They include catastrophic failures, unsatisfactory condi+ ons detected through surveillance or monitoring, or damage and other physical conditions found during maintenance work.

The investigation of these diesel generator problems can be carried out to various levels that, in most cases, are not clearly separated. In order to show a progression in the degree of detail, a distinction is made whenever ,

' possible between failure analysis, which covers the entire range, and its -

subset, root cause analysis. Failure analysis starts from the most apparent symptoms and progresses to the determination of the underlying failure or incipient condition. The root cause analysis attempts to find the cause(s) l of the underlying failure or incipient condition that could be related to ;

design or a procedure used in operation or maintenance. !

In general, the likelihood of performing a successful analysis is increased by the availability of a large amount of meaningful data. The quality uf those data and the manner of their retrieval are critical to their usefulness. The availability of data to address each issue identified in Section E.2 should be considered by the reviewer as the issues are reviewed.

A root cause investigation should be conducted very methodically since th4 root may be several levels below the visible symptoms, or there may be several synergistic causes, some more predominant than the others. The methodical approach to investigations is stressed in the examples of Section E.3.

E.2 ISSUES 10 CONSIDER WHEN REVIEWING FAILURE ANALYSIS AND ROOT CAUSE IN-VESTIGATION The specific issues that should be addressed when reviewing the failure analysis and root cause investigations of a reliability program are presented in this section.

Does the licensee collect and incoroorate the necessary informa-l tion for a failure and root cause investication?

! One or more of the following elements may be required for a successful root cause analysis of an identified problem:

4 E-5 4

e All parts removed from the unit in connection with the repairs necessitated by the problem.

e Reports on the condition of all adjacent parts and other visual observations made during disassembly (including, for example, verbal descriptions, schematics, and photographs).

e Clearance and alignment readings, measurement of piece.

parts (with properly calibrated measuring devices).

o Design and manufacturing data, e Operational data, o Test data, o Reports of fluid analyses (fuel, lube oil, cooling water,etc.).

e Documentation on all prior failures, preventive main-tenance, and modifications of the same unit (specific documentation).

e Documentation on similar failures in other units at the plant or at other plants (generic documentation).

are problems documented in a retrievable manner and in a manner that allows the use of both plant and censric information to solve a problem?

A distinction should be made between the problems experienced by the subject unit (specific documentation) and those experienced ',

in similar units (generic documentation). However, the organiza-tion of the data should be common to both types of documentation.

The data's organization will facilitate using generic documenta-tion in the EDG reliability program and reporting occurrences to the NRC and the generic data collection agencies (INPO, NPE, etc.). This organization should follow logical groupings and could, for example, use the following major categories for equip-ment "inside the boundary" (as defined in Table B-1):

e Engine structure and drive train e Starting system o Combustion air intake and gas exhaust systems e Valve mechanism (if so equipped) e Lubrication system o Cooling system

. E-6

i l

\

l l

e Generator and associated switchgear

{

l e Instruments and controls including the governor '

e Monitoring equipment l

The specific documentation should be very dynamic with regard to both inputs and outputs. It should be updated continuously and should be available to all involved personnel from management to the lead mechanic levels. To increase its usefulness, it may use PC-based storage and retrieval methods. Significant specific data should be codified to facilitate its release, subject to proper approval, to manufacturers, other plants, consultants, l etc., and to accelerate the identification of generic problems. i All abnormal conditions including failure should be recorded in

, the specific documentation, whether or not they are subject to any regulatory reporting requirements. Often "non-reportable" conditions can result in excessive maintenance ecsts due to accelerated parts replacement or can eventually lead to major failures if allowed to recur. The documentation should include operators' or other employees' opinions. This material, although often speculative can benefit the investigative team.

In many cases, the solution to a diesel generator problem will affect more than one of the subsystems or plant organizations (e.g., electrical and mechanical repair). When the repair data are subsequently recorded in the specific documentation, they should be entered in all applicable categories and properly cross-referenced. This can be facilitated by electronic data processing.

The generic documentation, which should use the same groupings, '

would be more static, but access to it should be as broad as for '

the specific documentation. Use of the same format will facilitate communications and review at all plant levels. The NPRDS provides a format for data exchange, but the scope and depth is not usually adequate for root cause investigations. !

However, the NPRDS records do identify if and when similar problems have occurred. Identifications of a similar problem i that was experienced by one or more EDG owners can initiate a beneficial exchange of technical information.

l Does the reliability orocram systematically acoroach failure and l l root cause investications?

Considering the complexity of diesel generators, the diversity in operations, and the variety of failure modes, it is not possible 4

to design a failure analysis tree for each type of failure that could be encountered. Some examples of analyses are provided in Section E.? to illustrate the concept.

, E-7

l 1

A failure and root cause investigation should be very systematic. '

This systematic approach is time consuming but is a necessary ingredient in a root cause analysis program. This implies that :

predetermined patterns be followed all the way through, even if a '

very early strong candidate is identified. A particular failure will often result from a combination of causes. Some causes will be more predominant and causes tend to aggravate each other. The structured approach will help sort out these interrelationships.

The predetermined patterns should, however, remain dynamic and be subject to modification as the root cause analysis progresses.

Figure E-1 describes a systematic failure and analysis root cause approach.

The diesel generator problem is detected through surveillance, monitoring, preventive maintenance, or catastrophic failure. In all cases, a failure analysis should be carried out. In the case of relatively minor and nonrecurring problems, this failure analysis could be very rudimentary, yet it should provide some assurance that the problem is minor. Adequate documentation should be provided in case the problem changes in severity or recurrence frequency.

The failure analysis should be carried out to the determination of the incipient condition. That incipient condition could be a condition such as the degradation or failure of a piecepart or a mechanism, or control out of adjustment or imp 3perly timed. On Figure E-1, the steps are identified as the determination of the proximate cause.

When the incipient condition or failure has been repetitive, has resulted in substantial downtime, or may lead to a major catastrophic failure if the failure should recur, the analysis should continue until the root causa has been identified. The decision to perform a root cause analysis should take into ',

account any available specific and generic information. 1 In the event the problem appears to be a generic one, it is still necessary to look into potential specific causes that contributed i to the appearance of the problem or may have increased its severity. Similarly, if in the early stages of the analysis a problem appears to be specific to the plant or unit, the results should be checked against the generic data as the analysis l progresses. It should be kept in mind that no two units are exactly identical due to slight modifications in design through the years, changes in manufacturing methods, status of implemen-tation of vendor-recommended modifications, and other differences in operating and maintenance histories.

Using an example of a connecting rod bearing failure, which falls in the structure and drive train grouping, we can see how the systematic approach is used. The suspected causes of the failure could be:

E-8 1

Wonitor EOG J Performance 7 1r failure or off normal condition obsmed 1r ,

Determine proximate cause Preblerr. Closecut (fci!ure coute anc!ysis) Assess if surveillonce er prfownce monitoring ir shouid be oltered Compere to pcst .

f allures / conch ns to indicate posWdie, f'

systematic ceuse j

_J System: tic ecuse 3, No rjt*.imetic ccese 1 :

' Perform rc< {

UM;0 onclysis L. <

h . r ._ .~ .

ReWe= other plent Generi: or Gewric Ocuse records (NPRDS), 1 plant epecifi: -

% #T'30 incastry groups, etc. ecuse? cure erists? l Plant . _ . .

l

_ ,!_, , ,I, ,,

5pe:ific Y88 Ceuse ir i u -

ReWew operctiono Determine if

]

procedures, instcll ; OPtf0tI00' Of design-relcted gg j e spe:ici monitoring ~

if required cause t

Design OperationcI Related Related

_ y Redesign to Change operations I correct problem to correct prob!emI

._J 1r u , ___

FIGURE E-1. SYSTEMATIC ROOT CAUSE APPROACH E-9

~

1. In the bearing itself e Poor design e Improper material application e Material not meeting specifications or presenting a defect e Improper machining of the bearing parts e Improper installation of the bearing parts

2. Outside the bearing proper e Abnormal operaticn of the diesel generator as a unit 4 .

e Abnormal firing pressure in the corresponding cylinder e Improperly machined or damaged shaft e Inadequate lubrication, etc.

Obviously the operating characteristics and physical condition of the EDG would be different for each suspected cause. However, the differences would generally be subtle. Only a systematic approach can identify the proximate cause and subsequently proceed toward establishing a root cause.

Are failure and root cause investications nerformed by an acoropriately cualified team usino an adecuate task plan? ',

There are three principal elements to the establishment of a fai-lure analysis task: (1) a plant organization that puts emphasis on that task, (2) a well-organized, complete, and easily accessible data base, and (3) a means of exchanging with other plants, manufacturers, consultants, etc., information relative to generic problems.

Traditionally, a root cause investigat.lon is conducted by a multid'.scipline test. Even if, on the surface, a problem appears

! n bo strictly mechanical, it may, for example, have its root in a lubrication problem that should be addressed by a specialist in this area. The team should ind ude plant personnel to ensure continuity ard involvement of plant management (see Appendix H I

19r details). Some team members should be knowledgeable of mate-rials testing eethods, their applications, and limitations to ensure that the most pertinent laboratory analyses will be per-formed. on any failed or damaged parts covered by the root cause investigation. It is also important that representatives of tne rect Cause knalysis team follow any major maintenance operation E-10

1 taking note of the condition of parts during disassembly and monitoring reassembly. Although the observers should be indepen-dent from the maintenance work force, an acceptable alternative would be to train the mechanics to perform the observations and to specify the documentation requirements in the maintenance procedures and work orders.

The task plan should stress continuity in assignments since re-peated observations of diesel generator conditions are necessary to acquire the expertise necessary for successful root cause analysis. It is also important to clearly establish the priori-ties of the team members. Root cause analysis requires a ;

methodical approach and may be very time consuming. To avoid distractions, interruptions, and superficiality, the key members i of the team should be able to give to this task the highest priority. If the plant organization does not allow this, it is acceptable to seek, on a continuing basis, outside help that can be entirely dedicated to the task.

The development of a data base meeting the requirements of a l failure analysis task requires close cooperation between the future users (from management to the lead-mechanic level) and the personnel with strong data management expertise. The volume of data that will eventually accumulate in that base will be far too large to permit substantial modifications of the system after it is placed in service. It is, therefore, very important to apply sufficient resources to this item during its development. If plant personnel lack sufficient expertise, outside help can be used. !

In a vacuum, it is not possible to determine if a problem is of a i generic nature. Also, user 5 should not wait for a problem to !

appear to try to determine, through contacts with other plants or !

manufacturers, if it has been experienced by others. This can increase the burden brought by the problem and result in addi-tional delays in making satisfactory repairs. For these reasons, it is suggested that the generic documentation be systematically included in the root cause data bank.

Do the on-scene mechanics suonort and contribute to the failure and root cause investication?

A large amount of very valuable information can be lost if observations are not made during the disassembly. For example, an observer noticed that a mechanic removing a connecting rod bearing cap experienced difficulty in turning a large nut because it was no longer making contact with the cap. This was caused by lack of depth in the thread. The resulting friction had probably caused the required torque to be reached before the proper bolt tension was achieved during installation.

l E-11

Other examples of valuable teardown information are:

e Uniformity of bolt tightening.

e Discoloration of parts, especially if not uniform across the engine.

e Marks on surfaces indicating movement or fretting that can be affected by cleaning.

e Judgment on fit-ups when measurements cannot be taken.

Many of the questions and uncertainties of the problems shown in the examples of Section E.3 are solved by the on-scene mechanics.

They need to be trained to spot and record pertinent failure cause and effect information.

Independent observations made during reassembly can also play a critical role. For example, an observer noticed sharp edges at the oil-spreading grooves of a batch of new bearings that were being installed by a mechanic. These sharp edges were the result of a manufacturing error and would have probably resulted in bearing damage.

Knowing what to look for and how to read the te11 tales requires a great deal of experience that can be gained only through long association with the equipment and effective training. Stability in assignments to a root cause team is, therefore, important and if it cannot be maintained with plant personnel, the team can include, on a continuing basis, an outside diesel specialist (in >

addition to any manufacturer service representative that may be invited to participate in the analyses, depending on the circumstances).

4 It should be the responsibility of the root cause team to save "

-l any damaged parts for future analysis and to preserve them from any alterations until a plan of action has been agreed upon. The team should also be responsible for making contacts with other plants, manufacturers, etc., when the problem has the potential l of being generic but is not sufficiently documented in the i existing data base.

Is the entire failure and root cause analysis team consulted and I keot aDoraised throuchout the entire reliability DroQram Drocess (see Ficure 2-1). includina corrective action and corrective action verification?

The team that evaluates the failure and the failure cause should continue to add technical insight to the reliability program process. When the failure and root cause analysis is completed,

, the team should review the proposed corrective actions with all

, appropriate personnel involved. Their recommendations should become part of the specific documentation and should, therefore, E-12 i

be in a format that will permit their integration in those data. i Any modifications implemented as a result of a root cause analysis should be referenced to the analysis in the docu-mentation to facilitate its tracking.

The root cause should not be a substitute for maintr. nance management but should continue to follow analyzed problems until full resolution.

Any revised operating procedures resulting directly from the root cause analysis, or indirectly through equipment modifications, should be reviewed by the operating personnel before any subsequent operation of the diesel generator unit. This can also be facilitated by electronic data processing.

, E.3 EXAMPLES OF FAILURE ANALYSIS Following are three examples of analyses that are given to illustrate the many possibilities that may have to be investigated and how misleading some of the symptoms may be. This reinforces the need for a structured approach.

E.3.1 Example 1 i

Symptoms: The enoi.m rolls over but fails to start.

The example assumes that the unit is equipped with an air starting system that injects high-pressure air into the engine cylinders.

The potential causes are illustrated in Figure E 2, which shows only a few levels of investigation ending with the identification of the proximate )

i cause. When one or several of the causes listed are identified as contri-buting to the problem, the decision has to be made whether to do a root i cause analysis, which may require continuing the investigation several J levels below the proximate cause.

It is important not to stop as soon as the first abnormal condition is found, but to continue the analysis until every possible cause at that level has been investigated. In the example, the unit may be found to roll over slower than normal due to a faulty air starting system.

However, even at the lower speed, it is possible that the unit would have started if all the other elements had been optittum. The other elements that were not optimum and this time contributed to a lesser degree to the failure to start may be the primary cause of the next failure to start, if no correction is made.

E.3.2 Examolo 2 Symptoms: Water leaking into the engine.

Cracked cylinder liners. !

Scuffed pistons. '

E-13

SYMPTOMS: tacent Ro us ovtR But DOES NOT START

_ _ _ _ j _ _ _ _. ____p____. l j . ____4-___.

.

CONTROLS &

VALVE ASSEMBLIES 7

' ' COMBUSTION AIR

AIR STARTING

FW L SYSTEM 7

sysitM7 (IF So tou Peto) GovtRuoR7 ',

Sv5 TEN 7 .

__7_'***'__'

5 <

' _8? ?*?_ 2 __

VALVE LEAEING BLOWER Blower BLOWER DRIVE ggngng7 VALVES?

INLET AIR BYPASS DAMPER STUCE OPEM7 FAILURE 7 ClostD7 nr 50 E0utPPED m

h NOTE: WHEN ONE OR SEVERAL OF THE ITEMS SHOWN ON FIGURE E.2 (PAGES 1 THROUGH 4) ARE IDENTIFIED AS THE CAUSE OF THE FAILURE TO START, A DECISION MUST BE MADE WHETHER TO CONTINUE THE ANALYSIS DOWN TO THE ROOT (S).

FIGURE E-2. EXAMPLE 1 l

.. . __ __ _ _ _ . __ _ _ _ = _ _ _ _ _ _ _ . _ . .

AIR STARTING SYSTEM ROLLING TOO SLOWLY?

IF YEs I

I I

nata AIR vatvE AIR START vatvEs LOW AIR TANK PRESSURE 7 ruttvoeEn? natrumcTson?

'T' I '

J l l PeEssuRE VALVES STICEING EXCESSIVE AIR COMPeESSOR CostTMM.S PILOT VALVE usage? raILumE? SETTras est raftvet? (MolsTueE)?

FAILURE?

FUGURE E-2. EXAMPLE 1 (CONTINUED) t

3f

~~

Y L

R E

7 P

O D E1 _

R E VR I

T O P li T _

I T R CC E

O O F E -

I I T A

EU B 9 1

L E

U F

T 7 _

A E L

D C E Y T C Y C GD sEO#7 TL E NI eP J

nP NET I t N I PE I E i ROs T L S OR M

T I P O T N

R LE _

E P sP7 s

U O oIT F R IRE o P TTS

_ . C E EPR JM T NU IP O N e * * ' e . . ,

. . )

D _

. 7 . E R U

. O 4 . N N I R .e.

e T E 9 M

E

. V s a. N O _

O P.

T . G C S (

_ Y I . e '*'e . . ,.

S I L

E E U D 7 L F 5 P E LR G EE M G UT A OFL L I X

E C F

? .

L E 2 R U R -

DE _

ENLF E7 O F E.

L E

M LI I

ES GtNw E U R I

UG F 0 ANFA FE R P U _

. e t T G O E I

. N F _

T O

N 7

'5 M$""

II M IE L _

7

_ R E E5 E T IM" _

I 'l _

7 _

.- L5

_ AIEE l"

I' l

NL EY7 s l E vLe lLt aUaUr MFvFo l

._ m*** _

l,

-- ._- ----._-_.-_-_a.-_ -- . - + - -- - a _- -

eO C088i#0L5 E

GovEd=0R I I I

GO'E"087 CONTROL 57 I I I 1 1 NOT SET AIR CONTROLS IMPROPER FALSE 04 LOW OIL IN 880 RESET TRIP TRIP START PRESSURE? GovEmm0R FOR SETTINGS? SIGeeAL57 POSITION? wvsRaWLICS?

DeERat:Ose?

b I

DepeeGED LOW OR M OIL ggygg LEVEL 7 WDRAULIC rune?

FIGURC E-2. EXAMPLE 1 (CONTINUED)

- - - - - - - - -n+--- -- n .-,,-----n - - - - - - - - - - , - - - , - - - - . -- - - - - - - - _ - - _ _ - -

Three possibilities as to the initiation of the failure had to be analyzed:

(1) cooling water leaking past seals that caused a degradation of the lube oil, (2) liners cracking due to defects in material or fabrication, or (3) pistons not round and being scuffed.

It could not be established that the leaks started prior to the scuffing.

Further, some scuffing was found in cylinders where there was no leak, i

Also, some pistons were found in the very early stages of scuffing in cylinders where the liners were not cracked. In those pistons, it was found -

that the piston pin bushing had started elongating through extrusion. This (

elongation eventually caused the piston to go "out of round," which itself resulted in the scuffing of the piston. The friction against the cylinder liner, which increased with the scuffing, caused localized overheating that resulted in the cracking of that liner. The bushing degradation was the incipient underlying condition.

The decision was made to continue the analysis in an attempt to identify the .

root cause for the bushing extrusion. Since this did not appear to be a !

generic failure, improper design or materitl specifications were eliminated.

The following possible failure causes remain:

o improper bushing manufacture. This could not be demonstrated, o Engine overloading, which could result from reported instability !

in the governor system. This could not be demonstrated using the available data, o Excessive cylinder peak firing pressure due to imbalance between the injection pumps. This was probably a contributing factor but not the primary factor. The pumps were adjusted, e Improper lubrication. This appeared to be the best candidate, Further investigation revealed excessive foaming of the lube oil..

This example is further illustrated in Figure E 3.

Since this is a significant failure, a root cause investigation is warranted. Using improper luorication as the proximate cause (per Figure E 1), the first step is to determine if it is a generic or specific cause.

If the cause (a lubrication problem) was generic, another plant may have already identified and verified a cure. In this case, assume no generic information was discovered. The inability to discover a generic cause does not conclusively prove the problem to be unique, it only means that no record of or experience with the problem is readily accessible. Thus, the problem should be approached as specific in nature. The cause is unlikely to be related to maintenance or operations, so the invettigation focused primarily on design issues. It was determined that there was incompati-bility between the material of the bushing and additives in the oil. A different lube oil was recommended and was subsequently used in the unit.

Incidentally, the above is an excellent example of a problem that should be in a generic data base. The material incompatibility is likely to exist in many diesels. With this information, other diesel owners would know to

}

E-18

}

l

=5 _

W 4 g

'- E g _! l s a l_ *1 s

r-

~S~ t imI~ E 9

..s -

y -

-,o .:.

g ,

E

~

! '~'

E "h E!

ls E i

l -

-s l - , , :-

G El l Og .EE Es 0 i** E E w '

i S'" % woi i -=b tr-Wgo i 3.

_ N3* -5* -

i5 8"si l

ges -

s it's w

=

2B

.w e G. E"9

-o e m *s<

57' :

Y5 t_., a9 l

5 I8 l es - ,- !

EEe E$

e25 w -

E3G 58 I

Or -

~

to.

.Wt w WE g

<=v

W e g

- e

(

x v

5 c5: s m p a 5 UNN vt u 5vg o u

=>

.NE

rig

~J e- \

I l

E ^5

-3 Stiv

<E .

88 wfa s

l l

E-19

l l

switch lube oils and check the pistons and cylinder liners. A catastrophic :

failure and a time-consuming and expensive repair could be avoided. '

i E.3.3 Example 3 t

Symptoms: Gas pressure built up in the surge tank for the cooling i system. !

Gas samples were taken and analyzed. They contained a high concentration of ,

carbon monoxide, which seemed to point to an exhaust gas leak. This could I not, however, be demonstrated by hydrostatic testing of the engine itself.

l !

Other parts of the cooling system were then included in the investigation, including the combustion air aftercooler loop. A review of the system indicated that, at some point, the cooling water pressure dropped below the air pressure in the heat exchanger. This could allow combustion air to .

enter the cooling water if there was a leak in the heat exchanger. With S

attention focused on the exchanger, a leak was found at the gasket that had either not been installed properly or was not under sufficient compression.

The leak was in an area extremely difficult to inspect and would have been much harder to detect, were it not for the systematic approach used.

This leak could explain the pressure buildup in the cooling system surge tank but not the high concentration of carbon monoxide. The failure root cause investigation, therefore, turned to the combustion air system. It was '

a found that the engine exhaust stack and the combustion air intake were close enough to explain the aspiration of exhaust gas.

This example is further illustrated in Figure E 4.

E.4 INTERFACES WITH OTHER REVIEW ITEMS

~

The root cause program interfaces with several of the other review items of the diesel generator reliability program. l 1

i Reliability Target (Review Item A) :

J The method used to calculate availability (described in Appendix A) includes ;

both failures to start and run and downtimes for repair and test actions. A ;

balance should, therefore, be struck between outage extensions that may be

, required to implement a root cause program and the risk of increased i failures to start and run, or long outages following major catastrophic 4

failures, if the root causes of too many problems remain undetermined.

Surveillance (Review Item B)

Elements of the surveillance program such as identification of critical failure modes or aging mechanisms can be used as inputs to the failure

.I analysis program through the documentation suggested in this appendix.

Conversely, the identification of the root cause of problems may require a modification of the surveillance plan.

l E-20

5YMPTORS: 6As tulLa WP IN CDOLING WAf tR

^ ' '

SURGE TANK CONTAINING C0 1

[XMAUST SAS LEAK th thGINE C00LANT7 LEAK POS$15LE IN MO OTHER PARTS OF

CDCLlhG SYSTEM i

Yts

'COMBU$Tich AIR I 6

\ t >

CAN LEAK FOUND lh AFTERC00LER RELAf t 70 1

HlGM CO CDh:thTRATION 7 YE'S l RELATIVE P051110% OF DIESEL EXMAUSTt STACK AND COMBUSTION AIR lhTAKE l

\ i '. l l

1 1

FIGURE E-4. EXAMPLE 3 i i

E-21

I i

Performance Monitoring (Review Item C) l 2

1 i' The results of performance monitoring are one of the triggers of failure 1 analyses. The monitoring data can also play a very important role in the '

root cause determination as detailed in Section E.2. i

{ Maintenance (Review Item D) !

It is stressed that much is to be gained from having designated members of (

the root cause analysis team independently observe all major maintenance '

operations, especially the disassembly of the units. This requires coordination with maintenance from a scheduling point of view. If the alternative solution of having the mechanics make and record the obser- l vations is selected, this should be made a part of the maintenance program. '

Accurate and complete records of all prior maintenance operations are often t l<

critical to a successful root cause analysis. j

9 Problem Closecut (Review Item F)

I The problem closeout procedure should include the suggested review by l operating personnel of any new operating procedures, resulting directly from '

the root cause analysis or indirectly through equipment modifications. This !

l review could be made jointly with members of the root cause team, i If the root cause was an improper maintenance procedure, both the original

and revised procedures should be reviewed with all appropriate maintenance

personnel, clearly identifying their differences, j DataSystem(ReviewItemG) i {

1 Appendix G. Section G.2, lists the elements that are necessary to a i successful failure analysis. One of these is documentation of all prior i

failures of the same unit, preventive maintenance, and modifications. This

! is referred to as the specific documertation. To avoid duplication of 4 efforts, the format used to record that information should be compatible l with data collection for the other tasks. It should be emphasized, however, that the specific documentation for root cause analysis should remain avail-able at all levels from management to the lead mechanic. l q

l Responsibilities and Management Controls (Review Item H) i Management should retain the responsibility for deciding to what level the j failure analyses should be carried.

I j

E 22

APPENDIX F PROBLEM CLOSE00T (Review Item F)

O e

1 i I

')

i f

l f

f 4

d i

i I

i l

l 1

F1 1

e TABLE OF CONTENTS EEt F.1 INTRODUCTION ........................ F-5 F.2 ISSUES TO CONSIDER WHEN REVIEWING PROBLEM CLOSE0VT PROCEDURES ........................ F-2 F.3 INTERACTIONS WITH OTHER REVIEW ITEMS . . . . . . . . . . . . F-7 9

1 I

F-3

F.1 INTRODUCTION The purpose of formal problem closeout procedures is to ensure in a timely way that effective solutions to detected EDG reliability problems have been devised and implenented. An effective solution is one that corrects the observed EDG reliability problem and does not create any other reliability or performance problems that are as bad as, or worse than, the corrected problem. Often measurements can be taken that are not part of the normal, established performance monitoring procedure but will provide assurance that the implemented corrective action is effective. The problem closecut review item should ensure that consideration is given in the EDG reliability program to providing for any additional monitoring or surveillance that would expedite the assessment of corrective action effectiveness.

Two elements are necessary for an effective problem closecut procedu. m e The problem closecut procedure should provide for the

. establishment of specific, numerically based criteria that have to be met before the detected reliability problem will be considered ,

to be corrected. The actual criteria should be based on the nature of the reliability problem and cannot be specified before hand.

e The problem closeout procedure should provide for any additional monitoring activity that might be necessary to ;/ ovide a timely judgment concerning the effectiveness of the corrective action.

Again, the additional monitoring used, if any, should be based on the characteristics of the detected problem and cannot be specific i beforehand.

An example of a numerically based closeout criterion is: "No failures, ,

including incipient failures, attributable to the detected failure cause for !

a period of months." Another example, applicable to a single demand test, is: "UTT pressure levels out at _ psi by 15 minutes after diesel generator start and does not increase thereafter during a 4 hour4.62963e-5 days 0.00111 hours 6.613757e-6 weeks 1.522e-6 months run of the diesel generator." Examples of special monitoring for problem closecut could be, for the first example above, any special surveillance that might be required to ensure that incipient failures due to the corrected cause do not appear; and for the second example above, continuaus monitoring of the oil pressure during the 4-hour run. The closecut criteria should be speci-fied before the trial period during which the corrective action effective-ness is being assessed. '

Review of a reliability program for EDGs should verify that there is an element of the program that deals specifically with problem closecut l procedures. Section F.2 specifies specific issues that should be dealt with under Review Item F. Section F.3 discusses the interfaces that should be I considered when reviewing the problem closecut item of a diesel generator reliability program and reviews other facets of the program. l l

F5

F.2 ISSUES TO CONSIDER WHEN REVIEWING PROBLEM CLOSEOUT PROCEDURES The specific issues that should be addressed when reviewing the problem closeout element of an EDG reliability program are presented in this section.

Does the EDG reliability orocram specifically address the problem closeout ortion of the reliability proaram?

For an EDG reliability program to be complete, there should be a formal problem closecut element of the program. The characteris-tics of the problem closeout procedure should be addressed expli-citly in the documentation describing the program. This documen-tation should address at least the issues presented below as additional questions.

~

Does the reliability orocram documentation specify a orocedure for _se3, tina criteria for problem closeout based on the results of the failure analysis or root cause investiaation?

Criteria for problem closeout should specify required l

measurements or observations for closeout, which may or may not l be in addition to surveillance routinely performed on the EDG and should specify an outcome from these measurements that define minimum conditions for success. Also specified should be the length of time or number of cycles over which these measurements should be taken and for which the minimum conditions for success apply. The nature of the closecut criteria is that they specify engineering or performance results that should be observed if the corrective action is to be judged effective. The criteria that are appropriate will depend on the nature of the reliability problem corrected and thus cannot be specified beforehand. Also, criteria that are too rigid or that are too extensive may divert resources away from more risky problems and are therefore as undesirable as criteria that are too lax, or no criteria. The discussion of the criteria in the EDG reliability program docu-mentation should reflect this balance. The discussion should also present a credible procedure for developing the criteria.

Does the EDG reliability orocram specify credibh _frmal closeout procedures?

Options for instituting formal problem closecut procedures include: (1) documentation of specified aspects of the problem closeout, including results of surveillance and monitoring related to the subject problem, and (2) a review committee to review any or all aspects of the problem closecut. A combination of the above could also be used. Again, the closeout procedures should be credible in the sense that they specify appropriate resources to be expended to ensure the timely correction of the problem. The reliability program documentation should reflect the dependence of the resources required for problem closecut on the severity of the reliability problem.

F-6

Does the reliability orocram documentation exhibit an under-standina of the types of special monitorina that could be used durina the period when the effectiveness of corrective actions is beina assessed?

The reliability program documentation should contain a discussion of the surveillance and monitoring that could be used to verify corrective action effectiveness. This discussion should include an indication of how the length of the trial period during which the assessment will be conducted is to be established. One option is to base the length of the trial period on the expected return-time for the problem if the problem had not been corrected.

F.3 INTERACTIONS WITH OTHER REVIEW ITEMS

, The problem closeout review item should be coordinated with Review Item G, "Data System," to ensure that the proposed EDG data system has the capa-i bility of supporting any additional monitoring requirements conceived of for i problem closeout. Thus, the data system should be capable of storing in a retrievable way information generated over the problem closeout period, even if this information is taken at a different rate or is different information from that which is normally taken and stored.

The problem closeout review item should also be coordinated with the review conducted to ensure adequate management of the EDG reliability program (Review Item H, "Reliability Program Management and Responsibilities") if the closecut procedures are to involve a problem closecut committee. This I problem closeout committee will be part of the management team for the i reliability program. Assurance should be obtained that the committee :

members have adequate background and authority to act in this capacity. I F-7 l

I APPENDIX G DATA SYSTEM (Review Item G)

O e

l l

l G-1 J

TABLE OF CONTENTS EAa.t G.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . G-5 G.2 ISSUES THAT SHOULD BE ADDRESSED WHEN REVIEWING AN EDG DATA SYSTEM . . . . . . . . . . . . . . . . . . . . . . . . G.5 G.3 INTERACTIONS WITH OTHER REVIEW ITEMS ............. G-10 REFERENCE FOR APPENDIX G . . . . . . . . . . . . . . . . . . . . . .

G-12 O

G3

~

_ _ _ _ - - f

G.1 INTRODUCTION The objective of this appendix is to identify review issues for the data elements that are necessary to support the EDG reliability program and for the system by which these elements will be collected and stored. A defini-tive and aggregate set of information is required to properly address the reliability and availability issues associated with the reliability program.

The data set should support the assessment of the specified goals and targets. The data set should also support the combined elements of the reliability program. Operating hours, number of demands, number of catastrophic failures, outage times, repair times, and other necessary information to achieve the requirements of all the review items are included in this appendix.

Data storage and retrieval may be performed on a computer or may be performed manually. In either case an organized system should be available l or be developed. This iaay be accomplished in several ways, for example, by using a capable, readily available data base management system on a computer or by setting up and maintaining an adequate file system for manual data l storage and retrieval. '

It is not necessary to duplicate and store all information in one specified location. However, all information (i.e., maintenance work orders, I completed test procedures, vendor manuals, etc.) should be stored in a systematic and easily accessible method. For example, vendor material and test procedures may easily be obtained in a well-organized olant library, while copies of completed test procedures may be available in a well-organized plant document room.

The data storage and retrieval process should be documented and procedures developed to ensure compliance with and support of the reliability program.

It is not the intention of this appendix to require the reconstruction o f.

failure, maintenance, and operating information related to the EDG prior t o.

the establishment of the reliability program.

G.2 ISSUES THAT SHOULD BE ADDRESSED WHEN REVIEWING AN EDG DATA SYSTEM The specific issues that should be addressed when reviewing the data system of an EDG reliability program are presented in this section. l Is the EDG reliability orocram information and data administered DroDerly and in a manner suDDortive to the other procram elements?

The failure data, outage data, and operating history data are required to estimate the EDG performance as described in Appendix A. For this reason, it is necessary to provide a secure, designated location for these data in order that the required calculations can be performed and updated on a timely basis. An appropriate plant individual or plant group should be designated as responsible to perform all data storage and retrieval tasks.

This data custodian will be responsible for issuing the required reports and for maintaining the EDG data set. A procedure should G-5

be developed to ensure that all required information is received by the data custodian in a timely manner, recognizing that com-pleteness of the data is an important consideration. Also any changes to documents or components directly affecting the EDG, such as procedures, test frequency, or modifications to the de-sign of the EDG, will be required to be reviewed by the data custodian in a timely manner to allow for a determination of any changes to the reporting requirements. Changes may affect items such as failure rate or reliability calculations, the determina-tion of a proper failure severity, or a change in the original boundary. The data custodian should also review all failure and maintenance reports to check for both accuracy and completeness.

The data custodian should document all changes and corrections to I the reports and provide feedback to the training process, described below, in an effort to improve the information that is ;

vital to the reliability program. -

Does the failure reDortino Dortion of the data system contain information of adeouate scoDe and deoth to suoDort reli- ,

ability calculations (ADDendix Al and Derformance monitorina fADoendix C)?

Failure reporting consists of describing the events leading up to and occurring during the failure process. Failure reports are necessary to calculate the reliability level of the EDG as described in Appendix A. This may be accomplished on a properly structured work authorization form, or a separate failure report form may be developed. The information in a failure report form should be documented as the information becomes available and not reconstructed after the fact. The failure report should include as a minimum the data elements listed below:

Originator: The person discovering the failure.

Department /0rganization: The plant department or organization of the originator.

Unique Document Identifier: A unique identifier to allow for tracking of the document. This may be a job order number or work i authorization number.

Component Identification: The unique identifier for the j component or piecepart as applicable. This identifier can be 1 related to the manufacturer, model number, and other engineering and spare parts data.

Component Type: A description of the component, e.g., valve, pump, relay, etc. The diesel generator is somewhat unique in the sense that the names of certain pieceparts may be confusing such as "valve." A method should be established to prevent such items as intake and exhaust valves from being confused with motor-operated and check valves. ;

I Equipment Location: Location of the failed equipment, i

G-6

Status of Component: The status of the component at the time of failure (i.e., operating, in test, etc.).

Condition of Parts or Materials Removed: Technicians on-scene appraisal of any parts (e.g., covers or nearby pieceparts) or materials (e.g., lube oil drained).

Plant Status: The status of the plant at the time of component failure.

System / Subsystem: The EDG system and/or subsystem of which the component is a part.

System / Subsystem Status: The status of the system or subsystem I at the time of the component failure.

~. Discovery Date:

Discovery Time:

Date of Failure: May be different from discovery date, if known.

)

Time of failure: If known.

l Failure Severity: Catastrophic /Immediate, Degraded or Incipient, !

using definitions similar to those in IEEE Std 500-1984 (Ref. G-1). Failure severity is important at three levels:

Piecepart Failure Severity:

EDG Subsystem Failure Severity:

EDG Failure Severity:

l Time and Date of Repair Initiation: The time and date repair actions commenced.

Time and Date of Repair Completion: The time and date the repair was successfully completed.

System Effect: The effect, if any, that the component failure had on the system.

Plant Effect: The effect, if any, that the component failure had on plant operations.

Detailed Failure

Description:

This should describe all events leading up to and during the failure process. This should include procedures, tests, test equipment, all instrument readings that were taken, pieceparts involved, a description of the evolution taking place, anything that may have been seen, heard, smelled, or felt that may be important in determining the root cause. The G-7

l cause of the failure if known. Any supporting documenta+ ion should be attached to this form.

Corrective Action Taken: This should contain enough informatica 1

{

to reconstruct exactly what corrective action occurred. An entry such as "rebuilt" is not adequate. An entry such as "rebuilt in accordance with Section 4.2.7 of Tech Manual" would be appropri-ate. A description of the process of rebuilding the failed component would also be acceptable.

i Spare Parts Used: Either list the spare parts used or attach the '

spare part ordering documentation.

Postmaintenance Retest Performed: The retests performed to verify component or system operability and test results.

~

The plant personnel (operators, maintenance technicians, engineering staff, etc.) who gather the failure information should be trained to properly fill out the forms. The failure i report should reference all procedures, instructions, and personnel involved and should include any information that may aid in the failure analysis.

A method should be developed to ensure that all components reflected in the boundary, as described in Appendix 8, are appropriately reviewed for reliability concerns affecting the EDG. Example methods include a list of all components determined to be within the EDG boundary or a special set of piping and instrumentation diagrams (P&lDs) and electrical schematics denot-ing the developed EDG boundary.

Most of the information described above is available via the NPRDS Failure Reporting System. However, since the NPRDS failure ,

reports do not include all the above information, the NPRDS failure reports, in and by themselves, are not considered adequate to support the EDG reliability program.

Does the operatino history oortion of the data system contain information of adeauate scope and deoth to support reliability calculations (Apoendix A) and Derformance monitorinQ (AoDendiX c17 A comprehensive operating history of the plant and of the EDG are required to be established. This should include, but should not be limited to, the following:

Demand Information The date and time of each EDG start attempt. The reason for starting the EDG (i.e., test, tech spec requirement, challenge, etc.) and whether or not the attempt to start the EDG was successful or unsuccessful.

G-8

Ooeratina Information The length of time the EDG operated for each successful start.

The operating parameters, if available, such as lube oil pressure and temperature, generator voltage and frequency, water temperature, etc.

Test Information This consists of the test frequency, test interval, and test duration for each EDG test and the reason the specific test was performed. All documentation should be available to support this information.

Ooeratina Characteristics Operating characteristics should be developed for all components I within the EDG boundary. This includes operating parameters for the components and information to determine reliability and availability of components, such as starting air compressor cycle and operating times, diesel subsystem status, and parameters checked or tested on a routine schedule.

Plant Ooeratina History A plant operating history should be developed and used by the data custodian to help determine the EDG operating frequencies and to aid in verifying the accuracy of the submitted reports.

Maintenance History A complete maintenance history of all components included in the EDG boundary should be developed and maintained. The maintenance history should be developed to support the EDG reliability program requirements, especially the requirements of the EDG performance tracking task. A list of all corrective and preventive maintenances and of all piecepart and component replacements should be included in the maintenance history.

Does the information base support reliability orocram activities.

esoecially the root cause investications suaaested in Aooendix E?

A set of documents generated external to the plant is necessary to provide assurance that problems experienced by owners of similar ED3s are not experienced. Technically qualified participants in the reliability program should have regular access to information such as NPRDS records Part 21 reports, 50.55(e) reports, LERs, and other pertinent information from consultants and vendors.

Other documents required in support of the EDG reliability program include:

G-9

1 l

e Associated portions of the Technical Specifications.

o Associated P& ids and electrical diagrams.

e Vendor manuals associated with the EDG, including all vendor reports and updates.

e Set of surveillance and test procedures and requirements.

e Set of EDG operating procedures.

e Set of Emergency Instructions associated with the EDG.

e The EDG design specifications and requirements.

e Recommended and implemented modifications (per vendor

, notices, NRC bulletins, etc.). l G.3 INTERACTIONS WITH OTHER REVIEW ITEMS 1

The EDG data system should have the capacity and flexibility to supuort all 1 the elements of the EDG reliability program that require historical perfor- !

mance data for their successful implementation. Specifically, the perfor-mance monitoring, performance evaluation, root cause, maintenance, and problem closeout all have a need for historical performance data. There-fore, the other review items that should be considered when reviewing the adequacy of the FDG data system are:

e Evi ;o of the EDG reliability target.

o EDG surveillance needs.

e EDG performance monitoring. ,

e EDG maintenance program.

e EDG failure analysis and root cause investigation. I 1

e Problem closeout.

e Responsibilities and management involvement.

Thus, there is a need to review the data system in light of the EDG reliability program needs and characteristics discussed in each of the other review items.

i The EDG data system is the primary repository for the information required by the NRC to evaluate EDG performance. Therefore, all information identi- l fied in Review Item A as necessary to evaluate EDG performance should be collected and stored by this system. The EDG documentation describing the proposed reliability program should identify this information explicitly and present a plausible description of the techniques to be used to collect and store it.

G-10

j

~~~

.. l

\

Review Items B, C, D, and E (surveillance needs, performance monitoring, l maintenance program, and failure and root cause analysis) all require historical data for their successful implementation, operation, or 3 i

adaptation to changing reliability characteristics. This information could '

be unique for each EDG reliability program and therefore cannot be specified in advance. Each submittal should be reviewed to ensure that the data system is capable of supporting the data needs of these other reliability program features.

Review of problem closecut procedures (Review Item F) may indicate that I special monitoring techniques will be used for problem closeout. The review 1 of the data system should verify that the data system is flexible enough to i accommodate these special monitoring schemes. !

l Finally, the data system should be managed to ensure that all the above needs are accommodated. Review of the EDG reliability program management considerations should ensure that they include day-to-day management of the

'. data system with qualified personnel assigned to manage and operate the system.

l

]

G-11

REFERENCE FOR APPENDIX G l G-1. Institute of' Electrical and Electronic Engineers, Inc., "IEEE Guide to the Collection of Electrical, Electronic, Sensing Component and Mechanical Equipment Reliability Data for Nuclear Power Generating Stations," IEEE Standard 500-1984, December 1983.

l l

l t

. i 1

I l

., l G-12

1 0

1 APPENDIX H RESPONSIBILITIES AND MANAGEMENT CONTROLS (Review Item H) l l

H-1

TABLE OF CONTENTS Paae H.1 INTRODUCTION . . . . . . . . . . .............. H-5 H.2 ISSUES TO CONSIDER WHEN REVIEWING MANAGEMENT CONTROLS . . . . H-5 !

H.3 INTERACTIONS WITH OTHER REVIEW ITEMS ............ H-8 1

1 I

H-3

l l

H.1 INTRODUCTION The purpose of this review item is to ensure that the management controls under which the EDG reliability program will be operated are adequate and that individual responsibilities for operating the program have been clearly defined.

A diesel generator reliability program is a management system for managing diesel generator reliability. The rules and procedures that flow from the management system are all based on a consistent philosophy, which states that a specified reliability target can be achieved by understanding the factors that drive a diesel generator's reliability and then applying reliability and engineering techniques in sufficient depth to ensure that the target is reached.

Management reviews and controls are necessary to ensure that the EDG relia-

, bility program results in achieving the reliability target for the diesel generators. Also, responsible individuals for implementing and operating the reliability program should be identified. These individuals should be qualified or suitably trained to carry out their assigned responsibilities.

Achievement of the EDG reliability target depends on there being adequate management review and controls of the reliability program, as well as quali-fied individuals responsible for implementing and operating the program who have the authority to manage the program to achieve the target. Even though consultants and vendors may assist the utility in implementing the EDG reliability program, the plant management retains the ultimate responsi-bility and is the key to the program's success.

Section H.2 presents issues that should be addressed when reviewing the proposed management plans for EDG reliability programs. Section H.3 discusses the need to coordinate review of the reliability program manage-ment with review of other items of importance to maintaining the EDG relia-bility target.

H.2 ISSUES TO CONSIDER WHEN R: VIEWING MANAGEMENT CONTROLS The specific issues that should be addressed when reviewing the management plan and assignment of responsibilities for an EDG reliability program are presented in this section.

Are there manaaement orocedures for reviewina and verifyino that the EDGs are meetina the reliability tarcet?

The review conducted under this review item should verify that procedures are in place to regularly assess whether or not the EDG reliability target is being met. These procedures should !

provide the schedule for this assessment and identify i responsibility for its completion. They should identify the computational techniques that will be used, including the data to be used for the assessment.

I H-5 I

Does the plant manaaement periodically perform detailed reviews of the EDG reliability Drocram?

In addition to frequent reviews to ensure that reliability targets are being met, detailed programmatic reviews should periodically be performed. These reviews should be performed by a team independent from the EDG reliability program and the day-to-day operation and maintenance of the EDGs. The emphasis of this review process is to ensure that the reliability program is operating in the closed loop process described in Section 2. The team should perform a technical review designed to ensure that problems are being detected and analyzed and, most importantly, corrected and closed out.

Does the implementation and operation of the EDG reliability Drocram have the uneauivocal suDDort of plant manaaement?

~

Probably the single most important factor that will determine the ultimate success or failure of the EDG reliability program is the degree of commitment to the program by the top plant management.

Indications of management commitment can be obtained from: (1) assessment of whether or not the projected resources allocated to implementation and operation of the program are adequate; (2) assessment of how high in the organization relatively detailed knowledge of the program exists (at a minimum, all operators and supervisors responsible for plant operations and maintenance should have detailed knowledge of the program); (3) assessment of the ability and readiness of line maintenance and operations personnel to describe advantages of the program; and assessment of the qualifications of the personnel assigned(4) to manage and operate the program (other than personnel performing routine operation that would be performed even if the program did not exist).

If the oroaram is to _be newiv set up or newiv modified. are there clear procedures and policies available that treat the procram implementation Dbase?

There is expected to be a transition phase for a newly established EDG reliability program or one that is newly modified to conform to the guidelines herein presented. It is a management function to ensure that the transition between no program, or incomplete program, and an adequate reliability program, is as smooth as possible. For instance, transition between use of prescriptive technical specification restrictions on EDG (before implementation of the program) and performance-based indicators (after implementation of the program) will require an evaluation of the performance-based indicators to ensure that they are adequate to meet the EDG reliability target.

Managing the transition requires treatment of it in a plan and then implementing the plan. The review of the EDG reliability program management plan should include verification that this problem has been recognized and that there is a plan to manage the transition period.

H-6

l l

Is there an individual with responsibility at the olant manacement level that is dedicated to implementation and use of a reliability oroaram for diesel aenerators and is this individual l resDonsible for imDlementina. oDePatina. and. if reauired, alterina the procram?

l 1

The licensee should give a technically qualified individual j adequate authority to run the EDG reliability program. If the EDG i reliability program has an influential "champion" that is in charge of implementing, operating, and, if required, altering the program, there is an added measure of confidence that the program will continue to be successful.

Do the oersonnel assioned to manace and operate the orocram have ,

a credible mix of aualifications that are reauired for such a procram?

The mix of qualifications necessary to successfully manage and run the EDG reliability program includes maintenance and diesel operations, diesel generator engineering design, engineering root cause investigation, and reliability assessment methods. The review should identify where each of these qualifications will come from and if authority over use of these resources resides with the personnel managing the EDG reliability program.

What is the relationship of manaaement of the EDG reliability Drocram to toD level plant manacement and to manacement of other functions in the plant?

Plant management at the top level and managers of other plant functions should recognize the reliability program as the vehicle for achieving the EDG reliability target. The review conducted under this review item should verify that this relationship among i plant management functions exists. An EDG reliability program .

could conceivably function correctly as either a program managed by line management or as a program managed by staff. However, the program will ap_t work if it is conceived of as simply an add-on to current surveillance and maintenance practices (i.e.,

entirely as a staff function). The EDG reliability program should reolace current surveillance and maintenance practices.

This is not to imply that the current practices (e.g., monthly testing and approaches to failure cause and root cause analysis) will not be adopted as part of the reliability program. To the extent that they provide a reliability focus to surveillance and maintenance, they will remain as part of the reliability program.

Thus, the program should be integrated into everyday plant operations.

Is there an identified mechanism for alterina the EDG reliability Drocram if this becomes necessary or Cost effective?

The EDG reliability program focuses surveillance and maintenance of the EDG according to the reliability of the equipment. Thus, management of the program should allow for the periodic reassess-H-7

ment of whether the current program is appropriate to achieving the EDG reliability target. Surveillance and maintenance activi-ties should be somewhat flexible in that more or less may ve needed, depending on the mechanisms that are driving the relia-bility. It is a management function to ensure periodic reviews of the programs' applicability. The review conducted under this review item should ascertain what management controls are in place to ensure that the reliability program will be focused by periodic assessments of the EDG reliability characteristics.

H.3 INTERACTIONS WITH OTHIR REVIEW ITEMS Review of the proposed management of the EDG reliability program should be coordinated with review of every other item. The success of the reliability program depends on the support of plant management and the skills of the personnel operating the program. If the program exists on paper but is never implemented, or is implemented and then abandoned or de-emphasized, the EDG reliability target will not be met in the long run. Improper attention to the EDG relitbility program by the plant will eventually become known to the NRC through the long-term trends in the EDG reliability estimate (as described in Appendix A).

The review of the proposed management of the reliability program should be coordinated with review of analysis of EDG surveillance needs (Review Item B) to ensure that there are assigned responsibilities for determining the EDG surveillance needs and preparing the surveillance plan as part of implementing the program. The coordinated review should also ensure that the individuals assigned responsibility for this task have the proper qualifications.

The review should be coordinated with review of the licensees' proposed EDG performance monitoring scheme (Review Item C) to ensure that performance monitoring is intended ay the licensee to be a continuing process and tg ensure that management c]ntrol mechanisms will allow this process to adapt to the changing reliability needs of the EDG.

The review should be coordinated with Review Items D and E (maintenance and failure analysis and root ceuse investigation) to ensure that qualified personnel will be assigned to handle these responsibilities.

The review should be coordinated with review of problem closecut procedures (Review Item F) to ensure that there are adequate procedures and management controls for problem closecut.

The review should be coordinated with review of procedures for storing and retrieving the data necessary for operation of the reliability program (Review Item G) to ensure that management controls are adequate for the continued operation of this critical function, to ensure that all necessary or potentially necessary data have been considered as cindidates for the data base, and to ensure that qualified personnel will be charged with the data input duty.

H8

1 I

l l

l APPENDIX I EMERGENCY DIESEL GENERATOR DATA REVIEW l

l 1

l l

I l

i l

l i

l I-l

l l

TABLE OF CONTENTS l l

Paag I.1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . I-5 1.2 HISTORIC EDG PROBLEMS .................... I-5 l l

I.2.1 Insights from NUREG/CR-4590 Data ........... I-6 !

I.2.2 Insights from a Collection of NPRDS Data ....... I-14 1.3 INTERFACES WITH OTHER EDG RELIABILITY REVIEW ITEMS . . . . . . 1-16 I.4 EMERGENCY DIESEL GENERATOR DATA BIBLIOGRAPHY . . . . . . . . . 1-17 LIST OF TABLES Tables 1-1 Subsystem Failures by Diesel Generator Manufacturer . . . . . . . . . . . . . . . . . . . . . . . I-7 I-2 A Ranking of Diesel Generator Subsystem Failure Occurrences ....................... I-8 I-3 Diesel Generator Failures by Components . . . . . . . . . . . 1-12 I4 Critical Component Ranking for Each EDG Manufacturer . . . . . . . . . . . . . . . . . . . . . . .

1-13 .

1-5 NPRDS-Derived EDG Insights ................. 1-15 1

1-3

l 1

l I

l I.1 INTRODUCTION The major objective of this appendix is to summarize historical diesel generator problems and to categorize these problems by subsystem and component. While analyzing historical problems, useful data sources will be identified and examples of how historical information can be used to supple-ment a reliability program will be provided. Although knowledge about the EDG at one's own plant is essential in order to administer a reliability program, a great deal can be learned by studying problems occurring at other plants. EDG equipment failure data have been collected and studied frequently, and all data collections and analyses have potential uses.

Historical diesel generator data give a good idea as to which subsystems and which components are likely to cause the most diesel generator failures.

The list of critical components and subsystems varies between diesel generators but can generally be predicted for each diesel generator manu-

- facturer. The list of problems that should be emphasized will be revised as the plant's reliability program proceeds, but this appendix can provide a useful starting point. Once the likely and potentially important problems are identified, preventive maintenance and surveillance policies can be focused. NUREG/CR-4590 and the Nuclear Plant Reliability Data System (NPRDS) (see bibliography in Section 1.4) were chosen for this study. Other references, such as vendor records and LERs, are suitable for similar studies.

Section 1.2 examines historical diesel generator problems from several perspectives. This section provides the user with a generic prioritization list for critical components and subsystems. If plant-specific historical information is not available, these lists can be used in setting up an EDG reliability program. Section 1.3 examines the interfaces between historical information and the review items of the EDG reliability program. Section 1.4 discusses the data sources used in this study. l l

I.2 HISTORIC EDG PROBLEMS ',

Two major sources were used to gain an understanding of what diesel generator problems are occurring in the industry. NUREG/CR-4590 contains an extensive data base of diesel generator failures that are categorized in several ways. The NPRDS data storage system contains a large amount of raw data that can be processed. Unfortunately, neither data source contains adequate information to accurately estimate the number of demands on aiesel generators or groups of diesel generators, so no failure rate calculation can be correlated to failure causes and modes. The data tables in this appendix are arranged by diesel manufacturer. Since the number of diesels made by each manufacturer varies greatly, the actual number of diesel failures for any manufacturer means little in itself and can be misleading i (for example, since there are more GM diesels than any other type, in most !

cases the GM diesels have experienced more failures, even when that failure category is of relatively low importance to the overall performance of GM diesels). No conclusions should be made as to the relative reliability level of the various EDGs, but insights can be derived for identifying the types of problems that can be expected to occur in the various types of EDGs. ;

15 I

I.2.1 Insichts From NUREG/CR-4590 Data The authors of NUREG/CR-4590 assembled a data base in order to study aging of diesel generators. The data base consists of 500 randomly selected j

failure records from each of the following sources: ,

Licensee Event Reports (LERs) l Nuclear Plant Reliability Data System (NPRDS) !

Nuclear Plant Experience (NPE) l Transamerican Delavel, Inc. (TDI) Owners Group j The randomly selected failures occurred between 1965 and 1984, but most l actually occurred in the later 1970's and 1980's. Relatively few plants !

were in service prior to 1975, and the data collection at those plants was sparse. Although failure causes were divided into those caused by aging and those not caused by aging rather than being categorized by more conventional l severity and cause breakdowns, there were several breakdowns useful for this study. The breakdowns by subsystem, component, and diesel generator manufacturer provide some historical evidence as to the best areas for a reliability program to focus.

Table I-1 contains the failure data from NUREG/CR-4590 categorized by subsystem and diesel manufacturer. The severity of failures cannot be gleaned from the information sources, but since the four information sources only contain catastrophic and significant noncatastrophic failures (by varying definitions), the failures should illustrate the types of failures that a reliability program should address.

Table I-2 provides an alternative display of the data from Table I-1. For each manufacturer, the highest number of failures was ranked as 1, second highest was ranked as 2, etc. The few ties that occurred were ranked according to the analysts estimation of the severity associated with failures in those subsystems. For each manufacturer, the ranking was.

terminated at the level where there were too few failures to tell difference. The "overall rank" column was obtained by ranking the sumsthe of failures listed in Table I-1. Engineering insight can be gained from Table 1-2, but further discussion of the nature of component failures is warranted. The following is a discussion of significant reliability insights, by diesel generator subsystem, starting with the subsystem with the most failures. The qualitative comments generally are based on the more detailed breakdowns of component failures and subsystems found in NUREG/CR- l 4590. !

Instruments and Controls (I&C1 The number of reported I&C failures was clearly larger than any !

other single type of failure. The I&C failures appear to be I about equally significant for each of the diesel generator manu-facturers, which can be expected since the I&C system design l varies from plant to plant, but not much from manufacturer to !

manufacturer. The reason for the large number of failures is the catchall nature of this category. The governor subsystem domin-ates the category, followed by startup-related I&C. The sub-system breakdown recommended for a diesel generator reli:bility I-6 <

l!

TABLE I-1.

SUBSYSTEM FAILURES BY DIESEL GENERATOR MANUFACTURER (NUREG/CR-4590 Data)

SUBSYSTD1 A]&Q &[&B QI Q gj Dj H2 IM !$2B g Engine Structure 2 0 0 12 13 $5 3 11 3 100 Drive Train 3 0 0 10 5 18 2 11 0 $1 Valve Mechanism 3 0 0 7 2 1 2 3 0 18 Startins 0 0 1 16 9$ 40 7 15 6 197 l Intake & Ezheust 10 0 0 10 31 41 0 17 1 113 I Fuel 1$ 1 0 21 44 75 14 26 8 215 o Lubrication 11 2 0 23 41 33 7 6 4 131 Cooling 21 1 6 5 65 36 15 18 1 172 Generator 5 1 0 6 49 27 1 7 3 105 l

Switchgear 15 1 0 11 86 60 5 0 4 194 j IK 36 4 9 73 169 112 23 40 14 496 Structural 2 0 2 0 1 5 0 1 0 9 Hech anic al 3 0 0 1 1 5 1 0 1 12 Electrical 8 0 0 11 53 18 11 4 3 i 112 Miscellaneoua 0 0 0 0 2 0 0 0 1 3 Unrelated 1 0 0 0 11 2 2 2 0 31 Noncorponents 1 0 0 1 13 5 1 0 1 23 l Total N d er

1 I

of Tailures 147 10 18 207 681 $33 9$ 164 49 I

1984 I

I Nanbet of Diesels in Service 18 3 5 11 84 49 8 31 .4 213 (1985)

Arbreviatiers ALCO - ALCO ALC3 - Allis Chamhers CAT Caterpiller CB

Cooper Besseer

@t -

Electro-Notive Division of General Motors m - Tattbanks Horse FBG

Nordberg TDI - fransamerican Delaval, Inc.

WCR - Worthington I-7

TABLE I 2.

A RANKING

OF DIESEL CENERATOR SUBSYSTEM FAILURE OCCURRENCES (NUREG/CR-4590 Data)

OtTRAI.L ID.111IR$ G ALCH GAI G.R 2! 2 H2 I21 EE PAW Engine Structure 5 11 4 7 8 10 Drive Tra % 9 10 6 11 Valve Mechanism 10 14 Starting 7 3 4 2 6 6 5 3 3 Intake & Exhaust 6 8 9 5 6-7 Tuel 3 3 7 2 3 2 2 2

, Lubrication S 2 8 8 5 5 6 Cooling 2 2 12 4 7 2 3 5 Generator 9 11 6 9 8 7 9 Switchgear 4 7 3 3 7 4 4 IEC 1 1 1 1 1 1 1 1 1 1 Structural 16 Mechanical 15 Electrical 8 6 5 11 4 6 8 Miscellaneous 17 Unrelated 12 12 Noncoeponents 10 13 Total Nuctor

of Tailures 147 10 18 207 681 533 95 184 49 1984 in Data Base i

This ranking is obtained free the information in Table 1-1 and is based on the number of failures shown for each manufacturer and in the

Total" coluem.

1-8

I program (Table B-1) separates these failures from other I&C because of the specific nature and reliability impact of these ;

failures. ,

Fuel Subsystem !

The fuel subsystem experienced the second or third largest number of failures for all the major manufacturers except for GM l i diesels. The GM fuel subsystem showed the seventh largest number l of failures. In general, fuel system failures were dominated by

injection component failures (injectors, injector pumps, injector nozzles) and by piping on the engine. GM diesels experienced l l

fewer injector failures (relative to other components) than any l other diesel generator. Injector reliability is closely related to periodic preventive maintenance and inspections, as well as to design.

S Startina Subsystem i

! The diesel generator starting system consists of components made i by diesel generator manufacturers and plant contractors.

The system, as defined in NUREG/CR-4590, consists of the air start mechanical components and their associated controls. The dominant failures occur in the starting air valve, controls, and starting motors. Catastrophic failure of starting motors results in the most risk, since the diesel generator should be air I started. Failures of the starting air valve and other controls !

are likely to be recoverable in a short time, allowing for a manual start of the diesel generator. Starting system failures are important for each diesel generator, especially far GM. GM has experienced failures of the starting motors somewhat more frequently than for other diesel generators. However, since no I certain count is available of how many EDGs use air start motors l and how many inject air directly to cylinders via a distributor ,

I instead of using an air start motor, no specific conclusion can be made. l The major failure cause is attributed as "adverse l environment: dust, humidity, chemicals, etc."

)

Switchaear Comoonents i

Switchgear component failures are important for all types of diesel generators. However, no trend or correlation to manu-facturer is noted. This category is mainly comprised of breaker and relay failures caused by poor manufacturing and construction.

Applicable reliability program activities include periodic inspection (non-teardown) and cleaning and more specialized condition monitoring such as ground detection and infrared surveys.

Coolina Comoonents Pump, heat exchanger, and piping failures each contribute about equally to cooling system failures. Cooling system failures comprise a significant part of the total number of failures for I-9

all but CB diesel generators, which have experienced very few cooling failures of any type. FM diesel ger.erator cooling subsystems appeared somewhat better than average due to few pump and heat exchanger problems. The following diesel generator manufacturers were noted to have specific problems:

ALC0 -

Piping TDI -

Intercoolers NBG -

Pumps Lubrication Comoonents The next most important diesel generator subsystem is lubrica-tion. Lubricating oil system failures were dominated by pump and heat exchanger failure, with filters, piping, and oil also contributing significantly. Although lube oil systems are

" specific to each diesel generator type, the variations across manufacturers were small. CB exhibited a high occurrence of filter problems, making lube oil the second most frequent source of failures. FM had more failures attributed to oil content, relative to the total number of lubrication component failures.

Intake and Exhaust Intake and exhaust failures are overwhelmingly dominated by the turbocharger. A significant percentage of these failures appear to be catastrophic, and the degraded and incipient failures often require very long repair outages, making turbocharger an important component. Turbocharger failure predominance does not appear to vary greatly with diesel generator manufacturer, but TDI experiences a little more than expected, and NBG has experienced the lowest relative number of these failures. Turbo-chargers are subjected to high stress levels in adverse environ-ments and should be addressed in a reliability program. ,

Electrical Comoonents Electrical components are important to most diesel generators; but tneir construction is specific to each plant. NUREG/CR-4590 breaks components into switches, wiring, transformers, controls, and others. Most failures fall into the "others" category, so no pattern emerges.

Generator The number of generator failures are fairly low, but significant, for each of the major diesel generator manufacturers except for GM.

GM's generator and voltage regulator failures appear more frequently than normal. Since GM doesn't manufacture these components, the high occurrence rate may be coincidental.

1 10

Enaine Structure l

The engine structure subsystem consists of the base, block, '

crankcase, main bearings, cylinder liners, and head, with the l most failures attributed to the crankcase. Engine structure '

ranks higher in failure frequency for CB and FM than for other manufact irers. CB failures were dominated by liners and FM i failur 'ere dominated by the crankcase, liners, and bearings.

These . ems probably are vendor specific. They will often be !

catastro pic to the diesel generator mission and usually will be !

expensive and time consuming to fix. Condition monitoring, such as vibration and clearanco measurements, and cylinder temperature l

trending are a key to reducing these problems. These problems i often require engine overhauls to correct, so it is important to predict them so that repairs can be done during a reactor outage.

The above-mentioned failures account for over 92 percent of diesel generator failures in the NUREG/CR-4590 data base. The remaining subsystems, in order of decreasing importance, are drive train, unrelated, noncomponents (mostly human error), valve mechanisms, mechanical, structural, and miscellaneous.

These failures are fairly rare, and there are insufficient data to make significant observations.

The failures in the NUREG/CR-4590 data base can be tabulated by component failures in a manner similar to that used for subsystem failures. Breaking failures down into critical components provides additional information about diesel generator failure characteristics. Table I-3 contains the number of failure records for each component and is backed up by Table I-4, which provides a ranking of diesel failure occurrences by component. As expected, the governor failure was dominant in NUREG/CR-4590. It was the leading contributor to the total number of failures for each manufacturer. The consistent governor failure rate is not surprising since, to the best of our information, all EDG governors are made by the same manufacturer (Woodward).,

Four out of the top eight components contributing to the number of diesel !

generator failures fall under the general category of I&C. Relays in the switchgear system show a large contribution, being the l largest contributor. Since relay fsilures are generally complete second i and catas- !

trophic rather than degraded or incipient, plants that experience switchgear j relay problems need to address them in a reliability program. Relays in !

standby systems tend to fail because of standby stress-related causes, such !

as corrosion, dust buildup, and other slow degradations. Surveillance tests that allow more frequent operation of control relays would decrease the exposure time for undetected failures, and the increased number of cycles would probably reduce sticking-type failures without significantly ,

increasing wearout. Unfortunately, most switchgear cannot be tested and cycled during normal plant operation, so condition monitoring, especially visual inspection, is often recommended. Other major I&C problems come from starting control 110, nonswitchgear relays, and sensors. The failure modes and their corrections are similar to switchgear relays.

The next major contributor to problems is the turbocharger. More turbo-charger failures tend to be degraded or incipient than catastrophic.

1-11

I l

l TABLE I-3. <

DIESEL GENERATOR FAILURES BY COMPONENTS (NUREG/CR-4590 DATA)

I COMPo s t gCg gl,2 QI Q RQ E D2 121 E!g IgIg;,

l 1

n Governor 19 0 4 15 66 33 4 14 4 159 1 Relays (switchgear) 6 0 1 6 44 24 3 0 1 85 1 Turbocharger 6 0 0 8 30 21 0 12 1 7s 1 Sensors (IbC) 4 1 1 11 22 17 4 5 3 68 Fuel Piping 2 0 0 14 7 22 3 9 1 $6 Break ers 6 0 e 4 20 16 1 0 1 $s fearting Controls 2 0 3 3 30 9 3 1 0 $1 Relays (It.C) 1 0 1 2 19 12 3 3 3 64 Starting Airvalve 0 0 0 6 12 11 3 9 4 45 Starting Motors 3 0 0 2 33 7 0 1 0 46 Injector Pumps 1 0 1 $ 1 17 4 11 5 45 startup I&C 2 1 0 2 19 15 1 4 0 44 CNorspeed Governor 3 1 0 11 li. 8 1 2 1 36 Voltage Regulator 0 0 0 4 14 13 0 2 2 35 Alarms and shutdowns 2 0 0 4 7 9 4 6 0 32 Cooling Systee Piping 9 1 0 3 $ 12 0 3 0 33 Cooling Best Exchangers 2 0 0 1 22 3 4 0 0 32 Crankeese 0 0 0 0 4 21 0 4 0 29 Control Air I&C 4 0 1 17 5 2 3 0 0 32 Lube 011 Beat Ezchangers 4 2 0 4 12 6 0 0 1 23 Lube 011 Pumps 4 0 0 1 17 2 1 0 0 25 Generator 4 1 0 1 13 5 0 2 0 25 w

TOTALS 84 7 12 124 421 287 42 C8 27 1092 i l l

i

I-12

TABLE I 4 CRITICAL COMPONENT RANKING

FOR EACH EDG MANUFACTURER (NUREG/CR-4590 Data)

C m NT M M G.61 0 22 2 E12 I21 h3 (by overall rank)

1. Governor 1 1 2 1 1 1 1 2

2. Relays (switchgear) 3 7 2 2 6

3. Turbocharger 4 6 4 4 2

4. Sensors (! Q 6 4 7 7 2 l 7 4 l S. Puol Piping 3 17 3 7 4

6. Break ers $ 10 6 6

7. Starting Controls 2 14 5 14 8

8. Relays (!&C) 9 11 9 10 A. Starting Airvalve 8 14 13 10 $ 3

10. Starting Motors 11 3 17

11. Injector Pumps 9 8 3 $ 1

12. Startup !&C 10 9 8

13. overspeed Governor 12 $ 16 16

14. Voltage Regulator 11 12 10 i

15. Alarms and shutdown: 12 18 15 4 6

16. Cooling System Piping 2 15 19 12 11

17. Cooling Heat Exchangers 6 20 $ !

18. Cr arJ e s s e 21 $ 9

19. Control Air IEC 7 1 20 11 .

20. Luhe oil Beat Exchangers 8 13 15 18

21. Lube 011 Pumps 9 11

22. Generator 10 13 10 6

The rank is derived from Table I-3, and represents the rank by manufacturer of each component.

l l

l l l

I-13 l

i

However, turbocharger repairs tend to require taking the EDG out of service for long periods. Any turbocharger repairs that cannot be scheduled during an outage will significantly affect diesel generator reliability.

Table I-4 also ranks, in as detailed a manner as possible, the component failure importances by diesel generator manufacturer. With a few notable exceptions, they generally follow the same overall ranking. The noticeable exceptions may be due in part to the random selection of failures that were included in the NUREG/CR-4590 data base. A more detailed analysis would be required to identify problems that are specific to diesel manufacturers, but the generic order would provide a good starting point for prioritizing reliability program activities.

I.2.2 Insichts From a Collection of NPRDS Data NPRDS uses a different set of component and subsystem boundaries than NUREG/CR-4590, so it is somewhat difficult to make direct comparisons between the data sources. NPRDS failure records generally provide good failure analysis and corrective action descriptions for failures of mechanical components (governors, turbochargers, etc.) but contain relatively poor electrical and electronic equipment failure analysis. The differences in subsystem and component breakdown between NPRDS and other sources, and the difference between individual EDG I&C, make it impossible to do a generic study of I&C problems without reading and classifying each failure report. However, with a specific plant problem, it has been proven beneficial to use NPRDS to analyze I&C.

Recent NPRDS data support the previous conclusions about mechanical components. Governor failures are important to each diesel generator manufacturer (about 1 in 15 failures) and are often catastrophic. Turbo-charger failures are important, more from their contribution to maintenance outage thn from frequent catastrophic engine failures. Many of the turbo-charger maintenances require involved work that is performed during raactor outages. However, catastrophic EDG failures due to turbocharger failures are important and occur fairly often.

Table I-5 shows the distribution of diesel generator failures contained in the NPRDS data base used for this study. The records were found using the NPRDS SEEK function for the EDG air start, EDG fuel oil, EDG lube oil, and EDG coolir.g subsystems, and the diesel-generator-related components of the emergency power system. A usable data base of 2458 was created.

Unfortunately, the NPRDS data base does not contain enough information to calculate an equipment online time or the number of demands, and thus failure rates cannot be estimated. Additionally, the NPRDS data are also influenced by a plant-specific interpretation of reporting requirements.

Failures should be classified by severity prior to entry into the NPRDS systim; only catastrophic and degraded fail m .s are entered. The classifi-cation by severity becomes judgmental. He.m er, the sample is believed to be large enough and random enough to iden ity any significant problems or strengths of EDGs. No glaring differen&s in the major contributors to diesel generator unreliability and unavailability were noted when comparing NPRDS data to NUREG/CR-4590 data.

1-14

~

l 1

l I

l TABLE I-5. )

NPRDS-DERIVED EDC INSIGHTS 5

-1 CATASTROFEIC DEGRADED *INCIFIENT TOTAL pureER l

< YENDCES FAlltills FAILURES M'%1. OF FAILtELIS l 4

(EDO Itself. No Subeyotes)

DC 121 2$5 13 339 C3 33

{

106 - 10 149 ALCO 13 16 1 32 FN 78 162 19 259

, NOM 4 17 2 27 WORT 4 11 0 13 TDI 8 9 0 11 sea (Cooling Subsystem)-

i DC 21 39 0 60 CB $ 9 0 19

. ALCO O $ 0 5 IN 11 93 2 106 1 NORD 2 9 1 12 WORT C 0 0 0 TDI 1 8 1 12 207 (Air Start Subsystem)

DC 72 344 9 425 CB 10 63 2 77

, ALCO 6 10 1 17 FM 35 235 4 274 NORD 6 28 3 37 WORT 3 32 0 35 !

TDI 17 58 '

4 3

944 (Fuel dal subsystas)

DC 7 34 0 61 CB 1 2 0 3

, ALCO 3 0 0 3 I

IN 6 35 0 41 i NOC 1 3 0 4 WCELT C 0 0 0 TDI 1 1 0 ,,,,,1 i

114 (Lube C11 Subsystee)

De 37 98 9 144 CB 4 10 3 17 ALCO 14 14 0 32 1 FN 13 b? 2 7$

PCC 3 13 4 20 WORT 1 2 2 3 TDI 1 9 22 ,,,11 305

Instptent Failures reports are not required to be submitted to NPRDS.

A i

I-15

Review of the data in Table I-5 provides se'eral insights and rankings. In the NPRDS category "EDG, no subsystem," R ich generally consists of the engine attached components, Cooper-Bessemer (CB) supplied engines showed a disproportionately high number of NPRDS records as compared to the number of diesels in service (11 percent of the NPRDS re'.ords are from CB diesels, but 17 percent of the "no subsystem" records are from CB diesels). CB engines tend to be older than most engines, so there may be a set of age-induced, or manufacturer-induced problems acting on these engines. The problems predominantly have a severity indicated to be degradedsbut the number of catastrophic failures is also slightly high. A more detailed, failure-by-failure analysis and categorization is necessary to identify specific problems and could refute the above hypothesis concerning aging.

Continuing down Table I-5, Fairbanks-Morse EDGs have experienced a disproportionately high number of degraded conditions in their cooling systems when compared to the total number of NPRDS records submitted from I other types of EDGs. This tendency is not apparent in Table I-l or I-3, so it may be a result of NPRDS reporting procedures. Another significant fact apparent from Table I-5 is that 29 percent of all cooling subsystem failures are classified as catastrophic. Most studies treat the coo'ing system as one unlikely to lead to catastrophic failure. Only engine failures, for which 30 percent of failures are catastrophic, has a higher fraction of catastrophic failures than the cooling subsystem. The air start, lube oil, and fuel oil subsystems experienced 15 percent, 23 percent, and 17 percent catastrophic failures, respectively.

Transamerican Delaval, Inc. (TDI) diesel generators tend to exhibit more air start failures, as compared to the total number of failures, than expected.

It may be misleading to attribute this phenomenon to TDI, since much of the starting system is not specific to the EDG vendor. GM also has experienced quite a few air start failures, as indicated in both Tables I-l and I-5.

The results of the NPRDS data breakdown are very similar to the results o f, the NSAC-108 study, which was performed over the same period.

I.3 INTERFACES WITH OTHER EDG RELIABILITY REVIEW ITEMS The collection and use of EDG failure information can be of significant use in developing an EDG reliability program. Operating experience should be eawn upon whenever possible. Surveillance needs, performance monitoring, and maintenance aspects would all benefit from operational feedback. In addition, the data collection system should provide offsite relatable experience. a means to access Generic information on diesel generator failures is also important for failure analyses and root cause investigations. Generic information is most useful when the analysis of a specific failure has begun, the nature of the problem is generally understood, and it has been determined that root cause analysis is needed. Information on the nature of problems, and their solutions, is generally obtainable from LER and NPRDS descriptions.

ever, How-detailed information often more difficult to find. on the root cause (or potential root causes) is Historical diesel data are most useful for initial searches for (1) how frequently a given problem or problem type has I-16

occurred and (2) who (meaning utility, vendor, etc.) has had similar prob-lem(s). The problem frequency is useful in prioritizing the root cause efforts and the identification of who has had similar problems gives an idea of who to contact for specific relevant experience.

However, extracting plant, or machine, specific conclusions should not be made from the condensed data bases provided in this appendix. The various limitations have been noted above.

I.4 EMERGENCY DIESEL GENERATOR DATA BIBLIOGRAPHY (with comments)

1. NUREG/CR-4557, "A Review of Issues Related to Improving Nuclear Power Plant Diesel Generator Reliability," J. Higgins, C. '

Czajkowski, A. Tingle, Brookhaven National Laboratory, BNL-NUREG-51969, April 1986.

' The data and recommendations of utility responses to Generic I Letter 84-15 and the recommendations for DG reliability by other groups (EPRI, vendors, NRC, etc.) are summarized and analyzed in l this NUREG/CR. The document is not a reliability data analysis 1 but does contain a large amount of information on views and concerns about diesel reliability.

2. NUREG/CR-4590, Vol. 1, "Aging of Nuclear Station Diesel Generators: Evaluation of Operating and Expert Experience,"

K. Hoopingarner et al., Pacific Northwest Laboratories, PNL-5832, '

August 1987.

This NUREG/CR contains a very large and elaborate diesel failure data base, with failures broken down by subsystem, component, diesel manufacturer, and various aging parameters. Any study of aging is important, but is beyond the scope of this task, so the i aging parameters were not used. An engineer administering a, diesel reliability program would find this a general.

reference as to the nature of other plant problems, good but it does not contain specific failure and root cause information.

3. NUREG/CR-2989, "Reliability of Emergency AC Power System at Nuclear Power Pl ant s , " R. Battle and D. Campbell, Oak Ridge i

National Laboratory, ORNL/TM 8545, July 1983.

This report contains the results of a reliability analysis of the onsite ac power system. It uses the results of a separate analysis of offsite power systems to calculate the expected frequency of station blackout. Included is a design and operating experience review and onsite power system models.

4. NSAC-108, "The Reliability of Emergency Diesel Generators at U.S.

Nwlear Power Plants," H. Wyckoff, Electric Power Research Institute, September 1986.

This report describes the EPRI effort toward organizing, investigating, and compiling a realistic data base of EDG l success / failure experience for the years 1983, 1984, and 1985.

1-17

~ ^'

\

EPRI chose not to count easily recoverable failures as a failure to start. The strength of the study is the concentrated effort to make the survey comprehensive, and to report the experience of all utilities in a rigorously consistent manner.

5. Nuclear Plant Reliability Data System (NPROS). Institute of Nuclear Power Operations (INP0).

The NPROS data for diesel failures between 1984 and the present l (Spring, 1987) have been collected. The data will be used to l augment the analysis of the NUREG/CR-4590 data base. Nearly 2500 .

NPROS failure reports were used for this study and could be used l for general insights into failures (e.g., by manufacturer, compo- '

nent, and subsystem) or for specific examples of diesel failures. ;

)

l I

i

)

i e

i l

I-18

.

ML20148S481

Contents

Text

SUBJECT:

SUMMARY

Enclosures:

1.0 INTRODUCTION

Subject:

Description:

Navigation menu

ML20148S481

Text

SUBJECT:

SUMMARY

Enclosures:

1.0 INTRODUCTION

Subject:

Description:

Navigation menu

Search