ML022270129

Presentation Material from Firstenergy for 08/15/2002 Public Meeting at R-III Headquarters on Root Cause of Davis-Besse Reactor Vessel Head Corrosion and Corrective Actions. Category 1 Meeting

ML022270129
Person / Time
Site:	Davis Besse
Issue date:	08/15/2002
From:	Myers L, Wuokko D FirstEnergy Corp
To:	Hopkins J NRC/NRR/DLPM/LPD3
Hopkins J
References
Download: ML022270129 (49)

Text

Davis-Besse Nuclear Power Station Management and Human Performance Root Causes August 15, 2002 1

Management and Human Performance Root Causes Introduction Lew Myers FENOC Chief Operating Officer 2

Agenda

• Introduction

- Lew Myers, FENOC Chief Operating Officer

• Management and Human Performance Root Causes

- Steve Loehlein, Root Cause Analysis Team Leader

• Corrective Actions and Summary

- Lew Myers, FENOC Chief Operating Officer 3

Desired Outcome Discuss the root causes:

• Management Oversight

• Corrective Action Program

• Technical Rigor

• Program Compliance And the key corrective actions to be taken 4

Management Conclusions Introduction

• Earlier Root Cause investigation and the NRC Augmented Inspection Team report both concluded that management had ineffectively implemented processes, and thus failed to detect and address plant problems as opportunities arose

• Root Cause Analysis Team was tasked to determine WHY the significance of the conditions in the plant was not recognized 5

Root Cause Team Introduction Team Composition

• Technical Root Cause Analysis Team Leader

• Root Cause Qualified FENOC Employees

• FENOC Oversight Support

• Process Expertise (Conger & Elsea)

• Davis-Besse Management Personnel Industry Input

• Root Cause Process Experts

• Organizational Effectiveness Experts

• Organizational Development Consultants 6

Root Cause Team Introduction Root Cause Team Root Cause Analysis Team

• Lead: Steve Loehlein (Beaver Valley)

• Bill Babiak (Perry)

• Mario DeStefano (Perry)

• Randy Rossomme (Beaver Valley)

• Lesley Wildfong (Conger & Elsea)

• Bill Mugge (Davis-Besse)

• Joe Sturdavant (Davis-Besse)

• Bobby Villines (Davis-Besse)

• Dick Smith (Conger & Elsea)

• Dorian Conger and Ken Elsea (C&E)

• Spyros Traiforos 7

Root Causes Steve Loehlein Root Cause Analysis Team Leader 8

Root Causes Problem Statement Understand why, over a period of years, Davis-Besse personnel failed to identify corrosion of the Reactor Pressure Vessel Head base metal 9

Root Causes Root Cause Statement There was less than an adequate nuclear safety focus

- There was a focus on production, established by management, combined with taking minimum actions to meet regulatory requirements, that resulted in the acceptance of degraded conditions 10

Root Causes Investigation Process 11

Root Causes Approach

• Scope Development

- Technical Root Cause results provided clues Errors occurred over several years, and in several areas

• Program effectiveness needed to be assessed

• The potential for boric acid to cause damage was an issue for the plant in 1998/99, but actions taken then did not result in detection of head corrosion 12

Root Causes In-Depth Evaluations

• Event and Causal Factors chart and Barrier Analysis techniques used

• Management Oversight and Risk Tree (MORT) technique used

• MORT Analysis Sections

- Technical Information Systems

- Corrective Action Program

- Hazard Analysis Process

- Task Performance Errors

- Management Support / Oversight

• Corrective Actions based on conclusions of data analysis 13

Root Causes Data Sources

• Technical Root Cause Analysis Report

• Interviews (more than 120)

• Documents (approximately 700)

• Over 20 years of Data 14

Root Causes Presentation Sequence of Data Analysis/Results 15

Root Causes Presentation Sequence of Data Analysis/Results

1. Boric Acid Corrosion Control and Inservice Inspection (ISI) Programs

2. Handling of Technical Information

3. Corrective Action Program

4. Hazard Assessment Process

5. Management Oversight/Risk Assessment 16

Root Causes Boric Acid & In-Service Inspection Programs

• Hazard-Barrier-Target Analysis evaluated the Boric Acid Corrosion Control and Inservice Inspection (ISI) Programs Model assumes that boric acid is the hazard, and the RPV head is the target Barriers included design, training, inspection for leaks and corrosion, cleaning, and corrective actions (nearly 50 in all) 17

Root Causes Conclusions

• The failure to clean the RPV head (a failed barrier in the analysis) prevented the team from analyzing the behavior that would have resulted had a bare head inspection been preformed

• The RPV head was not a focus in the process

• None-the-less, had the programs been followed as required, they were adequate to have prevented the serious head damage 18

Root Causes Handling Technical Information

• Evaluated using MORT analysis technique

- Evaluated the process the plant used to ensure that technical information was properly assessed and incorporated 19

Root Causes Conclusion

• The process for disseminating and incorporating technical information was adequate

• Personnel failed to correctly apply key industry information and plant knowledge about the potential harmful effects of boric acid 20

Root Causes Corrective Action Program

• Utilized MORT technique and Change Analysis

• Examined critical steps in process

- Initiation

- Operability Review/Categorization

- Cause Analyses

- Corrective Actions

- Trending/Effectiveness Reviews 21

Root Causes Conclusion

• Davis-Besse adequately identified and documented nonconforming conditions

• Personnel at ALL levels did not effectively implement the Corrective Action Process Operability/operational impact underestimated Categorization did not recognize significance Shallow cause analyses Inadequate corrective actions Inadequate trending of recurring equipment problems 22

Root Causes Hazard Assessment Process

• Evaluated using MORT analysis technique

• Examined how the organization used it to recognize and evaluate nuclear safety risks

• A key focus was on the application of the process for evaluating issues subject to 10CFR50.59, Safety Evaluations 23

Root Causes Conclusions

• Hazard Analyses process contained the necessary elements to ensure the design and licensing basis was maintained

• Process that required detailed analysis became less restricted over time Result was that in later years, the process was not applied to Containment Air Coolers, Radiation Monitor Filters, or boric acid on RPV head 24

Root Causes Management Oversight/Risk Assessment

• Started from Technical Root Cause Analysis Report

- Timeline of Key Events (Figure 26) from that report provides insight

- Plant conditions provided clues that problem existed

• Reactor Coolant System unidentified leak rate

• Containment Radiation Monitor Filter plugging

• Containment Air Cooler cleaning frequency

• Boric acid accumulations on the head 25

Root Causes Management Oversight/Risk Assessment (continued)

• Examined 22 Condition Reports (PCAQRs & CRs)

- Operability/operational impact underestimated by engineering and operations

- Low categorization, with no root cause analysis required

- Shallow cause analyses with focus on resolving symptoms rather than cause

- Corrective Actions deferred resolution or treated symptoms

- No collective significance recognized or evaluated; no visible senior management sponsorship 26

1.0 Timeline of Key Events 50 0.9 P ress urize rRe lief M onthly Avera ge

1. N umb er of Nozz les Not 0.8 Va lve Lea ka g e 40 0.7 0.6 30 0.5 Un ide ntifie d L eaka ge (gpm )

V isibl ein Vis ual Inspection s 0.4 20 0.3 0.2 10 0.1 0.0 0 PC AQR 96-055 1 due Rup ture disc bur st TM P re ss ur iz er c ode s af et y C R w rit ten on R CS to inability t o 10/ 17/98 va lve s repa ired when leaka ge a ssociated wit h comply wit h NG -E N- lea kage is .8 gpm. 5/99 da ta sca tt er . 3/01 Cont ainment Sump 00324. B&WO G le aka ge >1 gpm PCAQR 96-0 551 boric CR wr itt en on fluctuating At tributed to C RD f lange Safe ty E valuat ion acid on the hea d is R CS lea ka ge .25 gpm. 7/01 leakage t a kes cre dit fo r downgraded and close d visual inspect ion. RCS U/I leakage incre asing; aft er 2 years 9 month s.

L eaks not identif ied. 10/01 Co ntai nment Mo nit or BA M onthl y re pla c ement for preventive ma int enanc e 1 we e k 1 -2 dy 2 wk s 2 wk s 1 wk 1 -2 dy Filte r Repl acemen t AA M onthl y replac eme nt for preve ntive ma int e nance 1 wk 2 wk s 2 wk s 1 wk 1 -2 dy Low flow alar m Filter p ape r con tains ir on RP Vh ea d is no t Pla nt Engine ering Four H EP As C R 99- 1300 closed Independent Laborator y Sa mpl epo int is swappe d due to bor on o xide, c r edite d c ons ide red as a believe s the c ause o f inst all ed fo r wit hout re s ult s of parti cula te in an att e mpt t o reduc e bui ld- up - C R M a gna f lux. Cont ain men t source of ir on cor rosi on is CA C iron oxide det er mina tion o f sample indic at e the likely th ebor on build- up. 9/01 initiated. s ump sa mple s ha ve oxide. E ng. me mo plenum or pip ing o r remova l. cause. 9/99 or igin to be a stea m I odine ca r tridges a re 3/99 cor rosio n products. 7/99 to Opera tion s c onduit c orr osion . 7/99 ( TM ) 8/99 sourc e. 11/99 removed due t o det ecto r sat ura tions . 11/01 17 2 5 4 Fir st C AC cle anings C NRB is to ld t he run CR is suedbased on CR wr itt en t o Root ca use on Numbe r of Ai r Cooler Clean ing s s ince 0 2/24/92. 11/98 times bet we e n cleaning CAC plenum pr essure evaluate imp act o f c leaning of the PCA QR wr itt en t o will incre ase be cause decrea sing aft er cle aning. 12/00 CAC s no t Ma gnaf lux le vel is s igni fic an t work during perfo rmed unt il docume nt s te p in CAC CR wr itt en t o p lenum pr essur e. 11/98 de c re as ing. 7/99 12RFO . 6/00 docum ent st ep dr op 13 RFO .

in CA C plenum Work O r der t o Valve is le aking One nut is missing, Ins pec t ion ever y 6 P CAQRs in 5 Collective pre s sure. 1/01 r ep lace yoke o n st ar tup, valve is f unct iona l. 30 days - month s. s ignific ance o f a nd repa ck the dec ision to use Sec ond nut is decis ion not t o (1st r oot caus e ) ma nagemen t 3rd r oot ca use on R C-2 4/00 Reac tor Coo lant 2 Valv e va lve , BAC C as-is . (No missing, va lve is repac k. Plant is Re turn to powe r, issue on R C 2 -

E ffe ctiveness review of C R Pr ogram is no t s af et y inope rable - PC AQR in f orced o ut age. va lve is PC AQR 98- 0020 98-00 20 det er mines ac tion s addr esse d. e valuation ) 98- 1681. V alve is r einjected . (2nd r oot caus e ) RC 2 packing le ak, no C R we r eef fect ive . 10/00 5/98 5/98 in jec ted. 9/98 10/98 10/98 12/98 P CAQR 9 6-0551 Varying sizes of bori ca cid mounds P CAQR 98- 0767 M ost of t he hea d is cover ed in a n une ven la yer CR 00-1 037 La r ge deposit s of bo ron and la va-sca tter ed in var ious are as: Hea dno t c le ane d. a long wit h seve ral " fis t" sized lumps: Not co mpletely cle ane d. like f low t hrough the mo us e holes.

Calendar 199 5 199 6 199 7 199 8 199 9 200 0 200 1 200 2 Op er atin g Cyc les C yc le 1 0 C yc le 1 1 Cy cle 12. A Cy cle 12 .B C yc le 1 3 1 0RFO 1 1RFO 1 2RFO 1 3RFO 27

Root Causes Why did the organization not recognize the significance of the plant condition?

• Conclusions from other analyses were used to develop an overall understanding of failure to recognize significance of the plant condition

• Additional MORT analysis was done in Management Policy/Incentives

• Added numerous interviews insights 28

Root Causes Conclusions

• Beginning in the mid-1990s, management focus was on production concerns Rigor in assessing issues for their potential impact on nuclear safety diminished:

Taking minimum actions to meet regulatory requirements was interpreted to be adequate for nuclear safety Management style was less directly involved, and relied on subordinates to escalate concerns 29

Root Causes Conclusions (continued)

• Results were:

Plant was restarted and run for extended periods with some degraded components Personnel performed with the philosophy that issues were not serious unless they were proven to be Rigor in processes declined at the same time that the threat of head damage increased 30

Root Causes Root Causes - Management Oversight

• A less than adequate nuclear safety focus and a production focus, combined with minimum actions to meet regulatory requirements 31

Root Causes Root Cause - Corrective Action Program

• Inadequate implementation of the Corrective Action Program:

Addressing symptoms rather than causes Categorization did not recognize significance Less than adequate cause determinations Less than adequate corrective actions Poor equipment trending 32

Root Causes Root Cause - Technical Rigor

• Failure to integrate and apply key industry information and site knowledge/experience; and to compare new information to baseline knowledge 33

Root Causes Root Cause - Program Compliance

• Some steps in the Boric Acid Corrosion Control Procedure were not followed 34

Root Causes Root Cause - Contributing Causes

• Some decisions were made without considering the need for a safety analysis

• Corrective Action Program was not state-of-the-art 35

Root Causes Other Key Observations

• Alloy 600 material used in the original design of the CRDM nozzles was susceptible to cracking/leakage; the original gaskets in the CRDM flanges were susceptible to leakage

• Training was not provided to individuals performing inspections for boric acid

• Inspections activities and corrective action were not coordinated through the Boric Acid Corrosion Control (BACC) Coordinator

• BACC Procedure did not specifically reference the CRDM nozzles as one of the probable locations of leakage 36

Root Causes Other Key Observations (continued)

• Condition Reports associated with the boric acid issues tended to stay unresolved until significant degradation occurred

• There was little evidence of QAs involvement and the documented findings by QA were of mixed quality

• Monetary incentive program rewards production more than safety at senior levels

• Written policies do not support a strong safety focus

• Operations had minimal involvement

• Management had minimal entries into containment 37

Corrective Actions Lew Myers FENOC Chief Operating Officer 38

Corrective Actions Extent of Condition Davis-Besse Building Blocks ensure adequacy of systems, programs, and the organization to support safe and reliable operation.

Specifically:

• The System Health Assurance Plan provides for rigorous system reviews

• The Management and Human Performance Excellence Plan will ensure a strong and sustained safety focus

• The Program Compliance Plan ensures programs meet industry high standards of performance 39

Corrective Actions Return to Service Plan Restart Overview Panel Reactor Head System Health Resolution Plan Assurance Plan Bob Schrauder Jim Powers Program Compliance Restart Test Plan Plan Restart Action Plan Randy Fast Jim Powers Lew Myers Containment Health Management and Assurance Plan Human Performance Excellence Plan Randy Fast Lew Myers 40

Management Oversight/

Nuclear Safety Focus Corrective Actions

• New Senior Management Team with high standards

• Implement the Management and Human Performance Excellence Plan

• Case Study Training and Rebaselining of Standards (how the event happened, what barriers broke down, and what needs to be different in the future)

• Safety Conscious Work Environment Survey and Assessment

• Organization Effectiveness

• Four Cs (Compliments, Communications, Concerns, and Changes)

Management (Structured Approach)

• Management Observation Program

• Evaluation of Directors and Managers 41

Corrective Action Program Corrective Actions

• Complete a Program Compliance Plan which includes a detailed latent issues review of Corrective Action Program by outside consultants

• Strengthen the Corrective Action Review Board to enforce higher standards for cause evaluations and effective corrective action (chaired by Plant Manager or another director-level individual)

• Ensure criteria used for categorization of significant or repeat equipment failures are appropriate and utilized by plant personnel

• Routinely perform assessments of categorization 42

Corrective Action Program Corrective Actions

• Repeat conditions are to be treated as Significant Conditions Adverse to Quality (SCAQ)

• Review existing long-standing issues for possible SCAQ categorization and use root cause evaluation techniques to obtain resolution

• Require the use of formal cause determination techniques for root and basic cause evaluations to ensure analytical rigor is applied

• Define and implement training for cause evaluations

• Improve guidance on reviews of the effectiveness of corrective actions (focus on verifying causes have been fixed and provide training on revised guidance)

• Implement an effective site-wide equipment trending program 43

Technical Rigor of Programs Corrective Actions

• Rebaseline Standards and Expectations in each FENOC Group

• Establish Engineering Assessment Board to reinforce standards

• Establish FENOC hierarchy of documents for consistent standards for analyses of safety issues

• Establish a Periodic System Walk-down Program

• Establish a Periodic Engineering Program Review Process 44

Procedure Compliance Corrective Actions

• Provide training to applicable personnel and managers that includes:

- Need to remove boric acid from components

- Inspect for signs of corrosion

- Perform inspections for signs of boric acid in component internals

• Reinforce standards and expectations for procedure compliance and the need for work-practice rigor

• Implement Management Observation Program with weekly schedules (used at Perry and Beaver Valley)

• Perform independent assessments of procedure compliance

• Discuss procedure compliance regularly at morning meeting 45

Contributing Causes Corrective Actions

• Establish the FENOC decision-making process at Davis-Besse, including hazard analyses

• Perform Corrective Action Procedure Benchmarking 46

Other Relevant Corrective Actions and Improvements

- Design of New RPV Head

- BACC Procedure Revision to include CRDM Nozzles

- Training on BACC Procedure

- Coordination of Boric Acid Control Activities

- Timely Corrective Action Resolutions

- Reviews of Quality Assurance Audits and Surveillances Adequacy

- Realign Incentive Program to Increase Focus on Safety

- Establish Policies to Support Safety

- Operations involvement

- Management presence in the field 47

Davis-Besse Site Organization VICE PRESIDENT - FENOC CHIEF EXECUTIVE VICE FENOC OVERSIGHT OPERATING OFFICER PRESIDENT - FENOC L. William Pearce Lew W. Myers Gary R. Leidich MANAGER - OPERATIONS VICE PRESIDENT -

EFFECTIVENESS NUCLEAR Michael J. Ross Lew W. Myers PLANT DIRECTOR - DIRECTOR - SUPPORT DIRECTOR - NUCLEAR MANAGER MAINTENANCE SERVICES ENGINEERING J. Randel Fast Michael J. Stevens Robert W. Schrauder James J. Powers, III MANAGER - CHEMISTRY MANAGER - MANAGER - MANAGER -

AND RADIATION MAINTENANCE REGULATORY AFFAIRS PLANT ENGINEERING PROTECTION Peter D. Roberts Patrick J. McCloskey Joseph W. Rogers Robert W. Pell MANAGER -

MANAGER - WORK LEARNING ORGANIZATION MANAGER -

MANAGER - PLANT CONTROL DESIGN BASIS OPERATIONS Dave Gudger ENGINEERING Michael J. Roder C. David Nelson John J. Grabnar MANAGER -

QUALITY SERVICES MANAGER - NUCLEAR Linda M. Dohrmann MANAGER -

TRAINING OUTAGE MANAGEMENT William A. Mugge C. David Nelson (acting) MANAGER - SECURITY Gary A. Skeel MANAGER -

48 HUMAN RESOURCES Deanna L. Haskins New to Position

Summary CEO of FirstEnergy has set the standard of returning Davis-Besse back to service in a safe and reliable manner, and doing the job right the first time.

We are committed to meeting this challenge.

49