ML030240394

Firstenergy Nuclear Operating Co. (Fenoc)/ Evaluations Performed at Davis-Besse to Address USNRC Inspection Manual Chapter 0350, Item No. 1, Adequacy of Root Cause Determination/ Letter, Table of Contents - Attachment B

ML030240394
Person / Time
Site:	Davis Besse
Issue date:	01/09/2003
From:	Myers L FirstEnergy Nuclear Operating Co
To:	Dyer J NRC/RGN-III
References
1-1299
Download: ML030240394 (192)

Text

FirstEnergy, DAVIS-BESSE NUCLEAR POWER STATION Root Cause Analysis Report Failure in Quality Assurance Oversight to Prevent Significant Degradation of Reactor Pressure Vessel Head CR 2002-02578, dated 6-13-2002 REPORT DATE: 9-10-2002 Prepared by:_-w R.-Ji Tad-cl T oot C se Leadd Reviewed by: D.0 Poole Investigative Team Leader Approved by: L. W. Pearce ,o 7 --L.Co,,I Vice President, Oversight

TABLE OF CONTENTS TABLE OF CONTENTS ......................................................................................................... ii EXECUTIVE

SUMMARY

................................................................................................... 1 PROBLEM STATEMENT ..................................................................................................... 4 DESCRIPTION OF REASON FOR INVESTIGATION .................................................................. 4 CONSEQUENCES OF EVENT/CONDITION INVESTIGATED ................................................... 4 IMMEDIATE ACTIONS TAKEN .......................................................................................... 4 REMEDIAL ACTIONS TAKEN .......................................................................................... 4 EVENT NARRATIVE ......................................................................................................... 5 TEAM CHARTER .......................................................................................................... 5 EVENT DESCRIPTION ................................................................................................... 5 BACKGROUND DESCRIPTION .................................................................................... 6 KEY EVENTS ............................................................................................................ 6 DATA ANALYSIS ........................................................................................................... 10 METHODOLOGY ......................................................................................................... 10 ANALYSIS M ODEL ...................................................... .. .......................... 11 ASSUMPTIONS ........................................................................................................ 12 DATA COLLECTION .................................................................................................. 13 FUNCTIONAL TOOL ANALYSIS CHARTS ................................................................... 13 DATA REVIEW ............................................................................................................ 15 FACT LIST ................................................................................................................. 19 CAUSAL FACTORS .................................................................................................... 20 CONCLUSION ............................................................................................................... 21 EXPERIENCE REVIEW ..................................................................................................... 23 DAVIS-BESSE CATS ................................................................................................. 23 DAVIS-BESSE CREST .............................................................................................. 23 NUCLEAR INDUSTRY .................................................................................................. 24 CONCLUSIONS ............................................................................................................. 25 ROOT CAUSE DETERMINATION ......................................................................................... 26 ROOT CAUSE ............................................................................................................ 26 CONTRIBUTING CAUSES ............................................................................................ 26 Root Cause Analysis Report Table of Contents a ii

28 EXTENT OF CONDITION ....................................................................................................

. ............. .29 RECOMMENDED CORRECTIVE ACTIONS ..............................

29 Preventive and Remedial Actions ....... ,............................................................. ....... ................... 33 an ntAc i ns.........................

Enem ....................................

Enhancement Actions ............... ..........................

REFERENCES....................................................................................3 34 Docum ents reviewed ...............................................................................................

34 Personnel contacted ...........................................................................................

34 Methodologies employed ...................................................................................

...................................... 34 REVIEW METHODOLOGY OF AUDITS AND SURVEILLANCES AND ISEG REPORTS ...... 35 REVIEW METHODOLOGY OF QA TREND

SUMMARY

REPORTS 35 REVIEW METHODOLOGY OF TIMELINE .........................................

36 REVIEW METHODOLOGY OF PERSONNEL INTERVIEWS ....................................

NRC DOCUMENTS .................................. 36 REVIEW METHODOLOGY OF 37 Team M em bers ........................................................................................................

38 ATTACHMENTS ................................................................................................................

39 Attachment 1 - Feedback Model Diagram .........................................................

40 Attachment 2 - Event & Causal Factor Chart ..................................................

.................... 41 Attachment 3 - Functional Tool Analysis Chart, Prior to 1ORFO through End of Attachment 4 - Functional Tool Analysis Chart, End of 10RlFO 11RFO ..................... ... ............. ................ 42 through End of Attachment 5 - Functional Tool Analysis Chart, End of 11 RFO 43 Mid-Cycle Outage of 1999 .......................................................................................

Outage of Attachment 6 - Functional Tool Analysis Chart, End of Mid-Cycle 44 1999 through End of 12RFO ...................................................................................

........................ 45 Attachment 7 - Functional Tool Analysis Chart, Post 12RFO

..... 46 Attachment 8 - Standard Interview Questions for Oversight Personnel 48 Attachment 9 - List of Oversight Personnel Interviewed ................................

Attachment 10 - Standard Interview Questions for Selected Line Management 49

..... 51 Attachment 11 - List of Selected Line Management Personnel Interviewed

................. 52 Attachment 12 - List of Personnel Interviewed by the NQA Team Iii Root Cause Analysis Report

EXECUTIVE

SUMMARY

Davis-Besse (D-B) Condition Report (CR) 02-02578 identified the failure in Quality Assurance Oversight to prevent significant degradation of the reactor pressure vessel (RPV) head. The essence of this issue and the resulting root cause analysis was to understand the key aspects of the operation of the oversight function at D-B and why they did not cause positive change in the site line organization such that the head degradation would have been found at a much earlier stage. The investigation assumed that the oversight organization had both the opportunity to influence the line organization such that they would detect the degradation and also an opportunity to directly detect the degradation during oversight activities.

The team concluded that the Oversight function did miss opportunities to cause earlier identification and mitigation of the RPV head degradation. The analysis further indicates that standards within the oversight function were insufficiently differentiated from the standards of the station. It was determined that the root cause was that D-B's nuclear safety values, behaviors and expectations were such that oversight was not set apart, in terms of expectations and performance standards, from the balance of the station. This affected the ability of the oversight organizations to identify problems and effect needed positive change in station operations. This was particularly applicable to the implementation of the Corrective Action Program and resulted in the station tolerating conditions that were potentially detrimental to safety for long periods of time.

The team began with a model of an oversight organization likened to a signal processor feedback loop and developed over 400 facts/observations gathered from the following sources:

"* QA Audit and Surveillance Reports

"* QA Summary Reports

"* ISEG Reports and Correspondence

"* NRC Inspection Reports and Correspondence

"* Personnel Interviews

"* Miscellaneous Documents developed from other sources The investigation spanned the timeframe from late 1986 until the discovery of the degradation in early 2002. The analysis was broken down into five time periods. This was done to facilitate the investigation due to the extensive time involved and in recognition of changes that occurred in the oversight organization over time, the type and nature of information available about the ongoing head degradation, and in the opportunities available to detect the degradation. The five time periods were:

"* Prior to IORFO

"* Beginning of IORFO to the end of 1 IRFO

"* End of 11 RFO to the end of the 1999 Mid-Cycle Outage

"* End of the 1999 Mid-Cycle Outage to the end of 12RFO Root Cause Analysis Report Executive Summary

• 1

• After 12RFO The investigation performed Event & Causal Factor (E&CF) analyses for the following six management "functional tools":

"* Corrective Action Program

"* Root Cause Analysis

"* Operating Experience

"* Trend/Analysis

"* Culture/Values

"* Audits/Surveillances/Evaluations "Functional Tools" as used in this context refers to those cross functional processes used by either line management or oversight functions to identify and resolve problems at nuclear power plants.

Using the principles of E&CF charting, the team identified 27 initial causal factors that were consolidated into seven formal causal factors. From these formal causal factors the team identified one root cause and three contributing causes as follow:

Root Cause:

Since the mid- 1990s, D-B's nuclear safety values, behaviors and expectations were such that oversight was not set apart, in terms of expectations and performance standards, from the balance of the station. This affected the ability of the oversight organizations to identify problems and effect needed positive change in station operations.

Contributing Causes

"* The training for the RC-2 event was ineffective. It failed to improve the ability of both the oversight and line organizations to recognize corrosive conditions and their significance.

This contributed to the failure of the auditing team to raise a concern when auditing the Boric Acid Corrosion Control Program during 12RFO.

"* Oversight did not establish an effective method for assessing the oversight function. The process for providing oversight of the oversight function was less than adequate, feedback provided was mixed, and corrective actions were sometimes ineffective.

"* For a period of time, the management of the audit/evaluation process was not independent from the management of the corrective action process. This lack of independence allowed the oversight director to soften the thrust of an audit critical of the corrective action process during 2000.

Key corrective actions:

• Establish, document, communicate and hold Oversight and Process Improvement Department (OPID) personnel accountable to expectations that support the ability of OPID to detect adverse conditions, process information, and escalate issues in a manner that ensures they are resolved in a timely fashion. The goal of this corrective action is to elevate the ability of OPID to cause positive change to occur at FENOC stations when warranted. The corrective action section of this report contains details relative to appropriate expectations.

• Modify the values of the oversight organization to set itself apart from the culture of the station, assure that it always maintains the highest standards with regards to a questioning Root Cause Analysis Report Executive Summary 9 2

attitude, and continually drive the site organization to a higher level of excellence. The oversight organization must maintain sufficient independence in their thought processes that they do not inherit an unhealthy station culture but rather maintain a strong questioning attitude about station activities affecting safety.

• Provide organization, staffing, tools, training, office location, etc. for the oversight function appropriate to revised expectations; match all elements of resources and expectations.

e Develop an improved method for oversight of the oversight function.

A total of 17 corrective actions are recommended in this report. Details of remaining actions are in the corrective action section of the report.

Executive Summary

• 3 Root Cause Root Report Cause Analysis Report Executive Summary e 3

PROBLEM STATEMENT DESCRIPTION OF REASON FOR INVESTIGATION Station oversight functions were not successful in identifying or effecting resolution of issues related to corrosion of the Reactor Pressure Vessel (RPV) head. In this context, station oversight functions include Quality Assessment/Quality Control (QA/QC) audits, QA/QC surveillances, and Independent Safety Engineering Group (ISEG) activities. The investigation evaluated the interface with other station groups like the Station Review Board (SRB), the Corrective Action Review Board (CARB),

and the Company Nuclear Review Board (CNRB). Oversight's assessment of the Corrective Action Program (CAP) and Management Review Process were also evaluated.

CONSEQUENCES OF EVENT/CONDITION INVESTIGATED The organizational consequences, were a lack of timely identification of various significant conditions adverse to quality, and a failure to assure that the operating organization had taken necessary and sufficient actions to preclude degradation of the Reactor Pressure Vessel head. The nuclear safety consequences were a challenge to a principal fission product barrier and extensive boric acid corrosion of other equipment inside the containment building. In addition, the station suffered a loss of regulatory confidence, and is undergoing a costly outage to restore systems to an acceptable configuration and reestablish confidence in the operating organization.

IMMEDIATE ACTIONS TAKEN This Root Cause Team took no immediate actions. This Team was chartered to perform the Root Cause Investigation, beginning June 24, 2002.

REMEDIAL ACTIONS TAKEN This Root Cause Team took no remedial actions. This Team was chartered to perform the Root Cause Investigation, beginning June 24, 2002.

Problem atemern Problem Statement

• 4 0 dl Root Cause AnalysIs Report Cause Analysis

EVENT NARRATIVE TEAM CHARTER On February 27, 2002, the D-B staff issued Condition Report (CR) CR 2002-0891 reporting indications of through wall axial flaws in the weld region of #3 Control Rod Drive Mechanism (CRDM) nozzle. During the course of repair efforts on the penetration nozzle, the boring machine and the nozzle unexpectedly rotated approximately 15 degrees. Investigation of this condition under CR 2002-01053 identified significant corrosion around nozzle #3. As a result, a Root Cause Team was assigned to "Determine the root and contributing causes for Reactor Pressure Vessel closure head (RPV head) damage experienced at nozzle 3 and minor corrosion at nozzle 2, to support the operability determination for the station's as-found condition and the future repair plan." This Team was known as the "Technical Root Cause Team"; their report was completed on April 15, 2002.

During the performance of the technical issue analysis of the root cause evaluation for the degradation of the RPV head, five previous issues within the site's Corrective Action Program (CAP) were identified as related issues to this event. A Nuclear Quality Assessment (NQA)

Team was charged with performing a review of these five issues to provide a better understanding of the contributing management issues related to past corrective actions.

Additionally, the NQA Team performed comparisons to the 1998 root cause evaluation of the RC-2 boric acid degradation of body to bonnet fasteners to determine if corrective actions from that event were effective. The NQA Team identified several common f~ctors that contributed to past issues and the current degradation to the RPV head. These were identified in six additional CRs, including CR 02-02578, Oversight Effectiveness, which is the subject of this investigation.

The remaining five CRs are being addressed separately.

Condition Report CR 02-02578, Oversight Effectiveness, documented the NQA Team's observation that periodically ISEG and QA were associated with past corrective actions, however they did not effect positive change to aid in the proper resolution of those issues. In addition to citing specific examples, CR 02-02578 indicated that a review of oversight activities should be conducted to validate that current methodologies will not result in the same weaknesses identified in their review. This assessment is being conducted by a second-NQA-Root Cause Team through an investigation of why the oversight function was not successful in identifying or effecting resolution of issues related to corrosion of the RPV head.

This Root Cause Team was convened under the sponsorship of the Vice President, OPID, on June 24, 2002, to conduct this investigation of oversight activities.

EVENT DESCRIPTION The station event of concern is the level of degradation of the reactor vessel head that occurred before it was detected. The subject or "event" associated with this investigation is why the oversight function was not successful in identifying or effecting resolution of issues related to corrosion of the RPV head. In this respect, the investigation is focusing on why something did not occur. The investigation focuses on the characteristics of a well functioning oversight Root Cause Analysis Report Event Narrative e 5

organization and evaluates their effectiveness during precursor events and activities. Changes that occurred in key aspects of an effective oversight organization were included in the assessment to determine their impact, if any, on the event.

BACKGROUND DESCRIPTION D-B is a raised loop pressurized water reactor (PWR) manufactured by Babcock & Wilcox (B&W). The RPV head has 69 eQntrol rod drive mechanism (CRDM) nozzles welded to the RPV head. Each CRDM nozzle is constructed of Alloy 600 and is attached to the RPV head by an Alloy 182 J-groove weld. The RPV head is constructed of low-alloy steel and is internally of clad with stainless steel. There is a service structure surrounding the RPV head. The bottom the service structure support skirt has openings called "mouse holes" to permit visual inspections through the use of a pole-mounted camera.

During performance of inspections of the CRDM nozzles during 13RFO, significant degradation of the RPV top head base metal was discovered. The Technical Root Cause Analysis Report concluded that corrosion of the RPV head was caused by boric acid corrosion resulting from CRDM nozzle leakage. The CRDM leakage resulted from through-wall cracking of the CRDM nozzles caused by primary water stress'corrosion cracking (PWSCC). That Report also concluded that a reasonable estimate of the time-frame for the appearance of leakage on the RPV rate head from the CRDM nozzle cracking is approximately 1994-1996, and that the corrosion began to increase significantly starting at about 11 RFO in 1998 and acted for a four-year period of time. During this period, boric acid accumulated sufficiently and provided the necessary environment to begin significant RPV head corrosion. The pre-existence of accumulation of boric acid from other sources, such as CRDM flange leaks, may have accelerated the corrosion and increased its severity.

Additionally, the Technical Root Cause Analysis Report concluded that the accumulation of to boric acid on the RPV head allowed the nozzle leaks to go undetected and uncorrected in time top of the RPV head over a prevent damage to the head. Boric acid that accumulated on the nor period of years inhibited the station's ability to confirm visually that neither nozzle leakage RPV corrosion was occurring. The Report also noted that other evidence of the boric acid not leakage existed in the containment building but its association with possible nozzle leaks was recognized at the time. This evidence consisted of 1) iron oxide, boric acid and moisture found in containment atmosphere radiation monitor filters, 2) boric acid accumulations in the containment air coolers (CACs), and 3) boric acid accumulations on the RPV flange. While these conditions were all identified at the time, their collective significance was not recognized.

KEY EVENTS A summary of the key events relative to this specific investigation follows.

OA Summary Trend Reports - These reports were issued periodically, often quarterly, throughout most of the period of interest. The Reports contained assessments and conclusions of station performance as evaluated by the oversight organization.

Audit Reports of Corrective Action Program- These reports were issued periodically throughout the period of interest. These reports cbntained assessments and conclusions of effectiveness of the Corrective Action Program.

Root Cause Analysis Report Event Narrative e 6

Industry Experience with Boric Acid Corrosion Prior to 1988 - Several incidents of boric acid corrosion (including one event involving corrosion of the Turkey Point RPV head) occurred between the late 1970s and the mid-1980s. These events led to the Nuclear Regulatory Commission (NRC) to issue Generic Letter (GL) 88-05 in 1988. GL 88-05 required each license holder for a PWR to have a boric acid control program. In response to this Generic Letter, D-B issued a boric acid corrosion control procedure in 1989.

Leaking CRDM Flanges in the. 1990s - D-B and other B&W plants experienced leakage from the CRDM flange gaskets. As a result, D-B replaced its gaskets over several outages from 6RFO in 1990 through IORFO in 1996. However, D-B also experienced leaks with the new gaskets in 8RFO (1993), 1 IRFO (1998), and 12RFO (2000). Thus, in every outage from 7RFO through 12RFO, CRDM flange leakage was identified (either from the original gaskets or the replaced gaskets).

Station commitment to perform head inspections (4/94) - Station investigation concluded that inspections of RPV head were not a commitment; this decision resulted in a change of categorization of the condition to one of "not a significant condition adverse to quality." This change was approved by the QA Director.

IORFO (1996) - One peripheral CRDM flange exhibited signs of leakage during 1ORFO. This nozzle exhibited rust/brown stained boron at bottom of nozzle where it meets the RPV head; the head area in this vicinity also has rust/brown accumulation. The boric acid on other parts of the RPV head was powdery and white. The boric acid was very thin at the front edge with powder and small clumps of boric acid on top. Based upon a justification that the boric acid would not impact the RPV head given its high temperature, boric acid was left on the RPV head.

1 IRFO (1998) - CRDM nozzle 31 was identified as having a minor flange leak, and it was not repaired. Boric acid deposits were identified flowing out of the mouse holes in the southeast quadrant of the RPV head flange. The boric acid was a reddish rusty color. During the removal of boric acid from the RPV head, the boric acid was noted to be brittle and porous. Other than these areas of accumulated boric acid, the RPV head was judged to be basically clean. Based on the 1996 assessment that the boric acid would not impact the RPV head given its high temperature, boric acid was left on the RPV head.

Boric Acid Wastage of Body-to-Bonnet Nuts for RC-2 Pressurizer Spray Valve (1998) - In 1998, two body-to-bonnet flange nuts on RC-2 Pressurizer Spray Valve at D-B were identified as missing. The root cause analysis report for this event concluded that the nuts were missing as a result of boric acid corrosion. The NRC took escalated enforcement action against D-B for this event.

RC-2 Evaluation (1/99): QA concluded, in a Surveillance, that Corrective Actions from RC-2 event were inadequate.

RC-2 Investigation (3/99): RC-2 investigation concluded that QA was not proactive in getting involved with RC-2 issues.

Root Cause Analysis Report Event Narrative

• 7

Modification to Service Structure (3/99) - The oversight organization (ISEG) concurred with the planned deferral of a modification which would have facilitated inspection and cleaning of the RPV head.

Performance Based Audit Process (1/00) - During this timeframe, the oversight function began a transition from compliance-based assessments to a mix of compliance-based and performance based assessments. Performance-based assessments are intended to add increased value to the station by observing the in process performance of activities and may include areas which are outside the requirements of Appendix B.

OA Functional Alignment (2/00) - Thins refers to a change in organizational structure within the oversight group. This change allowed oversight to dedicate individuals to primary functional areas (e.g., engineering, maintenance, 'etc.) for a sustained period of time.

12RFO (2000) - Steam cutting occurred on CRDM flange nozzle 31, resulting in boric acid leakage. A pile of boron was identified on top of the insulation. The boron on the RPV head was a red, rusty color and hard. Additionally, boric acid had accumulated on the RPV head flange behind the studs flowing out of the mouse holes in the southeast quadrant. The boric acid had a red, rusty appearance. The cleaning of the RPV head during the outage was not fully successful, and some boric acid deposits were left behind on the RPV head. In interviews, the engineer stated that he was running out of time to continue cleaning the RPV head (the RPV head was scheduled to return to the RPV during the next shift). No written evaluation was performed to allow the boric acid to remain on the RPV head.

Audit of Outage activities, 12RFO (2000) - QA conducted a formal audit of outage activities; the scope of this audit included the Boric Acid Corrosion Control Program. The Audit Report (improperly) recognized thorough cleaning of RPV head.'

Fouling of the Radiation Monitor Filters in 1998-2001 - In 1998, fouling of the containment atmosphere radiation monitor filters occurred. There were boric acid and iron oxide deposits on the filters. The deposits had a "yellow" or "brown" appearance. From May of 1999 until April 2001, filter changes were required on an irregular 1 to 3-week interval (and sometimes once every I to 3 days). Accumulation of boric acid on the radiation monitor filters was recognized to be symptomatic of an RCS leak as soon as it occurred. Efforts were made, especially during the cycle 12 mid-cycle outage in 1999 and later during 12RFO in 2000, to locate the source of leakage, but without success. By November of 2001, filter replacements were required approximately every other day.

Containment Air Cooler (CAC) Cleaning in 1998-2001 - In 1998 and 1999, cleaning of boric acid from the CACs was needed nineteen times. Although the boric acid was generally reported to be white, a written post-job critique indicated a "rust color" was noticed "on and in the boron being cleaned away" from CAC 1. In June 2000, CAC plenum pressure again began to decrease, requiring resumption of cleaning. Thi's was followed by five total cleanings in June, August, October and December of 2000. Cleanings continued in 2001, with four more (total) in January, February, March, and May.

FENOC-wide Oversight Organization (1/01) - In January 2001, a FENOC-wide oversight organization was put in place. This organization continued the assignment of oversight Root Cause Analysis Report Event Narrative

• 8

personnel to each station, but brought all management of oversight activities under one Director.

This Director reported to the FENOC President, outside the station organizations.

13RFO (2002) - The boric acid degradation of the RPV head was discovered.

Continuous Assessment Process (5/02) - Conversion from assessments which were conducted during well defined, although infrequent, short time periods to a mode of making frequent performance based assessments in numerous areas. This approach was specifically designed to meet the commitments of the internal audit program, while providing flexibility for real-time evaluation of site programs and processes. Effective and timely communication with the line organization was emphasized through frequent periodic debriefs as well as quarterly audit reports. Any identified performance issues were documented and addressed promptly using the site-wide corrective action program.

Event Narrative

• 9 Cause Analysis Report Root Cause Report Event Narrative

• 9

DATA ANALYSIS METHODOLOGY In evaluating available root cause methodologies, the team determined that Event and Causal Factor (E&CF) charting was the most appropriate method for this analysis. This analysis is unusual in that it is establishing the cause of why oversight functions were not successful, i.e.,

why various activities did not occur. Also, the large number of elements of an effective oversight organization combined with the extended time period over which station events occurred led to modifications to the traditional E&CF method. The method used was based on the TapRoot@

process, and performed root cause analyses on various opportunities which were present for the oversight function to detect and effect earlier resolution of the degradation of the RPV head. This resulted in some unconventional aspects of the E&CF charting as discussed below.

The steps in the methodology were:

1. Develop Problem Statement and Mission

2. Plan Investigation

3. Collect information (including inte~rviews)

4. Analyze data consistent with the analysis model described below (Detector, Signal Processor, Output, Alarm, Values, Environment) (iterative)

5. Determine sequence of events that occurred within six major functional areas

); construction of Functional Too! Analysis charts for the five identified time periods

> develop E&CF Summary Sheets and Detail Sheets for various time periods

6. Identify Causal Factors for the varirous E&CF sheets

7. Evaluate Causal Factors to identify Root and Contributing Causes

8. Combine the Causal Factors and Causes to eliminate redundancy

9. Develop an integrated E&CF chart for the entire period of interest

10. Identify Generic Causes

11. Develop Corrective Actions

12. Prepare Report (ongoing during earlier activities)

13. Obtain independent peer check of results

14. Resolve comments from peer check process Data Analysis

• 10 Cause Analysis Root Cause Report Analysis Report Data Analysis 9 10

ANALYSIS MODEL function was To facilitate the collection and analysis of appropriate data, a model of the oversight to the defined (See Attachment 1). The oversight function was viewed as being analogous provided to facilitate feedback circuit for a process controller. A brief discussion of the model is to the overall an understanding of the data collection and analysis process. The model is applied oversight function as it acts to oversee the management control systems and management is the organization in the conduct of day-to-day plant operation. The oversight function Quality Control, and Independent Safety collection of activities taken by the Quality Assurance, by QC Engineering organizations (with the exception of critical characteristic verification and personnel) to ensure station activities are conducted consistent with regulatory requirements the influence of commitments in a safe and reliable manner. The model also captures management expectations on the oversight functions, to the extent that station management established expectations for performance monitoring above and beyond regulatory minimums.

control Process Controller: The management organization and the attendant set of management with its design and systems applied to operating and maintaining the nuclear plant in accordance licensing bases.

and Process Output: The products of the Operating Organization's work activities to operate bases. This would include, but not be maintain the plant consistent with its design and licensing limited to, maintaining the plant equipment within design limits, compliance with applicable and quality programs, and implementing procedures. Typical process variations to be detected fed back to the process controller would be failures to follow procedures or programs as prescribed, or adverse trends in organization or equipment performance.

or method Detector (D): As applied to this study, "Detector" was construed to mean any means be to identify problems or issues with the "Process Output." Typical means of detection would This would also internal audits, inspections, evaluations, surveillances, and field observations.

include review of external audit or inspection results. Likely failures to "detect" would include "misplaced detectors" (e.g., not having auditors or evaluators looking at the right things) and "miscalibrated detectors" (e.g., not properly trained or qualified, or not working to the proper expectations).

Signal Processor (S): As applied to this study, "Signal Processor" applies to the collection, analysis and reporting of "Detected" information regarding the "Process Output." Typical activities conducted in this portion of the feedback model would include signal integration from various "Detector" outputs, and generation of "Output" to the "Process Controller" or, in the event predetermined conditions are exceeded, a "Controller Out of Limits Alarm." Likely failures to appropriately process detector signals would include improper or nonperformance of an analysis of trends in equipment or programs, or a program audit that identifies issues but does not address their significance.

Oversight Process Output (0): The third step in the feedback loop is for the signal processor to provide output or feedback to the control device in the main process, the line organization. This is the oversight organization's opportunity to have a positive influence on the process and must be delivered with the appropriate information in an appropriate manner and then accepted and acted on by the receiving organization. The "Outputs" would normally take the form of Audit Reports, Surveillance Reports, Summary Trend Reports, and/or Condition Reports. In the context of this analysis, failures of the Oversight Process Output are identified when the

- -I.

e., I L__ *I - 44 Root Cause Analysis Report Data Analysis

• itI

oversight organization sends the correct feedback message to the line organization, however, the line fails to take appropriate or effective action.

Values (V): As applied to this study, "Values" applies to the individual, site, and corporate values that tend to establish the expectations for the QA oversight function, and for the line organizations' responsiveness to the QA outputs. Although "Values" do not have a specific failure mode, "Values" establish gains, bias, and set points for the "Signal Processor," "Output,"

and "Controller Out of Limits Alarm."

falls Controller Out of Limits Alarm (A): Aý applied to this study, if the process output signal frames an "Out of Limits" outside prescribed limits and/or is not restored within prescribed time alarm is sounded. Typical activities conducted in this portion of the feedback model would include comparison of the output signal (from the signal processor) to pre-established did expectations. The most likely failure to "alarm" would include cases where the organization not react sufficiently to a warning by the oversight organization(s), and the oversight organization failed to escalate the issue, e.g., repetitive shots on the same target.

Environment (E): As applied to this study, "Environment" is limited to those external, indirect influences that could cause the Feedback Circuit to drift from expected performance through mechanisms such as complacency due to good performance reviews. Like "Values,"

"Environment" does not have a failure mode; rather it influences gains, bias, and set points.

ASSUMPTIONS As discussed in the Event Description above this investigation involves analysis of why or something didn't happen, (i.e., why the oversight function was not successful in identifying analysis, the effecting resolution of issues related to corrosion of the RPV head). To facilitate the following assumptions were made by the Team.

he/she Assumption 1: If the "Rational Person" had knowledge of the Reactor Vessel head leak, would have acted to mitigate the condition.

Assumption 2: The Oversight Functional Organization can be modeled as the "Rational Person."

Assumption 3: The Oversight Rational Person had two distinct opportunities to significantly alter the ultimate outcome:

a. Assure through proper oversight functions that the processes used by the Line Rational Person were sufficiently robust and effectively used to detect and mitigate the condition.

b. Direct detection and action to mitigate the condition.

that there Assumption 1, although appearing obvious, establishes a precept of this investigation was no willful or intentional attempt tb allow the head to continue to degrade by any person having knowledge of the condition of the head. The second assumption simply extends 3 builds assumption I to an organizational level within the oversight organization. Assumption within the upon the first two assumptions and reflects the opportunities available to individuals resolution of the oversight organization and hence the organization as a whole to have effected head issue.

During the course of the investigationý conclusionsaredrawn for different time periods regarding the ability of the oversight organizaticn to have effected resolution of the issue through the opportunities stated in assumptions 3a' and 3b.

Root Cause Analysis Report Data Analysis

• 12

DATA COLLECTION Data collection began with the development of a listing of sources with the likely potential of identifying flawed detection, signal processing, output, and alarms. This was supplemented with likely sources of information that might reveal environmental influence or values that may have impacted the design or operation of the feedback process.

Once potential sources were identified, selection criteria for choosing documents and the associated attributes to review for were developed. Team members were assigned to document groupings, to review and capture pertinent data from the documents. This included development of a brief statement to explain the relationship between the data and potential impact on the Feedback Model, if any. This included a subjective assessment on whether the data was likely to have had a direct or indirect impact on the reactor head degradation event, and which element of the Feedback Model was affected. This data was tabulated in the CR 02-2578 Integrated Database (an Excel spreadsheet) to facilitate electronic binning and sorting by codes and dates.

Data was then reviewed by the Team collectively to peer check logic and coding. The data was then transferred to a working timeline consistent with the timeline developed by the NQA Team.

The Team reviewed the following data in order to gain insight as to the efficacy of the QA oversight functions. The selection criteria for documents to be reviewed and the review criteria are contained in the Methodologies Employed section of this report. The CR 02-2578 Integrated Database contains a tabulation of documents reviewed and other pertinent information developed by the Team from the following data set.

"* Audit Reports: Selected Audit Reports for the period 1/1990 through 2/2002.

"* Surveillance Reports: Selected Surveillance Reports for the period 1/1990 through 2/2002.

"* QA Summary Reports (These reports changed titles a number of times over the period of review) for the period 1993 through 1996, 1998 through 2001. No reports could be located for 1997.

"* ISEG Reports and miscellaneous ISEG correspondence for the period 1990 through 2000.

"* NRC Inspection Reports for the period 1/1990 through 5/2002.

"* Miscellaneous Documents - these documents were developed from sources while performing primary reviews, or from interviews conducted with QA and Line Management Personnel

" Interviews of QA Personnel and a sampling of other Station Personnel were conducted using the questions contained in Attachments 8 and 10. Attachments 9 and 11 provide a list of persons interviewed, and the CR 02-2578 Integrated Database contains a summarization of the interview results.

FUNCTIONAL TOOL ANALYSIS CHARTS Due to the unique nature of this investigation, e.g., trying to determine why something did not happen, the Team chose to define two distinct types of "Events." The first being the most significant to this Root Cause Analysis is described as a "Missed Opportunity Event," and the second is an "Event" derived by a slight modification to a TapRoot Event.

Root Cause Analysis Report Data Analysis

• 13

Missed Opportunity Event (MOE): circumstances related to oversight, which existed and had the opportunity to preclude or mitigate the reactor vessel head corrosion, but were missed. For this to be true, there needs to have been a reasonable opportunity for oversight functions to have precluded the failure of a particular barrier. Oversight organizations did have or reasonably should have had the information available to them. There needs to have been an oversight breakdown; either an oversight-type barrier was there and failed, or should have been there and wasn't.

The selected MOEs represent windows of time, during which, in the subjective opinion of the team, sufficient information was available and could or should have been recognized as indication of the impending or ongoing degradation of the RPV head. The time periods typically are tied to plant outages when significant additional information usually became available. As the MOEs move forward in time and the available information changes, the causes associated with the missed opportunity may also change. In order for the oversight function to have had a positive effect on detection of the head condition very early, different programs/processes/values would have had to been functioning effectively than at a later point in time when more definitive indications of the problems existed. The time periods chosen by the Team as MOEs are:

• The time period from the late 1980's up to the start of IORFO

• The start of IORFO through the end of 1 RFO

• The end of I 1RFO through the end of the Mid-Cycle Outage of 1999

• The end of Mid-Cycle Outage of 1999 through the end of 12RFO

• The time period after 12RFO Event: an action or activity in the sequence of events (SOE) leading to a Missed Opportunity Event or an Incident.

The investigation considered many facts, covering a significant time frame (16 years, 1986 2002). In order to better assess the various influences which were dominant during the various periods, the team adapted Event & Causal Factor (E&CF) charting to allow closer assessment of the issues. To do this, the term "Functional Tool Analysis" was coined by the team. It represents a variation on traditional E&CF charting. The Team identified six major programmatic tools used by line management and/or the oversight organization that could and should have been effectively used to lead to an earlier detection of the degrading condition on the RPV head. Each of the following functional tools was represented graphically by a horizontal slice for each MOE on the Functional Tool Analysis chart:

" Corrective Action Program (2AP) -- the procedures and processes used to identify, document, evaluate, and correct problems. While this includes procedures and computer software tools, the primary focdIs is its irmiplementation for resolution of problems.

"* Root Cause Analysis (RCA) - the procedures and processes used to perform root cause analysis, and their resultant products. Although this is a sub-set of CAP, the importance of RCA to the effectiveness of CAP caused the Team to evaluate it separately, as well as within the CAP itself.

"* Operating Experience (OE) - the collective process of obtaining, evaluating, integrating and acting on experience from the rest of the industry, as well as D-B experience, to prevent conditions adverse to safe operation.

Root Cause Analysis Report Data Analysis

• 14

"* Trend / Analysis (T/A) - the process of collecting, summarizing, analyzing various data hardware) that require to detect adverse trends in station performance (both process and further investigation and/or action to mitigate.

"* Audits / Surveillance / and Evaluations - the traditional tools used by QA Auditors or ISEG Engineers to evaluate processes or conditions.

"* Culture / Values (CJV) - the collective philosophy as demonstrated by actions of an and affects individual or entity. It represents the approach to dealing with issues and individually, when decision-making and actions of the organization, both collectively establishment of faced with certain conditions or situations. This especially applies to the priorities when faced with competing objectives.

Each attachment Attachments 3 through 7 are the completed Functional Tool Analysis charts.

and attached detail sheets covers one of the five time periods and includes a cover/summary sheet for each of the six Functional Tool Analyses.

into the individual Event and condition data collected from the data review was organized analysis a set of concluding functional tool analysis for each MOE. For each functional tool in the upper right hand "WHY" statements was developed and are represented by a black triangle were used to develop causal comer of the appropriate box on the chart. These "why" statements Near the end of the factors in a manner similar to traditional Event & Causal Factor charting.

factors identified through investigation, the team developed a standard E&CF chart using the key analysis.

DATA REVIEW of information. Much of the information was Note: during this review the team was exposed to significant amounts During the period of this investigation, another Root intertwined between the line organization and the oversight function.

contributors to the degraded RPV head. The two teams shared significant Cause Team was investigating management On occasion, this team developed observations and suggestions on corrective actions related to amounts of information.

occurred, such observations/suggestions were provided to the other improving effectiveness of line activities. When this related to methods that could improve the line's ability to manage corrective actions.

team. Several recommendations over 400 individual The accumulated data in the CR 02-2578 Integrated Database represents Functional Tool Analyses facts and relevant observations in support of this evaluation. The period and area of the discussed above were used to categorize each item into the associated time represent thirty assessment. The five time periods with six functional tools in each period of causal separate analyses. Each analysis is in effect its own E&CF chart with identification is to ascertain the factors for that functional tool in that time period. The intent of each analysis etc.) to the contribution of each given functional tool (e.g., CAP, Culture/Values, Trend/Analysis, related to corrosion failure of the oversight organization to identify and effect resolution of issues of the RPV head in the given time period.

Tool Analysis, As an example in referring to Attachment 3, under the Culture/Values Functional safety it can be seen that the team concluded that QA's failure to be the station nuclear not detected prior conscience is considered a causal factor in why the CRDM nozzle leakage was to IORFO.

each data In order to construct the Functional Tool Analysis charts, it was necessary to review Tool(s).

item in the CR 02-2578 Integrated Database and assign it to the appropriate Functional u*ata Ana.-lysis 4 Root Cause Analysis Report Data Analysis

• lb

Each of the data items in a given Functional Tool was then evaluated and a "mini" E&CF chart prepared. Examples are shown on the detail sheets of Attachments 3 - 7. A conclusion regarding the causal factor(s) associated with each of the six Functional Tools in a given time period was then summarized in the large box to the right of each Functional Tool Analysis Detail sheet and carried forward to the summary page for the time period. The 27 "why" or causal factor boxes on the time period summary sheets are the causal factors carried forward into the causal factor analysis below.

Each time period was evaluated using the information from the Functional Tool Analysis sheets.

A conclusion was drawn for each Functional Tool for each time period relative to Assumptions 3.a and 3.b discussed earlier. The intent of the individual conclusions was to answer the questions:

For Assumption 3a: "Was there sufficient information and opportunity available in this area during this time period for'the oversight organization to have influenced the line organization in a way that would have allowed them to identify the head degradation."

For Assumption 3b: "Was there sufficient information and opportunity available in this area during this time period for the oversight organization to have directly identified the head degradation." I An overall conclusion for the time period was also drawn and documented in the large box in the center of the Summary sheet.

Through the data review and analysis described above, a number of causal factor themes began to emerge. As the analysis progressed from time period to time period, similar causes were seen and in some cases became more pronounced. New causal factors were also identified during later time periods when new or additional information on the head degradation became available.

When the analysis of each of the five time periods was complete, an overall review of the results was performed, and the 27 initial causal factors were consolidated into the eight formal causal factors described herein. Through the data review and analysis, a number of key events and issues became apparent as significant contributors to the formal causal factors. Central to these key events and issues was the poor implementation of the corrective action program over the complete time period addressed by this analysis.

IOCFR50 Appendix B, Criterion XVI Specifically requires that the cause of significant conditions adverse to quality be identified and corrective action taken to preclude repetition.

Two of the aspects which influence the success of an organization in meeting this requirement are the corrective action program which is in place, and the actions of those who use the program. Although the D-B program Underwent substantial change during the era of concern (1986 - 2002), the team found no data 'that indicated that the program itself contributed to lack of success by oversight.

Rather, the actions of the station team did not accomplish the desired result. Opportunities to identify and prevent significant RPV head degradation were present for both the line organization and the oversight function. The team concluded that achieving a clear understanding of issues, making effective decisions regarding fixing abnormal conditions, effectively using industry operating experience (including D-B's own experience), and effectively analyzing station trends could have identified the issue early enough to have prevented substantial head degradation.

[SUPPORTIVE OF CAUSAL FACTORS: 1, 2, AND 7]

The failure of the oversight function to effectively challenge station theories regarding the causes and effects of boron on the RPV head (including other indications in containment) contributed Root Cause Analysis Report Data Analysis

• 16

heavily to oversight's inability to cause effective resolution of issues. The inability to successfully interject was caused primarily by a culture in the oversight function that mirrored that of the station. Degradation in station standards occurred in both the line organization and the oversight organization. Use of compliance based audits during most of this period resulted in an emphasis on meeting administrative requirements, rather than ensuring station conditions were fixed. As more and more data became available, especially during refueling outages, the opportunities increased to draw the conclusion that a significant issue was not being resolved.

Although, the effectiveness of the oversight function in identifying issues began to increase in 2000, several factors worked to render this improvement of modest value. Responsiveness of the line organization was less than adequate, the success of the oversight organization in escalating issues was less than adequate, and the oversight organization missed opportunities to clearly identify the issue. [SUPPORTIVEOFCAUSALFACTORS: I, 2, 3, and 5]

Several key opportunities existed for oversight to have used information in the corrective action program to address weaknesses in station performance as follow:

(D Operating Experience: A 1987 D-B Independent Safety Engineering (ISE) memo (ISE 87-10049, dated 5-28-87)) indicates that a major effort was made to become knowledgeable of both the Turkey Point-4 and the ANO-1 boric acid corrosion events in that era. The ISE memo documented that high temperature boric acid corrosion can occur up to 600'F, and that boron carries over in steam. Although the station was aware of numerous additional industry letters and NRC communications that occurred over the years, the temperature environment information was lost.

It appears that much of the basis for decisions made in the 1990s was a belief that boric acid corrosion could not occur at D-B operating temperatures (TAVE - 580'F). The opportunity was present for the oversight function to resurrect this information and challenge the station.

[SUPPORTIVE OF CAUSAL FACTOR: 7]

PCAQR 96-0551 addressed issues of boric acid on the RPV head identified in IORFO. The investigation phase of this PCAQR was not completed until after I IRFO, more than 21/22 years after it was initiated. During that time, two root cause reports were drafted but not completed and the PCAQR was eventually downgraded and closed without addressing the issue. The second of the root cause reports was prepared by ISEG indicating that they were involved in the resolution of the PCAQR. PCAQR 98-0767 documenting boric acid found on the RPV head during 1 IRFO was closed to PCAQR 96-0551, negating an additional opportunity to address the issue. [SUPPORTIVE OF CAUSAL FACTOR: I1

() RC-2: RC-2 is a pressurizer spray valve that experienced boric acid corrosion ini 998 (see CR 98-020). This event provided several opportunities for the oversight function; two examples:

"Trackingand trending: at the very time the root cause evaluation of CR 98-020 was closed (3-30-99), the station was five months into heavy cleaning of the containment air coolers. This was an opportunity for the oversight group to press the issue to get a clear understanding of what was going on in containment. Although the phenomena was complicated strongly by cultural beliefs, a high value on problem resolution at this point would likely have led to the proper understanding of the phenomena. [SUPPORTIVE OF CAUSAL FACTOR: 41

" Corrective actions: Numerous corrective actions were identified as a result of this event; two were assigned to the oversight function. The event itself provided an opportunity for oversight to key in on boron and ensure that corrective actions were effectively implemented, and that thorough extent of condition reviews were completed. Team analysis of the actions resulting from this situation, however, concluded that the corrective actions were largely Root Cause Analysis Report Data Analysis 9 17

ineffective. Even the actions assigned to oversight lost significance within 3 years.

Interviews with current oversight personnel showed that although many had heard of the event, few behave differently today as a result of it. The opportunity to challenge the effectiveness of the corrective actions, and particularly the formal effectiveness review completed by Quality Programs/Quality Improvement Unit (Memo dated 10-27-00), were missed.

[SUPPORTIVE OF CAUSAL FACTOR: 1]

T CR 98-1904 was a station wide assessment of 10 events that occurred in 1998 and indicated an overall decline in station performance. The results of the assessment should have made substantial improvement in key processes and activities such that the head degradation would have been identified sooner, however, it appears that despite the corrective actions from this CR, little overall improvement was made. [SUPPORTIVEOF CAUSALFACTOR: 1]

( CR 99-1300 addressed containment radiation monitor clogging issues. Over the course of addressing this issue, the station took a number of actions to address the symptoms of the issue but did not aggressively pursue the cause. Actions included installing temporary HEPA filter units in containment and modifications to the radiation monitor system. Actions on this issue combined with addressing the more frequent cleaning of the containment air coolers reflect an approach that addressed symptoms rather than the causes. [SUPPORTIVE OF CAUSAL FACTOR: I]

( Refueling outage audit, 2000 (AR-00-OUTAG-01, dated 6-7-00): 12RFO came only four months after key training initiatives related to the RC-2 event had been completed. Part of the scope of this audit was evaluation of the Boric Acid Corrosion Control Program and PCAQ 00-0782. (mhis PCAQ documented red/brown boric acid found on the RPV head at the start of 12RFO.) Had the training been fully effective, the audit team would likely have taken the initiative to make sure that the RPV head was fully cleaned prior to startup. InstEad, the audit team failed to verify claims that the head was clean, and provided positive words, in the report regarding the persistence shown by engineering in getting the head clean. Evidence shows that the head was not fully plean, and that some station personnel were aware of that. [SUPPORTIVE OF CAUSAL FACTOR: 61 As a footnote regarding the 12RFO audit, the investigation team concluded that the audit team was insufficiently experienced in the Engineering functional area. Of the three personnel involved, the supervisor was conducting his first audit, one auditor was a new hire (started employment during 12RFO), and the other auditor was known to be a weak performer. The assignment of such an inexperienced team to review a condition of such significance (BACC program), in hindsight, was less than adequate. However, the team concluded that this was not a cause, in that some management personnel at the-station knew of the conditions (i.e., had been to the training and seen the pictures). Given that the oversight function had been' unsuccessful in improving performance over the previous years, it was the judgment of the team that a clear identification of the problem at this time in this audit would not have made a difference. Had the oversight function recognized the signs present at the station prior to this outage, they likely would have approached the oversight function differently (i.e., in a more aggressive manner),

resulting in a different result from the audit.

() CAP audit, 2000 (AR-00-CORAC-01, dated 6-12-00): during this period, the corrective action program and the oversight function fell under the responsibility of one individual (a director).

When the audit team assessing the corrective action program in 2000 concluded that the program had serious weaknesses in ýonstruction and implementation, the director caused the message to be softened. He also cotivinced the Senior Management Team that the audit results exaggerated the weaknesses in the CAP. [SUPPORTIVE OFCAUSAL FACTOR: 51 Root Cause Analysis Report Data Analysis e l U

The team identified instances where both oversight personnel and management personnel chose not to enter containment during refueling outages based on ALARA considerations. The importance of personal observation as a key element of enforcing standards was not balanced against dose considerations. [SUPPORTIVE OF CAUSAL FACTOR: 8]

The team also considered the oversight methods employed to ensure that the FENOC oversight function itself is effective. As discussed below, there was a weakness at D-B in this area during this period. The checks in place on the oversight functions are external activities such as biannual audits required by external groups (D-B typically used Joint Utility Management Audits), review of station activities by the Company Nuclear Review Board (CNRB), and the Nuclear Regulatory Commission (NRC). [Note that INPO does not currently include the oversight function as part of its scope.] The message from this collective group was inconsistent. NRC and JUMA typically provided a positive message, particularly as it related to the effectiveness of the CAP; CNRB was sometimes more critical. Nonetheless, the effectiveness of this arrangement in ensuring the oversight function was effectively implemented was itself ineffective; it failed to cause the necessary changes at D-B to ensure that the D-B oversight function was executed effectively.

[SUPPORTIVE OF CAUSAL FACTORS: 5 AND 6]

An additional consideration that was identified during the investigation was the level of PWR and more specifically B&W experience within the station senior management. The relationship to this investigation is the level to which the oversight organization alters its oversight activity when the level of experience changes. There were key times during the events of this investigation when three of the five site senior managers had little or no PWR experience and a fourth had no B&W experience. During these times there is an increased risk that issues specific to a NSSS vendor may not be completely understood or appreciated. [SUPPORTIVE OF CAUSAL FACTOR 1]

An important activity for the oversight function is the ability to recognize adverse trends in station performance when the line organization does not (i.e., connect the dots). Once an adverse trend is identified, it is equally important to cause the line to recognize and act on the trend. An overview of the events between 1ORFO and 13RFO indicates that there were sufficient "dots" to have recognized the head degradation much sooner than occurred. Examples include: PCAQR 96-0551, containment air cooler cleaning frequency, containment radiation monitor filter clogging, CR 00-0782, lessons learned from the RC-2 event, the health of the CAP, NRC Generic Letter 97-01, NRC Information Notice 2001-05, and NRC Bulletin 2001-01. The oversight role relative to trending/analysis should include both oversight of the station's trending/analysis program and performance of independent trending/analysis. It is recognized that oversight does not have the resources to perform the same level of trending/analysis as the station and should therefore perform oversight of that process. However, oversight should also perform independent high level trending/analysis of key issues as a check of the station's trending/analysis. The trending/analysis performed by oversight should be based on a healthy skepticism of station activities, it should be based on oversight's role as the nuclear safety conscience of the organization, and should be structured to identify station issues that need to be addressed at the station/department management level. [SUPPORTIVE OF CAUSAL FACTOR 41 FACT LIST The CR 02-2578 Integrated Database provides a compilation of the facts selected by the team for analysis in this investigation. Also, refer to the Five Non-Conformances Integrated Timeline and related information prepared by the NQA Team.

Root Cause Analysis Report Data Analysis

• 19

CAUSAL FACTORS Collectively over the five time periods involved in the analysis, a total of 27 initial causal factors (ICF) were identified. Many of them apply to several periods or are reshaped over time; therefore, the initial 27 were combined as appropriate resulting in a total of eight formal causal factors (FCF). These eight causal factors are:

1. Low standards existed within the okersight function related to management of Corrective Actions. Examples: acceptance of ,weak cause evaluations by the oversight function; failure of line management to address CAP weaknesses identified by oversight; lack of clear expectations for classification of PCAQRs that address equipment issues on critical safety equipment; ineffective use of CAP data by line and oversight. [This theme developed in CAP/Root Cause Analysis: ICFs 1.1, 2.1, 2.2, 3.2, 4.1,412, 5.2] [FCF I contributed to Cause 1]

2. Oversight standards were not set apart from those of the station. As the station increased its tolerance for substandard equipment performance, the oversight function increasingly missed opportunities to influence positive changes in conditions detrimental to safety. Included in this cause is the failure of oversight to recognize the risk of management change and the need to adjust oversight or the environment surrounding problem identification and resolution.

The lack of organizational independence between the CAP and the Audit / Evaluation functions also contributed to this cause. [This theme developed over Culture / Values: ICFs 1.6, 2.6, 3.6,4.6, 5.6] [FCF 2 contributed to Causes I and 4]

3. The oversight function was unable to effect change. Examples: Oversight was not sufficiently critical to resolve underlying problems, including the CAP; Oversight failed to escalate issues sufficiently. [This theme developed in CAP: ICFs 3.1,4.1, 5.1) [FCF 3 contributed to Cause I]

4. Oversight did not require that a consistent and effective trending and analysis program be defined and documented in terms of content, degree of analysis, format, responsibility, etc.

Guidance was not provided on the expected use of information. Clear ownership of trending and analysis was not evident from station behaviors. [This theme developed over Trending / Analysis:

ICFs 2.4, 3.4, 4.4, 5.4] [FCF 4 contributed to Cause I]

5. The oversight process / audit program was not structured for intrusive and aggressive assessment of technical issues. Messages from external oversight were mixed. [This theme developed over Audits/Surveillances/Evaluations: ICFs 2.5, 3.5] [FCF 5 led to Cause 3]

6. The audit program was restructured in terms of organization, however, the expectations of oversight were still not sufficiently defined and/or communicated. The auditor did not raise a concern when reviewing CR 00-0782, despite the training from RC-2. [This theme developed over Audits/Surveillances/Evaluations: ICFs 4.5, 5.5] [FCF 6 led to Cause 2]

7. Expectations for Operating Experience Assessment Program (OEAP) in terms of oversight requirement and responsibilities w~re not being met. Oversight was not sufficiently critical of the OE program. [This theme expressed in OE Reviews: ICFs 1.3, 2.3, 3.3, 4.3, 5.3] [FCF 7 contributed to Cause I]

8. ALARA concerns created a reluctance to observe or verify certain field activities; this reluctance affected both oversight and management personnel. [This theme developed in ICF 4.6 and 5.6] (FCF 8 led to Cause 1]

Root Cause Analysis Report Data Analysis

• 20

CONCLUSION The team concluded that the Oversight function did miss opportunities to cause earlier identification and mitigation of the RPV head degradation. The analysis further indicates that standards within the oversight function were insufficiently differentiated from the standards of the station. The root cause was determined to be that D-B's nuclear safety values, behaviors and expectations were such that oversight was not set apart, in terms of expectations and performance standards, from the balance of the station. This affected the ability of the oversight organizations to identify problems and effect needed positive change in station operations. This was particularly applicable to the implementation of the Corrective Action Program and resulted in the station tolerating conditions that were detrimental to safety (boric acid on RPV head and other RCS components, containment air cooler degradation, containment radiation monitor fouling, unexplained RCS leakage) for long periods of time. In evaluating the results of the analysis against the Feedback Model, the team concluded that the predominant problem was that the calibration of the feedback model had drifted to an out-of-tolerance condition. This means that the model was adversely influenced by slow changes in culture within both the oversight function and the line organization. For oversight, these changes included lowering of expectations, reduction in nuclear safety values, and acceptance of lack of response to output.

The lack of a healthy escalation process was also noted as contributory, but closely related to inadequate expectations / values. The inadequate expectations / nuclear safety values also affected the Detection function.

The expectations for key programs such as Corrective Action Program, Operating Experience, and Trending/Analysis were not adequately defined in terms of assignment of responsibilities, results expected from the programs, and the Oversight functions' role relative to these programs.

Lack of independence betweep the Corrective Action Program and the Audit/Evaluation function may have impeded needed management escalation of deficiencies in the CAP. Facts. supporting this conclusion include the failure to identify some issues, the failure to correctly apply OE with regards to understanding and evaluating identified issues, the failure to identify relevant causes, the failure to correct issues in a timely manner, the failure to prevent recurrence of the same issues, the inappropriate timeliness of corrective actions, and the failure to properly evaluate the effectiveness of corrective actions. Since these weaknesses occurred across a substantial period of time, through evolving corrective action programs, it is further concluded that the weaknesses were not related to a specific corrective action program but rather to the implementation of the corrective action program (both by oversight and the line organization).

For a period of time, the management of the audit/evaluation process was not independent from the management of the corrective action process. This lack of independence allowed the oversight director to soften the thrust of an audit critical of the corrective action process during 2000.

Oversight did not establish an effective method for assessing the oversight function. The process for providing oversight of the oversight function was less than adequate, feedback provided was mixed, and corrective actions were sometimes ineffective. Over a period of years, assessments from JUMA, CNRB, and NRC were ineffective (did not identify existing issues) and sometimes inconsistent. Follow-up response to identified weaknesses was sometimes less than adequate.

The training for the RC-2 event was ineffective. It failed to improve the ability of both the oversight and line organizations to recognize corrosive conditions and their significance. This Root Cause Analysis Report Data Analysis

• 21

contributed to the failure of the auditing team to raise a concern when auditing the Boric Acid Corrosion Control Program during 12RFO.

Although recent corporate and organizational changes have corrected some of the conditions fundamental to the cause, additional preventive and remedial actions are required. The most significant of these is to take aggressive action to ensure that the nuclear safety standards held by the Oversight organization are appropriately upgraded and held separate and apart from the station to prevent collective or common, mode degradation.

Anlyss ReortDataAnaysis*

RootCaue 2 Root Cause Analysis Report Data Analysis

• 22

EXPERIENCE REVIEW Industry Operating Experience (OE) was reviewed in an effort to determine if there have been previous instances documented where the oversight organization was not effective in identifying and achieving resolution of significant plant issues. Three groupings of OE documents reviewed were (1) D-B CATS, (2) D-B Condition Reports, and (3) the INPO Website. The D-B and Nuclear Industry searches identified related issues. The results of the review were used to determine if there is a generic, broader scope issue that needs to be addressed and to evaluate the corrective actions taken in response to past occurrences to assure any new corrective actions will be effective.

DAViS-BESSE CATS A keyword search on "QA" was performed in Corrective Action Tracking System (CATS) resulting in 141 hits. The subject descriptions were reviewed and hard copies obtained for those that appeared to be related to the issue of oversight's failure to identify and effect resolution of significant plant issues. Most of the CATS items were QA's identification of issues in an audit, surveillance, or self-assessment, and CATS was tracking it to completion. Some other issues were administrative in nature and included documents that were missing signatures, timeliness, and procedure interpretations. Three items were selected for additional review: PCAQR 97-1019 that involved inconsistencies in implementation of QA procedures; CR 99-1889 that addressed INPO's questions regarding the adequacy of a QA surveillance; and CR 00-3108 regarding a reevaluation of a QA Audit closeout review on periodic housekeeping inspections not being performed.

Of these three, only CR 99-1889 is considered relevant to the issues of thig investigation. It "questions the adequacy of a QA surveillance completed in May 1999 that evaluated the site's effectiveness in resolving INPO/WANO issues from 1996 and 1998." The CR was categorized as "Important" and investigation identified several management factors that contributed to the inadequate surveillance. Corrective Actions To Prevent Recurrence (CATPRs) included a clearer definition of the surveillance scope, a critique before issuance, a lower threshold for using Condition Reports (CRs), including team members with expertise in INPO assessment techniques, ensuring that there is adequate time given for the surveillance, and using direct communication techniques. These corrective actions likely had some effect on the oversight organization as evidenced by more critical audits beginning in 2000 (e.g., AR-00-CORAC-01);

however, other factors identified by this investigation continued to hinder the overall effectiveness of oversight. The issues identified in CR 99-1889 are supportive of the causal factor I in this investigation. Associated Recommended Corrective Action 1 amplifies on and expands the management actions taken by CR 99-1889 in a manner that will effectively address the identified issues. No additional action is required as a result of this experience review information.

DAVIS-BESSE CREST The Condition Report Evaluation and Status Tracking (CREST) system at D-B was implemented in December of 2000. A keyword search was performed in CREST using "QA in the "Title" field. The following five CRs were selected for further review based on a subjective review of the title of the CR.

Root Cause Analysis Report Experience Review e 23

"* CR 01-0426, SA 2000-0126 1999 JUMA QA Audit CA

"* CR 01-2226, SA 314NQAP-2001 Results

"* CR 02-0485, NQA SA 2001-0120 Areas for Improvement

"* CR 02-1136, Lack of timely review of CR Corrective Actions by QA

"* CR 02-2694, NRC indicating that the QA Boric Acid Inspection Plan was not timely Review of these five CRs did not identify any additional considerations that need to be addressed by this investigation.

D-B Condition Reports were also searched using keywords: "NQA ineffective," "QA ineffective," and "QA failure to identify." One additional CR, 01-0427 regarding "Required audit element not performed" was identified for further review. CR 01-0427 was written as a result of self-assessment 2000-0126 and identified that previous corrective actions to address audit scheduling have not been fully effective. The specific issue addressed by the CR was that equipment qualification had not been audited as required since 1994. An extent of condition was performed to evaluate whether other required audits had been missed. The extent of condition review identified nine programmatic elements "which do not appear to have been assessed at an appropriate level or frequency commensurate with their importance to overall program effectiveness." One of these elements was oversight of the Operating Experience Program, which was also identified as weak by this investigation. The extent of condition evaluation indicates recognition by the oversight organization that they should have been more active in the oversight of OE; however, it was not completed until May 2001, and therefore had little effect on the opportunity to affect the RPV head event. It does represent an additional missed opportunity by oversight to have recognized weakness in the OE program sooner when it may have had an impact. The current FENOC continuous assessment process for performing audits and assessments inoludes comprehensive Master Assessment Plans (MAPs) to control the scope of activities assessed. Assessment of the OE program is included in current MAPs and is also the subject of Recommended Corrective Action 14. Therefore, no additional action is necessary based on this information from the experience review.

NUCLEAR INDUSTRY The final piece of Operating Experience reviewed with regards to the issue of the "QA inability to identify and prevent" was the INPO Website. A search of the "Just-in-Time" operating experience for "QA" indicated 85 hits but nothing significant with regards to this .issue.

Additional searches of the INPO Operating Experience Web Page were performed using keywords: "QA failure," "QA breakdown," "oversight breakdown," "Corrective Action breakdown," "OE Program breakdown," "insufficient QA oversight," and "insufficient quality oversight." No hits were obtained on these keyword searches. On "management expectations" there were 195 hits, on "management expectations and implementation" there were 65 hits, and on "insufficient oversight" there were 51 hits.

The most significant INPO Operating Experience with regards to the issue of the "QA inability to identify and prevent" was found while reviewing the hits on "insufficient oversight" on the INPO "Prevents Events" categories. Under "Quality Programs" a D-B head comparable, although far less significant event, was noted in SEN 163 (a recurring event concerning a high-pressure injection line leak). A cause category I'or Managerial Methods included insufficient use of operating experience. This again provides indication that earlier recognition of a weak OE Root Cause Analysis Report Experience Review

• 24

program could have helped the line and oversight organizations improve that program. As discussed elsewhere in this investigation, a stronger more effective OE program would have contributed to identifying the head degradation sooner. This aspect of the issue has been identified by the investigation, therefore no additional action is required from this experience review information.

CONCLUSIONS In summary, Operating Experience (OE) was reviewed to ascertain if the oversight's inability to identify weaknesses in key line functions and effect positive change had been previously identified. The following examples show where there were opportunities for QA to identify and improve their own processes:

" SEN 163 (4/22/97) --- This SEN provided indication of the importance of an effective OE program. At the time of the SEN and with the information it contained, it is unlikely that a clear connection would be made to the ongoing head degradation.

" CR 99-1889 (12/2/99) --- QA self-identified weakness in their surveillance process and took action to address some management issues. Corrective actions likely had some effect; however, other issues continued to hinder oversight effectiveness.

" CR 01-0427 (2/14/01) --- QA recognized weakness in their audit scheduling process and that nine programmatic elements had not been assessed to an appropriate level. One of these nine was the OE program. Corrective actions at the time of this CR were too late to have a significant effect on detecting the RPV head degradation. Had this been identified sooner, it may have had some effect.

The results of the experience review, while identifying some issues that could have been used in the past to identify and implement earlier corrective actions, determined that there is not a generic, broader scope issue that needs to be addressed. The corrective actions taken in response to past occurrences are complemented and built upon by the corrective actions in this investigation to assure that they will be effective.

Experience Review

• 25 Root Root Cause Report Analysis Report Cause Analysis Experience Review 9 25

ROOT CAUSE DETERMINATION The following are the root cause and contributing causes addressing the issue presented in the Problem Statement.

ROOT CAUSE I. Since the mid-1990s, D-B's nuclear safety values, behaviors and expectations were such that oversight was not set apart, in terms* of expectations and performance standards, from the balance of the station. This affected the ability of the oversight organizations to identify problems and effect needed positive change in station operations. Examples:

"* Oversight was not set apart in terms of performance standards. This affected their ability to influence station behaviors that tolerated conditions that may have been detrimental to safety.

"* Oversight did not influence the station to manage Corrective Actions so that problems were effectively identified and fixed so as to prevent recurrence.

"* The oversight program was not structured for aggressive and intrusive oversight of emergent technical issues; it was insufficiently challenging of the line organization.

"* Oversight behaviors reflected that ALARA was of higher concern than oversight of field activities.

"* Oversight did not evaluate the risk involved with changes (including management changes) and the potential need to adjust oversight or the environment surrounding problem identification

"* Use of both internal and external operating experience by oversight was not sufficient to result in effective application of this information to identify and prevent problems.

Oversight did not effectively oversee the station use of OE nor did oversight adequately incorporate OE into oversight activities.

"* The Trending and Analysis program failed to provide consistency regarding the content, degree of analysis, format, responsibility, expected use of information, and documentation. Failure (of the line organization) to perform effective trending / analysis inhibited the oversight function from using trend/analysis information to prompt the line organization to identify the deficient condition of the RPV head, and from identifying the condition themselves.

CREST Cause Code: H04 Management expectations were not well defined or understood.

TapRoot Basic Cause Category: Standards, Policies, and Admin Controls (SPAC) confusing or incomplete. The SPAC governing the oversight functions were inadeq uate to enable oversight to effectively accomplish its objectives.

CONTRIBUTING CAUSES

2. The training for the RC-2 event was ineffective. It failed to improve the ability of both the oversight and line organizations to recognize corrosive conditions and their significance.

This contributed to the failure of the auditing team to raise a concern when auditing the Boric Acid Corrosion Control Program duiring 12RFO.

adequate to CREST Cause Code; Q02 Corrective action for previously identified problem or previous event cause was not prevent recurrence. I TapRoot Basic Cause Category: Training - IUnderstanding needs improvement.

Root Cause Analysis Report Root C;ause Determination 9 26

3. Oversight did not establish an effective method for assessing the oversight function. The process for providing oversight of the oversight function was less than adequate, feedback provided was mixed, and corrective actions were sometimes ineffective. Over a period of years, assessments from JUMA, CNRB, and NRC were ineffective (did not identify existing issues) and sometimes inconsistent. Follow-up response to identified weaknesses was less than adequate.

CREST Cause Code: H04 Management monitoring of activities did not identify problems.

TapRoot Basic Cause Category: Analysis and Evaluation lacked depth.

4. For a period of time, the management of the audit/evaluation process was not independent from the management of the corrective action process. This lack of independence allowed the oversight director to soften the thrust of an audit critical of the corrective action process during 2000.

CREST Cause Code: H02 Supervisory Methods.

TapRoot Basic Cause Category: Analysis and Evaluation not independent.

Root Cause Determination

• 27 Analysis Report Root Cause Analysis Report Root Cause Determination

• 27

EXTENT OF CONDITION Although the specific driver of this investigation was boric acid corrosion of the RPV head, the investigation itself was on the efficacy of the D-B oversight functions as they related to that condition. The investigation examined the causes for missed opportunities for oversight functions to prevent or mitigate degraded equipment or components. As part of its Building Block plans, FENOC has established program, system, and functional area (organizational) reviews to determine the extent of condition. Specifically:

"* The System Health Assurance Plan provides for reviews of systems.

"* The Program Compliance Plan pro tides for reviews of programs.

"* The Management and Human Performance Excellence Plan provides for review of functions and organizations.

These reviews will be sufficient to determine the extent of condition of problems in systems, programs, and organizations and, by implication, any significant adverse conditions missed by oversight.

From a FENOC perspective, the extent of condition might extend to the oversight functions at both Beaver Valley and Perry. Therefore, an assessment is recommended (see Corrective Action

17) to determine whether similar problems exist at those plants. The assessment should focus on the evaluating whether those plants hak'e problems similar to the root causes identified in this report and the Root Cause Analysis Report for the failure to identify the significant degradation of the RPV head.

Finally, FENOC is performing an assessment of the adequacy of the CNRB, which will be sufficient to determine whether CNRB has been similarly affected.

Extent of Condition

• 28 Root Root Cause Cause Analysis Report Analysis Report Extent of Condition e 28

RECOMMENDED CORRECTIVE ACTIONS Note: Considering that this CR documents a situation where a function was unsuccessful in identifying or effecting resolution of a significant issue, any corrective actions will be geared to successful identification or resolution of issues within some time period. The Team felt that a significant issue (such as corrosion of the RPV head) should be identified and resolved as early as reasonably possible. In this particular case, the Team felt that identification and resolution during time Period 2 (extending from start of IORFO to end of I IRFO) should be expected. The Recommended Corrective Actions, then, are geared toward ensuring that, in the future, the oversight function will successfully identify and effect resolution of significant issues based on the type and nature of information which was available during Period 2.

Note: Since the oversight function at D-B is integrated into a single department that provides oversight for the other FENOC stations, it is recommended that the corrective actions for oversight be incorporated across all three stations.

Note: The FENOC Change Management Policy should be considered in implementation Preventive and Remedial Actions CAUSE I Since the mid-1990s, D-B's nuclear safety values, behaviors and expectations were such that oversight was not set apart, in terms of expectations and performance standards, from the balance of the station. This affected the ability of the oversight organizations to identify problems and effect needed positive change in station operations.

ASSOCIATED CORRECTIVE ACTIONS:

I. Preventive Action (PR). Complete Create a new position of Vice President, OPID, reporting outside the station organization.

Staff this position with an individual who has demonstrated high standards for safety and rigor, and is capable of providing leadership to enhance the values, behaviors, and expectations of OPID.

2. Preventive Action (PR). Owner = OPID / LWPearce Establish, document, communicate and hold OPID personnel accountable to expectations that support the ability of OPID to detect adverse conditions, process information, and escalate issues in a manner that ensures they are resolved in a timely fashion. The elements of this action include:

"* benchmarking of the industry for corresponding expectations

"* revision of the OPID Business Plan and appropriate implementing documents to address these expectations

"* face-to-face meetings on expectations by the OPID management team with OPID personnel using lessons learned from the D-B RPV head degradation event, and this root cause analysis to demonstrate the values by management

"* continued frequent reinforcement of the expectations by OPID management.

The goal of this corrective action is to elevate the ability of OPID to cause positive change to occur at FENOC stations when warranted. The expected changes include:

Root Cause Analysis Report Recommended Corrective Actions 9 29

"* emphasis on the importance of 10CFR50 Appendix B Criterion XVI regarding management of corrective actions; this expectation should highlight that the goal is to identify and fix problems (both equipment and process) to prevent recurrence, as opposed to emphasizing programmatic processing and tracking

"* emphasis on the importance of operating experience (both external and internal) in developing a full understanding of a condition

"* defining the role of the oversight organization relative to trending and analysis of station conditions. An expectation should be provided regarding how the oversight function will utilize trend and analysis to accomplish its role. The oversight function should provide oversight of trending performedby the station, and also perform independent high-level trending of key issues as a check on the effectiveness of the station's trending. Although it is recommended that the station line organization remain responsible for performance of trending and analysis, development of this expectation should review this arrangement for compatibility and achievement of the oversight function.

"* modification of OPID values to emphasize the importance of problem identification and resolution; adopt an OPID policy of zero tolerance for indication that problem identification and resolution is being discouraged by management behaviors

"* expectations for performance b sed assessment to critically assess activities and issues beyond Appendix B compliance, including emergent issues and effective use of OE, and accountability for same when information is available

"* emphasis on the importance of intrusive oversight and aggressively (yet professionally) challenging the organization to resolve issues

"* expectations for the process for escalating issues, emphasizing raising controversial issues to management in reasonable and timely manner; and for dealing with differing professional opinions

"* expectations for adjusting th6 amount and method of oversight when changed conditions produce a change in station risk

"* an expectation regarding when to assess in-process activities; examples include critical evaluations of station components, decision-making meetings affecting safety, restart readiness decisions, decisions to fund or defer modifications, interface between the Company and the NRC, etc.

"* emphasis on the potential for complacency within OPID based on positive feedback from external sources. The expectation should reflect the need to balance positive feedback from external sources with the need for continual critical self-assessment. (This is also related to CAUSE 3)

3. Preventive Action (PR). Owner = OPID / LWPearce Modify the values of the oversight organization in the OPID Business Plan to set itself apart from the culture of the station, assure that it always maintains the highest standards with regard to a questioning attitude, and continually drive the site organization to a higher level of excellence. Oversight's values should include respect for the reactor core, principal safety barriers, nuclear safety, equipment important to safety, and avoidance of "group think" and "cockpit mentality" (looking inside without considering what the outside is telling you). The oversight organization must maintain sufficient independence in their thought processes that they do not inherit an unhealthy station culture but rather maintain a strong questioning attitude about station activities affecting safety.

i

4. Preventive Action (PR). Owner = OPID / LWPearce Case nalsi RooReortRecmmede . orrec.iv Acios*

Root Cause Analysis Report Recommended Corrective Actions

• 30

Provide organization, staffing, tools, training, office location, etc., for the oversight function appropriate to revised expectations; match all elements of resources and expectations.

5. Remedial Action (RA). Owner = OPID / LWPearce Establish an expectation within OPID and with the FENOC stations that OPID will comply with the ALARA principles, but that these principles (i.e., dose budgets) will not be used as a basis for limiting access to areas for activities deemed important by OPID.

6. Remedial Action (RA). Owner = OPID / LWPearce Develop and implement a systematic approach to sharing lessons learned from this investigation with both Beaver Valley and Perry organizations. This action includes a face-to face feedback session with all OPID employees regarding lessons learned for oversight from the D-B head event including focusing on the importance of managing corrective actions, using operating experience; etc.

7. Remedial Action (RA). Owner = OPID / HWStevens Using industry benchmarking, integrate INPO Warning Flags for declining station performance into measures for evaluating station performance.

8. Remedial Action (RA). Owner = OPID / LWPearce Develop a plan for allocation of assessment resources during planned outages. The plan is to be tied to and approved by the outage milestone for naming of outage project managers. The plan shall reflect the need to dedicate OPID resources to oversight activities prior to committing to support of line activities.

9. Remedial Action (RA). Owner = OPID / LWPearct Upon development of changes to expectations of the oversight function, conduct appropriate reviews with line management at each station to ensure that the oversight values and expectations are effectively communicated to the station. Work with station management to ensure that they understand, accept, and embrace the value of an effective and critical oversight organization.

10. Remedial Action (RA). Owner = OPID / LWPearce Establish methods for formal reinforcement of OPID values on an on-going, periodic basis with OPID employees.

11. Remedial Action (RA). Owner = OPID / HWStevens Establish a schedule for periodic effectiveness reviews relating to the corrective actions in this CR to ensure that the changes in expectations and values become rooted in the culture of OPID.

Root Cause Analysis Report Recommended Corrective Actions o 31

CAUSE 2 The training for the RC-2 event was ineffective. It failed to improve the ability of both the oversight and line organizations to recognize corrosive conditions and their significance.

This contributed to the failure of the auditing team to raise a concern when auditing the Boric Acid Corrosion Control Program during 12RFO.

ASSOCIATED CORRECTIVE ACTIONS:

12. Remedial Action (RA). Owner = OPID / MAPavlick Use industry benchmarking to develop performance-based methodologies for OPID to assess the effectiveness of training for both' QA and the line organization resulting from emerging conditions (e.g., RC-2 or the D-B head).

CAUSE 3 The Oversight did not establish an effective method for assessing the oversight function.

process for providing oversight of the oversight function was less than adequate, feedback of provided was mixed, and corrective actions were sometimes ineffective. Over a period identify existing years, assessments from JUMA, CNRB, and NRC were ineffective (did not issues) and sometimes inconsistent. Follow-up response to identified weaknesses was less than adequate.

ASSOCIATED CORRECTIVE ACTIONS:

13. Preventive Action (PR). Owner = OPID / LWPearce Develop an improved method for oversight of the oversight function. Institute a formal requirement for findings and issues identified by the selected method to be incorporated into the CAP to ensure tracking and closure. Desired elements of the selected option would be development of performance objectives and criteria, and sharing of industry best practices.

Options may include such actions aý working with JUMA to develop a more rigorous process to drive oversight to excellence.

CAUSE 4 For a period of time, the management of the audit/evaluation process was not independent from the management of the corrective action process. This lack of independence allowed the oversight director to soften the thrust of an audit critical of the corrective action process during 2000.

ASSOCIATED CORRECTIVE ACTIONS:

11. Remedial Action (RA). Owner = OPID / LWPearce Reassign responsibilities so that the audit/evaluation function is independent from the CAP.

The current organization provides independence between the oversight organization and CAP. However, the assignment of responsibility for the CAP common process to an OPID Manager threatens this independence.

- - A Root Analysis Report ecommenuu ..urrwtuvu Recommended MLuUIu Corrective MAluW"O -

Root Cause Cause Analysis

Enhancement Actions Although the following items might not have prevented the incident from happening, they do have the potential benefit of adding assurance that similar types of events will not occur in the future. While there is not a direct linkage to a specific cause, a reference by the Team to the cause they were originally associated with is provided.

14. Enhancement Action (EA). Owner = OPID / MAPavlick (Refers to Cause 1)

Use industry benchmarking, and add an element to the Common Master Assessment Plan to assess the use of OE as part of the assessment of the appropriate functional areas. This action will require auditors to be familiar with existing OE prior to performing assessment activities.

15. Enhancement Action (EA). Owner = OPID / HLHegrat (Refers to Cause 1)

Benchmark the industry regarding assessment of behaviors (including management) to the scope of OPID tasks. Add assessment of behavior to the appropriate OPID Master Assessment Plans if appropriate definitive guidance is available.

16. Enhancement Action (EA). Owner = OPID / RLHansen, HLHegrat, HWStevens (Refers to Cause 1)

Each OPID NQA section review the ISE function to ensure that OE reviews are fulfilling the commitments made to NUREG-0737.

17. Enhancement Action (EA). Owner = OPID / RLHansen, HLHegrat (Refers to Extent of Condition review)

OPID NQA sections at Perry and Beaver Valley perform an assessment to determine if similar problems exist in the oversight functions at those stations; specifically, problems similar to the root causes identified in this report and the Root Cause Analysis Report for the failure to identify the significant degradation of the RPV head.

Corrective Actions Recommended Corrective Actions 9 33* 5J Cause Analysis Report Root Cause Recommended

REFERENCES

1. Attachment to Nuclear Quality Assurance Examination of Five Closed Nonconformances Related to Reactor Pressure Vessel head.

2. TapRoot incident Investigation Manual

3. FENOC Root Cause Reference Guide, Rev. 3

4. FENOC Condition Report Cause/Tr'end Codes Documents reviewed As listed in the CR 02-2578 Integrated Database Personnel contacted See Attachment 12 Methodologies employed The overall analysis method was Event and Causal Factor Charting, as modified for this event; refer to Data Analysis / Methodology for details. The principles of TapRoot were used to the extent practical. The paragraphs below describe the methodologies used for review of various types of station data.

REVIEW METHODOLOGY OF AUDITS AND SURVEILLANCES The team approach to the review of audits and surveillances was based on provided lists, which sorted these documents as far back as 1990. The lists were reviewed for potential items associated with the identification of boric acid and (Pressurized Water Stress Corrosion Cracking) PWSCC related issues. Other typical audit and surveillance selections included those having a focus on outage activities, operating experience and corrective action. The lists were hi lighted to identify desired items for team review. Hard copy of a number of more recent documents was provided by the D-B NQA staff from their file. Additional hard copies of other selected documents were obtained by the team using record searches and the microfilm files located at the D-B Records Center in the DBAB.

The selected items were reviewed by the team to identify audit/surveillance scope, ratings, and issues identified by these documents. Supporting documents such as checklists, PCAQRs, CRs and closure documents were additionally reviewed for several of these items as determined appropriate by the team. A summary tf te rev*iew of the documents obtained was entered into the team's database. Each item entered went through a group team review as to how each document was related to the review model of detection, signal processing, output, etc., and appropriately coded. This information was then used as input to the overall team conclusions concerning oversight influence of processes that could have led to earlier detection of the RPV head issue as well as self-detection of this issue by oversight.

In addition to the audits and surveilla es reviewed, additional keyword searches were performed to find additional information related to the issue"of boric acid Corrosion. This additional information consisted of various internal and external correspondence, corrective action Root Cause Analysis Report References

• 34

documentation (PCAQRs, CRs, self-asses,,nits5, procedures, and telephonecall documentation.

The JNPO network and internal D-B information management sysferhs were utilized to extract operation experience information related to the issue. This information was then used as input to the overall conclusion made in this report.

A search of the D-B Nuclear Records Management System (NRMIS) was performed. This database contains index entries for the quality and non-quality records maintained by Nuclear Records Management. NRMS was utilized to perform inquiry and retrieval of selected documents. These indexes can be displayed on the terminal screen. The actual record can then be viewed from the specified microfilm cartridge number and frame number of the indexed record.

REVIEW METHODOLOGY OF QA TREND

SUMMARY

REPORTS AND ISEG REPORTS QA Reports: Document types selected for review were all QA routine reports to station management that summarized the health of programs, processes, and plant equipment. The time period selected was 1994 through 2001.

The reports were reviewed to determine if QA was consistently reviewing programmatic indications along with the performance of plant systems, structures, and components, to evaluate the effectiveness of the managed systems in operating and maintaining the plant consistent with design and license bases. A subjective assessment of the nature and strength of message to line management was made for each report. This was compared with the factual information on plant performance depicted on the Five Non-Conformances Integrated Timeline, produced by the NQA Team.

ISEG Reports: ISEG records index was reviewed, and documents were selected for retrieval based on the following key words or phrases:

0 Routine reports

• Control Rod Drive Mechanism flange leakage

. CRDM leakage issues

• Boric Acid

• Operating Experience Assessment Program

• Operating experience

• Containment fan coolers

• Reviews of engineering products

-Related to PCAQRs or CRs identified on the Timeline The time period selected for ISEG reports was 1990 through 2000. The primary review was slanted toward the vigor of application of OE, and the ISEG messages being conveyed to line management about performance of systems, structures, and components, versus the factual events depicted on the Five Non-Conformances Integrated Timeline, produced by the NQA Team.

Miscellaneous developed source documents were identified during the review of records found under the above categories. Typically these were specifically referenced ISE correspondence, PCAQRs, QA correspondence, or reports.

REVIEW METHODOLOGY OF TIMELINE The NQA Team provided the Five Non-Conformances Integrated Tirneline, identifying key events related to RPV head degradatiorl, covering the period 12-29-86 to 3-14-02; included was supporting documentation fore the key PCAQRs/CRs. This information was Root Cause Analysis Report References

• 35

reviewed in its entirety to identify specific mention of activities related to the oversight function, as well as references to additional implications for the oversight function. Any information related to oversight was captured in the team database and evaluated by the team for implications on the oversight role.

REVIEW METHODOLOGY OF PERSONNEL INTERVIEWS The NQA Team had interviewed 34 individuals, and provided the notes from these interviews; a list of personal interviewed is provided as Attachment 12. Much like the Timeline, this information was reviewed in its entirety to identify specific mention of activities related to the oversight function, as well as references to additional implications for the oversight function. Any information related to oversight was captured in the team database and evaluated by the team for implications on the oversight role.

REVIEW METHODOLOGY OF NRC DOCUMENTS NRC related documents considered for this investigation included NRC Inspection Reports (IR) and correspondence between D-B and the NRC, both to and from. The initial review was of IRs from 1990 to the present (June 2002). These IR are available in the "NRC Correspondence" file of Folio Views and were identified in that database using a search criteria of "Inspection Report."

Each IR was selected to ascertain the subject of the inspection. Those with subjects of: Security, Operator Licensing Exam, License Operator Requalification, Y2K Readiness, Emergency Preparedness, Station Blackout, IST - Response to GL 98-04, and Fire Protection were excluded from further review. The remaining IR were word searched using the following key words with the intent of identifying the corresponding issues:

Key Word Corresponding Issue Boric Boric Acid Vessel Vessel Inspection Alloy Alloy 600 Leakage Pressure Boundary Leakage Reactor Reactor Coolant Leakage, Reactor Coolant Coolant System Inspection Head Head Inspection' BACC Boric Acid Corrosion Control Program 88-05 Generic Letter 88-05 97-01 Generic Letter 97-01 01-01 Bulletin 2001-01 86-108 Information Notice 86-108 and its supplements Quality Quality Assurance Cooler Containment Air Coolers CRDM Control Rod Drive Mechanisms Control Rod Control Rod Drive Mechanisms

• 5 Root Cause Analysis Rot aseAnlsi eprtRteene Report References e 36

Each "hit" on the above key words was reviewed to ascertain pertinence to this investigation. If there was pertinence the IR was flagged as a "trigger" and at least one entry was made into the CR 02-2578 Integrated Database. If appropriate, multiple entries were made. The majority of the reviews were done using the electronic version of the document in Folio Views. For some key reports such as the NRC AlT Inspection report and Resident Inspector's report for this event, and Resident's report covering the RC-2 issue, a hard copy was used for review. In addition, the Resident Inspectors reports for the time periods covering IORFO, I I RFO, and 12RFO were reviewed in hard copy even if there was not a "hit" on a key word. In the cases of these key reports, entries were made into the master database, as appropriate, to denote the lack of information on issues related to corrosion of the head.

For review of correspondence to and from the NRC related to this issue, a key word search of the "NRC Correspondence" Folio Views database was performed using the following: RC-2, Alloy 600, Boric Acid Corrosion, Generic Letter 88-05, and Generic Letter 97-01. Correspondence identified from this search was reviewed primarily to determine Quality Assurance involvement in outgoing correspondence (signature on a buck slip) and distribution of incoming correspondence to Quality Assurance. The overall objective was to determine the level of knowledge and involvement that Quality Assurance had in correspondence between D-B and the NRC related to the head degradation issues. Additional selected correspondence was reviewed as identified during the course of the investigation and entries made into the master database as appropriate.

Team Members In order to ensure adequate independence of the Root Cause Team, the Team was comprised of personnel external to D-B. However, to maximize the value to FENOC in terms of lessons learned, the Team was staffed primarily by oversight personnel from Beaver Valley and Perry.

The Team composition was:

Daniel C. Poole, Senior Management Consultant, Sentco, Inc.

Ronald A. Glus, Advanced Quality Evaluator, NQA Beaver Valley, OPID James D. Kloosterman, Supervisor Engineering Assessment, NQA Perry, OPID Russell J. Tadych, Senior Staff Engineer, FENOC Collective Assessment, OPID Kenneth E. Woessner, Senior Nuclear Evaluator, NQA Beaver Valley, OPID The NQA Managers at Perry (Henry L. Hegrat) and Beaver Valley (Ralph L. Hansen) supplemented the team. These Managers were the primary interviewers, and provided review of team activities on a real-time basis through daily telephone updates and one-day or two-day visits with the team each week.

Root Cause Analysis Report References e 37

ATTACHMENTS L. Feedback Model Diagram

2. Events and Causal Factor (E&CF)Charts

3. Functional Tool Analysis Chart - Prior to IORFO

4. Functional Tool Analysis Chart - End of IORFO through End of I IRFO

5. Functional Tool Analysis Chart - End of IRFO through End of Mid-Cycle Outage of 1999

6. Functional Tool Analysis Chart - End of Mid-Cycle Outage of 1999 through End of 12RFO

7. Functional Tool Analysis Chart - After 12RFO

8. Standard Interview Questions for Oversight Personnel

9. List of Oversight Personnel Interviewed

10. Standard Interview Questions for Selected Line Management

11. List of Selected Line Management Personnel Interviewed

12. List of Personnel Interviewed by the NQA Team ttacnments

• Root Cause Analysis Report Attachments

• 38

Attachment 1 - Feedback Model Diagram Attachments

• 39 Root Cause Report Cause Analysis Report Attachments 9 39

Attachment 1 Feedback Model, DB CR 02-2578

s Control Process Output

-- q

Attachment 2 - Event & Causal Factor Chart Root Cause Analysis Report Attachments e 40

- Period 1: Prior to IORFO (End date = 4-8-96)

BI.6 R1.6 81.6 R. 1.6

DB 02 Period 1: Prior to IORFO 0vr.ih (End Date a 4.8-96)

R1.3 R1A R M.O.E.i S-1990- 1994: 5 Station Oversight QA Summary Key PCAQRs re functions were not Trend Reports for BA issues were successful in identifying 1993 - 1995 were issued or affecting resolution of issued issues related to corrosi of RPV Head prior to 1.3

.4 1.4 1ORFO PCAQRs were 1993-1995 Reports Trend Summary process90-510, 93-098, focus on Program vs was in evolutionary state 94-295,94-912, and Equipment issues over 1993-1995 94-1338. 31, 32, 33, 34, 35) (31, 32, 33, 34, 35) 1.3 R W PCAQR 94-1338 was a R

Part 21 on Alloy 600 . r (392)

SSgn ifta' PA

[* 1.4 xpectations of Trending Program, in termsofcontent* degree of analysis,

'0 6 nof 0nn os..

1990 94: 5/9 - 200vs C..... m -- )94-295 issued l-)'94-295 is changed -- + Md9-05i SPCAQ*s leakage issued rel 8) 3I-17-94: PCAQRI. t°o' (5I NoCAOPCAQR58 4-27-94: deferred/not Service-S~trct :ure imlemented (98)

POAQRs are 90-120, Enhanced inspection is Action is submittal by 91-353,94-912 deemed not a QA Director (58)

• .. commitment (58)

PCAQRs lead to mindset that CRDM.

flange leakage is arii accepted condition Conditions lead to or su stn dard equlprntri mindset that RPV Head performance, the oversight function Inspection forCROM, ft seopf fttst nozzle cracking is not influenae positive changes In required conditions-detrimental to safety.

(H04)

Porfnd 2s 10RF0 through 11RFO Oi (4-8-96 through 5-17.98)

B2.4 IZ" B2 [F2.

8-22-96: QA 10196 - 3/97: CNRB Summary Trend reviews station 6/97: CNRB reviews 1-9-98: JUMA Audit 3-13-98: QA Quality Report for 1st half of performance (366, station performance 1-Report issued (162) Trend Report for 1996 issued (29) 367, 368, 369) (370) 4Q97 Issued (27) 2.4 5.4 9[ 5 [2.

Report focused on CNR9 questions disparity 6-17-97: CNRB Report indicat R o program ratherthan " etween audit specifics Identifies need for Equipment issues 29) and conclusions (368, higher level trend report strong audit a 367, 368, 369) (370) function" (1 J I 214 B2.5 12.5 B

QA Program was getting ha('rMi.ecd ages maym mixed reviews for hve p ded a more

)n-~ er~tmtren s external oversight aggressive audit program 94

[2.6 I6 a9 2.6 4-21-96: PCAQR 96-551 is written, documenting presence 7-29-96,12-23-96, 5-7-97:

8-22-96: Quality Trend Three Corrective Action of red/brown boric acid on - Summary for 1H96 [QAD Audits, covering most of periphery of RPV head, at 96-80099] Is Issued (29) 1996 and 1997, are Issued CRDM 67 (76)

(157, 158, 164)

• .s lB2.6 B2.6' PCAQR 96-551 demonstrates Focus Is on program e audits are: AR96-CORAC-02 lack of questioning attitude; performance. Wear and [ performed 8/96 to 11/961; buildup of red/brown boric Aging is noted as 2nd AR-97-CORAC-01 [performed acid is dismissed as a highest cause code, but no 2e97]; and AR-97-CORAC-02 (

"software issue." (76) discussion (29) 158, 164) 1814

'1  ! 21"*2.6 (tolera nce for Iorig standInb

\ equipment problems,* an 82.6 IIPNOwmig Station did not provide needed Improvements in response to QA outputs

El2 Period 2: 1ORFO through 11RFO (4-8-96 through 5-17-98)

During Period 2, Engineering had responsibility for disposition of most PCAQRs realted to RPV Head R26 ff.6 [2.6

DB CR 02-2578 Period 2: 1ORFO through 11RFO Oversight (4.8-96 through 5-17-98)

Effectiveness B~2.3

El Period 2: 1ORFO through IIRFO (4-8-96 through 5-17-98)

M.O.E. 2 2.1, 2.3 D23 L

CQAdid not effect change in OE program I

R2.3 Ovrstghtt did, not connect OE with Internal plant events A

22.1.2.2.3. R 2t1, 2.2 @2.1 PCAQR 96-551 (re boric Two efforts at Root Cause lORPVO) 7acidonfound on RPV head i open Evaluation through 96-551 PCAQRI forcompleted approval are not during eriod 2 (u4) throughout Period 2 (74) SJ

• Period 2 (74)

DB CR 02-2578 Period 3: End 11RFO through Mid-Cycle Oti Oversight (5-17-98 through 5-10.99)

Effectiveness B3.5 B3.5 a 3.1 3.5 3.5 3-30-99: RC-2 9-3-98: NRC IR 98-11 1-6-99: CNRB omments InvestIgatton concludes 3-30-99: RC-2 provides positive -udits not looking in rig '" AIQC were not Investigation concludes s ta tem en ts to w a rd a udit a dl n t o k ng I rt t ariden CAPt (207) and CAP (207) pdit (357,371) place (357 -71) proactive 1 Involved In getting InRC-2 Issues" QAdoonly TechhasSpecr esou r ce,to audits (112) (112)

L - i I,. - 11 B.5 3.5 83.5 CNRB comments audits Engineering had QA did not perform C should be targeted ( toward',,

hnigpr f I *dlspos~tion responsibtilty for of most \ prior to 2000; functional only oversight ecision-making and PCAQRs related to RPV Appendix B audits (73)

"rblemsolving (357, 3D71)* head 53.5 3.

r3QA did not do u comrpliancebased; does functional oversight of CIAAudit program Engineering; only not ncliude.-n-depth continues to get Appendix B audits fun*ctonal assessment mixed reviews from external oversight "

35 B3.5 It Is unlikely that Audit/Sure would identify lpfedude~rn An EngIneering error of more aggressive audtit.

sod Judgment re the RPV r 3.5 I A R 3.6 B3.6 na 3.6 B 3.1.3.2.3.

4-21 1-19-99: 9-2-98: ONRB questions PCAQR 96-551 remains PCAQR 97-224 remains 8-6-98: PCAQR 98-767 SRB consensus causing open through most of open fothroughout is closed to PCAQR - people to omit Root Period 3 (103) Pertod 3 (393)96-551 (108) Causes and CAs from PCAQRs (365)

-7, D3.6 B 3.6,4.t a 3.8 PCAQR 98-767 documents PCAQR was open for 33 CNRB concern re fist size clumps of boron months, had 9 extensions, on RPV head at 11RFO; Shad 2 failed RCEs, was lack of ownership of some was rust/brown color.

closed as Apparent Cause CAP (393) 5 3.6 (108)

(via downgrade) (103)

[]3.6 B 3.6 Leakage did not challenge possible leak on head (68)

PCAQR 98-767 was not PCQR had been PCAQR 96-551 challenges opn 27 months.

made a restart restraint for decision that GL 88-05 startup from liRFO (108) inspections are not a a 3.6 commilment (107)

CRDM flange leakage I I continues as an accepted condition

I 0 Period 3: End 11RFO through Mid-Cycle Ot Oesight (5-17-98 through 5-10-99) 3,2 alI 3.1.3.6

- Period 3: End 11RFO hrough Mid-Cycle Oti (5-17-98 through 5-10-99)

B.32.1 R3.13.11 10-21-98: QA evaluates (AR-98-CORAC-02) CAP 3-31-99: RC-2 1998: Plant experience'd as effective overall with investigation (CR98-020) 1-6-99: CNRB/AQS notes boric acid issues I Improvement needed In conlcudes QA needs Po self assessment of OE and collective more involvement in was performed

( this significance (395) CAPperiod (386) boiltingclmp rI33.3 on RP-2vav IA did not affect change in High volume of e1 ihslwtation OE program dgtrcabu CNRBIAQS~ ~

exrseinfrm1aQqetions ~ *. detrAQ noted Op -- 9.)RI frstato slow with te a tioan t abo, ut:i an Man. au isc m ets-ed o asrsgh aicone bu6tokno(39 (387) 7 3a mtrole ab~of leadership~tt~ CiA

• reductionadt

• Mntstaff identifiedan knowC events"d°nt metcause"Wof i wtof IsEGlw improvementfrsrto (384) as a concern but took no (389)

(387) action (372)

-- -- -' t --

Mngmt behavior,are B3.6 notsupporting CAP)

Staioncontlnuea to not put a high value on CAP

• I

36. and oversight R 3.6 Some mgmt felt INPO Esamples: Mgmt behavior at Mgmt discouraged mtgs, emphasis on Pis, issue identification 1 rating built arro ance and run for (84) pI ressure to operate, fearful of "PerfIndicators (91) 'VIgmt, RSE reluctant to push mod for head Inspections B[3.6 - A k n Increased Its tlerance for "

ý.qpedbrtp ace thejoverslght Cultu~re-Is nbt on S missed opprtkunIties to lncre.singq conoyeptyqopln~

issue identiflc-atoný 'and ." .*e .nl~liln ~.,~ .. .:i ./  ;

thoi~ough reO'solft'6oh

• no...

fi~ ....

ers!htio an Wh~~hn'trdh dusov eAer I

~~ ~ a - n.

R uBR0-I5 3.1 S Period 3: End t1RFO through Mid-Cycle 01 (5-17-98 through 5-10-99)

M M.O.E, 3 Station Oversight functions were not successful in identifying or affecting resolution of issues related 19 corrosion of RPV head, End d 11 RFO through Mid-Cycle Outage of 1999 R 3.1 PCAQR 97-224, written to improve CAP actions, is open throughout Period 3 S3.1 D3 2 R3.2 23.6 1-19-99: CA Surv (SR-98-MAINT-07) requests that investigation for PCAQR 98-1885 include decision to live with leakage on RC-2 (309)

E)3.2

0 . Period 4:

4

@44 A [B4.44 7-3-99: CR 99-1300 (iron QuliATrn 9817-99:

oxide on rad monitor filter) 7-25-99: NRC IR 99-08 reflects Summary Report for9179:A was Missed Opportunity for DB knew they had not achieved 6/99- 7/99 . . surveillance on trend and anlaysis to pre-code safgety valve leakage (QAD-99-80088) is EngIneering Is Issuej identify probtem (89) rate (220) Issued (16) (SR-99-ENGR-08ý (1184)

@ 4A 4 Reports fails to mention issed Opportunity for A, in 08 inspection plan makes no CAC cleanings and daily that QA fails to challenge mention of considering RPV rad monitor fifter work. QA engineering that assessment head as potential souce of falls to connect RCS teak suggestions are not being leakage (220) rate Issue, rad monitor seriously looked at. QA "filter, and CAC (16) accepts Engrg response that W. no now Engrg ectvtvltes will R4.4 be Initiated (184) 4

-7/99: Cycle 12 VJ Periodic Assessment of Radiation Monitoring QA fails to cite chapter System (NPE-00199) and verse regarding reports functional failures equipment degradations of rad monitors (38) In the plant; IAfocused on programs and process, not equipment (184) 4.4 QA and SEdid not onditio of RR leak documenting CNRB 10-8-99: Quality Trend CA es NRC corroborates corroborat fc assessment atstme't cSummary Report for 8/99 - 10-23-99: QA completes audit "thatCAP was effective at time concern re lack of 9/99 (QAD-99-70145) is issued of CAP (AR-99-CORAC-02) of RC-2, and remains effective' ownership of CAP, is (17, 18) (143)

(313) closed this Period (393) 4.1 4.1 Issues are CA timeliness, use of QOA notes weaknesses in TERMS, Extent of Condition, Strending of equipment, CR CR Initiaiton/categorization, data, and other sources and trendinglanalysis (143)of

El2 Period 4: End of Mid-Cycle Otg through 12RF (5-10-99 through 5-18-00)

I "Theweakness in Trend

• E tive Corrective.

and Analysis is being Action re Trend and identified to Station " nalysi* t*hot Management completed 4.

4 Oversight did not require that a I /consastentirendlngancdralysIs progam.

S wanot

_ defned and .doctmented n

,r rsp ab~lt Gul~~

17 and~ ~~, 'I~~no~d~t~rm 1 4 W.41 @. @ .1.4.6 4

B 1 a 4.1 Two major engineering groups I ssues are CA timeliness and

• are noted as exceeding station equipment tranding. Performance for goals on CR investigations:

completing Significant and Important station goal is 60 days, but these issues declined to the poorest groups are averaging 266 days, performance since the indicator was wih downward trends projected esiabi shed (19) (21) 5 _____________

B5-2 Period 4. End of Mid-Cycle Otg through 12RF(

Ovrih (5-10-99 through 5.18-00)

R14.5 B4.5 B4.5 S

B*4.6 B4.6 8-16-1 DB3 violation from RC-2, (QAI that some Man ý S7-1-9g: InresponsetoNRC acknowledges require maintenance practices and oversight Increased (314) additional assessments B.

8*J4.1 oe and2O (CRis thatt acin y r) s tha R 98were deuthet 0)es ron oisses o to boicn acd reatd );r

/C S i t9a1t onrs d er [ ctions (CA4 and CA9 from 1300 abnilyto oxinde aeut C/ 990 (boron wer lesta 4.

bli ty tov pefboron oronwhn der av bupnerte 00-78 (rb1 of RPV head);

ctiones A M inmos pe s ag l t M tha head ~ ~ ~~had 2 F )

'l tevos 3rdtoa

.JF reouredn5

• • ! of flowso S4.  :*.= fava-like 00-782 (rdbon

El2 Period 4: End of Mid-Cycle Otg through 12RF(

(5-10-99 through 5-18-00)

R4.6 B4.6

-vrih 0

34.3 Period 4: End of Mid-Cycle Otg through 12RFC (5-10-99 through 5-18-00)

[R . .o.1. 4 Station Oversight Functions 3-2-00: QA surveilance were not successful in R-00-OUTAG-01) on 12RFO Identifying or affecting adiness issilent on use of OE resolution of Issues related tq r outage preparation (182) f corrosion of RPV head, End &

Mid-Cycle Outage through 12RFO W~2 R 4.2 S4.6 A

DB CR 02-2578 Period 5: Post 12RFO Oversight (5--00 through 3-18-02)

Effectiveness B5...4 R5.4 "6-29-00 Audit report evaluating Trending and 7-7-00: Audit report on various 10-13-00: Quality Trend Analysis program 12RFO activities Summary for 7122 - 9.30-00 (AR-00-CORAC-01) (AR-00-OUTAG-01) (QAD-00-80151) is issued (22, 24, 138) is issued (135) is issued (1)

R* 5.4 []5.4 []5.4 Tre nd p ro g r ,:*

t r'h i rendsn wseautd X C No mention of C*

_( CP process paramei e oisusieniid 15, Rad Monitors,, ACs or btprovides little analysis noise dntfe\15 thought they are even still an (2 issue (1)

CAs from 1999 CAP audit OA recognizes The head had not been could not be closed; "thoroughly cleaning the th cleanednot actions judged not RPV head" (135) detectedby Q-A 5*

.4 1.4' R5.4 Weaknesses identified in' No/*adverse quality trends NormI" accomplishing main have been identified by No foral process exists fuction, i.e., identifying ] Trendraig2nlue)Ro Program in over 2 for detecting adverse aderse trends (22)/

CAPciv years (22)

• trends (22)

Rated CA ~ X' /gnl CAP rating includes Root Rated CAP marginal, Cause analysis including RCA (138) evaluations (138)

C. ,s 7 5. L5 - 5.1,5.2 R5.

7-21

-0r: Surveillance on PP 3-20-01: Surveillance on CARS 3-27-01: NRC lR 01-015 8-2-01: Quality Trend CAAP (AR-00-RPRWP-01) 1- (SR-01-CORAC-02) provides strong positive Summary Report for 2Q01 i reinforcement on CAP (and (SR-01-QUART-02) adds CAP is issued (181) is issued (310) RCA) (227) as functional area and rates satisfactory (7)

• en i* B/ Inconsistent criteria for i*5.1 R

Idenlifies wealkness in acceptance/rejection of sAP s C and ts RCAs used by Station Is receiving mixed messages re health of CAP QA Ineffective in getting line management'actio on CAP effectivenessIn nrotsufflctentfY Qvrtgta inderfylnjg Toresolve-crtttcal pfoblems, ir Lthei CAn.,or sightffed Neither ine norQA are effectively using CAP data. (H04, 80.6)

DB CR 02-2578 Perod 5: Post 12RFO Oversight (5-8.00 through 3-18.02)

Effectiveness R.5.4 5.4 5.1, S.4 8-1-01: QualityTrend 11-7-01: QualItyTrend 12-26-01: Audit report on Reg Summary Report for 2Q01 Summary Report for 3Q01 Affairs activities (SR-01-QUART-02) (SR-01-QUART-03) (AR-01-REGAF-01) is issued (6) Is Issued (8) is Issued (125) tx,4 ., I L, ;k-T-Tle follwn CRs were issu this Period related to boric acid or PWSeC: 00-4138, 001547, a 53 No probe as tiowhy Rad 01-039,01-487, 01-890, Monitors came off (a)1list Trend and Analysis of 01-1110 01-1 191 01-1822, (8] OE sent mixed 01-1857, 01-2769, 01-2795, message: rated Sat 01-2852, 01-2936, 01-3025, with 10 CRs (125) eelbAi1td(Timelune)

-(a)

' 5.4 dened to be* .0,A dd .not:-ade duat Report is broad "an L*R=.4 "overall Ssummary." F 1station a" i zIe la'-6dl tlher than "'+': .. e Trending Is evaluated; no process, ra nt (6) issues identified (125) equipmer I . ý It.,

Identifies CAP issues:

• .* timeliness, cause analysis, review effectiveness (125) 9 T N

B 5.1, 5.2 During this Period, some S5.6 employees have _ sOnEngr has concern re*

concerns about the CAP ngineer felt he had been owership of RPV head '

(expressed in 2002) strong-armed by outage inspection, and that the mgmt team when issue is not In the CAP presenting lava flow problem (80.1)

B5.6

_A employees feel CA*X Megvmt lack of concerin has not been effective in addressing CAP) observing and weakness (346) .2aluating lava boric ciaddflow (80) 85.2 L.6 SCR 00-782, CA employees feel mgmt SOwnership of RPV has an inappropriate - head inspections influence on outcome of \and not cleaning RCA Teams (351) entire head (80)*

Ak B I.

"1

• fi, 5.2 I

I 12 .

.2 Boric acid on head is an RCA continu ed to6SO r` acceptable condition contribute to Ineffective K", CA~h~s'ses.

CAP enfl b,ýýgt

R5ý4 El.R0-27 0 Perfod 5: Post 12RFO (5"-00 through 3-18-02) 2 5.4 A

S5.4 o 0184 OIIU tMIdINb was tot compIeted 6ý

[: 586

Period 5: Post 12RFO (5-8-00 through 3-18-02) 5.3 5.3 5.6 5.6 I

B 5.6 a 5.6 Management changes uring this period ALARA Smen

.and e~nragmn concerns may ha~ve h~e' ~conrveyes thos~cha~i1~, wrong message.

rI clouded eaxýpŽtati[ns of i ht placing ALARA above oversight functions assessment

5.5 El2 5.5 5.5 Period 5: Post 12RFO (5-8-00 through 3-18-02) 3-27-01: NRC inspection 1-10-02: JUMA audit report report (IR 01-05) is issued 3-18-02: NRC Inspection (AR-01-JUMA-01) is issued report (IR-O1-18) Is issued (227) (127) (229)

NRC concludes that QA 5 audits "effectively identified, evaluated, and NRC reviews CIAaudit JUMA rates CA and surveillances of corrected planct functions good REMP, and has no problems "

findings (229)

Mixed messages may have precludetid a more aggressive audit.,

5 Correspondence files for 5/88 to 3/02 show no apparent OA involvement in significant NRC issue correspondence (396)

CA'misses an opportunity for providing Input and getting informatfon on issues

-F- ý2 During this period there are signs of a changing culture Ba5.

B* 5.6

//

Recognition has that CA been effectvISEG B 6 Increased CAC tending to cleaning frequency (CR in addressing (346) weaknesses CAP reinforce ineffective CA CA is not consistent by accepting less tha 00-4138) indicates when issues are not put acceptance of 1*56 effectiv= CA (39,..

"*... 18).**** on CRs (345) extensive Work Around (4)

-' -" (41 pMgml interview Tolerance of station problems and lack of a 5.6 B5 6 ,tl k Q......,

Quality Trend Summary

"-C / Report for 2Q01 \* '[; ; ,:i  ::*; - .i:,:.:.

(SR-01 -QUART-02) not g f* a . e  :'ac 'h Sp exploring e rfo rm achange nc e (5 in

)

• oversightfu ctlon Inc~easlg1ymi

• * ' '* 'c

• i "e

• --'--*l*t f~r~ize R 5 'ch rlea nd *f>' eed the risk ofrna),agerenet to a5,,ve rsght o C//ulture may be slowly ý

// improving, but is still rhotu n i4po lr conducive to prompt oidenification and issue i I na resolution. ALARA concerns created a reluctance to observe or tderoughlica tionnelfy

• thorough resolution

• certain fleld-actlvlties. (04

.,..,:.-..5 .,- *. .... ..5-..,* H02} ..

5.5 - Period 5: Post 12RFO (5--00 through 3-18-02)

SM.O.E. 5

El2 Periods I - 5: Incident (Prior to 1ORFO through Post 12RFO) 4

- Periods I -5: Legend (Prior to 10RFO through Post 12RFO)

Legend

= Incident ( White ) = Assumption White = Event = Condition White

= Missed Opportunity Event Light Yellow

> tlltie = Deduction from facts; conclusion A

= a failed barrier = Causal Factor Note: some adaptations are made to traditional E&CF charting is this E&CF chart. As a result of the extensive time period involved, and the extensive = Text Symbol above block amount of data collected, key alterations are: Any color indicates Comment a) not every Event is sequenced in timeline order (doing so would overcomplicate the chart); some grouping with Funtional Tools is retained b) the depth of the reasons "why"'for the events and SAny color conditions varies, depending on information = x identifies the time Period (1 through 5) available 1: Prior to 1ORFO (prior to 4-8-96) c) use of terminology "Missed Oplortunity Events" 2: 1ORFO - 11RFO (4-8-96 through 5-17-98)

(see Report); see shaded box light yellow) 3: End IIRFO - Mid Cycle (5-17-98 to 5-10-99) d) Use of terminology "deduction from facts (a 4: End Mid-Cycle - 12RFO (5-10-99 to 5-18-00) conclusion)"; see shaded oval flight blue) 5: Post 12RFO (after 5-18-00) y identifies the Functional Tool 1: Corrective Action Program(CAP)

The layout of the charts are generally:

2: Root Cause Analysis (RCA) 3: Operating Experience (OE) 4: Trend / Analysis x.3 x.4 x.5 5: Audits, Surveillance, Evaluations 6: Culture / Values x.1 x.2 Note: within any shape block, codes may be found in x.6 parentheses at the end of information. These codes are of two types:

a) numeric only codes [e.g., (87)] identify the Master Tracking Number of the item in the investigation Integrated Database b) alpha-numeric codes [e.g., (H04)] identify the Cause Codes from the Corrective Action Program. Cause Codes are found only in Causal Factors.

Functional Tool Analysis Chart - Legend Conditions: Events (or Resultant Outcome):

Fact Fact C)

Deduction X Deduction (by team from facts) (by team from facts)

Shading:

(D Grey shading in any box represents input to Cross-hatch shading in any box the overall conclusion drawn for that represents information duplicated and Functional Tool Analysis. brought forward from the prior period.

Functional Tool Analysis

Conclusion:

Each Functional Tool Analysis includes a conclusion for that time period in the type of box shown. The triangle in the upper right hand corner indicates that this Functional Tool Analysis is a "causal factor".

Summary Sheet Information:

The Summary Sheet for each time period includes each of the shaded boxes from the attached supporting analysis. This includes the smaller "input" boxes and the large "conclusion" box.

Attachment 3 - Functional Tool Analysis Chart, Prior to 1ORFO Root Cause Analysis Report Attachments

• 41 Root Cause Analysis Report Attachments 9 41

Assumption 3: Th, Oveight Rational prsor had two disirtict opporenenim Wo signiticedy ide the idatuteoO e:

Functional Tool Analysis Chart - Summary Sheet - Period 1 - Prior to 1ORFO (4/8/96) Asture thsgh pmepo.esighb functosthat thepo Urie Rtiionsi Pero poxu.-e-d by the

-r suffieitly robust sd effectily sd todowect ard mitigateth ceedidoa.

1, Dan detc1-ion =d actioato aitigate the condition.

Assumption 3.a Assumption 3b 1.1 Corrective Action Process Although detection via CAP Detection not probable - insufficient

............. was not probable, had the opportunity

.. a

.N station placed more emphasis on aquipment problems that

00. Prmd.aocisr0Pro.eQ=thnetoi had the potentia to impact P0g oig toceee-ý~w~w eodere plant sa*ety this mould have lo,GRM Hl.Qelealo,9' "skity."u111"o~t. contributed to identifying the

-CAPr,ot ity toI0 -u.su sooner.

1.2 Root Cause Analysis Missed Not Applicable Not applicable Opportunity Event Evaluation of Root Cause Analysis data indicate there was little likelihood of impact during this time period Station Oversight functt uions Had the oversight 1.3 Operating Experience Process organrzation identified the ISEG was responsible for there not succetisfui in meak OEAP and tffected technical evaluation of OE.

identifying or affecting change, it would hawe They could and should resolution of issues reiated to contributed to the line have detntitied boric acid cdo E notailon corrosion of Reactor idendtryingboric acid corrosion and AJtoy600

, 0, In- oroisi Pressure Vetisel Head prior corrosion and Alloy 6OO PWSOC as important to Mr to CohPten notlid uL PWSCC as important to track as piant isse.

to tORFO track as plant issues.

S.............................

1.4 Trend I Analysis Insufticiont information Insufficient information was available ar this time was available at this time to achieve a meaningfui to achieve a meaningful trend. An etlent trend trend, An excellent trend t o 4n ,e r c0 omde , program would not haoe program would not haos

. . . . . . . . .,.. E dto c Dooro..1 Ccecteeln te, thi. preld: " i unlikely supported identification supported identifitation tnht a highlyefectie progrominorary one o the based on the available based on the available tunctional tcisa would alone booneiiowed detection information.

I information.

ofuthe event at this early stage. It tho CAP end OE prograrnswue, both highlyeffecthveaodoporting

,i, a cultrl..I environment without the moukeessen iontitied in thesanalyses itoi pussible that tha 1.5 Audits, Surveillance, Evaluations issue could have boon identifiedan catty an Not Applicable Not Applicable 1 ~ORO It wouid havecreqireod knowlredgegoing into 1 )HPOthatthe boric avidcorrosoen on the head cas likely to oocur and to have looked for it Evaluation of Audit, Surveillance, Evaluation data indicate there was little likelihood of impact during this time period Ovreright appears to have Station values have 1.6 Culture Values ------------  :-2I accepted (and commonly will already begun to form that will affect the ability accept) the same values as the station. When this to detect the head

• Hod,theCAWsoearrtt

1. occurs, it makes it even degradation for years.

, oitetrfc re.*nncadntpotamoo, h more difficult for oversight tu The acceptance and

......... ....... oenostrsehtinoeepobotve *meoe in detect and effect change in justlsication of equipment station -at.e. that would problems that affect have heiped to preclude the safety systems cannut be iakcr.0it fitnot/.

tedtI*K .otiedeeriStott II 0 . C ,: reI t- r.....

. . . . . . . .. . event. tolerated,

...... . . . ... . ... ... . r.. u...d- .. . ... . .

Page 1.0

Functional Tool Analysis Chart - Detail Sheet - Period 1 - Prior to 1ORFO (4/8/96) 1.1 Corrective Action Process Detail - Prior to 1ORFO C*t1ML~e. 0c =e. 92 '2rC1D22  % causalFactor -M1 206Pe/ Wý..Pne-e

..... HumanPart. 1H"$0(WMW0dW1068 pw-)

1.2 Root Cause Analysis Detail - Prior to 10RFO Little likelihoodof kmpact during this time period.

Organizational 1.3 Operating Experience Process Detail - Prior to 1ORFO acceptanceof Ovwesight feediback 46 - ISE-92-00128 SE dnw 0,Suevsry mnfethods 43 -ISE-90-OO153 continuIng problems fc dwrinfetv

~Exta&ti ~okrn for OEAP n RAC-01ruihr OEAP did nnt affec CORA-01 EAP change In OE modrpd-tely rCausal Factor - oo4-oca*u2r0 . 82E0.. &

arIght High -turma ot unctionstracked externalOE poac IN5-0t lns1,Isueoutde information Human Per. s SM* Npyl Ho "Ward*

.AO

.. ~6 Y 21etclng toM connect OY

=ln A Issue, 93-0098 94-0295, 94-0912 Page 1.1

LAl 0 -4 C

0.

C

'A a.

cn C

C 0 (0

C 0 C

'0 a 0 0

S 'A, 0

0 C 0 (D 11 0 0

-v 0 0 S

0

'1 0

CD CD iA 0.

0~

-Ti 0

00 I I

-v o

j; ai iI1i jI iii

-u ID V F'j

& a

.)

- Functional Tool Analysis Chart, End of 1ORFO through End of 11RFO Rot aseAaylsRpotA cfmflS. qe=

Root Cause Analysis Report Attachments

• 42

Assumption 3: The Ovsight Rtiera Prsion hod two distinct opporunides to signifolady ahtdie ulim-uteouoer*:

Functional Tool Analysis Chart - Summary Sheet - Period 2 - 1ORFO (4/8/96) through a, Aosurvdosogh pmpop eronight hancuos, dot thdproces.- cord by the Lien Reation P-eeo wo sffii6oedy robsst odnifrtioly esed todieect 11 RFO (5/17/98) ted mitigate thie ndidto.

b. De- derioo andscti.. to mitigaottihrcodidon.

Assumption 3.a Assumption 3.b The oversight organization PCAQR 96 0551 could 2.1 Corrective Action Process ...... should have been hblsto have led to discovery of detect and effect change to the head issue it had

. Lraodsnio/de a lsn orelsaa.ior , managementn s low value been effectively CoI =POOQby

• et i6ssr, ftiddeemso st ate by evuthi withty Feluolln placed on the CAP addressed in a timely manneot ISG assumed responsibility for this

• g.o.l.... . .. .l.. .r..idght fun.on.

investigation during this time period.

2.2 Root Cause Analysis Missed Same as for CAP aboveo Same as tor CAP above.

Opportunity See contribution to CAP above , trnterwstadtsCP

• mgmntm .pK P.

wernooondc mre5eibpoenrntht.#  ;, Event Had the o-ereight ISEG won now 23 Operating Experience Process ----- ------- -' Statton Oversight functionss organization identified the responsible tov the were not suiccessfuI in weak OEAP and affested overall effotrtvenesa of

,005, 0.44- se-ow.

identtfying or affecting change, itwould have the OEAP. They could

.......... .... lovsop~taroslvtOhEPe temO~dosrg t contributed to the line and should have resolution of issues relgted to SOversighttid not r llhetfrrfl

-,rttd dentlfyring boric acid identitfed boric and imaonnctOwith . corrosion 0f Reactor corrosion and Alloy 600 corrosion and Alloy 600

• Events

~ -".

Events Internal Plant

. . .. Pressure Vessel Head AFO 10 through RFO 11 PWSCC as Important to tack as plant issues PWSCC as isportant to track a. plant issues.

2.4 Trend /Analysis CA Summary Trend Suffioent information reports could have is now aeihable to

• Owrd tes. Sa rsqera o

n2eees  : OrerdrasOaths

, Gd d proorotnt ot n. included equipment isues and develop a meaningful tred related to boric

• long.term.fr.nd ' rede* Steseretd o e"n o Its accomplished acid. Plant information

• Trendlo iotff t ' svi overall conclusoan for this time period: meaningful trsed from "O-.t10 and In RFP-10 and RFO-it sufficient analysis, FO-tt along with information became avaitlable to detect the 02 should

.0hle head degradati.ve The CAP is eot have been connected, supporting effectve resolution of the issues as .tdeincd by PCAQR 96-0 51.

25 Audits, Surveillance, Evaluations Additional OE s also available, An effectrie Had the oudivscuredlleco With appropriate orgasization houid have detected the head program been structured technical expertise degradation in this time period Station with functional areas and conducting technical salvu. detracted from the ability I l(heCAP S... , .... ........ ... . 04.Mot Emrm thietlonr* and owernightto be effective, epectationr to go beyond complisnce Itcould hveo audits, f would he been more likely that

• Maxd Message  : Itteecinrew* ttd NitsmnIreheo Thes-ik05* piwaid gmns been more effective in identifyorgand correcting the heed ivsue would have boen identitted.

MY have precluded, ldetshiaErnnk,6qlnrrrta (%e" 0 nornrae ndpR " seeedtof.sei I Sa more aggressive

• n RV tn Unwed rght . programmaWt eudioprogram A weaklmsasae 2.6 Culture/ Values ...........: 7WPCAOP l ,. - When oversight is operating in an environment where neither The station values that have formed regarding the boric

• toleratetg Station dO-s, ot oquestioningatoitude etandhrg equip. prab ** v tlu Overasght they nor the CAP are valued. acid on the head continue to

• s. ltiOi rF fat 0 becomes difficult to dotect be reinforced. The values of

- e 00 a. ooeetolaesh.

wr

..- ;~ i* . ., . ..- ..

..- .. .... . . ... .. .. .. . asrdatfest needed change. the ovesight organization

• ouhsnattaWodrnoenmtetonsr 1 Ice

• c'MeIn t  :, CtRDM Fange MndoetthetRV 9402 DlRPVd hMod

• eare-hoemtrn saghtas W elý Clearly understood mere closely aligned with the

' Station does not etgtectattons should provide station, Therefore it is

• equtk CAG leakage acceptedt Headda/linpction Ieti- portPVheadA oversight the opportunity to ditcult to detest the gfeenirrgiOt( conditions not requiked ot *np voice CAP providepositive impact on the degradation in the presence station as a whole. of these values, Page 2.0

0 C

o 0, C~ o 03 C~i L I) 0 Li oo Jul k CDU ER 0

Functional Tool Analysis Chart - Detail Sheet - Period 2 - 1ORFO (4/8/96) through 11RFO (5/17/98) 2.3 Operating Experience Process Detail - 1ORFO through 11RFO

• ~sru~tranrop~intenoenght

• Ovortoightdid toot conhnctOEmigr lattoysalPtetth Causal Factor -1 O4, innrin n*-orrino*iteno.&

nee0..w0 )iduoy p

etotohy u..d

.srey .o1oh 0 n.*iI-~

162 AR97JUMA 01 *suewit OE 57I A Trn Sumnery OFAP 01 O 0E Mnc 0 Reports reiew LTA Screeninag (recurring)

HihSi- d Sco Organizational acceptance of Overnight feedback H02, Superansoy methods were urettoctir uaP ternal $0an0 p ih h Page 2.2

Functional Tool Analysis Chart - Detail Sheet - Period 2 - 1ORFO (4/8/96)through 11RFO (5/17/98) 2.4 Trend / Analysis Detail - 1ORFO through 11RFO 370- CNRB moutonolentify needfor higher IeOeItr0rndreport Causal Faetor 0 PooornPaore 0he oe, 17 20 hr natlien 00 hQiponmn ,olerratagrO etnolvY No 05n010.yroodOrnrrr Foote POOr.nr EqIdotee,.. HoanenPert. -. utdeaerctvoeophWeo y.

7- OrinlntyTren Report 4th Oti 97

- no OnolysIS of longtenortrendo 2.5 Audits, Surveillance, Evaluations Detail - 1ORFO through 11RFO ErnOelo had 73.OrgOnedo.No d400p nd1.-t dr 2000.

AppwI.

We, 7e.3030. 232,'203,207, Audit 316 NRO Heed

,102, JUMA mressage on audit audt function" Programn.-

nkI Causal Faotor . 0o

• Fre*e*n;-non.Wr 0004Meer-ner. EM n nor WeedAre oOo*o 00e luren tunctional ov-ight 4. olddlrdpr of Engineering,only d00teere needtobe leerr~.nno App.Baudits Human Part. -o- cse,oenrtorn Wo0ria waloreeSkILProWleny CIAAudit Program ----

'----5g---

wansgettingmixoed oehvepeod, revWaw for external a orore.greela oversight

• ood itprogr ' .s Page 2.3

Functional Tool Analysis Chart - Detail Sheet - Period 2 - 1 ORFO (4/8/96) through 11 RFO (5/17/98) 2.6 Culture / Values Detail - 1ORFO through 11 RFO clfeaqng isO, ""7t.v.........

• leed/inspection t*A*, .66* i,no ~orl nod lii.~ nridot , ' ;i cý,ndition - n to*aori fl-.drtunntor Jh

.oohaJ.. . , .o . .. n . .po.ii

. i.

.hango eere f5 c 94025byodos rio reUid 6 0 76 -PCAOR96661, iok, *75-PCAQPR We551' Causal Factor -- H04 ou,-.,

M .tlOn.

• a <di lo gW+i g e.. r. . tolerate Iong M u nt noon-upn -li-dog did astanding equip. prDod Hua PernUy trot--m.

. ,* ...o. . . .

• .-.0.dhi~flroian.*"{

• iNPOWarning Foag Human Perft -p---

27+.c0AD ou8- . ........* ornn~.

381-CNRB questions Stations' Station doeosnot

,CO c+,,c+,.

.........-.. ' +n.,...

TreadoSummar , ...

O!i0~iO 00 =,

oi.

3, C,,=,

valuation of Oversight

---,* .. +

,,ue

• 0

• Oversight, on............a. ' 379, 380 - CNRB Siatin<:oes S

, .atto, t.ooa, QA *týt ,.

pni* tquestions valuation Stations' ol CAP auCA.

Page 2.4

- Functional Tool Analysis Chart, End of 11 RFO through End of Mid-Cycle Outage of 1999 Aalyss Rpor RootCaue AttchmntS* q Root Cause Analysis Report Attachments a 43

Assumption 3: The Oesight Ratiorns Prsse had two distinct oppornuises to sigeifieady aIto theebusst=mcer*n:

Functional Tool Analysis Chart - Summary Sheet - Period 3 - 11RFO (5/17/98) through i binctions that [beprocesses used by the Assur through prpar ovossight Line R11.onrdP.uon wem sufficretly robust andefftndrseyused to deect End of Mid-Cycle (5/10/99) and mihgatethecondition

b. Direct detectionad adconto miigaoethe condiotso.

Assumption 3.a Assumption 3.b Although GA did address 3.1 Corrective Action Process -*

CAP issues ia their audits, Comprehensive

- ------- - - H0, Mianagement 6pcaiqinttseo the program did not assessment of the issues Oasraiahknsslonlh .

"andWorker Aocountability s GOeruight

. wao not ffielentby significanty improve. Even if the line did not act on the associated with RC-2 should have identified the

-i' esedhoarr I findings. GA should have s niodeance of the I npre

• CnsAP critical to resolve underlying continued to press the ongoing boric acid isauen

• problems, Including the CAP.

issues until effective withthe head.

change occurred.

3.2 Root Cause Analysis There were multiple If an assessment of the examples of weakness in RACMassociated with the

............. .............. the RCA process A borc acid issues was done

• n asahe=olmntI,*eses Puii reosl sn a hr

• oweedesasseraas~l n mauidght' t addressCOP , Missed should hboe identified the weakness and sffectod it hikelycould have led to datitiyfnrg the Issue with

. . a. . .s. . .. . .. . . . .tiprove mteot. the head.

Opportunity Event Had the oversight ISEG was now 3.3 Operating Experience Process .............. organization identified the responsible for the weak OEAPand attested overall uffectibvenessof

0I4,1nledeahyReutew change. would have thu OEAP. They could

-,005. In-Hou~e Eiqerienoes contributed to the line and should have "6 s ightdid not

' Expectations for OEAP In terms

,ofoversight eaquiruwontand identifying boric acid identified boric acid coneeltOEwith re5ponnsibilitiesae not beIng Station Oversight functions corrosson and Alloy 600 corrosion and AJloy600 PWSCC as important to PWSCC as important to

• Internal Plant 5 smut there not suc essful in S..z..* ?

Eveonts

. f  :. . . . . . . . . . . . . . . .

• identtfying Dr affectin g mrack as plant issues track as plant issues ,

resolution of isusreae to corrosion of Reactor GA Summary Trend reports Sufficient information is 3 .4 T re n d i A n a ly s is .. Pressure Vessel Head End of could have included now available to develop a Osm-opdtlM rfsbieOWs a assltat RhO- 11fhrough Mid-Cycle equq pmentosuesand meaningful trend related ar

. ,t er, e eroftg ahnaltya protes Outage of 1999 accomplished meaningful to bore, acid. Plant s bedei ra* ea h learnste, trend analysts. inormauon from RFO- 11

- -- -IN

- -- orenian ,eish soStun onseetm and RC-2 along with "equipmentissuee loag tienens rndes -

pw* oe*atbas ectsidsanshea iresreds -,

availeble OE sbeuldhave been connected.

Goarsi eanolusin tenthis tires. panlad 3.5 Audits, Surveillance, Evaluations ................ RFO- 11 through Mid-Cyolemore information Had the audit'surveillance With upproprtate became avaiable to dtaect the head program been structured tehncarlpriae degradation, however some of the with functional areas and condcttng techneael as. Pntrstr oeailern information was less direct, and may have etpectations to go beyond aodits, wto uld hase

- - - - - - - -_Message_____Ilk_ nslaetIMsatrattsedsa even masked deducing that the leakage oomplianoe it could has been more likely that the roetienretetimre55'ad'ierkenm could be related to RPVy The CAP ta not buee more effective in head ssue would have

[. . . j supporting effective resolution of the issues identisying and correcting been identified.

as evidenced by PCAQR 98-1904 Station programmatic wtaknees.

.. mor. ag..ge..n... .uealuenrts.

.. . a. . ..a..a.r.ee.e.u.hs.n.a n.e.u..

values detracted from the ubility of the CAP and oversight to be effective. An effective organization may have detected the head When oversight is operating in The station values that have 3.6 Culture / Values

• oerr! Iae se I dro Ytsi Aeorortabiiw A"s en esee httneo ed It rmen dagradation at this time period. an environment where neither formed regarding the boric r e "s they nor the CAP are valued, acid on the hbod contiue to anal Vh-ý Aet siste ordssneaW b torass. Ve n becomes difficultto detect be rewnforced. The volues of and affect needed change. the oversight organizuation

,_0t C8DpasaSWeskaga sefoaar Clearly understood were closely aligned withthe st ls e>aictations should provide station. Therefore it Is i.I.. aet.... .. a. . ..:

~

-s esrsaste c aseua maesa~e 5 intVn earrsr vooo The

.....-age n...s

... ns e CAastiaserue-AVot

,to e s ft.

aSgna s . n . o. vno pin .. g ora oese bsgt oversight the opportunity to provide positive impact on the difficultto dtaect the degradation inthe presence

~~

.~~~~~~~

PO~e rpelton donrignoequlad . u~es esesiant

. .srntr.re...s0 ..

suenbaan." -dUre-~g oe

. ---at u on.

--- --- station as a whole. of these values.

Page 3.0

Functional Tool Analysis - Detail Sheet - Period 3 - End of 11RFO (5/17/98) through End of Mid-Cycle (5/10/99) 3.1 Corrective Action Process Detail - End of 11 RFO through Mid-Cycle Cauml Factor -- t, 04ma~mwwrw*ý ý.

n4 We* AO- .Ok dU did NW&f*eOb.dy. CAP..oe4 Human Ped. -1 S.O* 0* N y High ýma 0.-r d)

Page 3.1

Functional Tool Analysis - Detail Sheet - Period 3 - End of 11 RFO (5/17/98) through End of Mid-Cycle (5/10/99) 3.2 Root Cause Analysis Detail - End of 11RFO through Mid-Cycle Causa[ Factor I 002.SuPe&4 MS',oOw& Itfltee I

206 - NRCIR-90-t1i Human Part. - coefa. btom Lrrproverrent assignmentOt of RCA 0.50St tec (L0e.ý 3.3 Operating Experience Process Detail - End of 11RFO through Mid-Cycle

' 004, Anuotyl Review, ]

.,Q05,Irs.HoueEporenleoe Er~rectetoforOEAPitsfeorm.

onfn-lght requiretcact and Causal Factor -- jý 005 Iauy~ea0ote&

C 0 ý0rfeOý~

og. into,..r~ nln 003 -1oo~

tnr0nEn~ne.

p-,

Human Pert -.- I LeeryP-aonY W-e krae000m &orftoerr Organzational acceptance of Oversight feedback

• Overslghididnot

• IntelnalPlant Page 3.2

Functional Tool Analysis - Detail Sheet - Period 3 - End of 11 RFO (5/17/98) through End of Mid-Cycle (5/10/99) 3.4 Trend / Analysis Detail - End of 11 RFO through Mid-Cycle Caual Factor .. o P0,0I Po000ý ý0 Th oo d 0c, alt *

.1 ft bardO...y a,, ýHuman Pwf. - o.vc, 10 8Iooj ACA00*0004.L6 CACC6K*g F00w*0

& 8BA0*0 C,*.-I0 ý WWI00 00*0 Page 3.3

Functional Tool Analysis - Detail Sheet - Period 3 - End of 11RFO (5/17/98) through End of Mid-Cycle (5/10/99) 3.5 Audits, Surveillance, Evaluations Detail - End of 11RFO through Mid-Cycle v

,( ,,~~~~~~. x.......

• po9Pr9r8P

-otCIno981o1oNR 20007 A.I CAOP 112 COCQC RC-2(CR 08000)

OISEG ro 12 00RC2 Ro t (CR R- Caoa P 98-420)OA poak0e In.110ng h.yb.aaoo01a.todo.

i.nOv RC-2 ... e. 'S aunt CnRnn 307.071*ookn

,-t. o o.-*og .go, i lov dA

'id not do la3labtlily 3

E functional oversight (SOFI of Engineering, only do - -o ieIu d, i d, App. B audits CAA- Pro-r undul Ingluothe

• nsookodtorhihnod aoor.... nt. RRV efa *uasanRooot oodiA ooo to.a i t= M AD- -Eo taUle a

• 04,

---

i -- *Tr Aotsioorfita k raU od

Hed "i Roa

• • ir**dd .ta

. .. . . . t. . .. . .

Page 3.4

Functional Tool Analysis - Detail Sheet - Period 3 -End of 11iRFO (5/17/98) through End of Mid-Cycle (5/10/99) 3.6 Culture /Values Detail - End of 11RFO through Mid-Cycle ---

Causal Factor -- O. i Msuer wo ww ýluuchtoz 4ui inourardIAWd. Wusdequal*

Human Pad. -- b AouInWtatY 5

dad to 9 Ilo PCAOR 0-5. a

ý51n~u 'uedid1.19960 no 5/5 to Yen ulmoueead~lI.1MR PwEvu Page 3.5

Attachment 6 - Functional Tool Analysis Chart, End of Mid-Cycle Outage of 1999 through End of 12RFO Attachments

• 44 Analysis Report Cause Analysis Root Cause Report Attachments

• 44

Assumptlon 3: The Oversight RationalPerson badteo distinct opportnties to igoificentlyaite the ulootot owtcomc Functional Tool Analysis - Summary Sheet - Period 4 - End of Mid-Cycle (5/10199) & Assuncthroughproper oversight functions tht tho processes used by tho through 12RFO (5/18/00) Line RmionslPerson - sufficietly robust end ftecti-ly n and mitigate the oondidon.

todefet

b. Direct detection and notionto mitigato dhecondition.

Assumption 3.a Assumption 3.b 4.1 Corrective Action Process ... CA Identified Issues

.a-oolted with the CAP, Sufficient data wee however, the line did not avaiebbe hr the CAP for theC f adequal.. ~ oitln to picwot-, ,Olweqht ýdvbet tOý take affective corrective availablein CAto hovawe deteted the i, -gtno ewntnfoboWt ~ detectionof RPV' .rcorrw.-hfntncnIwunoO acotion. CA should ha issue.

~o~nCAP v 4Head cotrditlon, escalated the iasues until resolution wam aehieved.

4.2 Root Cause Analysis ---- Missed There were multipte If an a.en of the examples of veakneos in ROAs associated with the Opportunity the RCA procese. GA boric acid Issues wee ehould hew idertified the done itlikely could have FrAewc to \ nyoo0aeowo

ý 4,tadewA Event weaknecs and effected Improvementt led to identifyingthe Issue with the head.

Had the oversight For the majority of thie 4.3 Operating Experience Process Sfation Oversight functions organization Ideltified the period. ISEG wea were not successful in weak OEAP and affectad responsible for the tdentect ng or affecting change, it would have overall effeatb neess of contributed to the line the OEAP. Theycould

-Erpaitntio. lorGEAP in tame resolution of issues relafed to Identifyingboric acid and should heve corrosion of Rogctor corrosion and Afloy 600 identified boric acid 4' 'oatpor~itdeeorbesi ntb n Pressure Vessel Head End of PWSCC as Important to corrosion and Alloy 600 Mid-Cycle Outage of 1999 track as plant Issues. PWSCC as Important to track as plant issues.

through RFO-12 Summary Trend repors Sufficient information Is 4.4 Trend/I Analysis nYý weuldhoew ieluded now available to develop a meaningful trend related equipment issues and accomplishad merningful to boric acid including Red btend analysle. GA could Monitor fiter clogging, haw escalated the need for RCS leak rates still Oealonc a luseionfor this timesperiod: elevted, nd change in correotive action in the Mid-CyPni through RFO-12 - Sufficient t andrnalya oenditg CAC fouling frequency.

• .........~~.*pwo.o.ncc...ueckne i iffctireoron endopportuniy wereaveilahle program.

lo detectthe head degradation. TheCAPhr cot eupportint effective resiotutonof the Issues asevidenced byCR99i-1300on Red Monitor filtercocgking,&CR 00-0782.

4.5 Audits, Surveillance, Evaluations Stationvaluesdetracted frow theabilityof The audintturvellanoe Sufficient eqrertle wme Ac program had been theCAPandoversightlto beeffectifoe. structured with functional provided. auditors wer at". netere wthe proncessn trained on BA, however effactloe organization weoeldhavedetected

'a- erees however the proocte opportunlites were theheeddegradation hrthisthanperiod.

change was in its infancy missed In the oversight Oversight hadmultipleopporfonites in this period. Trend and analyaclof plant events and more tiec Is needed before the change could of BACCP as itapplied to

&conditions, as wll as oversight of the have meeasurble ofectCR-00-0782.

.usale haw effect.

BACCPduringRFO-12 ware clearly opportunities fordetection.

I Managementchanges, hich h wl-ec of the o-nrigh 4.6 Culture/ Values 7. occurred Immediatelypriortor organization-ra sely asined diningO tn period,'d change Win thehaation. Th. Vak of

-tetiorw of owhereigh. trendingA analyss of plant

. wte~aneo.

.' .w ro~etno Hoever, theoeý nochange Ir peon

• Inadequate, GA rye antud tooardlBAon RPVnead failedto performsoreo cld wiren W eeeennaL~.h Als, shedegreeof changeInvaue " o rlgint due toexce-cc concern of CAPand overeigirtawenot ha ALARAgoals. Them value.

~~~~~~~~~~ aoroee -~ -,,n-en

-r -wne~

-o-r---- -- erab* notkee, Heron ovrerightdid rot make charges may have contributedtotw RPV head dagrdeetn -otWing stffltleotto c dleoot.n delteocted Page 4.0

Functional Tool Analysis Chart - Detail Sheet - Period 4 - End of Mid-Cycle (5/10/99) through 12RFO (5/18/00) 4.1 Corrective Action Process Detail - End of Mid-Cycle through 12RFO Causal Facto( -. BAUr~et Esoasns BB-P.O.~ '

r

• CAP 1.wa

• o8.*aht" Page 4.1

Functional Tool Analysis Chart - Detail Sheet - Period 4 - End of Mid-Cycle (5/10/99) through 12RFO (5/18/00) 4.2 Root Cause Analysis Detail - End of Mid-Cycle through 12RFO Causal Factor - 0 k -SuP0wO l M.od. ft.004 Humn 0 0 9 b hbf tothem Human perf. -3 c0moacvtono 4.3 Operating Experience Process Detail - End of Mid-Cycle through 12RFO Causal Factor - (W. kWy 99.49.. 0,aw-an~&

0.4400.ER,90.&

onattilq HumanPerf. -- I, S0aY*t40.thy Wud*"a*to.a i & Wouit9.'y HOuSwanad 9(tac04) 37-CR99.2482 To OUTAG.01 silent on0 roGM*8009090 SOER use of 0E for 4,9.40,82. outage prep.

Page 4.2

Functional Tool Analysis Chart - Detail Sheet - Period 4 - End of Mid-Cycle (5/10/99) through 12RFO (5/18/00) 4.4 Trend / Analysis Detail - End of Mid-Cycle through 12RFO

• 000,0000o10209o10010.04200009,00 ti Causal Factor -p,

• PrIor / weO Th 00 06p1100 w00 OouO tlnu h mly.

No Hua - *ef 01.009..Oemed.01e 16000, sO42oh Hu3 O Inadeduo, ifoO l a0 0010110,2000000,000 u lnn lm03mcnaofo 00,,00 r N

00,01003000,10

" 00000/,01" 01/00,03 0 r s21 0 S-

.C sm.I3030 O 003. 12 10 09012,020Oh143 -AR 0

• ,01000000,010,03 CORAC-02 MosuelTl10 T17.0-9.000 o

Qo WN 00$

00 poy

,30*

0 n . 0 0ns0 O , 0 m r hmtrending y b3mo and 0 s-, 0e o

. d.. . ..... ... ..... . 010000000 00

... . . . 0analysis 001000000

' 1 .wm*m rn I,..... ......

00,0.. .1.0. 0.. ..

co f---, ndffer

• o"o0303.eO nw.03a

• Arl10,000 Ii0,o t0;200 900..0...0.

Page 4.3

'1*

0 i

a-I" j him IIi U 0 0 I I)

C) 00 C14 0

0) 8 Cf) 0

'a c

w 10 0

al)

C,)

a) 75 all 0

0 L-

Functional Tool Analysis Chart - Detail Sheet - Period 4 - End of Mid-Cycle (5/10/99) through 12RFO (5/18/00) 4.6 Culture / Values Detail - End of Mid-Cycle through 12RFO Causal Factor - mu aa t stactton.

-ltptO0C W0.~Ard not.aoqat~y paonaltatted Soo..ins fftcuotýt.0 0t.,~t mooto nlBLB gAA0,0.0 0 3,1 to tloA la.

Human Perf. -- tA~labty 314-Itrto NOC CA. 3 aO.MgttttItlaOast S3.00000-tOt0S 0 taetBA.l Ci*

Etu,0000.ttotlbt, IVBI.tsotOsoLeAa

.oOttuCV Oct00. Ft.tn ala ttt.0 0 nOWtkgtOSS 30,0 314- n-ACgh I

nC~,

Bat001ItOW. 00

,96 - Mgmt In iw:

82- MgmtconcemlterviewRDMawy Rlang-~ae

7.

Catatp' .1vy 1ýao

• sooty u.,Voaatooloo I

.... . . 0 t ndgt tn 503tAM0W303 O t,tWV P0St0 a 00 t 000.00,0 L 00010.0.

St)W T0100.0 OOl

• 4.tJgOtlV.00.0.

C.onOed OCoS Bassolo a4tooo w~lh t~~batgavflW

-..-.-.-bk:t0 BA.0wtati. hell 003,0d 0 00TgJo atowB Actld 0aoaaataoa..Ia.tty

• r tt3¶

-4 Page 4.5

Attachment 7 - Functional Tool Analysis Chart, Post 12RFO Attacfltnents . qb Root Cause Analysis Report Analysis Report Attachments 9 45

01 01 U,

0 0

CL 0

(D 0

In 5n 01 0D 0

atijI 0 0

0 0

.S\K

.2 I-uk Cf, AwpF

• \ v' I H A

grl CD MINI 0

a.n

• MHIIt 0

U90111: 0i 0.

0T1 CD 0

CD 0

< 0.

I U j II M

0 11

Ji I'I I

[! d k Ii H

t I

0 iI ti I sJ jIIi jf o

o I iii

'K: 4 I Ii

'U,,

iii

• 1 0) 0 0.

U, Ii o

G) 0 4

0) C" 0

= IL

• 0 IL a

o C.'

0 0.

0) -

(U

• j. a

.12 0o I- U

.11<

• :1!

0 0 0 0

o CI C C)

C,'

IL ts

01 CD 0

m

d. 0 00 0

co (A "

02 z~

0

• 0  :-i M.

CL 00 CDl

.0.

5n

-I 01 to to to to 0

to 0

(a CL

'1 0

0 0

0 In 0

0 0

I i 1 RfpI Ii jIh iiI 0) 0 C,'

• C) I

sia 0 IN CL "II FL "I.

fill fil 41illf~0

- Standard Interview Questions for Oversight Personnel

1. What are the factors for selection of activities for oversight? Who has input to go beyond Appendix B requirements; was internal/external OE used for input? How were these factors documented or communicated in Business Plans, site policies or management expectations?

2. Were the above QA expectations consistently applied through various organizational and management changes?

3. Was oversight activity performed by review of paperwork / historical records, or by actual field observation, or both? How does QA consider work not performed as a part of the evaluation process? Apply same to follow-up to corrective action? Do you feel you have the required technical skills when dealing with hardware / design issues?

4. Are all problems and issues required, by expectations or procedures, to be addressed by entry into the CAWh? What is QA's response to any failures to meet such expectations.

5. Regarding interactions with the Line Organization...

"* What levels of the line organization do/did you regularly interact with?

"* What is the frequency of that interaction?

"* Would you characterize the interaction as cooperative or contentious / adversarial? Were there any parts of the organization that were consistent in their response, regardless of the issues discussed?

"* Did the level of justification required result in any reluctance to document issues?

"* Do you encounter resistance to use of the CAP to address issues? If so, from whom?

" What level of intervention, if any, was necessary to resolve conflicts? If intervention by upper management was necessary, was that perceived as a failure of, or as part of, the normal conflict resolution processes?

"* Were the documented responses an accurate reflection of actions taken for a particular issue? Was this consistent, and was it dependent on organization or individuals? How did this result in changes to QA approach to oversight?

6. Over the last two cycles, how was QA viewed by JUMA, INPO and the NRC? How was this perception transmitted to the QA staff and the line organization? What effect'did the feedback have on both the QA approach and reception by the line organization?

7. Do collateral assignments affect organizational independence, either individually or organizationally?

8. Historically, what behaviors by QA evaluators have been rewarded by line and QA management?

9. To what degree are the perceptions of line management factored into performance reviews?

Did this affect willingness to aggressively pursue issues or problem resolution?

10. Have QA findings been modified / softened to accommodate management desires (either QA or line)? Talked out of writing CRs? If modified, was there a change to the intent or a re characterization without change to intent?

Root Cause Analysis Report Attachments

• 46

11. To what degree does management approval determine the outcome of Root Cause Evaluation? Is the evaluation of qualified root cause evaluators changed through the approval process, or are the professional opinions preserved in CAP documentation? Has this ever been evaluated by QA or by outside reviews of the CAP?

12. What is your confidence level in the integrity of the organization when challenged with significant technical or operational issues? Do you have any concerns with the ethics of the organization under these circumstances? What is the level of confidence that the management decisions will not be adversely affected by economic, schedule, or production pressures? If at all, to what degree is it necessary to modify your approach to oversight to compensate for these concerns?

13. Historically, what has been the role of QA (both organizationally and individually) in the restart readiness process? Was QA's role viewed as that of participant or evaluator? Was this role consistent with expectations of line management? Has this been effective in affecting restart priorities or assuring problems were resolved before plant startup?

14. Regarding the RC-2 event...

• Did QA provide in-process oversight of repair activities?

• If so, were any problems identified, documented, and followed-up by QA?

• Did NQA evaluate the effectiveness of Corrective Actions for the event?

• Was NQA included in the population of personnel given follow-up training regarding Boric Acid?

• Was the RC-2 event perceived by QA/line as a failure of oversight?

• Did the RC-2 experience change the perception of QA's role in plant activities? What changes to oversight approach and reception occurred, if any?

15. What changes do you see that are appropriate for QA?

16. Any additional comments of any type?

Attachments

• 47 Analysis Report Root Cause Analysis Report Attachments

• 47

Attachment 9 - List of Oversight Personnel Interviewed

• The following individual were in the D-B Oversight Organization at the time of the interview, or were in it during key times associated with events related to the RPV head corrosion issue.

Charles Ackerman, DAvis-Besse Timothy Chambers, Davis-Besse Edward Chimahusky, former Davis-Besse RCS System Engineer Clarence DeTray, Davis-Besse Priscilla Faris, Davis-Besse Eric Grindahl, Davis-Besse Mark Koziel, Davis-Besse Mark Levering, FENOC Collective Assessment Engineer Mark Pavlick, Davis-Besse James Ratchen, Davis-Besse John Reddington, Davis-Besse Quality Assessment Supervisor Charles Rider, Davis-Besse H. Kent Rhubright, Davis-Besse Henry Stevens, FENOC Manager Quality Assurance, Davis-Besse James Vetter, Davis-Besse Quality Assessment Supervisor William Wagner, Davis-Besse Attachments

• 48 Cause Analysis Root Cause Report Analysis Report Attachments

• 48

Attachment 10 - Standard Interview Questions for Selected Line Management

1. From your perspective as an organization subject to QA oversight, what are the factors for selection of activities for oversight? Who has input to go beyond Appendix B requirements; was internal/external OE used for input? How were these factors consistent with and communicated by Business Plans, sit6 policies or other advertised management expectations?

2. Were the above QA oversight expectations consistently applied through various organizational and management changes?

3. Are oversight activity performed by review of paperwork / historical records, or by actual field observation, or both? (discuss in terms of history and now) In your area is QA involved in evaluation of work not performed as a part of the oversight process? Apply same to follow-up to corrective action (both initiated by QA and those initiated by others)? Do you feel QA organization has the required technical skills or expertise when dealing with hardware / design issues?

4. Are all problems and issues required, by expectations or procedures, to be addressed by entry into the CAP? What is QA's response if they identify any.failures to meet such expectations.

5. Regarding interactions with your organization...

"* What levels of the line organization do/did QA personnel regularly interact with?

"* What is the frequency of that interaction?

" Would you characterize the interaction as cooperative or contentious / adversarial? Are there any parts of the QA organization that are/were consistent in their response, regardless of the issues discussed? Are there any parts of your organization that are/were consistent in their response, regardless of the issues discussed?

"* Do the level of justification presented by QA result in any reluctance to accept issues?

"* Do you encounter resistance to use of the CAP to address issues? If so, from whom?

" What level of intervention, if any, was necessary to resolve conflicts? If intervention by upper management was necessary, was that perceived as a failure of, or as part of, the normal conflict resolution processes?

"* Within your organization, are the documented responses to a CR an accurate reflection of actions taken for a particular issue? Is this consistent, or is it dependent on organization or individuals? Has this resulted in changes to QA approach to oversight of your organization?

6. Over the last two cycles, how was QA viewed by JUMA, INPO and the NRC? How was this perception transmitted to the line organization? What effect did the feedback have on both the oversight approach used by QA and reception by your organization?

7. Do collateral assignments affect QA's organizational independence, either individually or organizationally?

8. Historically, what behaviors by QA evaluators have been rewarded by line and QA management?

Root Cause Analysis Report Attachments

• 49

9. To what degree are the perceptions of line management factored into QA performance reviews? Did this affect willingness to aggressively pursue issues or problem resolution?

10. Have QA findings been modified / softened to accommodate management desires (either QA or line)? -Have QA assessors been talked out of writing CRs? If modified, was there a change to the intent or a re-characterization without change to intent?

11. To what degree does management'approval determine the outcome of Root Cause Evaluation? Is the evaluation of qualified root cause evaluators changed through the approval process, or are the professional opinions preserved in CAP documentation? Has this ever been evaluated by QA or by outside reviews of the CAP?

12. What is your confidence level in the integrity of the organization when challenged with significant technical or operational issues? Do you have any concerns with the ethics of the organization under these circumstances? What is the level of confidence that the management decisions will not be adversely affected by economic, schedule, or production pressures? It at all, to what degree is it necessary to modify your approach to oversight to compensate for these concerns?

13. Historically, what has been the role of QA (both organizationally and individually) in the restart readiness process? Was QA's role viewed as that of participant or evaluator? Was this role consistent with expectations of line management? Has this beehi effective in affecting restart priorities or assuring problems were resolved before plant startup?

14. Regarding the RC-2 event...

"* Did QA provide in-process oversight of repair activities?

"* If so, were any problems identified, documented, and followed-up by QA?

"* Did NQA evaluate the effectiveness of Corrective Actions for the event?

"* Was your organization included in the population of personnel given follow-up training regarding Boric Acid?

"* Was the RC-2 event perceived by as a failure of QA oversight?

"* Did the RC-2 experience change the perception of QA's role in plant activities? What changes to oversight approach and reception occurred, if any?

15. What changes do you see that are appropriate for QA?

16. Any additional comments of any type?

Attachments

• U Cause Analysis Root Cause Report Analysis Report Attachments

• 50

Attachment 11 - List of Selected Line Management Personnel Interviewed The following individuals were in line management positions and interfaced with the Oversight Organization. They were interviewed for their perspective on the functioning of the Oversight Organization during key events related to the RPV headcorrosion issue. Mr. Zellers was with the NRC at Davis-Besse during key times.

Howard Bergendah!, former Vice-President, Davis-Besse David Geisen, Davis-Besse Design Basis Engineering Manager Mark Haskins, Davis-Besse Supervisor Self-Evaluation Program William Mugge, Davis-Besse Training Manager Robert Pell, forrmer Davis-Besse Operations Manager Michael Stevens, former Davis-Besse Maintenance Manager Kevin Zellers, Davis-Besse Engineer, Design Basis Engineering Attachments

• 51 Cause Analysis Root Cause Report Analysis Report Attachments 9 51

Attachment 12 - List of Personnel Interviewed by the NQA Team Guy Campbell, former Davis-Besse Vice President Edward Chimahusky, former Davis-Besse RCS System Engineer Robert Coad, former Davis-Besse Operations and Radiation Protection Manger Scott Coakley, Davis-Besse Outage Director Robert Donnellon, former Davis-Besse Director Engineering and Services David Eshelman, former Davis-Besse Plant Engineering Manager James Freels, former Davis-Besse Licensing. Manager David Geisen, Davis-Besse Design Basis Engineering Manager Prasoon Goyal, Davis-Besse B&WOG Material Committee Representative Daniel Haley, former Davis-Besse RCS System Engineer John Hartigan, Davis-Besse Mechanical Engineering Robert Hovland~former Davis-Besse Radiation Monitor System Engineer John Johnson, former Davis-Besse Corrective Action Program Lead James Lash, former Davis-Besse Plant Manager Peter Mainhardt, performed Davis-Besse Reactor Vessel Head Inspections Eugene Matranga, Davis-Besse System Engineering Glenn McIntyre, former Davis-Besse Mechanical Systems Engineer Kevin McLain, former Davis-Besse Reactor Operator Steven Moffitt, Davis-Besse Director Technical Services John O'Neill, former Davis-Besse PCAQRB Chairman Randy Patrick, Davis-Besse Shift Engineer Terry Ploeger, Davis-Besse Shift Manager Michael Roder, former Davis-Besse Shift Manager Joseph W. Rogers, Davis-Besse Outage Director Dennis Schreiner, former Davis-Besse Independent Safety Engineering Supervisor Andrew Siemaszko, current Davis-Besse RCS System Engineer Rebecca Slyker, Davis-Besse Regulatory Affairs Dennis Snyder, Davis-Besse Maintenance Henry Stevens, FENOC Manager Quality Assurance Lou Storz, former Davis-Besse Vice President Nuclear Joseph Sturdavant, Davis-Besse Regulatory Affairs Theo Swim, Davis-Besse Design Basis Engineering Andrew Wilson, Davis-Besse Maintenance John Wood, former FENOC Vice President Engineering Services Hoot Cause Analysis Report Attachments

• 52 Root Cause Analysis Report Attachments

• 52

Enclosure 2 Root Cause Analysis of Operations

FirstEnergy, DAVIS-BESSE NUCLEAR POWER STATION Root Cause tAnlyids Lor Lack of Operations Centrality in Maintaining, Assuring, and Communicafing the Operalional Safety Focus of Davis-Besse and Lack ofAccountability of Other Groups to Operations inFulfilling that Role CR 02-2581 DATE: November 22, 2002 Prepared by:

Valerie E. arnes, PhD Root Cause Lead Davis-Besse Spo nsor: " ý, ýz J..

Michael J. Roder Operations Manager S~Randel Fast Approved by:

S4Plant Manager Approved by:

perating Officer

Table of Contents Title Page 1.0 Executive Summary 4 1.1 Overall Conclusions 4 1.2 Problem Statement 4 1.3 Event Narrative 4 1.4 Data Analysis 5 1.5 Root Cause Determination 5 1.6 Extent of Condition 5 1.7 Corrective Actions 6 1.8 Experience Review 9 2.0 Problem Statement 11 2.1 Reason for the Investigation 11 2.2 Consequences of the Condition 11 2.3 Actions Already Taken 12 3.0 Event Narrative 13 3.1 Background 13 3.2 Event Narrative 14 4.0 Data Collection and Analysis 17 4.1 Input 17 4.2 Methodology 18 4.3 Results 19 5.0 Root Cause Determination 37 5.1 Root Cause 37 5.2 Contributing Causes 37 5.3 Related Observations 38 6.0 Extent of Condition 40 7.0 Corrective Actions 41 7.1 Corrective Actions for the Root Cause 41 7.2 Corrective Actions for Contributing Causes 42 7.3 Corrective Actions and Proposed Enhancements for Observations 44 8.0 Experience Review 46 8.1 Recent Assessments of Davis-Besse Related to Operations 46 8.2 Related Industry Experience 50 8.3 Differences between Previous and Proposed Actions 54 Root Cause Analysis Report I I I

Table of Contents Page Title Title Page 9.0 Root Cause Analysis Team 55 10.0 References 57 10.1 Davis-Besse References 57 10.2 NRC References 59 10.3 INPO References 60 10.4 Other References 61 Root Cause Analysis Report ii

Acronyms ALARA As Low As Reasonably Achievable CAC Containment Air Coolers CARB Corrective Action Review Board CR Condition Report FENOC FirstEnergy Nuclear Operating Company GL NRC Generic Letter HPEP Human Performance Evaluation Process IN NRC Information Notice INPO Institute of Nuclear Power Operations NRC Nuclear Regulatory Commission RFO Refueling Outage RO Reactor Operator RPV Reactor Pressure Vessel RP Radiation Protection SRB Station Review Board SRO Senior Reactor Operator SVP Site Vice President VT Visual Examination WANO World Association of Nuclear Operators Root Cause Analysis Report Ili

1.0 Executive Summary 1.1 Overall Conclusions Licensed operators' leadership role in station activities at Davis-Besse began to erode in the 1990's and reached its lowest point in the three years preceding the discovery of the Reactor Pressure Vessel (RPV) head degradation in March, 2002.

Over the past decade, Operations personnel believe that the "keys to the plant" were taken away from them. The Team concurs with this perception. It appears to the Team that management decisions with unintended consequences and, in some cases, management behavior that demonstrated an active disregard for the authorities and responsibilities of licensed personnel over a period of years, diminished Operations' ability and willingness to lead the site in assuring safe operations. The Team also concludes that the most significant barriers to Operations' leadership have been eliminated and that the corrective actions underway to strengthen Operations' role at the site are likely to be effective.

1.2 Problem Statement This report may be different from typical analyses of events at Davis-Besse because it focuses on the underlying reasons for an adverse organizational condition that developed over a period of years, rather than a single event. The purpose of this effort was to identify the root and contributing causes of the previous lack of Operations' centrality in maintaining, assuring, and communicating the operational safety focus of Davis-Besse and the lack of accountability of other groups to Operations in fulfilling that leadership role.

1.3 Event Narrative The Management and Human Performance Root Cause Analysis Report (Condition Report [CRI 2002-0891, August 21, 2002), which addressed organizational factors that contributed to the RPV head degradation, stated:

The Davis-Besse Plant had a significant outage in 1985. Since that time the plant has been a top performer, but starting in the mid-1990s a flattening or decrease in performance can be seen. The managers brought in during the 1980s event are gone and many of the managers developed during that period left the company and are now in key positions throughout the industry. Several of the plant evaluations both in-house and by outside organizations have noted this issue over the past three years. Actions were taken to improve this performance but not as promptly as needed.,

Root Cause Analysis Report 4 I I

The Management and Human Performance Root Cause Analysis Team noted that a significant barrier to declining plant performance should be licensed operators who promptly identify degrading conditions and aggressively pursue problem resolutions.

In the case of the RPV head degradation, however, that Team found that the Operations Department at Davis-Besse was not provided with important information about conditions on the head and missed a number of opportunities to demonstrate leadership in resolving the developing problem.

1.4 Data Analysis This Root Cause Analysis Team (Team) used change analysis, barrier analysis, and the Human Performance Evaluation Process (HPEP) to determine the root and contributing causes of the apparent absence of Operations leadership in assuring the safe operation of Davis-Besse Nuclear Power Station.

1.5 Root Cause Determination Based upon its analysis, the Team identified one root cause and three contributing causes for the erosion of Operations' leadership in station activities.

Root Cause

1. Senior management support for Operations' leadership role in assuring plant safety was lacking.

Contributing Causes

1. Staffing was inadequate to perform the tasks assigned.

2. Senior management failed to ensure that regulatory expectations for licensed personnel were effectively communicated and reinforced.

3. Senior management failed to assure that a safety conscious work environment was established and maintained in Operations.

1.6 Extent of Condition Based upon the information considered by this Root Cause Analysis Team and the information documented in the Management and Human Performance Root Cause Report, the Team believes that other station departments were also adversely affected by some of the same factors identified in this analysis. In response to the Management and Human Performance Root Cause Analysis Report, a Management and Human Performance Excellence Plan was developed and is being implemented.

The Management and Human Performance Excellence Plan includes a series of reviews of selected station organizations that include, for example, checks of whether there are clear lines of authority and responsibility within the organization; whether Root Cause Analysis Report 5

staffing levels and resources are sufficient to handle Aissigned responsibilities; whether individuals have a clear description of their assigned responsibilities; and whether interfaces with other organizations are clearly defined. In addition, a number of actions are being implemented to strengthen the safety conscious work environment across the site. The Team concludes that these efforts should identify needed corrective actions in other station organizations for adverse conditions that were created by the same causal factors that affected Operations.

1.7 Corrective Actions The key corrective actions are described below, arranged by causal factor:

1.7.1 Senior management support for ODerations' leadership role In assuring plant safety was lacking.

a. Extensive changes have been made in the officers, directors, and managers responsible for Operations, including changes in the Site Vice President, Plant Manager, Operations Manager, Operations Superintendent, and Operations Support Superintendent. These individuals value strong Operations' leadership.

b. Senior management is demonstrating support for Operations' leadership role by being visible and active in Operations' activities, such as shift turnover meetings, by appropriately mentoring and coaching Operations' personnel in resuming the leadership role, and by ensuring that Operations' priorities are addressed and supported in station decision-making. The Nuclear Quality Assurance organization will periodically assess the implementation and success of these activities.

c. A declaration from the chief executives will be issued and communicated to site personnel delineating Operations' leadership role.

d. The Operations leadership team will disseminate the results of this Root Cause Analysis and the corrective actions to other station managers, will ensure that other station organizations dedicate the time required to internalize the impact of Operations resuming the leadership role on their activities, and will resolve any issues identified in implementing the change.

e. Senior management will assure that no uneasiness remains among Operations personnel regarding the station's ability to operate safely prior to restart. Shift Managers will be charged with eliciting any outstanding safety concerns from their crews and for ensuring that the concerns are resolved. Davis-Besse will not restart until each Shift Manager is willing to state that he and his crew know that the plant is ready to restart, that Operations has regained and is performing the site leadership role, andcihat the plant will operate safely.

Root Cause Analysis Report 6 I I I I

1.7.2 Staffing was Inadequate to perform the tasks assigned.

Corrective Actions:

a. Continue hiring new personnel to be trained as equipment operators and continue training of RO and SRO candidates. Continue implementing current plans for additional licensing classes to replenish and maintain a sufficient number of licensed personnel for the tasks assigned, as well as to populate other departments with personnel who maintain active licenses.

b. Analyze the tasks currently assigned to Operations. Identify additional activities that Operations must perform to continue re-establishing and to maintain leadership. Determine the number of personnel and the qualifications required to perform the activities identified. Develop and implement short-term compensatory measures for staffing shortfalls that are identified.

c. Address Operations' compensation, as necessary, to ensure retention of current staff. Improve the station's competitive position in attracting desirable applicants. Continue current activities to develop and implement professional development plans for Operations personnel to ensure that career paths are identified and that future site leaders will be available and prepared to assume leadership roles.

d. Implement corrective actions for staffing needs identified in other station organizations to ensure staff capabilities exist to support Operations' priorities.

1.7.3 Senior management failed to ensure that regulatory expectations for licensed personnel were effectively communicated and reinforced.

a. Prior to restart, Operations and management personnel from other station organizations will receive corporate training regarding the roles, responsibilities and authorities of licensed personnel. This training course will be added to the core continuing training program for Operations and management personnel.

b. The Site Vice President will continue to make himself available to the Duty Shift Manager to assist in ensuring that personnel in other station organizations understand his expectation that they are accountable to the Duty Shift Manager and to Operations personnel and must support Operations' leadership role.

c. Management will ensure that an invitation is extended to NRC representatives to address newly licensed or upgraded operators when they are awarded their licenses, and to address Operations Department Root Cause Analysis Report 7 I I

personnel periodically to communicate and reinforce NRC expectations, as appropriate.

d. An Operations Standards and Expectations document has been issued to address, in detail, expectations, job standards, and responsibilities of Operations Department personnel. Knowledge of these standards and expectations will be reinforced by training and testing prior to restart.

e. A memo signed at the highest level defining the Shift Manager's role, responsibilities and authorities will be issued and conspicuously posted in selected areas throughout the site. This memo will be revised and reissued on an annual basis.

f. The Operations Leadership Plan will be reviewed and approved by senior management.

g. As part of the Management and Human Performance Excellence Plan, policies, procedures, program and job descriptions, and organizational interfaces are being reviewed for consistency with management expectations. As part of these reviews, opportunities to strengthen Operations' leadership role will be identified and incorporated. For example,

i. Requirements will be added to the corrective action process to ensure that Operations' concerns are adequately addressed in the prioritization, scheduling and resolution of condition reports.

ii. Operations' involvement in station decision-making processes will be strengthened. Operations' representation will be required at Management Review Board, Corrective Action Review Board, Station Review Board, and Project Review Committee meetings, and that Operations input will be sought in other station decision making processes, as appropriate. These important decision making meetings will not occur unless a designated Operations representative is present.

h. Licensed personnel will fully commit to resuming the leadership role.

1.7.4 Senior management failed to assure that a safety conscious work environment was established and maintained In Operations.

a. The Operations Standards and Expectations document will address the chilling effect in Operations by including expectations for Operations personnel to raise any operational concerns. It also contains the requirement for Operations personnel to demonstrate leadership in resolving concerns by continuing to escalate them through their management chain up to and including the President of FENOC until Root Cause Analysis Report 8 I I I I

resolution is obtained. Davis- Besse and FENOC senior management expect Operations personnel to inform the NRC of their concerns, if management does not address the concerns to their satisfaction.

b. Licensed operators will be delegated management authority for addressing and resolving safety concerns that are identified to them by other station personnel.

c. Consistent with their leadership role, Operations personnel at all levels will be given training in maintaining a safety conscious work environment to ensure that their leadership and oversight of station activities performed by personnel in other departments is conducted in accordance with management expectations.

d. Individuals appointed to the open Site Vice President and Operations Superintendent positions will be thoroughly screened prior to appointment to ensure that their management styles will support a safety conscious work environment in Operations.

e. Operations personnel and managers in all station organizations will comply with senior management requirements and NRC expectations for ensuring that a safety conscious work environment is maintained.

210 of 250 site supervisors have recently completed safety conscious work environment training for supervisors, and the remainder will complete training by December 4, 2002.

f. Reports from site-wide surveys and assessments of the safety conscious work environment in Operations will be provided to the Plant and Operations Managers, and any weaknesses identified will be promptly addressed and resolved.

g. Davis-Besse will complete implementation of the Safety Conscious Work Environment Action Plan as part of the Management and Human Performance Improvement Plan.

1.8 Experience Review The results of the experience review indicated that evidence of the decline in Operations' leadership role was identified prior to this Root Cause Analysis and that similar events have been identified across the nuclear industry. Some corrective actions were developed and implemented to address specific symptoms of the decline of Operations' leadership at the site, but it does not appear to the Team that the potential consequences for nuclear safety of the management and organizational issues occurring in Operations were previously considered. Changes in FENOC and Davis-Besse managers, as well as the managers within Operations, have been made and the new management team brings an operational focus and values a safety conscious work environment. In addition, an Operations Leadership Plan has been Root Cause Analysis Report 9 i I I I

developed and is being implemented that addresses the steps necessary to strengthen Operations as well as Operations' relationships with other site organizations. These actions are substantially broader and more comprehensive than the corrective actions taken for CR 01-2989 and previous assessments of Operations. Davis-Besse should perform reviews to ensure that the corrective actions specified in this report are effective.

Root Cause Analysis Report 10 I I

2.0 Problem Statement 2.1 Reason for the Investigation As documented in Condition Report (CR) 2002-0891, through-wall cracking was identified during thirteenth refueling outage (13RFO) in some of the CRDM nozzles on the Davis-Besse reactor pressure vessel (RPV) head. Further investigation of this condition in March of 2002 led to the identification of significant degradation of the RPV head base metal at nozzle 3 and additional corrosion at nozzle 2.

In April of 2002, a Root Cause Analysis Report was issued on the technical causes of the degradation of the Davis-Besse RPV head (CR 2002-0891 Technical Root Cause Analysis Report). That Report identified a number of management issues that were contributing causes to the degradation, and concluded that station personnel had failed to identify corrosion of the base metal of the RPV head over a period of years despite multiple opportunities to do so.

In August of 2002, a Management and Human Performance Root Cause Analysis Report was issued that stated that the Davis-Besse Operations Department did not take an active role in advocating actions to improve plant conditions. Condition Report 2002-02581, originated in June of 2002 by a member of the Management and Human Performance Root Cause Analysis Team, listed six occasions on which Operations personnel missed opportunities to demonstrate leadership in preventing the significant head degradation. The author of the CR stated, "This has raised the question of the centrality of Operations in maintaining, assuring, and communicating the operational safety focus for the site, as well as the accountability of other groups to Operations in fulfilling that role."

Although the information in this Report provides insights regarding the factors that caused or contributed to Operations' failure to prevent the RPV head degradation, the purpose of this Root Cause investigation and analysis was broader. The purpose of the current effort was to identify the root and contributing causes of the apparent absence of Operations leadership in assuring the safe operation of Davis-Besse Nuclear Power Station.

2.2 Consequences of the Condition Licensed operators hold a unique position within the commercial nuclear power industry. They are employees of the corporations that operate the nuclear power plants, but are also the only individuals within the industry who are licensed by the U.S. Nuclear Regulatory Commission (NRC). As individual licensees of the NRC, operators hold the special trust and confidence of the public. If the licensed personnel at a plant do not demonstrate leadership in maintaining nuclear safety as the highest operational priority, public health and safety are at increased risk.

Root Cause Analysis Report 11 I I I I

2.3 Actions Already Taken The Operations Department has developed and is implementing an Operations Leadership Plan. Accomplishments to-date include:

The administrative duties assigned to Shift Managers have been reduced, so that they are more available to provide leadership and oversight of activities in the field. In addition, the Shift Managers' work area was moved from the work control center counter area into an office.

An independent assessment of the potential and leadership skills of management personnel within the Operations Department has been completed to provide the basis for professional development plans.

Licensed SROs are assigned to the Fix-It-Now Team and to the Radiation Protection Department to ensure that an operational perspective is brought to those areas.

Benchmarking visits to three other stations reputed to have a strong operational focus have been completed and the results compiled for future improvements in Operations' work activities.

The Institute of Nuclear Power Operations (INPO) Operations First Line Supervisor course has been completed by the Unit Shift Supervisors and Shift Engineers to strengthen their leadership skills and enhance teamwork. All but two Shift Managers have attended the INPO Shift Managers course and the remaining two will attend within the next six months.

Aggressive hiring plans for Operations have been implemented and licensing classes have continued, despite outage demands, to ensure that sufficient and appropriately qualified personnel are available to sustain Operations' leadership role and to provide future leaders with a strong operational focus to other site organizations.

An Operations Standards and Expectations Directive has been completed.

Root Cause Analysis Report 12

3.0 Event Narrative 3.1 Background Davis-Besse achieved initial criticality on August 12, 1977. In its 25 years of operation, the organization has demonstrated cyclical safety performance.

Following the June 9, 1985 auxiliary feedwater event and extended shutdown, Davis-Besse began to improve its performance significantly. The managers were brought onto the site following the event focused on repairing equipment who that had been allowed to degrade or had not worked properly since the plant began operations. A strong Operations department was emphasized by senior management, and the number of licensed personnel was reported to have increased from 45 in 1985 to 100 by 1990.

In 1991, the plant manager published the following memorandum addressed to the Davis-Besse Operations Shift Supervisors:

Nuclear generating facilities have the potential to significantly impact the health and safety of the public. This potential impact places a special burden and responsibility on those who manage and command operations at the Davis-Besse Nuclear Power Station.

The first line of defense in protecting and assuring the health and safety of the public and the safety of personnel within the plant is the safe, conservative operation of the plant.

The Duty Shift Supervisor (SS) has the primary management responsibility, until properly relieved, for the safe, conservative operation of the plant. Accordingly, the Duty SS is directly charged with both the responsibility and the command authority of all shift operations, maintenance activities, and implementation of radiological controls under normal and abnormal conditions. Both the supervisor coming on shift and the supervisor being relieved shall make certain they review, convey, and understand plant status and on-going activities and that the activities are deemed to be in accordance with safety requirements.

You must constantly maintain the broadest perspective of operational conditions potentially affecting the general public, plant personnel, and the safety of the plant. Maintenance of this broad perspective shall be your highest priority.

Root Cause Analysis Report 13 I

I

Interviewees who worked at the plant before the 1985 event report that, in the period from approximately 1986 through 1990, the professionalism and expertise of Operations Department personnel increased. Before the 1985 event, operators were perceived as "arrogant," making it difficult to communicate with them to solve problems. An influx of new personnel who brought new attitudes, improvements to operator training, and a change in management expectations regarding professionalism allowed Operations to become more effective leaders at the site.

The increased number of licensed operators available also allowed them to be more involved in on-going work activities at the plant and enhanced the site's safety focus.

The majority of operators interviewed who were at the plant at the time also report clear senior management support for Operations' central role in decision-making.

It was the unanimous perception of the interviewees, both within and outside of the Operations Department, that Operations' leadership role at Davis-Besse began to erode in the early 1990's and reached its lowest point in the three years preceding the discovery of the RPV head degradation in March, 2002. The interviewees' common belief is that between the years of 1993 to the present, the "keys to the plant" were taken away from them.

3.2 Event Narrative The originator of CR 2002-02581 listed the following events and conditions as indicating a lack of safety leadership from Operations at Davis-Besse.

• F-The initial review of a condition report (PCAQR 96-0551) was accepted by Operations as an administrative issue, although the wording of the PCAQR indicated potential degradation mechanisms on the RPV head.

• Operations had minimal involvement in the identification of the leakage source responsible for iron oxide deposits documented in CR 1999-1300. Additionally, there was a lack of accountability to assure this source was definitively identified.

0 Operations demonstrated tolerance for long-standing issues, such as:

the need to clean Containment Air Coolers (CACs), boric acid dispersion on equipment in containment, and high unidentified leakage.

• Operations demonstrated a willingness to accept RC-2 leakage in re-starting the plant following 11 RFO.

• Engineering did not communicate trending parameters regarding unidentified leakage and RP (gaseous activity) to operators to ensure their awareness.

Root Cause Analysis Report 14 I I I I

A mode restraint was removed prior to the performance of work (CR 2000-1037) with words that the designated work "will be" performed.

This faith was unrealized as the expectation of the mode restraint was only fulfilled in part.

As indicated in CR 2002-02581, the Operations personnel interviewed were not aware of some of the key information related to the RPV head degradation until that information was first presented to them in the recent Case Study sessions, conducted during the weeks of October 11 through 24, 2002. For example, operators did not see videotapes that had been made of boric acid found on the RPV head during 12RFO in 2000 and were unaware that it was a dark red, rusty color.

Those interviewed were also not aware of the November, 1999, Sargent & Lundy report suggesting there was a steam leak high in containment that was causing boric acid corrosion and was responsible for the iron oxide mixed in with the boric acid deposits.

They were also unaware that the cleaning of the RPV head during past outages was not fully successful and that significant boric acid deposits were left on the RPV head.

Responsibility for the Boric Acid Corrosion Control program was assigned to Engineering and Operations did not provide oversight of the job. During 12RFO, a

Shift Manager who wanted to view the progress on the job was prevented from doing so by Operations management on the basis of dose concerns.

Observations made by radiation protection (RP) technicians who entered containment for the repeated cleanings of the Containment Air Coolers (CACs) and the frequent filter changes for the radiation monitors were routinely provided to Operations, but Operations personnel did not possess the technical information necessary to interpret correctly the changing conditions in containment. They received some limited training in the potential consequences of boric acid corrosion had following a 1998 event in which two carbon steel nuts had corroded away on the RC-2 spray valve. Some may have known from previous industry events at Salem and Turkey Point that crystalline boric acid from CRDM nozzle leakage left on the carbon steel RPV head would cause limited corrosion. However, the corrosion mechanism that was generally known at that point from industry operating experience was not the same aggressive attack that occurred at Davis Besse. Known corrosion rates were very slow and should not have resulted in wastage to this degree.

More importantly, operators were repeatedly assured by engineering and senior management personnel that the boric acid on the RVP head could not result significant corrosion, because the temperatures were so high that the boric in acid would remain in a non-reactive, crystalline form.

Root Cause Analysis Report 15 I I I I

If the operators had understood that aggressive corrosion on the RVP head was occurring, they were required by technical specifications to shutdown, cooldown and depressurize the reactor. However, they were told, and believed, that the boric acid was being removed from the RPV head in each refueling outage. When asked why Operations did not require that the plant be shutdown due to unidentified leakage in containment that was below technical specification limits and the presence of boric acid on the RPV head that they did not believe represented an immediate threat, a Shift Manager summarized his understanding of containment conditions at the time by replying, "Based on what?"

Root Cause Analysis Report 16 I I I I

4.0 Data Collection and Analysis 4.1 Input Leadership is a concept that exists in the human mind and arises from social interaction. Although organizational researchers have attempted to measure leadership objectively over the years, it is fundamentally a subjective phenomenon.

Understanding it requires access to the perceptions, interpretations and beliefs of those who are leading and are led. Subjective assessments associated with leadership, or any social interaction, may be based on inaccurate, incomplete, or distorted information. However, it the perceptions and interpretations of events that often best explain behavior.

Therefore, the primary information-gathering method used by the Root Cause Analysis Team was interviews. Many Davis-Besse personnel have worked at the station since the 1970's and three of the individuals interviewed were involved in plant construction and start-up. The long tenure of many of those interviewed made change analysis the root cause analysis technique of choice.

The passage of time distorts memory, however. Within 48 hours5.555556e-4 days <br />0.0133 hours <br />7.936508e-5 weeks <br />1.8264e-5 months <br /> of an event, the accuracy of an eyewitness' memories for the event decreases by 50%. As time passes, people seek to explain and interpret their own behavior in an event, and the explanations they create may unintentionally come to replace or distort the information they stored in memory. Further, people talk with others about significant events, and so one person's memories may become "contaminated" by others' over time. Because this investigation was not initiated until long after many of the events described here occurred, it is likely that some of the interview data collected are questionable.

A set of discussion topics was developed to guide the formal interviews, but the conversations were wide-ranging. The interviewer began by describing the statements in the Management and Human Performance Root Cause Analysis Report regarding the apparent erosion of Operations' leadership role at the site and asked the interviewees whether they agreed or disagreed with the statements. Given that all of the interviewees agreed that erosion had occurred, they were then asked to describe the bases for their conclusion and to provide specific examples of decreased Operations' leadership. They were also asked for their views on the reasons that the erosion had occurred. For each of the examples provided, interviewees were also asked whether the situation had changed and how it had changed. Therefore, when reading this Report, it is important to note that the "complaints" offered by the interviewees were intentionally sought by the Team.

Root Cause Analysis Report 17 I I I I

Interviewing began with equipment operators and proceeded vertically within Operations, with a minimum of 30% of the personnel in each job category contacted.

To the extent possible, information provided by an interviewee about another person was verified by contacting the other person. Reports of activities that would have involved several individuals, such as briefings and meetings, were verified by contacting at least one other individual who had been present. When available, copies of documentation related to the interview data were obtained or reviewed to validate the interviewees' recollections. The documentation included memoranda, e-mail messages, meeting minutes, standing orders, procedures, and CRs. In addition, a small sample of individuals from departments outside of Operations was interviewed. In the interests of readability, the percentage of interviewees who endorsed or disagreed with a particular interview item is not presented in this report.

Instead, except where noted, the information presented represents a summary of the interviewees' views and recollections, illustrated with some of the anecdotes provided.

Several other sources of information were also used. The Root Cause Analysis Team observed (1) meetings in which Operations personnel had the opportunity to demonstrate leadership, (2) crews in training perform in the simulator, and (3) turnover activities and conduct of operations in the control room. The Team also relied on data gathered by the CR 2002-0891 Management and Human Performance Root Cause Analysis Team.

4.2 Methodology The Team used the following methods to guide data collection and perform the root cause analysis:

• Change Analysis

• Barrier Analysis

• Human Performance Evaluation Process Cause Tree and Modules 4.2.1 Change Analysis Change analysis consists of comparing an event-free, prior, or ideal situation to the situation existing at the time an event (or adverse condition) is identified. The differences between the prior situation and the situation at the time of the event are then evaluated to determine their effect on the event (or adverse condition). The prior condition used for comparison in this analysis was the early 1990's, when Operations held a site leadership role. The current state of Operations' leadership, as described in CR 02-02581 and by interviewees, was the adverse condition analyzed.

Root Cause Analysis Report 18 I I I I

4.2.2 Barrier Analysis In a barrier analysis, management systems and physical barriers that could protect target from hazards are identified and analyzed. An evaluation is then conducted a to determine whether the barriers did not exist for the event or adverse condition in question, or if they did exist, why they failed to prevent it.

4.2.3 Human Performance Evaluation Process; The Human Performance Evaluation Process (HPEP; NUREG/CR-6751, April 5, 2002) was developed for the NRC and is intended for use by NRC inspectors to assist in the review of nuclear utility licensees' problem identification and resolution processes related to human errors. The HPEP can also be adapted for use in conducting investigations and causal analyses of human performance problems.

The HPEP Cause Tree may be used as a screening tool for identifying the range of possible causes for a human performance problem or as a checklist at the end of an investigation to ensure the breadth of issues considered was adequate. The Cause Modules discuss typical causes of human errors in nuclear licensee facilities and provide examples, based upon the research literature and industry experience.

As information was gathered, the Team used the HPEP Cause Tree and Modules to identify promising lines of inquiry and to rule out others. The following Modules were determined to be relevant to this Root Cause investigation:

0 Module 10: Attention and Motivation (Motivation only) 0 Module 13: Staffing 0 Module 14: Supervision

• Module 18: Coordination and Control 4.3 Results 4.3.1 Chanae Analysis Results A significant amount of change occurred at Davis-Besse in the past decade.

Interviewees perceived that there was a decreasing corporate willingness to invest in the staff and material condition of Davis-Besse during the 1990's. The interviewees believed that the corporate owners of Davis-Besse suffered financial pressure in the 1990's. Various explanations were offered, including the expectation that deregulation would cause increased economic competition, the assumption of significant debt, or several poor investments. Others reported being told that, because of the limited time left on the license to operate the plant, which limits the return on any investment, it was important to minimize capital improvements, and to maintain operations and maintenance costs as low as possible, in order to avoid plant shutdown.

Root Cause Analysis Report 19 I I I I

The Management and Human Performance Root Cause theport for CR 2002-0891 suggested that the perceived unwillingness to invest in Davis-Besse may have been due to individual corporate and site management's response to monetary incentive rewards based primarily on production, as follows:

The FENOC management monetary incentive program rewards production more than safety at senior levels of the organization. For example, the Nuclear Incentive Compensation Plan for 2002 provides for incentive compensation for various factors related to safety and production, and FENOC officers and plant directors are to receive most of their incentive compensation based upon production. This supports misalignment of the organizational priorities, and inhibits the transition of the organization to a safety-first philosophy.

Whether the cost-control measures implemented in the past decade represent a corporate response to financial pressures, the actions of managers motivated by financial incentives, or other factors, these measures appear to have had a significant adverse impact on Operations' leadership role, as follows.

4.3.1.1 Staffing Like many nuclear utilities at the time, a staffing study was performed for Davis-Besse in the early 1990's and the results were described as showing that the plant was over-staffed compared to other best-performing stations across the industry. Downsizing began and continued throughout the decade through layoffs, early retirement programs, and attrition. The number of permanent employees decreased from 1,134 at the beginning of 1993 to 717 at the end of 1997. As of January 1, 2002, the number of full-time, permanent employees at the site was 677.

One engineer interviewed stated that, as a result of the staffing study, engineers heard that layoffs were coming and left the site "in droves." He reported that so many left voluntarily that it was unnecessary to lay off anyone in Engineering. He also indicated that Engineering lost it's most qualified and dedicated people first.

Operators commented that the substantial loss of experienced engineers meant that in-depth knowledge and understanding of plant systems and design was also lost.

After the exodus, interviewees reported instances of having to locate past employees in order to obtain needed engineering information. In addition, system engineers were often assigned responsibility for multiple systems, whereas engineers in the past were able to focus on a single system. The consequence for Operations was not only that engineering responses to requests for analyses and assistance were delayed, but also that the quality of the information they received declined. Initially, when senior reactor operators (SROs) would request more thorough or detailed information, they found that some engineers were willing to provide it. Others would take the issue to their management, who would then discuss it with Operations management, and the SROs sometimes found themselves told to accept the information provided.

Root Cause Analysis Report 20 I I I I

Interviewees reported that, in response to the staffing study, senior management determined that Davis-Besse had more than enough licensed personnel and stopped hiring new non-licensed operators to be trained for eventual licensing. No new hires of non-licensed operators were reported to have occurred for a period of six years.

Attrition also affected Operations. Towards the end of the 1990's, the number of licensed operators fell below the 1985 head-count of 45. In 1998, shift crews fell to technical specification minimums. The SRO ranks were depleted by transfers of a number of SROs into other departments and the resignations of several SROs who left Davis-Besse for opportunities at other sites. No new SROs were licensed until 1998. Further, within a seven-month period in 2000-2001, prior to discovery of the RPV head degradation, two Shift Managers left Operations and one left the site altogether.

Compensation issues within Operations have contributed to the staffing problems.

Some data suggest that salaries within the Operations Department are below industry averages, including operator compensation at plants located in economically comparable areas. The perception of lower wages has adversely affected operator morale for several years and appears to have served as a distraction. During contract negotiations for the reactor operators (ROs) and non-licensed operators in the late 1999/2001 period, management attempted to rectify the problem. The operators' union, however, blocked the effort to increase compensation for operators without offering the same increase to others at the site, so salaries continue to be perceived as lower than industry averages. In addition to the impact on morale, a less attractive compensation package was reported to have made it more difficult to recruit top candidates at the non-licensed operator level. In 1998, management authorized Operations to recruit two non-licensed operators from a local fossil plant that is also operated by the corporation. Recruiting efforts for non-licensed operators outside of the corporation, undertaken in 1999, were only partially successful in that all of the open positions could not be filled.

Compensation packages for more senior operations personnel are also perceived to be lower than industry averages and progression through salary grades has been slower than for other plant personnel. The "standard rate" for a position is defined at Davis-Besse as the salary level for a job incumbent who is fully qualified and, typically, has been in the position for four to five years. Interviewees reported that no SROs or Shift Managers appear currently to be paid at the standard rate for their job classifications. Several individuals reported that some senior Instrumentation and Control Technicians were paid more than the highest-paid Shift Manager, based on an informal survey conducted by Operations personnel. Opportunities for better compensation, combined with other organizational factors, have contributed to the loss of a number of SROs to other plants over the past several years.

Root Cause Analysis Report 21 I I I

At the same time the number of licensed personnel waiS decreasing, Operations assumed additional responsibilities. For example, the Procedures group at the site was disbanded and Operations was required to take on responsibility for the development and revision of all operating procedures. In addition, Operations assumed full responsibility for staffing the fire brigade when Security and Maintenance personnel were re-assigned to other duties because of staff shortages in those departments. Audits and self-assessment responsibilities increased. These activities were added to on-going administrative tasks required by regulation and plant procedures, such as CR reviews, work order reviews, clearances and tagging, and operability evaluations.

There were several consequences of the reduced number of licensed personnel on-site, particularly SROs. Overtime hours for all Operations personnel were reported to have increased. Shift scheduling was changed to 12-hour rotating shifts when a new Operations Manager was brought in, late in 2000. The staff prefers 12-hour shifts because of the increased number of consecutive days off. However, the research literature predicts that the longer shifts and shift rotations, combined with the continuing need for overtime, increase fatigue levels and so increase the likelihood of operator errors. The SROs began taking paperwork home with them in order to address the growing backlog and to avoid recordable overtime hours in excess of regulatory limits.

Task-shedding also occurred. For example, SROs stopped attending many decision making meetings, because they were needed on-shift or were under schedule pressure to complete administrative tasks. Licensed personnel, particularly the Shift Managers, spent increasingly more time processing paperwork, and less time in making field observations, in training observations, or coaching and communicating with the crews. The SROs were reported to be discouraged from taking rotational assignments in other departments to broaden their experience, and promotions out of Operations were rare, because the staffing needs in Operations were too great.

Operations' leadership and an operational perspective on other plant activities have been significantly diminished as a result.

4.3.1.2 Decreased Operations Involvement In Station Decision-Making In the past decade, Operations has had a decreasing involvement in station decision making. Some programs and processes for which Operations was responsible were assigned to other departments to attempt to reduce the administrative burden on licensed personnel. In other cases, organizational and process changes were made as improvements, but had unintended consequences for Operations' leadership.

For example, in the past, the Corrective Action Program (CAP) was accountable to Operations and Operations played a significant role in its design and operation.

Operators and others interviewed perceive that the CAP, as currently designed and implemented, detracts from Operations' leadership role.

Root Cause Analysis Report 22 I I

Interviewees perceive that Operations has limited ability to track CRs or affect the corrective actions that are developed. The perception is that SROs see the CRs when they perform their required reviews for operability or technical specification concerns, for example, or operators generate CRs, but the CRs then "disappear" into the CAP process. One equipment operator stated that he had attempted to check on the status of several CRs he initiated. He said he was unable to retrieve them by searching in the CREST system because he had not retained the CR numbers, which appears to indicate a training need. Feedback to CR originators is not currently required by CAP procedures. Interviewees also noted that the owner assigned to a

CR is able to extend due dates without consulting with the CR originator or the SRO who performed the review. Further, non-licensed personnel are able to routinely assign and clear mode-hold restrictions from CRs without authorization by a licensed operator. In fact, a design engineer cleared the Mode 4 hold restriction on CR 2000-1037 that required the RPV to be cleaned of all boric acid deposits during RFO12.

Operations' input to the prioritization and disposition of CRs is perceived as diluted by the multi-disciplinary team approach to these activities. Categorization of CR significance is performed by a Management Review Board (MRB), using a

consensus-based decision-making process. Root cause analyses and corrective actions are evaluated by the Corrective Action Review Board (CARB), also based upon a consensus decision-making style. CAP procedures mandate that a quorum be present for these meetings, but do not require an Operations representative to be present at either the MRB or CARB meetings. Further, CAP procedures do not provide a mechanism for Operations to challenge MRB or CARB decisions that may be adverse to safe operations.

Participation by licensed personnel in other key decision-making activities also decreased over the past decade. Maintenance personnel interviewed stated that maintenance project review meetings typically occur without Operations involvement.

Operations representatives are invited to participate, but the perception is that they have been too overwhelmed to do so. In addition, the meetings are often scheduled when the SRO who is assigned to a system is on backshifts or is taking his days off.

Another interviewee reported that the Station Review Board functioned for 1.5 years without an Operations representative appointed. An Operations representative to the Station Review Board is not a procedural requirement. The Operations Manager was also previously not expected to attend senior management meetings. The current Operations Manager has invited himself to these meetings, however, and reinforces the present senior management team's operational focus.

Root Cause Analysis Report 23 1 1 1 1

4.3.1.3 Failure to Support Operations' Leadership Role in Maintaining Plant Material Conditions Beginning in the mid-1 990's, operators reported they had a decreasing ability to ensure equipment modifications and repairs were completed. The interviewees noted that the Shift Manager is authorized to initiate "Immediate Action Maintenance,"

but the criteria under which it will be implemented are stringent (e.g., imminent plant trip, power reduction, or shutdown; forced entry into a Technical Specification action statement which requires specific actions within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> or less).

A senior equipment operator left Operations about 2.5 years ago, in part due to concerns about degrading material conditions in the plant. In a memorandum submitted to the union steward, he wrote:

Material condition of the plant... is becoming a bigger issue as of late...

Main Steam Piping that is vibrating so bad as to damage insulation and cause a 1/2 in valve to shake itself apart... When I am in the plant I don't get a sense of safety... Too many leaks with nothing done or a Band-Aid attached to 'show progress.'

Another example of Operations' decreased ability to effect repairs is the Operator Workaround list. Although these items are now being addressed, the operators reported that many of the workarounds have been on the list for years. They described a former senior management attitude that many of the items were not a high priority, because they represented "only operator convenience."

Interviewees within and outside of Operations described a decreasing influence of Operations' on the prioritization of work requests during the 1990's. They perceived that the threshold for initiating work was raised, and that repairs and modifications that could not be justified in terms of a narrow interpretation of regulatory requirements were often denied or deferred. In addition, the Work Management Department was moved out of the production organization that includes Operations and RP/Chemistry. Work Management now reports to the Site Vice President (SVP) rather than to the Plant Manager, which appears to have negatively affected coordination between the two groups.

Interviewees reported an incident that occurred within the past two years in which a maintenance manager instructed his staff not to take direction from operators on shift, if it would interfere with completing scheduled work. This manager told his staff that, if an operator approached a maintenance technician with a request for an immediate repair, the maintenance technician was to call the manager at home to obtain permission. The instruction was not incorporated into plant procedures and was generally not implemented, but was described as indicative of existing barriers to Operations' leadership in ensuring equipment is fixed. Operators reported that it is still sometimes necessary to rely on long-term, personal relationships with some older maintenance personnel, who retain a "respect for Operations," in order to Root Cause Analysis Report 24 I I I I

accomplish timely repairs. The recent permanent assignment of a SRO to the Fix-It Now Team has improved responsiveness to Operations' priorities.

One interviewee outside of Operations also attributed some of the resistance to performing the repairs or enhancements requested by Operations to personnel shortages in the Maintenance and Engineering Departments, and indicated that those organizations were simply unable to respond timely to the workload. In his view, and that of others interviewed, Operations documented the items requiring maintenance, but eventually stopped pushing the formal work prioritization system for needed repairs other than the most safety significant. Their experience was that the resources did not exist and the work would not be performed.

4.3.1.4 Limited Support for Operations Command and Control Authority Operations' command and control authority over station activities that may affect plant safety while on-shift was also described as having been reduced over the past decade. Operations' responsibility for the safety of activities performed by other departments continues to exist in some policy statements, directives, and memoranda, and was occasionally mentioned by managers in the past decade, but the interviewees perceived that this responsibility was not supported by management nor reinforced in training.

Interviewees reported that, in the early years of Davis-Besse operations, the Shift Manager was authorized to make decisions on-shift and "inform Operations management later." Over the past decade, the interviewees' perceive that the autonomous decisions the Shift Managers were authorized to make dwindled. Some Shift Managers stated that there were few actions they could take, other than those involved in immediately responding to plant upsets, without first discussing the issue with management.

Those interviewed indicated that the plant departments run as "silos." They stated that there are four different directors with four different, and often competing, agendas. Interviewees reported that personnel in other departments are generally more responsive to the requirements of their senior managers, rather than to Operations, and that supporting Operations has not been a senior management expectation. In addition, when a cross-functional issue arose in the past several years, senior managers with responsibility for Operations were perceived as frequently acceding to the wishes of other managers in the interests of "teamwork,"

rather than supporting the operator. Over time, the expectation that operators would lead or provide oversight of activities performed by personnel in the other functional areas disappeared.

Root Cause Analysis Report 25 I I I I

As an example, operators stated that their access to contaminated areas in the plant was increasingly limited by more stringent ALARA policies. Operators were required to provide a justification for entering contaminated areas. Entries for the purpose of making field observations were increasingly determined to be unjustified by RP and Operations management. As a result, RP technicians, who continued to routinely enter contaminated areas, had more direct knowledge of existing material conditions and the progress of work activities in the plant than the operators. Since the RPV head degradation was discovered and a SRO was assigned to the RP Department, however, operators report that ALARA barriers to their field observations have been eliminated.

The SRO's command and control authority and other licensed operator responsibilities for providing oversight of all plant activities potentially affecting public health and safety also have not been emphasized by Operations management or in operator training. Several SROs and ROs indicated that they were not aware of this authority and were surprised when it was communicated in a recent meeting called by the NRC's Director of the Office Nuclear Reactor Regulation and the Region III Administrator on October 9, 2002. Most of those who attended the meeting responded positively to the message. When questioned about what he had been told in training regarding the responsibilities and authorities of a SRO, however, one SRO interviewed cynically replied, "Sure I can order it shutdown under the law. But, in times past, if we said 'the leak rate is increasing, shut it down,' we'd be outside the fence that night, looking in."

4.3.1.5 Management Styles As the foregoing quote suggests, the management styles of some managers above the Shift Manager level over the past decade did not support the leadership role of licensed operators. Some managers described appear to have confused the reinforcement of individual accouhtability with blaming. Interviewees also described managers who held them accountable to a different standard than the manager demonstrated with his own behavior.

Numerous examples were provided by the interviewees of senior site managers as well as managerial personnel within Operations using public humiliation and verbal abuse as methods to "promote accountability." Over the past decade, this type of behavior was described as being characteristic of one or more managers in Operations' management chain at all times, including some Site Vice Presidents, Plant Managers, Operations Managers, and Operations Superintendents.

Root Cause Analysis Report 26 I I I I

It appeared to those interviewed that some managers during this period were more interested in finding an individual to blame for problems than in solving them. For example, interviewees reported that a senior manager's first response upon hearing that two body-to-bonnet nuts were found missing on RC-2 in 1998 was to state that he wanted to find and discipline the mechanic who had not replaced them. They found it significant that he immediately assumed that the missing nuts were due to a personnel error, when it was soon identified that the nuts had dissolved from boric acid corrosion. In another instance described, a valve stem came off during startup following 12RFO and delayed the restart. Again, a senior manager's first question when told of the situation was to ask "Which operator closed that valve?" It was later identified that there was a stress cracking fracture in the valve stem that caused it to break.

A "shoot the messenger" approach to problems was also described. Another incident repeatedly mentioned was related to a root cause investigation regarding the unidentified leakage in containment. A systems engineer was publicly berated by a senior manager for reporting that the Team was unable to identify the source of the leakage while the plant remained at power.

In another incident, a reactor operator who had been told he was selected to attend the next SRO class, stated that he reviewed a Hot Topics list of "Warning Flags about Industry Operations Performance," published by INPO on the internet. He highlighted nine of the items on the list, such as "Management focus on production values overwhelms messages related to safety and conservatism" and "Large amounts of overtime are required for normal operations," and gave the list to a superior for possible discussion at a turnover meeting. The manager's response was, "Ithought you were going to become a SRO...," which the operator interpreted as a threat. Within the next two-three weeks, the individual found that he, in fact, would not be starting the class at that time. The individual did not believe he could prove that the decision was retaliatory and chose not to initiate a grievance or an employee concern. He is also not interested in pursuing the matter now, and believes that a similar situation would not occur under the current Operations managers. He described his interpretation of the incident, however, as an example of the climate in Operations at the time.

Previous managers were also described as sometimes directing personnel to take actions that violated procedures or conservative operating practices. Interviewees indicated that when staff would question a management decision or inform the manager that the actions directed were in violation of the procedures and that the procedures had to be changed to implement the actions, the questions were either not answered or the individual was chastised.

One example given was an operator questioning why the plant was re-starting after a short outage in the late 1990's with continuing unidentified leakage in containment.

He stated that the manager he questioned gave him a "dirty look" and ended the meeting without answering the question. The unidentified leak rate was below regulatory limits and so plant start-up may have been non-conservative, but was Root Cause Analysis Report 27 I I I I

allowable under the regulations., However, the operator's questioning attitude was clearly not reinforced.

The interviewees' also perceived that some managers were not interested in following procedures or conservative operating practices, if they were "in the way" of what the managers wanted to have done. One incident occurred during the 1998 event in which two body-to-bonnet nuts on RC-2 were discovered to be missing.

According to this report, when the mechanics who had entered containment discovered that two nuts were missing, they left containment and informed a maintenance supervisor, who then called a senior manager. The senior manager, who was not a licensed individual, directed them to re-enter containment and replace the missing nuts. The Shift Manager was not informed of the problem until the nuts had been replaced and a condition report was brought to the control room for his review. He stated that he "went ballistic" when he realized the containment re-entries had been made without his knowledge. In addition, no operability evaluation was performed prior to the replacement. The individual who described the incident stated that, if an operability evaluation had been done as soon as the absence of the nuts was identified, the evaluation method that would have been used would have shown that the valve was inoperable, and the plant would have to be shutdown. A later finite analysis showed that operating the valve without the two nuts was acceptable.

However, the Shift Manager's authority was circumvented, the required operability evaluation was not performed at the appropriate time, and the repeat entries of mechanics into the containment at-power to replace the nuts had the potential to cause a loss of coolant accident as well as catastrophic personal injuries to the mechanics, had their actions on the valve caused it to come apart while the plant was operating.

In a later discussion of the incident with another senior manager, this same individual questioned why an investigation 'was not conducted to determine whether nuts on other valves had similarly corroded. The question arose because it had been discovered that the missing nuts should not have been used in containment. They were made of carbon steel rather than stainless steel and so were subject to boric acid corrosion. The senior manager to whom this question was raised responded incredulously by saying, "What would you have us do? Shutdown?"

Although the individual was not a member of Operations and did not have the licensed authority to cause a shutdown, during a forced outage approximately one month later, this individual exhibited the type of leadership expected of Operations in ensuring that potential safety problems are thoroughly investigated and resolved.

Without taking credit for it, it was clear to the Team that the individual had created an opportunity to tour the area near RC-2 in order to ask for a demonstration of the method used to identify carbon versus stainless steel nuts. During the demonstration, another carbon steel nut was found. At this point, a complete investigation of the extent of the condition was performed and additional carbon steel nuts were found and replaced. i This type of subterfuge should not have been necessary to ensure that the full extentof the condition was assessed, but is another example, among several reported, of the informal means employed by some plant Root Cause Analysis Report 28 I I

personnel to promote operational safety despite the existing organizational and management barriers. This individual is now in licensing class, which the Team believes bodes well for the future effectiveness of Operations' leadership.

The effect of the previous management behaviors described was demonstrated in several additional comments made by Operations personnel regarding the previous climate in Operations. One operator noted, "Operations management goes as far as it can, but senior management over-rides them. They can tell you that the Shift Manager has the keys to the plant, but when the Plant Manager gives you a direct order, what can you do?" Another stated, "The NRC licenses me, not the plant.

We're not afraid to raise concerns; there was just no point in it." Anger and frustration were apparent when the operators described their inability to resolve degrading equipment conditions, problems in plant programs, personnel matters and other areas in which they believed Operations should be playing the leadership role.

Most of them reported a growing sense of "uneasiness" about the safety of operations. But, with limited authority and support, and repeated experiences of abusiveness from senior management over a period of 10 years, operators' willingness to aggressively pursue operational problems that, to the best of their knowledge, did not violate regulatory requirements, appears to have been diminished.

4.3.1.6 Management Changes In the period from 1993 through March, 2002, management changes within Operations and at more senior levels were relatively frequent. There have been three Site Vice Presidents since 1993 and the current Site Vice President is acting, so an additional change is anticipated. Three individuals have held the Plant Manager position, which is the director-level position to which Operations reports. There have been seven Operations Managers during this period, and nine individuals have held the Operations Superintendent position. The individual currently filling the Operations Superintendent position is assigned to the position in an acting capacity and has not volunteered to accept the position permanently. One equipment operator, who has been at Davis-Besse for 2.5 years, reported that he has had four Operations Superintendents in that time. A licensed operator stated that he has had 17 different superintendents and managers in the 20 years he has worked at the plant. Under these changing circumstances, consistency in the communication of management expectations has been lacking. Further, given the management behaviors described above, Operations personnel have had reason to believe that communications such as "You are the six most important men in the plant," as stated by a former senior manager to the Shift Managers, lacked sincerity.

Operations personnel interviewed range from cautiously optimistic to enthusiastic about the individuals in their current management chain above the Shift Manager level. In comparing the new managers to previous managers, interviewees stated that the new managers are visible in the field, ask questions and listen to the answers, explain their decisions, come to the support of the operators' leadership role in interactions with other plant personnel, seem to want them to learn and Root Cause Analysis Report 29 I

improve, and follow up on commitments. Several equipment operators indicated that this is the first time that Operations management has taken an interest in them and makes them feel they are an important part of the Department. The current managers were often described by the operators as a "dream team."

Because of the new management, Shift Managers and other Operations personnel at all levels expressed a willingness to pursue the leadership role at Davis-Besse, but some wariness remains. Poor choices in permanently filling the Operations Superintendent and Site Vice President positions, or further changes in those holding the Operations Manager and Plant Manager positions, could delay or stop the progress that has been made. Operations personnel may not be willing to fully commit to resuming the leadership role until it is clear that they will have consistent senior management support in doing so.

Senior management faces a significant challenge in filling the Operations Superintendent position permanently, however. In the past several years, few SROs who were asked to accept the Operations Superintendent position have been willing to do so. The primary reason described by interviewees was the perceived lack of management support for Operations and observations that previous Operations Superintendents were ineffective in the role. The lack of opportunities for advancement and a clear career path were also identified.

4.3.2 Barrier Analysis Results The barrier analysis focused on management practices and systems that could have prevented the erosion of Operations' leadership role at Davis-Besse. A management system consists of the policies, procedures and work processes that are defined by management to identify the goals and objectives of work and to control how work is accomplished in an organization. The causal factors identified in the barrier analysis are as follows:

4.3.2.1 Management expectations regarding the roles, responsibilitles, and authorities of Operatiohs were not appropriately defined or Incorporated into policies, procedures, position descriptions and performance evaluation criteria.

A review of station documentation indicated that legacy documents contain management expectations regarding Operations' leadership role but more recent documents are incomplete and some do not support it. For example:

"* A number of Standing Orders continue to exist and policy memoranda were published in the past several years that were not incorporated into procedures or the Our Conduct for Excellence: "Leading the Way" pocket manual.

"A position description for senior equipment operators does not list responsibilities for assuming a leadership role with regard to activities in Root Cause Analysis Report 30

the plant areas to which they are assigned and supervisory training was not provided to them.

Job performance evaluation criteria for supervisors, managers, and executives do not mention safety, whereas "safety consciousness" is a rating dimension for personnel at the individual contributor level.

Operating procedures include requirements for obtaining Operations Manager approval for actions that were more appropriately assigned to the Shift Manager.

As noted above, the CARB, Project Review Committee, and Station Review Board Charters do not mandate Operations representation at meetings.

The manner and specific types of situations in which Operations personnel should demonstrate leadership in setting high standards for station activities involving personnel from other departments are not documented.

Consistent and complete documentation could not have assured Operations' leadership role without management support for it. However, the fact that some plant policies, procedures and other forms of documentation have institutionalized a decreased role for Operations in maintaining safety is evidence of the erosion and appears to have contributed to Operations' declining leadership role at the site.

4.3.2.2 Management failed to ensure that regulatory expectations for licensed personnel were effectively communicated and reinforced.

NRC expectations for licensed personnel were identified and evaluated, because the NRC issues the licenses to individual operators. Management at Davis-Besse is responsible for communicating and implementing these expectations.

Paragraph (m)(2)(ii) of 10 CFR 50.54 states that "Each licensee shall have at its site a person holding a senior operator license for all fueled units at the site who is assigned responsibility for overall plant operation at all times there is any fuel in any unit." Over the years, the NRC has published policy statements and guidance regarding implementation of this requirement.

Root Cause Analysis Report 31

For example, following the Three Mile Island Accident in 1979, the NRC issued several requirements based on lessons learned from the accident. One requirement specifically addressed the Shift Supervisor's responsibilities (this position is now commonly referred to as the Shift Manager) and was later incorporated into TMI Action Plan Item I.C.E of NUREG-0694. In the Introduction to the original short-term recommendation 2.2.1.a, the NRC stated:

The ability of Shift Supervisors to carry out their responsibility for safe operation of the plant may be impaired by actions of utility management or by the individuals themselves. For example, management can impair a Shift Supervisor's command function by requiring a significant portion of his time to be devoted to administrative functions. The Shift Supervisor's command function can also be impaired by failure to recognize his leadership and decision-making responsibilities which go beyond those of the operators. If neither management nor the individual Shift Supervisors treat the Shift Supervisor position as that of a "manager" or "commander" of shift operations, the benefits to safety of clear delineation of a responsible individual in charge of plant operations may be reduced.

These expectations were further communicated by the NRC in Inspection and Enforcement Circular No. 81-02: Performance of NRC-Licensed Individuals While on Duty. In this circular, the staff stated:

NRC believes that a relationship exists between the professional attitude of a nuclear plant's operating and management personnel and the degree to which the health and safety of the public is protected...

Factors making up this professional attitude include knowledge of all aspects of plant status by licensed control room operators, maintaining an orderly and clean working environment, aggressiveness of the operating staff to prevent operational problems, and correcting observed deficiencies... All on-duty NRC-licensed operators and operating supervisors must be aware of and responsible for the plant status at all times. This includes supervisors being responsible for the performance of all personnel assigned to their shift who could affect plant safety, regardless of specialty affiliation...

In 1989, similar principles were reiterated in the NRC's Policy Statement on the Conduct of Nuclear Power Plant Operations:

It is essential that management at each nuclear power reactor facility establish and maintain a professional working environment in which the licensed operator may be fully successful in discharging his or her safety responsibilities...

Root Cause Analysis Report 32

Each individual licensed by the NRC to operate the controls of a nuclear power reactor must be keenly aware that he or she holds the special trust and confidence of the American people, conferred through the NRC license, and that his first responsibility is to assure that the reactor is in a safe condition at all times.

These NRC expectations for licensee management and licensed operators do not appear to have been consistently communicated to Davis-Besse personnel, as indicated by the operators expressing surprise at the statements made by senior NRC managers during the October 9, 2002 meeting. The operators were surprised that NRC management views licensed personnel as "agents of the NRC" in performing its mission to protect public health and safety.

Training to address regulatory requirements and expectations was available from FENOC legal counsel and has been provided at other FENOC sites, but was not incorporated into Davis-Besse operator training. This training has also has not been provided to site managers.

If these NRC expectations for licensed personnel had been effectively communicated and reinforced, operators at Davis-Besse may have been more successful in retaining a leadership role.

4.3.2.3 Management failed to assure that a safety conscious work environment was established and maintained In Operations.

A key aspect of licensed operators' ability to maintain a leadership role in assuring plant safety at a site is a safety conscious work environment. A safety conscious work environment is defined by the NRC as a working environment in which employees are encouraged to report safety concerns without fear of criticism or retaliation from their supervisors and are empowered to ensure that safety concerns are promptly resolved.

Within the NRC's Reactor Oversight Process, a safety conscious work environment is identified as a cross-cutting area. A cross-cutting area is defined by the NRC as a nuclear plant activity that affects most or all of the safety cornerstones that have been defined by the NRC and are assessed through NRC inspections and other oversight activities. Indicators of deficiencies in a safety conscious work environment that have been identified by the NRC include:

... the lack of effective evaluation, follow-up, or corrective action for concerns raised to the ECP or findings made by the licensee's QA organization; overall licensee ineffectiveness in identifying safety issues; the occurrence of repetitive or willful violations; a licensee emphasis on cost-cutting measures at the expense of safety considerations; and/or poor communication mechanisms within or among licensee groups.

Root Cause Analysis Report 33

Although the NRC has published expectations that nuclear utility licensees will establish and maintain a safety conscious work environment, these expectations were not fully implemented at Davis-Besse, as follows:

Expectations were not established that assign responsibility to managers at all levels for maintaining a safety conscious work environment, in which personnel are encouraged to raise safety concerns without fear of ridicule or retaliation and managers are obligated to work to resolve the concerns appropriately.

Training was not provided to managers to ensure they possessed the knowledge, skills, and abilities required to establish and maintain a safety conscious work environment.

Management behavior discouraged, and sometimes appeared to punish, a questioning attitude.

If a safety conscious work environment had been established and maintained in Operations, licensed operators may have been able to more aggressively pursue resolution of their growing sense of "uneasiness," even without conclusive evidence of a technical specification violation or other information to support an operability concern.

4.3.3 HPEP Results The HPEP Cause Tree and Modules are a variation of several root cause analysis techniques based on repeatedly asking the question, 'Why?", with regard to the circumstances surrounding an event or adverse condition. Questioning is stopped when answers are obtained that are outside of management's control.

4.3.3.1 Direct and Programmatic Causal Factors Two types of causal factors are described in the HPEP Modules: direct and programmatic causes. A direct cause of an event is the actions or conditions immediately preceding or surrounding the event that caused or allowed it to occur.

Programmatic causes are management and organizational conditions that allowed the direct causes to exist, and, hence, the event to occur.

Evaluation of the information gathered during the investigation using the HPEP Cause Tree and Modules identified the following direct causes (descriptions quoted from the HPEP):

Reward structure - The desired behaviors with regard to safety, productivity, and quality workmanship were not appropriately rewarded.

Root Cause Analysis Report 34

Insufficient staff available - Performance failed because adequate numbers of appropriate personnel were not available to perform the assigned work.

Resources not provided - Performance failed because supervision did not ensure that workers had the resources required to perform the task, including information, procedures, guidance or assistance in solving problems that arise.

Wrong aoals - Performance failed because supervision communicated, directly or indirectly, an emphasis on production or cost goals over safety.

• Questioninq attitude discouraged - Performance failed because supervision, directly or indirectly, discouraged workers from questioning work practices or instructions.

Two programmatic causes that accounted for the direct causes were identified:

Supervision and Management - Supervision communicates and reinforces management expectations and establishes goals and requirements for performance. Supervisory oversight may increase motivation to perform in accordance with expectations as well as detect and correct any errors that occur. Weaknesses in supervision, for example, may cause staff to choose production over safety goals in their work or to tolerate workarounds that may lead to errors. (Reward structure, resources not provided, wrong goal, questioning attitude discouraged)

Human Resources Planning - Most licensees develop some form of a business plan that defines organizational goals and objectives.

Business plans are often used to estimate the resources required to achieve the goals and run the business. Business plans may be used to determine staffing levels for the various parts of the corporate organization, sometimes without manpower planning and analyses of anticipated workload levels. As a result, there may be insufficient staff or staff may not have the required expertise. (Insufficient staff available) 4.3.3.2 Root and Contributing Causes The HPEP was also used to analyze the causal factors that were identified from the change, barrier and HPEP Modules analyses. The purpose of this analysis was to identify root and contributing cause(s) of the loss of Operations' leadership role at Davis-Besse.

The HPEP defines root and contributing causes as follows:

Root Cause Analysis Report 35

A root cause is the actions or set of conditions that, if eliminated or modified, would keep the event or adverse condition from recurring as well as prevent similar events or adverse events from occurring. A root cause is often responsible for multiple human errors or hardware failures, rather than single problems or faults. Root causes are more fundamental causes than direct causes, and are typically programmatic or management weaknesses.

A contributing cause is an action or condition that sets the stage for the event or adverse condition to occur. A contributing cause may be a long-standing condition or a series of prior events that, while unimportant in themselves, increase the probability that the event or adverse condition would occur.

The results of this analysis are reported in the next section.

Root Cause Analysis Report 36 I

5.0 Root Cause Determination Based upon the analyses described in Section 5.0, the Root Cause Analysis Team identified one root cause and three contributing causes for the lack of Operations' centrality in maintaining, assuring, and communicating the operational safety focus of Davis-Besse and for the lack of accountability of other groups to Operations in fulfilling that role. These causes are discussed in the following sections. Related observations made by the Team are also presented.

'/e Root Cause Aenior management support for Operations' leadership role In assuring S-'! nuclear safety was lacking.

The Management and Human Performance Root Cause Analysis Report concluded that a less than adequate nuclear safety focus (a production focus combined with taking minimum actions to meet regulatory requirements) was a root cause of the RPV head degradation. The current Root Cause Analysis Team concurs and concludes that the less than adequate nuclear safety focus was also a key contributor to the decline in Operations' leadership role in station activities. In fact, strong Operations' leadership should serve as a check on a growing production focus in management. Operations' leadership, however, requires that senior managers are willing to respect the authority and responsibilities of licensed personnel and to support and act upon Operations' concems that the margin of safety is decreasing -

before regulatory requirements are breached and safety is compromised. Therefore, the Team concluded that a lack of senior management support for Operations' leadership role was the root cause of this adverse condition.

5.2 Contributing Causes Three contributing causes were identified. Although none of these factors alone caused the erosion of Operations' leadership role, each of them contributed to it.

,' \t 1. Staffing was inadequate to perform the tasks assigned. There was an inadequate number of licensed personnel to complete assigned work without excessive overtime hours, to participate in station decision making, and to provide leadership in the activities of other station organizations. In addition, resources in other site organizations were insufficient to support Operations' priorities in assuring sustained safe operations.

Root Cause Analysis Report 37 I

S2. Senior management failed to ensure that rbAulatory expectations for licensed personnel were effectively communicated and reinforced.

Senior management did not ensure that station personnel at all levels understood the command and control authority of licensed operators on-shift. Appropriate roles, responsibilities and authorities were not documented in policies, procedures, and other documentation, expectations were not communicated and reinforced through training, and management behavior was inconsistent with the expectation that Operations would lead the site. Further, changes in management personnel resulted in the communication of inconsistent expectations regarding Operations' leadership role.

e (3. Senior management failed to assure that a safety conscious work environment was established and maintained in Operations.

Management engaged in behaviors that created a chilling effect in Operations by failing to encourage, and sometimes appearing to punish, a questioning attitude and the raising of safety concerns.

Aggressiveness in pursuing the resolution of operational problems was discouraged by repeated failures.

5.3 Related Observations The Team made three additional observations regarding factors affecting Operations' leadership role, as follows:

1. Operations personnel have not been provided all of the necessary tools and equipment to perform their tasks efficiently. Recent remodeling efforts have resulted in a lack of access to desks, computers, filing cabinets, and permanent telephones for some Operations personnel.

For example, Assistant Shift Managers and other SROs struggle to locate workspace for performing CR reviews, completing procedure changes and reviews, and other administrative tasks. Further, not all of the station databases and software required for Operations activities are accessible from every computer used by Operations personnel and some personnel require additional training in computer skills.

Root Cause Analysis Report 38

2. Communication across shifts is a challenge. Communication across shifts within Operations, and between Operations and other station work groups is a challenge under any circumstances. The lack of consistent access to e-mail for all Operations personnel contributes to communication difficulties. A Monday evening telephone conference has been instituted that includes the Operations Manager, the Superintendents, and Shift Managers and appears to have improved cross-shift communications. Additional technologies, such as the assignment of cell phones to all Operations personnel, and implementation of additional processes to enhance communications appear to be warranted to ensure that necessary information is communicated among all Operations personnel to support their leadership role.

Root Cause Analysis Report 39

6.0 Extent of Condition Based upon the information considered by the Root Cause Analysis Team and the information documented in the Management and Human Performance Root Cause Report in response to CR 2002-0891, the Team believes that other station departments were also adversely affected by some of the same factors identified in Section 6. In response to the Management and Human Performance Root Cause Report, a Management and Human Performance Excellence Plan was developed and is being implemented. The Management and Human Performance Excellence Plan includes a series of reviews of selected station organizations that include, for example, checks of whether there are clear lines of authority and responsibility within the organization; whether staffing levels and resources are sufficient to handle assigned responsibilities; whether individuals have a clear description of their assigned responsibilities; and whether interfaces with other organizations are clearly defined. In addition, a number 6f actions are being implemented to strengthen the safety conscious work environment across the site. The Team concludes that these efforts should identify needed borrective actions for similar problems to those identified with regard to Operations' leadership role at Davis-Besse.

Root Cause Analysis Report 40 I I

7.0 Corrective Actions This section repeats each of the root and contributing causes in Section 6, and then identifies applicable corrective actions.

7.1 Corrective Actions for the Root Cause Senior management support for Operations' leadership role was missing.

Corrective Actions:

a. Extensive changes have been made in the officers, directors, and managers responsible for Operations, including changes in the Site Vice President, Plant Manager, Operations Manager, Operations Superintendent, and Operations Support Superintendent. These individuals value strong Operations' leadership.

4. Senior management is demonstrating support for Operations' leadership role by being visible and active in Operations' activities, such as shift turnover meetings, by appropriately mentoring and coaching Operations' personnel in resuming the leadership role, and by ensuring that Operations' priorities are addressed and supported in station decision making. The Nuclear Quality Assurance organization will periodically assess the Implementation and success of these activities.

,d A declaration from the chief executives will be issued and communicated to site personnel delineating Operations' leadership role.

PThe Operations leadership team will disseminate the results of this Root Cause Analysis and the corrective actions to other station managers, will ensure that other station organizations dedicate the time required to internalize the Impact of Operations resuming the leadership role on their activities, and will resolve any issues identified in implementing the change.

,e<ýenior management will assure that no uneasiness remains among Operations personnel regarding the station's ability to operate safely prior to restart. Shift Managers will be charged with eliciting any outstanding safety concerns from their crews and for ensuring that the concerns are resolved. Davis-Besse will not restart until each Shift Manager is willing to state that he and his crew know that the plant is ready to restart, that Operations has regained and is performing the site leadership role, and that the plant will operate safely.

Root Cause Analysis Report 41 I I I

7.2 Corrective Actions for Contributing Causes

-7.2.1 Staffina was Inadequate to Perform the tasks assigqned.

Corrective Actions:

Continue hiring new personnel to be trained as equipment operators and continue training of RO and SRO candidates. Continue implementing current plans for additional licensing classes to replenish and maintain a sufficient number of licensed personnel for the tasks assigned, as well as to populate other departments with personnel who maintain active licenses.

k/A'nalyze the tasks currently assigned to Operations. Identify additional activities that Operations must perform to continue re-establishing and to maintain leadership. 'Determine the number of personnel and the qualifications required to perform the activities identified. Develop and implement short-term compensatory measures for staffing shortfalls that are identified.

o.e*ddress Operations' compensation, as necessary, to ensure retention of current staff. Improve the station's competitive position In attracting desirable applicants. Continue current activities to develop and implement professional development plans for Operations personnel to ensure that career paths are identified and that future site leaders will be available and pared to assume leadership roles.

'. Implement corrective actions for staffing needs identified in other station organizations to ensure staff capabilities exist to support Operations' priorities.

7.2.2 Senior management failed to ensure that regulatory expectations for licensed tmrsonnel were effectively communicated and reinforced.

.Prior to restart, Operations and management personnel from other station organizations will receive corporate training regarding the roles, responsibilities and authorities of licensed personnel. This training course will be added to the core continuing training program for Operations and management personnel.

e Site Vice President will continue to make himself available to the Duty Shift Manager to assist in ensuring that personnel in other station organizations understand his expectation that they are accountable to the Duty Shift Manager and to Operations personnel and must support Z 0rations' leadership role.

Management will ensure that an invitation is extended to NRC representatives to address newly licensed Ior upgraded operators when Root Cause Analysis Report 42 I I

they are awarded their licenses, and to address Operations Department personnel periodically to communicate and reinforce NRC expectations, as appropriate.

/An Operations Standards and Expectations document has been issued to address, in detail, expectations, Job standards, and responsibilities of Operations Department personnel. Knowledge of these standards and expectations will be reinforced by training and testing prior to restart.

2/A memo signed at the highest level defining the Shift Manager's role, responsibilities and authorities will be issued and conspicuously posted In selected areas throughout the site. This memo will be revised and reissued on an annual basis. I eJlThe Operations Leadership Plan wilt be reviewed and approved by senior management.

As part of the Management and Human Performance Excellence Plan, policies, procedures, program and job descriptions, and organizational interfaces are being reviewed for consistency with management expectations. As part of these reviews, opportunities to strengthen Operations' leadership role will be Identified and incorporated. For example, L Requirements will be added to the corrective action process to ensure that Operations' concerns are adequately addressed in the prioritization, scheduling and resolution of condition reports.

ii. Operations' involvement in station decision-making processes will be strengthened. Operations' representation will be required at Management Review Board, Corrective Action Review Board, Station Review Board, and Project Review Committee meetings, and that Operations input will be sought in other station decision-making processes, as appropriate. These important decision-making meetings will not occur unless a designated Operations representative Is present.

Licensed personnel will fully commit to resuming the leadership role.

7.2.3 Senior management failed to assure that a safety conscious work environment was established and maintained In Omratinns

/K The Operations Standards and Expectations document will address the chilling effect in Operations by Including expectations for Operations personnel to raise any operational concerns. It also contains the requirement for Operations personnel to demonstrate leadership in resolving concerns by continuing to escalate them through their management chain up to and Including the President of FENOC until Root Cause Analysis Report 43 i I

resolution is obtained. Davis- Besse and FENOC senior management expect Operations personnel to inform the NRC of their concerns, if management does not address the concerns to their satisfaction.

C Licensed operators will be delegated management authority for addressing

- and resolving safety concerns that are identified to them by other station Z.s~onnel.

istent with their leadership role, Operations personnel at all levels will be given training in maintaining a safety conscious work environment to ensure that their leadership and oversight of station activities performed by personnel In other departments is conducted in accordance with

,m)anagement expectations.

4. Individuals appointed to the open Site Vice President and Operations Superintendent positions will be thoroughly screened prior to appointment to ensure that their management styles will support a safety conscious work environment in Operations.

Operations personnel and managers in all station organizations will comply with senior management requirements and NRC expectations for ensuring that a safety conscious work environment Is maintained. 210 of 250 site managers have recently completed safety conscious work environment training for supervisors, and the remainder will complete training by December 4, 2002.

/"Reports from site-wide surveys and assessments of the safety conscious work environment in Operations will be provided to the Plant and Operations Managers, and any weaknesses identified will be promptly addressed and resolved.

g. Davis-Besse will complete implementation of the Safety Conscious Work Environment Action Plan as part of the Management and Human Performance Improvement Plan.

7.3 Corrective Actions and Proposed Enhancements for Observations Operations personnel have not been provided all of the necessary tools and equipment to perform their tasks and communicate efficiently.

Corrective Actions:

a. Provide permanent workspaces for Operations personnel.

b. Consider increasing the use of additional technologies and practices to support communication, such as the assignment of cell phones to all Root Cause Analysis Report 44 I I I

Operations personnel that can be used for individual communication as well as group distribution messages.

Root Cause Analysis Report 45 i I I i

8.0 Experience Review Section 7 of the Root Cause Analysis Reference Guide and Attachment 11 of the Programmatic Guideline for the Davis-Besse Condition Report Process state that a review of similar experiences at the plant and across the nuclear industry should be conducted to determine:

whether past occurrences of similar problems indicate a generic or broader scope issue,

• why prior corrective actions for similar problems were not effective, and

• whether the currently proposed preventive actions are different so as to be more effective.

This section also discusses why the currently proposed corrective actions are different from those taken in response to previous Davis-Besse and industry experience, and why the proposed actions should be more effective.

To identify relevant past experience, keyword searches of several databases were performed. The INPO Operations Experiences Database and Davis-Besse's CATS and CREST databases were searched. Search terms included: safety conscious work environment, safety focus, command and control, leadership role, management expectations, regulatory expectations, questioning attitude, command function, involvement, and accountability.

8.1 Recent Assessments of Davis-Besse Related to Operations Symptoms of the decline in Operations' leadership role at Davis-Besse were identified in four previous assessments. These assessments were performed within the past three years.

8.1.1 1999 RHR Organizational Assessment An organizational assessment was conducted by RHR International in June-July, 1999. The assessment identified many of the conditions that were described to the current Team regarding past organizational problems at the site, as follows:

• Headcount and cost-cutting

• The site had a pure operating orientation until the 1990s and a business planning mentality did not exist

• Reliability and cost have become critical success areas

• Many want to return to the basics Root Cause Analysis Report 46

• Perception is the site got behind 0 Silos exist among the different units 0 A gulf exists between the Directors and other levels

• Many managers avoid raising bad news

• Directors rely on command-and-control

• Key people were drained off 0 Little attention goes to attracting and developing talent

• Managers avoid rocking each others' boats 0 Criticism can be personal and blaming 0 Decisions are made within silos

• Little emphasis exists on safety and performance at all costs

• Management rarely scans for subtle problems The results of the assessment were communicated to senior site management, bul the current Team was unable to verify that corrective actions were taken at the time.

Senior management personnel at the site who received the information are no longer in those positions and were not contacted.

8.1.2 June, 2001 Operations Self-Assessment Prior to a scheduled INPO audit, Operations completed a self-assessment in June, 2001 (CR 2001-1458) to identify strengths and weaknesses in the following INPO performance areas:

Operations

• Safety Culture Plant Status Control The self-assessment identified only one weakness in Operations regarding the failure to include the Shift Manager as a member of upper management. Several corrective actions were developed and some were implemented at the time.

Root Cause Analysis Report 47

One corrective action that was implemented was holding an offsite meeting on October 2 9 th, 2001, for teambuilding purposes and to address needed steps for enhancing the Shift Manager's leadership position at the site. Minutes from that meeting show that the following items were discussed:

"* Insufficient staffing to accomplish the tasks assigned.

"* The need for an Operations Succession Plan.

"* The need for Shift Managers to be informed of personnel decisions made by the Operations managers that affected the individuals on their crews.

"* A plan to address pay issues for the SROs, Shift Manager professional development plans, and rotational assignment possibilities.

"* The unwillingness of Shift Managers to take the Operations Superintendent position because they believed that the management team would not support the Operations Superintendent and that the position was powerless.

"* The Shift Managers' desire to avoid attendance at the morning Managers Communication and Teamwork meetings because the meetings were perceived as "demoralizing."

Another corrective action to CR 01-1458 was to identify meetings and projects in which Shift Managers could participate to be more involved in management decision making. This action was determined to be "not a priority at this time, but may be revisited at a later date as part of continuing Shift Manager development."

8.1.3 September, 2001 INPO Audit Results of an evaluation conducted by INPO representatives during September of 2001 provided several additional indications of Operations' declining leadership role (CR 2001-2989). These included:

Operations management had missed opportunities since 1998 to take clear ownership of safety tagging problems. Operations had not taken ownership of those 'aspects of the process that crossed disciplinary boundaries, such as ensuring that the work scope planned was completely covered by the clearance requested.

Revised expectations for some work activities were not clearly established and communicated, and so were not being consistently implemented. Operators did not communicate to management that some of the standarls and expectations could not be implemented as written.

Root Cause Analysis Report 48

In some instances, shift management did not thoroughly question or challenge initial Engineering recommendations about the status of important equipment deficiencies.

Expectations for Shift Manager and Field Supervisor oversight of normal day-to-day plant operations were not established. The Shift Manager, Shift Engineer, and Field Supervisor were observed to spend most of their time in administrative duties.

A misalignment within Operations was identified in that expectations were not clearly and consistently reinforced, due, in part, to recent changes that had occurred in Operations management.

A number of corrective actions were implemented in response to CR 2001-2989 and some are still on-going. The effectiveness of the actions will be evaluated by December 22, 2002. The current Operations Leadership Plan includes additional items to address the results of the INPO audit. Corrective actions to CR 2001-2989 that have been implemented to-date include:

• Benchmarking trips to identify opportunities to enhance performance have been conducted;

"* The safety tagging procedure has been revised and staff within Operations have been assigned as owners;

"* Weekly telephone conferences and monthly meetings among Shift Managers and Operations management personnel have been institutionalized to enhance inter-shift communication; and

"* A revision to the Operations Standards and Expectations Directive has been completed.

8.1.4 2001 Nuclear Quality Assessment Audit A Nuclear Quality Assessment audit of Operations was performed during the period of August 13 through November 7, 2001. The audit team concluded that the Operations Program was effectively implemented and overall performance was rated as satisfactory. The audit team also noted, however, that Shift Engineers and SROs appeared to be overburdened by administrative duties while on-shift, there was a low number of staff available to perform procedure reviews and other administrative work, performance expectations in policies and procedures had not been updated, and that workspace for performing administrative duties was insufficient. The auditors noted that actions were being taken to address staffing shortages, but that the efforts had not been effectively communicated to Operations personnel. The audit did not address Operations' interfaces with other station organizations nor the effects of management styles on the Department.

Root Cause Analysis Report 49 I I i i

8.1.5 Conclusions The results of these assessments indicate that evidence of the decline in Operations' leadership role was identified prior to this Root Cause Analysis. Some corrective actions were developed and implemented, but it does not appear to the Team that the potential consequences for nuclear safety of the management and organizational issues occurring in Operations were previously considered. Again, however, in the absence of senior site management support for addressing the "chilling effect" in Operations or for resolving the types of organizational issues that were identified in the RHR report and subsequent assessments, it is unlikely that corrective actions that Operations might have considered taking would be effective.

8.2 Related Industry Experience Numerous examples of industry events that shared at least one characteristic with the adverse conditions found by this Team were identified in the INPO databases.

The four incidents that best matched the causal factors described in this Report are discussed in this section.

8.2.1 SER 93-28, Increased Leakage from an Unlsolable Reactor Coolant Leak after Repeated Sealant Inlections In 1993, Millstone Unit 2, then operated by Northeast Utilities, was shut down from 100 percent power when reactor coolant leakage from a manual isolation valve inside the reactor containment building !exceeded technical specification limits. The leak was first discovered after a reactor scram in May, 1993, and corrective actions were taken, but they were ineffective. the leakage did not exceed regulatory limits, so the reactor was returned to 100 percent power. Repeated attempts at on-line leak repairs were made over the next 2.5 months. During the final repair attempt on August 5, the leak rate suddenly increased to an unacceptable level and the reactor was manually shut down.

Station personnel who were invoIved in the repairs raised questions several times regarding the possible consequences for valve integrity of the repair attempts. A through-wall crack in the body of the valve was suspected at one point, and a recommendation was made to shut down the plant. The recommendation was not implemented.

Extensive analyses of the causal factors for this event have been published elsewhere and indicated that an emphasis on production over nuclear safety at corporate and senior management levels, and chilling effects, were root causes for the event. The INPO SER also noted the following:

Because senior management did not question the repeated attempts at repairing the valve, plant personnel saw this as implicit approval for continuing to repair the valve at-power.

Root Cause Analysis Report 50

Station management recognized that permanent repair required a unit shutdown to replace or rebuild the valve. However, it was decided to continue attempts to repair the valve at power to maintain production goals.

The decisions to continue attempts to repair the valve were based on engineering analysis of the structural integrity of the valve design.

Safety evaluations to assess the potential impact and consequences of catastrophic valve failure were not performed.

Station management justified continued operation because the overall leak rate was within technical specifications. This focus and mind set affected the judgment of many even when there was evidence of another potential problem with the valve.

Among other recommended corrective actions, the SER states:

Senior nuclear managers should periodically emphasize to nuclear organization personnel that nuclear safety considerations always take priority over production goals and that station personnel are expected to conduct nuclear-related activities to the highest standards. Examples of both proper and improper decisions affecting nuclear safety should be communicated to promote improved understanding of these management expectations. Plant staff periodically should receive reinforcement, through training and management coaching regarding their unique responsibilities for the safe operation of their nuclear units.

To meet these demanding responsibilities, plant personnel must hold themselves and each other accountable to the highest standards of performance. As nuclear professionals, nuclear plant staff members must make decisions that reflect an overriding emphasis on protecting the reactor core.

The Davis-Besse response to the Millstone event focused on the technical issues involved in the event, rather than the organizational issues. At the time, Davis-Besse was one of the few nuclear stations in the country that had already established an Ombudsman. The program had been in existence since the mid-1980s. Because the number of employee concerns brought to the Ombudsman was low, additional actions to respond to the safety conscious work environment aspects of the Millstone event did not appear to be necessary.

Root Cause Analysis Report 51

8.2.2 .SER 1-97, Nonconservative Operations During Isolation of a Reactor Recirculation Pump Seal Leak In September of 1996, Clinton Power Station personnel continued operations for approximately 16 hours1.851852e-4 days <br />0.00444 hours <br />2.645503e-5 weeks <br />6.088e-6 months <br /> with reactor coolant system pressure boundary leakage in excess of technical specification limits. The SER noted:

Operators and line managers were focused on decreasing a reactor recirculation pump seal leak to allow continued plant operation. As a result, operators did not take conservative actions to deal with this potential threat to reactor coolant system pressure boundary integrity.

Participation of line management personnel in the evolution, without clearly defined roles and responsibilities, resulted in confusion regarding the ultimate responsibility for making decisions affecting nuclear safety.

Lessons learned from industry operating experience were not used effectively to provide operator and line management focus on nuclear safety over production goals.

This event was reviewed by Davis-Besse Operations personnel and several actions were taken in response, including:

A simulator scenario was developed that included "pressure" from individuals outside the normal control room crew who were advocating that the plant should be kept on-line. The scenario was used as an introduction to a presentation on the event at Clinton and a discussion of management expectations regarding control room roles and responsibilities at Davis-Besse. All crews chose to conduct the plant shutdown required by Davis-Besse procedures and technical specifications.

Davis-Besse guidance documents were also reviewed and it was determined that existing guidance and procedures were sufficient to ensure clarity of roles and responsibilities.

Although Davis-Besse's response to this SER was aggressive and the crews performed well, the scope of the corrective actions was limited to control room roles and responsibilities. Operations' leadership in plant support activities, such as Engineering, Maintenance, and decision-making processes, was not similarly emphasized. However, as the RPV head degradation event and others have repeatedly demonstrated, Operations' leadership in plant support activities is as important as leadership in the control room to ensure that nuclear safety is maintained.

Root Cause Analysis Report 52

8.2.3 OE 9944, Lack of Operator Awareness of Plant Equipment Status A Quality Assurance Assessment at Clinton Power Station performed in 1999 identified several examples of Operations personnel lacking awareness of plant equipment status and failing to take ownership. For example, during shift turnover, complete information was not provided to the on-coming shift and on-coming ROs accepted incomplete answers to their questions. In one instance, control room equipment was released for operations for eight days before the information was communicated to the shift crews. Causal factors identified included the failure of shift and Operations management to consistently communicate, reinforce and hold personnel accountable, and shift managers and other shift personnel not fully internalizing the values of accountability and ownership. The Clinton Shift Managers noted that they allowed themselves to become involved in staff functions, such as tagout reviews, work order authorization reviews, and CR reviews in order to help others, rather than maintaining their management and oversight role.

Extensive corrective actions were implemented at Clinton to enhance Operations' awareness and ownership of plant equipment status. These included:

The Plant Manager developed a Model for Plant Operational Focus.

The model depicts organizational focus areas for Operations, Engineering, Maintenance, and Work Management, and their interrelationships. The Director of Operations reinforced these focus areas with the Shift Managers and their crews. The need for Shift Managers to take a leadership role in interfacing with other organizations was also communicated.

Communications between crews was enhanced by setting up a central voice mailbox for discussion of emerging items, safety issues, new CRs and other information. Access to the mailbox ensured that Shift Managers had the same information as soon as it was available to communicate to their crews.

Mentors for each Shift Manager were assigned to observe, coach and reinforce positive behaviors on the shift crews. Control room activities were monitored and personnel coached on appropriate responses to indicate ownership and accountability for understanding plant status and configuration.

This event report was disseminated to Davis-Besse Operations personnel for review, but it was not analyzed for applicability and no actions were taken to implement lessons learned at the site.

Root Cause Analysis Report 53

8.3 Differences between Previous and Proposed Actions There are a number of differences between the previous corrective actions that were implemented in response to the findings of the internal assessments of Operations and the reviews of external operating events and those proposed in this Report. The most important difference is that, since December of 2001, the entire top tiers of management at Davis-Besse have changed. In particular, a new position of Chief Operating Officer has been created and filled, a new Plant Manager from outside of Davis-Besse has been appointed, every Director has been newly appointed (several from outside of Davis-Besse), and all of the managers within Operations have been replaced. Additionally, a new Vice President of Oversight position has been created and filled, and this individual is charged with strengthening the safety conscious work environment at Davis-Besse.

At the same time, an Operations Leadership Plan has been developed that integrates the findings of previous assessments of Operations, industry experience, and the results of this Root Cause Analysis. The Leadership Plan represents an integrated response to strengthening Operations' leadership role at the site, and addresses necessary changes both within Operations as well as changes in the relationships between Operations and other site organizations.

These actions are substantially broader and more comprehensive than previous corrective actions. Davis-Besse should perform reviews to ensure that the corrective actions specified in this report are effective.

Root Cause Analysis Report 54 I I I

9.0 Root Cause Analysis Team The Root Cause Analysis Team consisted of two independent consultants who specialize in conducting root cause analyses and assessments of nuclear power plant organizational performance and a senior member of the Operations Department at Davis-Besse. The Team members were:

Valerie E. Barnes, PhD, Performance, Safety and Health Associates, Inc.

(President and Senior Research Associate), Team Lead - Valerie Barnes received her PhD in Social/Organizational Psychology from the University of Washington in 1985. She has managed or played a key technical role in numerous research and technical assistance projects for the U.S. Department of Energy (DOE),

the U.S. Nuclear Regulatory Commission (NRC), and other private sector and government sponsors undertaken to enhance the reliability of human performance.

She has assisted in developing and delivering training for DOE accident investigators and Board Chairpersons since 1995, assisted the U.S. Chemical Safety and Hazard Investigation Board to develop its investigation procedures and protocols, and recently published a guidance document to assist NRC inspectors in evaluating licensee investigations and root cause analyses for human performance problems.

Dr. Barnes has applied her expertise in more than 100 audits, inspections and event investigations that have addressed a variety of human performance issues.

Brian C. Haagensen, Performance, Safety and Health Associates, Inc.

(Managing Director and Executive Vice President) - Brian Haagensen is a senior management consultant with 30 years of experience in the nuclear industry. He has worked at 75% of the nuclear power plants in the country today. He was the project manager and lead expert for management and organization support to all NRC Diagnostic Evaluations Team Inspections from 1991 to 1995. He personally participated in three NRC Diagnostic Evaluations as the lead management and organization consultant (South Texas Project, Palisades and Maine Yankee). He was a certified NRC Operator Licensing Examiner from 1986 until 1995 and participated in over 75 exams throughout the country. He also supported numerous NRC inspections including emergency operating procedures inspections, Augmented Inspection Teams, Emergency Preparedness Inspections, Exercise Evaluations, Part 21 vendor audits and training inspections. He has an extensive background in corrective actions including support of the Indian Point corrective action program self assessment and preparations for the NRC 's 95003 multiple degraded cornerstone inspection. He has received formal training in root cause assessment techniques including MORT, Kepner-Trego, and HPIP. He was the lead operations representative on the NRC's shift staffing study conducted by Brookhaven National Laboratory, He co-authored the NRC's Human Performance Evaluation Process (HPEP) NUREG/CR-6251. He has a Masters of Science degree in physics and was a nuclear submarine officer from 1974 to 1982.

Root Cause Analysis Report 55

Douglas Ricci - FENOC (Davis-Besse, Operations Supervisor) - Douglas Ricci has worked in the field of nuclear energy for 36 years and for the Toledo Edison/Centerior Energy/Pirst Energy organization for more than 31 years. Doug was licensed as a Reactor Operator for almost 2 years and as a Senior Reactor Operator for over 22 years. Doug has held the positions of Reactor Operator, Assistant Shift Supervisor, Shift Supervisor, and Supervisor - Operations while at Davis-Besse Nuclear Power Station. He is trained as an INPO Root Cause Coordinator, an INPO Human Performance Fundamentals Course facilitator, and is a qualified Root Cause Evaluator.

Root Cause Analysis Report 56

10.0 References The following is a list of references reviewed in preparation of this Report.

10.1 Davis-Besse References Procedures 0 DB-OP-00000, Conduct of Operations 0 DB-DP-00022, Station Review Board 0 DB-FP-000005, Fire Brigade 0 GP-01, Conservative Operations 0 GP-25, Shift Expectations 0 DSP-90-00016, Command Responsibilities

• NT-OT-07007, Fire Brigade Training 0 NG-DB-00302, Davis-Besse Nuclear Power Station Fire Protection 0 NOP-OP-1001, Clearance/Tagging Program

• NOP-LP-2001, Condition Report Process

• Operations Standing Order 00-006, Configuration Control Action Plan 0 Operations Standing Order 00-011, Interim Operations Performance Requirements 0 Operations Standing Order 01-008, Failure to Perform DB-SC-03023 when EDG #2 Paralleled to D1 Bus

• Operations Standing Order 02-006, Interim Safety Tagging Guidance 0 Corrective Action Program Reference Guide Rev 5 9 Nuclear Operations Admin-1 Rev 17

• Delegation of Authority Admin-9 Rev 20

• Corrective Action Tech-3 Rev 18 0 Root Cause Analysis Tech-26 Rev 1

• Condition Report Process - Programmatic Guideline

• Charter - Corrective Action Review Board

• Charter - Davis-Besse Project Review Committee

• Davis-Besse Project Review Group Charter Rev 3 and Rev 4

• Duty Shift Supervisor Duties and responsibilities Tech-19 Rev 19

• FENOC - Root Cause Analysis Reference Guide Rev 3

• Corrective Action Policy, Tech-3 Rev 0 and Rev 15 thru Rev1 7 Potential Condition Adverse to Quality Reports

• 1998-0020, RC-2 with Root Cause Analysis Report

• 1998-0046, Insulation for RC-2 Removed for Inspection and Not Reinstalled 1998-0649, Inspection Results of Reactor Vessel Head Root Cause Analysis Report 57

• 1998-0650, Video Inspection Results CRDM Nozzle/Head Interface

• 1998-0767, Reactor Vessel Head Inspection Results

• 1998-0824, CAC's 2 and 3 Have Accumulated Boric Acid

• 1998-0915, Yoke on RC-2 is Corroded

• 1998-1642, Apparent Missing Nut

• 1998-1681, Missing Body to Bonnet Stud Nut

• 1998-1716, Functional Evaluation of RC-2 for Past Operability

• 1998-1797, Operations Accreditation Team Findings

• 1998-1799, RC-2 MWO Package Discrepancies

• 1998-1885, RC-2 Carbon Steel Nuts

• 1998-1887, Nut in Containment

• 1998-1904, 1998 Collective Significance Review

• 1998-1924, Functionality of RC-2 as a RCS Pressure Boundary

• 1998-1980, Containment Cooler Plenum Pressure Decreasing

• 1998-1981, HP-005ý Body to Bonnet Bolting

• 1998-1988, RC-2

• 1998-0020, Multiple Problems with RC-2 Condition Reports

• 1999-1614, LER 1998-009

• 2000-1001, RC-2 Spray Valve Problems

• 2001-1747, CARB Charter Compliance

• 2001-1748, Corrective Action Review Board Recommendations

• 2001-2862, Potential Adverse Trend in Unidentified RCS Leakage

• 2001-3025, RCS Leakage 0 2002-02584, Implementation of Corrective Action Program By Site Personnel

• 2002-02585, Management and Supervisory Oversight and Ownership of Plant Activities Audits

• Audit Report AR-00-ONF-01

• Quality Assessment Audit Report AR-02-OUTAG-01 Job Descriptions

• Plant Manager Davis-Besse Plant Operations

• Manager - Operations Davis-Besse Plant Operations

• Manager - Maintenance Davis-Besse Plant Operations

• Manager - Plant Engineering Davis-Besse Plant Operations

• Manager - Design Basis Engineering Davis-Besse Engineering and Services Manager - Quality Assessment Davis-Besse Nuclear Assurance Root Cause Analysis Report 58

0 Manager - Radiation Protection Davis-Besse Plant Operations 0 Director - Engineering and Services Davis-Besse Engineering and Services

• Open Position Announcement - Manager, Nuclear Outage

• Open Position Announcement - Manager, Nuclear Environmental and Chemistry Other Station Documents Human Resources Performance Management ownership for Excellence Guideline Forms: Executive level Competency/Behavior Forms 0 Human Resources Performance Management ownership for Excellence Guideline Forms: Manager level Competency/Behavior Forms 0 Human Resources Performance Management ownership for Excellence Guideline Forms: Supervisor level Competency/Behavior Forms 0 Human Resources Performance Management ownership for Excellence Guideline Forms: Individual Level Competency/Behavior Forms

• Root Cause Analysis Report Significant Degradation of the Reactor Pressure Vessel Head, CR 2002 -891 (Management and Human Performance Root Cause Analysis Report)

• Management and Human Performance Excellence Plan, September 5, 2002 0 Minutes of the October 29, 2001 Shift Manager Team Meeting

• Minutes of the October 26, 2002 Shift Manager's Meeting

• FENOC Nuclear Market Survey, Operations 2002 0 E-mail from Dee Laberdee Haskins to Doug Ricci, 11/12/02, Compensation Matrix 0 MPO-00-013, Operations' On-Shift Staffing, 6/9/99

• Nuclear Group Staffing Report, 1991-1997

• Operations Overtime Call-Out Reports (1998-2002) 10.2 NRC References

1. RC-2 NRC Special Inspection Report 350-346/98021

2. SEN 190, Pressurizer Spray Valve Bonnet Nuts Dissolved by Boric Acid Leak

3. Baker, T. (1995). Alertness, performance and off-duty sleep on 8-hour and 12-hour night shifts in a simulated continuous operationscontrol room setting (NUREG/CR-6046). Washington, DC: U.S. Nuclear Regulatory Commission.

4. Haber et al. (1995). Nuclear Power Plant Shift Staffing Levels: Site Data Collection Report (Accession No. 9510030160). Upton, NY:

Brookhaven National Laboratory.

5. Information Notice 79-20, NRC Enforcement Policy - NRC Licensed Individuals Root Cause Analysis Report 59

6. lE Circular 81-02, Performance of NRC-Licensed Individuals While on Duty

7. Information Notice 85-53, Performance of NRC-Licensed Individuals While on Duty

8. Information Notice 91-77, Shift Staffing at Nuclear Power Plants

9. Information Notice 95-23, Control Room Staffing Below Minimum Regulatory Requirements

10. Inspection Manual, Inspection Procedure 71707, Plant Operations

11. Inspection Report 50-346/02-03, NRC Augmented Inspection Team Degradation of the Reactor Pressure Vessel Head

12. Policy Statement on the Conduct of Nuclear Power Plant Operations (rec'd 1/23/89)

13. Regulatory Guide 1.114, Guidance to Operators at the Controls and to Senior Operators in the Control Room of a Nuclear Power Unit (May, 1989, Rev. 2)

14. Regulatory Guide 1.8, Qualification and Training of Personnel for Nuclear Power Plants (May, 2000, Rev. 3)

15. Shurberg, D et al. (1994). Identificationof Issues Associated with Nuclear PowerPlant Shift Staffing Levels, Task 1 Letter Report (Accession No. 951003074). Upton, NY: Brookhaven National Laboratory.

16. 10 CFR 55, Operators' Licenses

17. 10 CFR 50.54, Conditions of Licenses

18. NUREG-0800, Chapter 13, Operating Organization 10.3 INPO References

1. ACAD 97-004 Guidelines for Shift Manager Selection, Training and Qualification, and Professional Development

2. INPO Excellence in Human Performance (draft, August, 1997)

3. INPO 01-002 Guidelines for the Conduct of Operations at Nuclear Power Stations

4. OE 9944 Lack of Operator Awareness of Plant Equipment Status

5. SOER 81-12 Reactor Coolant Pump Closure Stud Corrosion

6. SOER 84-5 Bolt Degradation or Failure in Nuclear Power Plants

7. SOER 98-1 Safety System Status Control

8. SER 1-97 Nonconservative Operations during Isolation of a Reactor Recirculation Pump Seal Leak

9. SER 93-28 Increased Leakage from an Unisolable Reactor Coolant Leak after Repeated Sealant Injections

10. SER 46-80 Reactor Coolant Pump Closure Stud Corrosion

11. SER 35-81 Corrosion of Reactor Coolant System Piping

12. SER 11-82 Reactor Coolant Pump Closure Flange Stud Corrosion

13. SER 57-83 Cracking in Stagnant Boric Acid Piping

14. SER 72-83 Damage to Carbon Steel Bolts and Studs on Valves in Small Diameter Piping Caused by Leakage of Borated Water Root Cause Analysis Report 60

10.4 Other References

1. RHR International Davis-Besse Phase 2 Organization Study Results June - July 1999

2. Baker, K., Olson, J. and Morisseau, D. (1994). Work practices, fatigue, and nuclear power plant safety performance. Human Factors,36, 244 257.

3. Fiedler, F.E. (1967). A theory of leadership effectiveness. New York:

McGraw-Hill, 1967.

4. Howlett, H.C. (1995). The industrialoperator'shandbook: A systematic approachto industrialoperations. Pocatello, ID: Techstar.

5. Landy, F. and Trumbo, D. (1976). Psychology of work behavior.

Homewood, IL: The Dorsey Press.

6. Loftus, E. (1996). Eyewitness testimony. Harvard University Press.

7. Merritt,A.C. and Helmreich, R.L. (1996). Creating and sustaininga safety culture. CRM Advocate, 1, 8-12.

8. Muschara, T. (2000). INPO Human Performance Evaluations and Other Developments. In Proceedingsof the 2000 Human Performance Root Cause Trending Workshop. Philadelphia, PA: June 12-15, 2000.

9. R.C. Brown and Associates, Management Consultants (December 31, 1996). Final Report of the Focused Audit of the CL&P Nuclear Operations, R. C. Brown and Associates, prepared for the Connecticut DPUC.

10. Reason, J.T. (1990). Human error. Cambridge: Cambridge University Press.

11. Reason, J.T. (1997). Managingrisks of organizationalaccidents.

Burlington, VT: Ashgate Publishing.

Root Cause Analysis Report 61

Enclosure 3 Assessment of Company Nuclear Review Board

ASSESSMENT OF THE FENOC COMPANY NUCLEAR REVIEW BOARD Prepared for First Energy Nuclear Operating Company Prepared by Darrell G. Eisenhut August 13, 2002

Assessment of the FENOC Company Nuclear Review Board 1.0 Purpose and Approach This Report summarizes the results of an assessment, conducted during July-August 2002, of the FENOC Company Nuclear Review Board. The Purpose of this Assessment was stated to be:

Assess the Company Nuclear Review Board's (CNRB) past and going forward oversight role as it relates to the missed opportunity for identifying the reactor vessel head issue. Provide recommendations for improving the safety focus of the CNRB.

A large amount of material was collected and reviewed to better understand the FENOC requirements for the CNRB, and how the CNRB presently functions. These addressed its structure, composition, expertise of members, attendance, reporting, etc. A listing of documents reviewed is attached to this Assessment. A meeting of the CNRB (at Beaver Valley on July 16-17, 2002) was attended to obseve the functioning of the CNRB. Meetings, discussions, and interviews of CNRB members, management and staff were conducted, both at Beaver Valley, and also at Davis Besse during the week of July 22, 2002.

In addition, considerable information relating to the actual Davis-Besse reactor head degradation issue was reviewed including the sequence of events, selected Condition Reports, the Root Cause Report, the Return to Service Plan, and numerous other supporting documents. Also reviewed were several memoranda of a FENOC staff member (J Hultz) who was on distribution for CNRB information and who provided his reaction to that information to FENOC management. While this review of information provided a broad-based perspective on the history of the reactor vessel head issue, the scope of this assessment was limited and focused on understanding information provided to the CNkB, the information available to the CNRB, and the CNRB's response to that infornriation.

August 13, 2002 I

2.0 FENOC Requirements for CNRB The requirements for the Davis Besse CNRB are contained in Section 13.4.2 of the plant's Updated Safety Analysis Report (USAR). The USAR (Rev. 22, dated 11/2000) states that the CNRB "shallfunction to provide independent review and audit of designated activities... "

The USAR further states that the Company Nuclear Review Board shall review a number of technical areas, including:

f Significant operating abnormalitiesor deviationsfrom normal and expected performance of plant equipment that affect nuclear safety and,

h. All recognized indicationsof an unanticipateddeficiency in some aspect of design or operation of safety related structures, systems, or components.

I The FENOC Policies and Practices (Rev 14, effective 05/06/2002) for the Company Nuclear Review Board support, and elaborate on, the basic charter. It specifically notes:

- Section 2.0 - "The CNRB shallfunction to provide an independent audit and review of plant activities to assure itself" that the stations are "being safely operated and maintained "

- Section 4.2.1 indicates that the CNRB is responsible for "Informing the FENOC PresidentNuclear on any CNRB issue relative to the safe operation of the Davis Besse, Perry or Beaver Valley" plants, and

- Section 9.1 states that "CNRB members individuallyperforming review of an activity or action shall not have been responsiblefor that activity or action under review."

The FENOC Quality Assurance Program manual contains similar requirements.

August 13, 2002 2

3.0 Implementation of the CNRB function The implementation of the FENOC CNRB function generally follows the guidance in the company's USAR and Policies and Procedures document. For example:

- The CNRB is composed of both internal and external individuals having the required expertise,

- CNRB members routinely receive a wide variety of plant specific information for review in advance of meetings,

- CNRB meetings are held on a periodic basis at each station to review safety issues and to discuss overall performance (currently six times per year; twice at each station; for a minimum of three days each),

- CNRB meetings are reasonably well attended by both internal and external membership, and minutes indicate that a quorum was present,

- External consultant members periodically travel to the plants to observe activities and have discussions,

- External members review materials while "offsite" in preparation for discussions,

- The CNRB currently has one industry peer advisor as a member,

- In 2001, each external member devoted a significant amount of time (on the average about 480 hours0.00556 days <br />0.133 hours <br />7.936508e-4 weeks <br />1.8264e-4 months <br />) to the review of materials to support the CNRB review function, and

- There is no readily available means to estimate internal member's efforts.

August 13, 2002 3

4.0 Information Available to the CNRB The CNRB did not identify the severely degraded reactor vessel head condition present at Davis-Besse. Records also suggest that the CNRB did not inform senior company management of the issue, nor provide any specific recommendation regarding it to FENOC management.

A review of CNRB materials, including meeting minutes, 'hand-out' and presentation materials, and Subcommittee minutes, indicates there were several discussions at CNRB meetings that suggested a primary system degradation issue existed at Davis Besse. Examples of information provided to CNRB members and discussion topics at CNRB meetings included:

Boric acid plating out on containment air coolers (1-99),

Updates on reactor coolant system unidentified leakage (5-99, 3-01),

Frequent required change out of radiation filters (7-99),

Presence of iron deposits on filters and a requested SRI analysis (7-99), and Information that all nozzles were not inspected in previous outages (11-01).

Other supporting information that was available in review documents was more specific:

- Indications of rust or brown stained boron on the head (4-96),

- A finding of 'fist size clumps' of boric acid (4-98),

- A Southwest Research report noting iron oxide and corrosion (7-99),

- Red/brown boric acid leakage being noted (4-00),

- A discussion of accumulation of boron on the head (4-00), and

- A discussion of the need for a Project Plan for J-groove cracking (5-01).

(A more detailed listing of selected information that was made available to CNRB members is set forth in Attachment A.)

August 13, 2002 4

The availability of this information must be viewed in perspective. The CNRB received information on these Davis-Besse plant-specific issues during the same time period that the industry (including the CNRB members) was receiving operating experience reports of boric acid corrosion problems in the industry, and receiving several communications from the NRC identifying concerns regarding boric acid corrosion of carbon steel in PWRs. While this occurred over a long period of time dating from the 1980s, it was particularly focused on cracking of PWR vessel head penetration nozzles beginning in about 1997 and continuing through 2001. In fact, the NRC Information Notices, Bulletins, and Generic Letters were issued to call particular attention to the potential problems, and Davis-Besse was identified as a plant susceptible to the issue.

While this information was made available to the CNRB, no specific questioning or follow-up on the issue appears to have occurred by CNRB members. While there was limited discussion of the plant's reactor coolant leakage, the collective significance of the many indicators of problems appears to not have received appropriate safety focus. It should also be noted that plant management did not identify the collective significance of these indicators regarding boric acid on the reactor vessel head, or at least did not so inform the CNRB.

In summary, it is apparent that CNRB members received plant-specific information (and that additional relevant information was available if requested) and that considerable industry communication was provided regarding this issue, that should have suggested an ongoing degradation issue or concern related to the integrity of the reactor primary coolant system. Given the several specific alerts received from other nuclear industry organizations and those received formally from the NRC, it is reasonable to expect that the CNRB should have raised questions regarding the boric acid corrosion issue.

August 13, 2002 5

5.0 Conclusions and Recommendations The following summarizes the conclusions of the assessment and any associated recommendations.

" The CNRB failed to identify the reactor vessel head degradation issue. While the CNRB is an important safety review function for the station, it is but one of the several levels of safety protection that failed. The CNRB process failed and as such the issue should be viewed as broader that one "missed opportunity."

Senior management should ensure that all CNRB members recognize that the CNRB function failed to provide the expected level of safety protection at the station. Management should guard against any defensive reaction (e.g.,

"no one told us of the problem") and focus on the need to ensure that steps are taken to improve the chances of identifying such a concern if one were to occur in the future.

" Based on an observation of a CNRB meeting, the "Expectations" for the CNRB appear to not be clear. An observer has some difficulty is determining whether the CNRB meeting is more typical of a management review committee meeting, or a meeting whose primary purpose is to focus on technical/safety topics at the plants. In the CNRB meeting attended, there was generally too much attention directed at how better to manage the station, on reviews of technical matters of limited significance, and on somewhat administrative matters. There was less focus on technical and operational issues, and on potential safety issues.

It is recommended that senior management review the functioning, the structure, and membership of this important safety review entity for the station. Management should establish, or reiterate, the expectations for the CNRB particularly clarifying that its role is primarily a safety-focused organization and not a management unit of the station.

August 13, 2002 6

0 The CNRB meeting and the information presented and reviewed, had little focus on operational, technical, and safety topics. A review of presentations and meetings minutes from recent meetings indicates that they are generally superficial and not focused on specific operational or technical challenges for the station. They instead focus on general, more production-oriented indicators, with little emphasis on technical/safety oriented performance indicators.

I Senior management should ensure that the CNRB's focus is primarily on "safety, or matters that could directly affect safety." Production-oriented presentations should be avoided. Briefings should be re-directed towards operational, technical, or safety "issues" challenging the station. Specific factual information to support such issues should be presented, including better use of appropriate performance indicators.

The CNRB may not be meeting its charter to provide an independent safety audit function. It was indicated that certain of the "external" CNRB members are providing both "consulting services" to Davis-Besse, as well as serving in the "independent auditor" capacity. Such an intertwined dual consulting and independent auditing approach for some members may have led to a situation where the external members have lost their ability to objectively view information and avoid the "group-thinking" results. Members may have grown to be too close to the plant, its condition, and its management, and may now even "own" an aspect of performance problems. Certain plant conditions may not be as "obvious" to them as they could be for an 'independent" auditor. It was not possible to determine the extent to which this may have compromised the "independent audit" function as required by the USAR and procedures.

August 13, 2002 7

It is recommended that senior management clarify the roles and responsibilities for all CNRB members to ensure the integrity of the CNRB process. This should ensure that all members strictly adhere to the "arms length" independence requirement to better enable them to identify indicators of plant or performance deficiencies. CNRB members, including external members, should not even have an appearance of a conflict, and should be able to review various performance reports, including Condition Reports, apparent and root cause reports, plant operating experience and deficiency reports without having any prior involvement in the preparation of those reports.

A major source of "input" information for CNRB review should be the Nuclear Quality Assurance (QA) organization, and the supporting Corrective Action Program (CAP). It is recognized that separate internal assessments of each of these programs is underway within FENOC. Based on the review of several years of CNRB information, it does not appear that he CNRB effectively utilized this information and did not adequately review the health or effectiveness of either the QA or CAP program. In addition, it was not apparent that CNRB was effectively overseeing the QA Audits that are to be "performed under the cognizance of the CNRB."

It is recommended that the CNRB be specifically required to review the effectiveness of the QA and CAP programs on an ongoing basis at every meeting. QA should be required to inform the CNRB of the issues that QA has highlighted as major focus areas or concerns. Information regarding "what the CAP is telling the station" should routinely be provided to the CNRB. This information should help the CNRB define areas requiring attention. Management should establish clear guidelines to ensure that the CNRB conducts a rigorous, open review of the adequacy of the implementation of these programs, particularly since the CNRB has the responsibility to oversee and review the QA process.

August 13, 2002 8

" The present Chairman of the CNRB also has Nuclear Quality Assurance (QA) line management responsibilities. Since this approach has inherent conflicts, extreme care should be taken to ensure the proper relationship between nuclear QA and the CNRB.

Since the CNRB has the responsibility to oversee and review the QA function, appropriate information of QA findings, including audits, assessments, and field observations should routinely be provided to the CNRB members. Because QA in effect is "Chairing" the CNRB (a group charged with overseeing QA) the full CNRB must maintain additional focus on the effectiveness of the QA program.

" The unique, important role of the Operations department in guiding and directing overall station safety performance at the station was not apparent in CNRB discussions.

The emphasis on Operations (and supporting functions) by the CNRB should be strengthened. Consideration should be given to having attendees from Operations at meetings, the inclusion of Operational issues in meeting briefings, and a stronger role for Operations in Subcommittee meetings.

" Recognizing the present status of the FENOC stations, particularly Davis-Besse, the structure, meeting format, subcommittee structure and membership, and meeting frequency do not appear to be effective and efficient, and not optimal for conducting a rigorous safety review of performance at the stations. While it was concluded that certain of the approaches (e.g., using subcommittees) are similar in concept to those used throughout the nuclear industry, it was concluded that adjustments should be considered to improve the effectiveness of this safety review function.

August 13, 2002 9

It is recommended that FENOC management consider several adjustments to the CNRB as it now functions:

1. Clearly communicate management's view of the importance of the CNRB function, including the expectation that all members are expected to attend all meetings.

2. Reinforce that the principal focus of the CNRB in on maintaining personnel and worker safety, and then conduct the CNRB meetings in that manner. Ensure that all full CNRB meetings and Subcommittee meetings are conducted in a professional manner, all people are treated with respect, and that all views are presented.

3. The Full CNRB meeting agenda should be revised to allow adequate time for the key areas requiring focus. This should include additional emphasis on issues/findings within Operations.

4. The role of QA in supporting the CNRB should be clarified and integrated into the CNRB. The CNRB responsibility to review certain QA Audits conducted under its cognizance should be clarified.

5. The Subcommittee structure should be realigned to focus on the major functions of the stations: Operations, Maintenance, Engineering, and Regulatory & Oversight. All other station functions, e.g., industrial safety, radiation protection, chemistry, etc. can be assigned to one of these Subcommittees.

6. Strive to ensure that CNRB Subcommittee Chairman have prior actual plant experience in the key plant discipline where they serve (e.g., have a member with prior plant senior Operations experience serve as Chair of the Operations Subcommittee). The role of the Chair should be clearly defined and include being the "quarterback" of the subcommittee, the consensus builder, and should strive to encourage station personnel to identify, recognize, and develop solutions to their own issues.

August 13, 2002 10

7. External members should specifically be required to remain independent of other station work activities.

8. Expectations for Subcommittees should be clarified to require having membership from each of the other stations at every meeting.

Subcommittees should focus significant attention at the worker through mid-management level to better understand activities at the station.

9. Subcommittee Chairman should be given the flexibility to establish the agenda for the Subcommittee day, that is, the one day before the Full CNRB meeting. That agenda should be established based on prior material that was provided, and issues suggested by the subcommittee members, by management, by QA, or other means. They should not be required to all focus on only one particular focus area topic, thereby precluding a more thorough review of station performance.

10. Subcommittees should function as a "Team" and should be required to be "in the plant" for every meeting. As such, they should be able to provide their observations on issues of workers in the plant, material condition aspects, etc.

11. Information routinely provided to CNRB members should be reviewed to ensure it is adequate to allow the CNRB to reach meaningful conclusions.

Members should be requested to help with this re-evaluation.

Information provided should include the principal findings of QA and from the CAP process. Condition Reports of some level of significance should be provided to the Subcommittees for their review and consideration prior to the meetings.

12. Presentations to the CNRB should more clearly be focused on technical, operational, and potential safety.issues, and should provide enough information to allow the CNRB to reach meaningful conclusions.

August 13, 2002 I1I

13. Management should attempt to increase the use of peer-to-peer support for the CNRB. While job responsibilities often make it difficult for such members to participate for more than one day of a meeting, it should be considered since it helps avoid becoming isolated, can provide valuable information, and can support personnel development.

14. Consideration should be given to increasing the number of meetings per year for each station. Many plants in the US now have four regular meetings per year, although some have recently begun to transition to three. Meetings are often of two days of total duration, the first day being subcommittees and the second the Full Meeting, although some plants have 2 1/2 or 3 day meetings, or even longer. These are regularly scheduled meetings, and are regularly supplemented by telephone conference meetings to review technical specification changes. Plants in some form of difficulty (such as Davis-Besse) usually have additional meetings.

15. Consideration should be given to having the Full CNRB periodically review progress on the Davis-Besse Return to Service effort, and in particular, to conduct a formal review of the Readiness to Restart. The Charter of the CNRB should be carefully reviewed in this regard as one reading of the USAR would suggest that this is required. (See Section 2.0)

16. CNRB required reviews, e.g., technical specification changes, should not be allowed to be a major distraction to the technical or safety focus of the CNRB meeting. These required reviews can be accomplished by a variety of other means, including telephone conferences. Members could be encouraged to discuss questions or concerns with preparers prior to, and outside of, the full CNRB meeting. Large programmatic reviews (e.g.,

ITS, power uprate, license renewal) can be assigned to a subcommittee review with the full CNRB reviewing the Subcommittee performance. In all cases, the CNRB should follow its Charter and function as an "Audit" function, and not attempt to become a detailed review organization.

August 13, 2002 12

17. Several Administrative Processes used by the CNRB should be reexamined. These include:

a. The process used by the CNRB to "approve" an item should be reconsidered and realigned to better follow current industry practice.

For example, the present practice of "voting" raises administrative questions over just what is being voted on, whereas the emphasis should be on whether or nor any concerns or potential safety questions were identified by even one person.

b. The process of assigning "Actions" should not be allowed to become an administrative burden, since the stations already have a rigorous process for tracking actions, and the CNRB itself should certainly be able to establish its own follow-up actions. Subcommittees should not feel the need to assign administrative actions to themselves.

c. Recent minutes of CNRB meetings have been from 30 to 50 pages in length. It is doubtful that many people read the minutes or find them of value. It is recommended that they be greatly reduced and written more focused on important issues from the meeting. In addition, it is recommended that an Executive Summary (of no more than about two-pages in length) be written for senior management.

August 13, 2002 13

Attachment A Information Available to the CNRB: 1999 - present Information was reviewed to determine whether or not the CNRB members had available sufficient information to permit them to recognize the existence of indicators of a potential problem at Davis-Besse.

CNRB members receive information from a number of sources. Internal members are involved in the day-to-day activities and as such continuously receive an extremely large amount of information. Typical sources for external members include formal briefings and presentation material provided to both the full CNRB meetings and at Subcommittee meetings; information received during discussions; tours of the plant, information requests based on material sent in routine "mailings" to each member, and input from discussions with "workers" and first-line supervisors.

In addition, a very large amount of information including audits, inspection reports, operating experience reports, and plant Condition Reports is available for review and follow-up questions.

Members are also expected to stay current with other events within the nuclear industry, and as such should have been well aware of boric acid leakage issues in general, and particularly the more recent reactor vessel head leakage issues at other plants in the US. The industry has had historical problems with different types of boric acid leakage at plants, and significant problems were found at plants in the US.

The NRC has kept plants (and experts) informed of these issues, and regarding the more recent concern regarding CRDM J-groove cracking, Davis-Besse was identified as one of the more susceptible plants..

While it probably does not present a complete picture of all information presented or discussed at the CNRB meetings, a review of formal meeting minutes, 'hand-out' presentation material, and plant update information was reviewed. In addition, other August 13, 2002 14

information that was readily available to CNRB members at the time, for example, Audits and Condition Reports, was reviewed. In general, records from about 1999 to the present time were reviewed because they were more readily available.

The following summary highlights information available for CNRB review.

April 1, 1997, NRC Generic Letter 97-01, Degradation of Control Rod Drive Mechanism Nozzle and other Vessel Closure Head Penetrations January 7, 1999 - The 'hand-out' material for the Davis-Besse CNRB meeting included a discussion of boric acid plating out on the containment air coolers.

The Engineering/Licensing Subcommittee meeting summary notes that the CNRB had information that the reactor inventory leakage was currently higher than historical values, and that the upcoming May outage would provide an opportunity to identify sources of reactor coolant system leakage.

May 27, 1999 - The update material provided to the CNRB again includes the discussion that the reactor coolant system leakage was about 0.3 gpm.

July 21, 1999 - The minutes for the CNRB meeting states that the CNRB was "providedan update of the Reactor Coolant (RC) leak status and problems being encounteredwith containment radiationmonitors requiringfrequent change out of theirfilters. "

The minutes further indicate that "Chemistry identified the presence of iron on the filters. Southwest Research has been contractedto perform an analysis." The

'hand-out' material for the meeting also notes that concern 'still exists' regarding the frequency of filter changes required for the containment radiation monitors.

Containment entry was planned to pin-point problem areas.

[Note that while the CNRB was informed of "iron" on filters as early as 1999, the Southwest Research report was subsequently available for review and confirmed that Iron was an indicator of corrosion.]

In addition, CNRB Subcommittee minutes indicate that several important issues were all discussed: (1) an unidentified reactor coolant system leakage of 0.24 gpm, other identified leakage, and leakage over a long period of time; (2) a discussion of a build-up of boron on the containment air coolers and the belief that it was due to a water, not steam, leak; and (3) a discussion of radiation monitors and the need, coming out of the outage, to change filters every two days.

June 1, 2000 - The minutes for the CRNB meeting include a simple statement that "Outage efforts to fix RCS leakage have been successful. " There does not appear to have been any additional information to support, or any questions about this conclusion.

August 13, 2002 15

March 29, 2001 -The minutes for the CRNB meeting includes a specific discussion where plant management reviewed "the Plant'sproblems with Reactor Coolant unidentified leakage causing boric acid build-up on the Containment Coolers and the need to periodically enter containment to clean the coolers" and that "maintenance is looking at improving our cleaning efficiencies, " There is one CNRB member response "that the root cause is the leak and suggested that the focus needs to be on fixing the leak."

It is noted that this is the first time in the years of records reviewed, that there is a clear statement focusing on the safety issue, and it was nearly a year before the actual issue was identified, but there does not appear to have been any follow-up.

April 30, 2001, NRC Information Notice 2001-05, Through-Wall Circumferential Cracking of Reactor Pressure Vessel Head Control Rod Drive mechanism Penetration Nozzles at Oconee Nuclear Station, Unit 3.

August 3, 2001, NRC Bulletin 2001-01, Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles November 29, 2001 - The minutes for the CNRB meeting includes a discussion of the ongoing dialogue with the NRC regarding the drafting of a shutdown order to shutdown Davis-Besse before the end of the year. That summary notes a discussion on status, the comprehensive actions already taken by Davis-Besse, the need for contractor mobilization, and even a "political problem".

That same summary does not note any probing, safety-oriented questions or concerns from any CNRB members. This is noteworthy since the 'analysis section' of those same minutes clearly indicates that the evaluation is based on visual inspections and that "The inspection results afford us assurancethat all but 4 nozzle penetrations were inspected in 1996. All but 19 penetrationswere inspected in 1998. And all but 24 penetrations were inspected in 2000."

In addition, a number of Condition Reports (CR) were available for review as a follow-up to the discussions at the full CNRB and Subcommittee meetings. These CRs, also referred earlier to PCAQR reports, were rather specific in highlighting the issues. Examples include:

- PCAQR 96-551 (4/21/96) - Discusses the video tape of the CRDM nozzle inspection showing "several patches of boric acid accumulation on the RV head."

It also notes that the reactor head has "rust or brown stained boron." The document discusses the walk-down inspection requirements and the write-up notes Davis-Besse deficiencies relative to other B&W plants.

- PCAQR 1998-0767 (4/24/98) - Discusses the results of the video inspection that "indicated several fist size clumps of boric acid." It also noted "rust brown to white" lumps.

August 13, 2002 16

CR 1999-0882 (7/9/99) - Discusses the Southwest Research report (included) analysis of Iron on filters of radiation monitors, and cOhcludes "the iron oxide deposits are like corrosion products from an iron base compound within the system." This is the result of analysis work discussed at the July 21, 1999 CNRB meeting.

CR 2000-0782 (4/6/00) - Discussed the "red/brown" boric acid leakage from the weep holes, and included pictures. It further notes that Framatome completed the video inspection, and personnel from Framatome "examined the results of the inspection."

CR 2000-1037 (4/18/00 - This CR was a follow-up to CR 2000-0782 and discussed results of the Inspection of the Reactor head indicating "accumulation of boron in the area of the CRD nozzle penetrations through the head." It discussed the process for removing the boron deposited between the reactor head and the thermal insulation.

CR 01-1 191 (5/2/01) - This identified the need to develop a Project Plan to prepare David-Besse for response to a cracked CRDM J-groove weld following experience from Oconee and ANO.

There were also documents that provided a "mixed" message regarding inspection, cleaning, and management of the boric acid issue. The along with the other material noted may well have been the cause for an increased questioning attitude.

- Toledo Edison Quality Assessment Audit AR-OO-OUTAG-01 (7/7/00) - The Audit notes as a Positive Attribute within Engineering the "Aggressive cleaning of boric acid accumulation from Rx head" and "engineering displayed noteworthy persistence in ensuring boric acid accumulation for the reactor head was thoroughly cleaned."

- The Davis-Besse System Health Report, 4th Quarter 2001, includes information on several relevant issues. Sections on primary system leakage, containment coolers, and radiation monitors all mention the issue. It was noted that "J-groove weld failure can not be ruled out," analysis indicate the "presence of Boron from an active leak," and the need for frequent filter change-out in the radiation monitoring system.

August 13, 2002 17

Attachment B Documents Reviewed

1. Company Nuclear Review Board Policies & Practices, Rev. 14, FENOC, effective May 6, 2002

2. FENOC Company Nuclear Review Board Value Assessment, 2001, approved April 8, 2002

3. Company Nuclear Review Board (CNRB) Value Assessment, 2000, approved in 2001, but not signed or dated

4. CNRB White Paper, Short Summary, undated, received from A. J. VanDenabeclc in June 2002

5. CNRB Self Critique, July 20, 1999

6. Company Nuclear Review Board, Meeting Minutes, Davis-Besse

- Meeting of Sept. 3, 1998 (15 pages)

- Meeting of January 7, 1999 (17 pages)

- Meeting of July 22, 1999 (19 pages)

- Meeting of Sept. 2, 1999 (13 pages)

- Meeting of January 13, 2000 ( 20 pages)

- Meeting of June 1, 2000 (22 pages)

- Meeting of Oct 31, 2000 (33 pages)

- Meeting of March 29, 2001 (51 pages)

- Meeting of Nov 29, 2001 (45 pages)

- Meeting of May 23, 2002 (40 pages)

7. Letter, dated April 18, 2002, from H. W. Bergendahl to US NRC; transmitting Root Cause Analysis Report, dated 4-15-2002.

8. US NRC Inspection Report 98-002, for Perry Nuclear Plant, dated 3-12-1998

9. US NRC Inspection Report 96-006, for Perry Nuclear Plant, dated 12-9-1996

10. US NRC Inspection Report 99-01i, for Perry Nuclear Plant, dated 11-30-1999 II. FENOC Nuclear Quality Assessment - Perry, dated February 14, 2002

12. The Illuminating Company, Audit Report, Joint Utility Management Assessment, dated 3-2-1998

13. The Illuminating Company, Audit Report, Joint Utility Management Assessment, dated 2-14-2000

14. Letter, dated, June 26, 1997, Centerior Energy to the US NRC, Reply to Notice of Violation

15. CNRB Engineering & Licensing Subcommittee Minutes, meeting 99-001, January 9, 1999.

16. CNRB Engineering & Licensing Subcommittee Minutes, meeting 99-002, July 21, 1999.

17. Davis-Besse System Health Report, 4 th Quarter 2001

18. Davis-Besse I IRFO Post Outage Report, dated September 9, 1998 and associated "Briefing Slides"

19. Davis-Besse 12RFO Post Outage Report (undated)

20. Davis-Besse CNRB Value Assessment for 2000, Attachment A details

21. Beaver Valley CNRB Value Assessment for 2000, Attachment A details

22. Perry CNRB Value Assessment for 2000, Attachment A details

23. Davis-Besse CNRB Value Assessment for 2001, Attachment A details August 13, 2002 18

24. Beaver Valley CNRB Value Assessment for 2001, Attachment A details

25. Perry CNRB Value Assessment for 2000. Attachment A details

26. Davis-Besse Return to Service Plan (CD Version)

27. Handout of J. Martin, referenced in Davis-Besse CNRB Meeting Minutes of January 13, 2000 (page 13)

28. Davis-Besse CR 02-00846, "More Boron on Head than Expected," February 26, 2002

29. Davis-Besse CR 02-00685, "Boron Buildup on Reactor Vessel Head," February 21, 2002

30. Davis-Besse PCAQR 91-0353 regarding Boric Acid Corrosion Control, September 12, 1991

31. Davis-Besse PCAQR 96-551 regarding Boric Acid Accumulation, April 21, 1996

32. Davis-Besse PCAQR 98-0767 regarding several "Fist-sized" clumps of Boric Acid, April 25, 1998

33. Davis-Besse CR 2000-1037 regarding Boron accumulation, April 18, 2000

34. Davis-Besse CR 2000-0782 regarding "Boric Acid leakage from the weep holes," April 16, 2000

35. Davis-Besse CR 1999-0861 regarding recurring problem, sample lines full of water, May 10, 1999

36. Davis-Besse CR 1999-0372 regarding Unit Log Entries of radiation levels and possible RCS Leakage, March 6, 1999

37. Davis-Besse PCAQR 90-221, CRDM Flange Inspection Results, March 22, 1990

38. Davis-Besse CR 1999-0882, Boron Build-up on Containment Air Filter, July 7, 1999

39. Davis-Besse CR 1999-1300 regarding SRI analysis: Iron Oxide from corrosion on containment radiator monitor filters, September 23, 1999

40. Davis-Besse CR 01-1191 regarding CRDM Nozzle J-Weld Cracking Project Plan, May 2, 2001

41. Administrative procedure, Boric Acid Corrosion Control, NG-EN-00324, rev 3, May 29, 2002

42. Davis-Besse Quality Trend Summary, Ist Quarter 2002 Condition Reports, June 3, 2002

43. Summary of External Expert Hours billed to CNRB, Jan 2001 to present

44. CNRB Competency/Experience Matrix, Jun2 12, 2002, including resumes.

45. Davis-Besse Safety Culture Survey, 2002 Employee Survey Results, March 8, 2002

46. Davis-Besse Plant Update Briefing materials for CNRB, February 25, 1998.

47. Davis-Besse Plant Update Briefing materials for CNRB, April 23, 1998

48. Davis-Besse Plant Update Briefing materials for CNRB, September 2, 1998

49. Davis-Besse Plant Update Briefing materials for CNRB, October 21, 1998

50. Davis-Besse Plant Update Briefing materials for CNRB, January 6, 1999 5I. Davis-Besse Plant Update Briefing materials for CNRB, March 4, 1999

52. Davis-Besse Plant Update Briefing materials for CNRB, May 27, 1999 August 13, 2002 19

53. Davis-Besse Plant Update Briefing materials for CNRB, July 21, 1999

54. Davis-Besse Plant Update Briefing materials for CNRB, January 12, 2000

55. Davis-Besse Plant Update Briefing materials for CNRB, May 31, 2000

56. Davis-Besse Plant Update Briefing materials for CNRB, October 30, 2000

57. Davis-Besse Plant Update Briefing materials for CNRB, January 18, 2001

58. Davis-Besse Plant Update Briefing materials for CNRB, May 10, 2001

59. CNRB Presentation slides for Davis-Besse, March 27, 2001, H. Bergendahl

60. CNRB Presentation slides for Davis-Besse, July 20, 2001, H. Bergendahl

61. CNRB Presentation slides for Davis-Besse, October 4, 2001, L. Worley

62. CNRB Presentation slides for Davis-Besse, November 27, 2001, H. Bergendahl

63. CNRB Presentation slides for Davis-Besse, January 10, 2002, S. Moffitt

64. CNRB Presentation slides for Davis-Besse, February 28, 2002, J. Messina

65. CNRB Presentation slides for Davis-Besse, May 21, 2002, J. R. Fast

66. Toledo Edison QA Audit AR-00-OUTAG-0 1, July 7,, 2000

67. Minutes of CNRB meeting at Perry, February 28, 2002

68. Davis-Besse CR 02-03272, Multiple Failures to Comply with Regulatory Requirements, July 17, 2002

69. NRC Information Notice 80-027, Degradation of Reactor Coolant Pump Studs, June 11, 1980

70. NRC Information Notice 82-006, Failure of Steam Generator Primary Side Manway Closure Studs, March 12, 1982

71. NRC Bulletin 82-002, Degradation of Threaded Fasteners in the Reactor Coolant Pressure Boundary of PWR Plants, June 2, 1982

72. NRC Information Notice 86-108, Supplement 1, Degradation of Reactor Coolant System Pressure boundary Resulting from Boric Acid Corrosion, April 20, 1987

73. NRC Information Notice 86-108, Supplement 2, Degradation of Reactor Coolant System Pressure Boundary Resulting from Boric Acid Corrosion, November 19, 1987

74. NRC Generic Letter 88-005, Boric Acid Corrosion of Carbon Steel Reactor Pressure Boundary Components in PWR Plants, March 17, 1988

75. NRC Information Notice 90-0 10, Primary Water Stress corrosion Cracking in INCONEL 600, February 23, 1990

76. NRC Information Notice 94-063, Boric Acid Corrosion of Charging Pump Casing Caused by cladding Cracks, August 30, 1994

77. NRC Information Notice 86-108, Supplement 3, Degradation of Reactor Coolant System Pressure Boundary Resulting from Boric Acid Corrosion, January 5, 1995

78. NRC Information Notice 96-011, Ingress of Demnineralizer Resins Increases Potential for Stress corrosion Cracking of Control Rod Drive Mechanism Penetrations, February 14, 1996 August 13, 2002 20

79. NRC Generic Letter 97-001, Degradation o! Control Rod Drive mechanism Nozzle and Other Vessel Closure head Penetrations, April 1, 1997

80. NRC Information Notice 2001-05, Through-Wall Circumferential Cracking of keactor Pressure Vessel Head Control Rod Drive Mechanism Penetration Nozzles at Oconee Nuclear station, unit 3, April 30, 2001

81. NRC Bulletin 2001-01, Circumferential Cracking of Reactor Pressure Vessel Head Penetration Nozzles, August 3, 2001

82. NRC Information Notice 2002-1I, Recent Experience with Degradation of Reactor Pressure Vessel Head, March 12, 2002

83. NRC Bulletin 2002-01, Reactor Pressure Vessel Head Degradation and Reactor Coolant Pressure Boundary Integrity, March 18, 2002

84. NRC Information Notice 2002-13, Possible'Indicators of Ongoing Reactor Pressure Vessel Head Degradation, April 4, 2002

85. Reactor Pressure Vessel Head Degradation,'presented at ANS 2002 Annual Meeting, June 11-14, 2002, by Brian Sheron, US NRC

86. Davis-Besse CR 02-00891, Reactor Vessel 1 ead, February 27, 2002

87. Davis-Besse Procedure DB-DP-00022/R03, Station Review Board, effective March 2, 1998

88. Folder of Memoranda from Jay Hultz to Bill Kanda, regarding review of CNRB materials, October 2000-July 2001

89. First Energy Memorandum, Davis-Besse Employee Concerns Mid-Year Report for 2002, A.J. VanDenabeele to L.W. Pearce, dated July 24, 2002

90. Davis-Besse Approved CNRB Meeting Minutes, for May 23, 2002 meeting

91. Davis-Besse Nuclear Quality Assessment Surveillance Report, SR-02-OUTAG-01, approved June 24, 2002

92. Nuclear Quality Assessment Oversight of Davis-Besse Return to Service Plan, Rev. 1, July 22, 2002

93. Davis-Besse CNRB Action Item Status, July 24, 2002

94. Business Plan Monthly Performance Reports for Davis-Besse, Perry, and Beaver Valley Nuclear Stations, June 2002

95. Davis-Besse CR-02-02584, Implementation of Corrective Action Program by Site Personnel, June 13, 2002

96. Davis-Besse CR-02-02585, Management and Supervisory Oversight and Ownership of Plant Activities, June 13, 2002

97. Davis-Besse CR-02-0089 1, Control Rod Drive Nozzle Crack Indication, February 27, 2002

98. Davis-Besse CR-02-02363, Engineering Change Process Does Not Meet 10CFR 50 App B or ANSI 45.2.11 Requirements, May 31, 2002

99. Davis-Besse CR-02-02408, Collective Significance-Plant Modification Program Concerns, June 3, 2002 100. Davis-Besse CR-02-02419, Untimely Corrective Actions to Address Corrective Action Program Weaknesses, June 4, 2002 101. Davis-Besse CR-02-02434, Inadequate Engineering Rigor Applied to Activities, June 5, 2002 102. Davis-Besse CR-02-02606, Radiation Protection Corrective Action Program is Considered Unacceptable, June 14, 2002 August 13, 2002 21

103. Davis-Besse CR-02-02846, Containment Emergency SUMP Issues, June 27, 2002 104. Davis-Besse CR-02-02943, Containment Air Cooler Boric Acid Corrosion, July 2, 2002 105. Davis-Besse CR-02-02974, Past Operability and Reportability Reviews, July 3. 2002 106. Davis-Besse CR-02-03005, Operability Determination 02-2869 Rigor and Thoroughness, July 6, 2002 107. Letter, H. Bergendahl to US NRC, Supplemental Information Regarding LAR, June 4, 2002 108. Letter, US NRC to H. Bergendahl, RFI regarding Safety Significance Assessment of Reactor Pressure Vessel Head Degradation, June 24, 2002 109. Minutes of "Special" Station Review Board meeting, May 8, 2002 110. Memorandum, J.M. Vetter, to "Distribution", NQA Audit Notification DB-C-02-03, June 21, 2002 I 1. Davis-Besse LER 2002-001, Main Steam Safety Value Setpoints Greater Than Allowable Values, April 1I, 2002 112. Davis-Besse LER 2002-002-00, Reactor Coolant System Pressure Boundary Leakage..., April 29, 2002 113. Davis-Besse LER 2002-003, Fuel Movement in Spent Fuel Pool Without Required Door Attendant, May 9, 2002 114. Davis-Besse CR-02-03369, Quality Expectations, July 17, 2002 115. Davis-Besse CR-02-03404, Expectations for Program Reviews not Communicated, July 23, 2002 116. Davis-Besse CR-02-0286, Uncompensated Extra Hours, June 19, 2002 August 13, 2002 22