ML080390544

Braidwood/Byron Amag Investigation of Organizational Effectiveness and Decision Making

ML080390544
Person / Time
Site:	Byron, Braidwood
Issue date:	09/22/2003
From:	Bement R AmerGen Energy Co, Exelon Nuclear
To:	Office of Nuclear Reactor Regulation
References
FOIA/PA-2008-0046
Download: ML080390544 (35)

Text

J4 FINAL Braidwood/ Byron AMAG Investigation of ib rr Organizational Effectiveness and Decision Makinq

• • Investigation Dates:. September 22 - October 27, 2003 d-- lIItaI-e-

. Investigation Team AR#: 173510 for Byron Technical Root Cause Evaluation Report Investigation Team Leader:

"" Robert S. Bement, Site Vice President, Clinton Station

• Investigation Team Members:

• James P. O'Hanlon, Executive Consultant Arthur H. Rone, Team Manager, Institute of Nuclear Power'Operations David E. Henry, Executive Assistant to.Exelon Nuclear COO

. Russell R. Peak, Plant Engineering Manager, Clinton Station Dale F. Ambler, Senior Programs Specialist, Corporate Licensing and Regulatory Affairs Robert D. Coovert, Corporate Human Performance Manager Thomas J. Dougherty, Nuclear Oversight Manager, Limerick Station

, Executive Summary John L. Skolds, President and Chief Nuclear Officer of Exelon Nuclear formed an investigation team to identify areas for improvement in the Exelon -Nuclear Operational Decision Making process. This investigation was prompted by an overpower condition on Byron Unit 1 and 2 and Braidwood Unit 2 resulting from an Advanced Measurements and Analysis Group (AMAG) modification implementation. Specific objectives of the investigation were to:

1. Evaluate whether the initial implementation and analysis of the AMAG installation was conducted appropriately and with sufficient rigor.

2. Evaluate if decisions made were based on a full understanding of short and long term risks including the potential of a unit overpower condition.

3. Evaluate whether the roles and responsibilities for making and implementing decisions were established and understood by corporate, the sites, and vendors involved.

4. Evaluate the process utilized for challenging decisions made, and the results of those decisions, throughout the AMAG implementation process. Specifically, did the sites feel pressure to continue to operate at the higher power levels?

5. Identify insights and lessons learned to improve Exelon Nuclear's technical issue O resolution process.

Page 1 of 35 4 " '4 . '>" ~., * " . " "C "." ' '

FINAL In 1998 a corporate decision was made to utilize advances in feedwater (FW) flow measuring technology to improve the accuracy ofV-plant heat balances and achieve a 1% increase in thermal power through the 10 CFR 50.59 process. Advanced Measurement and Analysis Group, Inc. (AMAG) technology was selected based on performance and cost. Four Exelon plants (Byron, Braidwood, Dresden, and LaSalle) were modified with the new system in 1999 and the implementation completed at Braidwood in 1999 and at Byron in'2000. Questions-about the accuracy -of the AMAG system were raised by engineers*-at Byron,- Braidwood, Dresden, Mid-Atlantic -Regional - Operating -Group (MAROG), and .Stone-, and_ :Webster_.

Corporation both pre and post installation. Extensive-- troubleshooting -was- periformed in,ý response to sbomie -of these concerns but narrowly focused on -proving -that AMAG- -was,

,functioning-properly and did not resolve the issues of the engineers. Except for one instance, despite the concerns raised by the engineers, as well as other technical information, reactor power was not reduced at Braidwood or Byron Stations. There-was-a-mindset-that if-AMAG--

,was installed and operating properly it was- the-definitive measure -of plantý thermal power.

This caused a bias to accept the AMAG technology over other opposing opinions that were not fully considered and evaluated. In response to NRC questions on the differences between Byron and Braidwood, additional efforts to understand the difference culminated in the conclusion that AMAG was not providing a reliable measure of feedwater flow at Byron Units 1 and 2 and Braidwood Unit 2.

Irrespective of overall Exelon Nuclear performance, the team concludes that the foundation of the present management approach, that depends on robust processes and management focus on following those processes, was ineffective in resolving this specific issue. A lack of ownership and accountability, specific to this event, was identified. Decisionrs-re-g-&rdirgthe-AMAG installation and implementation-process were made Without a.complete Understanding -

-due--to-* poor -issue- management, _weak _oversight, -and iineffective --use --of- established

,processes:- Further, because information was compartmentalized and poorly documented and distributed, the organization was not able to take full advantage of the available information. ,-It is important to note--that-the-team-did7-nt -idi-ntify any- instances -of, an

-unwillingness -to raise-safety-concerns *or-willful -intent -issues. The identified causes are as follows:

1. Roles and responsibilities were fragmented. _D,-- 4 s A.,

2. Ownership and accountability were lacking.

3. Information was compartmentalized.

4. Technical questions were not pursued with-rigor and- differing technical concerns were nC1Utpiopsitioned. ,)

5. Both corporate and site oversight groups were ineffective.

6. Inapspropriate corporate pressure was applied-in one identified instance. t "

7. Corporate systems for problem identification and resolution were poorly used.

8. The rate of change was rapid and may have diluted senior management attention.

Included as Attachment 1 is a detailed timeline that provides a chronological perspective of the issues associated with the investigation team's review of the Braidwood and Byron AMAG implementation issue.

Page 2 of 35

FINAL Investi-gation Team MethodoloyV The investigation team and team leader were selected by the Chief Nuclear Officer of Exelon Nuclear based on their overall experience and impartiality with the issues surrounding the Braidwood and Byron AMAG issue. The backgrounds of all the team members represent over 150 years of applied nuclear power experience in the areas of Executive Management, Operations, Engineering, Nuclear Oversight, Licensing and Regulatory Affairs, and Training.

A critical attribute of this investigation was the infusion of expertise from outside of Exelon.

Specifically, the recently retired President and Chief Operating Officer (including the role of Chief Nuclear Officer) from Dominion Energy was brought in to augment the team.

Additionally, the team integrated an individual whose background includes organizational effectiveness expertise with the Institute of Nuclear Power Operations and being a former Vice President of Engineering in the industry.

An investigation team charter, approved by the Chief Nuclear Officer, was developed to assess the effectiveness of Exelon's organizational decisions made concerning the Braidwood and Byron AMAG installation for the time period from 1998 to present. The focus of this team charter was to identify missed opportunities in order to identify lessons learned and prevent similar events from occurring in the future. The focus was not on what the organization may have done well in assessing AMAG during this period.

To accomplish the objectives of the team charter, the team performed the following:

" Approximately 21 person-weeks to conduct the investigation.

" 38 interviews (including senior Exelon Nuclear executives).

" Reviews of over 70 documents (including condition reports, letters, technical reports, NRC correspondence, meeting minutes, etc.).

" A review of Braidwood and Byron plant performance data.

" Preparation of a detailed assessment report of the team's investigation.

" A detailed presentation was made to the Senior Team regarding the missed opportunities identified during the investigation as well as the Recommended Actions.

Conclusions by Objective Objective 1: AMAG Installation Process This assessment evaluated the issues surrounding AMAG installation to determine if initial implementation and analysis were conducted appropriately and with sufficient rigor. The most prevalent area of concern encountered throughout the review was lack of ownership by both site and corporate personnel, weak oversight of the vendor (AMAG), and failure to follow standard project management practices. The following are examples supporting the investigation team's conclusion:

Page 3 of 35

FINAL e mSite modification at Byron or Braidwood. In fact, of the four Exelon sites that were considering project ofmanagers installation were notonly the modification assigned one was to designated oversee thea implementation project manager.of the AMAG Corporate designated a project engineer for installation of the equipment at the sites, however, the individual selected did not have an I&C background which would have been very beneficial with this modification.

When AMAG personnel came to Byron to take the data after the modification was installed the corporate engineer had to go on site to escort the AMAG team because no one from the site would support the data gathering from the initial AMAG test. .

None of the Exelon sites supported the training that was offered as part of the modification contract. The original contract had enough money budgeted to send a maintenance, operation and engineering representative from each site implementing AMAG to the AMAG office for a week of classroom and lab training. None of the sites supported the four dates that the Cantera project engineer had arranged. The money budgeted for training was used to purchase additional equipment. AMAG stated that they were so frustrated by repeated site questions that they offered two days of training at Braidwood. This training, conducted at Braidwood, was poorly attended by the sites.

Interviews with AMAG, Westinghouse and Cantera engineering identified that there was no site buy in and that this was perceived as a corporate project.

Two engineers from the Mid-Atlantic ROG (MAROG) conducted an independent review of the plant indications resulting from the AMAG installation at the request of Byron as a corrective action from a condition report. The review concluded that the Byron- units were j~ operating at higher power levels than indicated. Tjhe_ team- was not able -to identify the ;v cause of the.overpower or the amount. Upon becoming knowledgeable of the MAROG, engineer's conclusions, the Site Vice President reduced power on both units to effectively .

remove any increase in power as a result of the AMAG installation until a question on differences in cycle fuel depletion between Braidwood and Byron could be addressed.

§ubsequently, a letter was issued .from-corporate Nuclear Fuels Management (NFM) that

-concludedthat-the differences seen in the development of reload cores nor the variations

,in cycle depletion data, since the implementation of A.MAG, can confirm or repudiate the "Validity of the AMAG results. Based on the conclusion of the NFM letter, the site restored power on both units. Thle-MAROG independent review team recommended a more in-Cepth investigation complete with suggested review topics and timelines to complete. 'k'/

This recommendation was not acted on. This was well documented in their report and 0, ,

included in the apparent cause evaluation (ACE). No owner was assigned to follow up on

/* their recommendations. The Byron Station Management Review Committee (MRC) and '

Plant Operations Review Committee (PORC) reviewed this ACE. The PORC approved _

the ACE despite the following comments documented in the PORC minutes: "There-is innsufficient informati0on to know if we are within the license basis or if the corrective actioni J-is to help determine that"- "After all the investigation, we still have open questions. C The reyvalciti~n--would show-hthat-there-is apotential ehnlcagance.' "Thiscouldbeta 1

, "n___UcleIa--safetyissue if an independent person disagrees-with -o b-t---c-i -1.-

ibn."

Nuclear Oversight (NOS) was minimally involved in reviewing the AMAG installation and the potential issues associated with the installation at Braidwood or Byron. When NOS Page 4 of 35

FINAL t" -

was interviewed during this investigation they indicated that NOS would not have looked at this because they did not have the technical expertise within the NOS organization.

A second general area of concern was the lack of rigor associated with the implementation of the AMAG modification at the sites. The following are examples supporting the issue:

Based on interviews with AMAG, Inc., the software program (DIAGNOSE), that eventually was used to detect the noise affecting the validity of the flow measurement, was not a standard required part of the modification. The DIAGNOSE tool for identifying noise contamination was available in 1999 during the installation at Byron and Braidwood.

However, in interviews with Westinghouse they identified a weakness in the rigor with which their equipment was installed and tested. Some installation teams at AMAG routinely checked for noise on all the feedwater loops and retained this information while other teams performed no checks or did not retain the information. At Byron Station, it is believed, based on interviews with AMAG personnel and information contained in the Byron technical root cause evaluation for this issue, that a noise check was performed on one loop on each unit, h6okwvee-tlii-t-sting information was-not-retained.

The modifications to install the AMAG equipment at both Braidwood and Byron included ,**J

ýorequirements to test the AMAG equipment to ensure measurementaccuracy, following the installation. The lack of testing of the AMAG eq-uipmentrcombined with the station's lack of knowledge of this equipment prevented the stations from discovering the noise contamination.

In general the relationship between Exelon and the Vendor was strained. Information was not shared from Byron with the AMAG team. AMAG was not aware of the correction factor changes until April 2002 when they came on site to look at the performance of the equipment in support of the ACE that Byron Station was performing. Westinghouse stated that most of their customers exhibited a greater level of involvement in the installation and testing of the equipment and wanted to know as much about the equipment as possible versus Byron Station, which exhibited minimal involvement in the installation.

Multiple tests were performed on the Byron feedwater lines comparing individual feedwater flow values to the common header flow values. As early as February 2002, data was available from these tests that would indicate that Byron was operating near the high end of the statistical limit or over the limit.

Objective 2: Decisions Based on Risk In 1999 and 2000, ultrasonic feedwater flow measurement technology provided by the AMAG was implemented at Braidwood and Byron Stations respectively. The connection betweenv n-indicated'feedwater flow and thermal power calculation -was well understood by'individuals Jinvo1ved -in AMAG implementation decisions. Additionally, there is considerable- evidence, from interviews and document reviews,- indicating that the consequence of AMAG inaccuracy I would be an overpower condition. However, there existed a prevailing belief that this was highly improbable. The miscalculation of the probability and, therefore, risk associated with non-conservative AMAG performance was caused by ineffective communication of Page 5 of 35

FINAL information, and unsystematic problem resolution. Information supporting these conclusions includes:

During the "Byron AMAG Resolution" presentation to the Corporate Senior Management Team in April 2000, several industry overpower events due to feedwater flow detection errors were discussed.

" Minutes from a November 21, 2002 Bryon PORC meeting convened to review ACE 91771 state, "There is insufficient information to know if we are within the license basis...", and "This could be a nuclear safety issue if an independent person disagrees with our conclusion." There was a lack of documentation associated with closure of these apparent open items contained in the PORC meeting minutes.

" Letters written by the Thermal Performance Engineers at Byron, Braidwood, and Dresden from June 1999 through July 2000 all discuss the possibility of operating at higher power than that indicated by the secondary plant thermal kit.

" An independent team from the Mid-Atlantic Regional Operating Group (MAROG) concluded in February 2002 that Byron Units 1 and 2 may be operating at reactor power levels higher than indicated.

" In a letter dated January 22, 2003, the NRC expressed their concern that "...Byron 1 may be operating above its licensed thermal power level."

_ Interviews and documentation reviews identified that, despite this clear-understanding -of the_

-onsequences associated with nonconservative AMAG errors, the overall.r-isk of this modification was underestimated due to an inhie&nt trust-that the AMAG system-was-

, Oprovidid*accurate information. Information that called AMAG performance into question was undervalued when making decisions relative to AMAG, despite-the-fagt that-in-situ testii-g6f-ý 1 th-dAMAG -syst6-m-was -rnot- performed to com-iparei-res-ults with- a-prec¢isionyflow-measuremt-eif mhm-eth-dI(e6.g:;tracer testing). Compounding this was the fact that information was not effectively communicated-to-all stakeholders in the AMAG project. There were several distinct reasons for the communication breakdowns:

" Poor documentation and communication precluded critical information from reaching the appropriate personnel. The DIAGNOSE tool was used during initial setup by the vendor, but this was not part of the formal vendor setup procedure, and baseline results were not documented. Also, "DIAGNOSE" was touched upon during the training but only a few individuals attended the training.

" Unexpected AMAG correction factor changes were identified as early as July 2000.

Westinghouse and AMAG were not informed of the changing correction factors until April 2002. The correction factor changes along with an observed high data rejection rate were significant enough to prompt Westinghouse to add criteria for acceptable rejection rate and standard deviation to the conditions stated in the Westinghouse Uncertainty Calculation (059-PENG-CALC-084, rev. 01).

N It is unclear as to the degree of knowledge by Westinghouse and AMAG prior to 2002 as to the concerns raised by three Exelon Thermal Performance Engineers regarding S ,'* potential overpower operations at Byron. This lack of Westinghouse and AMAG involvement represented a missed opportunity. Additionally, the Corporate Engineering Vice President reported that six months of data was obtained for diagnostic purposes, but Page 6 of 35

FINAL the information was not forwarded to Westinghouse as requested. Westinghouse did not raise the option of-using"DIAGNOSE" to troubleshoot correction factor discrepancies, and as a result Exelon was not aware of this option.

It is unclear how much information was available to NOS and NSRB, but it is clear that due to ineffective use of the Corrective Action Program, these oversight groups may not have been readily familiar with the condition or its significance and did not drive resolution of the AMAG issue.

Prior to implementation of AMAG at Byron station, a position supporting AMAG was developed and presented to the Senior Corporate Executive team (April 2000). This-presentation omitted key information and some- general statements thatwereiiricluled

ýwere icorrect(e.g: key secondary plant issues resolved). This presentation did not include information regarding the dissenting opinions of the Thermal Performance Engineers or Stone and Webster.

Due in part to a lack of assigned responsibilities, the approach to solving the Byron/Braidwood differences was fragmented. There is no evidence that all the key players were ever brought together to address the differences between Byron and Braidwood and fully understand what we "didn't know", in other words what was causing the differences. Instead, the evidence suggests that individuals and groups worked in silos to solve problems. It is important to note that the team did not identify any instances of an unwillingness to raise safety concerns or willful intent to withhold information.

Efforts to find the cause of the unexplained, unexpected differences between Byron and

• Braidwood were uncoordinated and lacking in rigor. Interview information indicated that there was an over-reliance on the AMAG technology instead of sufficient focus on resolving the differences between Braidwood and Byron or the Byron secondary plant parameters.

Further, investigation into the details of this event identified that specific to troubleshooting of the AMAG issue, accountability was not established, all involved parties were not assembled

' to ensure that the problem was fully understood, tasks and deadlines were not established, unexpected results were not resolved, and decision-making was not effectively challenged.

Finally, the Corrective Action Program was not effectively utilized to systematically document L and resolve the problem.

a N In January 2002, a condition report (91771) was initiated to document the unexplained,

% unexpected differences between Byron and Braidwood. An Apparent Cause Evaluation (ACE) was performed rather than a Root Cause Analysis (RCA). After ten months and numerous extensions, the ACE investigation concluded that the apparent cause was indeterminate. This ACE was approved by the Management Review Committee (MRC) and by the PORC, with no further action to identify the cause other than to continue to search for the cause of the balance of plant differences between Byron and Braidwood.

• ,Erom--th-e--AC EddcUhfiýta-eti-n_ *-it--was-Tunclear-as-to -what-methodology- was -used--to-ccontn this search:--

, Wh -ntheindepbe-fd*nt assessment-from the MARO G engineers concluded that-the-Byron J~/units maybe operating at power higher than indicated, a conditi6ni repio-rqvas not initiated-.

No individual was responsible to respond to the report, and the recommendation from this team was not performed. It is important to emphasize that the Site Vice President at Page 7 of 35

FINAL Byron Station, in response to the conclusions of the MAROG engineers, made the decision to set the AMAG correction factors to 1.0.

The Corrective Action Process was not utilized to document the dissenting opinions of the various Thermal Performance Engineers. Additionally, based on interviews conducted, there was not a complete understanding of the dissenting opinions at all levels of management at the sites or corporate. Additionally, CRs that were initiated to document CF variances were coded as "4d" rather than being driven to resolution. (Significance Level 4 means low-level problems, typically closed to immediate actions taken or other follow-up corrective actions and allows coding and trending of issues. Investigation Class D means no formal investigation / evaluation is required to determine the causes or to determine the corrective actions.)

Objective 3: Roles and Responsibilities for Making Decisions For the time period reviewed, the roles and responsibilities for making and implementing decisions associated with the AMAG issue were not clearly established and understood by corporate, the sites, or the vendors which contributed to a number of missed opportunities in resolving this issue. The corporate project engineer became the de facto project manager of the AMAG equipment at Byron when the site did not take ownership for installation.

Corporate engineering requested support from the Dresden thermal performance engineer and Stone and Webster. However, this support did not resolve the secondary plant issues in i June of 1999. Little progress was made until January 2000. The corporate thermal performance engineer had very little involvement in resolving the difference between the AMAG results at Byron and Braidwood, in fact, he stated his role was just oversight.

Corporate engineering assisted in the preparation of the AMAG Evaluation Report and the April 2000 presentation to senior management. Byron Station personnel believed corporate should have had the lead in resolving the differences between Byron and Braidwood.

Site roles and responsibilities were not clearly established in that Byron did not assign a project manager to support the installation of the AMAG equipment. As stated previously, the corporate engineer had to act as the AMAG escort. Byron was not working effectively to resolve the secondary indication issues and make a decision on the use of AMAG, until January 2000. Followingthei'lementation-of AMAG -at-Byron,- the--thermal-engineer-at .

Braidwood wrotea letter queStjoinig the performance of the-AMAG equipment at-Byron. -No_

C-IR was -written *it is Uncertain as t the degree of effectiveness-regardingifollow-up-on hisz -

c6ncern7--A condition report written in January 2002q6eti---d -the- differences between Byron and Braidwood following power uprate, resulting in an ACE being assigned. This ACE was extended numerous times and after ten months was approved with the apparent cause as indeterminate. ln:7January 2002,-Ybýe-dýb--rT-View ---ofle -Byron Station-NRC--Caie-Management database,- it ailpears-the NRC raised aquestion regarding the. condition report 1(9177-).- While-this issue was entered into the case- mahagement database-by Byron Stationr Regulatory Assurance there appeared to be no follow-up, based -on case management documentation, --until October. 2002. The April 2002 Westinghouse report recommended gathering of data to help identify and resolve the root cause for the correction factor fluctuation. This report was PORC approved at Byron Station and, based on the meeting Page 8 of 35

FINAL Sdocumentation, the changing correction factors were not questioned. Finally, this data was not supplied to AMAG until mid 2003.

Westinghouse and AMAG roles and responsibilities were not effectively established which complicated the communication between Exelon and the vendor. Both vendors stated that during troubleshooting activities key plant parameters and information was not communicated. Westinghouse did not ensure that AMAG was following rigorous processes and procedures for installation and troubleshooting of their installed equipment.

Objective 4: Process for Challen-pincq Decisions The process for challenging decisions associated with the AMAG issue was ineffective resulting in multiple missed opportunities (including independent oversight groups such as NSRB, PORC and NOS) to effectively address potential risks and prevent the overpower condition. Additionally, there was ineffective use of the corrective action process. In particular, three factors led to this situation: ineffective resolution of technical information and individual concerns, ineffective oversight by numerous individuals, organizations and entities, and, in one case, inadequate organizational behaviors associated with raising and resolving dissenting opinions.

The decisions associated with the AMAG implementation were ineffective with respect to the organization's resolution of technical information and individual concerns. Several pieces of technical information (e.g., letters from the three TPEs, Stone and Webster report and Westinghouse report) utilized informal documentation (electronic mail, letters with no addressee or copies, etc) and it is unclear as to the depth of distribution of this material.

Contributing to this situation was the ineffective use of the corrective action process in that this same information was not identified in condition reports. In essence, while individuals in the organization knew pieces of the issue, interviews and documentation reviewed indicate that no one individual had a complete picture of all the pieces.

- The second factor leading to an ineffective process for challenging decisions associated with the AMAG issue was inadequate oversight. Examples include:

- In the area of Engineering over'sight, three letters (one from NFM, and two from the Vice President of Engineering) were ineffective because they did not evaluate the full extent of discrepancies (information from the three TPEs, Stone and Webster report and the 4Westinghouse report).

o

• There was ineffective management decision-making when the Byron Station Management Review Committee (MRC) and Plant Operations Review Committee (PORC) decided to accept an apparent cause evaluation (ACE) that was determined to be indeterminate without commissioning a root cause analysis or another acceptable method to determine the cause.

N Completion of the ACE (91771) took ten months to complete and numerous extensions.

F__- The Byron MRC performance was weak in accepting the ACE considering ineffective jlý -resolution of the content in the ACE (i.e., recommendations not completed or effectively Page 9 of 35

FINAL

-/ tracked for completion) for which the investigation team could not find follow up documentation.

" The Byron Plant Operations Review Committee (PORC) was an ineffective barrier when they unanimously accepted the ACE (91771) and the supporting documentation as well A as its review of the 2002 Westinghouse report. Regarding the content of the minutes for the PORC meeting, there was a lack of documentation associated with closure0 of open items contained in the November 21, 2000 PORC meeting minutes.

" Based on a review of the NRC Case Management database for Byron Station, it appears that, on January 22, 2002, a question was raised by the Byron NRC inspector regarding CR 91771. This database entry results in a conclusion that Regulatory Assurance at Byron Station and corporate Licensing exhibited poor attention to detail based on the fact that the Byron Case Management item regarding this issue shows no documentation of follow up action for approximately ten months (when ACE 91771 was provided to the resident inspector).

" There was no documented evidence that identified Nuclear Oversight as having been effectively engaged in the AMAG issue including the following: inadequate field observations to review the issue, the length of time to resolve ACE 919771 or the conclusion of indeterminate for this ACE, or the Byron PORC review of ACE 91771.

When NOS was interviewed during this investigation they indicated that NOS would not have looked at this because they didn't have the technical expertise within the NOS organization. Additionally, through interview information obtained, on several occasions (three to five times) during the summer of 2002, the Byron Thermal Performance Engineer expressed concern to the Nuclear Oversight Manager regarding the lack of progress in solving the Byron/Braidwood differences identified in CR 91771. The Byron Nuclear Oversight Manager, through interview, was cognizant of the Byron TPE concerns, however, he could not produce documentation of Nuclear Oversight follow-up with these concerns or AMAG implementation reviews.

• Based on interviews and information reviewed, the presentation package used at the April 2000 information meeting with the senior executives did not contain previously discussed dissenting technical opinions. It is noted that the Byron Site Vice President was not in attendance at this meeting and he made the decision to implement. Further, interviews conducted did not identify any issues that would have indicated intentional withholding of information, however, a basis for why this information was not discussed could not be determined. The investigation team draws the conclusion, therefore, that this meeting was another missed opportunity instead of having- been _a potential- barrier to the overpower event. Iis -furthervnoted that the senior managers -inattendance at this m6-eting did not effectively-challenge the higher than expected Byron electrical power and-

• he differences between Braidwood and-Byron post( plementation.

• The investigation team has also identified that the review of the AMAG issue by the Byron Nuclear Safety Review Board (NSRB) was ineffective. The basis for this conclusion comes from a review of the NSRB minutes that addressed the AMAG issue. In aggregate, the NRSB review of AMAG resulted in no safety issue associated with the implementation at Byron Station. Specific excerpts from Byron NSRB minutes further substantiate this conclusion:

Page 10 of 35

FINAL From the April 2000 Byron NSRB minutes: "The Subcommittee reviewed the AMAG project. The AMAG presentation was well done and technically sound. It appears that the engineering organization has an adequate change process in place to implement AMAG. No safety issues exist."

From the May 2001 Byron NSRB minutes: "Power uprate activities were ongoing during subcommittee activities. A calculation error was discovered in an AMAG correction factor. The subcommittee will review the root cause analysis at its next meeting."

From the July 2002 Byron NSRB minutes: "The AMAG feedwater flow discrepancy issue was briefly discussed. The plant assured the subcommittee that the plant was operating below the licensed limit. The subcommittee was given an ACE and a Westinghouse report to read later, which provide technical detail."

Finally, the investigation team has concluded that there was a third factor that led to an ineffective process for challenging decisions associated with the AMAG issue dealing with whether pressure was applied that may have influenced decisions. At Byron and Cantera, interviews revealed that there was no pressure applied that would have inappropriately influenced decisions and actions. Hbwever,--at-Braidwood-Station,--the-majority-of interviews-- /- _

thermal perfO-rhance iden-tified-that-there was-a degree of perceived -plressure (changing the idicao to-yellow) being applied during the June October-20600-timeframe for-,being G.o(

a'le-to-achieve--the-same-electrical Qutput as Byron and thlt this pressurecame fr(m the corporate-Engiheering department. This conclusion is further substantiated by the following:

" Numerous challenge calls were conducted by the corporate thermal performance engineer to ascertain the basis for Braidwood not achieving Byron's output. There was no documentation to determine if Byron was being challenged for operating at a higher electrical output than Braidwood.

• At some point during the time period, the corporate Engineering department inappropriately changed the Braidwood thermal performance indicator to yellow as a result of not achieving the Byron electrical output. Through interviews, the investigation team was able to understand that Braidwood challenged this position which eventually resulted in the site thermal performance indicator being returned to white.

Obiective 5: Insights. Lessons Learned and Recommended Actions:

1) Roles and Responsibilities were fragmented.

Roles and responsibilities for making and implementing decisions were not clearly established and understood by corporate and the sites. An ad-hoc approach to project organization resulted in confusion as to ultimate responsibility for the correctness of the decision to implement AMAG and resolve ongoing technical questions and concerns.

Page 11 of 35

FINAL Recommended Actions:

- Communicate and reinforce expectations for roles and responsibilities to ensure actual management processes and decision-making is in accordance with the desired leadership style.

aReview the processes for issues management and verify they explicitly direct the assignment of clear roles and responsibilities at the outset of the processes.

9 Perform a self-assessment of project management processes and procedures to ensure clarity of Project Management roles and responsibilities between corporate and the sites.

2) Ownership and accountability were lacking.

" A lack of ownership, for the AMAG installation and implementation, was manifest throughout the period of time that the AMAG correction factors were being utilized at Byron. Data review and interviews indicate that operations viewed the AMAG modification as an engineering issue, the site engineering organization felt AMAG was a corporate responsibility, and some in corporate engineering reported that they had no decision making responsibility.

" There was a lack of accountability demonstrated; NOS-believed-they-could not review-the-AMAG-ssue because thlyla, edthe technical expertise; the -MAROG

,engineering, the Dresden and Braidwood thermal performance engineers, and the Stone & Webster reviews recommended more in-depth assessments, which were not conducted; the ACE was closed out as indeterminate without appropriately dispositioning the concern.

" In several instances individuals involved with the AMAG issue demonstrated accountability only for their narrowly defined tasks. 0 Recommended Actions:

X Review current corporate projects and prioritize to ensure the process, of ownership during the following phases of a project: design, installation, testing and operation is clearly defined and individuals understand what is expected of them.

0 Reinforce clear expectations that accountability transcends narrowly defined task assignments and that all are accountable for the overall success of the organization.

3) Information was compartmentalized.

• Information from the thermal performance engineers at Byron, Braidwood, and Dresden questioned the AMAG implementation at Byron Station. Due to poor issue management and oversight, decisions were made without a complete picture of the situation.

• Exelon personnel were -not -expert- on_the AMAG /system and did not receive vendor recommended training at the time of installation but instead received abbreviated training one year later which was not well attended. As a result the overall technical competencies in the organization were weak and technical personnel did not understand the capabilities and limitations of the system. The Page 12 of 35

FINAL true expert, AMAG, was not used effectively by Exelon to monitor and understand anomalies with the AMAG system. Requests for information by AMAG were not satisfied in a timely manner and multiple interfaces between Exelon and AMAG resulted in no one individual having a clear picture of performance or understanding of the issues either at Exelon or AMAG.

In April 2002 AMAG / Westinghouse asked for six months of data on performance of AMAG installed equipment. The data was not shared with Westinghouse until 2003 when the NRC raised issues.

Based on a review of the Byron NRC Case Management database, it appears that RC interest was much earlier than senior management knew. Senior NZ','/ management thought NRC interest originated in January 2003 because of questions the NRC received from a competitor of AMAG, whereas NRC interest

• .4 appears to have begun in January 2002 when the Byron Resident Inspector raised

• -,o < questions associated with CR 91771. This was documented in the Byron LfLRegulatory Assurance Case Management database as a January 22, 2002 "date ised" by the NRC inspector.

Recommended Actions:

" Modify the Operational and Technical Decision Making process (OP-AA-106-101-1006, Revision 0) to allow broader application. The current procedure appears to have all the necessary elements to successfully address issues similar to AMAG; however, the procedure as currently written would exclude issues like AMAG. The procedure should be utilized more often, and in all cases when answers to significant issues cannot be determined.

" Communicate and reinforce expectations for implementation and use of the changes made to OP-AA-106-101-1006.

" Establish and communicate expectations to contract with an independent third party, that has the requisite experience, when new technology is introduced that presents a potential safety risk and is beyond the current technical understanding and capabilities of the organization to monitor vendor performance.

" Review policies and communicate clear expectations to ensure critical communication and documentation between the corporate office, sites and vendors are properly reviewed, approved, shared, followed up, and properly maintained.

" Establish and reinforce expectations that decision makers must critically challenge assumptions that underpin decisions that impact safety. In addition, set expectations that those presenting information to decision makers present all relevant information and that conclusions are supported by rigorous technical analysis.

4) Technical questions were not pursued with rigor and differing technical concerns were not dispositioned.

Individuals appeared to be intimidated by the technology. didý(t-know enough--to ak-ueStions' seemed to be an acceptable excuse for not being appropriately Ilb involved and challenging decisions. There appeared to be a reluctance to use Page 13of35

FINAL outside help on this issue. Coordination of Westinghouse and AMAG involvement

• was weak.

The assumption was that if the AMAG system were installed properly and all the constituent parts were operating properly it must provide an accurate measure of reactor power. It followed under this assumption that since it was more accurate

' than secondary plant instrumentation it must be correct and the secondary plant instruments were less reliable. T hig-Ls ._-ption-drove --decisih mnakers and-ll6-w~d ~th6 to conclude -that the conflicting secondary plant instrumentationwas inaccurate.-(-This was an illogical conclusion since the secondary plant instruments were all biased in the same direction of similar proportions)

• Resolution of differing opinions was not effectively managed by those involved in the AMAG project. Most notably, a democratic vote was used in January 2000 rather than reaching consensus, prior to recommending AMAG implementation to the Byron Management Team. Dissenting viewpoints were not resolved, and not included in the presentation to the corporate executive team.

Troubleshooting and problem solving processes need improvement. For several years, Byron did not know why there was a difference between their units and the Braidwood units, why the secondary plant parameters were higher than expected, and why the correction factors were changing. Over time there was substantial effort regarding the AMAG technology, however, there was limited effort expended to understanding the secondary plant parameters.

Recommended Actions:

• Engage fleet resources when dealing with "Unsolved Mysteries" (i.e., lingering issues whose root cause in not understood). Independent evaluations and challenge boards are valuable resources for solving these issues.

SIncorporate into the Corrective Action Process controls that, on a periodic basis, site NOS and the site CAP Administrator will conduct a review of CRs to determine unsolved mysteries. I

" Ensure the modification process includes requirements for a verification of vendor installation, testing. and troubleshooting process/procedure,° and that these practices meet Exelon standards.

" Ensure the procurement process includes requirements for a verification of vendor installation, testing and troubleshooting process/procedure, and that these practices meet Exelon standards.

" Perform an assessment of the current troubleshooting and problem solving processes to incorporate lessons learned from this event.

" Develop a case study on this event which focuses on the organizational effectiveness and decision making issues identified during this investigation.

" Conduct awareness sessions across the fleet for the case study.

Page 14 of 35

FINAL

5) Both corporate and site oversight groups were ineffective.

9 Senior management missed an opportunity to resolve the issue when the focus of management attention was directed at why Braidwood was not achieving the same power level as Byron in spite of the fact that Braidwood was within its thermal kit and Byron was not.

• The effectiveness of the Byron PORC in accepting the ACE (91771) noted that "after the investigation we still have open questions, "The investigation would show there is potential technical arrogance", and "This could be a nuclear safety issue if an independent person disagrees with our conclusions". These comments should have had formal documentation to identify resolution, however, no such documentation could be found.

The NSRB meeting minutes for April 2000, May 2001, and July 2002 show the AMAG installation and discrepancies in feedwater flow were discussed to some degree. The team was unable to obtain documentation regarding presentation material made to the NSRB in any of these cases and, therefore, cannot conclude what was discussed, however, the investigation team did conclude that the NSRB did not drive resolution of the issue.

• The 2001 INPO corporate evaluation identified that "Weaknesses exist in critically challenging assumptions and decisions that can affect nuclear safety and have resulted in plant events and challenge station operations". The Byron power uprate in combination with feedwater flow recalibration project was specifically cited as examples of ineffective decision making. The corporate response to this Area for Improvement (AFI) resulted in modification to the Exelon management model but was not fully responsive to the specifics of the AFI and underlying causes. The AFI cited several examples of ineffective decision making similar to AMAG.

Recommended Actions:

• Nuclear Oversight will review their current operating practices to ensure that rigorous investigations are conducted to determine specifically "what is not known".

" Commission an NSRB self-assessment of their performance with regard to this issue. Communicate the results of this assessment to all current NSRB members and Exelon management along with expectations for future NSRB performance.

" Commission an independent assessment of the effectiveness of PORC in light of this issue. Implement and communicate the results of this assessment to all current PORC members at all sites along with expectations for future performance.

6) Inappropriate corporate pressure was applied in one identified instance.

• There was a degree of pressure (changing the thermal performance indicator to yellow) being applied on Braidwood Station during the June - October 2000 timeframe for not being able to achieve the same electrical output as Byron. This pressure came from the corporate Engineering department.

Page 15 of 35 1-

FINAL Recommended Actions:

a Based on feedback from the Byron TPE, Byron Engineering should evaluate the effectiveness of the Davis-Besse training (SOER 02-04) to clearly understand the message that was received. Perform gap training as needed.

0 Communicate the expectation that senior managers should maintain an oversight role to assure the organization is functioning as intended. When senior managers become the decision makers assure that at least one member of senior management maintains an oversight role.

N Evaluate the current MRM process to determine if the process is effective in identifying and resolving issues. Implement the actions stemming from this evaluation including clear expectations for future performance.

• Evaluate the current NSRB process to determine if the process is effective in identifying and resolving issues. Implement the actions stemming from this evaluation including clear expectations for future performance.

7) Corporate systems for problem identification and resolution were poorly used.

The June 2001 INPO Corporate Evaluation identified a weakness in the area of Organizational Effectiveness. INPO cited behaviors similar to those exhibited in managing the AMAG problem. Exelon's response to this Area for Improvement (AFI) was ineffective in addressing all of the insights of the AFI; rather, Exelon chose to respond to the issue in a broader sense associated with decision making.

There is no indicated action by Exelon that the specific issues in the AFI, similar to the issues found in this investigation report, were acted upon.

There was poor issue documentation and distribution; CAP procedures were not followed in some circumstances, CRs were not initiated on numerous occasions, information was communicated via email but not documented in the CAP Program, correspondence was informal (i.e., letters not dated or signed), and independent reports not signed as completed.

Recommended Actions:

Establish and communicate clear expectations for use of the Corrective Action Program (CAP). Reliance on e-mails to document potential adverse conditions rather than CAP contributed to the delays in resolving the AMAG problem and should be a basis for expectations communicated regarding this issue. When CAP was utilized, poor investigation and ineffective challenging of results further delayed resolution. Reinforce these expectations.

a Perform an effectiveness review of actions taken in response to the 2001 INPO corporate evaluation AFIs. Identify the required actions for any deficiencies noted from this review.

Formalize-the process o[pgerfoorming a7i assessment for, -at-a-minimum, -an J. ddeeriftate-AGE-_-(reference LS-AA-1 25-1003, Rev. 003, Apparent Cause Evaluatio~nManual)0...-

Page 16 of 35

FINAL

8) The rate of change was rapid and may have diluted senior management attention.

• The nuclear organization was going through rapid change during this time period including a company merger. In addition, there were several significant initiatives and issues being addressed, e.g., reducing refueling outage length, implementing extended power up rate at several sites, addressing Appendix R issues, and implementing a number of other projects. The accumulated effect of all these changes may have reduced management's attention to a risk significant issue.

Recommended Action:

Manage the rate of change to avoid unintended consequences and increased risk.

Incorporate into HU-AA-1 101, "Change Management", a periodic review of the degree of change occurring within the fleet and the associated risk resulting from the aggregate changes.

Additional Follow Up Actions Conduct an effectiveness review of the results of implementing the recommended actions contained in this report Strengths None Approved By:

Robert S. Investigation Team Lead Date Page 17 of 35

FINAL p References

1. AMAG Implementation Timeline (Attachment 1)

2. Investigation Team Interview List (Attachment 2)

3. Exelon Operational Decision Making Investigation Team Charter

4. INPO Document, "Principles for Effective Operational Decision-Making (December 2001)"

5. "Project Executive Summary" from 1998 Business Case Presentation

6. Braidwood Units 1 and 2 Feedwater Venturi Calibration Information.

7. Bryon UFSAR/FPR Change Review Form dated March 1999, Change Number 19-007.

8. Byron DCRs 990123 and 990124, March 1999.

9. Advanced Measurements & Analysis Group, Inc. document dated April 1999, "Standard Deviation of Flow Rate Measurement in a Steady State Turbulent Flow".

10. Braidwood PORC Meeting Minutes dated April 20, 1999 (99-027)

11. Byron Units I and 2 AMAG Test Results (including SPP 00-003 and SPP 00-004, May P 1999.

Braidwood Format for Design Change Package, DCP Number 9900049 approved June 12, 1999.

13. Braidwood Format for Design Change Package, DCP Number 9900050 approved June 12, 1999.

14. Joe Wooten (Dresden Thermal Performance Engineer) initial Byron plant review dated June 16, 1999 (e-mail date).

15. Stone and Webster Report, "Byron Thermal Performance", dated June 21, 1999 (e-mail date).

16. Dave Eder (Byron Thermal Performance Engineer) letter to Bill Kouba (Byron Site Engineering Manager) entitled, "Byron Ultrasonic Feedwater Flow Evaluation", dated July 1999. [Byron letter # 99-0109]

17. Joe Wooten (Dresden Thermal Performance Engineer review of "PEPSE Review of Byron Unit 1 Thermal Performance" dated January 2000.

18. ABB letter to USNRC dated January 6, 2000, "Submittal of CENPD-397-P, Rev. 01 -

"Improved Flow Measurement Accuracy using CROSSFLOW Ultrasonic Flow Measurement Technology", (proprietary information enclosed).

19. Draft Report No. XXX-00 (01/28/00), "Advanced Measurement and Analysis Group (AMAG) Implementation Evaluation Report".

Gary Loeb document dated February 14, 2000 entitled, "PEPSE Review of Byron Unit 1 Thermal Performance".

Page 18 of 35

FINAL

21. Report No. 001-00,(Rev. 2-20-00), "Advanced Measurement and Analysis Group (AMAG) Implementation Evaluation Report".

22. ABB Letter to USNRC dated March 13, 2000; "ABB CENP General Response to Public Meeting Regarding Caldon, Inc., Ultrasonic Flow Measurement Representation of Cross-Correlation Flowmeters Technology".

23. Safety Evaluation by the Office of Nuclear Reactor Regulation - ABB Combustion Engineering Nuclear Power Topical Report (CENPD-397-P, Revision 1), "Improved Flow Measurement Accuracy Using CROSSFLOW Ultrasonic Flow Measurement Technology", dated March 20, 2000.

24. Draft April 7, 2000 Presentation, "Byron AMAG Resolution - NGG Senior Management Meeting", (provided by Joe Williams).

25. April 7, 2000 Presentation, "Byron AMAG Resolution - NGG Senior Management Meeting".

26. Byron Nuclear Safety Review Board Meeting Minutes, April 18 and 19, 2000.

27. Byron Plant Operational Review Committee (PORC) Minutes for May 9, 2000 (PORC 00-41)

28. AMAG Training Course Checklist, May 9 and 10, 2000

29. Byron Plant Operational Review Committee (PORC) Minutes for May 11, 2000 (PORC 00-42)

30. Byron Safety Evaluation Summary Form, Tracking No. 6G-00-0079, Rev 1, approved May 12, 2000.

31. Byron Plant Operational Review Committee (PORC) Minutes for June 1, 2000 (PORC 00-48)

32. Don Hildebrant (Braidwood Thermal Performance Engineer) letter dated June 14, 2000, "Braidwood / Byron Differences Post AMAG Implementation".

33. INPO 2001 Corporate Evaluation Interim Report.

34. INPO Evaluation Report for Braidwood Station, May 2001.

35. Byron Unit 1 Licensee Event Report dated May 15, 2001, Docket Number 05000454, "Reactor Power Limit Exceeded Due to Improperly Calculated Feedwater Mass Flowrate Utilized in Reactor Power Calorimetric".

36. Byron Nuclear Safety Review Board Meeting Minutes, May 15 and 16, 2001.

37. USNRC Event Reports for May 15, 2001 - May 16, 2001 (includes Byron-I).

38. INPO Evaluation Report for Byron Station, July 2001.

39. USNRC Event Reports for October 2, 2001 - October 3, 2001 (includes Hope Creek).

40. USNRC Inspection Report to Oliver D. Kingsley dated December 14, 2001, "Byron Station, Units 1 and 2, Inspection Report 50-454/01-15(DRP); 50-455/01-15 (DRP).

0 41. Apparent Cause Evaluation (ACE) #91771 initiated January 29, 2002.

Page 19 of 35

FINAL

42. C. S. Hauser (Westinghouse) letter to Jeff Drowley dated March 15, 2002, "Exelon Nuclear Byron Units 1 and 2 CROSSFLOW System Performance Review".

43. Westinghouse Report (prepared by Advanced Measurement and Analysis Group, Inc.

and approved by Westinghouse Electric Company LLC) entitled, "Summary of Recent CROSSFLOW Related Activities to Support Byron Station", dated April 2002.

44. Byron Plant Operating Review Committee (PORC) Minutes, Thursday, May 2, 2002 (PORC# 02-028).

45. Byron Nuclear Safety Review Board Meeting Minutes, July 10 and 11, 2002.

46. Byron Plant Operating Review Committee (PORC) Minutes; Thursday, November 21, 2002 (PORC# 02-062)

47. James M. Armstrong, Director MAROG Nuclear Oversight (acting), letter to Exelon Nuclear Senior Management Team dated January 21, 2003 entitled, "INPO Corporate Evaluation Follow Up Assessment".

48. USNRC Letter to John L. Skolds dated January 22, 2003, "Licensed Thermal Power -

Byron Station, Unit 1"

49. USNRC Meeting Notice: "Forthcoming closed meeting with Exelon Generation Company, LLC (Exelon), to discuss licensed thermal power at Byron Station, Unit 1",

dated January 23, 2003.

50. Exelon Presentation to NRC dated January 24, 2003, Byron Thermal Power Measurement".

151. K. R. Jury letter to USNRC dated February 5, 2003, "Byron Station, Unit 1, Licensed Thermal Power Limit Verification".

52. USNRC Summary of January 24, 2003 meeting with Exelon regarding Byron Unit 1 Licensed Thermal Power Limit (TAC No. MB7319).

53. USNRC Inspection Report to John L. Skolds dated April 24, 2003, "Byron Station, Units 1 and 2, NRC Integrated Inspection Report 50-454/03-02; 50-455/03-02".

54. INPO Evaluation Report for Byron Station, June 2003.

55. George F. Dick (USNRC) letter to John L. Skolds dated July 8, 2003, "Licensed Thermal Power, Byron Station, Unit 1-Request for Additional Information (TAC No.

MB7378).

56. Keith R. Jury (signed by Kenneth A. Ainger) letter to USNRC dated August 15, 2003, "Byron Station, Unit 1 Licensed Thermal Power Limit Verification".

57. Byron Station Unit 1 Licensee Event Report, Docket Number 05000454, dated August 28, 2003, "Licensed Maximum Power Level Exceeded Due to Inaccuracies in Feedwater Ultrasonic Flow Measurements Caused by Signal Noise Contamination".

58. Braidwood Unit 2 Licensee Event Report, Docket Number 05000457, dated August 31, 2003, "Licensed Maximum Power Level Exceeded Due to Inaccuracies in Feedwater Ultrasonic Flow Measurements Caused by Signal Noise Contamination".

59. USNRC Event Reports for August 29, 2003 - September 2, 2003.

Page 20 of 35

FINAL O 60. USNRC Event Reports for September 2, 2003 - September 3, 2003.

61. USNRC SECY-03-0159 Weekly Information Report - week ending September 5, 2003, (includes Byron and Braidwood power reduction information).

62. USNRC letter to John L. Skolds dated September 9, 2003 regarding forthcoming meeting on September 18, 2003 for the Byron Station Unit 1 thermal power issue.

63. Presentation to NRC dated September 18, 2003; "Byron Station Thermal Power Measurement".

64. Braidwood and Byron AMAG Root Cause Report (AR# 173510-14), report date of September 26, 2003.

65. Document package provided by electronic mail from David C. Eder (Byron TPE) dated October 2, 2003.

66. Results of Byron Thermal Performance Engineer review of April 7, 2000 presentation package; provided via e-mail dated October 2, 2003.

67. OP-AA-106-101-1006, Revision 0, "Operational and Technical Decision Making Process".

68. HR-AA-18, Revision 0, "People*Solve Employee Dispute Resolution Human Resources & Administration".

69. Nuclear Oversight Field Observations.

70. Byron NRC Case Management document - open issues (printed October 24, 2003).

O71. LS-AA-125-1003, Rev. 3, "Apparent Cause Evaluation Manual".

72. LS-AA-125, Rev. 6, "Corrective Action Program (CAP) Procedure".

Page 21 of 35

FINAL Attachment 1 AMAG Implementation Timeline This timeline was developed by the AMAG Decision-Making Investigation Team based on personnel interviews; and reviews of letters, technical papers, condition reports, and plant performance data.

1998 In 1998, Exelon initiated action to procure and install improved feedwater flow measurement systems at eight of the ten units in the fleet. These modifications would increase the accuracy of feedwater flow measurement, and thus optimize electrical output at the stations. The ultrasonic flow measurement system selected for use at Exelon plants was manufactured and marketed by Advanced Measurement and Analysis Group, Inc. (AMAG) and Westinghouse Electric LLC.

The ultrasonic flow measurement- system, known as CROSSFLOWTM (but more commonly referred to as AMAG), uses ultrasonic technology coupled with cross-correction statistical techniques to measure feedwater flow by determining the duration of time it takes for flow turbulences to travel a known distance in the feedwater piping. The AMAG system uses two sets of transducers externally mounted on the common header feedwater piping or on each of the four individual feedwater loops. A high frequency signal is sent to the two sets of transducers and as the signal passes through the fluid, it is modulated by flow eddies in the fluid. These same eddies also modulate the second ultrasonic signal located approximately one foot further downstream. The difference between the two modulated signals is the displacement in time that it takes for the eddies to travel between the two sets of transducers.

The fluid velocity is then calculated by dividing the known distance between the two sets of transducers by the time delay.

AMAG is utilized as a calibration tool to correct feedwater venturi flow measurements through use of "correction factors (CFs)" to recover lost megawatts due to potential venturi inaccuracies. The feedwater venturis are periodically checked using a set of portable AMAG electronic devices and the venturi flow correction factors are manually updated in the process computer by Operators. In order to economize, Exelon decided to implement AMAG in a manner that did not permit the continuous feedwater flow monitoring by the transducers typically found at other facilities. Mounting brackets were installed at each plant, but only one set of hardware was purchased per site that would be shared between the units to periodically determine correction factors and calibrate the permanently installed venturi devices. This scheme differed from previous installations of the AMAG product at other non-Exelon sites in that the majority of other plants maintained continuous installation and monitoring of flow with the AMAG system.

Page 22 of 35

FINAL 1999 Corporate engineering designated a project lead for procurement of the equipment at the sites. However, the individual selected did not have an I&C background and relied heavily on AMAG representatives for technical issues. The corporate project engineer had an expectation that each affected site would designate a project manager to install and implement AMAG. Site project managers were not assigned to oversee the implementation of the AMAG modification at Byron or Braidwood. In fact, of the four Exelon sites that were considering installation of the modification, only LaSalle designated a project manager. It is not clear what the company expectations were for assigning project managers during this time period. In the spring of 1999, the corporate engineering lead became the de facto project manager for Byron, Braidwood, and Dresden. In June of 1999, when AMAG technicians came to Byron to obtain data after the modification was installed, the corporate lead had to go on site to escort the AMAG team because site personnel were unavailable to support AMAG.

Training was offered to Exelon personnel by AMAG, per the contract, prior to installation in 1999, but no one attended. The original contract included funding to send a maintenance, operations and engineering representative from each site implementing the modification to.-

the AMAG office for a week of classroom and lab training. None of the sites supported the four dates that the Cantera project engineer had arranged. Subsequently, the money budgeted for training was used to purchase additional equipment, and abbreviated training was provided one year later in May 2000.

The AMAG system installation was completed at Braidwood in April 1999 and at Byron in May 1999. The post modification testing included hardware installation verifications, but no in-situ testing to compare results with a precision flow measurement method (e.g.; tracer tasting). During the installation, AMAG personnel used their DIAGNOSE troubleshooting tool to test for hydraulic noise on one FW loop, but discontinued this practice after results were negative for that line. From interviews with AMAG, Inc., this practice was not a routine part of the installation procedure, and, in the case of initial testing at Byron, results were not documented. Once installed, data obtained from AMAG indicated that both Byron and Braidwood venturi flow devices were overestimating feedwater flow, and therefore, overestimating reactor power. Implementation of the AMAG correction factors at the stations was thought to provide a more accurate indication of reactor power, thus allowing both stations to gain electrical output from their generators. Prior to 1999, the four units at Byron and Braidwood each produced nearly identical gross electrical output (within + 2 MWe).

However, the AMAG correction factors obtained during the initial testing indicated that each Byron unit would gain approximately 12 megawatts more than the similar units at Braidwood.

This unexpected result was not documented in a Condition Report (CR).

Despite the difference in c.rrection factors, Braidwood implemented the correction factors and gained approximately 11 MWe per unit in June 1999. According to several engineers at Braidwood and corporate headquarters, the correction factors obtained at Braidwood were Page 23 of 35

FINAL consistent with expectations prior to implementation, and results on secondary plant process indicators showed good correlation with Braidwood's thermal kit. Byron did not implement the correction factors at this time due to concerns over the divergence between the stations.

In June 1999, Corporate Engineering requested that Stone and Webster review Byron and Braidwood plant performance and other key parameters in an effort to substantiate or refute the ultrasonic flow measurement results. The Stone and Webster report states, "In light of additional data, or more accurate readings there is no basis for concluding that the ultrasonic flow measurements are more accurate than the venturis. Further, an increase in reactor power based on the ultrasonic flow measurements seems imprudent without better data related to plant performance." No condition report was initiated to address this conclusion.

In June 1999, at the request of corporate Engineering, the Dresden Thermal Performance Engineer (TPE) evaluated the unexpected differences between Byron and Braidwood. The Dresden TPE concluded that, "In the event that a FW flow correction must be done, then until we have resolved the cause for the discrepancy between the units, I recommend that Byron not change their FW flow by more than Braidwood has((approximately 1%)1 This would allow the recovery of about\10 MWe per unit and would not give outside regulators any reason to question the results until we have a definitive answer." The Dresden TPE also concluded that no serious errors existed in either Byron or Braidwood plant PEPSE models, andrthere was no "smoking gun" to explain the potential differences. The Dresden TPE recommended a precision secondary plant thermal performance examination be conducted in the form of a full N4merican Society of Mechanical Engineers (ASME) Performance Test Code 6 (PTC6) test Kfor steam turbine testing). The information and recommendations provided by the Dresden TPE were not documented in a Condition Report, and the recommended testing was not performed.

In July 1999, the Byron Thermal Performance Engineer documented his position regarding AMAG implementation in a memo to the Byron Site Engineering Director. In that memo, the TPE states, "...1 do not support using the AMAG flow indications to recalibrate our existing feedwater flow venturis." Although the differences between Byron anq Braidwood appear to be the impetus for the review, he also cited disagreement between the corrected feedwater flow and key secondary plant indications as reason to question AMAG results. This position is not documented in a Condition Report.

Diagnostic testing was conducted in an effort to explain the station differences. For example, in October 1999, a test was performed at Braidwood to determine if the correction factor upstream of the venturis was indicative of "roughpipe" that may have resulted from the power washing practice at Braidwood through a comparison to another bracket installed well downstream in an un-pressure washed section (assumed "smoothpipe") of the same line.

Results were reported by AMAG / ABB-Clto agree within the accuracy of instrumentation, and therefore to eliminate the "roughpipe / smoothpipe" as a cause for the potential post-AMAG performance differences between Byron and Braidwood.

Page 24 of 35

FINAL O Throughout the remainder of 1999, Byron did not implement the AMAG correction factors.

There is no documented evidence that any substantial work was conducted in the last half of 1999 that resolved the concerns raised by the Thermal Performance Engineers. Additionally, based on interview information, the Site Engineering Director during this time period did not recommend AMAG implementation. Following the May 1999 initial testing at Byron, the ownership for resolving the plant secondary issues at Byron was not clearly defined until January 2000.

2000 In January 2000, a new Site Engineering Director took ownership for resolving the AMAG questions at Byron. A working meeting was held at Byron Station to understand and resolve the unexplained differences between Braidwood and Byron. An informal vote was taken following a review of the issues. The decision makers at the meeting voted to move forward with AMAG despite four Byron technical individuals, including the Byron Thermal Performance Engineer, voting against implementation. The Cantera AMAG project engineer attended the meeting, but abstained from voting because he felt it was the site's responsibility for implementation. The basis for deciding to move forward is unclear, however, during interview discussions, a prevailing thought provided was that there could be nothing identified as being wrong with the AMAG technology.

O Subsequently, Byron engineering, with support from Cantera, prepared the "AMAG Implementation Evaluation Report". This report recommended implementing the AMAG technology to the Byron senior management team. The final report (provided to the decision making investigation team without approval signatures or dates) did not resolve all the secondary indication issues and did not fully document the technical opposing positions (Byron and Dresden TPEs) previously discussed. Based on the recommendation to approve implementation contained in this report, Byron Senior Management approved the implementation of AMAG in March 2000.

In April 2000, prior to implementation of AMAG at Byron, the Site Design Engineering Manager made a presentation to the Nuclear Generation Group Senior Executive team and recommended implementation of AMAG at the station. In general, individuals in attendance at this presentation, were cognizant of the Byron TPEs position regarding the AMAG installation. However, based on interviews of some of the individuals present at this meeting, the presentation did not include information regarding the dissenting opinions of Stone &

Webster or the Dresden Thermal Performance Engineer, nor did it discuss the results of the January 2000 working meeting where the technical individuals voted against implementation.

Based on the presentation, the Senior Executive Team concurred with implementation at Byron. The investigation team did not identify any issues that would have indicated intentional withholding of information through interviews conducted.

O Following an NSRB subcommittee and PORC review, in the March/April 2000- timeframe, Byron implemented AMAG in May of 2000. With the correction factors applied, Byron's Page 25 of 35

FINAL electrical output exceeded Braidwood's, and no evidence was found during this investigation that would suggest any immediate follow up was initiated.

In May 2000, AMAG stated that they were so frustrated by repeated questions, from Byron, that they offered two days of training at Braidwood. The training conducted by AMAG at Braidwood was poorly attended by the sites.

Following implementation at Byron, the Braidwood Thermal Performance Enginrer wrote a letter in June 2000, which concluded:

"Enough indications (other than FWFamag) exist which demonstrate that the Byron.

Units are operating approximatelyl, 1.36% higher than the Braidwood Units. Due to this, it is expected that the Byron Units will produce approximatelyi*1.36%* more generation,.

than the Braidwood Units. However, -based upon the single measurement om FWFamag, all Units indicate nearly identical thermal power levels. The possible conclusions are:

1. The feedwater flow corrected by AMAG is in error, or

2. All of the other indicators are in error and in the same direction and of the, proper magnitude.

It is doubtful that item 2 above is true since it is unreasonable to assume that so many indications are in error, in the same direction and approximately ir the necessary.

amount.

Item 1 above raises doubt about our ability to correctly measure the actual thermal power levels at the Braidwood and/or Byron Units."

In the June 2000 report, the Braidwood TPE recommended, "Since FWFamag is the only allowed measurement for the determination of thermal power, and since it- is unreasonable to assume that all other Unit indicators are in error (in the same direction and magnitude), a review of the practices, methods, and installations of the AMAG instrumentation is indicated.

This review should be performed by an independent party with no vested interest in the outcome' of the results." No condition report was written and no owner was assigned to address the recommendation. Interviews indicated that Braidwood's management believed that the problem (electrical output differences between Byron and. Braidwood) was Byron's based on Braidwood having achieved the expected results from their thermal kit calculation.

2001 In October 2001, two condition reports were initiated to document thermal performance anomalies.

Page 26 of 35

FINAL

" The Byron TPE initiated Condition Report 78729 on October 5 to address the results of the Byron and Braidwood tests using precision instrumentation required to test uprate implementation,- and indicated that Byron units will exceed Braidwood units by approximately 119 MWeý following uprate. The disposition of this CR referred to the-'

technical reviews performed per the March 2000 Byron implementation study.

" The Byron TPE initiated CR 80251 on October 16, identifying issues concerning power uprate results. The CR documents Braidwood Unit 1 ability to achieve full power after uprate while Byron could not achieve full power. The corrective actions tied to this CR focus on what hardware or plant changes are needed to get Byron 1 to 100% power.

2002 In January 2002, Condition Report 91771 was generated by the Byron TPE to document the unexplained, unexpected differences between Byron and Braidwood plant performance. The Supervisor reviewing thips CR recommended an independent review. of the condition, but the evaluation was assigned to the Byron Thermal Performance Engineer who initiated the CR.

An Apparent Cause Evaluation (ACE) was initiated rather than a Root Cause Analysis (RCA).

Based upon a review of the Byron NRC Case Management database, it appears that In January 2002 (based on the "date raised" entry), the Byron NRC site resident inspector had question(s) concerning the unexplained differences between Byron and Braidwood documented in CR 91771. In October 2002 a copy of the ACE was given to the NRC site resident inspector. In December 2002 and January 2003 the NRC inspector reviewed the ACE and requested NRC Region III assistance with the review. (In January 2003, NRC Region III requested assistance from NRR in the review of the ACE. January 22, 2003 the NRC issued a letter to Exelon describing the potential to be operating above licensed power level.)

On February 10, 2002 two engineers from the Mid-Atlantic Regional Operating" Group (MAROG) were requested to perform an independent review of the Byron AMAG issues as part of the ongoing apparent cause evaluation (CR 91771). On February 15, 2002 their independent report was issued with the following conclusion and recommendation:

"Preliminary

Conclusion:

We agree with Byron Engineering and suspect that Byron Units 1 and 2 are operating at reactor power levels higher than indicated. This opinion is based on the many indications of higher than expected reactor power levels, e.g.,

plant output and fuel burnup. However, we were not able to identify the cause nor to quantify amount. We also agree that the most likely fault is with the AMAG correction factors applied to Byron at about 2.0 whereas Braidwood is 0.6. This opinion is based.

on the identification that the divergence began at the time of AMAG implementation,,"

and that this single change will result in the significant difference in plant outputs..,

Byron also uses an RCP heat input of 16.6 MWth where as Braidwood uses the standard' 16.0,,MWth., However, if the Byron RCP is truly more standard' (14.0MWth)ýý ,"

Page 27 of 35

FINAL then the overpower would only be about!0.6 MWthJ which might get about 0.2 MWe.

Thus this alone could not account for the entire divergence."

"RECOMMENDATION: A more in-depth investigation should be planned to either justify the difference, or to identify the fault, be it at Byron or at Braidwood. We fully expect that a full review of the activities at each plant surrounding the AMAG implementation will ultimately identify the cause of the difference and therefore will identify either the fault or the justification for the difference. THE TRUTH IS OUT THERE.

We anticipate that such an investigation will likely require 2-4 individuals and at least one month to as much as three months, There are rrAny factors and areas that should be investigated, e.g. the thermal kit models (PEPSE), pump performance indications for TDFW, MDFW, Condensate Primary and Booster pumps, venture loss coefficients, the AMAG measurements and calculations, and fuel consumption underestimates."

Based on these conclusions, the Byron Site Vice President decided to reset their correction factors on February 15, 2002 to 1, effectively derating both units approximately:23 MWe. t ,

In response to the MAROG engineer's concerns, Nuclear Fuels Management (NFM) issued a letter on February 20, 2002 (NFM-MW:02-074) stating, "This letter will conclude that neither the differences seen in the development of reload cores nor the variations in cycle depletion data, since the implementation of AMAG, can confirm or repudiate the validity of the AMAG results." On February 24, 2002, the correction factors for Byron were re-implemented based on NFM finding insufficient evidence to substantiate the MAROG engineer's fuel burnup issue. Secondary plant issues and recommendations contained in the MAROG report were not addressed at that time.

Troubleshooting in early 2002 focused on ultrasonic flow measurement results, the on-line calorimetric, and correction factor (CF) calculations to resolve CF differences, including:

" Based on information from the Braidwood and Byron root cause evaluation, Westinghouse / AMAG technical reviews at Byron performed in 2002 indicated that the AMAG instrumentation behavior observed on Unit 1 did not exhibit the consistent performance seen in other industry AMAG installations. Specifically, the calculated venturi CF varied unexpectedly as a function of power and the CFs appeared to change after power uprate was implemented. However, Westinghouse, based on the data available at that time, stated that the results of AMAG testing and evaluations indicated that the AMAG components were performing in an appropriate manner and that the test criteria for acceptable AMAG performance were being met. As part of the technical review, an AMAG bracket was installed on the feedwater common header at Byron Unit 1 in late February 2002.

" A test was performed on Byron Unit 1 in March 2002 that indicated the sum of the flows in the four feedwater loops plus tempering lines equaled the flow in the feedwater common_

header at the high end of the calculated -allowable statistical limit (difference of!0.699%'I Page 28 of 35

FINAL 91%',

with acceptance criteria of 0.106%.),at the coastdown power level of approximately (reference Calc 059-PENG-CALC-084 Revision 1 dated 4/24/02). A similar-test, performed prior to AMAG optimization, was performed at approximatel -

2/27/02 and 2/28/02 and indicated differences of"1.1-1.3%.(' 1 -

On March 29, 2002, CR 101618 was initiated due to unexpected and unexplained Byron Unit 2 FW venturi calibration data. The Unit 2 AMAG calculation suggested that a,0.6 /oincrease F -4 in power would be possible if the new correction factors were used. The largest change, in calculated correction factor, was on loop "2A". The new correction factor was not implemented because it was not corroborated by other plant data. It was also noted in the CR that AMAG recommends not implementing unexplained correction factor changes of more than 0.2% to 0.3%. The CR documents that significant changes have not been observed on Byron Unit 1.

In April 2002, a Westinghouse report was issued on CROSSFLOW TM related activities to support Byron station. The report indicated that while the system was not performing as consistently as other industry installations, the results of testing and evaluations indicated system performance criteria were being met. Recommendations included collection of continuous data for six months to trend for any sudden or gradual. changes in correction factors and to correlate them to changes in plant condi)ns. The CF changes were not challenged when this report was reviewed and approved by the Byron PORC. Additionally, a condition report was not written to document the issues and recommendations contained in this Westinghouse report.

In June 2002, following B2F23 (Unit 2 SG tube leak repair) the observed anomaly from March 2002 (reference CR 101618) with the Unit 2 correction factor constants was no longer present. A new correction factor of 1.69 was implemented. An Action Tracking Item was generated to perform long term trending on the Unit 2 AMAG constants. Data collection continued on Unit 2.

In July 2002, in response to a request by the Byron Site Vice President, The Corporate Vice President of Engineering issued a letter summarizing corporate engineering's review of the AMAG issue. Conclusions included; the Byron installation was correct, the equipment was performing within specifications, the data was being properly interpreted, the correction factor and calorimetric were being properly calculated. Corporate engineering recommended continued use of AMAG at Byron.

On August 30, 2002, the Corporate V.P. of Engineering issued a follow up letter, which concluded that, "...the Byron installation is correct." The letter further states, "It is a station responsibility to determine if other input values are correct. If this has been done and the station uses the calorimetric to determine core thermal power and maneuvers to stay under 3586.6 MWth, the station will not have nor will exceed licensed thermal power."

Page 29 of 35

FINAL Interview information indicated that, on several occasions (three to five times estimated) during the summer of 2002, the Byron Thermal Performance Engineer expressed concern to the Nuclear Oversight Manager regarding the lack of progress in solving the Byron/Braidwood differences identified in CR 91771. The Byron Nuclear Oversight Manager, through interview, was cognizant of the Byron TPE concerns, but, he could not produce documentation of Nuclear Oversight follow-up with these concerns or AMAG implementation reviews.

The Apparent Cause Evaluation (ACE) for Condition Report (CR) 91771 was completed in November 2002. This CR was initiated in January 2002 to identify the cause of the differences between Byron and Braidwood electrical output. After ten months and numerous extensions, the ACE investigation, begun in February, concluded that the apparent cause was indeterminate. On November 21, 2002 Bryon's Plant Operations Review Committee (PORC) convened to review the ACE. The PORC meeting minutes stated: "There is insufficient information to know if we are within the license basis..." and "This could be a nuclear safety issue if an independent person disagrees with our conclusion." No documentation could be found regarding resolution of these comments contained in the PORC meeting minutes. Additionally, this indeterminate ACE was approved by the Management Review Committee (MRC) and by the PORC, but no Root Cause Analysis was commissioned to identify the cause. Finally, the investigation team was unable to conclude that a risk assessment was performed for the indeterminate ACE as required by procedure.

2003 In January 2003, Westinghouse issued letter CAE-03-4 to address the unexpected CF identified on the Byron "2A" loop. (as noted in CR 101618, March 29, 2002). This report did not identify a known cause for the loop "A" discrepancies and recommended additional data be collected for a minimum, period of six months for each of the Byron units to correlate the unexpected changes in CF to specific plant operating conditions and to identify the reason for the CF variation. Six months of data had already been collected as recommended by Action Tracking Item in June 2002, but this information was not transmitted to Westinghouse.

On January 22, 2003, The NRC issued a Request for Additional Information (RAI) Letter to Exelon, stating, "Byron Unit 1 may be operating above its licensed thermal power level." The next day, CR 140753 was initiated to document the NRC's concerns On January 24, 2003, a Meeting was held at NRR between Exelon and the NRC to discuss the NRC questions. Exelon stated at this meeting that Byron was not operating above its licensed thermal power level.

On February 5, 2003, Exelon responded to the January 2003 RAI from the NRC. This response states: "...we have concluded that the ultrasonic feedwater flow instrumentation for Byron Station, Unit 1, was installed consistent with the guidance in RIS 2002-03; and that Byron Station, Unit 1, is operating within its licensed thermal power limit."

Page 30 of 35

" i" FINAL O In March 2003, an Exelon engineering test plan was initiated in response to continued NRC questions. The plan included installing FW main header ultrasonic flow measurement devices at Byron and Braidwood to compare with individual loops. The plan also initiated continuous on line data acquisition of Byron ultrasonic flow data to observe correction factor behavior during steady state and during various power maneuvers (i.e. coastdown, return to power following refuel outage).

In April 2003, in its inspection report, the NRC issued an Unresolved Item for the Byron power discrepancy. The inspection report states: "The inspectors considered the licensee's evaluation to be of appropriate scope and depth. However, based on the potential for Byron Unit 1 to be exceeding the licensed thermal power limit, and the technical complexity of the issue, the inspectors generated a task interface agreement with the Office of Nuclear Reactor Regulation (NRR) for additional review."

In May 2003, as a result of an Exelon corporate initiative to resolve Byron and Braidwood megawatt electric discrepancies, an AMAG bracket and instrumentation were installed on the feedwater common header at Braidwood Unit 1 to compare the flow in the common header to the sum of the flows in the four feedwater loops. Additional installments to further understand the CF changes continued through August.

O In July 2003, the NRC issued a second Request for Additional Information (RAI) Letter regarding potential overpower condition.

On July 18, 2003, CR 168199 was initiated to document AMAG constants on the Byron "2A" loop greater than "as found." Overall change of - 0.2% is within 0.69% instrument random error.

The results at full uprated power operation for the Byron Unit 1 common header test, documented in Westinghouse letter CAE-03-069 dated August 28, 2003, reported the difference between the sum of the AMAG measurements in the four feedwater loops and the common header was outside the acceptable statistical limits (difference of 1.572% mass compared to a maximum allowable statistical limit of.0.70% mass). -*

Review of the current and past collected information indicated.that the CF had a history of unexpected changes in the four individual feedwater loops. In addition, the CF appeared to vary as a function of power (not consistent with expected behavior) and individual feedwater loop flow measurements were non-linear with respect to the venturi output (again not consistent with expected behavior). Westinghouse / AMAG stated that the variations were associated with hydraulic noise contamination of the signal, creating a bias, either positive or negative, affecting the measured results. Westinghouse / AMAG also recommended in letter BCAE-03-70, dated August 29, 2003, that the flow measurement in the Byron Unit 2A loop be returned to the venturi due to hydraulic noise contamination. These issues were documented Page 31 of 35

FINAL in CR 173510. As a result of the notifications from AMAG / Westinghouse on August 28, 2003, Byron Station Management made a decision to return the AMAG correction factors to 1.0 on both units, pending resolution of the issue. This action returned the feedwater flow measurement methodology to the uncorrected feedwater venturis only.

On August 28, 2003, Byron notified the NRC of a potential violation of maximum power level on Units 1 and 2.

Braidwood performance was reviewed on August 30, 2003 as documented in Westinghouse letter CCE-03-78 dated September 2, 2003. Although some hydraulic noise contamination was seen on Braidwood Unit 1, Westinghouse / AMAG recommended Braidwood Unit 1 continue to operate in its current condition since the common header test performed in May 2003 verified the composite flow being measured by the four feedwater loops was accurate.

and valid (within 10.021%) and the CF had not been corrected or changed since that test.

Westinghouse also. recommended that the flow measurement in the Braidwood Unit 2A and 2B loops be returned to the venturis, since hydraulic noise was seen on these loops, but there was no common header data at that time to provide confirming data on the accuracy of the individual loops. These issues were documented in CR 173819 at Braidwood Station, which implemented the recommendations from Westinghouse for Braidwood Unit 2.

On August 31, 2003, Braidwood notified the NRC of a similar potential violation of maximum power level on Unit 2.

Westinghouse / AMAG review and evaluation of these issues, documented in the three previously identified Westinghouse letters, led to a preliminary conclusion that the inconsistent measurements in the four feedwater loops were being driven by a.variable affecting the ultrasonic flow signals (and ultimately the calculated time delay) measured by the AMAG electronics. Using frequency spectrum analysis, the variability in the time delay measurement was determined to potentially be the result of hydraulic noise contamination. A review of the AMAG installations at Braidwood and Byron indicated the presence of hydraulic noise contamination on several, but not all, of the individual feedwater line measurements as follows: Byron "IA" through "ID" (letter CAE-03-069), Byron "2A" (letter CAE-03-070),

Braidwood "1A" and "1B" (letter CCE-03-078), and Braidwood "2A" and "2B" (letter CCE 078). Subsequent data acquisition also identified hydraulic noise on the Byron "2B" loop.

This noise appeared at varying magnitudes at frequencies of approximately-5, 10, 15, and 20 hertz.! The hydraulic noise contamination was absent from the two initalled feedwater common header locations at Byron Unit 1 and Unit 2, and Braidwood Unit 1.

Numerous causes for the apparent noise were considered and investigated. Detailed evaluation by a Root Cause Team for this event determined that the root cause was noise contamination of the AMAG ultrasonic signal caused by acoustic resonant response of the feedwater piping system. Potential resonant frequencies were predicted using several theoretical methods. Dynamic pressure measurements were taken on each of the four Byron Unit 2 feedwater lines at a low point drain located near the flow measurement venturi.

Page 32 of 35

FINAL S and actualof plant Analysis response the dynamic and suggests pressure that the data shows principalbetween agreement resonance is likely that the theoretical segment of piping between the feedwater regulating valve and the steam generator. Data of the methods analysis also supports the existence of low frequency resonant system response consistent with the noise frequencies and reflects the magnitude of noise differences in the four feedwater loops (i.e., noise is highest in loop "2A"). The evaluation shows that the Byron feedwater lines natural frequency is below 25 Hz. Computer modeling discussed below supports an adverse impact on the CF at frequencies below 25 Hz.

AMAG provides a diagnostic program, "DIAGNOSE" with the system. Contained in DIAGNOSE is a tool for frequency spectrum analysis. The DIAGNOSE program was not part of the formal installation or check-up procedures and is typically not utilized unless there is a suspicion that there may be noise interference. Interview information and document reviews yielded a belief, on the part of AMAG, Inc., that the DIAGNOSE program was utilized on at least one of the loops at Byron Station during initial installation with no noise interference being identified, however, there is no formal documentation to support this. Noise was not suspected at the time of installation, nor in March 2002. In March 2002, AMAG, Inc. was preoccupied with investigating hardware issues; they did not identify any reason at that time to suspect noise or to utilize the DIAGNOSE program based, in large part, on the fact that noise was not present during installation. The activities in March 2002 were focused on the differences between Byron and Braidwood in terms of the physical AMAG installation. In late August 2003, this tool was used to identify the noise contamination. Until recently, there was no common header installation to allow calculation of statistical variance.

Page 33 of 35

'. 4 FINAL Attachment 2 Braidwood / Byron AMAG Implementation Investigation Team Interview List Gary Bal Plant Engineering Branch Manager Carl Dunn Site Engineering Director Don Hildebrant Thermal Performance Engineer Tom Joyce Plant Manager Mike Smith Plant Engineering Managjer Brad Adams Site Engineering Director Dave Eder Thermal Performance Engineer Bill Grundmann Regulatory Assurance Manager Ken Hansing Nuclear Oversight Manager Dave Hoots Plant Manager Rene Irby Regulatory Assurance Steve Kuczynski Site Vice President Kevin Passmore Plant Engineering Branch Manager Doug Spitzer Plant En ineerinq Manager Ken Ainger, Corporate Licensing Manager Joe Bauer Corporate Licensing Engineer Jeff Benjamin Vice President Licensing and Regulatoy Affairs Bill Bohlke Senior Vice President Chris Crane Chief Operating Officer Jeff Drowley Corporate Mechanical and Structural Manager Alex Javorik Corporate Equipment Reliability Director Keith Jury Corporate Licensing Director Bill Kouba Corporate Engineering Director Bill Levis Vice President Rich Lopriore Vice President Operations Support Jim Meister Vice President Engineering Chip Pardee Senior Vice President Nuclear Services Dave Wozniak Vice President Special Projects Jack Skolds President and Chief Nuclear Officer

!xe Ot -.

Chris Brennan Engineering - Kennett Square Gary Loeb Engineering - Kennett Square Les Guthrie Nuclear Oversight Director Tom Roberts Former Exelon employee Joe Williams Site Engineering Director - Clinton Station Page 34 of 35

FINAL I - e'- -

Advanced Measurements and Analysis Group, Inc.

Vahid Askari j Engineering and Operations Armando Lopez President Westinghouse Rhonda Doney _

Susan Hauser Project Manager Page 35 of 35