ML20246C352

Rept on Events of 890423 & 0505 Involving Backflow Through Auxiliary Feedwater Sys

ML20246C352
Person / Time
Site:	Comanche Peak
Issue date:	08/18/1989
From:	TEXAS UTILITIES ELECTRIC CO. (TU ELECTRIC)
To:
Shared Package
ML20246C349	List: ML20246C345 ML20246C352
References
NUDOCS 8908240340
Download: ML20246C352 (72)

Text

{{#Wiki_filter:m 6.

1w Attachment t> TXX-89596

' August-18, 1989 l 1 '- Page 1 of 72 L l:;- 1,-- - - l.' l TU ELECTRIC COMPANY COMANCHE PEAK STEAM ELECTRIC STATION DOCKET NOS. 50-445 AND 50-446-REPORT ON EVENTS OF APRIL 23 AND MAY 5, 1989 INVOLVING BACKFLOW THROUGH THE AUXILIARY FEEDWATER SYSTEM AUGUST 18, 1989 '8908240340 890818 ^ ) DR ADOCK 0500 5 I

Attachment ts TXX-89596 August 18, 1989 Page 2 of 72 TABLE OF CONTENTS I. . EXECUTIVE

SUMMARY

A. INTRODUCTION AND PURPOSE. B. DESCRIPTION OF APRIL 23 AND MAY 5 EVENTS C. .TU ELECTRIC INVESTIGATION OF ROOT CAUSES D. SIGNIFICANCE OF THE APRIL 23 AND MAY 5 EVENTS E. PRECURSORS F. CORRECTIVE ACTIONS, PREVENTIVE ACTIONS, LESSONS LEARNED AND ASSOCIATED IMPROVEMENTS G.

SUMMARY

AND CONCLUSIONS II.. INTRODUCTION AND PURPOSE III. DESCRIPTION OF THE APRIL 23 AND MAY 5 EVENTS A. DESCRIPTION OF THE APRIL 23 EVENT B. DESCRIPTION OF THE MAY 5 EVENT IV. TU ELECTRIC INVESTIGATION OF ROOT CAUSES A. INTRODUCTION B. INVESTIGATION FOCUS C. INVESTIGATION OF THE ROOT CAUSE OF THE CHECK VALVE FAILURES D. INVESTIGATION OF THE ROOT CAUSES OF OPERATOR ERRORS E.

SUMMARY

V. SIGNIFICANCE OF THE APRIL 23 AND MAY 5 EVENTS A. ACTUAL SIGNIFICANCE B. PIPING AND SUPPORT INTEGRITY C. IMPACT ON CONTAINMENT PENETRATIONS D. IMPACT ON INSTRUMENTATION E. POTENTIAL SIGNIFICANCE OF THE EVENTS IF THEY HAD OCCURRED DURING OPERATION VI. PRECURSORS A. NUCLEAR INDUSTRY EXPERIENCES WITH BW/IP CHECK VALVES B. PREVIOUS OCCURRENCES AT CPSES C. CONCLUSIONS VII. CORRECTIVE ACTIONS, PREVENTIVE ACTIONS, LESSONS LEARNED, 4 AND ASSOCIATED IMPROVEMENTS l l A. INTRODUCTION l B. CORRECTIVE AND PREVENTIVE ACTIONS l 1 4

l5. Attachment to TXX-89596 -August 18, 1989 Page 3 of 72 1. CORRECTIVE ACTIONS FOR HARDWARE 2. PREVENTIVE ACTIONS C. LESSONS LEARNED AND ASSOCIATED IMPROVEMENTS 1. MANAGEMENT AND SUPERVISION OF OPERATIONS 2. CORRECTIVE ACTIONS 3. COMMUNICATIONS AMONG OPERATORS AND SHIFTS 4. -PERSONNEL AWARENESS OF OPERATING EVENTS AND EQUIPMENT FAILURES AND THEIR IMPLICATIONS FOR SYSTEM OPERABILITY D. CONCLUSIONS VIII.

SUMMARY

AND CONCLUSIONS FIGURES FIGURE I - BW/IP PRESSURE SEAL CHECK VALVE-FIGURE 2 - FLOW PATH FOR THE APRIL 23 EVENT FIGURE 3 - FLOW PATH FOR THE MAY 5 EVENT APPENDICES APPENDIX 1 - RESPONSE TO NRC CONCERNS APPENDIX 2 - CHECK VALVE BACKLEAKAGE TESTING APPENDIX 3 - CHECK VALVE MODIFICATION AND MAINTENANCE HISTORY APPENDIX 4 - IDENTIFIED MATERIAL CONCERN-APPENDIX 5 - RADIOGRAPHY, INSPECTIONS AND COMPUTER ASSISTED DRAWINGS FOR BW/IP CHECK VALVES APPENDIX 6 - EVALUATION OF AFW CHECK VALVES AGAINST EPRI GUIDELINES L

Attachment to TXX-89596 August 18, 1989 Page 4 of 72 1. EXECUTIVE

SUMMARY

A. Introduction and Purpose On April 23 and again on May 5,1989, during hot functional testing of Comanche Peak Steam Electric Station (CPSES) Unit 1, backflow occurred from some of the steam generators through portions of the Auxiliary Feedwater (AFW) System to the Condensate 4 Storage Tank (CST). This event happened because of hung open check valves and coincident failure of Auxiliary Operators (A0s) to operate manual valves in the sequence specified by procedures. Texas Utilities Electric Company (TV Electric) established a Task Team to investigate these events. The NRC also established an-Augmented Inspection Team (AIT). This report discusses the TU Electric Task Team investigation results, and the operational weaknesses identified by the AIT Report and by the NRC in a meeting on July 17, 1989. l B. Description of the April 23 and May 5 Events On April 23, 1989, a partial blowdown of steam generators 1, 2 and 4 occurred through AFW System lines to the CST. The event occurred while realigning valves following a preoperational test. An Auxiliary Operator (A0) began to open an AFW pump test isolation valve while another A0 was closing the pump discharge valve. Operation of these two valves at the same time is not in accordance with the approved procedure which requires sequential valve operation. This operation coincident with hung open check valves created the backflow path from the steam generators to the CST. When the test isolation was fully closed, the backflow stopped and steam generator levels stabilized. Backflow occurred for approximately fifteen to twenty minutes. On May 5,1989, a partial blowdown of steam generators 1 and 3 also occurred through AFW System lines to the CST. This event occurred while aligning the system to perform an operability test. An A0 began to close an AFW pump discharge valve while the pump test isolation valve was being opened. This violated the approved procedure which requires sequential valve operation. This operation coincident with check valve failures created a backflow path from the steam generators to the CST. Backflow occurred for approximately twelve minutes and was stopped when the pump discharge valve was closed. Subsequently, the Reactor Operator (RO) directed an A0 to close the pump test isolation valve; however, it was inadvertently left one-quarter turn open due to mechanical binding. This alignment re-initiated backflow through the system. Eventually, the pump test isolation valve was completely closed. Backflow occurred for approximately sixty-six minutes primarily due tc the time i \\

~ Attachment te TXX-89596 August 18, 1989 Page 5 of 72 required to identify that the test isolation valve was partially open. C. TV Electric Investigation of Root Causes To assure a thorough investigation of the April 23 event the Executive Vice President for Nuclear Engineering and Operations established a multi-disciplined Task Team. TU Electric senior management emphasized that the Task Team should concentrate on a thorough and deliberate determination of root causes. Based upon its reviews, the Task Team determined that certain check valves in the AFW System had become hung open due to the discs becoming lodged beneath the seat lip (see Figure 1). The condition resulted from an elevation difference between the valve seat and disc created by incorrect reassembly instructions. The Task Team interviewed Operations personnel and reviewed available information to determine the root causes of the operator errors on April 23 and May 5. Investigation determined that valves were operated simultaneously due to a misunderstanding of the administrative controls governing the sequencing of procedure steps. In part, this lack of understanding was attributable to the absence of guidance in applicable Operations Department Administrative Procedures (0DAs). The Task Team interviewed operators to determine why the isolation valve was not fully closed on May 5. The Team determined that the operators believed the valve to be closed because of the resistance felt in closing the valve. The A0s were unable to visually determine the degree of valve closure because of the location of the valve with respect to its operator. D. Significance of the April 23 and May 5 Events These events did not and could not have resulted in any radioactive release because they occurred during preoperational testing and prior to fuel load. Therefore, the events did not pose any threat to public health and safety. The Task Team evaluated the impact of the backflow on piping and support integrity, containment penetrations, and instrumentation. Analysis identified several areas where piping Code allowable stresses were exceeded. Subsequent Ultrasonic Testing (UT) of the pipe verified that no plastic deformation had occurred. Thus, the stresses resulting from the elevated temperature were within the elastic range for the piping material and no piping needs to be replaced. One pipe support was visibly damaged and has been replaced. Additionally, analysis determined that ten supports were overloaded. Nonconformance Reports were written to require QC examination of the significant attributes of these supports and

j w. i ' ' Attachment to TXX-89596 i August 18, 1989 ~ -Page 6 of 72 3 no. deviations or deficiencies were found. Finally, containment penetrations were determined to be unaffected by the events. The . impact on some flow' transmitters is. still under evaluation. The TU Electric Task Team performed an evaluation of the potential effects of a sinnilar malfunction cf BW/IP check valves during plant operation.. In the absence of a line break nr a manual valve misoperation, the failure of.the valves lacks significance because of the ' absence of a backflow path. In the event of a loss of AFW. flow to'the steam generators for any L reason the Emergency Response Guidelines would require operator actions to commence Reactor Coolant System (RCS) cooldown' using systems other than the AFW and Feedwater Systems. E. Precursors-The Task Team reviewed industry experience with check valves and previous check valve problem at CPSES to determine whether other check valves experienced the same failure mode 'as the check valves involved in the April 23 and May 5 events. Although various concerns about the performance of check valves have been experienced by the industry, there was no generally available information prior to the April 23 event that the BW/IP check valves were likely to malfunction due to an elevation difference between the valve disc and seat. The Task Team did identify check valves at CPSES that may have failed in a manner similar.to those on April 23. These failures occurred in 1985, and on April 5 and 19, 1989. TU Electric has concluded that the existence of these failures indicate that improvements are warranted in the documentation, reporting, and evaluation of plant events and equipment failures. As. discussed below, TU Electric is implementing improvements in these areas. F. Corrective Actions. Preventive Actions. Lessons learned and Associated Improvements TU Electric is taking corrective actions for affected hardware, including the following: 1) BW/IP pressure bonnet check valves are being inspected and the discs and seats aligned as necessary, and post modification operability tests are being performed to verify that the valves are fully closed; 2) affected piping is being repainted, the damaged pipe support has been replaced, and other pipe supports have been inspected with no deviations or deficiencies found; 3) potentially affected flow transmitters will be recalibrates and replaced if necessary; and 4) binding of the isolation valve will be evaluated and corrected. l _--__.--__________J

Attachment to TXX-89596 August 18, 1989 Page 7 of 72 TU Electric is taking actions to address the root causes and prevent recurrence of events similar to those on April 23 and May 5,- including the following: o The valve reasseMly procedure has been revised to include a requirement for determining the elevation adjustment necessary to avoid interference between the disc and the seat. o The Operations administrative procedure which provides guidelines on the use of procedures has been revised to emphasize the requirement that procedure steps are to be performed in sequence unless specific exceptions are satisfied, o The Shift Operations Manager has developed and is implementing an action plan to enhance procedural compliance. The need to complete procedural steps sequentially will continue to be emphasized and will become part of Operator Requalification and Replacement Training, o Reach rod operators for safety-related valves will be evaluated for proper operability and human factors considerations. In addition, TU Electric has identified a number of areas where improvements could be made. These areas are discussed below. o Manaaement and Supervision of Operations - TU Electric is taking action to expedite the transition from a construction to an operating attitude, to provide Operations with greater control of the project, to improve the reporting of plant events and equipment failures to Operations management and supervision, and to enhance management's awareness of manpower needs for specific tasks. o Corrective Actions and Evaluation of Plant Events and Eouipment failures - TU Electric is taking action to improve the documentation and reporting of plant events and equipment failures, to increase the aggressiveness and timeliness of investigations of plant events and equipment failures, and to improve future team evaluations by TU Electric. i o Communications Amona Operators and Shifts - TU Electric is taking action to improve communications among operators and communications between shifts. o Personnel Awareness of Operatina Events and Eouipment i failures and Their lmolications for System Operability - TV Electric is taking action to increase the awareness of 1

a Attachment to TXX-89596 August 18, 1989 Page 8 of 72 Operations personnel to Work Requests and their implications for plant operability, and to improve the availability of information regarding plant events and equipment failures to Operations personnel. The specific improvercentr that TU Electric is making in each of these areas is discussed in detail in Section VII.C of this report. NRC concerns as identified in the AIT Report and during the July 17, 1989 meeting are discussed in Appendix 1. G. Summary and Conclusions The April 23 and May 5 events were of no immediate safety significance because there was no fuel in the reactor and Unit I was not radioactive. A similar event during operation coupled with a steam line or AFW line break could have resulted in loss of AFW. Operator action in accordance with procedures would have maintained the reactor in a safe condition. TU Electric is taking corrective action for the deficiencies in the hardware identified by these events. Additionally, TU Electric is taking action to address the root causes of the events and to prevent recurrence of similar events. Finally, TV Electric has identified lessons learned from these events and is taking actions to improve the management and supervision of Operations personnel, to improve corrective actions for plant events and equipment failures, to improve communications among Operations personnel and between shifts, and to improve personnel awareness of operating events and equipment failures and their implications for system operability.

Attachment to TXX-89596 August 18, 1989 Page 9 of 72 II. INTRODUCTION AND PURPOSE Comanche Peak Steam Electric Station (CPSES) is a two-unit Westinghouse pressurized water reactor (PWR) c ened by Texas Utilities Electric Company (TU Electric). During hot functional testing of CPSES Unit 1 on April 23 and May 5,1989, backflow occurred from some of the steam generators through portions of the Auxiliary Feedwater (AFW) System to the Condensate Storage Tank (CST) because of hung open check valves, coincident operator error and, on May 5, mechanical binding of an isolation valve. The NRC issued a Notice of Violation to TU Electric on May 18, 1989, based in part on the April 23 event. Additionally, on May 5, 1989, the NRC issued a Confirmation of Action Letter (CAL) which confirmed that certain actions would be taken by TU Electric in response to the events and which provided for an NRC investigation of these events by an Augmented Inspection Team (AIT). The results of the AIT investigation were provided in a letter to TU Electric on July 10, 1989. The letter described several operational weaknesses identified by the AIT during its investigation. Additionally, in a meeting at Rockville, MD on July 17, 1989, the NRC identified similar operational weaknesses resulting from the backflow events and other recent NRC violations. TU Electric informed the Nuclear Regulatory Commission (NRC) of these events on April 24 and May 6, respectively. Additionally, TV Electric established a Task Team on May 1, 1989 to investigate the causes and significance of these events and to recommend corrective actions. Based on the results of those investigations, TU Electric determined the events were potentially reportable under 10CFR50.55(e), notified the NRC on May 19, 1989, and provided an interim report to the NRC on June 19, 1989 which categorized the events as reportable (see SDAR CP-89-15,TXX-89429). Two INP0 Nuclear Network Notices were issued by TU Electric on May 17 and May 24, 1989. The May 17 Notice generally { described check valve backleakage. The May 24 Notice questioned i industry contacts concerning check valve backleakage due to mechanisms I other than valve distortion, debris or normal wear. To date no responses have been received. Additionally, on June 1,1989, TU Electric notified BW/IP of the defects that existed in its check valves and indicated that they may be potentially reportable under 10CFR21. I This report discusses the results of the investigation of the April 23 and May 5 events by the TV Electric Task Team, responds to the NRC's July 10 letter, and addresses the operational weaknesses identified by TU Electric and by the NRC at the meeting on July 17, 1989. The remainder of this report is divided into the following sections: o Section III provides a description of the events on April 23 and I May 5. l

Attachment to:TXX-89596 August 18, 1989: Page 10 of 72 o Section IV describes the investigations performed by the TU Electric Task Team and summarizes the results of the investigations, including identification of the.causes of the' events on April 23 and May 5. o Section V. discusses the significance of the events on April 23 'and -May 5. o Section VI describes prior deficiencies involving BW/IP check valves at CPSES and other plants, 'and discusses the. relevance of 'these deficiencies to the events on April 23 and May 5. o Section VII discusses TU Electric's corrective and preventive actions for the April 23 and May 5 events, the lessons' learned ~ from these events, and improvements being made by TV Electric. This.section also addresses the weaknesses identified by the NRC 'in its July 10 letter and at the meeting on July 17,~1989. o Section VIII presents TU Electric's conclusions as a result of. these events. o The six appendices provide ' additional information on TU Electric's response to NRC concerns; check-valve backleakage testing; check valve maintenance history; two unrelated material deficiencies relevant to BW/IP check valves; Task Team inspection techniques; and evaluation of AFW check valves against EPRI Guidelines, respectively.

Attachment to TXX-89596 August 18, 1989 Page 11 of 72 III. DESCRIPTION OF THE APRIL 23 AND MAY 5 EVENTS 1 A. Description of the April 23 Event: On April 23, 1989, a partial blowdown of steam generators 1, 2 and 4 occurred through AFW System lines to the CST. This blowdown created abnormally high temperatures in. system piping (greater than 2000F in AFW System piping and approximately 5000F in l Feedwater System piping) and reduced water levels _ in the three steam generators approximately 12% of the narrow range indication in 15 to 20 minutes. The event caused blistering and discoloration of the paint on the TDAFWP discharge piping. Prior to the event the plant conditions were as follows: 1. Reactor Coolant System (RCS) pressure control was in automatic 2. RCS pressure was 2235 psig 3. RCS temperature was 5570F 4. Stean. Dump control was in automatic 5. Steam generator pressure was 1100 psig 6. All Main Steam Isolation Valves (MSIVs) were open 7. Total steam generator blowdown flow was 45 gpm 8. Motor Driven Auxiliary Feedwater Pump (MDAFWP) 2 was in operation with a flowrate of 120 gpm 9. No fuel was in the reactor The event occurred while realigning Turbine Driven Auxiliary Feedwater Pump (TDAFWP) valves following a preoperational test. The TDAFWP flow control valves were fully open and the motor operated isolation valves were throttled and deenergized. The TDAFWP was started to provide flow to the steam generators for three minutes and was then tripped from the Control Room in anticipation of realigning it to the test header for a three hour run to perform a hot alignment check. The Reactor Operator (RO) used approved procedures to realign and run the TDAFWP to the test header. He briefed the Safeguards Building Auxiliary Operator (AO) and then sent him to close valve IAF-041 (TDAFWP DISCH ISOL), and open valve IAF-042 (TDAFWP TST ISOL). Upon reaching the TDAFWP room the A0 first opened valve 1AF-042 approximately 1/4 of a turn. He then proceeded to close IAF-041. The A0 turned the valve operator on IAF-042 one-quarter j turn because the open/close direction tags were missing on valve IAF-041, and he wanted to verify the proper rotation to open the valve. He did not realize that turning the operator this small amount could unseat the valve. The A0 then requested and was I provided assistance to operate these valves. When three other A0s l arrived at the TDAFWP room, one A0 began to fully open 1AF-042 and another A0 relieved the Safeguards Building A0 and continued to i close IAF-041. j 1 l l j

Attachment to TXX-89596- .{ N . August 18, 1989-4 Page 12 of 72 Operation of these two valves at the same time is' not in accordance with the approved procedure which requires that 1AF-041 be closed before IAF-042 is opened. This. operation, coincident with hung open check valves and the unseating of a Feedwater Isolation Bypass. Valve, FIBV,-{which is not intended to prevent backflow at pressures greater than containment design pressure) created an.open backflow path from the steam generators to the CST (see. Figure 2). The R0 noticed that steam generator water' levels were decreasing aslthe valves were being operated. The R0 increased MDAFWP 2 discharge flow to 400 gpm and noticed that only steam generator 3 was receiving flow at approximately 20 gpm. Recognizing that a potential backflow condition may exist, the R0 directed the Safeguards Building A0 to verify that valve IAF-055 (MDAFWP 02 TST ISOL) was closed.. The. Safeguards Building A0 reported back'that IAF-055 wac closed, but also stated that the paint on the TDAFWP discharge piping was bubbling. The R0 then told the A0 to close IAF-042. When IAF-042 was fully closed, the backflow stopped and . steam generator levels stabilized. Backflow occurred for approximately fifteen to twenty minutes. B. Description of the May 5.1989 Event On May 5,- 1989, a partial blowdown of steam generators 1 and 3 occurred through AFW check valves and lines to the CST. The blowdown caused paint discoloration of the MDAFWP 1 discharge - piping to steam generator 1 and TDAFWP discharge lines to steam generators 1 ar.d 4. An estimated 20% of narrow range level in steam generator I was displaced through the lower feedwater nozzle into the AFW System, while an estimated 11% of narrow range level in steam generator 3 was displaced from the lower feedwater l nozzle. Steam generator 3 did not blowdown sufficiently to cause hot water to reach AFW piping and discolor paint on the AFW line to steam generator 3. Prior to the event, the plant conditions were as follows: 1. RCS pressure control was in automatic 2. RCS pressure was 2235 psig 3. RCS temperature was 5570F 4. Steam Dump control was in automatic 5. Steam generator pressure was 1100 psig 6. Steam generator blowdown was isolated 7. All MSIVs were open L 8. All'AFW pumps were shutdown 9. No fuel was in the reactor The event occurred while aligning the system to perform an Auxiliary Feedwater operability test to familiarize Operations 1 1 )

,3j a

s y y6 N TAttachment to TXX-89596' 1 August 18, 1989' Q -Page 13 of 72 y personnel with Operation Test Procedures'(OPT). Both of the. MDAFWPs had been stopped for system realignment. An AO.and the Shift Technical Advisor. (STA) were sent-from the Control Room.to -begin the system alignment.: After arriving at the MDAFWP 2 room,' 'the A0 began.to close IAF-054-(MDAFWP 2 ISOL), and requested assistance in manipulating the' remainder of the AFW valves, including valve-1AF-055.: When an A0 arrived to provide-the requested assistance, the valves:were operated at the same time.- This violated approved procedures, which required that 1AF-054 be . closed before IAF-055_ is opened. ~ This operation, coincident:with hung open check valves and the unseating-of the FIBV, created an open backflow. path from.the steam generators to the CST (see Figure 3). Backflow occurred for approximately twelve minutes-until 1AF-054 was closed. After closing 1AF-054, the A0 verified the pump cross connect: valves were closed. MDAFWP 2 was then-started and data collected - in accordance with the OPT.. The pump was stopped and the A0 was instructed to realign the AFW System to. increase steam generatorL levels. The A0 cpened cross-connect valves 1AF-090 and 1AF-091-and attempted to c. lose valve 1AF-055. However, IAF-055 was - inadvertently left one-quarter. turn open (the A0 thought it was fully closed, but the valve was mechanically bound). This alignment re-initiated backflow through the system when MDAFWP 2 discharge valve 1AF-054'was re-opened. - When the A0 notified the R0 that the lineup was restored, the R0. started MDAFWP 2 to supply; flow to all steam generators.. The R0 watched water levels in. the steam generators for approximately twenty minutes and determined that levels were not responding correctly for a pump discharge flow. rate of 300 gpm. Suspecting a. problem with MDAFWP 2, the R0 stopped MDAFWP 2 and directed the A0 to prepare MDAFWP 1 for starting. MDAFWP 1 was started and the identical response was observed for pump flow rate and steam generator levels. The R0 stopped MDAFWP 1 and told.the A0 to close the AFW cross connect valves, IAF-090 and 1AF-091. The R0 then started MDAFWP 1 and 2 to supply the steam generators. and observed that levels were not increasing as he expected. Suspecting a backflow path existed, the Unit Supervisor went to the MDAFWP 2 room and helped the A0 fully close valve IAF-055. The R0 was then able to restore normal flow to the steam generators and observed the correct level response. Backflow for this portion of the event occurred for approximately sixty-six minutes. w, L .j i {

c.:'s . T Attachme'nt to TXX-89596 ?

August 18,:1989:

l; Page;14 of 72-L b

IVE LTU ELECTRIC INVESTIGATION OF ROOT CAUSES A.

' Introduction In. order to ensure a complete investigation of the. April 23 event, the Executive Vice President for. Nuclear Engineering and Operations established a Task Team'on May 1,-1989, to investigate root.causes for and identify corrective. actions. -The: Task-Team consisted of personnel from various organizations-including L0perations,iTesting, Results and Design-Engineering, Projects,.

Quality Assurance and Licensing.. Subsequently, the Task Team was also assigned responsibility for investigation of-the May 5 event.

' Establishment of the Task Team played -aikey role in. identification. of root causes.. The team provided a multi-disciplined approach to 'C , resolve potential. issues including operational; errors, equipment damage,u regulatory interface, quality control 'and scheduling. p

It is important to~ point out that,' from the beginning, TU Electric senior management emphasized that the Task Team should concentrate.

on a thorough and deliberate determination of. root causes. Timeliness', though.important, was considered secondary to comprehensive identification of root causes. Notwithstanding the above, TU senior. management's evaluation of' the Task Team's effort identified areas which could have been improved. 'Many of the personne1 rare assigned to the Task. Team-l only'part time. Consequently, the Task Team took'approximately six weeks to complete its: investigation. Initially Task Team responsibilities for interfacing with the NRC were unclear and communications with NRC were slow to develop. In addition, the-Task Team did not fully understand the information needs or expectations of the AIT. _ TU Electric recognizes that improvements can be made pertaining to event investigation by future Task n _ Teams.. These improvements are discussed in Section.VII. B. Investigation Focus In general, the Task Team investigation focused on two issues. First,-the Task Team directed tests, inspections and evaluations to determine the root cause of backflow through the AFW System. Second, the Task Team interviewed Operations personne1Jand reviewed available information to determine the extent to which operator errors on April 23 and May 5 contributed to the backflow events. Each of these.is discussed separately below. C. Investigation of the Root Causes of the Check Valve Failures Immediately after the Task Team was formed, a-testing program was developed and carried out to determine which check valves were hung open during the April 23 event. Additionally, the testing T c

~ ! Attachment toJTXX-895' 6~- 9 August. 18,.1989.. }Page.15.of 72 program,-in conjunction with a review of operat'or logs,

established the backflow paths' for the April 23 and May 5 events.

Identification of these flow paths;provided a basis for 1 identification and subsequent engineering evaluation of potentially overstressed piping, supports, containment' . penetrations and. instrumentation.- Appendix 2 discusses-the 'results:of this testing program.' As this: Appendix. indicates, numerous check valves.were determined to be hung open,; indicating -a generic. problem. -The Task-Team conducted reviews to determine the'cause of the backflow through the AFW System. The results of the reviews were-as follows:- o. The Task Team' reviewed the maintenance and modification ) .historiesL of. the check valves to determine if any shortcomings' could have resulted in the check valve failures. The'results of'this review are presented in Appendix 3. :As this' Appendix discusses, prior disassembly' and reassembly of' various BW/IP check valves in 1983= produced an elevation difference between the valve" seat' and-disc due to incorrect reassembly instructions. The instruction stated that the valve retainer, which' locates the disc assembly, was to be bottomed out. This technique created the aforementioned -elevation difference..As discussed below, the inadequacy in the' reassembly instruction only pertains to pressure seal check valves, of which there are fifty-seven in Unit I and 2. One-hundred-three bolted bonnet. valves were unaffected - because their design is such that a fixed vertical relationship exists between the seat / disc' assembly and seat ring. o The Task Team used radiography to determine ~ disc position 1 prior to disassembly and the Computer Assisted Drawing (CAD) program to determine the actual measurements of critical valve internal components. The results of the radiographs, the CAD. program, and inspections of BW/IP. valves are discussed in Appendix 5. Based upon these results',.the Task. Team determined that the. valve discs for pressure seal check valves had become hung open due to the discs becoming lodged. beneath the seat lip. In addition, the Task Team learned that available vendor information did not specify maximum disc axial play. ' Excessive axial play coincident with' L seat / disc elevation differences is viewed as a contributory L factor to valve failure. o The Task Team reviewed available industry experience with check valves in other nuclear plants to determine whether these plants may have identified a problem with BW/IP check valves that could have caused the backleakage on April 23 and I 'r'

Attachment to TXX-89596 August 18, 1989 Page 16 of 72 May 5. As discussed in Section VI.A, the Task Team did not identify problems related to the backflow events at CPSES from available industry information. Related information obtained during the course of the investigation of the events had not previously been identified to industry o The Task Team reviewed previous problems with check valves at CPSES to determine whether these problems and any common causes were present. As discussed in Section VI.B previous failures of check valves at CPSES had occurred, which may have failed in a manner similar to those of April 23 and May 5, indicating a need for improvement in the documentation, reporting, and evaluation of plant events and equipment failures. o Coincident with the Task Team investigation, two potentially significant material conditions in BW/IP check valves were identified during Station Service Water System testing. These conditions are unrelated to check valve backleakage, but are discussed in Appendix 4. o The Task Team evaluated the design of the AFW check valves using guidance issued by the Electric Power Research Institute (EPRI). As discussed in Appendix 6, this evaluation did not identify any factor that would relate to the sause of the hang up of the AFW check valves on April 23 and May 5. D. Investigation of the Root Causes of the Operator Errors Operations personnel under direction of the Manager, Operations conducted interviews with shift crews to determine the root cause of operator errors made during the April 23 event. As discussed in Section III, the event occurred following the simultaneous operation of valves IAF-041 and 1AF-042. Investigation determined that valves 1AF-041 and 1AF-042 were operated simultaneously due to a misunderstanding of the administrative controls governing the sequencing of procedure steps. In addition, the valve operator arrangements are unique. The R0 referenced the approved procedure and correct section for l instructions on realigning the TDAFWP to the test header. This section of the procedure indicates that closure of IAF-041 is Task 1 and opening 1AF-042 is Task 2. These tasks are numbered in sequence which requires they be performed in sequence. The ) requirement that Tasks 1 and 2 be performed in sequence was not 1 l fully understood or followed by the Auxiliary Operators. In part, ) l this lack of understanding is attributable to the absence of i i guidance in applicable Operations Department Administrative Procedures (ODAs). The ODA that provides guidelines for the 1 l 1 i!-..__

4 Attachment to TXX-89596 August 18, 1989-Page 17 of 72 preparation and review of operations procedures states that mandatory sequence of steps is assumed unless ;he steps are identified by bullets or unless the procedure states otherwise; however, another ODA that describes guidelines on use of procedures does not identify this rule. The Control Room and Auxiliary Operators are not responsible for the preparation of Operating Procedures and therefore are not as familiar with the -requirements of the former procedure as they are of the requirements of the latter procedure. The valve operator for 1AF-041 is mounted on the TDAFWP room floor and is connected to the valve stem by a series of reach rods and gear boxes. The arrangement of the valve operator and gears causes IAF-041 to be a reverse operating valve, clockwise to open and counter-clockwise to close. Due to the uniqueness of the valve operator, the handwheel for 1AF-041'is normally labeled to identify the closed direction. Upon arriving at the TDAFWP room, the A0 discovered that the direction label was missing and he was unsure of the-proper rotation' for closing the valve. To determine the proper rotation, the A0 took the valve operator for 1AF-042, which was labeled in the open direction, approximately 1/4 turn and observed the movement of its gear box. This allowed the A0 to determine the proper direction of handwheel movement for IAF-041. The A0 knew that 1AF-041 required approximately 40 minutes to operate due to the number of turns to full stroke and requested assistance. The A0 believed that 1AF-042 would also. require approximately 40 minutes to full stroke and did not think that the 1/4 turn on the handwheel would have unseated the valve. Three additional A0s were dispatched to the TDAFWP room to assist. The dispatched A0s were not adequately briefed on the evolution in progress and upon arrival, one relieved the operator closing 1AF-041 and another began to fully open IAF-042. This resulted in both valves being open at the same time. The Task Team also conducted interviews to determine the cause of the personnel errors related to the May 5 event. As in the April 23 event, the investigation revealed that the system was aligned by A0s using the approved procedure and correct section. However, Task 1 and Task 2 were not performed in the sequence specified in the procedure because the A0s did not fully understand the requirement to perform these tasks in sequence. The Task Team interviewed operators to determine why 1AF-055 was not fully closed on May 5. The Team determined that valve IAF-055 was a remote manually operated valve, that the A0s believed the valve to be closed because of the resistance felt in trying to close the valve, and that the A0s were unable to visually determine the degree of valve closure because of the location of the valve with respect to the operator. This valve binding caused the overall duration of the May 5 event to be much greater than the April 23 event. u-_-w__n.__

Attachment t3 TXX-89596 August 18, 1989 Page 18 of 72 The Task Team determined that the operators quickly identified the cause of the April 23 event to be parallel operation of valves. Considering that the operators on shift during the May 5 event were unaware of the April 23 event and were also dealing with an unknown problem, valve binding, the actions were considered timely and investigated logically. In summary, based upon its interviews of Operations personnel and review of documents, the Task Team determined that operators aligned the AFW valves in the wrong sequence on both April 23 and May 5. This was due to the failure to follow procedures, caused by a lack of understanding of the administrative requirements to perform procedure tasks in the sequence specified in the procedure. Contributing to this problem was confusion over which way the valve was to be turned. Additionally, the failure'of the operators to completely close valve IAF-055 on May 5 was due to its mechanical binding and the inability to readily verify closure. Operator actions and investigations for both events were considered timely and logical. In its report (pp. 31,35), the AIT states that the A0s did not comply with procedures governing manipulation of the valves because they believed that they could rely upon the check valves to prevent backflow through the AFW System, and because they were under time constraints to complete the valve alignment prior to the end of the shift. The Task Team determined that the A0s involved in the April 23 and May 5 events were not specifically provided with any directions to complete the valve alignment by the end of shift, but they may have taken it upon themselves to do so. The Task Team could not confirm that the operators relied on the check valves to seat when operating valves in parallel. The Task Team concluded that the primary cause of procedural noncompliance was a lack of understanding of the requirement to perform procedure steps in the sequence specified in the procedure. E. Summary Based on the above investigation and actions, the Task Team identified the following root causes of the April 23 and May 5 events: o Check valve failure occurred because of incorrect instructions for reassembly. The incorrect instructions, derived from vendor information, are applicable to pressure seal type valves only. When followed, these instructions created the potential for an unacceptable elevation difference between the valve seat and disc which caused the valve disc to become Todged beneath the seat lip. In addition, disc axial play had not been previously specified

Attachme;t-to TXX-89596 August 18, 1989 Page 19 of 72-by the vendor. Valve inspections done by.the Task Team and discussed with BW/IP indicated that disc axial play was, in some cases, excessive. This aggravated the problem created by the elevation difference. Fifty-seven pressure seal check' valves in Unit 1 and 2 are potentially affected. One-hundred-three bolted bonnet valves are unaffected because the ' valve' design prevents an elevation difference.during reassembly. However,. all bolted bonnet valves are being examined to assure the amount of axial play is within the design requirements. Corrective actions for the affected valves are' described further in Section VII.B.I. o Backflow on April 23 and May 5 was initiated because of hung open check valves coincident with.the failure of Auxiliary Operatort, to operate manual valves in the sequence specified by the procedures. A remote isolation valve, which operators thought was shut, also contributed to the May 5 backflow initiation and caused a delay in stopping the event. 'I ___________._e__--

1.; Attachment to TXX-89596 l August 18,~1989 . Page 20 of 72 V. SIGNIFICANCE OF THE APRIL 23 AND MAY 5 EVENTS This section evaluates the safety significance of the April 23 and May 5 events. The evaluation is divided into three parts: 1) evaluation of the actual significance of the events; 2) evaluation of the impact of the events on the integrity of the piping system and pipe supports, containment penetrations, and instrumentation; and 3) evaluation of the l. significance of the events if they had occurred during operation of the plant. Each of these is discussed separately below. ' ~ A. Actual Stanificance The April 23 and May 5 events did not and could not have resulted in any radioactive release because they occurred during preoperational testing prior to fuc1 load. Therefore, these events posed no threat to public health and safety. Furthermore, although water was diverted from the steam generators via the.AFW System to the CST, the decrease in the steam generator level was detected and AFW flow was restored prior to excessive loss of steam generator level. While these events did have a potentially significant impact on certain plant components, such as piping, supports, instrumentation and containment penetrations, these potential problems were identified, the hardware was thoroughly evaluated, and necessary corrective actions have been or will be taken by fuel load. B. Pioina and Support Intearity The Task Team performed a preliminary thermal blowdown analysis on the piping and pipe supports affected by the backflow from the steam generators for both events. This analysis iceluded a correlation of the level changes in the steam generators to backflow rates and mass / energy balances at piping junctures to determine piping temperatures. These temperatures were compared to the amount of coating discoloration observed on the p! ping. The coatings manufacturer (Carboline) performed a type test in which similarly painted metal coepons were heated at several different temperatures in an oven. The resulting discoloration of these coupons was then compared to the discoloration of the actual piping which provided support for the temperatures calculated in the mass / energy balances. After the preliminary temperature distributions for the affected piping were determined, a piping and pipe support stress analysis was completed. This analysis identified several areas where piping Code allowable stress was exceeded. Subsequent ultrasonic testing of the pipe verified that no plastic deformation had occurred. Thus, the stresses resulting from the elevated i

t

..ev Attachment'to TXX-8tS95 1

V 7 ~ August 18,'1989-Page 21 of.72 +, li temperature were within the elastic range for the piping material and no piping needs to be' replaced. Sixty-four supports were preliminarily identified as ' eing loaded b beyond their current design load. A more detailed analysis, accounting for actual. installation. tolerances and actual material allowables based on certified material test reports (CMTRs), determined that only ten supports were overloaded. - Nonconformance ' Reports were written for QC examination of the significant. attributes of these supports and no deviations or deficiencies : were found. The results of the completed blowdown thermal. analysis agree with I those of the preliminary analysis with the exception of a small segment of pipe (approximately twenty feet) on the MDAFWP discharge whose temperature was increased by 300F over that of the preliminary analysis. This will be factored into a final stress and support analysis. Should the final stress analysis indicate pipe stress over that allowed or any increased support loading beyond.that which was previously evaluated, the affected piping and supports will.be re-evaluated and reworked as necessary. C. Imoact, on Containment Penetrations - The' Task Team also evaluated whether the feedwater containment penetrations could have been degraded as a result of the backflow events given that exposure to the. temperatures. associated with hot o water for a sufficient period of time could result in concrete damage. Physical inspection revealed no damage to the penetrations. Additionally, analysis and UT inspection of the piping concluded that the penetrations were not affected by loads created.by piping that was in the backflow path. D. Impact on Instrumentation l Flow Elements (FE), Flow Transmitters (FT) and Temperature Elements (TE) could have experienced high temperatures as a result of the' backflow. A review of the instruments' design against a maximum possible temperature of 5570F was performed. The FEs / are metallic plates and hence unaffected. The TEs' qualification temperature is 20000F and therefore acceptable. The FTs, per R l discussion with the manufacturer, may lose calibration and may be damaged as a result of high temperatures. The FTs will be re-calibrated if possible and replaced if necessary prior to fuel load. 4 -) __ ____A

qa <. w. Attachment to TXX-89596 August 18c 1989 l l: Page 22 of 72 E. Potential' Significance of the Events if they had Occurred Durina Operation The TU Electric-Task Teaa performed an evaluation of the potential 4 effects of malfunctions of BW/IP check valves during. plant operations.. The check' valve disc hang up condition occurred in the AFW check valves and Main Steam (MS). check valves (steam supply lines). In the absence of a line break or manual valve misoperation, the failure of thz valves would lack significance. because of the absence of a backflow path. In the event of a loss of AFW. flow to the steam generators for any. reason the Emergency Response Guidelines would require operator actions to commence Reactor Coolant System (RCS) cooldown using systems other than the AFW and Feedwater Systems. l

Attachment to TXX-89596 August 18, 1989-Page 23 of 72 VI. PRECURSORS The Task Team investigated several CPSES check valve failures from 1983 to just prior to the April 23 event. In addition, previously received industry information was reviewed to determine if CPSES had properly reacted to that information. Finally, the Task Team contacted a number of sites who were thought to have purchased BW/IP check valves. l The Task Team conclusions are described below. A. Nuclear Industry Exoerience with BW/IP Check Valves The Tast Team investigated the nuclear industry's experience with BW/IP check valves to determine whether there was any indication that BW/IP check valves were prone to failure due to excessive valve disc elevation or axial disc play. Although various concerns about the performance of check valves were identified., there was no indication from industry that the BW/IP check valves were likely to malfunction from these causes. There have been a number of NRC Notices and Bulletins that raised concerns about the malfunction of check valves through various failure modes. There were no cases of check valve failure identified from NRC correspondence similar to that of the failure modes experienced at CPSES. Additionally, the Task Team determined that NRC Bulletins on check valves had been reviewed by CPSES and corrective actions taken as applicable. Similarly, the Task Team determined that NRC Notices on check valves had been reviewed for applicability and appropriate actions taken. The Task Team performed a search of the INPO Nuclear Plant Reliability Data System (NPDRS) to determine if BW/IP check valves had failed at other plants. A total of thirty-eight BW/IP check valve failures were identified; twenty-three of these failures were related to disc seating. Of these twenty-three, approximately seventy-five percent were caused by foreign material caught between the disc and seat, disc distortion, improper installation of the disc-stud-hinge arm assembly, or corrosion of materials. None of these BW/IP check valve malfunctions were identified as occurring through the failure modes experienced at CPSES. The Task Team contacted four plants to discuss problems with BW/IP valves. Three of the four plants had experienced backleakage and all expressed concerns with the general quality of their BW/IP valves. The three affected plants provided the following information regarding backleakage through their BW/IP check valves:

h-L Attachment to TXX-89596 L . August 18, 1989 L Page.24 of 72 l l l-1. A unique procedure had been supplied to St. Lucie for check l valve assembly. The procedure applies to 12 inch pressure ? seal bonnet model 73060 check valves and is used for. clevis, bonnet arm and disc assembly replacement. This procedure was designed to make up for variations 'in tolerances applied during body / neck fabrication. These variations in tolerance resulted in an unacceptable difference in elevation between the centerline of the disc and the centerline of the seat. The need for this procedure was recognized by BW/IP bofore L the valves were shipped to St. Lucie. 2. Diablo Canyon experienced seat leakage problems with BW/IP-pressure seal bonnet check valves. Diablo Canyon attributed its problems to uncertainty involved in aligning the disc parallel to the seat during assembly, although no non-intrusive techniques (radiography,. ultra-sonics, fiber-optics) were used to verify that rotational misalignment was the sole cause of their seat leakage problems. These uncertainties existed because there are no dowel pins or l' other type of positive positioning mechanisms designed into l the valve to ensure disc / seat parallelism. This problem is unrelated to the check valve failures that occurred at CPSES on April 23 and May 5. 3. McGuire also experienced problems with BW/IP check valves. These problems include significant bonnet leakage and three instances of greater than design leakage past the seat. The valves experiencing backleakage were replaced before the exact cause of the malfunction was determined. McGuire assumed that, because of the magnitude of the backflow, the disc was stuck in the neck of the check valve. TU Electric has not experienced a similar check valve failure. McGuire also modeled a BW/IP valve in a test loop and determined after experimentation that the bonnet should be raised to ensure proper seating. The assembly procedures at the McGuire plant have been revised accordingly. The Task Team's review of available industry experience with BW/IP check valves did not identify any problems that were related to the CPSES failure due to excessive valve disc elevation or axial disc play. Check valve leakage has been observed; however, this leakage was generally attributed to causes unrelated to valve reassembly, such as foreign material between the seat and disc or disc distortion. Based on discussions with McGuire it was determined that a similar failure mechanism had been identified; however, this information was not disseminated to the industry. B. Previous Occurrences at CPSES The Task Team conclusions pertaining to previous CPSES check valve failures are discussed below.

..s Attachment to TXX-89596, August 18, 1989-Page 25 of 72 - Prior to April 1989 'In 1983, check valve parts were found in the Component Cooling Water (CCW) heat exchanger. A valve disc had become detached because a weld which held the disc retaining nut had cracked allowing the retainer nut to back off. Further investigation L found that the' failed weld was a tack weld instead of the i specified fillet weld. j A modification recommended by BW/IP was made to replace. tack welds with fillet welds holding the disc nut to disc stem. In addition, during this same time frame, various check valves within AFW and other systems had been disassembled for flushing and draining operations and then reassembled. The modification and reassembly after flushing and draining are . germane to the backflow events only because they most likely caused the check valves to become hung open. At the time, the incorrect instructions for reassembly were apparently followed, which created the elevation difference between the seat and disc. In May 1985, a CPSES Problem Report documented that damaged snubbers along with a cracked disc seat and bent stud on IMS-042 (steam inlet check valve to the TDAFWP) had been found in the AFW turbine steam supply line. Revision 0 of the failure analysis stated that the cause was the bonnet being too low in the body. The report was later revised based upon input provided by BW/IP to-state that the damage was caused by unusual flow conditions in the piping system (water hammer) coupled with the bonnet being installed crooked. Through discussion with BW/IP, TU Electric personnel agreed that the failure was not due to the bonnet being too low in the valve. The corrective action included modification to the valve to accept the stated flow conditions by lengthening the disc stop. In addition, the piping and supports were modified to minimize the consequences of water hammer upon turbine pump start. In retrospect, the problem with valve IMS-042 may have been attributable in part to the incorrect assembly instruction. A more thorough discussion with BW/IP and a more in-depth investigation by TU Electric in 1985 might have confirmed that the reassembly procedure was incorrect. Both TU Electric and the AIT noted that post assembly backleakage testing had not been specified or performed for any of the i aforementioned valve disassembly operations. In addition, the AIT j noted that no post maintenance test or surveillance requirements J were specified from 1985 to the recent hot functionals. It is TU Electric's position that applicable provisions in Section XI of the ASME Code do not require that check valves be tested other than in the forward direction. However, it should be noted that in 1988, TU Electric revised its post-modification test

Attachment to TXX 89596 ' August 18, 1989 Page 26 of 72 procedures to require post work testing for backleakage. Therefore, TU Electric had taken action to procedurally address this issue prior to the backflow events. Surveillance testing was not performed on the ATW System after 1985 because operability requirements in accordance with Technical Specifications were not applicable. The AFW System was, however, included in the plant layup program and was maintained in wet layup with hydrazine-treated water for most of that period. Thers was no evidence of corrosion contributing to the failures experienced in the AFW System. Backleakaae on April 5 On April 5,1989, while filling steam generators following draining to attain in-specification chemistry, a report of water flowing into the TDAFWP Room was received by the Control Room. Investigation revealed that the source of the water. was backleakage through the TDAFWP piping. A flowpath was found from Steam Generator 4 AFW supply line to the TDAFWP room through-a clearance-tagged open vent valve. This flowpath indicated that check valve IAF-106 in TDAFWP supply line to Steam Generator 4 was not seating properly. At the time, an instruction was being written to forward flush the T0AFWP supply lines to the steam generators with Reactor Makeup Water, and it was decided to add a section to this instruction in order. to determine if the check valves in the remaining TDAFWP supply lines were seating. The flush identified that two of the remaining check valves were not properly seating. Work Requests were written and a post Hot Functional (HFT) due date of May 26, l 1989 was assigned. Testing, radiography and CAD techniques performed after the April 23 and May 5 events determined that the failure of these valves to seat was due to an elevation difference between the valve disc and seat. The Work Requests for these valves did not quantify the amount of leakage. Therefore, the organizations which review procedures and Work Requests were not alerted, nor did they pursue the severity of the problem. As a result, the backleakage was not documented on a higher-visibility document such as a PIR or Nonconformance Report (NCR). 1 l-Corrective action for this above concern is described in Section VII.C. I April 19. 1989 Event On April 19, 1989 a miniflow check valve, IAF-069, for MDAFW Pump 2 was identified as deficient after the pump's suction relief was l l l t-

6 Attachment to TXX-89596 August 18, 1989 Page 27 of 72 noted to be lifting. Two Nonconformance Reports were written to document the condition of IAF-069 and the valve was disassembled and inspected. As a result of this inspection, a deformation of the disc stem and face of the stop was identified. This damage was caused by tapping of the valve disc against the stop as a result of turbulence produced by an upstream orifice. To correct this problem, the disc stop was built up an additional.125" to help keep the disc more in the flow stream when the valve is open, thereby reducing tapping. In addition to the deformation, approximately.175" of axial play was noted in the valve disc. At that time, acceptance criteria for axial play were not available at CPSES. However, after discussions with BW/IP, the disc stud bushing was trimmed which resulted in reduced axial play. Following the April 23 and May 5 events, testing revealed that the rework was successful. The Task Team concluded that although the April 19 deficiencies, tapping and axial play, were not the primary cause of the April 23 failure, the relationship between the two failures was not identified. Design engineering personnel were not aware of the April 5 check valve failures. They therefore believed that the 1AF-069 failure was isolated.

r. consequently, engineering concentrated on the readily identf fiable deficiencies in 1AF-069 of tapping and axial play.

Extensivc investigatory techniques such as radiography and CAD were not thought to be needed, and were therefore not developed or used. The Task Team could not positively conclude that the discs were hung open in the check valves involved in the April 5 and April 19 failures. The valves involved in the April 5 failures were not opened or inspected prior to the April 23 and May 5 events. The valve involved in the April 19 failure was not radiographer prior to disassembly. Therefore, it was not possible to determine the pre-existing disc / seat relationship. However other indications such as the observed amounts of water on April 5 and the lifting of a pump suction relief on April 19, indicate that the discs were most likely hung ? en. April 23. 1989 Event The April 23 event is described in detail in Section III.A, above. PIR-110 was written to investigate this event. The Manager, Operations recognized that simultaneous opening of 1AF-041 and 1AF-042 was in. correct and that it had initiated the event. However, he felt that the error was isolated to the shift in question. Therefore, procedure noncompliance was not pursued with other shifts. 1 l l

1. e Attachment-to TXX-89596

' August' 18, 1989 Page. 28 ~of 72-The AFW operability test was performed on May 5 with Operations-department management's knowledge of the April 23 event. The potential for creating another backflow condition had 'been recognized by Operations management; however, it was~ concluded that the test procedures'placed sufficient controls on the L operation of manual valves so that backflow would not occur. Several similar tests had been conducted between April 23 and May 5 without difficulty. Operations management believed that it was appropriate to proceed with the test on May 5 because even if a generic problem existed in the BW/IP check valves, isolation valves in the AFW System would provide adequate protection against backflow and as stated before, at that time, Operations management felt that the operator error on April 23 was an isolated event. In retrospect, the operator errors could have been more fully investigated and investigation results provided to all Operations personnel. C Conclusions Documentation of available industry experience with check valves did not identify any concern with the elevation of the valve disc - related to the valve seat for BW/IP check valves. However, several precursor events at CPSES _such as the 1985 failure of check valve IMS-142 and'the April 5 and April 19 failures did involve BW/IP check valves that may have had the same failure mechanism as the check valve failures which occurred on April 23 and May 5. In addition, Operations management could have more thoroughly investigated the April 23 operator error prior to _ permitting the performance of additional testing on the system. Based upon these precursors, TU Electric is making improvements in the thoroughness of its evaluations of the causes of equipment failures, the documentation and reporting of equipment failures, and the evaluation of the effect of equipment failures on the. operability of plant systems. These improvements are deceribed in Section VII.C. i l i 1

7_ y q Attachment;to TXX-89596 ~ August 18, 1989L Page 29 of 72 VII. CORRECTIVE ACTIONS, PREVENTIVE ACTIONS, LESSONS LEARNED, AND ASSOCIATED IMPROVEMENTS -l 'l A. Introduction The events on April 23 and May 5 involved deficiencies in the' . BW/IP. check valves. Additionally, the events themselves adversely affected certain hardware at CPSES. Subsection B below identifies the corrective actions that TU Electric is taking for this hardware. In addition, Subsection B describes the actions being taken by TU Electric to address these root causes and. prevent' recurrence of events similar to those on April 23 and May 5. Lessons learned, together with associated improvements made by TU Electric, are discussed in Subsection C, below. This Subsection also addresses the weaknesses discussed by the AIT in its July 10 ~ letter and by the NRC at the meeting on July 17. Appendix 1 lists the conclusions and recommendations identified by the NRC and states how TU Electric has addressed each one. All actions described in the following Subsections will be completed by fuel load of the respective units unless otherwise noted. B. Corrective and Preventive Actions 1. Corrective Actions for Hardware TU Electric is taking the following corrective actions far the hardware at CPSES: Discs for the BW/IP check valves - As discussed in Section IV above, backflow occurred through the BW/IP pressure seal check valves on April 23 and May 5 when the valve disc hung open. -This failure was caused by'an elevation difference between the disc and seat during valve reassembly, resulting in the disc lodging under the seat lip. A potential contributing factor was an unspecified and, in some cases, excessive axial play in the valve disc. To address these problems, TU Electric is taking the following actions: a. Unit 1 BW/IP pressure seal check valves are being inspected and/or reworked by eliminating the elevation difference between the valve disc and seat. The rework is being accomplished by taking critical dimensions and using these dimensions to establish the amount of the retainer ring backout for the valve bonnet. The amount of permissible backout is being specified by engineering. I

Attachment to TXX-89596 August 18, 1989 Page 30 of 72 b. Pressure seal and bolted bonnet BW/IP check valves will be inspected to determine disc axial play. Based on the results of these inspections, the valves will be reworked as necessary in accordance with vendor specified tolerances. c. Rework performed in accordance with either action a. or

b. above, is being inspected by QC personnel and post-modification operability tests are being performed, including verification that the valve fully closes.

d. Inservice testing requirements are being established to ensure closure of BW/IP and other check valves for which l backflow is a safety function. This review will identify which valves will be tested prior to fuel load and will also note exceptions and their milestones. Damaae to BW/IP Check Valves - The backleakage through BW/IP check valve IAF-069 on April 19, 1989, was caused by damage to the valve disc and body due to turbulence produced by an upstream orifice. To address this problem, TU Electric is taking the following actions: a. A full inspection of IAF-069 was performed. Evidence of tapping on the valve stop was noticed and a deformation of the disc stem and face of the stop was present. Additionally, approximately.175" of axial play was noticed (this amount of axial play would be acceptable under the subsequently-developed BW/IP installation tolerances). It appeared that the axial play resulted from fabrication and was not the result of operation. In order to properly seat the valve, the disc assembly was taken apart and the axial play was reduced from approximately.175" to.060" .075". Also, the disc stop was built up an additional.125" to repair the deformed area as well as to help keep the disc more in the flow stream when the valve is open. The valve has received post-repair tests and is now operating properly. b. Kalsi, Inc. is performing an evaluation of CPSES check valves in response to INP0 Significant Operational Event Report (SOER) 86-03. Following receipt of this evaluation, TU Electric will determine whether to increase the distance between orifices and check valves. If it is determined that check valve failure is not imminent, implementation of the design changes may be deferred until after fuel load. Additionally, periodic post fuel load internal inspection of check valves will be performed to monitor and trend wear in the check

j: Attachment to TXX-89596 August 18, 1989 Page.31 of 72-valves. If these inspections reveal excessive wear TU Electric will initiate design changes to increase the distance between the orifices and check valves. Damaae to Pioe Components - The events on April 23 and May 5-' caused visible damage to a piping support, caused paint to blister or discolor on certain AFW piping,.and resulted in stresses in AFW piping that exceeded Code allowable limits. To address these problems, TU Electric is taking the following actions: a. YU Electric.is repainting the affected piping, b. TU Electric has replayed the damaged ' pipe support. c. TU Electric has inspected piping and supports based on preliminary blowdown thermal analysis. Upon completion of the final stress and support analysis (using the. finalized blowdown thermal analysis temperatures), they will be reevaluated and reworked as necessary. These and any other follow-up actions will be discussed further as part of SDAR CP-89-15. 2. Preventive Actions TU Electric is taking the following actions to address the root causes of the events on April 23 and_May 5 and to prevent recurrence of similar events: Installation Procedures for the BW/IP Check Valves - As discussed in Section IV above, backleakage occurred through the BW/IP check valves on April 23 and May 5 because of an elevation difference between the valve disc and seat due to inadequate reassembly instructions. Additionally, the lack of criteria governing axial play in the valve disc may have contributed to the backleakage..To address this problem, TU Electric is taking the following actions: a. The onsite valve assembly procedure has been revised to include a requirement for determiiting the elevation q adjustment necessary to avoid interference between the j disc and the seat. b. The accepteble range for the axial play dimension has been determined by BW/IP and will be included in its instruction and site procedures. c. A 4" BW/IP check valve was bench tested following l adjustment in accordance with the revised procedure discussed above to verify the adequacy of the J procedures. The test showed the procedure was adequate. i l

Attachment to TXX-89596 August 18, 1989 Page 32 of 72 d. The Quality Assurance department has taken additional action to assure that components and material procured from BW/IP will meet quality requirements. BW/IP has been placed on "Special Status" on the Approved Vendors List. This "Special Status" requires Engineering to develop critical characteristics on safety-related parts and components purchased from this vendor. These critical characteristics will be checked in the shop and during receipt inspection activities. This "Special " Status will be maintained until sufficient confidence has been reestablished in the quality of material supplied by BW/IP. Adherence to Procedures - As discussed in Section IV above, the failure to follow procedures on April 23 and May 5 was caused by a lack of understanding of the need to perform the steps in the sequence written in the procedure. To address this problem, TU Electric is taking the following actions: a. The Manager, Operations met with the personnel involved in these events and counseled them on procedure usage and procedure compliance. b. The administrative procedure which provides guidelines on the use of procedures has been revised to emphasize the requirement that procedure steps are to be performed in the sequence specified in the procedure, except as otherwise stated in the procedure, allowed by emergency operations procedure rules of usage, or permitted by the Shift Supervisor with appropriate documentation of the deviation. c. Administrative procedures have been revised to add applicable procedures for the AFW System to the list of procedures required to be available and referenced when performing field work. d. The Shift Operations Manager has developed and is implementing an action plan to enhance procedural compliance. As part of this plan, a memorandum on procedure compliance was provided to the Shift Supervisors, who in turn discussed the memorandum with their respective crews. The Manager, Operations and/or Shift Operations Manager also met and discussed the memorandum with each crew. Additionally, a workshop was held by the Manager, Operations with Operations Department Senior Reactor Operators (Shift Supervisors, Unit Supervisors, Shift Technical Advisors and Staff), including Training and Plant Evaluation personnel, to discuss the April 23 and May 5 events and procedure compliance. -_J

' Attachment to TXX-89596 l

August 18, 1989' Page 33 of.72 e.

Quality Assurance is performing an overview of-implementation of selected. procedures that control operation of selected systems necessary for safe shutdown. This overview will be performed by personnel wha possess technical experience in operation, maintenance and testing. These overviews will continue until they are deemed unnecessary by the Director of Quality Assurance. f. -In November 1988, TU Electric instituted a program for performance-based audits and surveillance. This. approach to audits and. surveillance emphasizes direct observation of plant ac'.ivities in-progress by personnel who are qualified in the activities being observed.- It also stresses the technical adequacy of the procedures being.used, as~ well as the performance of the personnel who are implementing them. The Quality Assurance department will re-evaluate this program based on identified lessons learned from the April 23'and May 5 events. Any necessary program enhancements will be made after completion of this evaluation. In addition, compliance-based verifications will continue'to be performed to assure personnel are adequately implementing program requirements (i.e., procedures and instructions) which govern their activities until they are deemed unnecessary by the Director of Quality Assurance. Manipulation of Remote-Operated Valves - As discussed in Section IV above, the event on May 5 was caused, in part, by a mechanically bound isolation valve. To address this concern, TU Electric is taking the following actions: .a. Reach rod operators for other safety-related valves will be evaluated for proper operability and human factors considerations. This evaluation will include consideration of factors such as whether a valve is operated in a direction that is opposite to the usual direction for valve operation, whether the valve is operable, the ease of operation of the valve, and the gear ratios and time required to operate the valve. b. Safety-related reverse-operated valves documented in the above evaluation will be marked to indicate the direction of operation. c. The cause of valve IAF-055 binding will be determined and corrected.

I j* ,4 ' August 18,!1989-Page 34-of. 72: 'Other Actions to Preclude Recurrence - In' addition to the actions discussed above,uTU Electric is taking the following actions to help prevent recurrence of events similar to those. which occurred on April 23 and May 5: P The events-on April 23 and May 5 were documented on PIR-a.. .110 and-PIR-129, respectively, for purposes of obtaining ~ corrective action. A discussion of these PIRs and-i- associated issues: such as; industry experience with check valves is being added to the licensed and non-licensed-operator requalification and replacement training' programs. Operations personnel will receive

training in this part of the.requalification program prior to fuel load, b.

Technical Specification surveillance test procedures for the AFW pumps are being revised. The revision will' require the discharge; pipe downstream of the test loop to be checked for. elevated temperatures'that would: indicate backleakage through check valves 30 minutes after the test.. 1 c. When requesting personnel.to provide assistance in performing a plant evolution, reactor operators and I auxiliary. operators have been directed to brief the f personnel on the evolution and applicable procedures. prior to performing them. d.- The' Shift Operations Manager.has counseled the Senior Reactor Operators (SR0s) on the importance of maintaining proper system status and the risks involved l in leaving a valve lineup in an indeterminate condition. C. Lessons learned and Associated Improvements In addition to the corrective actions and preventive actions discussed in the preceding section, TU Electric. has evaluated the events on April'23 and May 5, the' precursors to these events and the response to these events to determine lessons. learned and identify corresponding improvements. In performing this evaluation, TU Electric has: considered the findings and recommendations in the NRC's AIT Report'and the weaknesses l identified by the NRC'in the meeting on July'17,1989. ? TU Electric has identified a number of areas where improvements could be made. In some cases, the areas overlap, and some improvements are common to more than one area. The areas,- together with TU Electric's corresponding improvements, are discussed below. L

.e ' Attachment to TXX-89596 August 18, 1989 Page 35 of 72 1. Manaaement and Supervision of Operations - Several of the circumstances discussed in this report indicate that improvements can be_made in the management and supervision of operations. These improvements are as follows: o' Transition from a Construction to an Operations Attitude - To further instill an operating attitude in all Operations personnel prior to fuel load, TU Electric is taking several actions, including: 1) directing personnel to immediately evaluate the impact of events and equipment failures on the operability of components and systems; 2) directing personnel to evaluate events and-equipment failures for deportability under -10CFR50.72 and 50.73 and the Technical Specifications;

3) deleting the provision in ODA-408 which allows procedurer for off-normal evolutions to be issued without review by the Station Operations Review Committee (SORC), and requiring test procedures issued.

after September 2, 1989 to be reviewed and approved through post operating license processes; and 4) eliminating temporary programs and more fully implementing permanent operational programs. o Greater Control of the Pro.iect by Operations -TV Electric is taking several step-to provide Operations with greater control of the project, including: 1). reassignment of management of the Transition Team from the Projects organization to the Vice President, Nuclear Operations; 2) development of an integrated schedule by Operations;. 3) completion of system and area turnovers to Operations; 4) reassignment of the responsibility for the power ascension program from Startup to Operations; and 5) requiring Operations approval for scheduling incomplete construction items to be completed after fuel load. o Improvements in Notification of Operations Manaaement and Supervision of Events and Eouioment Failures - TU Electric is taking several steps to improve notifications to Operations management and supervision of events and equipment failures, including: 1) Operations personnel have been instructed to provide greater detail in problem descriptions on Work Requests to alert management to the severity of problems; 2) SR0s are now reviewing Work Requests for potentially significant multiple equipment failures and are l notifying management of such failures; 3) operators have I been directed to request assistance from systems

,4 Attachment to TXX-89596 August 18, 1989 l Page 36 of 72 ] j engineers to help evaluate problems involving j plant systems; and 4) the CPSES morning meetings on operation and plant events have been enhanced through greater participation by all project organizations. o Time and Manoower Needs for Specific Tasks - TU Electric is taking several steps to provide additional assurance that Operations management and supervision are aware of the time and manpower requirements for specific activities, including: 1) workshop training has been provided to Shift Supervisors on planning and controlling plant evolutions, including ensuring that manpower levels are adequate for routine evolutions; 2) A0s have been instructed to identify any need for additional manpower and to identify any problems with access, work conditions, etc. during pre-evolution briefings; and 3) activities performed near the end of a shift will be planned to ensure that the activities can be performed prior to the end of shift or that relief will be available for the personnel performing the activities at the end of the shift. 2. Corrective Actions Several of the circumstances discussed in this report indicate that improvements can be made in the corrective actions for plant events and equipment failures. These improvements are as follows: o Documentation and Reportino of Events and Eauipment Problems - TU Electric is taking several steps to enhance the documentation and reporting of plant events and equipment failures, including: 1) Operations procedures have been revised to encourage Operations personnel to document human factors concerns inside or outside the control room; 2) Operations personnel have been instructed to document the significance of problems, including leakage amounts, on Work Requests; and 3) the Condition Report (CR) program has been initiated for the documentation of non-hardware problems. o Acoressiveness and Timeliness of Investigations of Plant Events and Eauipment Failures - TV Electric is taking several steps to increase the aggressiveness and to improve the timeliness of investigations of plant events and equipment failures, including: 1) the PIR program will be refined to include provisions for failure mode analyses and human performance evaluations; 2) PIRs are being discussed in the CPSES morning meetings on L

i. ;. Attachment'to TXX-89596 L August 18, 1989 Page 37 of 72 operations and plant events to provide for immediate management review and determination of whether multi-discipline evaluations are warranted; 3) operators have been directed to request assistance from systems engineers to help evaluate problems involving plant systems. o Improvements in Task Team Evaluations - Based upon the experience with the Task Team investigation of the April 23 and May 5 events, TV Electric has. learned several lessons that will be applied as appropriate to any similar investigations in the future including: 1) establishing the team promptly after the event; 2) utilizing a multi-discipline team; 3) having dedicated, full-time team members; 4) designating a single point of contact with the NRC to ensure that the NRC is provided with completm wnsistent, and timely information; and

5) establishing a clear line of communication and direction from management to the Task Team. These lessons will be formalized in an incident investigation procedure.z 3.

Communications Amono Operators and Shifts Several of the circumstances discussed in this report-indicate that improvements can be made in communications among operators and between shifts.. These improvements are-as follows: o Communications Amono Operators - Administrative procedures will be revised to provide _for the prompt transmission of plant incident information to Operations personnel. o [gspunications Between Shifts - Administrative procedures ha've been revised to require'that an oncoming shift be notified of the " Lessons Learned" by the preceding shifts, including plant events, significant PIRs involving operator error or involvement, and unexpected system or component responses. Shift Supervisors are now required to brief the oncoming crew on plant statu!., upcoming evolutions ~ on the next shift, and current lessons learned or FIRS. The Manager, Operations now briefs a crew returning to shift work after a long period off-shift to notify them of events during this period and scheduled events during the next week. Shift Orders have been enhanced by including policy changes, corrective actions e for PIRs, and other general information. A working copy of evolutions is now maintained by Operations until the evolution is completed or terminated by the Shift Supervisor.

Attachment t3 TXX-89596 August 18, 1989 Page 38 of 72-4. Personnel Awareness of Operatina Events and Eauipment Failures and of Their Imolicatiojis for System Operability Several of the circumstances discussed in this report indicate that improvements can be made in awareness by Operations personnel of operational events and equipment failures and of their implications for system operability. These improvements are as follows: o Imoact of Work Recuests - TU Electric is taking several steps to provide additional assurance that Operations personnel are aware of Work Requests and their implications for plant operability, including: 1) open l corrective and preventive maintenance Work Requests have been reviewed, and any operability concerns and mode restraints have been identified; 2) control room operators have begun to review Work Requests generated during the previous 24 hours to identify significant failures, potential impacts on plant operability, deportability of the Work Requests, and the priority of the Work Requests; and 3) operators have been directed to request assistance from systems engineers to help evaluate problems involving plant systems. o Availability of Information - TV Electric has taken or is taking the following actions to improve the availability of information regarding plant events and equipment failures to Operations personnel including: 1) making current PIRs available in the control room and referencing the PIRs in the station log; 2) discussion of PIRs at the CPSES morning meetings on operations and plant events; and 3) implementation of a system status program that may include, for example, the use of laminated prints that can be marked to indicate system or coniponent status, o Shift Loa Information - TU Electric will take the following actions to improve the documentation of equipment problems in shift logs: 1) problems causing initiation of a PIR will be referenced in the Station s Log with its PIR number; 2) Technical Specification Limiting Conditions for Operation (LCO) will be tracked i in the Unit Log and will be discussed during the shift turnover process. i

,,..s Attachment to TXX-89596 August 18, 1989- ' Page 39 of 72 : D.- Conclusion TU Electric has evaluated.the events on April 23 and May 5, their

o. -

impact on hardware,.and implications for operation. Based upon this evaluation, TU Electric is taking corrective' action.for the affected hardware, has taken corrective action to address the root )- causes of the events and to prevent recurrence of similar events, l and is making improvements to. address the lessons learned from the i events. 1 l l l j l

f i N E L.__ ' w ,W N , Attachment to TXX-895961

August 18.-1989; hge 40 ofs72 s

43 LVIII.:!SUMARYANDCONCLUSIONS.' o TU. Electric performed an investigation of the root causes and significance of the April' 23 and _May 5. events..The events were caused by defects in BW/IP check valves.as a. result of an inadequate reassembly procedure, 'and by a failure of Operations personnel to-L follow procedures while manipulating isolation valves. Additionally, the event. of May _5 was caused in part by a mechanically bound isolation valve :These events had no actual safety significance because there 'was no fuel in:the reactor'and. Unit 1 was not radioactive.- If'a. E similar event occurred 'during operation, operator action would have: maintained the reactor in a-safe condition. TU Electric-has-taken corrective action for the deficiencies in the hardware identified by these events, including inspection and modification of BW/IP. check valves. Additionally, TU Electric-is - taking action to address the root causes of the events and to prevent

returrence'of similar events, including revision _of the' assembly procedure for the BW/IP check valves,' providing additional training on'-

compliance with procedures and clarification of the procedure ~ governing. manipulation of the AFW. isolation valves. Finally, TU Electric has identified lessons learned from these. events and is taking actions to !

improve the managtwnt and supervision of Operations personnel, to improve corrective actions for plant events and equipment. failures, to simprove communications'among Operations. personnel and between shifts',-

and to improve awareness of operating events and. equipment failures and their imp 1.ications for system operability. p f 4 _'-__m_ _m. m

A t TXX-89596 _ RETAINER f' 7 ~ / j ?l$" / /"" N ~ ENEATH EAT LIP DISC ei x w Ags x / / y_ s y l FIGURE 1

s f, N n 7 r S Y v d E 0 I I r r r r i I i .~ e i t a ~ r i g y; I { lI

• e_ "4 -

m v Ty - fY. ?. s :- ef ew e P 9 F r e c v v gY. V q-ry - r r F n O o e g g Bn d d T s c v N ruv cl 3 w#'tD . 4:g. g m.Y. gtel1 3 3 g, 9 ia F E a r r. t= lA A LC V M j, t t i . s . s d. E s &,A 9 e .v: = c 8 8+. ,9ce T T I 9 1 w. .o _c 3 I 2 's- ,e ? ,,,s xy5 y=) I! L HIE I I R 6 h IE ,i P r.$ A A p l s- '-is ',.'w ob w i. ., d v-7 R. r r2 eGf o s 6 t 4 s4 i 0 _e, s 5 s s i 4 r r n I o r 1 Ai a A a f t a a t ]el , [I. ' 8 , I l. a L ~ = = m P Y P v 5 3 V D V F 2'* n N N F1A a A0T A0" ws S T D D EP M M M ,D a I E m-m T E A G S K A N N R g E A O T ~ a D T w N S O >f* gz C m a S , ENM w

a 9 ', Q 7~-i G N ^_. Q, j 7 N OW_ @vl

• i i

I v U-e. t r e r i t r ,J k i ~, { V e; n,Y e9 r F n,Y m v v 9 r F 1 l; 1 L k R* s t v s ia g g

Me-$

3 . m, t g, Dgo t tcv t T ara t i s t N m1 1 i" 1 1 1 4 2 5 d f 9 - &1 qn* o c'* E c a' V m* a E r t 8 9 1 h,< 5 l xyg y 5~ ~- ~ .? I: 5 h, h Ie i se Y n -ra . jh A k k I ( S I M ,Y FI PI ~ m. i i' W'ft gE vU v rN o ~ s 5 g t D:r.R^ r-s e 3m j o e S , i - s 4 o im a n r-i i

r. v I

g ~ a ~ a .r u T r m, w 8. a ~ Y P 3 v V 9 t N AC iF1a 2 D N J mfd km 3 rA0t t a s r t IJ M H I w E e-T E g A G ) S A T N NR S E E A O C D T( T N S O f k3= S! 4hm"

r Attachment to TXX-89595 August 18,-1989 Page 44 of 72 APPENDIX 1 RESPONSE TO NRC CONCERNS In addition to the actions taken to correct the deficiencies identified as a result of the April 23 and May 5 events, TV Electric has taken a number of actions to address the root causes and prevent recurrence of these events. Furthermore, TU Electric has implemented improvements to address a number of lessons learned as a result of the investigation of these events and the possible precursors to the check valve failure. These actions are described in Section VII of this report. The NRC staff made-a number of conclusions and recommendations -in the AIT Report. Additionally, on July 17, 1989 Mr. Warnick, the Assistant Director for Inspection Programs of the Office of Special Projects of the NRC, enumerated similar weaknesses to senior TU Electric management during a meeting in Rockville, MD. TU Electric either has or will address the NRC staff's concerns and recommendations as set forth below. A. Mr. Warnick's July 17. 1989 Concerns About CPSES Goeration and Corrective Actions ImD1emented by TU Electric 1. NRC Concern Operators and Startup personnel failure to follow procedures. Valving errors to start the 2 backflow events, PT-01-02, PT-37-01, and PT-64-03.. TV Electric Action TU Electric is taking a number of corrective actions to improve future compliance with procedures (see Section VII.B.2). 2. NRC Concern Operators' lack of sensitivity to the position of valves. Changing the AFW valves out of the proper order of sequence. TU Electric Action In addition to placing increased emphasis on compliance with procedures, TU Electric has provided training / workshops on avoidance of the April 23 and May 5 events and the risks associated with improper valve line-ups (see Section VII.B.2). 3. NRC Concern Operators' failure to recognize the significance of check valve backleakage during the precursor event.

' Attachment to TXX-89596 4 August 18, 1989 Page 45 of 72 TU Electric Action TU Electric is taking a number of steps to ensure that the significance of equipment failures is documented and that Operators are aware of the impacts of equipment failures (see Sections VII.C.2 and VII.C.4). 4. NRC Concern Operators' failure to make sure supervision was aware of the three check valves that had significant backleakage (precursor event). TV Electric Action TU Electric is taking several corrective actions to ensure that Operations management is aware of future events and equipment . failures (see Section VII.C.1). 5. NRC Concern Supervisors' failure to stay informed of plant evolutions and problems (the system flushing to solve the chemistry problem and the RHR valving problem during the remote shutdown' test. If check valve had failed, it would have put RCS water to the RWST.). TV Electric Action TU Electric is implementing several corrective actions to improve reporting of equipment failures and plant events to management and supervision of Operations, and to improve the documentation' and reportino' of events and equipment problems (see Sections VII.C.1 and VII.L. 2 and TXX-89430 dated June 26,1989). 6. NRC Concern Failure to accurately and adequately document the extent of a problem (the precursor event Work Request said, " repair check valve leakage"). No TDR on RHR event. No TDR on PT 44-01 and QA person doing surveillance did not issue a surveillance deficiency. TU Electric Action TU Electric is tr. king a number of actions to enhance documentation and reporting of future events and equipment failures (see Section VII.C.2 and TXX-89430 dated June 26,1989). 7. NRC Concern Weakness in the documentation of equipment problems in the shift log. l l i

~. I Attachment to TXX-89596-l- August 18, 1989 l-L Page 46 of 72 TU Electric Action l TU Electric is implementing a number of actions to improve l communications on equipment problems and events between operators and shifts (see Section VII.C.3). 8. NRC Concern l Failure.to recognize inoperable equipment. -TU Electric Action TU Electric is taking a number of. steps to enhance the timeliness and aggressiveness of corrective action and to enhance the awareness and impact of operating events and equipment failures on system operability (see Sections VII.C.2 and VII.C.4). 9. -NRC Concern Failure to recognize and document equipment out-of-service. TU Electric Action .. A number of steps are being taken to enhance evaluation, documentation, and investigation of equipment failures and work requests (ree Sections VII.C.2 and VII.C.4).

10. NRC Concern Lack of adequate communications between the operating shifts.

TU Electric Action i TU Electric is taking a number of steps to enhance communications within Operations and between shifts (see Section VII.C.3). II. NRC Concern Weakness in the exchange of information at shift turnover (Precursor event and April 23 event). ] TU Electric Action TU Electric is taking several actions to enhance communications between shifts (see Section VII.C.3).

12. NRC Concern l

Supervision / Management review of problems documented on work requests (Precursor event). j i

Attachment to TXX-89596 August 18, 1989 Page 47 of 72 TV Electric Action TU Electric is taking a number of steps to enhance the documentation, investigation and reporting of events and equipment problems and improve reporting of events and equipment failures to Operations management and supervision (see Sections VII.C.1, and VII.C.2 and VII.C.4).

13. NRC Concern Failure of persons with knowledge of the precursor check valve problems to raise the information to management.

TU Electric Action TU Electric is taking a number of actions to improve documentation of events and equipment problems, and to improve the reporting of such events and problems to Operations management and supervision (see Sections VII.C.1 and VII.C.2).

14. NRC Concern The slowness and lack of direction initially demonstrated by TU Electric following the April 23 event.

TU Electric Action TU Electric is taking action to improve the aggressiveness of investigation of events and equipment failures and to enhance future Task Team investigations (see Section VII.C.2).

15. NRC Concern The perception that " Projects and the Schedule" were driving decisions at the time of the precursor event and the start of HFT.

TV Electric Action TU Electric is taking several actions to improve the control of the project by Ope ations (see Section VII.C.1).

16. NRC Concern The perception that the Operations staff are not in control of the plant.

TV Electric Action TU Electric is taking several steps to increase Operations control over the project (see Section VII.C.1).

4 Attachment to TXX-89596 August 18, 1989 Page 48 of 72 B. Conclusions in the NRC's AIT Report on the April 23 and Nav 5 Events and Corrective Actions Taken by TU Electric 1. NRC Conclusions (4.1.1) The identification of three inoperable check valves in the TDAFWP supply lines on April 5 should have been aggressively pursued. Instead, it was assigned a normal work request priority. This event reflects a lack of understanding of the system operability implications of failed components and a lack of aggressiveness of Operations management to follow-up on the results of the system flush they had specifically scheduled to determine the scope of the original identified check valve problem. This event was clearly a missed opportunity to discover the full extent of the check valve problem in time to prevent the April 23 and May 5 events from occurring. TU Electric Action' TU Electric is taking a number of actions to ensure timely and aggressive investigation and corrective action for future events and equipment failure. Furthermore a number of actions were taken to enhance documentation and reporting of events and equipment failures (see Sections VII.C.1, and VII.C.2 and VII.C.4). 2. NRC Conclusions (4.1.2) The overall response by control room personnel to both events (falling steam generator levels) was weak (see paragraph 2.1.2). TU Electric Actions TU Electric has implemented a number of corrective actions to preclude the reoccurrence of similar events which, including training on the April 23 and May 5 events, will improve response of control room personnel to events of this type (see Section VII.B.2). 3. NRC Conclusions (4.1.3) Continuing to test the AFW system after the April 23, 1989 event with known multiple failures of check valves without taking appropriate precautions shows a potential lack of respect for degraded plant conditions. It also shows lack of communications between shifts. TV Electric Actions Operations management did consider the degraded condition of the check valves before concurring that testing activities could

t\\ ' Attachment to'TXX'-89596 ' August'18,1989 .l _ Page 49 of. 72 : 6 proceed. It was concluded that administrative controls in place would compensate for the. problems identified if. properly implemented.. Notwithstanding, TU Electric is taking several steps - to. ensure that Operations personnel are aware of operation events and equipment failures and of their impact on plant-operability, and to enhance communications (see Sections VII.C.3 and VII.C 4). 4. FJ3fifonclusions' (4.1.41 It tvok an inordinately long period of time' for Operations to adequately identify the May 5 event and to report it as such, especially.considering that it had a greater magnitude of severity thun the April 23 event. The applicant's originally stated intent of including this event within the first PIR (110)= appeared to be slow. In fact, PIR-89-129 was 'only written at the NRC's AIT insistence. TV Electric Actions TU Llectric is taking actions to enhance the timeliness, reporting and evaluation of future events and equipment. failures (see Section VII.C.2).

5..

NRC Conclusions (4.1.5) The out-of-sequence operation of valves in the May 5 event, occurring 12 days after a fundamentally identical out-of-sequence valve operation in the April 23 event, reflects a significant' weakness in the applicant's ability to prevent an operational error from recurring'. TU Electric Actions TU Electric is taking actions to improve adherence to plant procedures; aggressive documentation, reporting, and evaluation of events and equipment failures; and communications between shifts (see Sections VII.B.2, VII.C.2 and VII.C.3). 3. NRC Conclusions (4.1.6) Sending only one auxiliary operator near the end of shift to operate valves IAF-041 and 1AF-042 reflects a lack of understanding in the control room regarding task manpower requirements. l TU Electric Actions 'TU Electric is taking several steps to provide additional assurance that Operations management and supervision are aware of manpower requirements for specific plant activities (see Section VII.C.1).

Attachment to TXX-89596 August 18, 1989 Page 50 of 72 7. NRC Conclusions (4.1.7) The AIT considers the difficulty of operation of valves IAF-041 and 1AF-054 to be a contributing cause to the April 23 and May 5 events, but of minor safety significance. The AIT supports the applicant's intent to make these valves easier to operate. TU Electric Actions Actions are being taken to facilitate manipulation of remote-operated valves (see Section VII.B.2). 3. NRC Conclusions (4.1.8) The evaluative process, which ultimately determined the root cause i for the check valve failures appeared to be unnecessarily protracted in that it required almost six weeks from the inception of the AFW Task Team until the development of a definitive root cause and corrective action program. This protracted process, although not directly related to any regulatory requirement, is an example of the applicant's lack of management aggressiveness in the resolution of a safety-significant issue. This issue involved 1 the multiple failures of passive components in a system intended to mitigate the consequences of an accident. For an NT0L plant, the applicant's response did not reflect the style of proactive Operations management philosophy normally associated with safe reactor plant operation. The AIT notes that when the applicant's Project Management took charge of the Task Team on May 26, 1989, efforts were significantly more timely and reflected a stronger commitment'to corrective action. The applicant's Task Team went 4 to the vendor Borg-Warner and n.ade things happen. This aggressive attitude by management brought to light the root cause and brought about a corrective action plan in a timely manner. TV Electric Action TU Electric is taking action to improve the aggressiveness and timeliness of investigation of plant events and equipment failures and to improve future Task Team evaluations (see Section VII.C.2). C. TV Electric Implementation of NRC Staff Recommendation 1. NRC Recommendation (4.2.1) Create a minimum equipment list that would aid Operations personnel to make jt.dgements regarding the effect of failed components on system operability. TU Electric Action Due to the nurnber of modes of equipment failure and the fact that the significance of the failure of a specific piece of equipment

Attachment to TXX-89596 August 18, 1989 Page 51 of 72 is dependent on plant configuration and what other equipment remains operable, TU Electric does not believe that a reliable minimum equipment list can be created. Furthermore, because a minimum equipment list would not be comprehensive (anticipating 4 the significance of every piece of equipment in every plant l configuration), plant operators might place undue reliance on such I a list and fail to perform probative analysis of the significance E of equipment failure not on the minimum equipment list. TU Electric believes that equipment failures must be evaluated on a i case-by-case basis. TU Electric is upgrading its program for the evaluation of equipment failure by requiring prompt review of the i impact of maintenance work requests and additional engineering support for operation (see Sections VII.C.1, VII.C.2 and VII.C.4). 2. NRC Recommendation (4.2.2) Assign system engineers the in-line task of reviewing all work i requests related to a given system. The engineer would evaluate i the impact of all component failures in regard to system operability. TV Electric Action Operators are being directed to request assistance from system engineers to help evaluate problems involving plant systems. Other actions are also being taken to enhance evaluations of Work Requests and impacts of equipment failures on operability (see Sectir:as VII.C.2 and VII.C.4). 3. NRC Recommendation (4.2.3) Provide training to control room personnel and supervisors regarding manpower requirements for certain types of plant evolutions. TU Electric Action Workshop training is being provided to Shift Supervisors on planning and controlling plant evolutions, including ensuring that manpower levels are adequate for routine evolutions (see Section VII.C.1). 4. NRC Recommendation Provide continued emphasis on training plant personnel to comply with procedures. Steps are to be performed in sequence unless otherwise specifically approved.

Attachment to TXX-89596 August 18, 1989 Page 52 of 72 TU Electric Action The Shift Operations Manager has developed and implemented an i action plan to enhance procedural compliance. As part of this 1 plan, a memorandum on procedure compliance was provided to the Shift Supervisors, who in turn discussed the memorandum with their respective crews. The Mantger, Operations and/or Shift Operations i Manager also met and discussed the memorandum with each crew. Additionally, a workshop was held by the Manager, Operations with Operations Department Senior Reactor Operators (Shift Supervisors, i Unit Supervisors, Shift Technical Advisors and Staff), including Training and Plant Evaluation personnel, to discuss the April 23 l and May 5 events and procedure compliance. Emphasis on procedural compliance will continue to be emphasized in recurrent replacement training for operators (see Section VII.B.2). 5. NRC Recommendation (4.2.5) Provide better communications between Operations staff, especially during shift changes. TV Electric Action TU Electric is taking several actions to enhance communications between operators and shifts (see Section VII.C.3). 6. NRC Recommendation (4.2.6) Provide a large and conspicuous plant status board in the control room, sufficient to provide significant " night order" information and to facilitate the transfer of information between shifts. TU Electric Action TU Electric is implementing a system status program that may include the use, for example, of laminated prints that can be marked to indicate system or component status (see Section VII.C.4). 7. NRC Recommendation (4.2.7) Initiate an immediate design revision to separate the 3-inch miniflow check valves from their associated orifices. The present configuration, if not corrected, lends itself to an exceptic : ally short lifespan for the check valves due to flow turbulence and valve tapping damage (see paragraph 2.3.3). TV Electric Action TU Electric is conducting evaluations to determine the effect of flow turbulence and valve tapping on the 3-inch miniflow check valves. Appropriate corrective action will be taken.

l Attachment to TXX-89596 August 18, 1989 Page 53 of 72 APPENDIX 2 CHECK VALVE BACKLEAKAGE TESTING Results Engineering developed a specific test prncedure to determine which check valves in the Feedwater and Auxiliary Feedwater Systems leak past their seats. The testing was initiated on April 28, 1989 and concluded that the check valves in the TDAFWP (IAF-078, 86, 98 and 106) and HDAFWP (1AF-075, 083, 093,101) supply lines failed under backflow conditions. The check valves in the main feedwater upper penetration (IFW-195,196,197,198,199, 200, 201, 202) did not leak past their seats. Performance and Test personnel tested the check valves in the TDAFW and MDAFW supply lines using ODA-408, " Nonstandard Alignments and Evolutions," procedures 1-89-053 and 1-89-055. The testing consisted of isolating the valve, connecting the upstream side of the valve to the nearest drain, pressurizing the downstream side of the check valve and measuring the decrease in pressure and flow across the valves after the upstream connections were i opened. Results are as follows: Test Test No. Valve GPM Leakaoe Prfssure (PSIG) 1-89-055 1AF-075 5.32 99 l-89-055 1AF-078 5.47 100 1-89-055 1AF-083 5.42 98 1-89-055 1AF-086 5.52 100 1-89-055 1AF-093 5.42 96 1-89-055 1AF-098 5.47 100 1-89-055 1AF-101 5.42 95 1-89-053 1AF-106 5.01 95 The AFW Pump Discharge Check Valves were tested by ODA-408 Procedure 1-89-058. The tests for the MDAFWP check valves 1AF-051 and 1AF-055 were performed by isolating the valves and pressurizing the-downstream side. When the upstream test connection was opened, no leakage was detected. The TDAFWP check valve (IAF-038) was tested in a similar fashion except that the upstream test connection is on top of the pipe, so the vent was cracked open while coveed with a soapy film to detect air displacement with the upstream pipe pressurized. Pressure on the upstream side could not be stabilized, although no air leakage was detected. The pressure problem was attributed to boundary valve leakage from a valve other than the check valve, and a radiograph (RT) performed on IAF-038 and confirmed that it was closed. Results of the test of AFW pump discharge check valve are as follows.

Attachment to TXX-89596 - August 18,-1989 i Page 54 of 72. Test' Test No. Valve' GPM Leakaae Pressure (PSIG) 1-89-058 1AF-038 0 50 1AF-051 0 78 1AF-065 0 71 LThe AFW miniflow recirculation check valves were tested by ODA-408 procedure. ' l-89-060.- The test was performed by crosstieing the_ recirculation header to the pump discharge. header to provide CST head pressure against the downstream side of the check valves. Leakage was collected at the upstream drain valve. Results are as follows. 1 Test Test No. Yglyg GPM Leakaae Pressure (PSIG) 1-89-060 1AF-057 7.81 21.5 1-89-060 1AF-069 0.0185 21.5 Because of inconvenient test connections, the recirculation check valve for the Turbine Driven Pump,-1AF-045, was not tested; instead radiography was used to determine the status of the valve. RT indicated that the valve was hung open. The low leakage rate through 1AF-069'is attributed to the reworking of the valve -internals that was performed in response to.the April 19, 1989 event. The Main Feedwater pump discharge check valves were tested by ODA-408, "Non Standard Alignments and Evolutions," Procedure 1-89-059. The test was performed by isolating the valve and pressurizing the downstream side. When the upstream test connection was opened,-leakage was collected. Results are as follows. Test Test No. Valve GPM Leakaae Pressure (PSIG) 1-89-059 1FW-006 0.817 120 1-89-059 IFW-013 8.62 120 The FIBVs were tested by ODA-408 Procedure 1-89-068. The test was performed by applying pressure beneath the air operated bypass valve seat and charting the leakage as pressure was increased. The attached charts show leak rate through the FIBVs as a function of applied pressure. As these charts demonstrate, leak rates for each of the FIBVs (except the FIBV for SG 4) increased sharply when back pressures reached approximately 100 to 300 psi. From these charts, it was concluded that the FIBVs would have isolated against-containment atmospheric design pressure as reouired by the design,1 but that they were not sufficient to prevent backflow from the steam generators into the AFW-System during conditions involving the higher pressures on April 23 and May 5. Therefore, the Task Team determined that the path of the backflow included the FIBVs. l

p, li l.

.g,

i -Attachment to TXX-89596 l" - August 18, 1989 l Page 55 of 72 l In conclusion, the' Task Team was able to determine which check valves in'the Feedwater and Auxiliary Feedwater-Systems failed where subject to backflow L conditions. This determination was useful in establishing the backflow paths during.the April 23.and May 5 events. Additionally, the Task ~ Team determined that a number of check valves were subjected to backleakage. l l2 IAlthough' the FIBVs satisfied their design performance requirements, TU Electric is revising its procedures to require isolation of these valves when the main feedwater system is not in operation supplying flow to the steam generators. Additionally, TU Electric is conducting a review to determine whether similar valves exist in safety-related systems and whether additional protection would be provided by requiring isolation valves upstream of such valves to remain closed during particular plant conditions. i m_m_

Attachment to TXX-89596 Feedwater Isolation Bypass Valve . August 18, 1959

8. wok.,. ras axvits - twv-zia7 Page 56 of 72 600

. 500 i q 400 " iN 5 300 \\ E i S 200 j ' ~ 100 8 E~~ ~ ~ 0 0 100 200 300 400 Applied acckpressure (psig) Feedwater Isolation Bypass Valve Bocklockage Test Results - 1HV-2128 500 7 400 1N 300 sE 2 200 k 'I 100 0 O 100 200 300 400 Applied Backpressure (psig)

l; 1 Attachment to TXX-89596 Foodwator isolation Bypass Valvo l ' August-18, 1989 m m,7 c.,,,,, Iwv-stas Page 57 of 72 s00 500

• .400 g

lN E 5.' 300 s* / t 3 200 100 - I M _ -U. IC[' ~"' 0 0 100 200 300 400 Applied Backpressure (psig) Feedwater Isolation Bypass Valve Bockleakoge Test Results - 1HV-2186 500 eb0 q -Eb E 300 sa o 3 200 100 r:-.:- MA j, _, _' ; -- - -a 0 -7 + O 100 200 300 4co AppEed Sockpressure (rsig)

t l Attachment to TXX-89596 August 18, 1989 Page 58 of 72 APPENDIX 3 CHECK VALVE MODIFICATION AND MAINTENANCE HISTORY A search of the historical files was performed to determine if any onsite modification or maintenance performed on the valves could have been responsible for their recent failure. A repair program of 1983 was of special interest. A modification had been made to replace tack welds holding the disc to disc stem, and disc stem to .11sc nut. This modification was made because of the potential for valve internals to come apart during operation, and as a result of a recommendation by BW/IP. Only three of the valves that feiled tests in 1989 were modified on site during 1983. However, the valve internals had all been removed at one time or another in order to perform the necessary inspection during the 1983 modification. Additionally, during the past years, internals of some valves have been removed for routine system flushing. Valve internals, which were removed in 1983 or for subsequent flushing, were reassembled in accordance with CP-CPM-9.18 and CP-20B-1, "Borg Warner Maintenance Manual," the BW/IP Inspection Plan for Check Valves and MMI-1002, "Borg Warner Check Valve Inspection." Review of documents indicated that the reassembly of the valves was performed in accordance with approved procedures. In addition, records show that QA control and QC verification was properly applied to each activity. Also, a representative of BW/IP was present during the 1983 modification. There is no evidence of noncompliance to these procedures. However, the procedures lacked adeouate detailed instructions to ensure proper reassembly because they did not provide instructions for aligning the valve disc and seat. Therefore, the Task Team concluded that the BW/IP check valves were imnroperly reassembled due to the inadequate assembly instructions based on vendor information. In addition to the work performed in 1983, check valves have been subject to other maintenance and modification activities. The attached table lists each ) Unit I and Common BW/IP check valve and its maintenance and modification history. The information in the attached table was compiled from a review of work travelers, Inspection Removal Notices and QC inspections associated with these valves. Additionally, previous work documents, including Nonconformance q Reports (NCRs), Problem Reports (prs) and PIRs were reviewed for any unusual i trends or noncompliance with specifications. In order to determine whether any trends existed, characteristics of each valve and its associated 1 maintenance and modification activities were identified and placed into one or more categories. These categories included: o Size - nominal pipe size, o R:iting - the pressure rating of the valve.

Attachment to TXX-89596 August 18, 1989 Page 59 of 72 l o Internals removed for inspection - if valve internals were removed for inspection during the 1983 overall repair program. o Disc Assembly Modification - if full fillet welds were not present, and the disc assembly was taken apart or modified in any way to make the recommended repair. o Owners form NIS if this form was present, it ensured that the repair was performed on site by Brown & Root. o Bushing Modifications - if the axial clearance was changed at any time for any reason other than the 1983 modification, o Internals remceed for flushing - if the internals had ever been removed by operators and/or maintenance activities other than the above. o Downstream of an orifice - if the check valve is operating in the area of an orifice. o Post work inspection by BW/IP - if any indication was given in the modification documents that the vendor was present for the modification work or made a separate inspection at a later date. o Separate passivitization - if the internals were ever removed strictly for rei.. oval of rust. Note that rust removal was performed in conjunction with the 1983 modification. o Internals transferred - if the internals of the valve as it is now installed differ from those originally shipped with the valve. o Valve failing - if the valve was shown through testing not to hold back pressure, or if, through radiograph, it was determined to be restricted from closing. As the attached table demonstrates, most of the categcries do not exhibit any correlation with valve failures. For example, none of the valves that failed during testing had been subject to separate passivitization, transfer of valve internals, or bushing modifications to adjust clearances, and only three of the thirteen failed valves had been subject to modification. Therefore, the Task Team concluded that these maintenance and modification activities were not the cause of the backleakage. The table also identifies a correlation, in either whole or part, between valve failure and four categories; 1) valve size of 3" or 4" inches $ an)d 4)

• 2 internals removed for inspection; 3) internals removed for flushing vaive downstream of orifice / turbulence. The first three of these factors all 1

indicate that inadequate vendor assembly instructions were the cause of the valve failure; i.e., the inadequate instructions only pertained to 3" and 4" ] inch valves, and the inadequate instructions were used during reassembly

Attachment t3 TXX-89596 . August 18, 1989 .Page 60 of 72 + following removal of valve-internals. With respect to the last category, valve downstream of orifice / turbulence, TU Electric is considering an inspection program for valves near orifices and is evaluating the need to move.

these orifices, as is discussed in Section VII.B.1 and in Appendix 6.

2 n some cases, it was necessary (and common practice in the industry) to I remove the internals of a check valve in order to perform high velocity flushing of a piping systeci. In addition, in a few cases, the internals of a check valve were removed to facilitate draining cf a system. The occasional removal and reassembly of valve internals does not adversely affect the function of a valve, provided that these activities are performed properly. l

Attachment ~to TXX-89596 ~ August 18, 1989 Page 61 of 72 111iM i %4 2 VALVE O / 1AF-009 3" 150 X X X X X i -014 6" 150 X X X X X -024 6" 150 X X' X X X -032 8" 150 X X X X X -038 8" 900 X X X X -045 3" 900 X X X X -051 6" 900 X X X X -057 3" 900 X X X X X X -065 6" 900 X X X X -069 3" 900 X X X X X X _ -075 I4 " 900 X X X X -078 4" 900 X X X X X X X __ 063 4" 900 X X X X -086 4" 900 X X X X X _ -093 4" 900-X X X X -098 4" 900 X X X X X x! I lX __. _101 4" 900 x x l -106 4" 900 X X X X X -167 8" 150 X X X i X l i l l

---

_-__a

. 7; ..q 3Ittachment'tsTXX89596. + y August?l8, 1989: i Page 62 of;72: / .= ~,j $N $@ h. l M 02 2 N tr I~g ~wR T c2 52 s ~8 2 g $5 "f cs 5g y *3 g d = t

• N

'g-o .g gg s r 31E E~2 j~f!c u % c& 5~5 s U ; " l E l i ta g l l $ $ $ s $

• g,u

-M g~a se -n { j L VALVE icc.0003 -3" 150 I Y-Y 0004 3* 150 I 0317 10" 150 X-Y I VENDOR -0602 '4" 150 Y WELDED VENDOR- -0651 4" 150 Y WELDED 0690 4"' 150 Y-I Y ~T -0693 4" 150 Y Y Y I l -0697 4" 150 Y I 0713 8" 150 Y I I i 1DD-006 3" 150 'I T Y ~ ( -018 3" 150 Y Y Y I .021' 3* 15C__I I I I 1 g2 0l+8 i 3" 1?O I' i 1DD-065 3* 150 X l IDD-106 3" I 150' I l I q YvA.001 C" 150 Y' 002 6" 150 I j .I l I ICA-0016 3" 150 Y Y 2 X -f I i _ _ l- !.. l - .I

.= - -.,. ~, Attachment to-TXX-89596 LAugust 18, 1989 7 iPage 63 of 72 = 5 .g 2 6 .R ,h I d 6 - h U ,E! ese x a un ll 22 gl na$$ as.e 8~t:s n , :$.t" L n. nf-L u 5 E $55$$ f# $l?$$$$5 }$$$ 1CT-013 '10" 130 'T T -025 -16" 150 I I I T-lI INTERNALS I TRANSFERRED -042 ~10" 150 I I I -047 4" 300 Y I i -048 4" 3002 I I-I -063 4" 300 ! I -Y 300lI -064 4" Y I -065 10" 150' I T T I I -077 16" 150 I I I j -094 10" 150' T I X Y I -142 16" 150s T T I T -145 16" 150 I T I T _ -148 16" 300 Y I T T -149 16" 300 I I g_ I l l l l b ' LEAKLD AT IIRT l

f. Ij i

I lI Ig(-024 6" 150' T Y ' EVEN POST REty3g .x I -34? 3' ! M1I I 'Y

I 150l Y

~'I WELDED VENDOR -346: 3" 4 ~ VENDOR 403 3" 150 Y I WELDED 406 3" 150 Y !I .X Y 452 3" 1 I i 463 1 Y l l l l l-I i

p Attachment.to TXX-89596 August 18, 1989 l-Page 64 of 72 / 5 0 $e*$4E/E!r?$~8- @ ~ a g $8ssGil 2R R efle n ~ -/ e co~ s i*t u-[a ~ = ~ E 'U. j 'B a' e ES f E! n'k E2 Ie Rfue 2* g VALVE 1W-191 6" 900 X -192 6" 900 X -193 6" 900 X -194 6" 900 X -195 6" 900 X 196 6" 900 X -197 6" 900 X -198 6" 900 X 199 6" 900 X l -200 6" 900 X -201 6" 900 X l I l l -202 6" 900 X 3 l l i 1Ms.142-4" 900 x X 1 X_ j I I l l l I l 1 _ ___.-_3 !I { 143 _4" 990 1 3__ L I l I { i o i 1 i nE-AO.L_ML" ' 11'3 ! L1 L_L X 1 I j 10"!150 -oca X 7 1.,_3 7 .m J 3s _ m. a u _.! I 1 i l i l 150 l 1 l 4 X 1 X L ._. y_ I i i I I i i l I i _}

Attachment to TXX-89596 August 18, 1989 /@ Page 65 of 72 U N S".Q y Q o U 6 SJ g $'~ j ~ s$Ej"g~EU j u [ 55, O N C, p s Efill!lpl5N[affllhE*N!lE g y EQ E S*2 N h m E ) VALVE ~ ISU-014 4" 150 X X X X X -016 10" 150 X X -017 10" 150 X -04E 4" 150 X X -388 10" 150 X X X 3 VENDOR -389 10"!150 X wn nFD I l i ICT-030 3' i 150. X X l i 1 l l' 1 l ~~ i l l l i c c S I j_ 3 I l 8 l I. I i I L .1 l l 1 1 i ) l l 1 i l 1 j i I i l I

4 ~ Attachment to TXX-89596 L August 18, 1989 Page 66 of 72 APPENDIX 4 IDENTIFIED MATERIAL CONCERNS An unrelated deficiency pertaining to BW/IP check. valves that occurred during Station Service Water System testing was identified during the Task Team investigation. A swing arm on check valve ISW-048 failed because of a pre-existing flaw. The preliminary indication of the failure mode on ISW-048 was the. presence of a pre-existing flaws and hot cracking resulting from improper casting and/or heat treatment coupled with the aggressive chemistry of the Service Water System. Analysis of.two other swing arms from Unit 2 (2SW-0048 i and 2CT-0149) did not reveal the same type of flaws that were present in the L . failed swing arm but did suggest a potentially insufficient heat treatment. The two swing arms destructively examined were subject to a.relatively low service stress. In addition three more intact swing arms (which have seen varying degrees of service) have been destructively examined and show no signs of any preexisting flaw. In order to firmly establish the condition of all the Unit I-swing arms, all-the valves will be non-destructively examined. The examination will consist of: 1. Visual 10x inspection 2. Wet fluorescent penetrant particle testing 3. Replication on two zones of each arm In addition, an evaluation of the perosity observed in the clevis of a spent fuel valve (1XSF-004) was performed by the manufacturer. This evaluation consisted of an x-ray to determine extent of porosity and a revie's of the design calculations. The rcview concluded that the pare was Satisfactory for its intended rervice. In addition, an engineering evaluatica is baing perfmved to deteruine the maxiuum amount (size) of parosity which could be accepted without exceeding allowbie stresc in the remaining cross-section. A preliminary review of the i stress in th3 clevis indicated an extremely low service stress (approximately 6 ksi is ioposed) when compared to the allowable stress (approximately 34 1 ksi), i ibe material deficiencies were reported to the EC an June 26, 1989 as potentially reportable under 50.55(e). Safety significance of these l 'deficiewies are still under ovaluation and will be repcrted to the hRC as I part of SD M CP-09 19., .l 1 eplication is a process by which a surface is polished and an acetate tape R is applied, peeled off and microscopically examined. This provides a topological examination in which hot cracks can be detected. l

8. '

l. Attachment to TXX-89596 L August 18, 1989 Page'67 of 72 APPENDIX 5 RADIOGRAPHY. INSPECTIONS. AND COMPUTER ASSISTED DRAWINGS FOR BW/IP CHECK VALVES The Task Team utilized radiography (RT), inspections, and Computer Assisted' Drawings to help determine the cause of the backleakage through the BW/IP check valves. The results of these activities are discussed below. Radioaraohv L Twenty-one check valves were radiographer. Ten of these valves appeared to be l hung open (i.e., the top of the disc hung up under'the seat lip at the 12 o' clock position). Of the ten open valves, eight were four-inch AFW valves and two were three-inch AFW pump recirculation valves. Two other four-inch valves (IMS-142 and IMS-143) appeared to be seated improperly. Although the disc in these two valves did not-appear to be lodged under the seat, the discs were not in contact with the seats over the lower halves. The remaining nine check l valves appeared to be properly closed. The attached table provides specific valve radiograph results. Radiographing these valves played a key role in the identification of the root cause of the backflow. This technique showed that there was a difference between seat / disc elevation and that the disc was lodged beneath the seat lip. Inspections Fourteen of the radiographer valves were disassembled and inspected. The fourteen inspected valves included the twelve valves that were determined to be open as a result of the radiographs. The attributes subject to inspection inc1cded axial. play, seat angle, proper alignment, machining of the disc edge, and retainer position. The attached table shows the results of these inspections. As this table demonstrates, there does not appear to be any correlation between the inspected attributes and the valves that were determined to be open. For example, the inspected valve with the largest amount of axial phy in the disc (valve 1FW-198) was closed, while other vaives with less Exial play were open. Therefore, the Task Team concluded 1 that none of these attributes, in and of its' self, was the root cause of the hung open valve discs. Computer AsJi.ittELD_rawinos Using' CADS, 2D and 3D drawing rodels were created for the as-found condition of 1AF-106 (4" 900# Pressure Bonnet Swing Check Valve) and ifW-198 (6" 900t Pressure Sonnet Swing Check Valve). These drawing models simultted the potential for hang up and improper c1csere I of the check valves. The models were prepared with dimensions obtained from manufacturing drawings and dimensions taken from disassembled valves. Also, J

' Attachment to TXX-89596 August 18 1989 Page 68 of.72 input was obtained from the BW/IP representative onsite. The models - demonstrated that variation in disc elevation (and to sone extent disc stud axial play) affects valve operation. Conclusions Based upon the radiographs and the CADS, the Task Team determined that the backleakage through the check valves was caused by hung open valve discs.due to an elevation difference between the valve disc and seat (and, to a much lesser extent, excessive axial play). Using this information, the Task Team reviewed the vendor manual for the BW/IP check valves and determined that the manual did not provide adequate instructions for ensuring that the valve disc is at the same elevatien as the valve seat, and that BW/IP had not provided . acceptance criteria for axial play. c I l C_i___.__

y 1httachment'toTXX-89596U W nr 4 [ g,g g W$ AugustL18, 1989-g a Page 69~of 72~ g l l *g j E 19l VALVE 1AF-009 3" 150 1AF-014 6" 150 1AF-024 6" 150 1AF-032 8" 150 1 AF-038 8" 900 1AF-045 3" 900 I OPN N O 7.7 N N .223 1AF-051 6" 900 1AF-057 3" 900 I OPN Y 0 5 Y N .142 1AF-065 6" 900 1AF-069 '3" 900 I -CLSD Y 0 F3 Y-N .078 1AF-075 4" 900 I OPN Y 0 12 N N .165 1AF-078 4" 900 I OPN N -0 5 Y N .180 1AF-083 4" 900 I OPN Y 0 5 N N .206 1AF-086 4" 900 I OPN Y 0 5 Y N .193 1AF-093 4" 900 I OPN Y 0 12 N N .265 1AF-098 4" 900 I OPN N O 5 Y Y .147 1AF-101 4" 900 I OPN Y 0 5 N N .210 OPNlN 1AF-106 4" 900 I O 5 N N .197 l 1AF-167 8" 150 1L[,-121_., 6" 600 l 1 W-192_,,,,6d ' __60J [ l I i 1 37-12] 6" lt9E l XfE:,124 ft."__J00 i n if,E ;,1.9J,. ' _ff " 600 I CLSRt,_ I lyW 196 ,,,J.*,,,,,,100 I CMD 16!:.lf.I.. O" ff0 I CMD p 1D7,-19s - 6"' 600 I CLSD,. Y. .150 5 I Y .315 j 1 1 l ,1 W-199_ _ 6 " 600 I CLsD 6 f 1W-200 4" 600 I CLsDl l 1W-201 6" 600 I CLSD 1W-202 6" 600 I CLSD iMs-142 4" 900 I OPN Y O 3.2 Y Y .194 iMS-143 4" 900 I OPN Y 0 3.8-Y N .124 4" 300 x

• RT UNCLEAR 1CT-04g 3

11G Mif)hd- @" ._ 33@@. 52 i m l C99 nwm reare l

h' ' Attachment'to TXX-89596 August 18, 1989 Page 70 of 72-APPENDIX 6 EVALUATION OF AFW CHECK VALVES AGAINST EPRI GUIDELINES The BW/IP check valves in the CPSES AFW System' were evaluated against the criteria in EPRI Report NP-5479, " Application Guidelines for Check Valves in n Nuclear Power Plants," to determine whether any inconsistencies between the l'

CPSES check valves and.the.EPRI guidelines may have resulted in the backleakage through the CPSES check valves.

The EPRI Report states that the following six factors should be considered in determining the application of cher.k valves: 1) valve sizing; 2) valve closure time; 3) structural compatibility; 4) valve seat leakage limits; 5) valve - orientation; and 6) piping arrangement. The results of the Task Team's evaluation of the CPSES AFW check valves against each of these factors is discussed below. .l. Velve Sizina The 4" and 6" AFW check valves showed no sign of wear associated with improper sizing. Therefore, the Task Team concluded that valve sizing was not a cause of the BW/IP check valve failures. 2. Check Valve Closure-Time AFW System design does not require any specific check valve closure times. Closure times were not a factor in the AFW check valve failures. 3, Effuctural ComnittjAility EPRI guidelines rccomend a riargin in pressure boundary thickaess to account for wastige due to erasion /corrusion. The mininm! valve thickness for this margir is dependent on design pressure and temperature, the ANSI pressure rating, and the ANSI body thickness of the valve. Under the EPRI guidelines, the minimum valve body thickness should be 0.411" for a typicai 4" valve in an AFW supply lir,e to the steam generators (conservatively arsuring system design temperature is 2000F). The minimum body thickness for a 4" AFW valve at CPSES is 0.509". Therefore the BW/IP check *;alves that crere evalu dcd conformed with tha EPRI guidelines. 4. Seat Leakace Limiti Backleakage through the AFW check valves on April 23 and May 5 was not caused by seat leakage, but instead by hung open discs. Therefore, this factor is not relevant to the root cause of the check valve backleakage.

d Attachment to TXX-89596 r August 18, 1989 Page 71 of 72 5. Valve Orien'tation EPRI guidelines state thatL swing check valves should be installed in horizontal runs. Check valves in the CPSES AFW System have been installed in horizontal runs. Therefore, orientation is not a consideration in the check valve failures. 6. Pipino Arrangement EPRI guidelines recommend that check valves be located at least 5 pipe diameters downstream of fittings such as elbows and tees and 10 diameters downstream of in-line disturbances such as pumps, control valves, and orifices. The following table lists all of the check valves in the AFW System and their proximity to. upstream disturbances.

1. of Pipe Nearest Upstream Diameters Valve Disc Valve Component In Between Failure Open 1AF-069 Breakdown Orifice 3

Yes Yes3 1AF-057 Breakdown Orifice 3 Yes Yes IAF-045 Breakdown Orifice 6 Yes Yes IAF-075 Flow Orifice 7 1/2 Yes Yes ' 1AF-083 Flow Orifice 7 1/2 Yes Yes IAF-093 Flow Orifice 6 1/8 Yes Yes IAF-101 Flow Orifice 6 1/4 Yes Yes IAF-078 Flow Orifica 4 1/2 Yes Yes IAF-086 450 Elbow 0 Yes Yes Flow Orifice 4 3/4 Yes Yes 1AF-098 Flow Orifice 4 3/8 Yes Yes IAF-106 Flow Orifice 4 1/2 Yes Yes IAF-024 Globe Valve 19 No N/A 1AF-014 Globe Valve 19 Mo N/A h 1AF-051 900 EL - 2 5/8 No H/A Iff-032 Globe V&lve 2 1/3 No N/A 1AF-065 900 EL 3 1/8 No N/A 1AF-038 Enlarger 900 2 3/4 - 2 1/2 No N/A As indicated above, many of the #FW check valves at CPSLS are closer to - upstrean fittings and other devices than recommended by EPRI. The majority of these valves also exhibited backleakage under test conditions and were determined to be hung open as a result of radiographs. However, the Task Team concluded that the proxicity of tM c. hack valves and upstream fittings and devices cas not a factor in the backleakage through the check valves on April l 23 and May 5 Although proximity between the valves, and upstream fittings and 3 devices might result in increaxed turbulence at the check valve, such turbulence would not cause the v&1ve disc to hang up. i

---

,__------.-__-------,--.--------,---------a--.

d' Attachment to TXX-89596 ' August 18, 1989 Page 72 of 72 l Although proximity between the check valves and upstream fittings and devices was not a cause of the AFW check valve leakage on April 23 and May 5, it was a factor in the backleakage through valve IAF-069 on April 19, 1989. As discussed in Section VII.B.1, TU Electric will evaluate whether to increase the distance between check valves and upstream orifices based upon an evaluation being performed by Kalsi, Inc. i i i 37he val /e was n'at radiographer prior to disassembly, but other indications, such as the pump sitction relief valve lifting, indicated the disc was most l. likely open. _. _}}