ML20034F065

Insp Repts 50-327/93-02 & 50-328/93-02 on Stated Dates. Violations Noted.Major Areas Inspected:Regulatory Issues & Followup Issues

ML20034F065
Person / Time
Site:	Sequoyah
Issue date:	02/23/1993
From:	Branch M, Merschoff E NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION II)
To:
Shared Package
ML20034F051	List: ML20034F050 ML20034F060 ML20034F065
References
50-327-93-02, 50-327-93-2, 50-328-93-02, 50-328-93-2, NUDOCS 9303020237
Download: ML20034F065 (67)

See also: IR 05000327/1993002

Text

{{#Wiki_filter:. , ~ _

1. "D

UWIT ED STATES . cO y[o,, 8 ' WOCLEAR REGULATORY COMMISslON [. REGION ll n S ~j 101 MARIETTA STREET.N.W.

f AT LANT A, GEORGI A 30323 %, / , ..... FEB 2 3 ;g93 - Report Nos.: 50-327/93-02 and 50-328/93-02 . Licensee: Tennessee Valley Authority 6N 38A Lookout Place ' ; 1101 Market Street Chattanooga, TN 37402-2801 Docket Nos.: 50-327 and 50-328 License Nos.: DPR-77 and DPR-79 i Facility Name: Sequoyah Units 1 and 2 - Inspection Conducted: January 11, 1993 through January 29, 1993

I i Team Members: L. Keller, Resident Inspector, Summer G. MacDonald, Reactor Inspector, RII

W. Orders, Senior Resident Inspector, Catawba a J. Shackelford, Reactor Inspector, RII ! S. Shaeffer, Resident Inspector, Sequoyah J. Shea, NRR, Division of Reactor Projects.I/II j ,j - , - t [ i A- 1 13-93 ' Team Leader: M. Branch, Chief Date Signed - Test Program Section Division of Rea r Saf ' , , '/

2[hG Approved by: E. Merschoff, D Wec or Date Signed Division of React Projects

I ,

i I . ! .l i 9303020237 930223 PDR ADOCK 05000327

o PDR . ~ , e '- >

'

i

. . l EXECUTIVE SUMMARY r BACKGROUND ! On December 31, 1992, at approximately 9:50 p.m., Sequoyah Units 1 and 2 i experienced a reactor trip from approximately full power due to undervoltage , conditions being sensed by protective relays on the units' electrical boards. i The electrical disturbance was caused by a fault occurring in the plant's l switchyard. The fault caused a loss of most of the distribution for the 500 i KV switchyard and also resulted in a loss of the intertie transformer between

the 500 KV and 161 KV switchyards. (Preferred power is supplied by the 161 KV l switchyard to the common station service transformers (CSST) for both units). ! The 161 KV switchyard experienced a voltage perturbation for approximately 88 i cycles during the electrical transient. This condition resulted in both unit's 6.9 KV shutdown boards isolating from offsite power and all four + emergency diesels generators (EDC) starting and loading onto the shutdown

boards. The licensee made a notification of unusual event (NOVE) based on the

unknown cause of the electrical transients and remained in this condition ! until offsite power was determined to be reliable and the shutdown boards were . transferred back to offsite preferred power. Both units were stabilized in

MODE 3 after the event. ~ ! Initial reviews indicated that all safety systems performed as designed. i However, during recovery of Unit 2 after the reactor trip, the operators, ! hampered by minimum steffing, disabled both high head safety injection pumps .i for approximately 1 minute due to a loss of a suction flow path for the pumps. The licensee subsequently made an additional notification to the NRC based on ! ' the disabling of both of the Unit 2 high head safety injection pumps. Other problems were also identified concerning the thermal barrier booster pumps and ' the Unit 2 CCP auxiliary lube oil pumps. !

Within approximately two hours after the initiation of the event, both units ! had been stabilized in MODE 3 with offsite power having been restored. At [ this time the intertie transformer bank was still out of service. The NOUE ' was exited at this time. The licensee immediately convened an incident ! investigation team to review the event. On January 1, 1993, the licensee ! convened a post trip review safety committee meeting to evaluate the event and j address corrective actions required prior to either unit's restart. The l licensee assembled a separate investigation team to investigate the cause of

the electrical switchyard fault, since it was determined that the electrical l fault was caused by a newly installed breaker that was being tested with its i primary fault protection bypassed. ! ! NRC's FOLLOWUP 0F THE EVENT ' ! Based on a preliminary review of the December 31, 1992, event in combination I with several other recent operational transients it was decided that a special - team inspection would be conducted by Regional and NRR staff. The purpose of i this special inspection was to independently review and evaluate recent ' oper; tional events at Sequoyah. A detailed review of the December 31, 1992, dual it reactor trip was performed. Other reviews performed included two f , addit ,oaal dual unit events and other similar events / transients that may have -l j ' ! -l .-- .. . .

t . + -.- , '

11 i related root causes. The adequacy of procedures, personnel performance, and equipment deficiencies were evaluated for the events reviewed. The accumulated information related to the events was then examined for potential ' generic probable causes to ascertain if any commonality existed. OVERALL FINDINGS A significant weakness was identified with regard to the training of Reactor

Operators (RO) for response to the December 31, 1992 dual unit reactor trip l event. The Unit 2 operators were challenged by limited staffing and had not received training for this situation. This limited staffing directly- contributed to decision errors by the Assistant Shift Operations-Supervisor (ASOS) and resulted in errors and procedure violations by the operator,. resulting in the eventual stopping of both of the coolant charging pumps

(CCPs) which degraded the high head charging system and the ability to protect the reactor coolant pump seals. _ . , The team determined that during portions of the December 31, 1992 event recovery, poor command and control practices (i.e. communication and repeatbacks) contributed to several operator errors. i Management control weaknesses requiring improvements were identified in numerous areas, including training, staffing, switchyard controls and testing of equipment, secondary plant vulnerability to transients, and work control practices. r

! SPECIFIC AREA FINDINGS 1. Training / Staffing and Performance of Control Room Operators ' failure to train the operating crews with the minimum number of _ operators allowed to operate the unit by plant management and allowed by administrative guidance, significantly degraded the operators' ability

to properly recover Unit 2 from the 12/31/92 event (i.e. train.with 2 R0s per unit however, frequently operating with only 1 on one of the two ' ' units). Staffing directly contributed to the Unit 2 excessive cooldown (not controlling auxiliary feedwater flow) and the decision to use

normal boration instead of the required emergency boration. i During the Unit 2 recovery from the 12-31-92 reactor trip, command and control weaknesses, misleading plant status information, and lim!+.ed l staffing appears to have resulted in the operators responding to symptoms instead of the overall event. ! 2. Control of Switchyard Activities , Control of risk activities was not effective and needs to be better j understood by both the plant and the Customer Group (CG). ! Communication between the site and CG load dispatchers was poor. l 3 4

.-. -

- .

. ' iii i The plant's control / understanding of Technical Specifications (TS) and general design criteria (GDC) 17 requirements involving grid stability and the 'importance of the intertie transformer was weak. , lack of detailed testing procedure for the December'31, 1992 event resulted in unnecessarily bypassing many levels of primary fault ~ protection and resulted in the December 31, 1992 switchyard fault being ! wide spread, affecting both units and all emergency busses. , Detailed testing procedures were not incorporated into breaker testing. , The lack of procedural guidance resulted in the bypassing of the anti- ' pump time delay relays. The mechanical damage that resulted from , pumping of the breaker cause the switchyard breaker electrical fault on December 31, 1992, due to dielectric failure caused by particulate , contamination. 3. Material Condition and Controls of Balance of Plant (B0P) Activities Needs Improvement , The Secondary Plant Reliability Study should improve B0P performance if the recommended improvements are evaluated and implemented in'a timely . manner. A large number of B0P work orders (W9s) are scheduled as outage activities and many present plant trip risks. Screening of B0P work orders does not always consider plant trip risks. Management's expectations of system engineers' ownership of systems has ' not been effectively communicated. Troubleshooting / tuning of B0P equipment is not always controlled.

The design / operation of the condensate /feedwater system presents ' operational and maintenance challenges. 4. Material Availability Contributing to Plant Material Conditions i Material availability and parts surplusing contributes to delays in correcting some of the plant material conditions as evidenced by a high number of W0s on material hold (approximately 12%). Sequoyah's management has not effectively controlled this surplusing activity as l evidenced by 13% of the W0s on material hold are a result of surplusing. 1 5. In general, the post trip review and the incident investigation (II) processes are sound. However,.the post trip review for the December 31, ' 1992, dual unit trip was inadequate. i 6. The shedding of the emergency busses during the December 31, 1992 event for a 1.5 second fault, due to an extra level of bus protection (i.e. , 80% voltage relay) which is not described in TS, complicated the plant's 1 ' response.

1 4 . . i

_ l i . , '

1 . l l- iv 7. GDC 5 (shared systems) requirements are adequately described in the ! Updated Final Safety Analysis Report (UFSAR). However, a concern ' associated with emergency raw cooling water (ERCW) system operations and f limitations for some combinations of units / modes and heat loads was identified. Additionally, shared systems in themselves present several

' operational / maintenance challenges, (emergency board preventive maintenance (PMs) and EDG maintenance, as well as CA maintenance). '

REGULATORY ISSUES i Violations of TS 6.8.1 with multiple examples (3) were identified for the failure to follow and/or inadequate procedures. One involved failure to- follow the CG/ plants interface controls relating to switchyard switching orders; one, which may be widespread, involved the failure to perform B0P activities under the plant's work control procedures; and one for a failure to evaluate missing the required calibration of the 80% degraded voltage relays. Apparent violations of TS 6.8.1 with multiple examples (5) were identified for , the failure to follow and/or inadequate procedures. Three of the examples involved the failure to follow Abnormal Operating Instruction A01-34 (emergency boration) during the December 31, 1992 event; one involved an inadequate procedure to control the configuration of the thermal barrier booster pumps (TBBP) switch position which complicated the December 31, 1992 event; and one involved an inadequate post trip review of the December 31, 1992 event. An apparent violation of 10 CFR Part 50, Appendix B, Criterion XVI, Corrective Action, was identified with two examples. One involved the failure to promptly correct problems associated with the lube oil lights for the Unit 2 charging pumps and the other for a failure to correct the inadequate switch configuration for the thermal barrier booster pumps that was identified.during system testing. A weakness in the previous application of 10 CFR 50.59 involving the ' failure to update the TS for the 80% undervoltage relay was identified. The background information reviewed by the team from the 1982 time frame was unclear as to the need for a specific TS for the 80% relay. The team noted that the licensee is currently planning modifications to the existing bus voltage protection scheme making obsolete the 80% relays and an amendment to the TS will be submitted for NRC review as part of the modification package. Additionally, the 10 CFR 50.59 in question was a product of the.1982 era and many improvements have been made to the prccess which should prevent future occurrences of this problem. NRC FOLLOWUP ISSUES An unresolved item involving applicable TS requirements to describe actions for controlling the switchyard intertie transformer status was identified. An inspector followup item was identified for the NRC to evaluate the identified weakness with the shared ERCW system ability to dissipate 2 units heat loads under certain combined modes of operation.

__ , . . . - ' , , , TABLE OF CONTENTS Page EXECUTIVE SUMMARY .......................................... i 1. INTRODUCTION.......................................... 1 - , A. Background.................................. 1 ' B. Team Objectives............................. 1 ' 2. Review Of Recent Events............................... 2 ! 3. DECEMBER 31, 1992 EVENT REVIEW........................ 2 ! A. Event Description........................... 2 B. Sequence of Events.......................... 3

C. Failure of Breaker / Initiation of Event...... 6 D. Review of Operator Staffing and Training.... 8 E. Review of Operator Actions, Equipment, , and Procedures with regard to the Reactor Trips....................................... 11 F. Regul atory Signi ficance. . . . . . . . . . . . . . . . . . . . . 14 4. REVIEW 0F ADDITIONAL OPERATIONAL TRANSIENTS........... 16

5. COMMON OR SHARED SYSTEMS.............................. 17 , A. OVERALL DISCUSSION............................... 17 B. SECONDARY SYSTEM VULNERABILITY / RELIABILITY AND DESIGN LIMITATIONS.......................... 21

C. MATERIAL CONDITION ............................. 22 ! , 1) SWITCHYARD MATERIAL CONDITION............. 22

2) B0P MATERIAL CONDITION.................... -23 ' 6. MANAGEMENT CONTROL AND OVERSIGHT OF WORK ACTIVITIES... 24 , A. CONTROL OF SWITCHYARD ACTIVITIES................ 24 ! B CONTROL OF SECONDARY SYSTEM ACTIVITIES.......... 26

C. WORK ORDERS ON MATERIAL RESTRAINT............... 27 D. REVIEW 0F SYSTEM ENGINEERING.................... 27 , t 7. POST TRIP AND INCIDENT INVESTIGATION PROCESS REVIEW... 28 -l 8. REVIEW 0F RELATED ISSUES.............................. 30

A. UNDERVOLTAGE PROTECTION SCHEME ISSUES........... 30 [ B. PLANT GRID INTERFACE............................ 31- .;

9. PROBABALISTIC RISK ASSESSMENT......................... 33 . i 5 [ . > -- - - . . . - - - - - - - - - - - - - -

, .. .- , 9 L 2 Page 10. OVERALL CONCLUSIONS.................................... 35 ' 11. EXIT INTERVIEW........................................ 39 APPENDICES - FIGURE 1: SEQUOYAH SWITCHYARD DRAWING......... 1 APPENDIX 1: OCTOBER 26, 1992 CONTROL AIR (CA) WATER INTRUSION..................... 1 APPENDIX 2: DECEMBER 15, 1992 DUAL UNIT RUNBACK DUE TO LOSS OF CA................... 1 APPENDIX 3: NOVEMBER 20, 1992 LOSS OF SWITCHYARD INTERTIE TRANSFORMER................ 1 APPENDIX 4: DECEMBER 8, 1992, UNIT 2 RUNBACK DUE ' TO LOSS OF CA....................... 1 - APPENDIX 5: PERSONS CONTACTED................... 1 APPENDIX 6: LIST OF ACRONYMS AND INITIALISMS.... 1. ATTACHMENTS 1: EXCERPTS OF THE 1990 NRC STAFF'S POSITION ON SEQUOYAH INTERTIE TRANSFORMER ........................ 1 ATTACHMENT 2: EVENT REVIEW MATRIX................. 1 s > l t t u , ! < s i e

e ! ' . . . 1. INTRODUCTION , A. Background ! l The purpose of this special inspection was to independently review j and evaluate recent operational events at Sequoyah. A detailed l review of the December 31, 1992, dual unit reactor trip was

performed. Other reviews performed included two additional dual i unit events and other similar events / transients that may have ! related root causes. The adequacy of procedures, personnel < performance, and equipment deficiencies were evaluated for the ! events reviewed. The accumulated information related to the , events was then examined for potential generic probable causes to l ascertain'if any commonality existed. i B. -Team Objectives j . Based on a preliminary review of the December 31, 1992, event in i combination with several other recent operational transients it l was decided that a special team inspection would be conducted by l Regional and NRR staff. The team objectives were as follows: ' , i. Perform a detailed review of recent single and dual units i transients at' Sequoyah for possible followup. ! ii. For the three recent dual unit transients and any other i

significant transients identified above perform a detailed -review of the events and assess personnel performance, j equipment response, and procedure adequacy. Additional

reviews to determine recurrence controls will also be

conducted.

iii. For the December 31, 1992, switchyard initiated, 2 unit I reactor trip, perform the following additional reviews: l t a. Develop a detailed sequence of events. l } b. Evaluate plant response to the event. l - Equipment performance

- Operators performance

- Procedures performance j iv. For the significant transients. reviewed above evaluate Management controls of plant and switchyard activities. , ! v. Review the IPE/PRA results for Sequoyah as they pertain to

.. the reviewed events and evaluate the plants use of the PRA. vi. Review the use of shared systems at Sequoyah and evaluate i the methods and effectiveness of work controls for 80P

activities. l I ! l !

' . , l .

' ' ! 2 , 2. REVIEW 0F RECENT EVENTS l The team conducted a review of Ils and plant LERs for the period of l August through December 1992, for possible followup. A matrix detailing the initial review findings was developed by the team and is included as Attachment 2 to this report. As noted at the bottom of Attachment 2 the , team did not review recent configuration control events as they are ' being evaluated as part of other regulatory actions. The team did note; however, that configuration control appears to be a continuing problem ! at Sequoyah and management actions need to be focused to correct the s identified problem. The team independently evaluated the various events to determine the adequacy of procedures, personnel performance, and equipment. In addition, root cause determinations and licensee , corrective actions were also evaluated for effectiveness. Subsequent ' areas in the inspection report utilized the accumulated information related to the reviewed events to determine if any commonality existed. Other events reviewed are appended to this report. The details of the

December 31, 1992, event is described in paragraph 3 of this report. , ! ! 3. DECEMBER 31, 1992, EVENT REVIEW [ A. Event Description , ! On December 31, 1992, at approximately 9:50 p.m., Sequoyah Units 1 ! and 2 experienced a reactor trip from approximately full power due

to undervoltage conditions being sensed by protective relays on ! the units' electrical boards. The electrical disturbance was ! caused by a fault occurring in the plant switchyard. The fault

caused a loss of most of the distribution for the 500 KV ' switchyard and also resulted in a loss of the intertie transformer . between the 500 KV and 161 KV switchyards. (Preferred power is ! , ' supplied by the 161 KV switchyard to the common station service ! transformers (CSST) for both units). The 161 KV switchyard i experienced a voltage perturbation for approximately 88 cycles

during the electrical transient. This condition resulted in both

unit's 6.9 KV shutdown boards isolating from offsite power and all

four emergency diesels starting and' loading onto the shutdown

boards. The licensee declared a NOUE based on the unknown cause ! of the electrical transients and remained in this condition until

offsite power was determined to be reliable and the shutdown

boards were transferred back to offsite preferred power. Both units were stabilized in MODE 3 after the event. Initial reviews indicated that all safety systems performed as designed; however, during recovery of Unit 2 after the reactor i trip, the operators disabled both high head safety injection pumps for approximately 1 minute due to a loss of a suction flow path for the pumps. The licensee subsequently made an additional ! notification to the NRC based on the disabling of both of the Unit i 2 high head safety injection pumps. Other problems were also ! r 2 i r ? , - _

_ . r ~ . . , . f 3 identified concerning the thermal barrier booster pumps and the ' Unit 2 CCP auxiliary lube cil pumps. Details of the above issues l are addressed later in this report. At approximately 11:48 p.m., Both units had been stabilized in MODE 3 with offsite power having been restored. At this time the . intertie transformer bank was out of service. The NOUE was exited i at this time. The licensee immediately convened an incident ! investigation team to review the event. On January 1,1993, the l licensee convened a post trip review safety committee meeting to ' evaluate the event and address corrective actions required prior to either unit's restart. A separate investigation team was

assembled to investigate the cause of the electrical switchyard fault. > On January 2,1993, the licensee inspected a newly installed breaker which connects the 500 KV switchyard to the high side of the intertie transformer (PCB 5058) and determined that this l breaker had failed and caused the above fault condition and } subsequent dual unit reactor trip. Additional reviews of the licensee's evaluations to determine the failure mechanism of the l breaker are discussed later in the Inspection Report.

On January 2, 1993, the 500-161 KV intertie transformer bank was ! returned to service. Unit I was restarted on Janurary 3, 1993, I , and returned to approximately 100 percent power on January 6, ! 1993. l

On January 2, 1993, Unit 2 was restarted and reached MODE 1 . operation on January 3, 1993. However; due to concerns regarding i the design logic for the RWST and VCT suction supply valves for the CCPs, the unit returned to Mode 3 operation to perform testing 1 of these components. After satisfactory testing was accomplished, j the unit was again restarted on January 9 and returned to ' approximately 100 percent power on January 11.

1 , B. Seouence of Events j t - The following sequence of events was developed from information 1 gathered from control room logs, data printouts, individuals j involved in the event, and from reviews of the licensee generated i sequence of events.

- t DATE / TIME ITEM DESCRIPTION l November 20, 1992 PCB 5054 fails. PCB 5054 was repaired and [ returned to service by utilizing heads

from PCB 5064. (see figure 1) - ! November 23, 1992 Decision was made to replace PCB 5058 with ' a new breaker. Heads from the original

h ' ,

,_ i, e ' . 4 4 PCB 5058 would be used on PCB 5064. December 13, 1992 PCB 5058 taken out of service. , December 14, 1992 TVA management (site and customer service)- discuss precautions, schedule, and , possible problem areas involved with the installation of the new PCB 5058. . December 17, 1992 Site preparation in the 500 KV switchyard- l begins for new PCB 5058. ' December 31, 1992 PCB 5058 installation complete; tests and , inspections conducted throughout the day. , i 3:00 p.m.(EST) Shift turnover. One of two R0's for unit 2 calls in sick; no replacement was provided. 7:20 p.m. Disabled primary protective ' relays i receiving currents from PCB 5058 current

transformers in order to verify the ' ! phasing on the relays. Protection for the intertie bank during this period was provided by secondary relays. 9:37 p.m. PCB 5058 was closed. Began phasing of the

relays. 9:48:32 C-phase to ground fault occurred on PCB ~ , 5058. The fault results in the feeder breakers for the 500 KV switchyard opening. For a short period of time the 161 KV switchyard backfed, via the intertie transformer, into the otherwise , isolated 500 KV yard causing a perturbation on the 161 KV switchyard. . 9:48:32-33 The switchyard transients resulted in both ! units tripping on RCP undervoltage. RCPs ! remain running on both units. - 9:48:33 The fault was cleared by transformer #5- l (intertie transformer) .161 KV winding ! ground relay, which was the appropriate ! relay operation with the primary relays . disabled. The total duration of the fault

was 88 cycles (approximately 1.5 seconds).

' The normal feeder breakers for the shutdown boards on both units opened as expected due to the operation of the 80%

_. . . . . 4 5 loss-of-voltage relays, this resulted in both shutdown boards for both units being deenergized. All EDG's subsequently , started and provided power to their

respective shutdown boards as expected. ! Load shedding and sequencing occurred per design; with the exception of the thermal . ' barrier booster pumps, which did_not , sequence back on due to their handswitch ' position. Letdown isolates on both units. Unit 2 R0 does not notice that. letdown isolated. ! '! The following sequence details additional events which occurred on Unit 2. No significant recovery problems on Unit I were encountered. ' , 9:49.08 Both Unit 2 CCPs start automatically due ' to load sequencer (only one in service prior to trip); VCT level starts , decreasing. t ' 9:55 - 10:00 p.m. Plant cooldown occurs below 540 degrees F. Procedures require emergency boration (AOI 34). ASOS ordered normal batch borate i rather than required emergency borate.

ASOS directs R0 to take manual control of MDAFWP to control RCS temperature. 10:00 p.m. Licensee declared an unusual event in accordance with criteria in their emergency plan, due to loss of 500 KV !' switchyard and both reactors tripped. , 10:08:03 VCT level drops to 7%; CCP suction valve , swapover to RWST occurs. Both CCP's still J running, letdown still isolated. I

Due to indication of low lube oil pressure on both of the operating CCPs the ASOS directs R0 to place "B" CCP in PTL, SOS i ' aware of decision. Thermal barrier - booster pump was recognized as.not being reconnected after load shedding. 10:09:41 Letdown was re-established. ) 10:10:19 VCT low level alarm clears (>13%). ASOS i directs realignment of CCP's to VCT. 10:11:18 LCV-132 & 133 opened at approximately 18 ) _ _

. _ . . ! < . a , t 6

percent VCT level (suction from VCT). ' 10:11:35 RWST valves were manually closed by the , operator. l t 10:11:36 VCT valves - 132 & 133 start to close, RWST valves do not reopen. 10:11:49 ASOS sees RWST/VCT valves closed (green lights on MCB) and directs RO to stop "A" [ CCP. "A" CCP placed in."stop", both CCP's " now secured, entry into LCO 3.0.3. -. Approximately 20 seconds after the "A" CCP

was stopped, the SOS started the TBBP's for both units. 10:12:03 VCT valves 132 & 133 were manually ' reopened by the operator. - 10:12:51 "A" CCP started. Charging' system j subsequently stabilized and TS 3.0.3 was- , exited.

11:13 p.m. Shutdown boards were realigned to offsite

power. i 11:48 p.m. Licensee exited unusual event. t C. Failure of Breaker / Initiation of Event ! FAILURE OF PCB 5058 Subsequent to the failure of PCB 5054 on November 20, 1992, as ' described in Appendix 3, TVA decided, due to obsolescence and lack of spare parts for the Hitachi breakers, to replace PCB 5058 with - a new ABB breaker which had originally been purchased for use in l another TVA 500 KV switchyard. Modification No. DCN M09079A

consisted of removing the existing Hitachi air blast breaker, i installing the new ABB SF6 gas breaker, and connecting the power, control and annunciator circuits. The new ABB PCB was built for 125 VDC control power and it was field modified for Sequoyah's 250. , VDC control scheme. The modification and breaker testing was complete and . switching had been performed to energize the breaker to perform final relay , phasing checks. All four of the primary protective relays were

simultaneously bypassed for phasing checks by. opening.their i respective trip cutout switches. During relay phasing,

approximately 12 minutes after the breaker was energized, a phase j to ground fault of approximately 20,000 amps occurred on C phase. Since all primary protection was bypassed the fault lasted 88 i cycles until the backup relay protection cleared the fault. , j l -. - .

- . . . . , ! ! - l .

! ! 7

Breakers tripped at five other TVA switchyards and SQNP PCB 5074 [ tripped on a pilot ground signal to isolate the 500 KV switchyard. i Unit 1 generator breakers (PCBs 5034 and 5038) and Unit 2 generator breakers (PCBs 924 and 928) tripped. PCBs 5054, 5058, 934, and 938 tripped on backup ground protection to isolate the 500-161 KV intertie transformer. The 161 KV switchyard fed the fault through the intertie transformer and 161 KV switchyard ' voltage dropped to approximately 62% (100KV) during the fault j transient which lasted for 1.5 seconds. Low 161 KV switchyard i voltage caused low voltage at the 6.9 KV shutdown boards and the i 80% 0.5 second undervoltage relays actuated. ! The team reviewed the event data with the licensee and concluded that relay and breaker operation was normal for the given j

• -

conditions. However, had any of the four primary protective . relays remained in service, the fault would have cleared in about j 3.5 cycles and most likely would not have resulted in a trip of i either generating unit. ! > - ' After the breaker failure occurred the licensee obtained support from the vendor. Vendor representatives inspected the failed C

phase pole unit in the field and requested that the license ship l , the failed unit to the factory for a detailed failure analysis. , The team inspected the failed C phase breaker pole unit and -{ interviewed licensee personnel to determine the possible causes of

the fault. Internal examination of the failed C phase indicated

< arcing damage between the closing resistor assembly and the i , breaker tank. A white powder coating of arcing by-products was } I present on the internal assemblies and tank bottom. The arcing { caused molten aluminum splatter and some damage to the tank

surface in the area of the closing resistor primarily concentrated l , on the bottom tank surface. The greatest damage was between the i rear closing resistor shield and the tank. The shield supports i , ! and a portion of the shield surface was burned off. There was j minor damage to the closing resistor. linkage and guides.

There was very little observable damage to the externals of the breaker. The team observed some black arcing deposits at the joint between the tank and the frame. . l ! l The gas pressure in the failed C phase unit after the event was j still normal indicating that the failure was not due to lack of sufficient dielectric. Moisture tests performed prior to l - energization were satisfactory. The team reviewed the post i failure gas analysis results and confirmed that there was no l 1 evidence of dielectric contamination on either A or B phase. The l results showed the presence of arcing by-products in C phase. CG 1 electricians reported a strong sulphur smell while obtaining gas i samples from the C phase pole unit. Hydrogen sulfide is an arcing by-product in an SF6 gas breaker. ! ! f t ! ! I i

~-. . -. = - . ! !

.

. , .

i ~ 8 i i Review of the vendor test results prior to shipment indicated that

the breaker satisfactorily passed all acceptance tests including l high potential conditioning to 860 KV. i The team reviewed the post installation testing performed by the

licensee and determined that the licensee had difficulty meeting ! breaker timing requirements. Measured breaker timing was longer i than vendor requirements. The licensee contacted the vendor and t determined that the specified timing only included breaker

interrupting time and closing / trip coil energization time. ' Auxiliary relaying time was not included in the timing .i specification. The licensee modified the breaker timing test set circuit connection points to allow the test set to operate the , breaker without the auxiliary relays. The breaker anti-pump 20

cycle time delay relay was bypassed by this testing. Vendor ! drawing No. 996D114 sheet 1 Rev. 4 provides a caution note that ' states "Do not attempt a closing operation without a minimum 20

cycle delay after a tripping operation. Breaker damage may ' occur." l The team inspected the B phase pole unit and observed significant ! mechanical damage to the closing resistor guide and linkage. The l team concluded that the probable cause of the C phase breaker j

fault was dielectric failure caused by particulate . contamination. l The particulate contamination was most likely caused by mechanical ' ' damage to the closing resistor assembly due to rapid cycling j during breaker timing testing. i

The licensee performed post fault equipment checks.and began - )

switchyard restoration. A visual inspection of the switchyard l indicated that no additional equipment had sustained damage. Oil j samples from the intertie transformer were analyzed and indicated , ' no internal fault had occurred. The team inspected equipment in -!

the 500 KV and 161 KV switchyards including the intertie and i common station service transformers and found no damaged i equipment. The protective relays and their cabinets were- l inspected and no abnormalities were noted. The team concluded

that licensee activities to determine the cause of the fault and ' the restoration of the switchyard were appropriate. ! , ' D. Review of Operator Staffino and Trainino l f The inspectors reviewed the December 31, 1992, event with regard l to operator training and control room staffing. Technical - Specification (TS) 6.2.2 requires that each on duty unit shift be

composed of at least the minimum shift crew shown in Table 6.2-1.

This table requires the following minimum shift crew composition

for operation of both units in MODES 1,2,3, or 4: j t f

, z

. - __

. a \\ ' 9 e e ! Position Number of Individuals Reauired To Fill ' Shift 1 (*) Supervisor (SOS) - ' Senior Reactor 1 (*) Operator (AS05) Reactor Operator 2 (**) Auxiliary Operatur 2 (**) l Shift Technical 1 (*) Advisor ' (*) - Individual may fill the same position on opposite unit ' (**) - One of the two required individuals may fill the same position on opposite unit. As of November 1992, operation staffing guidelines were to schedule four Ros per shift; however, RO staffing was allowed to ! return to the minimum TS required staffing level of three R0s if staffing the fourth R0 required overtime usage. Subsequently, in late November 1992, Operations Management initiated changes which

directed that shift staffing would always include four R0s with the only exception being if forced overtime was needed. This exception was also contingent upon the availability of a RO l trainee to staff the fourth R0 position in a training capacity ' under the direct supervision of the licensed unit personnel. SSP 12.1, Conduct of Operations,. Revision 2, Section 3.1.1.I, authorizes the trainee to perform restricted CR functions. In ! this position (as the fourth RO), the utilization of the trainee during a unit transient was also restricted by the licensee.

Other changes made by Operations Management in late November 1992 included guidance to staff four ASOSs for each shift. These were i the guidelines in place up to the time of the event, j -i The R0 staffing during the December 31, 1992, event consisted of

two R0s on Unit I and one R0 on Unit 2. This staffing level l resulted due to one R0 calling in sick. Per the aforementioned l Operations guidance, an RO trainee filled the fourth R0 position. i The inspectors reviewed the use of the R0 trainee during the ! response to the Unit 2 reactor trip. It was identified.during the j review that the trainee was utilized to reduce the MDAFW pump flow i under a licensed operator supervision; however, the inspectors ' concluded that the R0 trainee was not involved in the unit i recovery evolutions until the unit conditions were adequately l ! ,

_._ _ . _. __ _ __ .. _ _ _ -- ' , - 1

. i 10 ( } stabilized some period cf time after the event and after'the j operators had transitioned out of the emergency procedures.

The inspectors also reviewed the effects of the Unit 2 minimum RO l staffing level during the event with regard to the operator's i ability to stabilize the unit after the reactor trip occurred. l The inspectors concluded that the minimum level of staffing ! directly delayed the recovery of the unit and challenged operators i bey.and their provided training. This conclusion was based on

discussions with the involved operations personnel, review of logs > and component parameters, and review of the applicable Emergency l Response Instructions. Routine operations training is performed i with two R0s on shift for the simulated accident scenarios. Due i to the training not incorporating the possibility of frequent

utilization of minimum staffing levels (one RO), the Unit 2

operators (one R0 and one ASOS) were not well prepared to handle f the event. Due to delays associated with the one R0 having to perform primary and secondary system recovery evolutions, as l required by the applicable procedures, an RCS cooldown resulted.

The inspectors and the licensee concluded that the cooldown below i 540 degrees on Unit 2 was a direct result of operators not l adequately controlling auxiliary feedwater after the reactor trip.

A second example of how minimum staffing contributed to unit recovery problems invoked the ASOS's decision to normal batch borate in response to the cooldown versus emergency borate as j required by A01-34, Emergency Borate. Although the operators did' l not follow the requirements of A0I-34, the decision not to ' emergency borate, in part, was made due to the ASOS's reluctance i to commit the one R0 to the emergency borate process, which l requires the operators attention while in progress. The staffing " of one R0 contributed to this decision, in that by choosing the

' normal boration path, the R0 was not restricted to munitoring only that activity. In addition, the inspectors considered the ES-0.1 procedure weak, in that it instructed the operators to borate per e A01-34. For clarity to the operators, the procedure should state

to " emergency borate". Further details regarding specific operator error and other identified deficiencies are detailed later in the Inspection Report. Although the TS required staffing was met, the inspectors concluded that training encompassing the operating staffing levels

at the time of the dual unit trip event was inadequate. Under the r more frequent single unit transient, minimum staffing would not be a problem as the second RO on the unaffected unit is instructed to assist in the recovery of the affected unit. However, for the 3 December 31, 1992, event which involved a dual unit reactor trip, J training for the manning of three R0s in the CR was inadequate and j affected the timeliness of recovery actions and increased the j safety challenges to the unit. This subsequently contributed to ? , an utnecessary RCS cooldown, an inappropriate decision not to ! ' emergency borate in response to the cooldown, and posed undue -l ! I I

. . L

_ _ ~ _ _ __ . _ __. _ __ __ . _ _ _ - ._ , . ~ '

l r 11 ! i challenges on the operators in recovery of the unit. ! , The _ inspectors reviewed R0 staffing levels for the month of December 1992. The results indicated that there were 12 shifts !' ' where one unit was staffed with only one R0. All but one of these shifts was supplemented with an RO trainee; however, as stated

before, the trainee was prohibited in assisting in response to an i event, and therefore could not assist in critical recovery . , evolutions. The inspectors expanded the review to include the months of September, October, and November 1992. During these months, 41, 31, and 24 shifts, respectively, were staffed with one ' R0 on one unit. The inspectors concluded that the licensee ! frequently staffed the CR with the TS minimum required levels and ! that the licensee's training program did not reflect this common ' staffing practice. The inspectors identified this as a significant weakness. The inspectors also reviewed the current number of R0 licensed i personnel at Sequoyah in conjunction with the routine usage of l overtime. At the time of the event, 34 R0 licenses existed; l however, 26 were actually performing on shift duties. ! Subsequently, with the licensee operating on six rotational

shifts, there are approximately 4 R0s per shift. The inspector , concluded that this per shift staffing accounted for the routine

reduct..,n to minimum staffing levels (3 R0s). A review of ! 9 overtime usage was performed for approximately a one year period ! l prior to December 1, 1992. The review concluded R0 overtime usage i was fairly routine. The licensee also agreed that there was a l routine use of overtime for the R0s. TS 6.2.2.g states that l , j adequate shift coverage shall be n.aintained without the routine ! use of overtime. The objective shall be to have operating

personnel work a normal 8 hour day, 40 hour week while the units ! are operating. The inspectors concluded that, as of December 31, i 1992, the 26 R0 assigned to shifts was acceptable; however, at , this staffing level, the objective of the TS overtime limits was !

difficult to satisfy. The licensee informed the inspectors that

currently, six R0s were undergoing license R0 training, with a

projected graduation date in the Spring of 1993. This additional

staffing should alleviate some of the concerns addressed above. I E. Review of Operator Actions, Eauipment, and Procedures with reaard i to the Reactor Trips i . The inspectors reviewed operator response to the event for both Unit I and Unit 2. The actions taken by the operators on Unit I appeared to be appropriate. As described above, Unit 2 was i staffed during the event with one R0 and one ASOS. This condition contributed to a failure to adequately control plant parameters , and a failure to follcw an emergency boration procedure. In the ! following paragraphs, it was concluded that the Unit 2 operators inadvertently closed safety-related valves and mispositioned CR j l , i ! t ~j , >

.

, . . 12 o handswitches such that the required control functions were inhibited. These actions ultimately resulted in the degredation of all high head injection flowpaths for a short period of time. ,

The electrical failure which initiated the dual unit transient was

previously discussed in section 2.C. of this report. During the event recovery, the Unit 2 ASOS acted as the emergency procedure reader, and directed the one R0 to stabilize the unit. Approximately one minute after the reactor trip, Unit 2 operators entered Emergency Instruction E-0, Reactor Trip or Safety injection, Revision 12. After establishing that a safety injection was not required, the crew transitioned to ES-0.1, Reactor Trip Response, as allowed by the procedure. Upon entry in , to ES-0.1, at approximately two minutes after the reactor trip, . the procedure (step 2) requires the monitoring of T-ave to t preclude an excessive cooldown (below 540 degrees F). At this i time, T-ave was noted as being approximately 548 degrees F and , decreasing. Concurrent with these actions, the one R0 initially took manual control and backed down the TaAFW pump. The MDAWFPs were not throttled back at this time due to the R0 performing other actions per the applicable procedure. In addition to the

above, the Unit 2 ASOS noticed that the thermal barrier booster pumps (TBBP) were not running; however, he elected not- to start the pumps due to an electrical loading concern for the running , EDGs. It should be noted that all four TBBPs, two per unit, were not running. At approximately 7 minutes into the event (9:55 p.m.), as required by step 4 of ES-0.1, the Unit 2 operator again referred to Tave and identified that the RCS temperature had decreased to below the 540 degree F procedure limit to approximately 537 degrees F. With this information, the ASOS returned to step 2 in the procedure which required the entry into Abnormal Operating Instruction (A01) 34, EMERGENCY B0 RATION, due to the cooldown condition. A01-34 requires that the operators emergency borate due to the cooldown condition. Utilizing A01-34, the ASOS determined that a boration of 135 gallons w ,s required. However, due to some unfamiliarity with the procedure and a reluctance to commit the only R0 to the emergency boration process, the ASOS directed the R0 to normal , borate tnrough the blender at a rate greater than 10 GPM. After i the normal boration was initiated, the R0 took manual control of the MDAFW pump to reduce the RCS cooldown. Choosing the normal boration flowpath, which was contrary to procedure, ultimately contributed to a VCT level reduction. This was due to letdown being previously automatically isolated and, by proceeding with the normal boration flowpath, the automatic VCT . makeup capability was inhibited. Subsequently, at approximately I 20 minutes into the event the drop in VCT level below 7 percent caused an automatic swapping of the suction supply to the CCPs from the VCT to the RWST. Due to electrical load shed and < sequencing, the one running CCP stopped at the initiation of the ,

. - . . - . _ - .. -- -. .- ._ - - - . ! .

. . i 13 ! e ' event which caused letdown to isolate. Subsequently, both CCPs started due to the sequencing of the pumps onto the shutdown , boards as designed. The operations crew initially did not l realized that letdown was isolated. However, after the RWST VCT ' ' swap occurred, the ASOS realized that letdown was isolated. After verifying normal pressurizer level, the ASOS directed the R0 to i stop one of the two operating CCPs due to an indication of low

lube oil pressure for both of the operating CCPs. However, since i the blackout relays were still actuated, the pump handswitch had

' to be placed in the pull to lock (PTL) position. Problems with ! the indication of low lube oil pressure are discussed later in , this report. The inspectors reviewed the placing of the t handswitch to the PTL position and determined that the disabling , of the 2B-B CCP was within the guidance of the licensee's , emergency operating instructions; however, a review of the

operator logs did not reveal that the appropriate LC0 was entered ! at the time due to ongoing operator activities. > Following the above evolutions, letdown was established by the R0.

During discussions with the operators, the inspectors became aware ' that the letdown was established by utilizing procedural guidance- ! d however, the governing procedure, ES-0.1, did not require the i establishment of letdown at this time. The inspectors considered ! the operator actions to establish letdown appropriate; however,- > they recognized that the evolution was performed outside of

procedural guidelines. By 10:11 p.m., the R0 and the ASOS had verified that the VCT level , had recovered to 18 percent and a decision was made to swap the - ' a e running CCP suction back to the VCT from the RWST. The RO opened the VCT valves (2-FCV-62-132 and 2-FCV-62-133). Available ' data indicated that the valves reached the full open position at 22:11:18 hours. After the R0 saw only the red full open light and ! checked CCP parameters, he closed the RWST valves (2-FCV-62-135 l . and 2-FCV-62-136) at approximately 22:11:35. However, as later 1 determined, the operator left the RWST handswitches in the A-Auto .' position, rather than the A-P Auto position, thereby defeating the ! automatic re-opening of the valves an a low VCT level. Per l available data, the VCT valves were then indicating a closing , (less than 95 percent full open) at 22:11:36. It was later . . determined that the R0 inadvertently closed the VCT valves in

error. When the RO noticed both the red and green lights on the

i handswithches for the VCT valves, he alerted the AS0S that all i suction to the operating 2A-A CCP was about to be lost. The R0 ! then held the 2A-A CCP in the off position (the 28-B was already

in PTL) and letdown was again automatically isolated. With no l high head safety pumps operable, the licensee entered TS LC0 3.0.3. Approximately one minute later, the R0 had reopened the t - VCT suction supply valves and the operator subsequently restarted i the 2A-A CCP. The total period of time both CCPs were inoperable l was later determined to be one minute two seconds.

i , - l 1 . t

. , . -- . . . ,

t

i ' '

- 4 14 , ! Also during the period of time that both CCPs were inoperable, the ' operators recalled that all four thermal barrier booster pumps l (TBBP), two provided for each unit, were not running. This, in i combination with no charging pump / seal injection flow, resulted in a total loss of all thermal barrier cooling / seal injection for a ' period of approximately 21 seconds. This condition existed due to the momentary loss of offsite power event de-energizing the i running TBBPs. Consequently, upon re-energization of the shutdown ! boards by the EDGs, the TBBPs failed to start as designed due to . an improper TBBP handswitch position. Once the last CCP was . stopped, actions were taken to immediately start the TBBPs. The

licensee later determined that the TBBP handswitches were in the j wrong position due to procedural inadequacies. The procedural

problems are addressed later in the Inspection Report. The , significance of reactor coolant pump operation without the TBBPs l is further discussed in the paragraph 9. ! F. Reoulatory Sionificance ! The inspectors reviewed the regulatory significance of the events

of December 31, 1992, which were involved and/or contributed to j the degradation of the high head safety injection capability. l - During the recovery of Unit 2 after the reactor trip, the operators failed to follow the requirements of A01-34, - Emergency Boration, Revision 7. The purpose of this

procedure is to provide the necessary actions to initiate

emergency boration when the reactor is shutdown. The

procedure is entered, in part, as a response to an RCS i cooldown below 540 degrees F. Contrary to the above, on December 31, 1992, operators failed to follow the , requirements of A01-34, in that, a normal boration flowpath + was utilized, whereas an emergency boration flowpath should , have been established. The team concluded that this

ultimately contributed to a VCT level reduction and an ! t automatic swapping of the suction supply to the CCPs from the VCT to the RWST.

- A01-34, Emergency Boration, Revision 7, provides ! instructions to operators to adequately align and restore ! components utilized in normal charging operations and l boration flowpaths. This includes steps to return CCP - suction to the VCT from the RWST source. Contrary to the i above, during the recovery of Unit 2 after the reactor trip,

the operators did not utilize and failed to follow the ' requirements of A0I-34 during restoration of the VCT as the suction to the operating CCP, in that, the RWST i handswitches, 2-HS-62-135 and 2-HS-62-136, were not pulled i to the A-P AUTO position after manipulation. This resulted

in disabling of the process function to automatically open j the RWST suction supply valves on a closure of the VCT l suction supply valves. ! ! .

.. . ' . 4 15 - A01-34, Emergency'3 'ation, Revision 7, provides

instructions to operators to adequately realign and restore components utilized in normal charging operations and securing boration flowpaths. This includes steps to return CCP suction to the VCT from the RWST source. Contrary to. the above, during the recovery of Unit 2 after the reactor trip, the operators did not utilize and failed to follow the requirements of A01-34 during restoration of the VCT as the suction to the operating CCP. This resulted in the operator inadvertently closing the VCT outlet suction supply valves, 2-LCV-62-132 and 2-LCV-62-133. At the time, these valves were providing the suction supply to the operating CCP. This action resulted in a loss of all suction to the CCPs and precipitated the operators to stop the last running CCP. This action caused a degradation of the high head injection . ' system flowpath for approximately 62 seconds. < - Tennessee Valley Authority Design Criteria for the Component Cooling Water System, SQN-DC-V-13.9.9, specifies that the thermal barrier booster pumps (two per unit) shall be loaded to the diesel generators simultaneously with the Component Cooling System pumps after a loss of offsite power. Sequoyah UFSAR Section 9.2.1.3.3, Thermal barrier Booster Pumps, states that the purpose of the TBBPs is to provide the- additional head necessary to overcome high head loss through the thermal barriers. Each of the four pump motors receives electric power from normal or emergency. sources and is connected to one of the four shutdown boards. Placement of the control room thermal barrier booster pumps control switch in the A-P AUTO position allows the above requirements to be met. System Operating Instructions 1-S0-70-1, Component Cooling Water System - A Train, Revision 3 and 2-S0-70-1, Component Cooling Water System - A Train, Revision 1, provides the procedure requirements for the thermal barrier booster pump control room handswitch positions. Contrary to the above 1-S0-70-1 and 2-S0-70-1 were inadequate, in that, the procedures required the 1 subject handswitches to be in the A-AUTO position instead.of ~ the A-P AUTO position. This resulted in a failure of the thermal barrier booster pumps on both units to start as required on December 31, 1992. .i The above four regulatory issues involve a failure to follow the ' requirements of TS 6.8.1 and are identified as examples of ' Apparent Violation 327,328/93-02-02. i - A condition adverse to quality had existed for several years and was not promptly corrected, in that the 2A-A thermal , barrier booster pump failed to start as required by the-test

procedure due to an improper switch alignment. As a result .)

. ' . . 16 of the failure to promptly evaluate and correct this deficiency, on March 14, 1992, during the performance of 2- SI-0PS-082-026.A, " Loss of Offsite Power with Safety t y Injection-DG-2A-A Containment Isolation Test", the hand 1 % switches for all four thermal barrier booster pumps (2 per

unit) remained inappropriately aligned. Subsequently, on [ December 31, 1992, a momentary loss of offsite power event . occurred and the thermal barrier booster pumps failed to 1 automatically restart as designed. This condition resulted

in a loss of thermal barrier cooling water to the reactor ! coolant pumps. This condition was exacerbated in Unit 2 due

to operator actions which had also secured reactor coolant pump nal injection flow due to problems' associated with the

VCT suction valves. These conditions resulted in total loss ' of all thermal barrier cooling coincident with seal injection for a period of approximately 21 seconds. ' - A condition adverse to quality for the Unit 2 charging > pump's low lube oil pressure lights has existed since . October 1990. This condition has resulted in spurious low 'I lube oil pressure light actuation during normal operations. ' On December 31, 1992, following a loss of offsite power j event, the low lube oil pressure lights for.both Unit 2 charging pumps actuated which contributed to the control

room operators placing the 2B-B charging pump in pull-to- . , lock, which in turn, complicated the recovery from the t ' event. The above two regulatory issues involve a failure to follow the - requirements of 10 CFR 50 Appendix B, Criterion XVI, Corrective , Action and are identified as examples of Apparent Violation ) 327,328/93-02-03. i In addition to the above regulatory issues and the previously discussed training and staffing weaknesses, the inspectors

concluded that the command and control function during the Unit 2 i operations crew response to the reactor trip was weak. This was

concluded from reviews of the applicable procedures utilized,

interviews with operators, and review of the licensee's evaluation of the event. Specifically, the inspectors concluded that had proper procedure usage been maintained in combination with command repeat back of control board manipulations, the operator errors , involving the mispositiening of equipment likely would not have i occurred. The use of repeat back is routine for operators training on the plant simulator; however, this is a further example of a condition in which the licensee does not necessarily .; train in the same manner in which they operate. 4. REVIEW 0F ADDITIONAL OPERATIONAL TRANSIENTS i As stated in paragraph 1.b.ii above the team also evaluated other !

, t , ' . .t 17 operational transients at Sequoyah that have occurred within the last 5 months. The details of the reviews are contained in Appendices to this , report. The review of the events indicated additional areas that

required pursuing by the team. The subsequent paragraphs summarizes the i details, concerns, and conclusions of the teams findings. l ! 5. COMMON OR SHARED SYSTEM REVIEW ll A. OVERALL DISCUSSION ' The team reviewed selected systems to determine if the requirements of - 1 General Design Criteria 5 (GDC 5) were properly implemented at the

station. General Design Criteria 5 describes the requirements for sharing of structures, systems and components and states: " Structures, systems, and components important to safety shall not be shared among nuclear power units unless it can be shown that , such sharing will not significantly impair their ability to perform their safety functions, including mitigating an accident in one unit, and an orderly shutdown and cooldown of the remaining i units." - i The team also reviewed shared systems design and operations for impact ' on reliable dual unit operation. The SQN UFSAR describes the compliance of the SQN design with GDC-5. The UFSAR identifies the emergency raw cooling water (ERCW), component cooling water (CCW), fire protection, spent fuel cooling, fuel oil storage tanks, preferred and emergency electric power, chemical and

volume control, condensate, radioactive waste and control and auxiliary F building ventilation systems as shared, safety related systems. In

addition, the DC Power System is shared to the extent that the vital inverter in one unit are energized by the DC power channels assigned ' primarily to power loads of the other unit. In addition, the team reviewed the licensee's IPE to assess the

importance of shared systems. In general, support system faults account ' for approximately 48 percent of the total projected core damage . frequency at the Sequoyah facility. Among the primary support systems, ' ; shared systems comprise the largest contributors in terms of overall risk importance measures. The licensee's recent IPE analysis had recognized these facts and provides useful insight into the importance j of shared support systems at the Sequoyah facility. Among the most ' important shared support systems are: i) Component Cooling System, ii) Electric Power, iii) Essential Raw Cooling Water, and iv) Ventilation l Systems. l CONTROL AIR SYSTEM: j i The non-safety related portion of this system is a shared system common to both units. The compressed air system at SQN is comprised of several interconnected systems; the Station Control and Service Air (non ,

.. . . _ _ _ _ _ _ _ _ . ' .. d 18 essential) system and the Auxiliary Control Air (essential air) systems. System Description four service and control air (SCSA) compressors provide instrument and control air to safety and non-safety related components and service air for station service air needs. A separate service air compressor provides air only into the service air portion of the system. The SCSA air is distributed to essential and non-essential headers. Non- essential air headers serve non-safety related components and safety- related components that fail to a safe condition upon a loss of air pressure. Essential air headers serve safety-related components. The essential air header isolates from the SCSA and non-essential portions of the system upon a loss of control air pressure. Two auxiliary air compressors are installed to supply air to the essential air components upon a loss of non-essential air or the SCSA compressors. The non-essential air system was designed to be a shared system. Common loads are distributed off either the Unit I air header or the Unit 2 header. The headers are interconnected at numerous points in the system. It is possible to close isolation valves and split the system into separate Unit 1 and Unit 2 systems; however, because of the

distribution of common loads, the total loads on the split system would l be unequal. Currently, the controls of the four air SCSA compressors i are not configured to control the air compressors on a unitized basis. The team reviewed the design of the system from an operating / maintenance perspective. Currently, the licensee runs with an average 1.5 compressors loaded. With four installed SCSA compressors, sufficient compressor capacity exists to allow routine preventative and corrective

compressor maintenance. The licensee's preventative maintenance program l includes a semi-annual overhaul of the compressors. The design of the !' l controls for the SCSA; however, is very maintenance limiting. It is not possible to electrically isolate the manual compressor loading controls from the automatic compressor sequence controls. As a result, the station does not have any routine PM on the control cabinets and corrective maintenance is approached reluctantly, as was observed with the December 15, 1992, loss of air event. A similar reluctance to do corrective maintenance on the Auxiliary Control Air system as evidenced by the number of longstanding minor work items. Several of these had been deferred more than once for various operational reasons. The team also concluded that the lack of control room indication for the control air system severely limits the operator's ability to respond to a loss of control air event prior to the onset of a plant transient. The licensee has identified this lack of indication as a system weakness and have planned the installation of pressure indication during the upcoming refueling outages on both units. At the end of the inspection , period, the licensee was evaluating the need to install system low l pressure alarm annunciators. The team concluded that the evaluation of this potential modification should be expedited. _ _ _ _ _ _ _- . --- - - - _ _ - -_ __- - _ - ____ _- _-_ - _ - __ - - -_ - ____ - - - __ - --_____ _ . . . .

. $- .j

, e 19 SAFETY RELATED ELECTRICAL SYSTEM: The team reviewed the design of the safety-ralated electrical I distribution system which the UFSAR describes as a shared system. The 6.9 KV shutdown boards and 480 V shutdown boards provide redundant ' sources of power for safety-related components. The boards are , separated into two trains; however, the trains are not unitized. For

example, Unit 1, Train A 480 V shutdown board 1Al-A supplies Train A safety-related loads necessary for both Unit I and Unit 2 operation. As a result, the loss of any one of the eight 480 V shutdown boards places - both units into a TS eight hour action statement. Similar TS

requirements are imposed on both units on the loss of any one of the four 6.9 kV shutdown boards or loss of any one of the four 120 Vital AC , power channels. . The team identified that because of the limiting shared design of the

safety-related electrical system, the licensee has no preventative

maintenance in place for the 6.9 kV or 480 V shutdown boards. The ' system engineer for the 480 V system recognized this weakness and

initiated development of a PM package for the 480 V. shutdown boards in ! 1991. However, the PM has yet to be implemented due to the requirement l to have all boards operable to support either unit's operation. The licensee is considering asking for TS relief by extending the A0T for a single board. In addition, the licensee did experience a failure of one of the 480 V shutdown boards in November 1991, during maintenance - activities. The board was inoperable for six days while repairs were made. At the time, both units were shutdown due to unrelated circumstances. Had the failure occurred during normal operations, a dual unit shutdown would have been required.

The inspectors also noted that the four Standby Diesel Generator sets are all required operable by the TS to support operation' of either unit and have a 72 hour A0T if a single EDG is inoperable. This combination 't would most likely require a dual unit shutdown in the event major , corrective maintenance was required and also appears to discourage any extensive PM as well. i ESSENTIAL RAW COOLING WATER , The team examined the ERCW system for compliance with GDC 5 and for > influence on dual unit operation. The function of the ERCW system is to provide the necessary cooling or makeup for safety-related plant , equipment and components in response to adverse plant operating , conditions (design basis events) which impose safety-related performance requirements on the systems being served. ' ! System Description: The ERCW system is described in UFSAR Section 9.2. The team reviewed ! the system configuration, power supplies and controls and found no j 1 ,

. . . ' ,

..t j i ' 20 ! deviations from GDC-5 criteria. ! The team reviewed the design basis of the ERCW system. The UFSAR and. design criteria associated with the ERCW system indicate-that certain , combined modes of operation are not within the design capability of the- ERCW system. In particular, it had been recognized that the modes which

cannot be adequately supported are both units in shutdown or one unit in ' shutdown and the other unit in a LOCA condition. The design criteria. I state that should a LOCA occur in one unit, then the non-accident unit i must be maintained at hot standby until heat loads in the accident unit are low enough to allow the non-accident unit to proceed to safe'

shutdown conditions. However, the inspectors noted that the. configuration whereby one unit is in a refueling mode and +he other unit is in an accident condition had not been addressed in the design criteria for the system. Further, based on_the heat loads associated with a refueling condition as originally assumed in the design criteria , and UFSAR, then the ERCW would only be able to remove about 25 percent i of the required heat loads as listed in the UFSAR. Thus, the operating ' configuration whereby one unit is in refueling and the other unit is in an accident configuration is outside the original design basis of the

plant. j t Based on these observations, the licensee performed a preliminary calculation in an attempt to gain an insight into the actual heat loads ! and heat removal capability for the described configuration. It was . determined that if actual plant operating data were used, and if certain . " nonessential" heat loads were removed, then the ERCW system would have been able to effectively remove the necessary heat to support the plant , configuration. However, the current plant surveillance' procedures do not envelope the required flow rates to ensure that adequate ERCW capability is maintained at all times. .l ' It is possible that under certain postulated accident and heat loading. conditions, coincident with certain ERCW flow rates as allowed by the plant surveillance requirements, that the ERCW system would not be able , to remove the necessary heat to support all of the required safety- related equipment to mitigate an accident in one unit and a refueling configuration in the non-accident unit (i.e. preliminary calculations

indicated that approximately 1,200 gpm would be required, whereas the surveillance requirements mandate a minimum of 950 gpm. Actual system flow rates as determined during the most recent surveillance were on the order of approximately 2,000 gpm). The licensee agreed to perform further evaluations of the actual heat loads expected during this configuration as well as the minimum required mitigation equipment. The results of this investigation should provide a definitive answer as to whether this configuration should be considered as a design basis condition of the uni +.s, and whether surveillance requirements should be ' modified to assure.some minimum level of heat removal capability during refueling configurations. The final evaluation of this issue is identified as Inspector Followup Item 50-327,328/93-02-05, Review of ERCW Design Basis for Shutdown Configurations. t , . . . -

-. - i ' ' i , i ' 21 l The team reviewed the licensee's maintenance practice on the ERCW pumps and confirmed that system was maintained within its current design basis ' configuration during preventative and corrective maintenance. CONCLUSIONS:

In conclusion, the team did not identify any systems not in compliance l ! with GDC-5. However, the team identified several instances, including _ condensate, compressed air and safety related electrical systems, where

the design of a shared system allowed very limited opportunities for _ ! preventative or corrective maintenance. The restrictive design of the_ , control air system was a significant contributing factor to the December 15, 1992, dual unit runback event and continues to inhibit the I performance of minor maintenance items. The licensee's study of shared system / dual unit vulnerabilities must be completed and appropriate i followup actions implemented to help reduce dual unit challenges. , , B. SECONDARY SYSTEM VULNERABILITIES / RELIABILITY AND DESIGN LIMITATION , LICENSEE EVALUATION OF SHARED SYSTEM VULNERABILITIES: j The licensee has initiated a review of issues related to the reliability ! of the secondary plant. The study tiled, Secondary _ Plant Reliability l Study, will draw together data on component relialbility, system ? vunerability, shared system interaction, maintenance scheduling process l control, componnent maintenance history, industry and plant staff

experience and will develop recommendations on how to improve the t relialability of the secondary plant. While the team was on site, the

licensee was conducting a review of shared systems that have the _ l potential to cause dual unit trip or dual unit power reductions. The .! licensee's review included shared systems important to safety and those . ! necessary for reliable power operation. The trip or reductions evaluated included both automatic actions and manual actions, including i TS required shutdowns. '

By the end of the team inspection, the licensee had compiled a list of systems that had the potential to cause a two unit trip or power reduction. The licensee's list included'23 of the total of 43

identified shared systems. The licensee indicated plans to identify ' specific failures in these systems that would lead to dual unit events. ! The licensee also indicated that the results of this study would be i factored into the Secondary Plant Reliability Study. ! i FEEDWATER AND CONDENSATE SYSTEM DESIGN LIMITATIONS j! Through the review of the recent secondary plant induced transients I which resulted in turbine runbacks and reactor trips the team discovered { what appeared to be a challenging design. The design of the condensate j system requires the operation of all of the installed pumps for 100% j power operation. Additionally, the feedwater system-relies heavily on i the amount of water provided by the feed-forward design of the feedwater i i

I l l . __ .,

. . '

22 heater drain system. Therefore, the Sequoyah design employs a rapid (200%/ min) turbine runback if the feedwater heater drain system is not ' available to prevent a reactor trip on low-low SG level. - An analysis of the design of the logic circuit which initiates a main turbine runback when there is a high level in the number 3 heater drain ' tank, revealed possible oversights. The 200%/ minute runback is initiated if heater drain tank bypass valve LCV-6-105B begins to open. This valve is one of two which open to bypass the contents of the number i 3 heater drain tank to the condenser when level in the tank is high. The design intent of the runback is anticipatory in nature and was- ' devised to reduce unit load to a point that the feedwater system can . supply adequate flow to the steam generators if the feedwater ' constituent afforded by the feed-forward capacity from the number three ' heater drain tank / pumps is lost. In the team's opinion, the circuit is not an adequate precursor of the transient against which it is designed to protect, in that high level in the tank does not necessarily presage i the loss of that flow. It should be noted, that the licensee performed an evaluation of the runback circuit in 1991 which indicated that the runback, as designed, is unnecessarily challenging and has plans to modify the circuit to be more representative of the intended protection.

The need of 100% of the installed equipment for 100% power operation has

resulted in a large number of WO on hold and clasified as outage work. -l C. MATERIAL CONDITION , 1). SWITCHYARD MATERIAL CONDITION

To assess the reliability of the switchyard the team inspected the j switchyard equipment and related maintenance documentation. The i team noted an air leak and temporary strip heaters near the

pressure regulator for PCB 5054 which is one of the two 500 KV ' supply breakers for the intertie transformer. The air compressor cabinets showed some oil leakage and some oil was noted on the . ' pressure switches. The air compressors for many of the Hitachi breakers had extensive run hours and the oil appeared to be emulsified. 'I The team inspected selected support equipment in the 161 KV switchyard. A temporary high pressure hose was connected to one air reservoir, this was utilized to allow one compressor to provide pressure for two breakers during maintenance or for ' emergencies. The team noted an oil leak on the rear oil cooler of ! PCB 948. In general the team concluded that the present material condition of the switchyards was marginal and needed maintenance. ? The team reviewed the practices and history of preventive and corrective maintenance for 500 KV and 161 KV switching equipment in the Sequoyah switchyard. 'a team examined the PM schedules and procedures as well as corrective maintenance records. i . - -

. . , , I ' ' . . 23

Preventive Maintenance: l ! The vendor manual for the Hitachi air blast power circuit breaker provides a recommended schedule for inspection and overhaul of the i Hitachi PCBs used in the Sequoyah switchyard. The vendor

recommended a daily visual inspection, a comprehensive series of inspections over a two year or 750 breaker cycles interval, and a ten year breaker overhaul. The two year maintenance schedule i provides for the inspection of the interrupter moving contacts, the air operating system mechanical linkages and valves, the air operating system pressure regulator, the bushings and the breaker

control panel contacts. ' The interaction of the Customer Group and the Nuclear Power group is governed by SSP-6.52, " Activities of Customer Group at Sequoyah

Nuclear Plant." According to SSP-6.52, the Site Maintenance

Manager is responsible for oversight and maintenance and testing conducted by the customer group onsite. The onsite customer groups report functionally to the site maintenance manager. The i customer group is responsible for testing and maintaining PCBs and compressors and is required to establish methods to control,- ' document and track maintenance of switchyard _ equipment. ! The site maintenance group turned over responsibility for maintenance of switchyard equipment to the customer service group starting in 1988. The turnover process was completed in approximately 1991. A review of maintenance activities prior to

1988 revealed that the 500 KV breakers received little preventive maintenance and required frequent corrective maintenance. The customer group established and implemented a maintenance program , for the PCBs that incorporated many of the vendor' recommended i items. l The Hitachi breakers appear to have experienced a significant number of failures and have not demonstrated good reliability. Spare parts and lack of training of maintenance personnel on

correct maintenance practices for the Hitachi breakers also contributed to their low reliability. This was evident by the

November 20, 1992 event. ' 2). B0P MATERIAL CONDITION .As discussed in several sections of this report 80P material condition I has been identified as the cause of many of the secondary plant induced operational transients. The design of the feedwater/ condensate system ' does not lend its self to at power PM or to some degree corrective maintenance of equipment. Additionally, it was apparent to the team ! I that effective predictive maintenance / testing was also restricted and not being performed. A walkdown by the team noted outstanding WR tags i hanging on important BOP equipment such as a dome air leak'on a l regulating valve on the main feedwater pump, and the CA system compressors and air dryers. The team also attended P0D meetings and

. - . -- . .

. - ,-

. 24 I r noted that planned CA cooler water leaks were discussed ~ and that a note ., about very dirty lube oil for one of the main'feedwater pumps remained-- ~ on the POD for several days without generating a great deal of

management attention. ! The team also reviewed the licensee's draft SIP which had an area dealing with B0P material condition. The team noted that although the l SIP described numerous outside and inside findings of poor B0P material ' conditions the corrective actions listed appeared-to be narrowly focused . at correcting appearance (painting, insulating). The licensee ! responsible manager for development of the SIP indicated that other areas of the plan included additional corrective actions- such as the , SPRS, etc. The licensee's improvements in the B0P area have to be ' properly focused since the current design restrictions make improvements difficult. ' 6. MANAGEMENT CONTROL AND OVERSIGHT OF B0P WORK ACTIVITIES A. CONTROL OF SWITCHYARD ACTIVITIES ' i Nuclear Power Group and Customer Group share responsibility for switchyard activities at SQNP. Policy and Organization Manual, . Intergroup Agreement No. 6 - Customer Group, specified organizational responsibilities and requirements for switchyard activity control. Site specific requirements were outlined in Site Standard Practice (SSP) 6.52 Rev. O. The team reviewed these documents to determine how switchyard

activities were controlled. l Nuclear Power Group is responsible for operation of the switching of the equipment under control of the CG dispatcher at the ADCC. Nuclear' Power Group is responsible for oversight of the CG switchyard equipment - maintenance and testing evolutions. The SSP defined the equipment - maintenance responsibility split between CG and NPG. Organizationally the CG maintained a dedicated support group at SQNP which functionally reported to the Maintenance Manager. Customer Group consisted of an engineering test unit reporting to a supervisor and an , electrical maintenance unit reporting through a switchyard foreman to l the Engineering Unit Supervisor.

CG maintenance and test work activities were controlled under the site work planning process. CG work on safety-related equipment was

performed per plant maintenance and sis. The team reviewed several , procedures and determined that the level of detail and step by step compliance requirements-were adequate. CG was required to use PM r instructions where required for activities on non-safety-related equipment. The team observed that some PM procedares were used, but . > that for certain evolutions, such as breaker timing testing,- the - guidance of the CG Field Test Manual (FTM) was utilized in lieu of procedural control. l t ,

. > ' l 7 l Work plan WO 92-15 264-00, the work document for the post installation I testing on the new ABB PCB 5058 was reviewed by the team. The work plan required that specific testing be performed in accordance with the CG g Field Test Manual. The team verified that the FTM did not have a ' specific section for breaker timing testing ABB SF6 breakers. No other , specific procedures were utilized for the testing, only the generic

guidance provided by the FTM and the requirements contained in the ~ vendor manual. The ABB vendor manual did not contain specific detailed ' guidance on breaker timing testing. ABB drawing 996Dll4 sheet 1, Rev. 4 , provided a caution note regarding potential breaker damage when ' operating the breaker without the 20 cycle closing delay. The initial breaker timing results did not meet vendor specifications. TVA contacted the vendor and ABB indicated that auxiliary relay time was not included in the timing specification. TVA then changed the test points for the breaker timing set and performed timing checks without

the 20 cycle closing time delay. It appears that the breaker timing set ' closing and opening signal settings, with the closing delay timer i bypassed, allowed the breaker to be rapidly cycled and mechanically damaged. This damage appears to have been the probable cause of the electrical failure of the C phase pole unit which caused the dual unit

trip. The lack of procedural controls for this important evolution was { viewed by the team as a significant weakness. TVA recognized the need for control and management of the installation of the new ABB breaker and took . steps to provide appropriate control of the evolution. Project Managers from both the CG and the SQNP site were assigned to coordinate the installation. The SQNP CG switchyard electrical maintenance group was supplemented by a dedicated team of TVA . CG construction electrical personnel assigned to the project. The SQNP CG engineering unit personnel were assigned the testing tasks on the new breaker. A TVA corporate electrical maintenance specialist who had been to the ABB factory for training was assigned to the project. I Subsequent to the installation of the breaker, the post installation testing was performed. Many of the tests could have been and were performed with the breaker deenergized. However, the final relay phasing checks must be made with the breaker energized and under load. The final PCB installation was discussed Dy the project PMs and SQNP operations. SQNP CG and the ADCC developed the switching orders to j energize PCB 5058 and perform final relay phasing. Plant operations was ' briefed and concurred with the switching orders. i The switching orders called for all four primary protective relays to be disabled simultaneously by opening their trip cutouts for relay phasing i checks. The protection would be provided by secondary protective < relays. The SQNP CG and the ADCC personnel decided that the probability I of failure of the SQNP intertie transformer bank or other 500 KV equipment was significantly less than a switchyard or intertie trip resulting from incorrect wiring. This decision was made despite the i wiring checks, and CT polarity and ratio tests and many other breaker ' and circuit functional checks which had been performed. The team asked i ! l ) >

._ , 1 , 4 . 26 ! if any thought had been given to adjusting the time delay on the secondary relaying _and was told that this had not been considered. f ! The team concluded that the licensee's risk evaluation was weak in that ) a thorough consequence analysis did not appear to have been performed. i Operations personnel were not aware that an electrical fault with only , secondary protective relaying available would yield a dual unit trip. ' TVA recognized that the PCB 5058 installation would require additional

management and control yet the critical step of energizing and loading the breaker was performed without good management controls. The risk evolution was performed without a thorough consequence analysis, and no i steps were taken to provide compensatory measures while disabling all ! primary protective relaying. Operations personnel did not fully - ' understand the risk and they did not question the switching orders as , written. SQNP management oversight of this evolution was judged as weak, particularly in allowing the evolution during a period when the units were not fully staffed. ! In response to the 12/31/92 event, TVA instituted interim measures for switchyard work control performed by the TVA CG. These measures were

developed by TVA SQNP, CG, and Corporate staff. One of the changes was to list significant CG work activities on the SQNP Plan-of-the-Day. CG

will be required to identify and evaluate non-routine work to minimize removal of protective devices and the length of time the protection is removed. The non-routine CG activity evaluation must be presented to the Operations Manager three days prior to the scheduled activity and it . shall describe possible impact on the switchyard and unit generators and ! discuss alternative actions considered. Activities performed by the CG outside of normal / emergency switching activities that require trip cut outs to be open shall be approved by the Operations and Maintenance Managers. Additionally, CG shall routinely provide input to SQNP regarding transmission grid conditions which could impact SQNP operation. The team reviewed the interim measures and determined that t they would significantly improve switchyard work control and control of risk significant activities. B. CONTROL OF SECONDARY SYSTEMS ACTIVITIES

Site Standard Practice procedure, SSP-6.21, Maintenance Management ' System Initiation Of Work Requests, specifically requires that work requests be initiated for maintenance on all installed plant equipment except for very simple tasks such as erecting a ladder. The purpose of _ ' the licensee maintenance management program is to control all levels of i maintenance repair activities to include treading of failures and the assurance that operations maintains configuration management of the ' systems as well. Based on the teams review of events discussed in Appendices to this report, it was determined that operations personnel and system engineers

have for some time, been involved in a practice of performing informal, undocumented, uncontrolled maintenance on both units' condensate feedwater/ heater drain tank level controllers and associated equipment.

!

. _ . . ., t . . ' l J j , ! 27 . ! This " field tuning" approach to maintenance is not as specified in the ! site administrative procedures that controls maintenance activities for B0P equipment, and is counter productive to the overall process of l achieving and maintaining a high degree of equipment reliability.. The

team also noted that responsible levels of site management were aware of ! and condoned this practice. Management should reevaluate controls of

B0P activities and make the necessary modifications to the program and ! then convey appropriate expectations pertaining to procedure adherence , and the presumed quality standards pertaining to maintenance activities. ! i

As discussed in APPENDIX 5 of this report the failure to perform l maintenance of level controller 2-LIC-6-106 on December 8,1992 is a i violation of SSP-6.21, Maintenance Management System Initiation Of Work ! Request and is identified as an example of Violation 50-327, , 328/93-02-01, Failure to Follow Procedures. l

C. WORK ORDERS ON MATERIAL RESTRAINT ! ! One of the common themes recognized by the team for the events evaluated I was lack of timely corrective action. In some cases this was due to i unavailability of parts. At the time of the inspection the material j ' restraint work orders represented 12% of the overall corrective ~ maintenance backlog (105 out of 859). From the team's perspective, this { percentage was relatively high. This percentage is representative of i , the numbers throughout 1992, and was in excess of the licensee's goal of ! 5%. Out of the 105 work orders that were on material restraint,18 were j the result of parts having been purged via the licensee's inventory

reduction program. The main criteria for which parts were selected to j be purged was usage history.. The team noted that as the plant ages

- certain parts with a low usage history may have increased demand, due to ! equipment wear over time. This may have an adverse effect on parts ! availability as the plant ages given the licensee's system of purging ! ^ material based on usage history. The team concluded that-licensee

management needs to take steps to reduce this backlog and maintain it ! within their goal. Sequoyah management's attention needs to be placed .

on the inventory reduction plan and it's potential to adversely affect j the units. f D. REVIEW 0F SYSTEM ENGINEERING 4

The team reviewed the system engineer program for factors that j contributed to the recent dual unit transients. The system engineering i program is under the direction of the Technical Support Manager and consists of approximately sixty engineers subdivided into four I 2 organizations: Reactor Engineering, NSSS Systems, BOP Systems and I&C

] and Electrical Systems. Licensee procedure SSP-8.50, " Conduct of

Technical Support," is the governing administrative guidance for the [ Technical Support Program. i The team reviewed the staffing of the Technical Support section. ! SSP-8.50 states that in general, systems engineers will be cognizant for

! ! , i _

.- , '

_ , 28 one to four systems. A review of system engineer assignments confirmed that the licensee adheres to these guidelines. The system engineers are predominantly TVA senior engineers. The Technical Support organization is implementing a formal qualification process that includes classroom based systems and fundamentals training, training on specific work processes, simulator training and a formal written and oral examination process. The team interviewed the Manager of the Technical Support organization to determine his expectations for the conduct of the system engineer program. The Technical Support Manager has been involved with the system engineer program since it was developed in its current format in , 1989. He recently assumed the position as Technical Support Manager ' after filling the position in an acting capacity for approximately one year. The Conduct of Technical Support Manual outlines responsibilities of system engineers, The manual lists 33 job tasks of varying complexity. Of those, the Technical Support Manager indicated that he emphasized performance of system walkdowns, maintenance of the system notebooks including trending information and resolution of problems i encountered by other organizations with a system. He expressed a view that the system engineering group is a support organization and not owners of the systems. This same view was expressed by maintenance and operations personnel as well. The team noted that this was distinctly different than the system engineer expectations described by the new site Vice President. The team determined that this difference in - expectations by senior site management had not been effectively communicated. The team interviewed a number of system engineers to determine their understanding of their job and to assess knowledge of their systems.

All system engineers interviewed demonstrated a strong knowledge of ' system operation and of the current status of the system. The team did determine some weakness in knowledge of design basis of the system ' although the SEs understood the operation and performance of the as- built system. In general, system engineers demonstrated a good ' knowledge of the outstanding work items against their systems and took

an active interest in trying to see that these items were worked off. 7. POST TRIP AND INCIDENT INVESTIGATION PROCESS REVIEW The team performed an evaluation of the licensee's post trip review and , incident investigation program to determine its effectiveness. The ' procedure which delineates the process, is Site Standard Practice i SSP-12.9, Incident Investigation And Root Cause Analysis. The overall purpose of the process, as specified in the procedure, is to conduct a l thorough evaluation of events in order to determine, in part, the root i cause and the corrective actions to be taken to prevent their recurrence. The process is comprised of two basic components; "a post trip review" to determine the basics of what happened and what short term corrective actions are required before the unit can be restarted, l and a more thorough " incident investigation" to determine, in part, long = term corrective actions. , - -

. . i l . -

. > I 29 . The team concluded that the procedure and the process are basically sound. However, in the evaluation of the post trip review performed on Unit 2 for the 12/31/92 reactor trip, the team observed a number of i- deficiencies which are described below. The team concluded that these ! deficiencies resulted in an inadequate post trip review, r < The team performed an in depth evaluation of the dual unit reactor trip of 12/31/92 during which it was noted that some important aspects of the - event were not addressed in the initial post trip review performed on Unit 2. One of these oversights involved the fact' that the post trip review concluded that the VCT outlet valves had isolated a second time because level in the VCT had decreased to the isolation setpoint-. This conclusion incorrect. The reviewer had made the conclusion based on discussions with the operators involved in the event; however, did not review and verify VCT level chart recorder traces which revealed that - level had in fact not decreased. This erroneous conclusion resulted in the unit being restarted and then forced to return to Mode 3 to resolve

the discrepancy. A second oversight involved the fact that the ' handswitches for the RWST to CCP pump suction were misaligned (left in J A-Auto as opposed to AP-Auto) by the operator, a fact that the_ post trip ! review did not detect. It was for these reasons that the team concluded , that the post trip review conducted for the 12/31/92 event was ' inadequate. This is identified as an example of apparent violation 50- 327,328/93-02-02, Failure to follow procedures. The team also evaluated the Incident Investigations (II) associated with the October 26, 1992, control air system water. intrusion event, and the necember 15, 1992, dual unit runback event, the November 20, 1992 loss , f switchyard intertie transformer event and the December 8, 1992 Unit 2 [ runback. The team concluded that, that in general, the incident investigations were broad based and thorough. However, the 11 for the October 26, 1992, event was deficient in the following areas. The licensee's II concluded that the root cause of water intrusion into the control air system was condensate accumulation in the #2 receiver, due to a sediment buildup that obstructed the drain line. The II further concluded that the accumulation of detris was due to ineffective blowdown technique. The source of the sediment was corrosion products from the control air system. The licensee corrective action consisted of modifying their receiver blowdown technique, writing of work orders to correct the , secondary equipment that did not respond as expected during the transient, and writing a PM to clean the inside of the receivers on a five year frequency.

The II concluded that there were no precursors to this event. However, during the II, the plant manager directed a separate, independant i investigation be performed by ISE to evaluate the adequacy of SQN responses to this event. The ISE evaluation concluded that a more , thorough evaluation of previous drain line problems might have prevented ! ! _

. - , , ' , 30 ! this event. Additionally, the ISE identified that there were safety- rela ed' equipment on Unit 1 (6 locations) which needed local particulate filters installed (already in place for the same components for Unit 2). i This corrective action had not been identified in the original II.

,

The probable root cause for the sediment accumulation in the receiver drain line is inadequate preventative and corrective maintenance. A more thorough evaluation of previous drain line events could have t prevented this event.

! Although, the ISE independent review recognized that there were precursors for this event, and provided sound corrective action , recommendations, an adequate comprehensive II would have attained the i same results. For these reasons the team concluded that the incident investigation process could also be improved.

' 8. REVIEW 0F RELATED ISSUES During the course of this inspection the team identified several related

issues described below that required additional review. l A. UNDERVOLTAGE PROTECTION SCHEME ISSUES The Sequoyah undervoltage protection scheme, as described in TS consist l of one set of 70% loss-of-voltage relays with a 1.5 second time delay

' for EDG start, and a second set of 70% relays with a 5 second time delay for shutdown board load shedding. This would mean that if shutdown

board voltage recovered to the pickup setting of the shedding relays i before the 5 seconds had elapsed, there would not be stripping of the shutdown boards. In October 1986, a modification was made that provided a set of relays , which open the feeder breaker to the shutdown boards when the voltage on

the normal feeder to the shutdown boards drops below 80% of nominal for greater than 0.5 seconds. This results in a deenergized shutdown board i until the 70% relays described above actuate, and the EDG ' subsequently ties onto the bus (approx. 10 sec.). Note that with this modification, even if offsite voltage recovers after the 0.5 second time delay, the

plant is still subjected to load shedding due to this 80% relay. This - ' relay in effect circumvents the protection scheme described in TS table 3.3-4.7. As of January 22, 1993, a TS amendment had not been submitted , to recogonize the presence of this modification. The team noted that

the modification was necessary due to concerns with potential operation ' with a degraded bus voltage slightly above the 70% level for a prolonged- period of time (up to 5 minutes due to degraded voltage relay time ! delay) resulting in motors tripping on thermal overloads. This > legitimate concern underscores the need for the 80% relays to be - included in TS, with' appropriate surveillance requirements included. , The licensee provided correspondence from the early 1980 time frame between TVA and the NRC that discussed the instalation of the 80% relay.. The team reviewed this correspondence and concluded that the 10 CFR 50.59 performed in the early 1980s did not recogonize the need

! ,

. _ , . ! . . 31 . i for a TS change and this is a weakness. The licensee's 10 CFR 50.59 . review process has been upgraded several times since early 1980 and , these upgrades would produce a different product if the same

modification was reviewed today. Additionally, the licensee is proposing to modify the current undervoltage protection scheme and make i obsolete the 80% relays in question. The licensee will submit proposed TS changes associated with this modification for the NRC staff's review during the summer of 1993. The team discussed the licensee's' plans with the NRC staff and find the licensee actions acceptable. . The 80% relay described above, opens the normal feeder breaker via a set of Agastat 7000 series relays which are set at 0.5.second. These Agastat's were calibrated on an 18 month PM frequency. The procedure ' used to calibrate these relays was MI-13.1.6. The last time this - calibration was performed for the Unit 2 relays was May 1989. According , to the licensee's procedure for rescheduling or deferring maintenance - (SSP-6.3), if a PM task cannot be performed within the scheduled due date, including the grace period, a technical justification must be wriden by the responsible . system engineer in order to defer the PM. Contrary to.this requirement a technical justification was never [ performed for the Unit 2, 80% Agastat relays (LVI & LV2). This matter is identified as an example of violation 328/93-02-01, Failure to Follow , Procedure. The licensee completed a technical justification to defer 1 this PM until the next maintenance opportunity (not to exceed 2 months). ' The technical justification was subsequently developed by the licensee and provided to the team for review. The team found the justification ' to defer the PM until the next maintenance opportunity to be adequate. . B. PLANT GRID INTERFACE f

During the review of the November 20, 1992, failure of PCB 5054 event, the team noted that a 161 KV grid undervoltage alarm was received

subsequent to the loss of the 500/161 KV intertie transformer bank. The l plant entered TS LCO 8.3.1.1.C and dispatcher controls were required to - restore 161 KV grid conditions following the loss of the intertie transformer bank. The team was concerned about the availability of an immediate offsite power circuit as required by 10 CFR 50 Appendix A, criterion 17. !

The team reviewed the licensee controls related to the plant grid interface. Potential problems with 161 KV switchyard voltage were initially identified by TVA's CAQR process in 1987. The CAQR indicated [ that there would be problems maintaining 161 KV switchyard voltage with

. projected 1987 grid loading with the SQNP 500/161 KV intertie ' ' transformer bank out of service. ISEG report 87-10-SQN-1 recommended - improvements to resolve the plant grid interface problem. A plant grid I interface study was issued February 20, 1990 which evaluated several , options for resolution of the plant grid interface problem. The. study recommended that existing CSSTs be replaced with automatic tap changing CSSTs. Calculation SQN-EEB-MS-TIO6-0007 was issued on January 12, 1990, ! - for procurement of new CSSTs and showed a minimum grid voltage requirement of 153 KV for steady state loading and 155KV for transient

, ,

_,

, e

- 32 loading. New CSST A was operational on September 18, 1991, CSST C was operational on August 30, 1992, and CSST B was operational on November 30, 1992. , The team reviewed the interface agreement between the plant'and the 3 customer group as it related to plant grid interface. The plant provides updates of the APS to the CG. The CG then updates the analysis ' of grid conditions based upon the new SQNP auxiliary power system , loading and grid load data. The CG had not yet completed their analysis - i based upon the SQNP December 10, 1992, APS update. The agreement further required annual meetings on plant grid interface and the team observed that these meetings were being held. The team reviewed the switchyard letter 18 and actions required by the ADCC dispatchers to control grid conditions. The inspectors travelled to the Chickamauga ADCC to walk through the required switchyard letter

actions with the dispatchers to determine the recovery time. The ADCC had the ability to recognize a loss of the intertie transformer bank via an alarm or by the computer status screen. The inspectors judged that the dispatchers could follow the steps in the switchyard letter in 10 minutes if the appropriate initial conditions were met. Training on the implementation of the switchyard letter 18 had been provided to the < dispatchers. The dispatchers were aware of the requirements and had a copy of the letter. For the 10 minute recovery time to be met certain , intertie transformers and generating plants had to already be on line. The team noted that it was not standard practice for the dispatchers to contact SQNP operators to notify them of activities which could put the grid out of compliance with the required initial grid. conditions. , While reviewing the dispatcher restoration actions the team noticed a copy of a switching order for SQNP which had not been followed in sequence order. Site Standard Practice SSP-6.52 Rev. O, " Activities of i Customer Group at Sequoyah Nuclear Plant", requires in step 3.2.a that ! clearances and grounding be performed in accordance with site and Customer Group requirements. Customer Group Operating Procedure Letter No. 9 dated June 1, 1992, section 14 requires that after verifying switching orders to be correct and adequate, the operator will then perform the operations exactly in the order given, carefully observing equipment for proper operation. He will report the time for each - operation, including the time that clearance tags were placed. Contrary to the above requirements, on January 1,1993, during first shift, . Sequoyah operators performed the required switching evolutions out of sequence. This item is identified as a part of violation 93-02-01, failure to follow procedures. ' The team reviewed the plant grid interface issue. The NRC's staff's evaluation of TVA's 1990 design and administration controls for ' compliance with GDC 17 ~and the implementation of TS controls-is contained in attachment 1. The team discussed the NRC staff's position described in attachment I . I

- _ - - - -.-. . - - . - - . - . - . ! - 2 i - s' a l 33

with the licensee. The licensee agreed that until final resolution of the grid stability issue they would enter TS 3.8.1.1.c immediately if ' the intertie transformer is lost. The licensee described to the team i several switchyard improvements including the installation of auto-tap

change CSRT that have taken place that may modify the NRCs staff ! position described in attachment 1. The licensee agreed to submit to ! the NRC the following information so that the NRC could rereview the modified off-site power system and the licensee interpretations and TS ! controls of the off-site power sources and equipment. As soon as , possible the licensee will provide to the NRC; 1) a copy of CSST-design ! < change package along with the supporting calculations, 2) the completed j grid load study, and 3) their interpretation of how TS requirements will be applied to planned and unexpected loss of switchyard equipment such . as the intertie transformer to include entry and exit conditions and any ! t- load dispatcher controls. This item is. identified as URI 50-327, ! 328/93-02-04, Plant Grid Interface Review, pending licensee submittal of

! the above information for NRC review and resolution.

i

9. PROBABILISTIC RISK ASSESSMENT

A. General Comments k i The inspectors reviewed the available probabilistic risk related i , information associated with the Sequoyah facility to gain further i insight into the safety significance of recent events at the site. Th.is j review consisted of examinations of previous PRA studies associated with l Sequoyah as well as interviews and discussions with the TVA i Probabilistic Risk Analysis organization. l 1 The licensee had submitted a level 2 probabilistic risk analysis (PRA) j in support of the Generic Letter No. 88-20 requirements for an i individual plant examination (IPE) in early September of 1992. The ' initiating event data and systems reliability data which were used in ! - the analysis were based on generic industry data and updated using l ' plant-specific reliability data. The plant-specific failure and ! maintenance data used in the reliability updating methodology covered 4 the period from May 1988 through June 1991. Since the period of collection for reliability data (5/88-6/91), the . units had experienced a number events which would warrant an evaluation ' of the estimates used in the IPE submittal. In particular, a number of

reactor trips had occurred since June 1991 which would point to a higher i reactor trip frequency than that used in the PRA. Additionally,

, independent component failures in the instrument air system also pointed [ to a possible reduction in overall system reliability relative to that

, which was assumed in the analysis. Discussions with the licensee's risk j analysis organization indicated that no plans had been made to update or j reevaluate the reliability data in light of the new information. In

. order for the PRA to provide a useful benefit to overall plant ! ' operations and safety, it is important that the most recent and ! realistic plant data be incorporated into the analysis. , , ! !

t

.- - . . . . O P

, . 34 Additionally, the licensee had only recently made efforts to incorporate the reliability information used in the IPE into the new effort to

examine secondary plant reliability issues. This type of use of PRA/ reliability results should prove to be highly beneficial in factoring _ risk and reliability information into a useful plant i improvement program. ! B. Event Evaluation , The information available from the licensee's recent IPE submittal was " used to evaluate the 12/31/92 switchyard event. The event is most closely modelled as a momentary loss of offsite power in which the RCPs i do not trip. The EDGs successfully start and load onto their respective-

' buses and the reactor trip function is successful. An equipment- alignment error resulted in the unavailability of both TBBPs to restart , ' after power was restored to the shutdown boards. During the event response, the operators intentionally secured all RCP seal injection due l to problems associated with the VCT/RWST suction valves and confusion concerning sufficient oil pressure to the' centrifugal charging pumps. r Based on the sequence of events, it was determined that thermal barrier ' cooling was unavailable for a period of about 20' minutes and that seal ! injection was unavailable for approximately 1 minute. It is estimated

that both RCP seal cooling functions were simultaneously unavailable for l 21 seconds. i Current industry guidance indicates that the RCPs can operate for only

very short periods of time without- both seal injection and thermal ' barrier cooling. Thus, the simultaneous. loss of both functions is modelled as an RCP seal LOCA. The licensee's recent IPE submittal , conservatively models any simultaneous' loss of thermal barrier cooling

and seal injection as an RCP seal LOCA. (Subsequent investigations by . the licensee indicated that no discernible RCP seal degradation had ' occurred in this particular scenario.) Therefore, from an accident initiation standpoint, the events on 12/31/92 were a significant precursor to a reactor coolant pump seal LOCA scenario. A detailed analysis of the 12/31/92 event using the licensee's IPE model , was performed by the team. The results of the 12/31/92 event evaluation

underscore the necessity to maintain RCP seal integrity from a plant 3 risk perspective. The results of the CCDP analysis indicate that the event which occurred was indeed a significant precursor to a reactor coolant pump seal LOCA scenario. The plant configuration which had ! existed prior to the event (i.e. improper TBBP hand switch positions) contributed to a measurably higher plant risk than that which had been . originally assumed in the IPE analysis. -! C. Summary The inspectors determined that the licensee had not been aggressively , using the information available from their IPE study. The plant risk and i reliability information contained in the analysis should provide a - ! useful database for plant improvement programs (i.e. secondary -l l i

. .-

. 4 1 - 35 i l reliability study, plant trip reduction evaluations). However, no-

evidence of a concerted, proactive effort to implement these results has 5 been seen. Additionally, even though the IPE study had identified the > availability and operation of the thermal barrier booster pumps.as important to safety from an overall risk perspective, the licensee had I not instituted adequate controls governing the configurations associated with the TBBPs. The IPE model had the capability to provide meaningful. modelling of recent plant events. However, the licensee had not i recognized the capabilities of this engineering tool to support overall plant operation ard safety. , r 10. Overall Conclusion , A. The Unit 2 operator's response to the 12-31-92 dual unit reactor trip event was severely handicapped by operator under staffing. The operator's response was plagued by errors as well as procedure

violations which ultimately resulted in stopping the only i operating CCP which degraded the ability to protect the RCP seals. ' Seal failure is a significant contributor to the CDF. t i B. Improvements of Management Controls Over Several Areas Are Needed 3 i 1) Failure to train the operating crews in a TS minimum staff i composition significantly degraded the operators' ability to , properly recover Unit 2 from the ~12/31/92 ovent. .The ! Licensee trains with 2 R0's on the boards at the simulator

but the units are frequently operated with only 1 R0 on one of the two units. Inadequate training at this staffing ! level directly contributed to the' Unit 2 cooldown'(not - controlling Aux Feed) and the decision to use normal L boration versus emergency boration (this resulted in the low VCT level with no RCS letdown and no automatic VCT makeup + capability available). l ! (a) The team concluded, that during the 12/31/92 dual unit { reactor trip, the Unit 2 Reactor Operations staff ! performance was deficient in the following areas / activities: - Failed to follow requirements of.A01-34 (emergency " boration). - Mis-positioned the RWST to CCP suction valves t handswitches by not returning them to A-P AUTO

position as required. ! - Performed unit restoration activities outside of I emergency procedure guidance (VCT realignment, letdown ' and TBBP restoration). - Inadvertently closed the VCT to CCP suction valves which resulted in a loss of boration flowpath and made

s !

. . -. .. ! . ' .- , , 36

'J the only operable high head pump inoperable. - Failed to enter the appropriate LCO when one CCP was > placed in the PTL position. .

- Failed to maintain ideal command and control ! functions. ' (b) Procedure 1&2-S0-70-1-was inadequate in that the procedure required that the switch positions for the ' - TBBP be placed in A AUTO instead of A-P AUTO. This resulted in the pumps being incapable of automatically , restarting as designed. Additionally, during blackout t testing switch position concerns were identified but , not corrected. , 2) Switchyard Activities , a) Coordination of Work Activities Between Plant and ' Customer Group Needs To Be Improved.

i - Control of hi-risk activities needs to be evaluated by both parties. Examples in which

controls could have been improved . include but - are not limited to, the occasion on 11-20-92 i when troubleshooting was performed while the system grid was degraded, and the event.of 12- 31-92 in which the licensee was performing high ! risk switchyard activities even though there was l limited on_ site staffing. - Communications needs improvement. Mis- . communications during the 11-20-92 switchyard event resulted in the operation of the wrong , breaker and a switchyard fire. ! - Procedure adherence needs improvement. Switching orders are not always performed in the sequa ce listed by the dispatcher, as required , by procedure. Plant verification of the 1 importance of equipment being removed from t service was evident-but dispatcher orders involving removal of protective devices are not

routinely challenged.

- Plants staff _needs to assume more control of

grid interface to ensure that TS and GDC 17 requirements are met. During the 11-20-92

switchyard event, it took 2 hours to re- . establish grid stability due to needed ! generation sources not being immediately available. .

. > l- . $ s l- ! 37 - Customer Group procedures need improvement. The lack of a detailed test procedure for the December 31, 1992 switchyard breaker testing , l- resulted in unnecessarily bypassing of many levels of primary fault protection and resulted i l in the switchyard fault being widespread,

l affecting both units and all emergency busses. - Additionally, the lack of a detailed test

procedure appeared to be the most likely cause 1 of mechanical damage to the switchyard breaker (i.e. bypassing the anti-pump time-delay relay). 3) B0P Activities l ' a) Material condition and control of BOP activities needs improvement.

l .t (1) SPRS needs to be completed and recommended [ l improvements evaluated and implemented in a

l timely manner. l ! (2) Maintenance improvements delineated in the Draft SIP appear to be narrowly focused and targeted at improving appearance instead of.actually

correcting the problems (paint, insulate, etc.). j 1 (3) Based on interviews and discussions, System a Engineers are not always effective in improving ! the condition of their systems. Plant + management's expectation of system ownership by i , the System Engineers has not been effectively ! l communicated. During the December 8, 1992, Unit i 2 runback, the System Engineers troubleshooting . ! / maintenance of the FWHDT level controls was not " controlled by the WO process.

l (4) The design of some of the secondary systems ' challenges the licensee's ability to operate and l maintain them. The sharing of support systems (i.e. needed for either unit operation) and the restrictive design of systems such as the condensate /feedwater system (ie 100% of the

equipment needed for 100% operation) are two

l examples of these challenges. Following are ! l selected examples of related concerns: , - Condensate and Feedwater System l l Sequoyah design / operating practices for

l 100% power operation makes raaintenance ' l difficult. Currently approximately 300 l W0s indicating needed maintenance are on ! ! s ?

. _ . ., k . , . . , . 38 . hold and scheduled as outage activities. ! The design includes non-standard auto ! turbine runbacks from the #3'FWHDT control circuit which-in the past three years has resulted-in at least 11 turbine runback challenges. This design appears to be _

' unnecessarily challenging and a propr, sed runback logic circuit modification based on previous' studies as far back ts 1991 appears to have bogged down due to_being ! included in a large scope, c<er increasing ! system modification packaca.

- Control Air System

There are currentl.v 37-open WO on the CA I ! system, 13 of which are > 3.5' months old- and 8 of these open W0s are on the Essential Air system of which 6 of the 8 are > 2 month old. Resolution of recommendations from the Engineering review of CA system should be evaluated and implemented in a timely , ' manner. b) Material availability was a contributor to delays in , correcting some plant deficiencies and has led to a i higher number of WO on material hold than the goal t (actual no 12%, goal 5%). Sequoyah management do'es

not appear to have control of the ongoing inventory reduction process that is surplusing parts that are

needed by the plant (currently 12 % of WO on hold for ! material availability are a result of surplusing).

, C. In general, the post trip review.and II processes are sound. However, the post trip review for-the 12-31-92 dual unit trip was inadequate. Additionally, several of-the-IIs ' reviewed appeared to be narrowly focused. -;

D. Other Issues ' 1) The 80% undervoltage relay modifications that installed the relay and changed the time delay should have recognized.through the 50.59 Process that TS , needed to be changed. Additionally, plant procedures i that require periodic calibration of the relays are i not being followed and no technical-justification was-

performed for test interval extension.

' > i

.. t , . , .

. 39 ,. 2) Control of Intertie Transformer The licensee agreed to enter TS 3.8.1.1.C and to .- ' control grid stability, with good lines of ' communication between plant and dispatcher, whenever the intertie transformer is out of service. The licensee also. agreed to. timely submit to the NRC their understanding of TS restrictions associated with the intertie transformer along with the supporting current-

grid load study and design calculations associated i with the CSST modification. 11. Exit Interview The inspection scope and results were summarized on February 5, 1993, i with those individuals identified by an asterisk in Appendix 5 of this ' report. The inspectors described the areas-inspected and discussed in , detail the overall conclusions.and the inspection findings listed below. l The licensee acreed with the characterization of the December 31, 1992, Unit 2 less of CCP flow with no operating TBBP as a precursory event to a RCP seal LOCA as described in their IPE/PRA. Although reviewed during ' the inspection, proprietary material'is not contained in this report. With the exception of providing additional information regarding the 80% undervoltage relay, dissenting comments were not received. Item Number Description and Reference -; 327,328/93-02-01 A Violation of TS 6.8.1 for failure l to follow and/or inadequate

procedures with multiple examples. 327,328/93-02-02 An apparent violation of TS 6.8.1 for failure to follow and/or in - adequate procedures with multiple examples. 327, 38/93-02-03 A apparent violation of 10 CFR Part

50, Appendix B, Criterion XVI, Corrective Action with two examples; 327,328/93-02-04 An URI for the NRCs review and evaluation of the plant / grid interface controls. 327, 328/93-02-05 An IFI for the NRC to review the- shutdown capability of the shared

ERCW system. 2

t t 8 0 8 i sr E. _ _ , 5 . r

_ a. e . g - . ~ - - . .- a - t - .r o . - . i r,- . s r r _. I f

- v a g l FIGURE 1 , 3

-

s r = . . ! I k 5 a

i E. , e t l

e r f ~.

i r .ao av vano s t - s . 1 , . - ?- ~d'M

i

- i .- w . . . . s t p 5. =: s ,- ,

6*e- o u. s. s . - .

ans er' y va. e,u, m. : ar g , , w ..., .,. i' g,ggs } soo-ra.s n y 7 siss *.s . , , .. r >6*6, o e . .* "" b / \\ siss 92s ,,,,

,,,, (b \\N ,* a , dwar;maf e ; m. s M < 4 333, 7 6 O6J e / so.o 4 s so.s s ' %j n . rr sms + y' manyuup opt- ' harp adammb* i l 7o*c* o=c g . . p . ses - 5 -

an@ephe y a amm ,*

TR . ca. .C.s , 6 lc h O C 4-2 a0/3rD/400 *** - 4 -<g a soo in -32.a .

b I

• s3 sic

1. t s. wD.1

, ei ,9 m3 4 ,, . Le 1,fM-' ,, .o . , .

• SC),G

vas sare.6 ,J er.aue e,,.. a .f.'*.'..*.' . A ,,7.3_ =* u... w. , , , , - o 606 s,

to.t. so. a= .ND ET #,( ,,e Y "ED * i <

6D6 616 , 4 f g I mac p-c Q

3 r.' L ' * C.s.

+ 2 e Yone. sea. st1 c. zr . , 6 6 6 ^ 4/ siant e L g N 84 L .'.* ' __

4 s m 3, y-c- l*-- 3.u*.= aua ' , - E.. s. s. Ta. * * * g j Q*"" %

I l g . l ,s 7 3 s 4.u..suo aua . , t== fe==.* O g==O }. O t . s. s. Ym. + c*

e

I = -'= C

.sQ3 ,g, 7 twee scio.= t )-( %

• ,

, Q**" o 6C4 af , = ,

a _ g . w staal oha 7.mp*f l sp3 . N p.rs .? 'I g e. - i

'hne.epf e

,, .i . N ' _. 3 C . 6DC 6O6 ~ {

o 5 p l [f

[,ed=h r0 6 g M s. 9 .T T . A.c. t .

• T 'r6,0 M "

4 5 , b*c E g .' 92o !-=D 6 p . h $ -' 7- ? Ya6 6L ~ \\ * ,s <

• 10

- ~. p. % . % > c, l s

o < wns c_ N - g, s.s. 3 .

-- 666 o ~ gl g a. a re .o 2.c3..soo ats. , u. s. s . E & ,s b'r-3 4 5 ~ / I

ir 'a. me. 2 % g r. , , . , , , , , , , ,

6 o M .r.o .. E $ . N a " %n. 4 s e > . { { W "*' ' 'M e asia ta. m. r 6 is't r'a".s'"n

• ' '

" t. s . s. Yn.

• p-

4 4 F , a + r..,,' . >7= l ,s . . a, t.

• r

>-< ""* T* * "6 T I

i mti tra sus to.se in. ... s ,- o- e a 4 - I - - p.-o j 55881 israe/3o ... e I R 4 w

I ed D d-.Md ,. /O r- rt ,, )I 8 2'l ' A " i - ~~ - i o

9 a eLg bL f H Sr. . 2'18 co.m. 22,.s.s. vn . .. .e . a + c- t

g

. te se on at ? W -

y .rss ....ssi n

a./ ...-s e n t. . A po ,,, ,, ,ar r eo, o =====. m ,,,,6 ., ta w een e k g g ,- 4p esc 'a' "'* e ' C f* V 1 & 6_.* Al vv YARO N j l

- ,y, k yy 5 M, cono.=L 31sa a e s .e s w - cm. s. s. in. +c.. ,y q I$

_ .,e e rs yy vn.t is a.eae .. T

• • .s .I

?o - rood =0 T0*E* 80- ' ' ' I'88***** a t ,g y4 rF y in. e. .~ ?- t_ t* . e r

• s

a ;;

y E, f 1 'I

Ea . _6_ _a.* ~a t a

g . . - . t, '

. ,

, . 4 . l t APPENDIX 1 -

OCTOBER 16, 1992 CONTROL AIR SYSTEM WATER INTRUSION EVENT '

l. Event Summary On October 26, 1992, approximately 1000 gallons of water was inadvertently introduced into the control air system which supplies both units. This water intrusion caused major secondary transients which resulted in a Unit I turbine trip / reactor trip, and a Unit 2 runback. The cause of the water intrusion was , condensate accumulation in the #2 receiver tank, due in part to an ' obstruction in the drain line. This event is described in NRC

Inspection. Report 92-34. , 2. Personnel Performance A review of documentation associated with this event and interviews with selected personnel involved indicate that

individuals involved responded appropriately. -! 3. Equipment Performance During the Event ! The water intrusion resulted in malfunctions of numerous . controllers and valves in the condensate and feedwater systems. t In addition to the equipment affected by the water intrusion, there were numerous examples of secondary equipment that did not , respond as expected during the event. , 4. Procedures Associated With the Event j 4 The procedures associated with this event were appropriate. , Procedure compliance was satisfactory. t I 5. Licensee Corrective Actions The licensee's Incident Investigation (II) for this event- l (S-92078) concluded that the root cause of water intrusion into ! the control air system was condensate accumulation in the #2

receiver, due to a sediment buildup that obstructed the drain ! line. The Incident Investigation further concluded that the l accumulation was due to ineffective blowdown technique. The

source of the sediment was corrosion products from the control air r system. The licensee's corrective action consisted of modifying h their receiver blowdown technique, writing work orders to correct i _ the secondary equipment that did not respond as' expected during ! the transient, and writing a PM to clean the inside of the- l receivers on a five year frequency. l i ! , i . r - - - . . . - , .

. -. .

I ,' i ..' i 2 The II incorrectly concluded that there_were no precursors to this. event and the potential for this event was unforeseeable. During i the II, the plant manager directed.a separate, independent ! investigation be performed by ISE to evaluate the adequacy of

SQN's response to this event. The ISE evaluation concluded that a t more thorough evaluation of previous drain line problems might have prevented this event. Additionally, the ISE identified that there were safety-related equipment on Unit I (6 locations) which needed local particulate filters installed (already in place' for ~ the same components for Unit 2). This corrective action had not been identified in the original II. i 6. Conclusions - The probable root cause for the sediment accumulation in the . receiver drain line is inadequate preventative and l ' corrective maintenance. ! - A more thorough evaluation of previo.s drain line events -! could have prevented this event. j , The ISE independent review recognized that there were ! precursors for this event, and provided sound corrective l action recommendations.

i '

i [ ! , " , i i - ! .l ! ,

! 8 J l l i . ,

, . . .

. . . ' . 4 APPENDIX 2 DECEMBER 15, 1992 DUAL UNIT RUNBACK DUE TO LOSS OF CA

! 1. Event Summary ' , On December 15, 1992, the plant experienced a reduction in control air pressure with both units at 100% power. The partial loss of i air pressure caused certain components to realign to their " loss of air" position and as a result, both units experienced turbine runbacks. The event is described in detail in Inspection Report 92-36. The team reviewed the licensee's II for this event and evaluated the licensee's root cause analysis and corrective actions implementation. The licensee's Incident Investigation S- 92-094 for the event was completed January 14, 1992. 2. Personnel Performance Control Room personnel and AU0s responded effectively to the transient despite the lack of control air system alarms and indications. The inspectors reviewed the training available to operations personnel concerning failures of the control air

systems. Initial training for licensed operators emphasizes the

complexity of a loss (total or gradual) of control air event and ! indicates that the operator response is' impeded by an inability to predict the exact sequence of component failures and provide precise pro:edural guidance. The requalification system module and job performance measures for non-licensed operators provide a

good description of the local operation of the control air ' compressors and dryers. Licensed operator requalification cycles have included scenarios involving loss of essential and non- essential air events. The team concluded that licensed and non- ' licensed operator training was adequate to cope with a loss of air ! event. ' , 3. Equipment Performance The control air compressor sequence selector switch was first identified as deficient with the initiation of Work Order (WO) 92-12610-00 on August 17, 1992. The WO was assigned Priority 2

status for immediate action by the work control organization. Priority 2 items are those that " hinder or has the potential to i hinder station operation." The action identified was to troubleshoot, repair or replace the switch. The switch is not ' carried on the vendor list of recommended spare parts and due to the age of the part, significant delay was experienced in obtaining a replacement.

! The onshift Unit ASOS and SOS revised the priority of the WO to t Priority 3 and deferred the troubleshooting of the switch due to l the risk of tripping both units if control air was lost. The WO

was deferred at that point (September 1,1992) until the system i , '

I r-m . . . . . -

a - , . . , . 2 engineer developed a detailed trcubleshooting action plan. A plan , was developed, but troubleshooting was again deferred by the operations staff until the required parts were obtained. A series

of problems delayed receipt of the proper replacement switch. .The dual unit transient occurred on December 15, 1992, before the

correct replacement part was received and installed. ' 4. Procedure Review The team reviewed the procedures available to the operators for -

responding to the loss of control air. Procedure A01-10, Loss of Control Air, provides guidance for controlling the units in a shutdown condition following a loss of nnn-essential air. The procedure assumes that the loss of air has been of sufficient magnitude that the unit (s) has(have) tripped. The procedure does not provide guidance for use during a gradual loss of control air. Given the difficulty in predicting the sequence of equipment failures during a gradual loss of air, the team found the approach of A01 10 adequate. However, the licensee's recent loss of air experience provides an opportunity to improve both training and

the A01's based on the observed sequence of events. The team noted that the licensee's II did not address the adequacy of procedures ner did it recommend that training evaluate the event for possible program improvements. 5. Licensee Corrective Actions In II-S-92-094, the licensee identified the failure of the air compressor sequence selector switch due to loose internal parts as the direct root cause of the event. The II also described several contributing factors including delay in completing WO 92-12610-00, i weakness in risk management of the degraded control air system and lack of preventative maintenance on the selector switch. The licensee further identified that the lack of control room indication for the control air system hindered the' operators ability to respond to the loss of control air in a timely fashion. The licensee identified four corrective actions in response to the , weaknesses described above. A " Degraded /Out of Servic( Equipment with Significant Risk" list was developed to provide a vehicle for - routinely identifying outstanding, risk significant work. items to senior site management's attention. The list is currently maintained by the Work Control Supervisor and published daily in the Plan' of the Day. It is intended to provide'a means for the work control and shift operations staff to asses the risk of performing-or deferring work items and to establish an appropriate work schedule. In addition, the-licensee has recently initiated the Secondary Plant Reliability- Study (see Section 5.B) as one of the corrective action in II-S-92-094. . Finally, the licensee will- + evaluate the need to install additional control room indication for the control air system. ! h i

- .. . . i

. ! . '

. t 3 ! i The team evaluated the weaknesses identified by the licensee in ! the II. The scope of the weaknesses and corrective actions ! identified were appropriate. However, the team concluded that the ! design of the control air control circuitry prevents routine i preventive or corrective maintenance on the control panels. ! Because of the restrictive design of the control air controls, the , licensee deferred necessary corrective maintenance which in turn i led to the aforementioned transient. In addition, the Secondary l Plant Reliability Study will need continued support at all levels ! of the organization to achieve effective plant. improvement. i ! !

i

1

i f I ! ! ! I i 1 i , ! " ! I 3 4 E J ,. -. - - -_,2 - , - - - . +

. . ' . . APPENDIX 3 NOVEMBER 20, 1992 LOSS OF SWITCHYARD INTERTIE TRANSFORMER 1. Event Summary , On November 20, 1992, CG personnel were changing the settings on Intertie Transformer Bank Differential relay 587T, for the Plant Bowen line upgrade. Personnel error in failing to install an i insulating spacer introduced a ground on the 250 VDC system. Additional personnel errors during troubleshooting for the ground caused an inadvertent trip of the Intertie Transformer Bank Feeder Differential Relay at 08:20 a.m. which opened PCBs 5054,5058,934, and 938. 500KV Bus 1, section 2 PCB Phase discordance alarm was ' received when PCB 5054 mis-operated. A 161KV switchyard grid ' undervoltage alarm was received. Two CG engineering unit personnel who were performing breaker timing tests on PCB 5064 heard PCB 5054 leaking air. Operations personnel entered LCO e 3.8.1.1.C for loss of both offsite power circuits. PCB 5054 automatically closed due to low air pressure. A switchyard electrician erroneously notified the dispatcher that PCB 5058 was ' leaking air rapidly. The air leak was in fact from PCB 5054. The , dispatcher instructed the SQNP ASOS to close PCB 5058. Closing PCB 5058 placed a load of approximately 1500 Amps through_ PCB 5058 and 5054. With low air pressure PCB 5054 had low dielectric strength and began arcing and one head caught on fire. The dispatcher was notified by the CG electrician that PCB 5054 was on fire and the dispatcher told the SQNP AS0S to open PCB 5058 which extinguished

the fire on PCB 5054. At 10:13 a.m. the switchyard grid , undervoltage alarm cleared. At 11:17 a.m. the plant exited LCO 8.3.1.1.C for units 1 and 2. Intertie transformer bank was back- ' in service at 05:07 p.m. NRC Inspection Report 92-34 provides a discussion of the event. ' , > 2. Personnel Performance ' This event was plagued with personnel errors. The CG technician's failure to properly assemble the relay after changing relay , settings created the ground on.the 250 VDC system. Proper post maintenance testing would have detected the ground. i Troubleshooting the ground ultimately led to the inadvertent trip

of PCBs 934, 938, 5054, and 5058. Performance of inadequate post i maintenance testing due to lack of communication of TC0 switch , status caused the plant to believe that the ground had cleared. , Lack of communication during job turnover on ground " troubleshooting caused further unnecessary troubleshooting. , Incorrect communication between switchyard maintenance personnel and the dispatcher led to incorrect PCB operation causing the switchyard fire. . W - -

. . - . - - _- ! . . ' . ' 2 , The team reviewed the training records for several engineering i < unit personnel and found that the training was being conducted per

the requirements of the CG QA manual. The dispatchers had ! received training in the implementation of the Sequoyah switchyard l letter 18. The team reviewed the steps to assure proper grid conditions required in the Sequoyah letter and determined that the

, dispatchers could complete their portion of the actions within ten ! minutes. However to rapidly stabilize the grid following a loss of

the Sequoyah units the dispatchers must have certain combinations of generatir,g plants on line or in standby status and certain 500 ' KV grid intertie transformers available. This means that performing scheduled or corrective maintenance on this equipment , ' can affect SQNP grid stability. The dispatchers must closely coordinate these activities with SQNP operators so that activities

at SQNP affecting the SQNP intertie transformer bank are not performed when critical grid maintenance evolutions are in progress. The team noted that while the dispatchers appeared knowledgeable in the switchyard letter actions they did not ' routinely communicate grid status to the SQNP operators. On November 20, 2992 the Raccoon Mountain intertie transformer was . ' off line for maintenance yet dispatchers did not inform SQNP operators. Both SQNP operations personnel and CG dispatch ' personnel must be aware of grid conditions to properly manage high risk activities at SQNP. On November 20, 1992 , the SQNP , operators and the CG dispatchers allowed work on the switchyard , ' 250 VDC control power system while the Raccoon Mountain intertie transformer was undergoing maintenance. The Raccoon Mountain

intertie transformer is explicitly called out as being required ! for operation with the SQNP intertie in or out of service. Because of personnel error under these conditions SQNP was in LCO ' 3.8.1.1.C for a period of approximately 2 hours. The dispatchers ' and the SQNP operators should have recognized these conditions and l 1 not allowed maintenance which could have affected the SQNP

) intertie transformer bank until the Raccoon Mountain intertie ! transformer repairs were complete. Approximately 2 hours were

4 required to complete the repair. The operations and CG dispatch , i ' personnel did not fully understand the risk activities for the evolutions in progress. 3. Equipment Performance ' , The equipment performed as expected except for a PCB 5054 Hitachi 500 KV breaker. The Hitachi breakers at SQNP are the only ones of

their kind in the U.S. and were original plant equipment. The ~ inspectors reviewed the maintenance program and history for the ' breakers and concluded that the reliability for these breakers has

been poor. The Hitachi breakers are air blast type breakers and have required more maintenance than other air blast type breakers

at TVA. PCB 5054 was overhauled during October 1992 and yet ! experienced internal air leaks, mis-operated and shorted causing a , fire on one of the breaker heads. Improper maintenance practices

' i !

-. _ - -. - - ,.

.

.. 3 were indicated by the licensee as a possible contributor to failure. Originally the plant's electrical maintenance personnel were responsible for performing the maintenance on these breakers. During 1989, the CG was assigned the responsibility for the maintenance on this equipment. Currently the CG is still responsible for the Hitachi breaker maintenance. Possible interchanging of parts, and incorrect use of lubricants deviated from the vendor manual recommendations and contributed to the failure. Once the breaker mis-operated the proper alarm was generated. Once air pressure was lost the PCB attempted to close automatically as designed however the breaker failed to fully close. Subsequent application of load through the partially closed breaker caused an arcing fault and a braaker fire. 4. Procedure Review The team reviewed the procedures and documentation pertinent to the event and determined that lack of detailed procedures was a contributor to the problems experienced. Had there been explicit step by step instructions utilized to control the work activities the CG technician would most likely have correctly assembled the protective relay and not introduced a ground on the 250 VDC system. The work activities of CG technicians on non-safety- related equipment is not controlled by detailed step by step procedures as it is for safety-related equipment. The CG Field Test Manual is utilized to provide direction for most CG field calibration and test activities. The field test manual was reviewed by the team and found to provide general guidance and not detailed step by step instructions for performing work. , . 5. Licensee Corrective Action The team reviewed the licensee incident investigation report and found the report to be adequate. The licensee identified the root { cause of the inadvertent trip was personnel error in removing a relay case while troubleshooting for a ground on the 250 VDC ! control power system. The root cause of the ground was improper relay reassembly after relay recalibration. Lack of communication - i and incorrect communication were significant complicating factors l during the event. The root cause of the failure of PCB 5054 was

! an internal air leak caused by a defective epoxy bond on the transfer valve. i The licensee identified several possible breaker failure contributing factors as identified below: Interchanging of components in the Hitachi 500 KV breakers

without proper manufacturer's data. - i t

_____- -_ _ _ _ _ . _ _ _ _ __ __ _______. . _ _ _ _ _ _ _ . _ _ _ _ _ _ _. _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _

- ' , . ! . 4 L

Deviation from maintenance practices identified in the Hitachi vendor manual. Inadequate spare parts inventory which has led to the use of

old components. " Inadequate communication between TVA and Hitachi on 500 KV !

breaker maintenance issues since installation.

Improper lubrication and cleaning of breaker components. ~

The team concluded that the evaluation of root causes for the

event was adequate and agreed with the root cause findings. The licensee proposed corrective actions for recurrence control. Training will be provided to CG engineering unit personnel on the event particularly emphasizing proper communications and post

maintenance testing. The need for training of the CG electricians by the vendor will be assessed. CG with vendor support will evaluate the need for corrective maintenance on the Hitachi breakers. CG will evaluate the existing Hitachi breaker maintenance procedures and spare parts inventory. Long term ! replacement / modification of the Hitachi breakers to reduce ! maintenance and improve reliability will be evaluated. Additionally, the CG will evaluate the Transformer Feeder Differential Relays to ensure that they will drop their trip targets. ! 6. Conclusions The event demonstrated a weakness in communications between the i SQNP operators and the CG dispatchers. The control over high risk activities affecting the plant, switchyard, and the grid needs to - be strengthened. Training should be provided to both parties on =i the equipment and activities which pose a risk to Sequoyah operation under degraded grid conditions. CG Engineering should be consulted whenever SQNP operations and CG dispatch personnel are not sure of the possible consequences of faults or failures. The event demonstrated a fundamental weakness in work control over CG engineering unit work activities. Existing management controls

over CG work activities have not been effective as seen by the ' personnel errors, failure to communicate, and failure to perform . proper post maintenance testing. Additionally, CG electrical maintenai.ce personnel deviated from vendor maintenance . ! recommendations. The team concluded that some of the corrective actions will prevent recurrence, however the team noted that the

corrective actions did not evaluate the adequacy of the existing CG engineering unit procedural controls.

, t l . . .

. - . ! ' ,- ' APPENDIX 4 DECEMBER 8, 1992 UNIT 2 RUNBACK j . i 1. Event Summary . 5 On December 8, 1992, Sequoyah Unit 2 experienced an automatic unit { runback from 100% to 75% power followed by a manual power ' reduction to 28%. The transient was initiated when a metering ! orifice cleanout plug in a pneumatic level controller, LIC-6-106, , failed to reseat after a system engineer had depressed it in an r effort to stabilize the device's output. The controller regulates the level in the number 3 HDT. Ultimately, the system engineer's

actions caused HDT bypass valve LCV-6-105B to open which diverted ! the contents of the tank to the condenser, resulted in an

automatic and subsequent manual turbine runback, a trip of the 2A l main feed water pump and a manual start of both motor driven ~; auxiliary feedwater pumps. ! t For a detailed description of the event, refer to inspection , report 50-327,328/92-36, t ! For the purposes of this report, this event was analyzed to ! evaluate personnel performance, to determine if associated , equipment functioned properly, ascertair. the extent to which

procedures were used and their adequacy, evaluate the adequacy of ! the licensee's corrective actions, and to assess the transient's ! safety significance. 2. Personnel Performance l During the subject event, the performance of the system engineer, ! the associated operations personnel, and the responsible parties ' in the site's management organization could be improved in the - areas and for the reasons detailed below. , i The team determined that Operations personnel and system engineers

have for some time, been involved in a practice of performing l informal and undocumented maintenance on both units' condensate i feedwater/HDT level controllers and associated equipment. The ! subject events troubleshooting was performed without a work order i document. The inspectors reviewed the work performed without a - procedure and concluded that performing this type of plant

maintenance without a work procedure was a violation of the site's i procedures. Details of this violation are further explained in ! paragraphs 4 and 7 of this Attachment. This practice may be l counter productive to the overall process of achieving and .; maintaining a high degree of equipment reliability. The team also i noted that responsible levels of site management were aware of ! this practice; however, did not recognize that it was in conflict ! with site work control procedures. , l The team also noted that during the event, the operators assumed i ! -f

e - t . . 2 manual control of the main feedwater pump master controller as i well as the main turbine governor valve _ limiter and manually ran the unit back to 28% reactor power to combat a 25% turbine

runback, an event the unit is designed to mitigate automatically. ' It is the team's conclusion that the crew over reacted to the symptoms exhibited by the transient and in so doing, exacerbated r the event. The team also determined that the licensee does not have an Abnormal Operating Instruction (A01) to assist the < operators in coping with a unit runback. This fact may have contributed to the operators' actions. l 3. Equipment Performance . Level Controller LIC-6-106 The licensee attributed this particular malfunction of the level i controller to mechanical binding of the metering orifice cleanout i plug as well as dirt / rust particles in the air supply regulator i and controller relay. The team noted; however, that prior to the event, the controllers on both units had required continual. maintenance (approximately every 6 weeks on unit 2 and every 90 ' days on unit 1) for deficiencies such as setpoint drift and/or erratic controller behavior. A review of Sequoyah's forced outage history and discussions with the licensee's staff revealed that

although the problems with the level controller were well known and documented, there had been no root cause identified for the comnonent's persistent malfunction. The team also determined that adequate maintenance trending information was unavailable since i much of the associated " field tuning" work such as setpoint a changes, was performed outside the. licensee's maintenance program, employing no work request or procedure. The team noted, that even after this event, and its associated incident investigation, the i licensee had not determined a root cause for the components' -t recurrent malfunctions. Turbine Runback Logic l An analysis of the design of the logic circuit which initiates a

main turbine runback when there is a high level in the number 3 ! heater drain tank, revealed possible oversights. The 200%/ minute runback is initiated if heater drain tank bypass valve LCV-6-105B > begins to open. This valve is one of two which open to bypass _the ' contents of the number 3 heater drain _ tank to the condenser when i level in the tank is high. The design intent of the runback is i anticipatory in nature and was devised to reduce unit load to a- l point that the feedwater system can supply adequate flow to the

steam generators if the feedwater constituent afforded by the j feed-forward capacity from the number three heater drain

tank / pumps is lost. It should be noted, that the licensee performed an evaluation of the runback circuit in 1991 which

indicated that the runback, as designed, is unnecessarily challenging and had plans to modify the circuit to be more ! i s

_ - -- .. - ! . ' i , . - i 3 ' I representative of the intended protection.

2A MFP Trip The licensee performed an evaluation to determine why the 2A main feedwater pump had tripped during the transient. During this analysis, the licensee discovered that pressure switch 2-PS-54-10A ! had actuated, which is designed to initiate a feedwater pump trip

when low seal injection water supply pressure is detected. The

setpoint of the pressure switch was checked and was found to have

drifted to a setpoint 19.5 psi higher than its normal setpoint of

235 psi. It was also noted that the time delay associated with

the trip had drifted to a setpoint of 11.5 seconds instead of the ! intended 20 seconds. The licensee also determined that pressure switch 2-PS-54-10C (low seal water injection pressure alarm) had failed to initiate a low pressure alarm as designed to warn i operators of an impending trip of the feedwater pump. Calibration j checks confirmed that the alarm pressure switch setpoint had i drifted to a setpoint 24 psi lower than the feedwater pump trip setpoint. A review of applicable transient data indicates that the transient lasted less than 20 seconds. Accordingly, the licensee concluded that the changes in the setpoints for the i protective instrumentation was the reason the 2A pump had tripped. . Upon questioning the licensee about the provisions in place to ! assure that this instrumentation is routinely calibrated and i functionally verified, the inspectors confirmed that prior to this ! event, the licensee had no preventive maintenance procedures in j pl ace. ! The team also concluded that the operators exacerbated the ! transient on the feedwater pump by rapidly increasing the pump l " speed in their efforts to manually combat an event that the plant

is designed to handle automatically.

, 1 . 4. Procedure Perfortrance ! The team noted that neither a procedure nor work request were used i . by the system engineer when he was performing the aforementioned " field tuning" evolutions nor by the accompanying non-licensed operator who adjusted level controller LIC-6-105 to increase the- [ setpoint of HDT bypass valve LCV-6-105B. Through reviews of i applicable documentation and interviews of responsible parties, i the team determined that maintenance on these level controllers I has routinely been performed outside the site's maintenance !

program. During interviews, the team determined that the practice > of performing this maintenance informally and at the Operations

group discretion had evolved because the feedwater HDT level ! ' controllers were requiring maintenance (setpoint changes or other l ' minor adjustments) on an " increased frequency", and that the equipment had experienced these deficiencies for "a long period of time". The team also concluded through interviews, that responsible levels of managemeat knew this practice was occurring;

,

. - ,

f . .

r , 4 . however, did not recognize that it conflicted with site work control procedures. Site Standard Practice procedure, SSP-6.21, Maintenance Management System Initiation Of Work Requests, , specifically requires that work requests be initiated for maintenance on all installed plant equipment except for very simple tasks such as erecting a ladder. ' The team also determined that although System Operating Instruction S01-5.2, Number 3: Heater Drain Tank, allows the operators to make necessary adjustments of these level controllers - during startup and/or shutdown of the HDT pumps, there are no provisions for adjusting the equipment at power. The team determined that even though the HDT level controllers l required continual setpoint changes, there is no periodic ~ instruction for accomplishing the task. In addition, during the event review, the inspectors identified that the operators took manual control of the unit to combat a minor operational transient for which the unit is designed to . , recover automatically. This appeared to the inspectors to have ' not been necessary in recovery of the unit. Subsequently, the inspectors determined that there was no Abnormal Operating instruction to assist the operators in mitigating a unit runback. . t Due to the high potential for unit runbacks at Sequoyah, the need i for such a instruction for operators may be warranted. , 5. Licensee Corrective Action , As it pertains to this event, the team concluded that the licensee's corrective actions were deficient in the following areas. '! The problems associated with the chronic malfunction of the feedwater heater level controllers has been documented and well known to the licensee, including appropriate levels of management. ! The malfunctioning of this equipment has been the source of many plant transients, including 11 main turbine runbacks, dating back- ' to 1989. Yet, even after the most recent event, the licensee does

not know the root cause for the repetitive disfunction of the equipment nor have they adequately investigated to determine one. i The licensee identified in 1991 that the logic associated with the , main turbine runback which is initiated when HDT bypass valve LCV- t 6-105B is not full closed is unnecessarily challenging to the units and is not an adequate precursor of the transient against which it was designed to protect. However, two years after that identification, the modification to alter the circuitry has not . been installed. ' Responsible levels of licensee management were aware that the aforementioned practice of performing maintenance outside the

.

, , 5 . requisites of the maintenance program was ongoing, however, did

not recognize the performance of these practices were in violation of site work control procedures. '

Due to concerns raised during review of the December 31, 1992, l event, the licensee reprioritized the modification.to eliminate the current runback configuration. 6. Conclusions and Regulatory Significance Operations and system engineering personnel routinely perform informal and undocumented maintenance on . installed plant equipment and this practice poses a transient _ hazard for the plant. The i team concluded that managements' threshold for the performance of , work without the required work controlling procedures was not conservative and did not support compliance with site work control - procedures. The performance of informal and undocumented ' maintenance on installed plant equipment is identified as a violation of Site Standard Practice procedure SSP-6.21, Maintenance Management System Initiation Of Work Requests. This is identified as an additional example of Violation 50-327, 328/93-02-01, Failure to Follow the Requirements of TS 6.8.1. In addition, the following weaknesses were identified by the team during review of the event. , - The licensee has not determined the root cause of the need for continual maintenance on the number 3 HDT level controllers and have not addressed the repetitiveness of the , problem. - The licensee's corrective action associated with changing the logic affiliated with the turbine runback on high level ' in the HDT has not been aggressive, in that, the need for - , system modification was identified in 1991, but has yet to be implemented, even though in that period of time, there- ~ have been a number of associated plant transients. - The licensee does not have preventive maintenance procedures , in place to assure that the instrumentation associated with

certain main feed water pump trips is routinely calibratedf and functionally verified. _ - The licensee does not have in place any A01s to assist the ' operators in coping with a unit runback. l . ^ s . . w . -

. . ' ' . . ! APPENDIX 5 l

PERSONS CONTACTED Licensee Employees

• R. Fenech, Site Vice President
• R. Beecken, Plant Manager

L. Bryant, Maintenance Manager

L. Bush, Acting Operations Manager

• J. Bynum, Vice President

• M. Cooper, Site Licensing Manager
• T. Flippo, Site Quality Assurance Manager

J. Gates, Technical Support Manager - C. Kent, Rad.ological Control Manager

• M. Medford,.Vice President

. R. Rausch, Modifications Manager i H. Rogers, Acting Technical Support Manager ! J. Smith, Regulatory Licensing Manager. <

• R. Thompson, Compliance Licensing Manager

!

• P. Trudel, Nuclear Engineering Manager

l J. Ward, Engineering and Modifications Manager ~ N. Welch, Operations Superintendent i NRC Employees

!

• S. Ebneter, Regional Administrator-

,

• E. Merschoff, Director, Division of Reactor Projects

!

• A. Gibson, Director, Division of Reactor Safety

.

• G. Lainas, NRR, Assistant Director, DRP

• F. Hebdon, NRR, Project Directorate 11-4

!

• W. Holland, Senior Resident Inspector
• P. Kellogg Chief, DRP Section 4A

j

• Attended exit interview.

-l

Other licensee employees contacted included control room operators- ' shift technical advisors, shift supervisors and other plant personnel.

i f

! i i i !

i .i $ -- - - - - - . - - -

? , i . , . , .._ APPENDIX 6

, List of Acronyms and Initialisms l ABB ASEA Brown Boveri + A01 Abnormal Operating Instruction

APS Auxiliary Power System i ASME American Society of Mechanical Engineers ! ASOS Assistant Shift Operations Supervisor

AVO Assistant Unit Operator B0P Balance Of Plant , CA Control Air

CCDP Conditional Core Damage Probability . CCP Coolant Charging Pump

CDF Core Damage Frequency

CFR Code of Federal Regulations i CG Customer Group ! CIPTE Complex or Infrequently Performed Tests or Evolutions i CR Control Room CS Containment Spray CSST Common Station Service Transformer DC Direct Current DN Deficiency Number DRP Division of Reactor Projects ' EDG Emergency Diesel Generator l ERCW Essential Raw Cooling Water j ESF Engineered Safety Feature FSAR Final Safety Analysis Report FWHDT Feedwater Heater Drain Tank GDC General Design Criteria ! GPM Gallons per Minute HDT Heater Drain Tank

II Incident Investigation

IPE Individual Plant Examination .i ISE Independent Safety Engineering ISEG Independent Safety Engineering Group ISI Inservice Inspection , KV Kilovolt " LCO Limiting Condition for Operation LCV Level Control Valve .; LOCA Loss of Coolant Accident ~ .MCB Main Control Board , MDAFWP Motor Driven Auxiliary Feedwater Pump ' NOUE Notice of Unusual Event NRC Nuclear Regulatory Commission l NSSS Nuclear. Steam Supply System OCC Operations' Control. Center ' PCB Power Circuit Breaker

PERP Plant Evaluation Review Panel l PORC Plant Operations Review Committee ! PRA Probabilistic Risk Analysis PTL- Pull to Lock- ! QA Quality Assurance , ! l . - - - - - = . .

. .- - . ,

, 2

! RCP Reactor Coolant Pump i RCS Reactor Coolant System

RHR Residual Heat Removal ' R0 Reactor Operator RWST Refueling Water Storage Tank SE System Engineers ' SI Surveillance Instruction SIP Site Improvement Plan ' S01 System Operating Instruction SOS Shift Operating Supervisor SPRS Secondary Plant Reliability Study SQN Sequoyah Nuclear SQNP Sequoyan Nuclear Plant SRO Senior Reactor Operator SSP Site Standard Practice TBBP Thermal Barrier Booster Pump ' TS Technical Specifications . TVA Tennessee Valley Authority UFSAR Updated Final Safety Analysis Report URI Unresolved _ Item VCT Volume Control Tank 4 VDC Volts Direct Current i WO Work Order l . r s , i

, L i - , ,__ ___ ____ m

. - . ___ _ _ _ __ - t ! a [

s 5 !

ATTACHMENT 1 l 1 EXCERPTS OF THE 1990 NRC STAFF'S POSITION ON SEQUOYAH INTERTIE TRANSFORMER > i The NRC staff reviewed the Sequoyah Independent Safety Engineering Group ! (ISEG) report 87-10-SQN-1, "AC Power System Plant Grid Interface Review Report." The staff's evaluation on the recommendations in the ISEG report is , presented below. The evaluation is based on the 1988 configuration of the

switchyard. ! Plant Grid Interface: ! ! Based on the existing Sequoyah switchyard design and a specific plant / grid ! interface problem scenario, ISEG report 87-10-SQN-1, dated June 27, 1988,_

indicated that the requirements of General Design Criteria (GDC) 17 would not

always be met. The problem described assumes that the 550-161 KV intertie transformer is taken out of service and that the TVA grid is operating at some ' peak load condition. With these assumptions, the analysis (performed by TVA)

shows that the grid system will not reliably supply the onsite electric i distribution system loads following a second grid contingency, i.e. a full , load rejection of both Sequoyah units. Thus, the-offsite grid system will i thus not meet the requirements of GDC 17 immediately following loss of the 500-161 KV intertie transformer. However, once the load dispatcher for the ' TVA grid system reconfigures the system (i.e. adding additional generating capacity or reducing loads), the offsite grid system will' meet the requirements of GDC 17. The assumptions in the FSAR regarding grid system stability are still valid.

GDC 17 requires in part that: ! I Provisions shall be included to minimize the probability of losing 1 electric power from any of the remaining supplies as a result of, or ! coincident with, the loss of power generated by the nuclear power unit, the loss of power from the transmission network, or the loss of power

from the onsite electric power supplies. To meet these provisions, it has been the staff's position, historically, that the utility's offsite grid system must remain stable (as demonstrated by j analysis performed by the utility) in the event of loss of the nuclear unit generator, the largest other unit on the grid, or the most critical transmission line. A stable grid, following loss of the nuclear unit i ' generator or generators, will minimize the probability of loss of the remaining supplies (i.e., the offsite grid system and onsite power supplies). Likewise, a stable grid, following any single grid system contingency, will minimize the probability of loss of remaining supplies (i.e., the nuclear unit generator or generators and onsite power supplies). The licensee, in section 8.2.2 of the UFSAR states that the eight 161 KV transmission lines connected to the 161 KV switchyard, the 500-161 KV intertie tranformer bank, and the five 500 KV transmission lines (i.e. grid system) ' have sufficient capacity to supply the total required power to the plant's electrical auxiliary power system under normal, shutdown and loss of coolant -i - , y r- r -p v v - --- - - - - - - - - - - - - . - - - - .

.

. 2 accident (LOCA) conditions for any single transmission (i.e. grid system) > contingency. The UFSAR statement-(l) represents assumptions in the FSAR . regarding grid system stability, (2) meets the above staff position and GDC ! 17 requirements, and (3) is, therefore still valid. l ' Technical Specification Requirements The inoperability of the intertie transformer bank causes a reduced capacity in the grid system such that it can no longer reliably supply power at- - required voltage levels immediately to the two preferred circuits connected to the onsite electric distribution system at Sequoyah. Therefore, when the intertie transformer becomes inoperable, the grid system along with the two , preferred offsite circuits must be considered inoperable. .Since the grid system's caprcity to supply safety loads is not specifically addressed in the technical specifications, the definition of operability dictates that the Sequoyah plant should enter TS 3.0.3 LCO. However, since physical loss of two offsite circuits is defined in TS and since loss of capacity in the offsite

system causes loss of both offsite circuits, the staff concludes that it is , appropriate and acceptable for Sequoyah to enter the LC0 associated with the loss of both offsite circuits.

.! s

1 l , e % i 5 ! ! i ?

! i ,

4 t ! . - . . . - . . - .

. , . - - . . . . 1 , 1 r l ATTACHMENT 2 Most Prot)able Causes for Recent Sequoyah Operational Events i .. ! ' EVENT INITIATION CAUSE CAUSALFACTORS PROBABLE ROOT CAUSES REMARKS 12 31-92 Switchyard Electrical Failure of Switchyard Lack of Detailed Proc, Training on New Equip, lack of Adequate Mgt Control Obsolescence of Switchyard ! Breaker Fault Breaker due to Dielectric Failure Method of testing, Overall Work Controls Over important Activities Equipment Requires Equipment l Potentia 81y due to Mechanical Upgrade Damaga 12-31 92 Dual Unit Switchyard Breaker Fault and Operator Staffing / Training, Failure to Follow Lack of Adequate Mgt Control, Operators Recognized Trip & Plant Bypassing Protective Relaying Procedure / Operator Error, Procedural Weaknesses, Training / Staffing Consequence of No Seal Response Misleading Information to Oper, Not Injection & No TBBP (RCP Seal) Understanding Consequence of Bypassing Protective Relaying 80% Relay Mods May Have Unnecessanly Required Load Sheadmg 10-26-32 1 Unit Trip Loss of Control Air Pressure Other Unit Runback Water in Air Receiver, Corrosion in Drain P: ping. Lack of Adequate Predictive and Lack of System Eng Design of #3HDT Runback Logic, Air Blowdown PM's on Air System based on involvement with Problem Method, Failure to Extensively Evaluate other importance. Resolution Corrosion Related Sequoyah Problems, Adequacy of GL 88-14 Program to Resolve Industry inadequate Corrective Actions To Sharing of System impacts Both Problems Associated with CA Systems, Shared Resolve GL and Other Air Ssstem Units CA System (Design), Timeliness of Resolution Concerns. Design Review items for CA System Design (Shared CA System, Material). 12 15-92 Dual Unst loss of Control Air Pressure Failure of Selector Switch, Systern Design Makes inadequate Sensitivity to Risk Sharing of System impacts Both Runback On-line T/S of Control Ckt Risky, Lack of Spare Associated with Operating with a Units Parts Delayed Repairs (4 M), Lack of CR indication Degraded Control Switch of Impending Failure, WR Prioritization did not Recognize Risk of Failure Material Availability 11-20-92 Loss of Tripping of 500 KV Supply Bkr Lack of Detailed Procedure, Personnel Error. Lack of Adequate Work Controls Closing of Wrong Bkr resulted in Intertie Transformer Communications, Obsolescence of Equipment over important Activities Switchyard Fire (Spare Parts) (Cannibalizing of Equipment), Lack of Knowledge of Grid needs

12-8-92 Unit 2 Unplanned, Uncontrolled No Work Plan, No Procedure, Unauthorized inadequate Mgt Oversight of Control Bend on #3HDT Level Runback Maintenance on #3HDT Level Equipment Manipulation, Sys Engineer Not Maintenance Activities in the BOP Too Restrictive (Design) Controller Understanding Role, Failure to Document / Trend Equip Prob Nonstandard Runback Logic (Designi . Configuration Control items Associated With CS & RHR HX Throttle Velves, the 12-24-92 RWST Cooldown, and the Failure to Open the CS Pump Suction Valve Prior to Mode Change Were Not . Reviewed as Part of This Evaluation Since They are Being Reviewed as Part of Recent Enforcement Actions. . . _ . _ _ _ . , _ _ _ . _ . _ _ _ _ _ ~ . _ _ . , _ . _ . . _ . _ . . . . ~ - . . . - . - . . _ . _ ~ . _ . _ . _ _ , . . - . _ }}