IR 05000315/1985035

Insp Repts 50-315/85-35 & 50-316/85-35 on 851029-1108.No Violation Identified.Major Areas Inspected:Failure of Unit 2 Reactor Trip Breaker to Actuate During Unit Trip on 851029

ML20151L477
Person / Time
Site:	Cook
Issue date:	12/19/1985
From:	Norelius C NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION III)
To:
Shared Package
ML20151L475	List: IR 05000315/1985035 ML20151L468
References
50-315-85-35, 50-316-85-35, NUDOCS 8601020021
Download: ML20151L477 (16)

Text

. .

U. S. NUCLEAR REGULATORY COMMISSION AUGMENTED INCIDENT RESPONSE TEAM Reports No. 50-315/85035; 50-316/85035 Docket Nos. 50-315; 50-316 Licenses No. DPR-58; DPR-74 Licensee: American Electric Power Service Corporation Indiana and Michigan Electric Company 1 Riverside Plaza Columbus, OH 43216 Facility Name: Donald C. Cook Nuclear Power Plant,' Units 1 and 2 Inspection At: Donald C. Cook Site, Bridgman, MI Inspection Conducted: October 29, 1985 through November 8, 1985 Team Members: W. L. Forney, Chief C. W. Hehl, Chief Section 1A, DRP-RIII Section 2A, DRP-RIII J. T. Beard D. L. Wigginton Operating Reactor Licensing Project Assessment Branch, NRR Manager, NRR

~

-V. D. Thomas G. J. Toman, Consultant Engineering / Generic Franklin Research Center Communications,IE:HQ B. L. Jorgensen J. H. Neisler Senior Resident Inspector, Reactor Inspector, D.C. Cook Region III J. K. Heller T. E. Taylor Resident Inspector, Reactor Inspector, D.C. Cook Region III C. L. Wolfsen E. R. Swanson Inspector, Senior Resident Inspector D.C. Coo Palisades c .glYh C / 2//Mf1

-

_ Approved By: C. E. Norelius, Director Division of Reactor Projects Date Region III B601020021 851219 *'

'PDR ADOCK 05000315 G PDR -

L

. .- .. .. .- .. _ . -.. - - .- - .

N

, . .

.

.t l l

l TABLE OF CONTENTS page i General Discussions ...................................... 3 1 Reactor Trip Review i' . Sequence of Events .................................. 4 Plant Parameter Review . . . . . ........................ 5 Operator Actions . . . ................................ 6

, t i . Source and Promulgation of Trip Signal . . . . . . . . . . . . . 6 i r

' I Reactor Trip Breaker Evaluation Breaker History . . . . ................................ 7 j- Breaker Failure Investigation ....................... 10 i

4. ' Confirmatory Action Letter . . . . . . ........................ 13 i

i l ' Pe rs o n s ' Co nt acted ' . . . . . . . . . . . . . . . ~.' . . . . . . . . . . . . . . . . . . . . . . . 15

. Open Items .............................................. 15 t Unresolved' Item ......................................... 15

' Additional NRC Actions .................................. 15-i

Conclusion .............................................. 16

.

T l

4

'

,

t

'

i

\1 .

-

- s

'

e g W

1 ( y w -

4' r' T gy,f W w- y V W v -

97 % V Mw-7- D w

- . - - - . - -... _ ~.--- . . -. - _ . .

. .

,

I General Discussions f

. On October 29, 1985 the Donald'C. Cook Unit 2. plant was operating at approximately 80 percent of full power, having just stabilized at that

level following a gradual escalation from 50 percent. The Unit had been

.in service for about a week, following an extended outage begun in mid-July

- 1985 to identify and repair leaking steam generator tubes. The bulk of

this previous operating week.had been spent with the plant at 30 percent power, while steam generator chemistry was adjusted within specifications i

"

and a boric acid chemical flushing program of the steam generator secondary side was performed. Unit 2 steam equivalent to about 1.5 percent power was being utilized'for Unit 1 system testing. Re' pairs to a containment

- radiation. monitoring instrument (ERS-2300) were in' progress, as was a test of the steam generator level set II mismatch portion of the reactor protection syste At 1:58 p.m. EDT the reactor tripped, from an indicated loss of reactor

coolant flow on loop "B". No such loss of flow was apparent in the main

! control room, nor were any activities known to be in progress which might be expected to affect either reactor coolant flow, the reactor coolant pumps, or the instrumentation for monitoring these item The operators commenced post-trip recovery, stabilizing the primary and J ..second ary.p_an l t A few minutes into the recovery, the reactor operator

' observed that one of.the two reactor trip breakers (RTB "A") had not t -opened. Further, the anticipated motor operated main feedwater isolation valve closure from reactor trip coincident with low average primary coolant temperature had not occurred for two of four main feed lines. However, since the feedwater regulating valves had closed in

response _.to the feedwater isolation signal, the failure of the two main i feedwater isolation valves to automatically close did not pose any l operational concern The licensee notified the Nuclear Regulatory Commission, pursuant to 10 CFR 50.72, of the.RTB failure'and the apparent partial feedwater

,. isolation failure. The root cause of the reactor trip was as yet .

' unidentified at the time of.this notification. Due to the potential i significance of the RTB failure the NRC conducted several telephone

'

discussions with licensee management concerning the' necessity for preservation of evidence pending:a thorough, disciplined investigation

.

into the unanswered questions ~ surrounding this event. . Following these

. discussions on October 29, 1985, the'NRC dispatched to the site an

'

. Augmented Incident l Response Team (AIRT). The.following day, October 30,

'

'1985, a Confirmatory Action Letter was sent to the licensee' detailing

. certain commitments made by licensee management during the abova

,

. telephone discussions.

I This: report details the' activities'of_the AIRT,-the licensee and their

! respective consultants in gathering information for developing conclusions

<

'

concerning the cause"and course of the automatic reactor trip, the actions

- of the plant' operators during this event, the cause'of'the RTB failure' ,

and associated Confirmatory Action' Letter activitie t

'

I

, *

- 3,

~

'

'

t

• , e -

J -- ,. e--,, - ,. ,,,p-w - - y , , 9 -n -w w,n,,,, y -,- e

.

. .

2. Reactor Trip Review Sequence of Events The following edited sequence of events was derived by review of the Plant Computer (Prodac 250) and~0perational Sequence Monitor (OSM) output information on binary parameters (components on/off, open/ closed,etc.). Detailed evaluation of the Prodac 250 and the OSM outputs establish the following composite time-lin msec turbine stop valve "A" shows closed reactor coolant pump No. 22 breaker shows open reactor coolant low flow above 30%-TRIP SIGNAL reactor coolant pump bus under frequency (first of four repetitions)

reactor coolant pump No. 22 breaker shows closed reactor coolant pump No. 22 breaker shows open reactor coolant' low flow above 30%-TRIP SIGNAL reactor coolant pump No. 22 breaker shows closed turbine stop valve "A" shows open 100 msec reactor trip breaker (RTB) "B" trips 300-500 msec reactor trip turbine trip and P7 left circuit turbine trip actuated power range negative rate trip-reactor trip

" West" and " East" main feedpump turbine trips i right circuit turbine trip actuated 556 msec RTB undervoltage, "A" de-energized 565 msec RTB undervoltage, "B" de-energized 0.6 sec-1 sec generator motoring indicated turbine stop valves A, D, B and C show closed 3.15 see turbine trip - reactor trip 4-5 sec loops 1 and 3 feedwater flow low mismatch reactor trip

! 5.5 - 6.2 sec all four steam generator. lo-lo level trips 6.5 sec loop 2 feedwater flow low mismatch reactor trip 29.7 sec unit differential - main generator trip Discrepancies or peculiarities noted in review of the sequence of

.

events'(binary) items included the following:

l (1)' Although recorded by the P250,Et he OSM failed to record the l ~ expected power range negative rate trip signal following the

l l

- -

. . - -

. .

actual reactor tri Subsequent investigation of this item identified a defective OSM circuit " card". The defective circuit card was replace (2} Despite the Prodac 250 and OSM outputs, no actual low flow or reactor coolant pump No. 22 breaker "open" condition occurre The spurious solid state protection system (SSPS) input signals to this effect were traced to a momentary voltage reduction on the power source carrying the relays for these actuation (This is discussed further in Paragraph 2.d). The early (first 100. msec) turbine stop valve " closed" indication and RCP bus under' frequency indications are also attributed to this momentary voltage reductio ) The RTB undervoltage (breakers A and B coils de-energized)

indications came in near the end of the 200-600 msec range observed in previous trip reviews, but both came in only a few msec apar b. Plant Parameter Review The following information was derived by review of selected strip charts and the output of the Prodac 250 computer monitor concerning analog data and major plant system parameters.

,

Reactor coolant system temperatures, pressurizer pressure and pressurizer level dropped substantially post-trip. Temperature went as low as 510 degrees F, pressurizer level went below the setpoint (17%) for heater trip'and the pressurizer heaters automatically secured;.and pressure dropped from about 2220 to about 1980 psi All of these parameters were affected by the relatively low decay heat in the reactor core, the continued excess heat removal occasioned by the supply of steam to Unit 1 via the auxiliary steam header for heating and for overspeed trip testing being performed on the Unit 1 main feed pumps, and the full-flow of " cold" auxiliary feedwater for about the first two minutes. All parameters recovered normally when the supply steam to Unit 1 was secured (the main steam isolation valves were closed) and the turbine-driven auxiliary feedwater pump was secured, both motor-driven pumps were left in servic Steam generator levels and steam pressure behaved normally for a -

high power trip, though steam flow and feed flow channels did not all ." track" each other correctly on the Prodac 250 computer. Job Orders (the licensee's method for initiating maintenance) were written for-correctio Feedwater isolation valves FM0 201 (loop 1) and FM0 204 (loop 4) did not close as initially expected with reactor trip and low average primary coolant temperatur Subsequent investigation-determined the " reactor trip" input to the control logic for these valves comes from auxiliary contacts (which close when the breaker opens) on the

"A"' reactor trip breaker. Since RTB "A" failed to open, the auxiliary

, - __

. . - - . . - - _ - . .,

. . ,

a

,.

, . .

.

-: contacts were never closed and the valves did not receive that portion of their required actuation logic. Thus,.they did perform as designed. As'noted above, all four loop feedwater regulating

-

valves ~(which are not. single-train oriented) did isolate.

i Operator Actions f At the time of the reactor trip, the. Unit 2 Control Room was staffed

~

by'a Unit Supervisor(SRO), a Reactoc Operator No. 1 (RO) and a l Reactor Operator do. 2 (under training). This staffing met facility j < license requirements. The operators apparently had no prior 1 indication of any problem. They had just stabilized reacter power at'about 80 percent'following a several hour ramp-up in power (approximately 3% per hour) from-50 percent. All major centrol

systems were in automati i'

The first problem indications .noted by the operators .were; multiple alarm actuation ~s followed closely by rod bottom lights, indicating l a reactor trip. The Reactor Operators No. 1 and No. 2 focused their attention on recovering the primary pint and its support system Auxiliary feedwater automatic start was verified effective, then  !

focus went to recovering pressurizer level, restarting pressurizer

'

heaters and restoring normal letdown. The Unit Supervisor after verifying correct automatic actions began working to recover steam  ;

generator levels.. He isolated the~ main steam stop valves to limit

'

1 the primary system cooldown ~until various steam supplies could '

be. secured.

i-l- A few minutes after the t'eip, an operator noted that the "A" reactor trip breaker had not opened. By this time the Operations Superintendent

'

(SRO) was present in the control room. Following discussion with the Unit staff, he made the decision not to trip RTB "A"

using the manual

'

" shunt" trip' from the main control room. This decision was consistent  !

L with approved licensee procedures for post-trip l recovery; i.e., there

!- is no procedural requirement to use the manual shunt trip following a i automatic reactor trip provided all rods. indicate in and the reactor e

is subcritical.

i . .

-

In the opinion of the AIRT,' operator a'ctions durir.g and following this ~!

event were prudent and correc ,

i-

_ So'urce and Promulgation of Trip ~ Signal-

, - - .

.

The reactor trip initiating' signal was an indicated reactor coolant.

. . loop-low flow (from the' reactor coolant pump No. 22 breaker "open"  : relay)-above P8 (30 percent power). _.This was;a consequence, as were

[

several other false early event indications, of a momentary voltage 4

-

reduction on Contro1~ Room Instrument Distribution (CRID) inverter-No.E2.- 1CRID II became.an_early suspect in the' licensee and AIRT U reviews.because it is.the Class IE power supply associated with the

! -RPS relays.which actuated. Protection Set II (testing activity.in ,

progress), and radiation monitor ERS 2300 (which was being worked on at the' time of-the event were also suspect). Also pointing to problems _with CRID II was the observation that the Prodac 250 and

~

OSM outputs had a 'similar appearance to previous reactor trip events ,

involving loss-.of a CRID inverte .t

,

._w.- . , ~ - . -,, . - - _ . , . . . a, _ _ , ,J,--- ,. ,,,R- 4-,

l

. .

Other early oncerns involved the potential that the solid stat protection system (SSPS) had not correctly transmitted a protective signal to the "A" reactor trip breaker, or that there was a relationship between the failure / problem which initiated the spurious low reactor coolant flow signal and the subsequent failure of RTB

"A" to actuat The SSPS was verified to have provided a proper actuation signal when it was established the undervoltage trip attachment on RTB "A" had, in fact, actuate The sequence of events record shows the de-energization of the undervoltage coil, and separate independent verifications were subsequently made which established the indicated coil voltage in the control room panels and the coil voltage measured locally at the breaker panel were both essentially zero. This information separated the two issues of unknown source /cause of the trip and failure of the RTB to actuat Prior to release of the SSPS for general troubleshooting, selected

"as found" determinations of proper system functions were made by performance of the following surveillance tests:

• • 2 THP 4030 STP 145 " Reactor Protection Logic Surveillance"

.

• • 2 THP 4030 STP.137-1 " Reactor Coolant Pump No 2 Bus Under-frequency Bus 2 C Surveillance Test"

• • 2 THP 4030 STP.188 " Reactor Coolant Pump Breaker Position Reactor Trip Test" Successful completion of these tests, as verified by the AIRT, confirmed that the SSPS functioned correctl In pursuing the cause of the trip signal, the licensee developed investigative procedures involving the SSPS Protection Set II, ERS 2300, and the CRID II inverter. These procedures were discussed

with, concurred in, and selectively observed by members of the AIR On November 3, during continuing investigations of ERS 2300, which had been returned to service just prior to the reactor trip event, but which had failed again November 1, a licensee ~ Control and Instrument technician identified a degraded (scorched) and intermittently failing transformer-to the " photo-helic" control flow circuitr This transformer is powered from CRID II. It is believed that the " heavily loaded" CRID II (normally operating near full-amperage capacity) suffered a momentary voltage drop during intermittent shorting of the ERS 2300 control circuit transforme 'This monentary ' voltage drop was apparently of sufficient magnitude (about 60 v.) to actuate some of the smaller, more sensitive relays powered by the CRID II in'erter, v including the " reactor coolant pump breaker open" relay which precipitated the low flow reactor tri . Reactor Trip Breaker Evaluation Breaker History In review of the Donald C. Cook Unit 2 reactor trip breaker "A" failure event of.0ctober 29, 1985 the AIRT reviewed the following documentation:

g -- , r -a w

.

. .

• Condition Report 12-07-85-1445, dated July 23, 198 A crack was discovered in- a D8-50 stationary contact pole bas * ~ Job Order 43332, File EE-SG-DB-50, dated July 27, 198 " Perform acceptance test on DB-50 breakers for Unit 2. Perform applicable portion of test procedure 12 MHP 5021.082.023."

• Receipt Inspection Report ASP 8007, dated August 20, 198 D.C. Cook receipt inspection report for refurbishing Unit 1 DB-50 breaker * . Job Order 14087, File EE-SG-DB-50, dated August 20, 198 " Reactor trip breaker "A" smoked trip coil."

• Condition Report 2-08-85-1572, dated August 20, 198 " Reactor trip breaker "A" had smoke coming from inside."

• Receipt Inspection Report ASP 7949, dated July 29, 198 D.C. Cook receipt inspection report for refurbishing Unit 2 DB-50 breaker '

• **2 THP 4030 STP.144, Revision 1

" Reactor Trip Breakers Surveillance Test"

'

• **2 THP 4030 STP.180, Revision 1

"SU(1) Instrumentation Checks Prior to Startup"

• **12 MHP 5021.082.023, Revision 1

" Maintenance, Inspection and Repair Procedure for Westinghouse Type DB-50 Air Circuit Breakers Installed as Reactor Trip Breakers and Reactor Trip Bypass Breakers."

During training / dry-runs to perform **12 MHP 5021.082.023 in mid-July, 1985 while both Units were shut down (Mode 5/6), maintenance personnel

,

identified and documented that a DB-50 stationary contact pole base exhibited a crack on the top of the base plate. As a result, on about July 25, the Unit 1 and Unit 2 reactor trip' and trip bypass breakers (8 total) were sent to the vendor (Westinghouse) for refurbishment. The refurbishment included: replacement of twenty-four (24) pole bases; replacement of eight (8) undervoltage trip attachments; installation of four (4) trip counters (Unit 1 only, Unit 2 to be

_

performed.at a later date); and, additional parts as determined by inspection at the refurbishment facilit The Unit 2 breakers were received back at the Donald C. Cook plant site on July 29, 1985 and the Unit 1 breakers were received back at the plant site on August 19, 1985. The receipt inspection for-the breakers (Unit 1 and Unit 2)

consisted of inspection for: storage / packaging compliance; quality-l l verification; surface contamination; shipping damage; nameplate verification; and review of the Westinghouse Quality Release ,

documentation. All of the breakers. passed receipt inspectio '

>

- - . ... -. . . . -. . _

-

r i

. .

During acceptance testing on July 30, 1985 for the Unit 2 DB-50 breakers using ** 12 MHP 5021.082.023, maintenance personnel found

additional cracked base plates on each of.the four breakers. The base plates.were replaced at the plant by Westinghouse personnel

Following this on-site base plate replacement the Westinghouse j personne1' witnessed successful-acceptance testing by D.C. Cook i personnel. During this inspection, no Job Order, plant approved procedure, or Westinghouse procedure documenting these repairs wert available for review by the inspectors. On November 8, 1985, the j Maintenance Superintendent informed the inspector that the repair

,

was apparently not done by a procedure, however, it was accomplished ,

1. by a qualified technician under supervision of a lead engineer and

! in the presence of Westinghouse quality assurance personnel. Failure i to document the above identified on-site repair by Job Order or approved procedure may be a violation of plant administrative requirements. .This is an unresolved item pending additional review.-

(Unresolved Item (316/85035-09)).

During the AIRT on-site investigation, the licensee was asked to

,

obtain/ provide detailed Westinghouse documentation of the July 1985 refurbishment-at the Westinghouse facility. This information was

~

t provided to the resident inspector on November 6, 1985. Inspector

! review of the licensee'_s procedure used during the on-site acceptance testing determined that steps in **12 MHP 5021.082.023 pertaining to lubrication.of the UVTA were' marked "not applicable" because "the ,

! UVTA was lubed at.the factory, therefore steps pertaining to before

lube do not apply". The Westinghouse refurbishment document reviewed i

by the inspector stated that lubrication may be necessary if UVTA test l results are unsatisfactory, h~owever, the refurbishment document did

not require lubrication'and did not require verification ~that the

[ UVTA.was lubricated. The inspector questioned if the "not applicable"

notation in Procedure ** 12 MHP 5021.082.023'was appropriate, since I

,

the licensee did not have documentation that the UVTA's were lubricated at the factory. This is an open item pending_further review by-the resident' inspectors (0 pen Item 315/85035-10). Other than-the cracked base plates (identified by the licensee), the lack of documentation for onsite repair (identified by the inspector) and

'

!

,

' questions pertaining to lubrication of the UVTA (identified by the t:

inspector) no additional problems wer2 identified during acceptance

,

testing.

1 A review of equipment-history for the reactor trip breakers since

,

refurbishment determined that during control rod drop timing testing j- _on August 20, 1985, the Unit-2 reactor trip breaker _"A" was found to y have hadismoke coming from'inside the breaker cubicle._-Condition-

.

i

'

Report 2-08-85-1672 and. Job Order 14087'were written to' investigate .

and effect' repairs. The. condition report' documented that the !

closing coil was producing the smoke. The reason for the smoking ;

, coil could.not be definitely determined, but overheating due to a ;

misaligned or_. hung-up shutoff contact (which kept the co !

,

,

continuously energized)'was suspected; these. parts were adjusted.

3 -(A functional test.(**12 MHP 5021.082.023) of the_ reactor trip i

breaker was satisfactorily performed and the trip breaker declared !

-operabl

t E , _

_9 f

.

-

. .

. -

- - -

.

.

. ..

. .

The inspector also reviewed the Unit 2 bi-monthly breaker surveillance test (**2 THP 4030 STP.144) performed during the period of January 1985 through October 1985 and the Unit 2 startup breaker surveillance test (**2 THP 4030 STP.180) performed on October 22, 198 These surveilloces were performed on the required freonency. The inspector noted that,St<ps 7.11 and 8.11 of STP.144 require verifi. cation that the UVTA voltage meter indicates approximately 0 volts. It is unclear why the procedure does not specify an acceptance value or range. For-examo'e, the range of values recorded for Step 7.11 was 0 to 1 voP . 1.c equivalent steps in STP.180 specify an acceptance value

f

nc t' a '5 volts. This was discussed with licensee managemen b. Breaker Failure Investigation At 1:58 p.m. EDT on October 29, 1985, the Unit 2 "A" reactor trip breaker failed to operate on receipt of a valid trip signal from the reactor protection system. The in-series "B" reactor trip breaker functioned correctly and the reactor was shut dow Immediately following the trip, personnel from the site resident inspector office and the Palisades resident inspector office

,

verified the licensee's initial actions and assessed the status of the plan i addition, agreements were reached,.via telephone, between Mr. C. W. Hehl (NRC Section Chief - Region III) and Mr. W. G. Smith, Jr. (Plant Manager, Donald C. Cook) pertaining to quarantining the Unit 2 "A" reactor trip breaker and other suspect components pending further evaluation and restrictions on startup of Unit 2 and Unit 1 (in Mode 3, completing post refueling testing). A Confirmatory Action Letter ( See Paragraph 5 below) was subsequently issued on October 30, 1985 documenting these and other licensee /NRC agreement Region III dispatched a Augmented Incident Response Team (AIRT) team consisting of a team leader (W. L. Forney, Region III Section Chief)

and a region based specialist. The team arrived on site at appruimately 8:00 p.m. on October 29, 1985. An additional region based specialist arrived on site at approximately 8:00 a.m. on October 30, 1985. Team members from NRC Headquarters ari Franklin Research Center arrived on site at approximately 8:00 p.m. on October 30, 198 The licensee's site team consisted of a team leader (M. P. Alexich, Executive Vice President, AEPSC) team members from the site and corporate office, and representatives from the breaker vendor. The licensee site team was onsite the afternoon of October 30, 198 Late on October 29, 1985 the Region III AIRT Team members met with licensee onsite management and agreed on the approach for

,

determining why the breaker did not function. The " quarantined" condition of the breaker and other suspect parts was reverified at

'

this tim .

- . .-- ..- - - . - .- - ~ . - - - - - - .

. .

,

'

~0n October 30,.1985 the licensee developed a procedure (un-numbered)

for in situ inspection of the. failed breake ~

The procedure I stressed the preservation of evidence of the cause of the breaker  ;

failure. Therefore the inspection was visual only, and the breaker

'

F components.were not touched during the inspection. An NRC inspector l' witnessed the inspection and'the ." dry-run" on a " spare" breaker that preceded the inspection. The " dry-run" contra-indicated certain ,

!. proposed uses of a borescope for performing the visual inspection, j .and the " live-run" was performed without these potentially -

problematical steps. No indication of the cause for the breaker

.

failure was identified during the visual inspectio It appeared i the undervoltage trip attachment (UVTA) which should have tripped the' breaker via actuation of the trip bar, had moved from its normal

, position and appeared to be touching the trip ba '

.

I After the breaker was inspected, the licensee developed a proceaure

!' (**2 MHP-SP-105 "Special Procedure for Inspection.of Unit 2 .

Reactor Trip Breaker A, Serial No. 324Y7277B") on November 1, 1985 for the removal of the undervoltage trip assembly (UVTA), removal of

'

i a the circuit breaker, and the-electrical and maintenance analysis of

} why the breaker failed to tri !

The procedure was developed, reviewed and approved in'accordance with

! the licensee's administrative Procedures. Procedure development was i

followed by a " dry-run" of the functional portions of the procedure, j performed in the presence of the AIRT Team. After the dry runs were l~ satisfactorily completed, the licensee documented the as found-L condition of the Reactor Trip Breaker A. This. document included j v'erification by voltage measurements that the "A" breaker UVTA coil

had deenergized in response to the trip signal, and meas ping and i recording the location of various breaker components. Dowmentary  !

photographs were taken throughout the performance of the g ocedur l l The circuit breaker trip function was dis'abled by tying the UVTA [

,

reset lever and the trip bar of the circuit breaker in place, .such '

that an accidental trip would not occur to upset the "as found"  !

condition of the breaker or endanger personnel engaged in the t

.

• .

investigation. The UVTA was removed from the circuit breaker,  !

bolted to a specially prepared board.and moved to a secure location l

in'.the electrical machine. shop.

.The ties placed to' prevent the circuit breaker from tripping were

removed and the breaker tripped using a calibrated spring scal i

,

-The' scale was initia..y pulled horizontally, being attached to . ,

the. trip bar by a string run over a clamp above~the breaker. .An  ;

'

administrative upper pulling force limit of 70' ounces was use The force necessary to trip-the breaker was 36 ounces, as measured i

<

horizontally. The. circuit breaker was removed from its cubicle and  !

, i ten (10) repetitions of the. force measurements-performed. - The  !

forced measured by'five tests pulling vertically on the trip bar ~

l ranged from:18 to 22.. ounces'and averaged 20 ounces; the normal  ;

upper acceptance, limit in this_ test.is 31 ounce j

-

.  ! i

>- .;

'

11- l

!

r

'

._, _

.- -J _ .-.. - -

. _ , ~ , . ,.,, _ _ , , .

- - _ - _ _ _ _ _ . _ __ _. _ _ . _ _

i

. .-

i' The UVTA and RTB were. removed to the electrical shop for further

,

testing. The UVTA was inspected in the electrical shop. No

indications of the failure were identified during this inspection.

.

The UVTA was replaced on the circuit breaker and the coil wires

connected to a test source. The coil was energized, the breaker i reset a'nd the coil de-energized. Upon coil de-energization, the L breaker tripped normally. This test was repeated four times on

'

slowly decreasing voltage with satisfactory results. The trip margin measurements were then performed using the applicable

portions of **12 MHP 5021.082.023. The trip bar was loaded with

twenty ounces of excess weight and the UVTA cycled (energized / .

4 de-energized)'five times. The breaker failed to trip each tim The weight on the trip bar was reduced to sixteen ounces and the UVTA cycled three times. The breaker failed to trip each tim The weight on the trip bar was reduced to ten ounces and-the UVTA i cycled three times. The breaker failed to trip each tim Five i tests with five ounces attached resulted in.one failure, one clear

success,_and three' successes characterized by breaker actuation on a t

second UVTA lever _ strike on the trip bar after initially reboundin All. weights were removed from the trip bar and the UVTA cycled three times. The breaker tripped e'ach time. This testing established

,

inadequate lifting force output from the UVTA as the root cause of q the "A'! trip breaker failure of October 29, 198 On November 2, 1985 the UVTA from the "A" trip breaker was placed in L a double locked box (Plant Manager and Westinghouse each had a key

to one lock). The box-and the "A" trip. breaker were placed in a

! locked cage located in the. electrical machine shop, and remained t there at the completion of the inspection, pending shipment to the

,

vendor for further investigatio '

. After the "A" trip breaker and. associated'UVTA were isolated, a _

course of action leading to startup of Units 1 and 2 was discussed.

! :The licensee committed to perform the.18 month surveillance /

. maintenance procedure (**12 MHP 5021.082.'023; which includes the 20 ounce trip margin verification) on the remaining breakers (four  :

I installed.in Unit 1, three installed in Unit 2 and the replacement

! breaker.for the."A". reactor trip breaker).- The four Unit 1 reactor trip-breakers and the replacement breaker for Unit 2 successfully

~

! "

completed the, surveillance / maintenance procedur On each of_these !

.

breakers the UVTA was' lubricated as required by the Surveillance  :

l- procedure. ~When the surveillance / maintenance procedure was commenced i

- on the Unit 2 "B" reactor trip breaker, the licensee decided to '

conduct the " trip force margin test" without prior lubrication of'the UVTA. The change in sequence was decided on because the lubrication /

lack of lubrication was the only obvious difference between the  ;

Unit _2 "A" breaker which failed to trip and the-five breakers which ;

had just su'ccessfully completed the surveillance / maintenance  !

[. procedur During the performance ~of the trip-force margin test with

<

20 ounces on the trip bar, the '!B" breaker failed to trip when the jl

,

UVTA was'de-energized. Thejtest was repeated with no weight on the r p trip bar; the breaker trippe The test was repeated with ten ounces ~ j [

[

,. l

. . . >

g .

. _ -

. .

on.the trip bar; the breaker trippe The test was repeated with twenty ounces on the trip bar, the breaker did not trip. -The test procedure was modified to trip the breaker using a spring scale and measure the tripping force. The test was performed four times. The breaker, still with twenty ounces on the trip bar, then tripped satisfactorily using the UVTA. This testing was performed for the

"A" bypass breaker, and "B" bypass breaker and two spare UVTA; all passed. The " degraded" UVTA for the '.'B" breaker was removed from the breaker and isolated in the same manner as the UVTA for the "A" breake The cause of the failure of Unit 2 reactor trip breaker "A" to function correctly on receipt of a valid trip signal on October 29, 1985 (and several additional times during subsequent testing) was concluded to be inadequate force on the breaker trip bar on de-energization of the undervoltage trip attachment. The UVTA root-cause failure mechanism remained unknown at the conclusion of this inspection. Investigation into the UVTA failure mechanism will be conducted by Westinghouse with oversight by the licensee and the NR . Confirmatory Action Letter A Confirmatory Action Letter from James G. Keppler (NRC-RIII) to John E. Dolan (AEPSC) was issued on October 30, 1985 as noted in Paragraph 1 above. The line items of the Confirmatory Action Letter and statur of licensee implementation are as follows: Conduct a thorough review to determine if a trip signal actually was received by the Reactor Protection System (RPS).-(Confirmatory Action Letter 316/85035-01).

(Closed): A trip signal (RCS low flow above P-8) was received and correctly processed and output by the RPS. See Paragraph 2.d abov Determine why the reactor trip breaker failed to tri (Confirmatory Action Letter 316/85035-02).

(0 pen): The reactor trip breaker failed to trip due to inadequate-tripping force from the undervoltage trip attachment. This was a new device installed in July 1985 by the vendor at his facilitie The root cause of the undervolt.qe trip attachment inadequacy remains under investigation by the vendor ~with licensee and NRC oversigh See Paragraph 3.b above, ' Determine why two of the.four steam generator feedwater isolation l . valves failed to close as anticipated following the trip.

l (Confirmatory Action Letter 316/85035-03).

! (Closed): Feedwater isolation valves FM0-201 and 204 failed to close due to incomplete actuation logic directly related to the~

~

,

!

l 13 i

_ . . ._ _ __ _ . - _ _- . _ .

=

f

.

, . .

fat'sure of reactor trip breaker "A" to ope They performed as designe See Paragraph-2.b abov ' Maintain all affected equipment related to the affected Reactor

. Protection System train, to. include' inverter power sources, in the

'

"as fou'nd" condition. Therefore, take no action such as removing or tripping the reactor trip breaker which would destroy or cause to be i lost, (other than necessary to protect the health and safety of the

public) any evidence which would be needed to investigate or .

i reconstruct this event.~(Confirmatory Action Letter 316/85035-04).

(Closed): The specified equipment was protected and controlled

'

-as. required, and was released for incident investigation and reconstruction only as specifically approved by the NRC AIRT Team based on the Team's review of and concurrence with

, procedures developed specifically to maximize collection of-

valid information. -See Paragraphs 2 and-3 abov Review operator actions taken immediately following recognition i of the~ failure to trip and determine if these action's were in accordance with your procedures and policies. (Confirmatory Action d

Letter 316/85035-05).

(Closed): Neither licensee nor NRC reviews of operator actions 3 identified any violation of licensee procedures and policies or l other requirement See Paragraph 2.c above.

!

' Develop troubleshooting plans and procedures, and provide these to the NRC site team for their review and concurrence prior to

, initiating'any trouble shooting of the affected equipmen (Confirmatory Action Letter 316/85035-06).

(Closed):-The licensee implemented this item as specified,

which served as the basis for release of the equipment for event' investigation / reconstruction as noted for-item'

~

!

l 316/85035-04 abov See also Paragraph 2 and 3 above.

4-

! Submit a formal report of your' findings and conclusions to the NRC-

' Region III Office within 30 days- (Confirmatory Action Letter

.

316/85035-07).

-(0 pen)

At the conclusion of the inspection ~ the report date-had not-yet. arrive ~

h .' 'We further understand that startup of.both Units 1 and'2 will not

- occur until'authoriz'ation to restart is'obtained from.the-Regional

. Administrator'of his designee. (Confirmatory Action Letter i 316/85035-08).

! (0 pen): At the conclusion of the' inspection,--the licensee had neither

!

. requested nor received authorization to. restart either Unit 1 or

~ Unit'2; but-the expectations of NRC concerning information J

..

-

'

14:

.

l'

, ~- - . - , - . . .- - --

. . - - . - _ . - _ _ . _ _ . _ _ . . . . _ - . _

-

,

c l~

requirements in support of the anticipated request, and the

mechanisms for_ submittal and processing of the request had been

-

fully di'scussed with licensee representacive !

.

5; Persons Contacted t

M. P. Alexich, Executive V. ice President, AEPSC

! W. G. Smith, Jr. , Plant Manager B. Svensson, Assistant Plant Manager

'A. Blind, Assistant Plant Manager

, K. Baker, Operations Superintendent

,

'

J. Stietzel, Quality Control Superintendent J. Allard, Maintenance ~ Superintendent L.'Gibson, Technical. Superintendent'-Performance

, A. McCabe, Major Transmission Equipment Engineer, AEPSC q:'

B. Auvil, Nuclear Safety.and Licensing Engineer, AEPSC J. Anderson, Electrical Generation. Engineer, AEPSC

C. Miles, Control and Instrument Supervisor P. Helms, Control and Instrument Assistant Supervisor i S. White, Maintenance Foreman R.-Sims, Shift Technical Advisor '

R. Ptacek, Shift Technical- Advisor

- C.

Springman, Shift Technical Advisor R. Blythe,. Shift' Supervisor

! J.~ Hylok, Control and Instrument Foreman

M. -Parvin, Site Service Manager - Westinghouse

C. Geiss, Technical Service. Representative - Westinghouse i A. Deb,. Technical Service Representative - Westinghouse-

'

R. Schreiner, Field Service Engineer Westinghouse

_

The inspectors also contacted a number of licensee and contract

! employees and informally interviewed operation, technical and

maintenance. personnel during this period.

,

6 .- Open-Items I .Open Items are matters which have baen discussed with the licensee, which will De r'eviewed'further by the inspector, and which involve some action on tha part of the NRC or licensee or both. -An open item disclosed g  :

during the inspection is discussed in Paragraph _ Unresolved Item

.

,

! Unresolved items are matters about which more information is required-Lin order to' ascertain whether they are acceptable, violations, or l deviations. An unresolved itemzis discussed in Paragraph .

.

JAdditional NRC Actions

!

p

~

' ~As a' result of this event 'on November 5 1985 the NRC issued

.

, , .

=IE Bu11etin:85-02-requiring _ licensees who did _not have automatic shunt-trips. installed as part of their reactor trip breakers.to perform testing _on theLundervoltage trip assembly (UVTA).

,

w L

'

l-i- 15

.m-

,

-,e? * -s 4-

_ _ _. . _ __ _ _ . _ _ _ _ _ - _ . - _ . r 4 L On November 5, 1985 NRR assumed lead responsibility for the NRC in addressing the generic aspectslof this. event and following the UVTA testing-to be conducted by Westinghous On November 6, 1985, NRC Region'III granted concurrence on the licensee's request for restart _of both Units 1 and 2. This concurrence was granted based on_the licensee having-satisfactorily-met the requirements of the October 30, 1985 CAL, with exception of submittal of a formal report of the findings and conclusions

- regarding this' event, and the licensee having installed the

automatic shunt trip feature on Unit 2.

t

Conclusion The. actions of the NRC' Augmented Incident Response Ter.m and of the

. licensee were directed toward conduct of a thorough and disciplined investigation of a failure of a reactor trip breaker to actuate in

! response to a. valid reactor trip signal of October 29, 1985. The

, ,

investigation proceeded.in a manner which was designed to assure

- preservation of evidence.

. .

The.following significant facts were ascertained: The reactor protection system functioned correctly in processing the trip signal, and the undervoltage trip attachments on both breakers j "A";and "B" were de-energized.

[ The undervoltage trip attachments (UVTA)lon both trip breakers were

. degraded, with the lift force output on the "A" trip breaker

~

!

attachment at such a low level as to be unreliable in tripping the l_ breaker. This was the root cause of.the-breaker failure.

j The performance of plant operators, and the operation of other major

'

or safety-related plant systems,.were as expected and designed.

l Though'the reason for the degraded UVTA performance was not conclusively

. - determined by-this inspection, arrangements were made to make that-

determination. Meanwhile, all plants.without~ diverse automatic trip breaker actuation-(via the' automatic shunt trip) were instructed by IE'

Bulletin 85-02 to perform such testing'as necessary to demonstrate

' >

acceptable UVTA performance.

i l

^

I

,

f" m .

1:

[- 1b +

-

'

a e 4 '

. - - - - , e- 4a- --M'T *-W 'we4 t '4*- --T * &-- t-* . -e -i==u= -

y* ,-Up- swr __g.-tv