ML13309B247

Forwards Reactor Trip Breakers. Info on Programmatic Aspects of Surveillance Test Failures Will Be Provided Later

ML13309B247
Person / Time
Site:	San Onofre
Issue date:	04/13/1983
From:	Dietch R Southern California Edison Co
To:	Harold Denton Office of Nuclear Reactor Regulation
Shared Package
ML13309B248	List: ML13309B247 ML20073D630
References
IEB-83-01, IEB-83-1, NUDOCS 8304140066
Download: ML13309B247 (54)

Text

REGULATURY IN* MATION DISTRI3UTION SYSTE&RIDS)

ALCESSION ObR;830414006b P0C.DATE: 83/04/13 NOTARIZED; NO DOCKET #

FACIL:50-361 San Onofre Nuclear Station, Unit 2, Southern Californ 05000361 50-362 San Onofre Nuclear Station, Unit 3, Southern Californ 05000362 AUTHNAME AUTHOR AFFILIATION DIETCHR.

-Southern California Edison Co.

RECIP.NAME RECIPIENT AFFILIATION

DENTONHR, Office of Nuclear Reactor Regulation, Director

SUBJECT:

Forwards "Reactor Trip Breakers," Info on programmatic aspects of surveillance test failures will be provided later.

I~~

a t

e r

DISTRIBUTION CODE: 80018 COPIES RECEIVED:LTR A ENCL f SIZE:,j#

TITLE: Licensing Submittal; PSAR/FSAR Amdts & Related Correspondence NOTES:J Hanchett Icy PDR Documents. ELD Chandler Icy, 05000361 NRR Scaletti Icy.

J Hanchett Icy PDR Documents. ELD Chandler icy, 05000362 RECIPIENT COPIES RECIPIENT COPIES ID CODE/NAME LTTR ENCL ID CODE/NAME LTTR ENCL NRR/DL/ADL 1

0 NRR LB3 BC 1

0 NRR LB3 LA 1

0

ROOH, 01 1

1 NTERNAL: ELD/HDS2 1

0 IE FILE 1

1 IE/DEPER/EPB 36 3

3 IE/DEPER/IR8 35

.1 1

IE/DEQA/QAB 21 1

1 NRR/DE/AEAB 1

0 NRR/DE/CEB.

11 1

1 NRR/DE/EHEB 1

1 NRR/DE/EQB 13 2

NRR/DE/G9 28 2

2 NRR/DE/HGEB 30 1

1 NRR/DE/MEB 18 1

1 NRR/DE/MTEB 17 1

1 NRR/DE/SAB 24 1

1 NRR/DE/SGEB 25 2

2 NRR/DHFS/HFEB40 1

1 NRR/DHFS/LQB 32 1

1 NRR/DL/SSPB 1

0 NRR/DSI/AEB 26 1

1 NRR/DSI/ASB 1

1 NRR/DSI/CPB 10 1

1 NRR/DSI/CSB 09 1

1 NRR/USI/ICS8 1b 1

1 NRR/DSI/METb 12 1

1 NRR/DSI/PSB 19 1

1 8

22 1

1 NRR/DSI/RS 23 1

104 1

1 RGN5 3

3 DAMI/MIB 1

'0 XTERNAL: ACRS 41 6

6 6NL(AMDTS ONLY) 1 1

DMB/DSS (AMDTS) 1

.1 FEMA-REP DIV 39 1

1 LPOR

.03 1

1 NRC PDR 02 1

1 NSIC 05 1

1 NTIS I

1 OTES:

3 3

TOTAL NUMBER OF COPIES REUUIRED: LTTR 57 ENCL 50

Southern California Edison Company P. 0. BOX 800 2244 WALNUT GROVE AVENUE ROBERT DIETCH ROSEMEAD, CALIFORNIA 91770 TELEPHONE VICE PRESIDENT 213-572-4144 April 13, 1983 Mr. H. R. Denton Office of Nuclear Reactor Regulation U.S. Nuclear Regulatory Commission Washington, D. C. 20555 Gentlemen:

Subject:

Docket Nos. 50-361 and 50-362 San Onofre Nuclear Generating Station Units 2 and 3 Enclosed are seven copies of SCE's report on the technical aspects of the March 1 and 8, 1983 surveillance test failures of the San Onofre Units 2 and 3 reactor trip breaker under voltage devices. These test results were previously described to the NRC by letters to NRC Region V dated March 11 and 24, 1983.

This report does not address the ongoing and completed investi gations regarding programmatic aspects of this issue. Such aspects include, for example, control of vendor information, control of hardware configuration, etc. Information on the results of these investigations will be provided at a later date.

Please contact me if you have any questions or comments.

Very truly yours Robert Dietch Vice President cc:

Mr. John Martin, NRC Region V 6304140066 830413 PDR ADOCK 05000361 P

PDR

4/12/83 SAN ONOFRE NUCLEAR GENERATING STATION UNITS 2 AND 3 DOCKET NOs. 50-361 and 50-362 REACTOR TRIP BREAKERS APRIL 1983 Southern California Edison Company San Diego Gas & Electric Company City of Anaheim

~ City of Riverside 8304140074 830413 PDR ADOCK 05000361 P

_ PDR

SAN ONOFRE UNITS 2 AND 3 RETURN TO POWER REPORT REACTOR TRIP BREAKERS OUTLINE I. Executive Summary II. Scope and Purpose III. Introductory and Background Information A. Reactor Protection System (RPS)

1. Design Criteria

2. Design Description B. Reactor Trip Breakers (RTB's)

1. Design Criteria

2. Design Description C. RTB Initial Procurement and Testing

1. Procurement History

2.

Acceptance Tests - Vendor

3.

Acceptance Tests - Startup

4. Startup and Surveillance Tests D. March 1 and 8, 1983 RTB Surveillance Test

1. Description of Tests

2. Test Results IV. Investigative Programs A. RTB Investigative Tests

1. Description of Tests

2. Test Results B. Evaluation of Licensee Administrative Procedures

1. Control of Vendor Information, Tech. Manuals

a.

Vendor Supplied Documentation

b.

Vendor Supplied Services

c.

Vendor Recommendations on RTB

2. Control of Hardware Configuration'

a.

Spare Part Programmatic Controls

b. Reactor Trip Switchgear Spare Parts

3. Maintenance Program

a. Procedures and QA/QC Requirements

b. Maintenance History and Records

c. Vendor Maintenance Activities

4. Surveillance Program

a. Technical Specification Requirements

b. Procedures and QA/QC Requirements

c. Surveillance History and Records

5. Reporting Requirements

a. Technical Specification Requirements

b. Reporting of March 1 and 8, 1983 Surveillance Results

6.

Non-Conformance Reports

7.

Compliance With IE Bulletins and Circulars

a. Initial Reviews

b. Follow-up Procedures

8.

Post-Trip/Restart Reviews C. Evaluation of Capability to Mitigate ATWS

1.

Procedures for Mitigating ATWS

2. Control Room Layout/Design

3. Operator Training/Knowledge D. Findings/Conclusions Regarding RTB Failure V. Proposed Corrective Actions A. Short Term Corrective Action Leading to Restart

1. Control of Hardware, Vendor Information, Vendor Personnel

2. Maintenance Procedures and QA/QC Requirements

3. Surveillance Procedures

4. Technical Specification

5. Operator Training B. Long Term Corrective Actions Subsequent to Restart

1. Follow On Activities

2. Evaluation of RTB Improvements VI.

Conclusions

Executive Summary As a result of failure of reactor trip circuit breakers (RFBs) to function at Salem 1 (IE Bulletin 83-01), SCE performed its 18 month surveillance test at San Onofre Units 2 and 3 which independently tests undervoltage (UV) and shunt trip functions of the RTB. This was done even though the RTB are of a different design than Salem and were not required to be tested by the bulletin. The surveillance test was performed in early March 1983 when San Onofre Units 2 and 3 were both shutdown and in different stages of their respective startup test programs. Four of the total of eighteen RTBs tested failed to trip following actuation of their UV devices. All eighteen tripped following actuation of the shunt devices.

SCE responded promptly to the failure of the UV devices to trip the RTB by conducting a comprehensive investigation which is discussed in this report. The purposes of this investigation, and resulting findings and corrective action, are to ensure that the UV devices operate reliably and to ensure that the SCE startup and operating programs include any lessons learned from the failure of the CV devices to operate reliably. However, it is important that the following facts be considered in reviewing the experience with RTB UV devices at San Onofre Units 2 and 3:

o The UV device is one of two diverse methods used to trip the RTBs.

Unlike other designs, the UV device is not required to function for the reactor protection system to complete its design basis protective action at San Onofre Units 2 and 3. Reactor trip is initiated by opening the RTBs automatically or manually, and both signals actuate both UV (deenergize to operate) and shunt (energize'to operate) trip devices. During our investigation the shunt device always functioned to trip the RTD, and, therefore, the RTBs at San Onofre Units 2 and 3 have performed so as to meet their design basis function.

o Failures of UV devices to function, and work performed in response to these failures, have occurred during startup testing at San Onofre Units 2 and 3. As a result, the initial failures in March and July 1982 were interpreted as problems with set up and adjustment which are typical of startup testing. The failures during surveillance testing in March 1983 were recognized by the operating phase surveillance program as symptomatic of other potential problems and as requiring in depth investigation.

The comprehensive SCE investigation discussed in this report has identified factors contributing directly to the failure of the UV device.

The report term corrective actions required to obtain satisfactory breaker performance have been identified and implemented.

-2 The UV device has a much smaller force margin to trip the RTB than has the shunt device. Nevertheless, with upgraded maintenance and surveillance testing, the UV device will function reliably. The shunt device reliability for tripping the RTB has always been very high in the San Onofre Units 2 and 3 design, and operation of this device alone fully satisfies the reactor protection system design basis.

-3 II.

Scope and Purpose This report provides a summary of the comprehensive review performed by the Southern California Edison Company (SCE) subsequent to discovery of the reactor trip breaker undervoltage trip mechanism operability problem on San Onofre Units 2 and 3. The purpose of the report is to describe the corrective actions that have been implemented as a result of our review.

Following a brief discussion.of the reactor protection system (RPS),

reactor trip breakers (RTB), and surveillance tests identifying the operability problem, the point is made that the UV trip device is not required to function for the reactor protection system to perform and complete its design basis protective action.

The report then describes SCE's investigative program, including the specific breaker tests that were performed on the test bench (NRC witnessed) and subsequent testing conducted by SCE at our Electrical Test Laboratory. Discussions are provided summarizing our evaluation of administrative procedures, including; control of vendor data, control of hardware configuration, maintenance and surveillance programs, reporting requirements, non-conformance reports and compliance with IE Bulletins and Circulars.

The report also discusses evaluation of post-trip and restart reviews as well as capabilities to mitigate the consequences of Anticipated Transients Without Scram (ATWS). Based on the RTB investigative test results, SCE has identified enhanced maintenance and surveillance procedures for the RTB. Based on the evaluation of administrative procedure, SCE has also identified both short and longer term corrective actions which we believe are appropriate to strengthen our administrative system.

The report concludes that with the implementation of the enhanced procedures for maintaining the RTBs, San Onofre Units 2 and 3 can safely resume operation and startup testing.

-4 III. Introductory and Background Information NRC IE Bulletin No. 83-01, Failure of Reactor Trip Breakers (RTB's)

(Westinghouse DB-50) to Open On Automatic Trip Signal, issued on February 25, 1983, discussed the February 25, 1983 failure of the DB-50 RTB's to open automatically upon receipt of a valid trip signal at Salem Unit 1. The reactor was manually tripped from the control room about 30 seconds later, and the event was successfully terminated without core damage.

The Salem Unit 1 RPS is designed to automatically open the RTB's using.

only the undervoltage trip device; manual actuation of the RTB uses both the UV and shunt trip devices.

The failure of the RTB's to trip automatically was attributed to sticking of the undervoltage trip device.

In contrast with Salem Unit 1, the San Onofre Units 2 and 3 RPS provides both automatic and manual signals to actuate both the UV and shunt trip devices of each of the RTB's and the design configuration utilizes eight channelized RTB's. This section of the report discusses the design criteria and function of the RPS, the 'design and function of the RTB as part of the overall reactor protection action and the role of the undervoltage trip device as part of the RTB.

Even though IE Bulletin No. 83-01 did not require testing of the RTB's at San Onofre Units 2 and 3, the UV trip devices on the RTB's at San Onofre Units 2 and 3 were independently tested on March 8 and 1, 1983 respectively. The results of these tests are discussed in Section III.D of this report.

-5 A.

Reactor Protection System (RPS)

1. Design Criteria The design bases for the Reactor Protection System are presented in FSAR Section 7.2.1.2 and are summarized as follows:

The RPS is designed to assure adequate protection of the fuel, fuel cladding, and RCS pressure boundary during anticipated operational occurrences. In addition, the system is designed to assist the ESFAS in limiting the consequences of accident conditions.

a. The system is designed in compliance with the applicable NRC General Design Criteria of 10 CFR 50 Appendix A, IEEE 279-1971, IEEE 338-1971 and is consistent with the recommendations of Regulatory Guide 1.53 and Regulatory Guide 1.22.

b. The system is designed to alert the operator when any monitored plant parameter is approaching a condition that would initiate protective action.

c.

The system is designed so that spurious protective action will not be initiated during normal operation of the plant.

d.

Four measurement channels are provided for each plant parameter monitored by the protection system, with the exception of the control rod positions.

e. The four measurement channels are independent and isolated from each other utilizing: separate sensors monitoring the channel parameters, separate wire trays or conduits for the channel interconnecting cabling, separate channels for the RPS cabinet mounted signal processing equipment and separate instrument ac power buses backed up by separate batteries for each channel.

f.

The four measurement channels provide trip signals to six independent logic matrices, arranged to effect a two-out-of-four coincidence logic, each having outputs to four independent trip paths for each actuation signal.

g. When one of the four measurement channels is taken out of service, the protection system logic can be changed to a two-out-of-three coincidence logic for actuation of plant protective action.

h. After initiation, manual reset of the actuation output signal is possible following the clearing of its input signals.

i.

System functions requiring operator attention or action during routine plant operations are displayed and/or controlled at the operators main control board.

j.

Annunciation is provided at the main control board of all operations at the RPS cabinet that could affect the function of the system.

-6

2. Design Description Figure III.A.2-1 is a simplified functional diagram of the reactor protection system. As can be seen the RPS is a four channel system and can be divided into several areas defined as:

Measurement Channels, Bistables, Logic Matrices, Logic Matrix Relays, Trip Paths and Trip Circuit Breakers. FSAR Section 7.2 provides a detailed discussion of each of these areas. The following is a summary discription of the RPS operation.

Measurement channels consist of sensors and signal conditioning equipment whose purpose is to convert the parameters being measured (pressure, temperature, etc.) into signals usable to the RPS bistables or calculators. These signals are provided in the form of analog voltages.

Signals from the measurement channels are sent to voltage comparator circuits (bistables) where the input signal is compared to predetermined setpoints. Whenever the measurement channel signal reaches the setpoint, the bistable output will de-energize associated bistable relays.

Tripping of a bistable results in a channel trip which is characterized by the de-energization of three bistable trip relays.

Contacts from the bistable relays of the same parameter in the four protective channels are arranged into six logic ANDs, designated AB, AC, AD, BC, BD, and CD, which represent all possible two-out-of-four combinations. To form an AND circuit, the bistable trip relay contacts of two like protective measurement channels are connected in parallel (e.g., one from A and one from B).

This process is continued until all combinations have been formed.

Since there is more than one parameter than can initiate a reactor trip, the parallel pairs of bistable trip relay contacts for each monitored parameter are connected in series (Logic OR) to form six logic matrices.

The six matrices are designated AB, AC, AD, BC, BD, and CD.

This logic requires two or more bistables monitoring the same parameter to be in a tripped condition before a reactor trip can be generated.

Each logic matrix is connected in series with a set of four logic matrix relays. Each logic matrix relay is associated with a trip path. The trip path is made up of six series contacts (one from each logic matrix). The contacts are also in series with a trip circuit breaker control relay (initiation relay).

For each actuation signal, the above logic causes the de-energizing of the four trip path initiation relays whenever any one of the logic.

matrices is de-energized.

INPUTS FROM NSSS MEASUREMENT CHANNELS 1 2 3 4 5.6 7 8 9-N 1 2 3 4 5 6 7 8 9-N 1 2 3 4 5 6 7 8 9 -N 1 2 3 4 5 6 7 8 9-N BISTABLES CHANNELSA CHANNELSB CHANNELSC CHA LOGIC MATRICES A-B LOGIC

)

A-C LOGIC GIC )

B-C LOGIC B-D LOGIC

)

C-D LOGIC A4 A

C4 AD4 BC4 t504 CD4 LOGIC 0

MATR IX AB2 E

AC22 BC2 BD?

CD2 RELAYS E

E TO 120 Vac TO 120 Vac TO 120 Vac TO 120 Vac VITAL INSTRUMENT VITAL INSTRUMENT VITAL INSTRUMENT VITAL INSTRUMENT BUS No.1A BUSNo.1B BUSNo.1C BUS No.1D ABI AB2 AB3 AB4 ACT

=AC2 AC3 =AC4 AD1 =AD2 AO3 AD4 TR PTSBC1 BC2 BC3

=BC4 TRIP PATHS B

BD2 B3 BD4 TRIP CDl CO2 CD3 CD4 CIRCUIT BREAKER CONTROLRELAYS K1 K2 K3 K4 (grNrAT ION AELAYS) 480 Vac-30 480 Vac-30 BUS No. 1 BUS No.2 MAIN CIRCUIT No.1 BREAKERS No.2 MOTOR GENERATORS Nol 240.Vac-30N

+125 Vdc BUS No. 2 SYNCHRONIZER25 Vd BUS No.3 BUS TIE MANUAL r K2 K3 K3 TRIP No.1 UV ST UV STS

-2VdUV VBSNST

+125 Vdc BUS No 1

1 1

+125 Vde BUS No. 4 MANUAL K -

TRIP CIRCUIT TRIP No.2 BREAKERS K4 K4 UV UV ST

{UV ST An UV ST CEDM POWER INDIVIDUAL CEDM CEDM CONTROL INDIVIDUAL CEDM SUPPLIES POWER SUPPLIES SYSTEM POWER SUPPLIES CONTROL CEDM CEDM CDM CEDM CEMD ELEMENT COILS CCOILS ILS

)

COILS COILS COILS DRIVE MECHANISMS FIGURE III.A.2-1 SIMPLIFIED FUNCTIONAL DIAGRAM OF THE REACTOR'PROTECTION SYSTEM

-7 The trip path initiation relays then transmit trip signals to the RTBs.

The design criteria and function of the RTB using these actuation signals is described in the next section.

-8 Reactor Trip Breakers (RTB's)

1. Design Criteria The reactor trip switchgear cabinet assembly, including the reactor trip breakers, is specified in SONGS 2.and 3 FSAR Table 3.2-1 as:

Quality Class 1 (safety related)

Seismic Category 1 (design basis earthquake)

Electrical Class 1E It should be noted that the seismic qualification test plan does not require the breaker closing circuit to function during a DBE; however, it must not inter ere in any way with the trip function of the breaker.

The reactor trip signals are generated in the reactor protection system through the six logic matrices as described in Section III.A.1.2. The reactor trip signals are in turn transmitted to both the undervoltage and shunt trip devices on each RTB.

Each of the four trip path initiation relays (K relays) sends an actuation signal to both the UV and shunt trip devices on a pair of RTBs.

The pair of RTB receive dc power from the same vital bus that provides ac power for their associated initiation relay. This can be seen schematically in Figure III.A.2-1 where the Kl relay provides the actuation signal to RTB numbers 1 and 5.

The reactor trip breaker design is such that failure of either the undervoltage trip device or the shunt trip device does not eliminate the safety function of the reactor trip breaker.

The trip circuit breakers will complete their protective action of interrupting power to the control rods using either the UV or shunt trip devices.

The RTB undervoltage device and the shunt trip device compliment each other in that the undervoltage device trips the breaker upon loss of control voltage while the shunt trip device trips the breaker upon application of control voltage (see Figure III.B.1-1).

FU, FU

~tnt tarttt efLattD Dut

• t.

Wrtoomto towaratef 30 OT5 o ae wser sewoa.

sre EL32 ES 10 0

O3SE I

CTt 2

oIoV PIC n L

C.t I

2 SEu2 2L2S?

ttt.

3Li~

olLoo Z

200\\31 ILTRIPJ

n.

.Y SVC OB AOT 5NTj 32.~b-oT 3L3OS i-.

SOURCE'Z 52 G

52.

ENTERS U*OL StMI tMTI-1 APS -

To HT. 6a T P4Oa1. 2L UNIT PANEL 2

or O3 UIT 3 PANEL BREAKER CLOSIN APPROVED a-,REFERENCE 5_

2'DESIGN 30TCB-2 TC D

SG U-ozC 5'23-944-A5 OTS OTB OT B

OT 0T OTS OT O 8 xo1-orGOING TERMINAL BLOCK IN PL33/3L.33 PC.- PLANTr COMPUTER 52 5 t St S.

15 SL.

2. AIR CIRCUIT BREAKER -T.ooV* 600A.

b b

2 LEMERGENCY TRIP PUSHBUTTON AT O7 OT O TWITCHGF.AR TO HAVE A CLEA.R PLASTIC PROTECTIVE Con.R WITH PQOVISION 1 T B B 1OT 0T8 OTS C

OTBore

-d EMER8.

.EER&

FOR LOCKING WITH WIRE SEALS.

PC.

ANN 2L3Z

__51__9411OT 8 ot~al6/4 078B o-rs PC PR SAFETY 1ELATED

'116 ONLy REV.

Descm*noN cREV.DESCRIPTION

'***s a

a.

e a to DrAW.N BY:

D~T OT loESCRIPTION P itS CO IA'~l OV tl ArttO _!_

r~

~ ~

~

a scK rearcilli An aHCK v.,, L-.elT--Too DEl ETEDf.COirACTS 52/8AJ c

ELEMENTARYWIRING DIAGRA F., C. 5 s J /.;

4,.N I. 1 PE R.REV.RED:1370--REACTOR rI P CiRCUIT BR hER Cv 86(

inc-t*3 GRAbtD TO AIL=,)

.a 4-

- Rc) 1~'!'

- l c,j:

-V v* T~y.

I l D.,RA

/I GRE

,c:-/3 7. - 4 12 - '3 3 0O I(i.R Q 97 -

9 A%~ i rtr.

C A,'.:ATU~t or:.:: rv ov r cr 9

________________,*__\\_v.,,Em t-VY~ Ac.:.: ;t 1. m ::Ato c o *Andy.

FIGURE III.E 1-1

-9

2.

Design Description The details of the breaker latching mechanism and the function of the tripping devices are shown pictorially in the following sketches:

a. Breaker Assembly, Trip Bar and Latch Figures III.B.2-1 through 8 provide simplified representations of the AK2-25 latch mechanism operation.

b. Shunt Coil Assembly Figure III.B.2-9 depicts the AK2-25 shunt trip device in both the "tripped" and "not tripped" positions. The device requires 125 Vdc and is energized to trip by actuating the tripper bar in a counter clockwise direction.

c. Undervoltage (UV) Coil Assembly Figure III.B.2-10 depicts the AK2-25 undervoltage trip device in both the "tripped" and "not tripped" positions.

This device requires 125 Vdc and is normally energized.

Upon a loss of voltage the device actuates to rotate the tripper bar in a counterclockwise direction.

-10

d. Existing Design Modifications The RTB in use at SONGS Units 2 and 3 have one additional modification to the foregoing design description. CE recommended a design change which SCE implemented to place a diode in parallel with the UV coil to provide a discharge path for the collapsing field and prevent contact damage to the RPS "K" relay. This diode is shown in Figure III.B.1-1.

It should be noted that the diode is not installed on the breaker, but in the switchgear cabinet assembly, so that when the breaker is installed or in the "racked out" position, the diode is in the UV coil circuit, but when the breaker is removed from the cabinet, it is not. The requirement for simulating this diode when preventive maintenance and testing is performed is discussed under test results in Section IV.A of this report.

This diode also results in a slower collapse time for the UV coil magnetic field, causing the UV trip device operating time to increase by about 30 msec.

This change may also result in a slight loss of UV device tripping force by reducing the kinetic energy of the UV device armature; however, when the enhanced breaker maintenance is performed as described in this report, the overall RPS response time remains within acceptable limits.

FIGURE Ifl.B. -1 GE AK-2-25 CIRCUIT BREAKER UPPER STUD BREAKER BASE STATIONARY CONTACTS MOVABLE CONTACT INSULATED LINKS STOP."A" OUTPUT CRANK CLOSING SPRING PIVOT TRIP LATCH P:VOT PIN STOP "B" (BLOCKING CAM)

LES LOWER STUD CLOSING SOLENOID ARMATURE C.LOSINGSOLENOID THE CIRCUIT BREAKER NECHANIS-1 HAS BEEN SIMPLIFIED CONSIDERABLY IN ORDER TO CLARIFY ITS OPERATION.

STOP "A" REPRESENTS A PORTION OF THE MECHANISM THAT PREVENTS THE LINK FROM MOVING BEYOND THE POINT INDICATED.

STOP "S."

IS A BLOCKING CAM THAT PREVENTS THE MECHANISM FROM MOVING AND CLOSING TH-E CONTACTS UNTIL THE CLOSING SPRING IS CHARGED.

THE TRIP LATCH LOCKS THE !IECHA*ISM IN THE CLOSED CONTACT POSITION UNTIL RELEASED BY THE TRIP PUSHBUTTOM, SHUNT TRIP DEVICE, OR THE UNDERVOLTAGE TRIPPING DEVICE.

IT IS SHOHN ON THIS SKETCH AS A PICTORIAL REPRESENTATION OF ITS FUNCTION.

FIGURE III.B.2-2 UPPER STUD MOVABLE BREAKER BASE CONTACT STATIONARY CONTACTS I NSULATED STOP "A" L__

CLOSING OUTPUT SOLENOID CLOSING SPRING TRIP LATCH PIVOT PIN LO,-ER STUDj j BLOCKING CA CLOSING CONDITION 1 SOLENOID THE CIRCUIT IS OPEN AND THE OPERATING SEQUENCE IS ABOUT TO START.

THE SPRING LOAD KEEDS THE CONTACTS OPEN.

FIGURE lit.B.2-3 UPPER STUD BREAKER BASE

-MOVABLE O A E STATIONARY V

CONTACTCONTACTS CLOSING SOLENOID ARMATURE CLOSING SPRING INSULATED L I NK' STOP "A" Z,-7 OUTPUT CRANK PIVOT TRIP LATCH PIVOT PIN BLOCKING. CAM LOWER STUD CLOS:NG CODITION 2 SOLENOID THE CLOSING SOLENOID IS ENERGIZED AND THE ARMATURE EXTENDS.

ENERGY IS STORED IN THE CLOSING SPRING BUT THE BLOCKING CAM PREVENTS 1-I THE OUTPUT CRANK FROM MOVING.

FIGURE III.B.2-4 UPPER STUD BREAKER BASE STATIONARY CONTACTS MOVABLE CONTACT CLOSING SOLENOID ARMATURE CLOSING SPRING INSULATED LINK STOP "A"l OUTPUT CRANK CRA'NK PIVOT PIVOT PIN TRIP L.ATCH PVTP1 LBLOCKING LOHER STUD CAMS CLOSING CONDITION 3 SOLENOID THE BLOCKING CAM RELEASES THE OUTPUT CRANK AND THE SPRING TENSION CAUSES THE CRANK TO ROTATE ABOUT ITS PIVOT AND IT DRIVES THE MOVABLE CONTACTS

\\

AGAINST THE STATIONARY CONTACTS.

THE TRIP LATCH HOLDS THE MECHANISM IN POSITION AND THE CIRCUIT DET11EE THE UPPER AND LOWER STUDS IS COMPLETED..

FIGURE III.B.2-5 UPPER STUD BREAKER BASE STATIONARY CONTACTS MO1VABLE CLOSING SOLENOID CONTACT AR: AT URE INSULATED CLOSING LINKS SPRING OUTPUT PIVO PIVOT PIN TRIP LATCHI Jo LOWER STUD CONDITION 4 THE CLOSING SOLENOID IS DE-ENERGIZED AND THE ARMATURE REMAINS IN ITS POSITION DUE TO THE OVER-CENTER TOGGLE ACTION.

CLOSING SOLENOID

FIGURE III.B.2-6 UPPER STUD BREAKER BASE STATIONARY CONTACTS MOVABLE.

CONTACT INSULATED STOP. "A"

.LINKS OUTPUT CRANK CLOSING SPRING

,PIVOT rLOSING TRPLCHPIVOT PIN S1 LEN 01 D MATURE LOWER STUD BLOCKING CAM CONDITION 5 THE TRIP LATCH IS TRIPPED BY THE TRIP PUSHBUTTON, THE SHUNT TRIP DEVICE, OR THE UNDERVOLTAGE TRIPPING DEVICE. THE OVER-CENTER TOGGLE ACTION COLLAPSES AND THE MECHANISM IS NOW FREE TO MOVE.

THE SPRING TENSION CAUSES THE OUTPUT CRANK TO ROTATE ABOUT ITS PIVOT, PULLING THE INSULATED LINKS AND CAUSING THE CONTACTS TO 4.

OPEN.

THE SYSTEM WILL REMAIN IN THIS DE-ENERGIZED POSITION UNTIL THE CLOSIN-TRIPPING CONDITION IS RECTIFIED AND THE CLOSING SOLENOID IS SOLENOID ENERGIZED.

FIGURE III.B.2-7 TRIP LATCH ROLLER PIVOT TRIP SHAFT T

LTRIP LATCH TRIP LATCH ROLLER THE TRIP LATCH AND TRIP LATCH ROLLER ARE IN THIS POSITION WHEN THE CLOSING SOLENOID IS NOT ENERGIZED AND THE TRIP LATCH IS IN THE RESET POSITION. THERE IS A GAP BETWEEN THE ROLLER AND THE LATCH.

WHEN THE CLOSING SOLENOID IS ENERGIZED, THE TRIP LATCH ROLLER IN ITS HOLDER TURNS

.COUNTER CLQCKWISE, CLOSES THE GAP, AND IS RESTRAINED BY THE TRIP LATCH.

THE TRIP LATCH AND ROLLER REMAIN IN THIS POSITION UNTIL THE BREAKER IS TRIPPED.

WHEN ANY OF THE TRIPS ARE ACTIVATED, THE TRIP SHAFT ROTATES IN A COUNTER CLOCKWISE DIRECTION AND THE TRIP ROLLER ROLLS ALONG THE CONTACTING SURFACE OF THE TRIP LATCH UNTIL THE TRIP LATCH SWINGS OUT OF THE WAY, LEAVING THE ROLLER AND ITS HOLDER TO ROTATE FREELY ABOUT ITS PIVOT. THIS CAUSES THE OVER-CENTER TOGGLE ACTION IN THE MECHANISM TO COLLAPSE AND THE CONTACTS OPEN.

THE ROLLER RETURNS TO ITS ORIGINAL POSITION WHEN TRE CONTACTS OPEN AND THE MECHAJNISM IS DE-ENERGIZED. THE TRIP LATCH WILL REMAIN IN THE POSITION INDICATLD UNTIL THE CAUSE FOR TRIPPING'IS RECTIFIED AND THE TRIP LATCH IS RESET.

FIGURE III.B.2-8.

MANUAL TRIP PADDLE UNDERVOLTAGE TRIP PADDLE SHUNT TRIP PADDLE TRIP SHAFT TRIP SHAFT BEARING TRIP LATCH TRIP LATCH ROLLER TRIP SHAFT SKETCH NOTE: THIS SKETCH DOES NOT PORTRAY THE ACTUAL SHAFT OR THE LOCATION OF THE VARIOUS TRIPPING PADDLES WITH RESPECT TO EACH OTHER OR TO THE SUPPORTING BEARINGS. IT IS A PICTORIAL REPRESENTATION OF THE TRIP SHAFT AND ITS FUNCTIONS.

FIGURE III.B.2-9 313 NOT TRIPPED e A I'I" 040 10 0

ALOL SVL TRIPPED Figure 9. (,695C161) Shunt Trip Device

1.

Nut

6.

Screws

11.

Armature Arm

2.

Frame

7.

Magnet

12.

Trip Paddle

3.

Spring

8.

Coil

13.

Mechanism Frame

.4. Rivet

9.

Clamp

14.

Trip Shaft Clamp

5.

Weight

10.

Armature

r 0

FIGURE III.B.2-10 12 3e

'2 4-4 6

13 7

12 ThII 9

TRIPPED 20

1. Mounting Screw

2. Frame

3. Armature 184.

Spring

5. Shading Ring

6. Adjusting Screw

7. Locking Nat

8. Bushing S9.C

-- -- -10.

) a et 15 S1r.ws

.4

12. Coil

13. Bivet

14. Adjusting Screw 11

15. Locking Wire

16. Mounting Nut 13

17. Mechanism F*a-ne

18. Trip Paddle Clamps 12

19. Trip Paddle

20. Adjusting Screw 9

7 NOT TRIPPED Figure 28. -0152C9206)

Undervoltage Tripping Device

C. RTB Initial Procurement and Testing

1. Procurement History The San Onofre Units 2 and 3 Reactor Trip Switchgear (RTSG) Systems were procurred as a part of a general procurement package intended for several CE plants. A procurement history of this generic package and the documents that confirm compliance with SCE's project specifications are described below. This documentation is for the RTSG that included the original AK2-25-2 RTB's, and is available for audit. Spare parts procurement would have to verify compliance with these specifications as well as certify complete functional equivalency.

1. A Request for Quotation (RFQ) was issued by CE for a generic order of four (4) RTSG systems which invoked CE General Specification 00000-ICE-3008 Rev. 01 and CE general RTSG design drawings.

2. Unit Electric Control, Inc. (UEC) among other vendors provided a response to CE Request for Quotation which was received during the first quarter 1975.

3.

CE performed a Bid Evaluation of the bidders that responded to the CE RFQ recommending Unit Electric Control, Inc. as the preferred vendor for the subject equipment.

4.

CE issued a Master Purchase Order to Unit Electric control invoking CE General Specification 00000-ICE-3008 Rev. 01.

5. A supplemental quotation request was issued by CE to Unit Electric Control, Inc. modifying the requirements for packaging, shipping, receiving, storage and handling (i.e., the ANSI N45.2.2-1972 requirements now imposed on the vendor).

6. Unit Electric Control, Inc. issued a response to the supplemental request and implemented the requirements of same.

7. Purchase Order Release #1 against the Master P.O. for San Onofre Units 2 and 3 was issued by CE to Unit Electric Control.

This Release Order invoked CE Project Specification 1370-ICE-3008 Rev.

01 in addition to the general requirements and requirements for seismic/ environmental qualification.

8. Supplement 1 to the Master Purchase Order was issued by CE requiring deletion of coil status indication lights (which were deleted to enhance breaker control circuit reliability) and changing the RTSG heaters from 120V AC to 240V AC.

9. The following Request for Approval or Review (RAR), Technical Change Request (TCR), Deviation of Contract Request (DCR) submittals were-made by Unit Electric Control against the SCE Unit 2 and 3 RTSG contracts.

-12

a. RAR's 1 through 16 were submitted by UEC during the time period October 1975 to April 1976. These RAR submittals pertained to Inspection and Test Plans, Drawings, IMQP, Qualification Plan and Results, Tech Manual, and Function Test Procedures and Results. All RAR's were approved by CE for compliance with CE specifications.

b. TCR's 1 through 4 were submitted by UEC in October 1975.

TCR's 1 through 3 were not approved by CE and the requested changes were not accomplished. TCR 4 was approved and.

allowed a change in the CE specified painting requirements.

c. DCR's 1 through 3 were submitted by UEC during the time period of October 1975 to February 1976. DCR 1 was disapproved by CE and the requested change was not allowed.

DCR 2 allowed the deletion of terminal block cover wire numbering. DCR 3 allowed UEC not to submit Progress Reports on RAR's. DCR's 2 and 3 were approved by CE.

10. A conditional Certificate of Equipment was issued in December 1975 which formed the basis for shipment of the RTSG's for both San Onofre Units 2 and 3. An unconditional Certificate of Equipment was subseqently issued in June.1976 to include CE review of the as-certified drawings, instruction manual and test data.

11. The following lists the as-shipped breaker configuration for SCE Units 2 and 3 by breaker serial number and location.

Unit 2 Breaker Serial No.

Location 256A4002-656-3 TCB-l 256A4002-656-15 TCB-2 256A4002-656-24 TCB-3 256A4002-656-29 TCB-4 256A4002-656-1 TCB-5 256A4002-656-8 TCB-6 256A4002-656-26 TCB-7 256A4002-656-33 TCB-8 256A4002-656-18 TCB-9 Unit 3 Breaker Serial No.

Location 256A4002-656-5 TCB-l 256A4002-656-14 TCB-2 256A4002-656-25 TCB-3 256A4002-656-17 TCB-4 256A4002-656-2 TCB-5 256A4002-656-7 TCB-6 256A4002-656-34 TCB-7 256A4002-656-45 TCB-8 256A4002-656-12 TCB-9 The above are the original AK2-25-2 breakers, shipped with the RTSG.

-13

2. Acceptance Tests -

Vendor Reactor Trip Switch Gear acceptance tests required and performed by the RTSG vendor.were as follows:

A. As required by CE General Specification No. 00000-ICE-3008 Section 5.10.3 Bench Tests were performed by the vendor of the San Onofre RTSG prior to installing equipment into the RTSG cabinet. This testing included:

1. Tests to check defects in mechanical assembly

2. Tests to check defects in electrical wiring

3. Checks for grounds, shorts or open circuits

4. Actuation checks to ensure:

a. Proper closing operation

b. Proper tripping operation

c.

Proper status indication as defined by breaker position

d.

Proper functioning of auxiliary relay contacts The above testing was satisfactorily completed and documented.

B. In addition to the individual equipment bench tests described in (A) above the RTSG system underwent additional testing as required by CE General Specification No. 00000-ICE-3008 Section 5.10. This testing included:

1. Hi-Potential Testing,

2. Point-to-Point Continuity Testing,

3. System operational testing (this procedure is incorporated in the CE Instruction Manual for the reactor trip circuit breaker switchgear for San Onofre Units 2 and 3).

C. Quality Assurance inspection of workmanship as required by Combustion Engineering Quality Control Specification (00000-WQC-11.1, Rev. C) was performed.

The above testing and inspection were satisfactorily completed, a Certificate of Equipment was issued by CE, and the RTSG was delivered to SCE.

The following summary of the RTSG equipment qualification data and references reflects testing and analyses performed prior to delivery of the RTSG to SCE. The follow-up seismic study by Wyle Laboratories (Item 6) confirmed the adequacy of the previous seismic qualification.

(1) The RTSG was seismically qualified to the requirements of IEEE-344, Seismic Qualification of Safety Related Electric Equipment for Nuclear Power Generating Station, 1971.

-14 (2) The RTSG environmental test was documented in accordance with the requirements of IEEE-323, Trial Use Standard, General Guide for Qualifying Safety Related Electrical Equipment for Nuclear Power Generating Station, 1971.

(3)

San Onofre Units 2 and 3 Reactor Trip Circuit Breaker Switchgear was seismically and environmentally tested/analyzed by Wyle Laboratories, Huntsville, Alabama.

(4)

The results of the testing/analysis were approved by CE following the requirements of Combustion Engineering "Sellers Procedure for Submitting Approval Review (RAR)" NPD-MPI-6 Appendix.

(5) Test results were defined in CEN(94)-S Data Sheet No. 3 for the RTSG seismic test documentation and in CEN(95)-S Data Sheet No. 3 for the RTSG Environmental Test documentation as referenced in Section 3.10 and 3.11 of the SONGS FSAR for NSSS equipment.

(6) Additional Analysis information was provided to SCE by Wyle Laboratories Analysis Report No. 26321 Section 6.11 as response to the NRC Seismic Qualification Review Team (SQRT) general concerns for San Onofre Units 2 and 3.

The RTB's are intended to be used in the reactor trip switchgear cabinets which are located in a non-harsh radiation and temperature environment. Consequently, environmental testing includes only the upper limit of the normal room temperature and does not test or analyze for radiation effec s. The RTB are in an area that is expected to be less than 10 rads total 40 year dose, so their components are considered qualified for this service. Room temperature is regulated by the plant HVAC. Any potential effect of temperature or radiation on maintainable parts (such as lubricant) will be detectable through the enhanced surveillance program described later in this report.

3. Acceptance Tests - Startup As described in the San Onofre Units 2 and 3 FSAR, the Startup Test Program includes requirements for prerequisite and preoperational testing of equipment including the reactor trip switchgear.

The preoperational testing is discussed in Section III.D.6 of this report. The prerequisite testing of Reactor Trip Breakers was as follows:

a. All type AK2-25-2 breakers (18 total) were successfully tested with Startup generic test GT-400-05. This was a generic'air circuit breaker test, and contained the following elements:

1. Meggering of insulation resistance to >50M ohms

2. Recording of X, Y, and trip coil resistances.

3. Recording of minimum voltages to trip (<70 VDC) and close

(<100 VDC)

-15

4. Verification of correct functional operation of auxiliary and test switches.

5. Verification of correct mechanical adjustments of contact wipe and latch adjustment.

6. Visual inspection of arc chutes, barriers, control switches and wiring, racking and locking mechanism, all switches and wiring and hardware.

7. Closing solenoid resistance was recorded.

Although not specifically required by GT-400-05, it was SCE's practice to record undervoltage device pickup and dropout voltages during this test.

b. Equivalent prerequisite testing was performed on the four breakers which were procured as spares and subsequently installed as replacements on Unit 3.

The fifth spare breaker is untested and has never been installed.

4. Startup and Surveillance Tests The acceptance tests discussed above have been performed prior to placing RTB's (including RTB's initially supplied as spares) into service. In addition, the following preoperational,.surveillance and maintenance tests were performed prior to the March 1 and March 8, 1983 surveillance tests:

a. Preoperational testing:

RTB's were tested as part of the Plant Protection System (PPS) preoperational test, PE-357-01, and the PPS response time test, PE-358-01, on both San Onofre Units 2 and 3. All RTB's installed at the time of the tests responded properly.

These tests, however, trip the breakers on combined UV and shunt device operation (as would occur during RPS protective action) and thus would not reveal UV problems.

1) The PPS preoperational test is a comprehensive test of the entire system through RTB operation and Engineered Safety Features actuation (pump starts, etc.).

The RTB test in Section 8.10.2 of PE-357-01 is a functional trip test which looks for breaker operation only, and does not time breaker opening or otherwise detail its operation.

The Unit 2 preoperational test was performed on May 30, 1981, and the Unit 3 preoperational test on August 25, 1982.

No RTB failures occurred.

2) The PPS response time test was performed on both Units 2 and 3 to meet the requirements of FSAR 14.2.12.72S.

This test recorded electronic response times from the sensor to the RTB's or ESFAS subgroup relays, as appropriate. Mechanical response times of pumps and valves were measured in other

-16 tests and added on to these results to yield total ESFAS response times. In the case of RTB's, the time from the de-energization of the matrix relays to the opening of the RTB's was measured as a subcomponent of total RPS response time. The acceptance criterion was <120 ms. 'The longest time (last RTB to open) was recordedin each case; Unit 2 was tested in November 1981 with the longest time being 60 ms.

and Unit 3 was tested in August 1982, recording a maximum time of 100 ms. No breaker failures occurred during these tests.

b. Surveillance testing Surveillance testing of Reactor Trip breakers is performed in accordance with S023-II-1.1, "Plant Protection System - Channel Function Test,," S023-II-3.1, -3.2, -3.3, -3.4, "Plant Protection System Channel Response Time Tests," and S023-II-11.161, "Reactor Breaker Undervoltage and Shunt Device Circuit Test."

These tests are performed on the RTB's installed in the switchgear cabinets, and on replacement RTB's as discussed below:

1) The PPS Channel Functional Test is a comprehensive test of the protection-system from sensor output through RTB operation and Engineered Safety Feature actuation similar to the PPS preoperational test discussed previously in this report. This test is performed at least once per month in accordance with the Technical Specifications, and for the RTB's is a functional trip test using combined undervoltage and shunt trip operation.

As operation of the shunt or UV device satisfies protection system design criteria, this test verifies RTB function in accordance with design requirements. The RTB's have never failed to trip during this test.

2) The PPS Channel Response Time Tests measure electronic response times from the sensor to the.RTB's or ESFAS subgroup relays, as appropriate. Mechanical response times of pumps and valves were measured in other tests and added on to these results to yield total ESFAS response times. These four test procedures (one per Channel) are essentially identical to the preoperational response time test discussed previously in this report. As stated previously no RTB failures were experienced during response time testing.

3) The RTB Undervoltage And Shunt Device Circuit Test separately tests the UV and shunt device trip functions. This test is required to be performed at least once per 18 months and following maintenance or adjustment of the RTB's, in accordance with the Technical Specifications.

On Unit 2, this test was initially performed monthly and after any repair or replacement of RTB's. Following correction of UV trip failures on 4 Unit 2 RTB's in early 1982 and successful

-17 completion of 2 more monthly tests on each RTB, SCE reduced the frequency of this test to that specified by CE and the Technical Specifications. Two additional Unit 2 test failures were observed in July 1982 following UV coil adjustment. An RTB functional test was performed to verify function in accordance with the criteria for plant protection system design. The separate UV and shunt device surveillance was not completed before resuming the rod drop tests and LER 82-176 was submitted to document this occurrence. The surveillance test was successfully completed in July 1982 (Unit 2) and in August, 1982 (Unit 3).

An additional surveillance test was performed on the Unit 3 RTB in October 1982. There were no further UV device surveillance tests until the March 1 and 8, 1983 surveillance tests discussed later in this report.

c. Maintenance testing Maintenance, including preventive maintenance (PM), corrective maintenance and testing as part of maintenance is performed in accordance with Work Orders. The maintenance relating to Reactor Trip Breakers is discussed below:

1) The preventive maintenance program was developed from vendor, NSSS and Engineer-Constructor (CE and Bechtel) experience and recommendations.

MPEG-025 is a generic overhaul procedure for 600 volt and below air circuit breakers.

Work in accordance with this procedure has not been performed on the RTB's. MPES-008 is a specific maintenance procedure for the RTB's developed in response to IE Bulletin 79-09.

This procedure addresses overall lubrication and other general maintenance items as well as specific steps pertaining to UV design in accordance with the guidance in IE Bulletin 79-09.

MPES-008 was performed on the RTB's originally installed in Unit 2, in March through May 1981 but has not been performed on the RTB's originally installed in Unit 3 or on RTB's supplied as spares. Preventative maintenance planning had scheduled this to be done at the first refueling on the basis that the RTB's, as delivered, would not require PM before that time. Maintenance procedure S023-I-4.36 was developed from MPES-008 for use at first refueling but has been cancelled because, through an administrative error, S023-I-4.36 was initially issued with key information from MPES-008, deleted. This information has been added to a new procedure (S023-1-4.66) which also incorporates the recommendations in Sections IV.D.2 and V.A of this report.

Although there is no Technical Specification requirement to perform a baseline PM for all equipment, it is SCE's pracitce to do this and it was done for the Unit 2 RTB's. For the Unit 3 and spare RTB's, however, SCE relied on the delivered condition of the RTB's and did not specify PM until the first refueling.

-18

2) Corrective maintenance on the RTB's was performed in response to problems documented in Non-Conformance Reports (NCR's); a Work Order is then written against the NCR to correct the problem. For the Unit 2 RTB undervoltage trip failures in early 1982, the NCR was dispositioned to correct the problem per vendor direction, based on a perception that failure of the UV device to function reliably made SCE procedure MPES-008 suspect (it had been used earlier for PM on these RTB's) and that vendor assistance was needed to ensure that no unusual problems existed with these safety related components. The Work Order specified that the undervoltage coil pickup voltages be reset to vendor technical manual (S023-944-352) criteria by SCE electricians under vendor (General Electric) supervision. For the July 1982 RTB adjustment to facilitate breaker closing, SCE electricians reset the undervoltage coil adjustments on all Units 2 RTB's to vendor technical manual criteria without vendor assistance. When two RTB's subsequently failed the July 1982 undervoltage surveillance tests, CE recommended.that a vendor service representative check the UV coil adjustments for all Unit 2 RTB's. The UV coil adjustments were then reset (some outside the vendor technical manual limits) by the vendor representative to obtain suitable RTB (close and UV trip) operation and the NCR was dispositioned to accept these settings as-is, based on the perception that the vendor representative-determined settings were the best technical information available. For the Unit 3 and spare breakers, a Work Order had been written to perform both MPEG-025 and MPES-008 maintenance procedures, but was then cancelled in favor of work orders to adjust per vendor technical direction consistent with Unit 2. Following these activities, the surveillance test was repeated on all RTB's for Units 2 and 3 with satisfactory results.

Further discussion of the above maintenance process is provided later in this report.

-19 D. March 1 and 8, 1983 RTB Surveillance Tests

1. Description of Tests The NRC issued IE Bulletin No. 83-01 "Failure of Reactor Trip Breakers (Westinghouse DB-50) To Open On Automatic Trip Signal" on February 25, 1983.

Even though IE Bulletin No. 83-01 did not require testing of the General Electric RTB's installed at San Onofre Units 2 and 3, surveillance testing was performed in accordance with station procedure S023-II-11.161 "Reactor Breaker Undervoltage and Shunt Device Circuit Test."

This procedure tests the UV device and shunt trip device independently as described in Section III.D.4 of this report.

2. Test Results Testing was performed on all Unit 3 RTB's on March 1, 1983. One breaker, RTB 4, type AK2-25-2 serial #256A4002-656-17, failed to trip on its undervoltage test. It performed properly on shunt trip.

NCR-3-243 was written to document the nonconformance. Since this was viewed as an isolated failure, and the CEDM cabinets were not energized, the failure was not considered to be reportable pursuant to the Technical Specifications at this time. Testing on Unit 2 was delayed by the presence of a failed power supply in the PPS. After the power supply was repaired, testing of Unit 2 RTB's commenced on March 8, 1983. This testing revealed three RTB's (all type AK2-25-2) which did not trip on undervoltage:

RTB#

Serial #

1 256A4002-656-3 4

256A4002-656-24 6

256A4002-656-8 All breakers successfully passed their shunt trip test. NCR-2-163 was written to document these failures and the NRC was notified of the surveillance failures.

-20 IV.

Investigative Programs As a result of the March 1 and 8, 1983 surveillance failures of the GE Type AK2-25 RTB at San Onofre Units 2 and 3, SCE conducted an investigation to determine the cause and ramifications associated with the surveillance failures. This section of the report discusses SCE's efforts in the following specific areas to determine the root cause of the problem and to identify specific corrective actions and follow-up activities for resolution of RTB concerns and related programmatic deficiencies:

A. RTB Investigative Tests

1. Description of Tests Following the Reactor Trip Breaker surveillance tests of March 1 and 8, in response to IE Bulletin 83-01, SCE initiated further investigation and testing of San Onofre Units 2 and 3 Reactor Trip Circuit Breakers. This further investigation and testing has been performed in three parts: (a) initial investigative tests March 12-17, 1983 at San Onofre, (b) in-depth testing March 26 - April 1, 1983 at the SCE Electrical Test Laboratory in Alhambra, (c) test results of nine RTB's at San Onofre Unit 2 obtained during baseline preventative maintenance and independent testing by Franklin Research Center (requested by NRC). A discussion of investigative test parts a, b, and c follows; part d will be provided directly to the NRC by Franklin Research Center.

a. Initial Investigative Tests (March 12-17, 1983)

1) Following an introductory meeting with representatives of NRC and Franklin Research Center, SCE initiated an investigative plan for determining problems with Reactor Trip Circuit Breakers. A procedure was developed and made available for review and comments by NRC and Franklin Research Center.

Comments were provided by the above organizations for SCE consideration and were factored into the final investigative procedure.

2) A General Electric Company field service engineer, who had previously provided vendor assistance on Reactor Trip Breakers, was called in.

In addition, Combustion Engineering provided contact with the General Electric Company to request the assistance of a factory expert in the investigation of the Reactor Trip Breakers. The General Electric Company provided Mr. Max B. Fornwalt, Senior Project Engineer, who arrived on Sunday afternoon, March 13, 1983. It should be noted that Mr. Fornwalt authored the General Electric Service Letter that subsequently became part of IE Bulletin 79-09.

3) Following the assessment of comments, a final investigative procedure was prepared. Work Orders were generated to implement the investigation. Reactor Trip Breaker TCB2 (functioning breaker) was tested to verify the adequacy of the investigative procedure. Reactor Trip Breakers TCB1 and TCB6 (two of the three malfunctioning Unit 2 breakers) were then tested with the same procedure. Each of these breakers was visually inspected and tested for undervoltage trip while still installed in the switchgear cubicle, then removed and tested for trip shaft torque, undervoltage trip response time, and undervoltage device pickup voltage. The RTB lubricant was revitalized as necessary and changes were made in the UV pickup voltage levels.

4) The observations of the initial investigative tests are as follows:

a) Two breakers were missing the locking wire for undervoltage coil pickup voltage adjustment. One of these (malfunctioning breaker TCB1) had a lower than nominal pickup voltage (101 VDC vice 106 VDC) and the other (functioning breaker TCB2) a higher than nominal pickup voltage (112 VDC vs. 106 VDC).

b) The as-found trip shaft torques all exceeded the 1.50 pound-inches specified by IE Bulletin 79-09. The as-found torque was slightly higher in TCB2 (functioning UV device trip) than TCB1 (malfunctioning UV device trip) but, as noted above, TCB2 also had a higher than nominal pickup voltage; TCB2 also exhibited erratic behavior (slow trip) when its pickup voltage was lowered to 100 VDC.

c)

Both malfunctioning breakers tripped satisfactorily on repeated undervoltage trip tests with as-found or lower undervoltage pickup adjustments when the bearings lubricant was revitalized to reduce trip shaft torques to less than the 1.5 pound-inches specified by IE Bulletin 7 9-09.

d) The shunt trip device successfully tripped the breaker if the undervoltage device failed to do so.

e) The GE Factory Representative stated that he could find no evidence of improper handling or mechanical damage to the Reactor Trip Breakers examined. Further, all mechanical adjustments were satisfactory with the exception of undervoltage device pickup voltage and a minor increase needed in the TCB6 overtravel adjustment (made after successful testing of the breaker).

-22

5) Preliminary conclusions of the initial investigative tests were as follows:

a) The major contributing factor to improper Reactor Trip Breaker operation on undervoltage was due to insufficient or degraded lubricant in the trip shaft and latch roller bearings.

b) The secondary contributing factor was the undervoltage device armature pickup voltage being set below the recommended 106 VDC.

b. In-Depth Tests (March 26 - April 1, 1983)

1) Based on the results of the initial investigative tests discussed in Section IV.A.l.a above, a more detailed inspection was performed on one of the Reactor Trip Breakers to obtain as much quantitative information as possible with regard to the dynamic operation of this breaker under various conditions. The breaker subjected to this in-depth test and inspection was Reactor Trip Breaker TCB-4 from Unit 2, the third of the three breakers which malfunctioned during surveillance testing in early March, 1983. The other two malfunctioning breakers (TCB1 and TCB6) had been previously tested and readjusted as discussed in Initial Investigative Tests,Section IV.A.l.a of this report.

2) The work was performed by SCE at our Electrical Test Laboratory with the aid of a General Electric service representative. The breaker was tested on a bench and when operational tests were made, the breaker was secured to the bench by the same breaker flanges which support the breaker when installed in the cabinet. High speed photography was used to assess breaker trip performance. A magnetic oscillograph was also used to record the following parameters as required for dynamic tests on the breaker.

o Breaker main contacts (3) o Shunt trip coil current o Shunt trip coil voltage o Closing coil current o

Undervoltage trip device current o

Undervoltage trip device voltage o

Auxiliary "b" switch contact

3) The breaker was visually inspected and.tested in the as received condition and baseline measurements were obtained for all parameters, including undervoltage coil pickup and dropout voltage, trip shaft torque, trip response time, undervoltage coil armature air gap, and all electrical component resistances. Adjustments were varied and tests repeated to determine optimum settings and limitations. The

-23 investigation also included inspecting, cleaning, and revitalizing the trip shaft and latch roller bearings, and cleaning and adjusting the undervoltage device. UV device pickup voltage was also investigated.

4) The results of the in-depth investigative tests were as follows:

a) As-received trip shaft torque was greater than the 1.5 inch-pound limit and as-received undervoltage device pickup voltage lower than the 106 VDC limit specified by IE Bulletin 79-09. Cleaning and revitlizing the trip shaft and roller bearings reduced the trip shaft torques to less than 1.5 inch-pound and successful UV operation was obtained. This confirms the preliminary conclusions of the initial investigative tests discussed in Section IV.A.l.a of this report.

b) Considerable variation of the UV device pickup voltage setting will result from variations in the UV device coil temperature during pickup voltage adjustment. A minimum of 30 minutes is required for the UV device coil to reach a stable thermal state.

c) The optimum adjustment for undervoltage device armature pickup voltage is 106 + 2 VDC at a "cold" UV device coil temperature of 700 to 950F.

d) The diode installed across the UV device coil for surge protection of the PPS relays delays the breaker response time (nominal 30 millisecond difference) although it remains within allowable values. This diode is installed in the reactor trip breaker cubicle wiring and is not present on a removed breaker. Therefore, a diode is required during Reactor Trip Breaker bench testing.

e) Excessive clearance between the UV device armature magnet and restraining rivet reduced the effective throw of the armature by permitting it to move up against the rivet rather than rotating.

The as-found clearance of 0.018 inches exceeded the 0.001 to 0.010 range recently recommended by the manufacturer; a somewhat narrower range (0.003 to 0.006 inches) will provide improved performance and is consistant with the G.E. factory range of 0.001 to 0.010 inches.

f) As-received condition of the trip latch roller bearing (rough operation and excessive clearance) may have resulted in variation of trip shaft torques with roller position but did not affect trip reliability when pickup voltage and trip shaft torque were within desired range.

-24 g) There is ample design margin in the shunt trip device.

Operation of the breaker with the shunt trip was satisfactory down to approximately 30 VDC; the voltage available is nominally in excess of 130 VDC.

h) The undervoltage response time of the breaker is faster and more consistent for a well lubricated breaker than one with degraded lubricant.

5) The conclusions of the in-depth investigative tests were as follows:

a) Two major contributing factors to improper Reactor Trip Breaker operation on undervoltage are degraded lubricant and undervoltage device armature pickup voltage adjustment; this confirms the preliminary conclusions of the initial investigative tests discussed in Section IV.A.l.a above.

b) A third major contributing factor to improper RTB operation on undervoltage is armature/magnet/rivet clearance in the UV device.

(This may have been isolated to this breaker).

c) A minor contributing factor to breaker response time variation could be damage to the trip latch roller bearing.

d) The undervoltage device armature pickup voltage should be adjusted to 106 + 2 VDC at a "cold" UV coil temperature of 700 to 850F.

e) The UV device dropout voltage should be measured (after pickup voltage adjustment) with the coil energized a minimum of 30 minutes.

f) The Reactor Trip Breaker response time should be measured on clean, lubricated and properly adjusted breakers to establish a baseline. Further response time tests during breaker surveillance would detect any degradation of breaker performance from the baseline, thus identifying potential incipient failures.

g) The Reactor Trip Breaker response time on the bench should be measured with a test diode connected across the UV device coil (testing in the cubicle would have the installed diode in the circuit).

h) With preventative maintenance as dictated by the results of UV response time tests during enhanced surveillance, there is adequate design margin in the UV devices; no changes are needed to the undervoltage trip design.

-25 i) There is ample design margin in the shunt trip device; no changes to the shunt trip design are needed.

c. During the period April 6th through April 10th, 1983 preventative maintenace (PM) procedure S023-I-4.66 was implemented to maintan the breakers and establish the base line preventative maintenance data for Unit #2 reactor trip breakers.

1) As Found Valves a) On six of the nine breakers, the undervoltage device armature to rivet clearance was not within the required range.

b) Seven of the nine undervoltage device pickup voltages were less than the required 104 to 108 volts.

c) Six of the nine trip shaft torque valves exceed the required 1.5 in-lbs.

2) Final Post-Maintenance Base Line Valves a) All trip torque valves were less than 1.26 in-lb. (most were in the range of 1.0 to 1.1 in-lbs.)

b) All RTB's trip times were less than 70 msec.

c) No other bearing problems were found. It is therefore concluded that the bad bearing found on TCB-4 during the in-depth investigation was an isolated occurance.

d) All TCB's operated satisfactorily in all respects following completion of the preventative maintenance.

2. Results of Investigative Tests The conclusions of the investigative tests performed by SCE are discussed in Sections IV.A.l.a and IV.A.1.b of this report.

Pertinent recommendations based on these conclusions are provided in Sections IV.D, V.A.1 and V.A.2 of this report.

-26 B. Evaluation of Licensee Administrative Procedures (Later)

-27 C. Evaluation of Capability to Mitigate ATWS

1. Procedures for Mitigating ATWS Prior to the March 1 and March 8, 1983, events regarding the Reactor Trip Breaker undervoltage trip function, Emergency Operating Instruction (EOI) S023-3-5.1 "Emergency Plant Shutdown" (Revision 8) (this procedure is for any plant trip situation and includes the plant's ATWS procedural steps), included an instruction in the first step of the "Immediate Operator Action" section requiring operators to verify that all Reactor Trip Breakers are open and reactor power is decreasing.

If the reactor is not tripped, the procedure then requires that all four manual reactor trip pushbuttons be actuated. In addition, the following specific ATWS actions are called for:

a. De-energizing load centers B15 and B16. This interrupts power to the control element drive mechanism motor generator sets and therefore, removes power from the CEDM's regardless of Reactor Trip Breaker position.

b. Manually initiate emergency feedwater actuation signals.

c. Initiate emergency boration.

Revision 9 of EOI S023-3-5.1 was issued on March 25, 1983, and implemented additional ATWS steps which require the initiation of a manual reactor trip when an automatic trip set point is being approached or has been reached regardless of whether the reactor has tripped automatically and to manually trip the turbine.

2. Control Room Layout/Design The control room layout drawing (Figure IV.C.2-1) identifies control board locations of indications and instrumentation which can be used to identify whether or not an ATWS event is occurring and controls available to aid in the mitigtion or prevention of such an event. All locations depicted on Figure IV.C.2-1 are numbered relative to the sequence in which they are contained in the "Immediate Operator Action" section of Revision 9 of EOI 5023-3-5.1, "Emergency Plant Shutdown."

Figure IV.C.2-1, as drawn, shows only the San Onofre Unit 2 side of the Control Room, but with the "same hand" arrangement of the control room indication and controls identified in this drawing are located as shown in either the Unit 2 or Unit 3 control room area.

As shown on Figure IV.C.2-1, the indications and instrumentation which can be used by operators to detect and initiate ATWS, are easily accessable to operators.

4 II (D

) RPS Remote Operntor's Module 2 or more channels indicate trip Depress RX trip pushbuttons Depress RX trip pushbuttons Rcnctor trip hkr stntus panl verify all 8 TCs open Excore linear power recorders verify power <6% F.P.

CEA Bottomed Indication BAnncinto verify all CEA bottom lites on OA De-energize MCC 11521 (619 De-energize NCC Bl6 7A Initiate EFAS 1 & 2 UNT 271 Initiate EFAS I & 2 I

2Initiate emergency boration:

2A CVCS Valves & Blend Control Emerg. boration path Gravity feed valves for baorxN, 6v Manually trip turbine Verify feedwater flow to S/Cs 17 J

/o Annunciator panel 50A31 Mzr Safety Vv out. Tempi-HL De 50A49 CEDMCS Bus Undervoltage Boronometer -

verify RCS between P --

n----------------.1750 and 2250 ppmB before stopp na ing emergency boration FIGURE IV. C. 2-1 CONTROL ROOM LAYOUT/DESIGN ATWS INSTRUMENTATION/CONTROLS

-28

3. Operator Training/Knowledge Following the issuance of IE Bulletin 83-01, on-shift training sessions were conducted for each operating shift reviewing the ATWS operations included in EOI S023-3-5.1 "Emergency Plant Shutdown." In addition, each licensed operator was required to review the IE Bulletin referenced above and IE Bulletin 83-04, when it was issued. After the revision of EOI 5023-3-5.1 on March 25, 1983, all licensed reactor operators were required to review and acknowledge, in writing, this review of the revision to the procedure. In addition, the five operating shifts will receive, as part of the requalification training program, formal classroom training and discussion relative to the revised procedure. This will occur over a five-week period, which started the week of March 28, 1983.

-29 D. Findings/Conclusions Regarding RTB Failure

1. The reactor trip breaker (RTB) undervoltage (UV) trip device is not required for the San Onofre Units 2 and 3 reactor protection system (RPS) to perform its design basis protective function; failure of the UV trip device does not affect the ability of the San Onofre Units 2 and 3 RPS to perform its protective system function due to the presence of the shunt trip.

2. From the results of the investigation of four RTB's discussed in Section IV.A of this report, it is concluded that the cause of the unreliable operation of the undervoltage (UV) trip function for the Reactor Trip Breakers was a combination of the small design margin in the force provided by the UV trip device and the following:

o Degraded lubricant on the trip shaft bearings or latches.

o Incorrect setting of the UV device pickup voltage.

o Excessive clearance in UV device armature hinge area.

3. Despite the small design margin in the force provided by the UV trip device, the UV trip feature of the RTB can be made to operate reliably when enhanced maintenance and surveillance techniques are used. This conclusion is based on the results of the tests and inspections of three RTB's at San Onofre, the results of in-depth testing of one RTB at the SCE Electircal Test Laboratory and the results of the review of RTB maintenance and surveillance history.

The enhanced maintenance and surveillance needed to ensure reliable operation of the UV trip feature, based on these results, are as follows:

a. Mkintenance The enhanced maintenance procedures for RTB's should include the following:

1) Trip shaft torques, UV device pickup voltages, and UV opening times should be determined for the breakers in their as-found condition:

a) Measure and record the pickup voltage of the UV device with its coil surface temperature within 70 to 850F.

Also, measure the pickup and dropout voltages after the UV device coil has been energized at 130 volts DC for thirty minutes. Ambient and coil surface temperatures should also be measured and recorded.

-30 b) Measure the opening time of the circuit breaker with full loss of voltage to the UV device. This time is to be measured from the loss of DC control voltage and to be breaker main contact part. Three operations and measurements are to be made using an oscillagraph recording device, with a test diode connected across the UV device coil.

c) Measure and record the trip shaft torque which is required to trip the circuit breaker. This measurement is to be made three times using a specially made tool to fit on the trip shaft.

2) Prior to the current plant restart, the RTB should be disassembled to the point where the operating mechanism can be removed. The trip latch roller should be inspected for damage and all bearings and bearing points should be checked for mechanical wear and freedom of rotation. Clean all parts with isopropyl alcohol.

Apply CRC 5-56 to all bearings and bearing points.

All excess CRC 5-56 should be removed to prevent the collection of dust and contaminants.

3) Remove the UV device to permit inspection and to check the adjustment of the clearance between the UV device armature and the rivet (0.003 to 0.006 inches). The mechanism portion is to be cleaned with isopropyl alcohol and then apply CRC 5-56 to all pivot points. Excess CRC 5-56 should be removed.

4) After reassembly of the circuit breaker, the mechanical adjustments should be checked and adjustments made as required.

5) Repeat the circuit breaker performance measurements:

a) Check and adjust the pickup voltage of the UV device when its coil surface temperature is within 70 to 850F.

This should be set to 106+ 2 VDC. Also, measure the pickup voltage after the UV device coil has been energized at 130 volts DC for thirty minutes.

b) Measure the opening time of the circuit breaker with full loss of voltage to the UV device. This time is to be measured from the loss of DC control voltage and to the breaker main contact part. The criteria for the main contact part shall be approximately 70 milliseconds or less subject to confirmation by baseline testing of all breakers. Three operations and measurements are to be made.

c) Measure and record the trip shaft force to trip the circuit breakers. This force shall not exceed 1.5 pounds-inches. This measurement is to be made three times using a specifically made tool to fit on the trip shaft.

-31

b. Surveillance Enhanced surveillance for RTB's will be made to monitor the performance of the circuit breakers for indications of degrading UV trip operation, in order to determine the need for (and to permit performance of) restorative maintenance before UV trip reliability is affected. The enhanced surveillance for RTB's includes the following:

1) After the circuit breaker maintenance is completed, a surveillance test should be made. This should include three UV opening time tests. The opening times for the three tests should not exceed the allowable limits shown in Item 5b under maintenance, above.

2) Repeat the surveillance test four more times at approximately 30-day intervals (i.e., monthly).

3) After four sequential months of UV trip function surveillance testing without UV trip failure or degradation, the surveillance testing interval may be increased to at least once per 62 days (i.e., bimonthly).

4) After eight more sequential months of successful UV trip testing, the surveillance testing interval may be increased to at least one per 92 days (i.e., quarterly).

5) After 12 more consecutive months of successful UV trip testings, the surveillance testing interval may be increased to at least once per 184 days (io.e., semi-annually).

6) If the average of the three UV trip times or the scatter of the three trip times during each test increases significantly from the baseline surveillance tests, or exceeds the baseline-adjusted limits, the maintenance procedures should be repeated and new surveillance tests initiated.

7) Surveillance and maintenance frequency should continue to be adjusted on the basis of surveillance and maintenance test results. The minimum RTB maintenance frequency should be one per 12 months (i.e., annual PM). Unit 3 surveillance (and maintenance) should be initiated at the frequency in effect for Unit 2 at that time.

4. The reactor trip breaker UV device surveillance failures were due to the small design margin in UV trip force and the hardware condition resulting from the absence of the enhanced surveillance and maintenance techniques which had not been developed or identified by the industry as needed at that time. This conclusion is based on the review results discussed in paragraphs 2 and 3 above. Based on this conclusion and the fact that equipment difficulties are commonly experienced during the startup and construction phases, it must be

-32 further concluded that incorporation of the potential improvements to SCE administrative procedures and implementations which have been identified as a result of the reviews discussed in Section IV.B of this report might have reduced the likelyhood but would not have precluded the UV trip function malfunctions observed.

5. A procedure which addresses ATWS was in place at San Onofre Units 2 and 3 prior to the Salem RTB trip failure. This procedure, S023-3-5.1, "Emergency Plant Shutdown" has been revised to improve operator response to an ATWS event by requiring immediate manual reactor trip and then manual turbine trip when an automatic trip setpoint is being approached, regardless of whether the reactor has tripped automatically. The San Onofre Units 2 and 3 Control Room is already configured such that the instrumentation and controls to detect and mitigate ATWS are easily accessible to the operators.

6. The existing SCE operator training/retraining program contains sufficient provisions for promptly notifying operators of important procedural changes and for additional reinforcement through follow-on training and retraining.

-33 V.

Proposed Corrective Actions Short term and longer term corrective actions are summarized below based on discussions provided elsewhere in this report.

A. Short Term Corrective Action Leading to Restart

1. (Later)

2.

(Later)

3. Surveillance Procedures As discussed in Section IV.A.1, surveillance testing of the RTB will incorporate conclusions identified during the investigative tests conducted by SCE. The details of this surveillance testing will be as described in Section IV.D.3.b of this report.

If at any time during this surveillance program, degradation or failure of the undervoltage device occurs, the failure will be analyzed, corrected, and used to enhance the preventative maintenance procedure to prevent recurrence. Additionally, the surveillance testing program will be re-zeroed to the beginning of the interval phase in progress to ensure successful correction of the problem prior to extending the test interval.

When Unit 3 attains initial criticality, its testing and maintenance will be initiated at the same periodicity in effect for Unit 2.

The monthly P.P.S. testing under S023-II-1.1, which tests the RTB shunt and undervoltage trips together, will be unchanged.

4. Technical Specification SCE will review the need for Technical Specification changes after the appropriate maintenance and surveillance frequencies have been identified.

5. Operator Training As discussed in Section IV.C. of this report, the "Emergency Plant Shutdown" procedure for San Onofre Units 2 and 3 operating instruction for any plant trip situation including ATWS procedural steps has been modified to require the initiation of a manual reactor trip when an automatic reactor trip set point is being approached or has been reached regardless of whether the reactor has tripped automatically and to manually trip the turbine.

All licensed reactor operators have been reviewed and acknowledged in writing, review of the revision to this procedure. Additionally the five operating shifts will receive, as part of the requalification training program, -formal classroom training and discussion relative to the revised procedure. This will occur over a five-week period which started the week of March 25, 1983.

w

-34 SCE is confident that the procedural revision along with the review of the revised "Emergency Plant Shutdown" procedure which is being reinforced through the operator requalification training program will enable operators to detect and mitigate ATWS.

-35 B.

Long Term Corrective Actions Subsequent to Restart (Later)

X

-36 VI.

Conclusions

1. From the standpointof the reactor protection system design, the UV coils are not required to operate in order for the breakers to trip under any design basis condition.

In addition, the conservatice reactor, protection system design at San Onofre provides added assurance (compares to similar vintage plants of other designs) that an ATWS event will not occur.

2. Exhaustive breaker testing in SCE's laboratories was performed on one of the breakers that exhibited unsatisfactory performance.

SCE was able to do this evauation because of its experience and capabilities for testing of this nature that are largely unique to SCE. The experienced staff and facilities for such testing do not exist in most utilities throughout the country. This testing identified three key areas where additional attention needs to be devoted during maintenance and survillance. These are:

a) Lubrication of key components b) Performing UV coil adjustments at a known temperature c) Maintaining the UV armature to rivet clearance within recommended close tolerances Based on the results of the investigative testing, it is clearly understood how to adjust and maintain the breakers to provide a high degree of reliability for the UV trip function.

3. The proposed maintenance and surveillance programs will provide this high degree of reliability. The initial conservative frequency will be adjusted based on actual experience.

4. Based upon the above, and the fact that the shunt coil has never failed to trip the breaker, there is a very high degree of assurance that the RTB will operate if called upon to do so. Therefore, the health and safety of the public is assured and any and all restrictions limiting operation of San Onofre 2 and 3 associated with the RTB should be removed.