Text

SER Lifting Restrictions on Exceeding Power Documented in Confirmation of Action Ltrs & 0429 Based on Corrective Actions Re Reactor Trip Breaker Failure
	ML13309B266
Person / Time
Site:	San Onofre
Issue date:	04/30/1983
From:	; Office of Nuclear Reactor Regulation
To:	;
Shared Package
ML13309B265	List: ML13309B264; ML13309B266;
References
	TASK-***, TASK-TM NUDOCS 8305180804
	Download: ML13309B266 (58)
	v • d • e

j a

Safety Evaluation Report Concerning The Reactor Trip Breakers at San Onofre Nuclear Generating Station, Units 2 and 3 Docket Nos. 50-361 and 50-362 Southern California Edison Company, et. al.

by the U.S. Nuclear Regulatory Commission Office of Nuclear ReactorRegulation April 1983 8305180804 830502 PDR ADOCK 05000361 s

PDR

TABLE OF CONTENTS Page I. INTRODUCTION AND

SUMMARY

I II. BACKGROUND INFORMATION.............

II-1 A.

Reactor Protection System (RPS).

-1 B. Reactor Trip Breakers (RTB)...............

11-4

1.

Design Criteria, Basis, Safety Classification.

11-4 2

Design Description.

11-4

a. Breaker Assembly, Trip Bar and Latch......

II-4

b. Shunt Coil Assembly 11-5

c. Undervoltage (UV) Coil Assembly.........II-5 C.

RTB Surveillance!Tests, March 1 and 8, 1983-.......

11-5

1.

Description of Tests.....

11-5

2.

NRC Staff Action Resulting from Tests.....

II-10

3.

Licensee Investigative Program...

I....

II-10 III. INVESTIGATIVE PROGRAMS.......

1 A.

RTB Investigative Tests III-1

1.

Initial Investigative Tests (March 12 to 17, 1983).

III-1

2. In-Depth Tests (March 26 to April 1,' 1983)..

111-2 B. Evaluation of Licensee Adminstrative Controls......

111-4

1.

Control of Vendor Information, Technical Manuals.

111-4

2.

Control of Hardware Configuration.....

111-5

3.

Maintenance Program...

........... 111-7

a. Procedures and QA/QC Requirements...

11....

I1I-7

b.

Maintenance History and Records'..H

1.

III-8

c. Vendor Maintenance Activities....

III-9 4

Surveillance Program............

III-10

a. Procedures and QA/QC Requirements....

III-10

b. Surveillance History and Records......

111-12

c. Adequacy of Technical Specification Requirements....

111-13 San Onofre RTB SER v

74 --

l", 7-V

TABLE OF CONTENTS (Continued),

Page

5.

Reporting of Failures.................

111-14

6.

Compliance with IE Bulletins and Circulars..

111-16

7. Post-Trip/Restart Reviews...............111-18

8. Results of Previous Evaluations of Licensee Administrative Controls...............

11-20

a.

SALP.......

111-20

b.

INPO...................

111-20

c. Licensee Action Addressing Previous Evaluation Findings.

111-21 C. Evaluation of Capability to Mitigate ATWS..

111-22

1.

Control Room Design..111-22

2.

. Reactor Trip and ATWS Procedures....

111-25

3. Operator Training/Knowledge..111-26 D. Licensee Conclusions Regarding Causes of RTB Failures,.

111-28' E. Staff Evaluation and Independent Conclusions.

111-29

1. Potential Consequencesof Failure of RTB UV Mechanism..n.

..111-29 2.,

Summary of Contributory Causes............111-29

a. Design Considerations....a-32

b. Maintenance.....................

111-32 c.,

Surveillance.................... 1-4

3. Implications For Other Plant Systems-and Components.

111I-35 IV. PROPOSED CORRECTIVE ACTIONS AND BASIS FOR RESTART........IV-1 A.

Control ofHardwares RegadingCauss.o. RT..

IV-1 B. Control' of Vendor Information and Personnel.......

.... IV-1 C. Maintenance Procedures and QA/QC Requirements.......IV-2 D. Surveillance"-Procedures and Technical Specifications...

IV-3 E.

Long Term Corrective Actions.................

IV-4 F.

Operator Training.

IV-4 G. Conclusions Regarding Adequacy, of Program.........IV-4 FIGURES

1. Simplified Functional Diagram of the Reactor Protection System......................................

11-2

2.

GE AK-2 Trip Device..............

11-3

3. Trip Shaft Sketch.......................

11-6

4.

GE AK-2-25 Circuit Breaker.....

11-7

5.

Shunt Trip Device......

-8

6.

Undervoltage Tripping Device.........

11-9 7.'

Control Room Layout..........

................. I-23 San Onofre RTB SER vi E.7

TABLE OF CONTENTS (Contihued)

Page APPENDICES A. PRINCIPAL NRC STAFF REVIEWERS;.....

A-1 San Onofre RTB SER vii 7-F

I INTRODUCTION AND

SUMMARY

NRC IE Bulletin No. 83-01, Failure of Reactor Trip.Breakers (RTBs) (Westing house DB-50) to.Open On Automatic Trip Signal, issued on February 25, 1983, discussed the February 25, 1983 failure of the DB-50 RTBs to open automatically upon receipt of.a valid trip signal at Salem Unit 1. The reactor was manually tripped from the control room'about 25 seconds later, and the trans.ient was successfully terminated.

As a result of failure of reactor trip circuit breakers (RTBs) to function at Salem 1 and.the issuance of IE Bulletin 83-01, one of the SanOnofre licensees, Southern California Edison Company (SCE), performed RTB surveillance tests at San Onofre Units 2 and 3, independently.testing undervoltage (UV) and shunt trip functions of the RTB. This was done even though the RTBs are of a dif ferent design (GE-AK-2-25) than Salem and were not required by the bulletin to be tested., The surveillance tests were performed on March 1 and.8, 1983 when San Onofre Units 2 and 3 were both.shut down and in different stages of their respective startup test programs. Four of the total of 16 RTBs tested-failed to trip following.actuation of their UV devices.

All 16 tripped upon actuation of the shunt devices.

After being notified of the RTB failures at San Onofre 2 and 3, the NRC staff issued IE Bulletin No. 83-04 on March 11, 1983, which notified owners of plants having breakers other than those covered by IEB 83-01 of the events at San Onofre, and advised them that their RTBs should also be tested.

After the San Onofre RTB failures occurred, a series of tests was conducted to determine the causes of the failures.

These tests were conducted at the San Onofre site (March.12-17, 1983) and at the SCE Electrical Test Laboratory (March 26-April. 1, 1983). The NRC staff met with SCE to discuss the RTB issue at the San Onofre site on March 12-17,'1983, and in Bethesda. Maryland on April.12, 1983.'. A special inspection relating.to RTBs was conducted by NRC Region V personnel.during the period March 12-25, 1983, and is described in Inspection-Report No. 50-361/83-13, dated April 14, 1983..0nApril 15, 1983, SCE submitted a report describing their investigations and the conclusions they reached, and on April 22, 1983, SCE provided a letter committing to propose revised.Technical Specifications for RTB surveillance testing within 30 days and committing -to revise the post-trip procedures within 45 days.

The principal SCE conclusions are summarized as follows:

1. The design of the San Onofre 2 and 3 reactor protection system (RPS) in cludes a high degree of redundancy and diversity. The control element assemblies (CEAs) in each reactor are divided into two groups; each group is controlled by a series-parallel circuit comprised of four RTBs. Each of the four RTBs is energized by a separate power supply (battery). Each RTB contains 2 coils, a shunt coil which is energized to trip and an under voltage (UV) coil which is deenergized to trip.

San Onofre RTB SER 1

77 7:

2. The failures at San Onofre 2 and 3 involved only the UV coils. No shunt coil failures were. observed. Due to the redundancy and diversity in the RPS design, the failure of any or all the UV coils would not have prevented any of the RTBs from tripping, if a trip demand-had occurred.

3. The principal. direct causes of the observed.failure of the UV devices to trip the RTBs were a combination of the small design margin between the force provided by the UV device and the tripping force required, and (a) degraded lubrication of trip shaft and latch roller bearing, (b) too low a.setting of the UV device armature pickup voltage, and (c) excessive clearance between the UV.armature and a' restraining rivet which causes, the armature to rotate around its pivot.

4. The above deficiencies can be corrected by appropriate maintenance and sur veillance of the RTBs.

No design'changes are required.

The staff has reached the conclusions given below as a result of the detailed investigation program developed by the licensee with staff input, close surveil lance.of the program by the staff asit was being conducted, and evaluation by the staff of the results of the program.

1. The. SCE management,, response to IEB 83-01 (the Salem reactor protection system under-voltage breaker failure problem) was to perform tests on the San Onofre. RPS breakers to.determine their adequacy.

The staff concludes that this action demonstrated proper management concern for, reactor safety. When it was determined that some of the San Onofre 2 and 3 breakers failed to meet performance requirements the licensee reported the failures to NRC as required by.the Technical Specifications.

This action further demonstrated the appropriate management concern.

2. The SCE management responded.to NRC's request for a careful and detailed investigation by holding the plant in a shutdown 'condition and maintaining the breakers in the. as-failed position for staff inspection. This provided a basic starting'.point from which to develop the detailed investigations.

This action contributed greatly to the confidence the staff has in the identification of the causes of the.failure and the recommended solution.

3. After review of the history of the control of hardware, vendor information and personnel, maintenance, and surveillance the staff concluded, as did the licensee, that deficiencies in many of. these areas contributed.to the failure of the breakers. The staff and licensee concluded that any long term solution to the problem would require improvements in these areas.

4 The investigation plan developed by SCE, b'ased on staff recommendations, provided the necessary test results to clearly characterize the cause of the breaker failures and the major contributing factors requiring correction to eliminate the problem.

5. The control room design, post-trip and ATWS procedures and operator train ing are adequate to permit the plant operators to respond appropriately to normal reactor. trips and ATWS events.

San Onofre RTB SER 1-2

~'.

6. The San Onofre 2 and 3,reactor protection system provides a considerable degree of redundancy of circuit breakers and redundancy and diversity of trip actuation coils. There are four independent paths to provide power to each of two groups of control rods.

These four-paths are each provided with two reactor trip breakers, for a total of eight reactor trip breakers. For each.half of the control rods,,the minimum number.of breakers that must function to drop.the rods is two; for all rods, four.

7. The licensee has provided (1) revised administrative controls to assure equipment operability and (2) functional tests and revised procedures to assure surveillance testing of the independent functions of the under voltage and shunt devices. This will assure proper assessment of opera bility following maintenance.

8. The deficiencies in maintenance, requirements..and surveillance methods found in 'reviewing the maintenance history of the reactor trip breakers have been' or will be corrected through the actions taken by.SCE and commitments 'made to take further actions.

Based on the above conclusions reached by the staff regarding the causes of the reactor trip breaker failures, and the measures that have been and will be taken by the licensee to remedy these problems, the staff concludes that continued operation of San Onofre Units 2 and 3 will not endanger public health and safety.

San. Onofre RTB SER I-3

II.

BACKGROUND INFORMATION A.

Reactor Protection'System The San Onofre 2 and '3.Reactor Protection System (RPS) is designed by Combus tion Engineering to sense several plant variables and to actuate a trip of the reactor (emergency shutdown) in the event that any monitored plant variable reaches an abnormal value (setpoint). The RPS consists of multiple measurement channel.s.(instrumentation), bistable trip devices, coincident logic matrices, series trip paths and output relays, and reactor trip circuit breakers. This equipment is depicted in simplified schematic form in Figure 1. When a suf ficient number of breakers are tripped,,power to the control element assemblies (CEAs) is interrupted causing them to fall:into the reactor by gravity and thereby terminating the nuclear reaction process.

The overall.functions of the protection system are to assure that fuel design limits are.not exceeded during a plant transient (anticipated operational occurr rence), and to sense the onset of accident conditions and function'in conjunc tion with engineered safety feature systems 'to limit the consequences of accidents. The RPS is designed to comply with applicable NRC regulations, including the General-Design Criteria in Appendix A to 10 CFR 50. Because it is.a safety-related system, the RPS must'be designed, constructed, installed, operated, maintained, and tested in accordance with the, Quality Assurance criteria in Appendix B to 10 CFR 50.

Of the General Design'Criteria,'the'most germane to the issue at hand is GDC 23 which requires that, for conditions such as loss of electric power, the RPS must fail to a safe state. Traditionally, this criterion has been applied by the NRC staff as requiring.the RPS design to be such that it intrinsically causes auto matic reactor trip upon loss of power to the RPS. 'Therefore, designs include undervoltage (UV) trip mechanisms as a part of the reactor trip breakers.

The UV mechanism is energized during normal plant operations and will trip the breaker either when power is lost or when power is interrupted by an automatic or a manual protection signal.

The UV mechanism, its pickup-voltage adjustment spring, its interaction with the main tripping mechanism, and the critical bear ings of the main tripping mechanism are depicted in Figure 2.

An additional feature 'of the RPS design at this plant is the actuation of a shunt trip coil *in.the reactortrip breaker. The shunt.trip for every reactor..

trip breaker is actuated by any automaticlor manual protective-signal.

The shunt trip uses shunt (separate) power to trip the breaker. For the 8-breaker configuration, four separate safety-related dc power sources are used to pro vide power to the shunt trips.

During a reactor trip, the UV mechanism is deenergized and simultaneously the shunt device is energized to trip each reactor trip byeaker. If either of these diverse and independent mechanisms operate, the individual breaker will perform its safety-related function.

San Onofre RTB SER II-1

INPUTS FROM NSSS ME UcEMENT 123458789-N 123456789-N 123456789-N 1 23456789-N BISTABLES C-ANNELSA HNELSB

LOGIC MATRICES A-LOGIC A-C LOGIC A-0 LOGIC B-C GIC B-D LOGIC AB4 C4 AD4 BC4 d U4 4

CO4 LOGIC AB2A AC2 2

SC2 E

8A02CD RE LAYS E

-TO 120 Vac TO 120 Vac TO 120 Vat.

TO 120 Vac VITAL INSTRUMENT VITAL INSTRUMENT VITAL INSTRUMENT VITAL INSTRUMENT BUS No.lA AB1 BUS No.oB.IC BUS No. 10 AC1 AC2 AC3 AC4 AD1 AD2 AD3 AD4 TRiPPATHS BC1

= BC2 BC3 BC4 T01 =8B02 803 804 CCD2 CD3 CD4 TRIP ColCD CIRCUIT BREAKER CONTROL RELAYS K1 K2 K(3 K4 (INITIATION RELAYS) 480 Vac-30 480 Vac-30 BUS No. 1 BUS No.2 MAIN CIRCUIT No.1 BREAKERS No.2 MOTOR GENERATORS No.1 No.2 240 Vac-3 SYNCHRONIZER

+125 Vdc BUS No. 2

+125 Vdc BUS No.3 BUS TIE MANUAL Kn723 K3 TRIP No..

2 UV U ST 7

UV ST UV ST

+125 Vdc BUS No.1

+125 Vde BUS No. 4 K1 MANUAL

(--

TRIP CIRCUIT TRIP No. 2

'1BREAKERS K4 K4 UIV

.4.

UV ST UV ST UV ST CEOM POWER NDIO UALIE CEM CONTROL INDIVIDUAL CEOM SUPPLIES POWER SUPPLIES SYSTEM POWER SUPPLIES CONTROL CEOM CEDM CEDM CEDM CM CEMO ELEMENT COILS COILS COILS COILS COILS COILS DRIVE MCHANISMS, Figure 1 Simplified Functional Diagram of the Reactor Protection System San Onofre RTB SER IH-2.

-~~~~;

s-.--

R-,~--

rI P SW4FTr 4Z~rA7V*

s 44 N

LATCHW A1P LAMMC P~oT PCP

~~ Co(

Figure 2 GE AK-2 UV Trip Device (energized)

San Onofre RTB SER II-3

The configuration of the multiple trip breakers is shown in Figure 1.

A con-.

siderable degree' of redundancy of circuit breakers is provided by thiscon figuration' to perform-the protective function.' There are'two parallel paths to provide power to each of the two groups of control rods. These 4 paths are' each provided with, 2' reactor trip breakers, for a total 'of 8 reactor trip breakers.:

For each half of the control rods, the minimum <number of breakers that must function to drop the rods is two; for all rods, f'ur. Conversely, as many as four breakers could fail' to 'trip and all control rods would be dropped..

II.B. Reactor-TripBreakers (RTBs)

II.B.1. Design 'Criteria, Basis, Safety Classification The reactor trip switchgear cabinet assembly, including the reactor trip breakers., is-s-pecified in San 'Onofre.Nuclear Generating Station.:(SONGS) 2 and 3 FSAR Table 3.2-1 as:

Quality Class 1 (safety-related)

Sei.smi~c Category I (design, basis earthquake)

Electrical' Class 1E The seismic qualification test plan does not require the breaker closing cir cuit to function during a DBE; however, it must not interfere in any way with the trip (opening) function of the breaker.

The reactor trip breaker design criteria is such that failure of either the undervoltage trip device or the shunt trip device does not eliminate the safety functionage trip eico'r theshnt brak ripdvc.osntelmnt h aey function of the-reactor trip breaker.' The trip circuit breakers will complete their protective action of interrupting power to the control rods using either the UV or shunt trip devices.

The RTB undervoltage device and the shunt trip device complement each other in that the undervoltage device trips the breaker upon loss of control voltage while the shunt trip device trips the breaker upon application of control voltage.

II B 2. Design Description II.

B.2.a.- Breaker Assembly, Trip Bar, and Latch The details of the breaker latching mechanism and the function of the tripping devices as summarized here are applicable to GE AK 2-25-2 electrically operated breakers which are closed by a solenoid coil.

The armature of the solenoid is linked to the breaker mechanism and the armature movement, operating through the mechanism, closes the breaker and elongates the closing spring which contains the potential energy to open the breaker.

The breaker is tripped open by the' displacement of a mechanical trip latch, which allows a toggle linkage, supporting the movable contacts in the closed position, via the closing spring, to collapse. This trip latch is rigidly.

fastened to a trip shaft which runs horizontally across the breaker. All of the features provided for tripping the breaker operate through striker arms, San Ohofre RTB SER II-4

which rotate-the mechanical trip latch by moving against trip paddles fastened on the trip shaft (see Figure 3).

The local manual (mechanical) breaker trip button, overload devices, shunt trip, and undervoltage (UV) tripping device' all operate on the trip shaft to trip (open) the breaker.

When the control voltage is low~or has been removed, as when the breaker has been, pulled out for inspection or maintenance, or has received a protection system trip signal, the breaker 'is tripped open by the UV device. When this occurs, the breaker is tripped by displacement of the trip latch off a latch roller. The trip force is ip a spring which causes a toggle-mechanism to col lapse, resulting in the opening of the breaker. The trip latch mechanism in the energized and deenergized positions are seen in Figures 4 and 5.

Il.B.2..b.

Shunt Coil Assembly The shunt trip device is shown in Figure 5. -In physical appearance it is sim ilar to the undervoltage tripping device. Upon receipt of a control voltage, the coil is energized whi'ch creates -a magnetic force which pulls on the armature.

This magnetic force displaces the armature' causing it to strike and displace the shunt trip paddle on the breaker trip shaft. Thus the breaker is tripped (opened) by receipt of control power.

II.B.2.c. Undervoltage (UV) Coil Assembly The undervoltage tripping device is shown in Figure 6. When energized the mag netic force created by the coil picks up and holds in place a spring-loaded armature.. The voltage at which the armature moves into place is called the pickup voltage.

Pickup voltage is varied by an adjusting screw.. By turning the adjusting screw, one end 'of the spring used to move the armature is changed.

This change results in differences in spring-elongation (stretch) and hence the

..potential energy to initiate the trip. The higher the pickup voltage, the higher the spring potential.energy available for initiating the trip., The upper. limit on pickup voltage is caused by inability to energize the UV' device (i.e., re-close the breaker).

The undervoltage tripping device is activated by removing the control voltage which releases the potential energy in the spring causing the armature to strike and displace the undervoltage trip paddle on the trip shaft.

II.C.

Reactor Trip Breaker Surveillance Tests, March 1 and 8, 1983 II.C.1. Description of Tests SCE has stated that the tests run on March 1 and 8, 1982 were performed to verify proper breaker operation after the issuance of IEB 83-01 (February 25, 1983). This IEB applied only to Westinghouse DB-50 Reactor Trip Breakers, but SCE decided to test the GE AK 2-25 breakers used in San Onofre 2 and 3.

On March 1, 1983, the surveillance.test was completed on Unit 3 breakers. One problem was identified, a failure of.breaker TCB4 to trip on deenergization.

of the UV'device.

Non-compliance Report (NCR) 3-243 was written on March 3, 1983, to document this problem. As is discussed in Section II.B..5.b of this report, the licensee concluded that this failure was an isolated instance, and therefore reporting of this failure to the NRC was not required. The staff concurs with the licensee s action.

San Onofre RTB SER II-5

MANUAL TRIP PADDLE UNDERVOLTAGE TRIP PADDLE SHUNT TRIP PADDLE TRIP SHAFT TRIP SHAFT BEARING TRIP LATCH

-TRIP, LATCH ROLLER TRIP SHAFT SKETCH NOTE:

THIS SKETCH DOES NOT PORTRAY THE ACTUAL SHAFT OR THE LOCATION OF THE VARIOUS TRIPPING PADDLES WITH RESPECT TO EACH OTHER OR TO THE SUPPORTING BEARINGS.

IT IS A PICTORIAL REPRESENTATION OF THE TRIP SHAFT AND ITS FUNCTIONS.

Figure 3 Trip Shaft Sketch San Onofre RTB SER II-6 C

7

ROLLER PIVOT TRIP SHAFT TRIPTLATCH TRIP LATCH

-ROLLER THE TRIP LATCH AND TRIP LATCH ROLLER ARE IN THIS POSITION WHEN THE CLOSING SOLENOID IS NOT ENERGIZED AND THE TRIP LATCH IS IN THE RESET POSITION. THERE IS A GAP BETWEEN THE ROLLER AND THE LATCH.

WHEN THE CLOSING SOLENOID IS ENERGIZED, THE TRIP LATCH ROLLER IN ITS HOLDER TURNS COUNTER CLQCKWISE, CLOSES THE GAP, AND IS RESTRAINED BY THE TRIP LATCH.,

THE TRIP LATCH AND ROLLER REMAIN IN THIS POSITION UNTIL THE BREAKER IS TRIPPED.

It WHEN ANY OF THE TRIPS ARE ACTIVATED, THE TR P SHAFT ROTATES IN A COUNTER CLOCKWISE DIRECTION AND THE TRIP ROLLER ROLLS ALONG THE CONTACTING SURFACE OF THE TRIP LATCH UNTIL THE TRIP LATCH SWINGS OUT OF THE WAY, LEAVING THE ROLLER AND ITS HOLDER TO ROTATE FREELY ABOUT ITS PIVOT. THIS CAUSES THE OVER-CENTER TOGGLE ACTION IN THE MECHANISM TO COLLAPSE AND THE CONTACTS OPEN.

THE ROLLER RETURNS TO ITS ORIGINAL POSITION WHEN THE CONTACTS OPEN AND THE MECHANISM IS DE-ENERGIZED. THE TRIP LATCH WILL REMAIN IN THE POSITION INDICATED UNTIL THE CAUSE FOR TRIPPING'IS RECTIFIED AND THE TRIP LATCH IS RESET.

Figure 4 GE AK-2-25 Circuit Breaker San Onofre RTB SER*

II-7 1

3

$2 o

-9 NOT TRIPPED 0

A

one

-a

46 TRIPPED

1. Nut

6. Screws

11.

Armature Arm

2.

Frame

7.

Magnet

12.

Trip Paddle

3.

Spring

8.

Coil

13.

Mechamism Frame

.4.

Rivet 9,

Clamp

14.

Trip Shaft Clamp

5.

Weight

10.

Armature Figure 5 Shunt Trip Device San Onofre RTBe SER II-8 44 Rie Clm 4

rp hf lm

Figure 6 Undervoltage Tripping Device 2

T9h 36 TRIPPED

1. Mounting Screw

2. Frame to

3. Armature

4. Spring NOT TRIPPED

5. Sbading Ring S. Adjusting Screw

7. Locking Nut 8 Bushing

10.

Anet 3

11. Screws

12. Call

13. Itirwt

14. Adjusting Screw Sa Oo Locking Wire

16. Mounting Nut

17. Mechanism Trame 18.'Trip Paddle Clazmps

19. Trip Paddle 7

.12

20. Adjusting Scr&ew A'I San;Onofre RIB SER II-

On March 8, 1982, the surveil.lances run on Unit 2 reactor trip breakers identi-.

fied the failure of-TCBI, 4, and 6 to trip on deenergization of the UV devices.

NCR 2-163 was written at this time documenting that condition. In conformance with the plant Technical Specifications, these events were reported to the NRC on March 10, 1982, and Licensee Event Reports (LERs) 83-19 and 83-23 were i-ssued for Units '2 and' 3, respectively.

II.C.2.

NRC' Staff Action Resulting 'from the March 1and 8, 1983 Tests Three initial actions were taken by the NRC'upoh notification of these failures.

First, a Confirmatory Actioh.Letter was sent to Southern California Edison on March 11, 1983. This Tetter advised SCE of the NRC's understanding thatthey would take no action to close the reactor trip breakers except for testing with the rods deenergized, or add positive reactivity on Units 2 and 3 until the matter~was.resolved to thesatisfaction-of the NRC. Second,,also on March 11, 1983,'the NRC issued IEB 83-04 on thesubject of failure of the undervoltage trip function of GE AK-2 reactor trip breakers., This bulletin requested that all operating reactor facilities (except those-with Westinghouse DB-50 breakers which were covered by IEB 83-01) perform tests on undervoltage devices, review the maintenance program on the breakers,,notify all licensed operators of the failure-to-trip event at Salem and the -testing.failures at San Onofre, and finally, toreport to the NRC the 'various results of the program just described.

Third, March 12, 1983, a team of six NRC personnel and two NRC contract personnel joined the San Onofre Unit'2 and 3 Senior Resident Inspectors onsite to examine in detail the breaker failures at Units.2 and 3. The following is an outline of the agenda for the discussions this team held with the licensee:

a.

Discussion of theobjectives of the site visit.

'b. Facilities requirements for the visit (security, etc.)

c. Paper/hardware needed.

d. SCE briefing - description of reactorprotection systemand breakers.

e. SCE summary discussion of recent events.

f. Plant tours (photographs taken) and test witnessing.

g. Detaileddiscussions of the events with various plant personnel.

h.

SCE definition.' of issues (preliminary)..

i. SCE recovery program (preliminary).

The testing conducted by the licensee during the staff site visit are described below. The last members of the team to leave the site exited on March 17, 1983, except for the Seni6'r Resident Inspectors who continued follow-up actions.

II.C.3. Licensee s Investigative Program With the arrival onsite of an NRC team on March 12, 1983, the licensee (SCE) assembled a team of engineers, technicians, and managerial personnel.

That day, a draft procedure was developed which defined the steps necessary to determine the cause of the breaker failures.

Concurrent with this effort, the site docu ment control center began extensive file searches to recover all documents re-.

levant to these breakers. The NRC requested copies of applicable procedures, manuals, and drawings, plus a complete history of the breakers. This history was to.include the design basis and criteria that was used for purchase and in-stallation, procurement documents, acceptance testing documents, surveillance San Onofre RTB SER I-10

test history, a record of maintenance and modifications, and copies of responses to relevant pastIE Bulletins and Circulars.

In order to validate the special-procedure,.SCE performed detailed static and dynamic tests on reactor trip breaker TCB 2, a Unit 2 breaker which had per formed satisfactorily, with a General Electric technician and a General Electric design engineer in attendance. This provided base-line data that was used on TCB 1, a failed Unit 2 breaker.

With the results and experience gained on those two breakers, the licensee tested the next failed breaker, TCB 6. Failed breaker TCB 4 from Unit 2 was sent to SCE's Electrical Test Laboratory in Alhambra, California, for in-depth testing. The NRC staff requested that failed breaker TCB 4 from Unit 3 be sent to the Franklin Research Center in Philadelphia, an NRC contractor. The' FRC personnel will perform independent tests on the breaker.

In-depth testing was conducted between Marcb,26 and April 1,

'1983 by SCE at its Electrical Test Laboratory with the aid of a General Electric service repre sentative. The' breaker was tested on a bench and when operational tests were made, the breaker was secured to the bench by the same breaker flanges that support the breaker.when installed in the cabinet. High speed photography was used to assess breaker trip performance. A magnetic oscillograph was also used to record a number of.performance parameters during dyhamic tests on the breaker.

The breaker was visually inspected and tested in the as-received condition and baseline measurements were obtained for all parameters, including undervoltage coil pickup and dropout voltage, trip shaft torque, trip response time, under voltage coil armature air gap, and all electrical component resistances.

Adjustments were varied and tests repeated to determine optimum settings and limitations. *The investigation also included inspecting, cleaning, degreasing, and regreasing the trip shaft and latch roller bearings, and cleaning and adjust ing the undervoltage device.

UV device pickup voltage was also investigated.

The results of this testing is described in the next section of thi.s report.

San Onofre RTB SER II11 4

'77

III.

INVESTIGATIVE PROGRAMS As. a result of the March 1and 8, 1983 surveillance failures of the GE Type AK 2-25 RTBs at San Onofre Units 2 'and 3, SCE conducted'an investigation, to determine the cause and ramifications associated with the surveillance failures.

III A. RTB.Investigative Tests Following the reactor trip.breaker surveillance tests of March 1 and 8, 1983 in response to IE Bulletin 83-01, SCE initiated furthertesting of, San Onofre Units 2 and 3' reactor trip circuit breakers. This'furthe'r testing was performed in two parts:

(a) initial investigative tests conducted from March,12 to 17, 1983 at San Onofre; (b) in-depth testing conducted from March 26 to April 1, 1983 at-the SCE Electrical'Test'Laboratory in Alhambra.

III.A.1. Initial Investigative Tests. (March, 12 to 17., 1983)

a. Following an introductory meeting with representatives of NRC and Franklin Research Center, SCE initiated an investigative plan for determining pro blems with RTBs., A procedure was developed and made.available for review and comments by NRC ahd'Franklin Research Center. Comments were provided by the above organizations.for SCE consideration and were factored into the final investigative procedure.

b. A General Electric Company service technician who had previously provided vendor assistance on RTBs was called in. In addition the General Electric Company provided a design engineer to assist in the investigation of the RTBs.

c. Work orders were generated to implement the final investigative procedure.

Reactor trip breaker TCB2 ('a functioning breaker) was tested to verify the adequacy of the investigative procedure.

Reactor trip breakers TCB1 and TCB6 (two of the three malfunctioning Unit 2 breakers) were then tested with the same procedure. Each of these breakers was visually inspected and tested for undervoltage trip while still installed in the switchgear.

cubicle', then removed and tested for trip shaft torque,. undervoltage trip response time, and undervoltage device pickup voltage. The RTB lubricant was revitalized'.as.necessary and changes were made in the UV pickup voltage levels.

d.

SCE stated that the observations of the initial investigative tests are as follows:

(1)- Two breakers were missing the locking wire for undervoltage coil pick up voltage adjustment'. One of these (malfunctioning breaker TCB1) had a lower than nominal pickup voltage (101 Vdc versus 106 Vdc) and the other (functioning breaker TCB2) a higher than nominal pickup voltage (107 Vdc-versus 106 Vdc).

San Onofre RTB SER Ill-1

(2) The as-found trip shaft torques all, exceeded the 1.50 pound-inches specified by IE Bulletin 79-09. The as-found torque was slightly higher inTCB2 (functioning.UV device trip) than TCB1 (malfunctioning UV device trip) but, as noted above, TCB2 also exhibited erratic behavior (slow trip) when its pickup voltage was lowered to 100 Vdc.

(3) Both malfunctioning b.reakers tripped satisfactorily on repeated'.

undervoltage trip tests with as-found or lower undervoltage pick up adjustments When the bearing lubricant was revitalized to reduce trip shaft torques to less.than the 1.5 pound-inches'specified by IE Bulletin 79-09.

(4) The shunt trip device successfully tripped the breaker.

(5) The GE design engineer. stated that he could find ho evidence of improper handling or mechanical damage to the reactor trip breakers examined. Further, all mechanical adjustments,were satisfactory, with the exception of under-voltage device pickup voltage and a minor increase needed in the TCB6 'overtravel adjustment (made after successful testing of the breaker).

e. SCE's preliminary conclusions based on the initial investigative tests were as follows:

(1) The major contributing factor to improper reactor trip breaker opera tion on undervoltage was due to insufficient or degraded lubricant in the trip shaft and latch roller bearings.

.(2)

The secondary contributing factor was the urdervoltage device armature pickup voltage being set below the recommended 106 Vdc.

III.A.2. In-Depth Tests: (March 26 to April 1, 1983)

Based on the results of the initial investigative tests discussed above, a more detailed inspection was performed on one of the reactor trip breakers to obtain as much quantitative information as possible with regard to the dynamic opera tion of this breaker under various conditions.

The breaker subjected to.this in-depth test and inspection was reactor trip breaker TCB4 from Unit 2, the third of the three breakers which malfunctioned during surveillance testing in early March 1983. The other two malfunctioning breakers (TCB1 and TCB6) had been previously tested and readjusted as discussed above.

The work was performed by' SCE at its Electric Test Laboratory with the aid of a General Electric service representative. The breaker was tested on a bench and when operational tests were made, the breaker was secured to the bench by the same breaker flanges that support the breaker when installed in the cabinet.

High speed photography was used to assess breaker trip performance. A magnetic oscillograph was also used to record the following parameters as required for dynamic tests on the breaker:

Breaker main contacts (3)

Shunt trip coil current San Onofre RTB SER 111-2 7:

7zT

Shunt trip coil voltage Closing coil current Undervoltage trip device current Undervoltage trip device voltage Auxiliary "b" switch contact The breaker was visually inspected and tested in the as-received condition and baseline measurements.were obtained for all parameters, including' undervoltage coil pickup and dropout voltage, trip shaft torque, trip response time, under voltage coil armature air gap, and all electrical component resistances.

Adjustments. were varied andtests repeated to determine optimum settings and limitations. The investigation also included inspecting, cleaning, and re vitalizing the grease in the trip shaft and latch roller bearings, and cleaning and adjusting the under-voltage device. UV device pickup voltage was also investigated.

The results of the in-depth investigative tests were as follows:

(1) As-received trip shaft torque'was greater than the 1.5 inch-pound limit and as-received undervoltage device pickup voltage lower than the 106 Vdc limit specified by IE Bulletin 79-09.. Cleaning and revitalizingthe grease in the trip shaft and rol-ler-bearings reduced the.trip shaft torques to less than 1.5 inch-pound and successful UV operation was obtained. This confirms the preliminary conclusions of the initial investigative tests discussed in Section III.A.1 of this report.

.(2)

Considerable variation of the UV device pickup voltage setting will result from variations in the UV device coil -temperature during pickup voltage adjustment. Thirty to 60.minutes is required for the UV device coil to reach a stable thermal state.

(3) The optimum adjustment for undervoltage device armature pickup voltage is 106 +/-2 Vdc at a "cold" (de-energized) UV device coil surface temperature of 700 to 85PF.

(4) The diode installed across the UV device coil for surge protection of the RPS relays cause some delay in the breaker response time (nominal 30 millisecond difference) although it remains within allowable values.

This diode is installed in the reactor trip breaker cubicle wiring and is not present on a removed breaker. Therefore, a diode is required during reactor trip breaker bench testing.

(5) Excessive clearance between the UV-device armature magnet and restraining rivet reduced the effective throw of the armature by permitting it to move up-against the rivet rather than rotating. The as-found clearance of 0.018 inch exceeded the 0.001 to 0.010 range recently recommended by the manufacturer; a somewhat narrower range (0.003 to 0.006 inch) proposed by SCE will provide improved performance and is consistent with the GE factory range 9f 0.001 to 0.010 inch.

(6) As-received condition of the trip latch roller bearing (rough operation, and excessive clearance) may have resulted in variation of trip shaft San Onofre RTB SER 111-3

torque with roller position, but did not affect trip reliability when pickup voltage and trip shaft torque were-within the desired range.

(7), There is ample design margin in-the shunt trip device. Operation of the breaker with the shunt trip was satisfactory down to approximately 30.Vdc; the voltage available is nominally in excess.of 130 Vdc.

(8) The undervoltdge response time of the breaker is faster and more con sistent.for a well lubricated breaker than one-with degraded lubricant.

III.B.

Evaluation of Licensee Administrative Controls SCE has committed to comply with Regulatory Guides 1.30, 1.33, 1.64, and 1.88 as part of the Quality Assurance program for operations of San Onofre 2 and 3.

III.B.1. Control.of Vendor Information, Technical Manuals NRC guidance in this area is found in.Regulatory Guides 1.30 and 133.. Sec tions 2.2.5c and 3.2 of Regulatory Guide 130 require the manufacturer's in structions to be available at the site and that this fact be verified. Sec tion 5.3.5.4 of Regulatory Guide 1.33 requires that maintenance procedures reference applicable sections of related vendor manuals and equipment operat ing and maintenance.instructions'.

It states that in some cases these documents may constitute adequate procedures in themselves. It requires that such pro cedures receive the same level of review and approval as operating procedures.

During an NRC inspection conducted between October 26 'and December 4, 1981, the inspector identified a licensee weakness in this area. This finding was docu mented in NRC Inspection Report No. 50-361/81-28. The finding, at that time, was that.the licensee had not established a system to assure that vendor infor mation was coordinated, controlled, and evaluated for potential effects on maintenance or surveillance procedures. Based upon an NRC inspection conducted during March 12-25, 1983 (Inspection Report No. 50-361/83-13), it appeared that the licensee had not at that time established such a system. For example, difficulties were encountered in determining if the reactor trip breaker technical manual onsite was.the correct version and a technical manual for the undervoltage devices was never located onsite.

In Section IV.B.1.a of their April 15, 1983 Reactor Trip Breakers Report, the licensee acknowledges.the potential for problems associated with the control of vendor information and technical manuals.. The licensee is implementing a new, comprehensiveconfiguration control program to centralize responsibility for.

control and management of all vendor supplied information received thus far and in the future. This program is scheduled to be implemented by the-completion of the plant startup phase. Until this new program is fully implemented the licensee will utilize administrative-procedure controls to reduce the possibil ity of incorrect vendor data being used for safety-related operations at the site.

The staff has examined the licensee's statement regarding programmatic controls of vendor supplied information in the April 15, 1983 report. The staff con cludes that this program contains sufficient controls over vendor supplied information if effectively implemented. The staff has determined that the San Onofre RTB SER 1.11-4

licensee has provided specific guidance to.the licensee's maintenance.organi zation regarding the full use of vendor supplied documentation on site for the planning and accomplishment of maintenance activities. The staff has also determined that an improved procedure-for administrative controls for the utilization of vendor supplied information is in draft form and is in the near, term approval process. The licensee has held meetings with responsible super visors and the procedure writers to ensure the correct handling of-vendor supplied information. 'A letter dated April 8, 1983.has been~ issued by the licensee defining the responsibilities regarding the retrieval and use of vendor supplied information relating to maintenance procedures.

Based on the above and the fact that the control of vendor/NRC data was not the major cause of the incorrect maintenance action on the RTBs, there is' reasonable assurance that the control of vendor information and technical manuals can be andIwill be accomplished in an acceptable manner. Further, we conclude that the licensee's actions to date provide sufficient ahd 'acceptable bases for plant restart. The.NRC staff will continue to review the ongoing implementation of the new control system and document its findings in its Regional Office inspection reports.

III.B.2. Control of Hardware Configuration Section 5.2.7 of Regulatory Guide 1.33 requires that maintenance and modifica tions be performed 'in a manner to ensure quality at least equivalent to that specified in the original design, as verified.by appropriate inspection and performance testing. It requires that this.work be' preplanned and performed in accordance with apppropriate instructions, procedures, and drawings. It requires the establishment of means for assuring the quality of these activities and documentation thereof. Section 5.2.7.2 of Regulatory Guide 1.33 requires that design activities associated with modifications be accomplished in accord ance with Regulatory Guide 1.64.

Section 8 ofRegulatory Guide 1.64 requires documented procedures for design changes and that these changes be justified and.subjected to design.control measures commensurate with those applied to the original design. It requires the organization which reviews a design change to be competent in the specific design area and to understand the requirements and intent of the original design.

It also requires that each design change be transmitted to all affected persons and orgahizations.

During.the March 12-25, 1983 NRC inspection, it was found that the reactor trip switchgear cabinet'assembly, including the, reactor'trip breakers,.are.specified as Quality Classi1, Seismic Category 1, Electrical Class. 1E in"the FSAR, Table 3.2-1,. and were supplied by Combustion Engineering as part of the NSSS contract.. The licensee s equipment list (Drawing No. 90009), Revision 55, page 61 of 70, classifies the reactor trip breakers as Quality Class 1, Seismic Class 1. Therefore, classification of the RTBs on the licensee's equipment list agrees with the classification assigned the quality class and seismic category by the FASR, and is acceptable.

The licensee has determined that General Electric type AK 2-25 circuit breakers are not.used in any safety-related application, other than in the reactor.trip switchgear, at San Onofre Units 2.and 3. The licensee further stated that San Onofre RTB SER 1.II-5

Westinghouse type DB-50 breakers are not used in safety-related applications at San Onofre Units 2.

and 3.

During inspections of the reactor trip breakers and Undervoltage coils, the inspectors observed the following:

a. Trip Circuit Breakers (TCBs) 5, 6, 7, and,8 in Unit 3 switchgear were type AK 2-25, whereas all other Unit 2 and 3 TCBs were type AK 2-25-2.

b. Two different part numbers for the undervoltage trip devices.were observed.

Part No. 269C282-G2 was stamped on the under-voltage (0V) device name plate for Unit 2 TCBs 6:,

7, and 9, while Part No. 269C282-G5 was stamped on all other UV devices.

All UV devices were labeled as 125 Vdc.

Further. examination identified that.the parts list in the type AK circuit breaker technical manual identifies Part.No. 269C282-G2 as an ac Instantaneous Undervoltage Device and Part No. 269C282-G5 as a dc Instantaneous Undervoltage Device. All UV devices are supplied with 125 Vdc Class lE power in the San Onofre Unit 2 and 3 applications.

Based upon these observations, the inspectors requested that the licensee identify the properibreaker and UV device configurations necessary to be utilized in the reactor trip breaker switchgear. On March 25, 1983, the Inspector received information from the licensee indicating that:

TheAK' 2-25 and AK-2-25-2 breakers are functionally identical and inter changeable.

The AK 2-25-2 was supplied as original equipment and the AK 2-25 was supplied as spares.

All UV relays are of the type.utilized in dc applications.

As a result of the difficulties encountered in obtaining information on breakers and UV device design configurations, the inspector concluded that the licensee had not at that time established or effectively implemented a program to effect controls over the specification and maintenance of thedesign configuration of safety-related equipment extending to the component level.

In their April 15, 1983 Reactor Trip Breakers Report, the licensee identfied specific problems with the procurement of 5 spare RTBs.

The problems involved among other things, the issue of sub-component configuration control (specif ically the UV assemblies)

The licensee QA organization conducted a compre hensive audit of both the NSSS and the breaker vendor in April 1983 to identify actions necessary to correct the identified problems. The licensee has requested the NSSS to initiate corrective action to correct the licenseels audit findings.

The staff has confirmed that none of the breakers involved in above discussion of spare breakers are installed in Unit 2.

The 5 breakers in question are currently installed in Unit 3 or in storage.

All 5 breakers have had "NCR hold" tags placed on them to prevent their use in.Unit 2 or 3 until all discrepancies have been resolved. The staff has reviewed all nine of the RTBs installed in San Onofre RTB SER III-6

Unit 2. This review has determined that all of the RTBs installed in Unit 2 are. -the correct breakers as specified in the original procurement specifications identified by Combustion Engineering.

Based on the-above, we conclude that reasonable assurance exists as to the quality and configuration of the RTBs.installed in Unit 2. The NRC staff will follow the closeout action regatding the procurement and-configuration controls of the 5 breakers discussed' above.

The findings of this effort will be documented in an NRC Regional.Office inspection report. We conclude that the licensee's actions to date provide sufficient and acceptable bases for plant restart.

III.B.3. Maintenance Program III.B. 3.a.

Procedures and QA/QC Requirements In addition to the requirements noted in Section III.B.2 above, Regulatory Guide 1.33 (in section 5.2.7.1) requires the development of a program to main t.ain safety-related items at the quality required for them to perform their intended functions. It requires early development of procedures required for maintaining equipment expected to require recurring main'tenance, with altera-.

tions to improve performance as operating experience is gained,. It requires a preventive maintenance program which prescribes the frequency and type of main tenance to be performed.

It requires evaluation of malfunctions to determine proper corrective action.. Section 5.3.5 of Regulatory Guide 1.33 specifies what must. be incTuded in maintenance procedures,'.while Section 9 of Appendix A of Regulatory Guide 1.33 'provides additional guidance concerning maintenance procedures.

Section A6 of Appendix A of Regulatory Guide 1.88 specifies that records re flecting plant design modifications are lifetime records and that records of principal maintenance activities (including inspection, repair, substitution or replacement items) be maintained for 5 years.

The NRC inspection of March 12-25, 1983 determined that the San Onofre 2 and 3 maintenance program and procedures generally complies with the requirements of ANSI N18.7-1976,* "Administrative Controls' and Quality Assurance.for the Opera tional Phase of Nuclear Power Plants," and 10 CFR 50, Appendix B, "Quality Assurance Criteria for Nuclear Power Plants and Fuel Reprocessing Plants."

One programmatic concern was identified, however, relating to the lack of a specific requirement to perform a prompt evaluation of any malfunctions iden tified during maintenance. 'The original program did require an evaluation.

However, the time period to perform it was not specified.

In the licensee's April 1983 Reactor Trip Breakers report (Section IV.B.6),

the licensee describes changes that have been made and will be made to proce dures 'for evaluation of nonconforming conditions identified during maintenance.

The staff has reviewed the procedures which have been completed. The staff.

has determined that the appropriate reporting requirements have now been

ANSI N18.7-1976 is endorsed by Regulatory Guide 1.33.

San Onofre RTB SER 111-7

included in these procedures. The changes described resolve the concern identified in this area and provide r easonable assurance that maintenance related nonconforming conditions will, be effectively handled.

This area will be reviewed by' the staff after full implementation and the findings documented in a Regional Office inspection report. We conclude that the licensee's actions to date provide sufficient and acceptable bases for plant restart.

III.B.3.b. Maintenance History and Records In June, 1980,,the licensee issued Maintenance Procedures MPES 008, "Under voltage Tripping Device of GE AK-2-25 Circuit Breakers in the Reactor Trip Switchgear," in response to IE Bulletin 79-09 requirements. The procedure was revised in March, 1981, to correct difficulties encountered while using the procedure on the Unit 2 reactor trip breakers in January,.1981.

Procedure MPES 008 was used:for maintenance on~all Unit-2 RTBs during the.

April-May, 1981 time period and again used on Unit 2 TCB2 during June, 1981...

This maintenance was the last formal preventive maintenance performed, using MPES 008, on the Unit 2.RTBs until M.arch, 1983.

Surveillance of the'Unit 2 RTBs on March 25, 1982 identified that TCBs 4, 6, 7, and 8 failed to function as required.. A maintenance work order was written, and a vendor representative called in, to correct the situation.' The instruc tions on the work order were to adjust the RTBs."per direction of the 'vendor representative."

No documentation of the specific maintenance tasks authorized or performed was prepared or completed, other, than to transfer a RTB from Unit 3 to replace TCB8. The vendor technical manual and procedure MPES 008 were avail

<able but:were apparently not utilized in the performance of this maintenance.

During the period of July 8-10, 1982 it was identified that Unit 2 TCBs 4, 6, and 7 would not shut on the first attempt. A work order written to correct

.this situation di-d not specify or reference' the required repair method or the retest requirements following repair. Maintenance personnel documented that adjustments were made on relays to trip and reset at factory required voltages, however, details as to which relays were adjusted and'what voltage acceptance criteria were used was not.provided. On July 12, 1982, Unit 2 TCBs 4 and 7 again failed to'trip on undervoltage.

During the period of July 14-21, 1982 a work order was implemented to check the undervoltage coil adjustments for all Unit 2 RTBs.

Informal data sheets, and discussions with the.foreman,indicated that an extensive inspection of the breakers was performed. However, the work order neitherspecified nor docu mented a repair method. The tasks performed were documented from memory by the foreman in March, 1983, eight-months later. This work was performed with the assistance of a vendor representative. A nonconformance report written pur suant to this maintenance activity identified that the undervoltage coil pickup voltages did not conform to the criteria of procedure MPES 008.

The "as left" undervoltage coil pickup voltages were documented to range from 90-119 Vdc and were accepted by the licensee based on verbal authorization of the.vendor re presentative. This range of pickup values deviated substantially from the recommended criteria.of 106 Vdc provided in IE Bulletin 79-09 and MPES 008.

San Onofre RTB SER III-8

The staff has reviewed the revision to MPES 008. Additionally it has reviewed the maintenance action taken on all RTBs which established baseline data and restored all RTBs to a fully operable status -in accordance.with the require ments of the vendor manual and IEB 79-09. Based on the above, there is reasonable assurance that the RTBs are fully operable and that the required records of maintenance performed now exist for future use. We conclude that the licensee's actions to date provide.sufficient and acceptable bases for plant restart.

III.B.3.c. Vendor Maintenance Activities The licensee's system for controlling vendor activities is contained in Test Instruction (TI) 16 (Vendor Monitoring), Revision 3, dated August 13, 1982.

The purpose.of the procedure is to establish a uniform method for procuring and monitoring vendor services requested by startup personnel.and applies to vendor personnel who perform or direct the performance of work during the startup and test programs..

Examination of reactor trip'breaker (RTB) maintenance history indicates that:

1.

On about March 25, 1982, in response to the failure of four Unit 2 RTBs to trip when the UV coil was deenergized, the licensee's maintenance organiza tion and a vendor representative adjusted all UV coils.

2. During the period of July 14-21, 1982, a vendor representative performed work on all Unit 2 RTBs.

The overhaul purportedly included lubrication.

3. During the period of August 10-21, 1982, the licensee's maintenance organization replaced the UV trip devices on four Unit 3 RTBs and, with a vendor representative, adjusted all Unit 3 UV coil settings.

The NRC inspector examined the maintenance documentation (Work Orders.and Nonconformance Reports) utilized to effect the above activities and observed that the activities of the vendor representative(s) were not controled as required by TI-16, in that approved procedures did not appear to be utilized which specified the work performed by the vendor representative. Work performed by the maintenance organization was, however, documented and specified by procedures with quality-control inspections required at various hold points.

The inspectors discussed with a vendor representative who had performed work on the RTBs the extent.of, adjustments made and the criteria used by the vendor representative. These discussions indicated that the vendor representative was not specifically familiar with the technical manual criteria of'certain adjust ments and that some-adjustments may have been made which appear to be reserved for factory setting.

The, licensee's.procedure TI-16 does not appear to contain any criteria or provision for the assessment of vendor representative capability, training, or qualification.

In summary, it appears that vendor representative(s) performed work on the breakers and that the licensee exercised inadequate control over vendor repre sentative activities in that:

San Onofre RTB SER 111-9

The capabilities, training, and qualification of vendor representatives was not assessed prior to the start of work.:

Detailed work plans or procedures were not prepared to control the work activities.

Documentation of work activities conducted was insufficient.

In their April 15, 1983 Reactor Trip Breakers Report the licensee acknowledges the potential for inadequate control of vendor supplied services. In response to this situation the licensee reviewed all (about 20,000) work orders to identify those cases where a vendor performed or directed work without using'a licensee procedure.

Forty-one :such cases were identified and are beingreviewed to determine the appropriate correctiveactions.

The staff has determined.that the licenseehas commenced, and is in the process of conducting increased training in verbatim compliance and in supervision of vendor work to ensure that programmatic controls of vendor services are effective. The licensee expects this training to be completed by May 31, 1983.

This action when fully' implemented will provide reasonable assurance that, future activities performed by or under the direction of a vendor representa tive.will be accomplished with due regard to quality and safety.

The licensee's actions in this area 'will be reviewed by the NRC staff and the

'findings documented in a Regional Office inspection report.

III.B.4.

Surveillance Program During the March 12-25 inspection, NRC inspectors examined the surveillance pro gram associated with the reactor trip breakers for Units 2 and 3 and discussed the program with plant management, plant engineers, and.technicians. The follow ing documents were reviewed:

Technical Specification 3/4.3.1, Reactor Protective Instrumentation Procedure.S023-11-11.161, Revision 2, dated February 10, 1983; Reactor Breakers Undervoltage and Shunt Trip Device Circuit Surveillance Test Numerous Letters Listed in a March 14, 1983 memorandum to A. E. Chaffee (NRC) from H. B. Ray (SCE) on the Subject of UV Device Surveillance Testing.

Numerous Surveillance Test results listed in a March 13, 1983 memorandum to A. E. Chaffee (NRC) from H. B. Ray (SCE) on the subject of Surveillance History.

III.B.4.a. Procedures and QA/QC Requirements Section 5.2.8 of Regulatory Guide 1.33 requires a surveillance testing and in spection program to assure that safety-related items will continue to operate San Onofre RTB SER III-10

satisfactorily. Section 5.2.19 requires, as part of the test program, surveil lance tests to provideassurance that failures or substandard performance do not remain undetected and that the required reliability of safety-related, systems is maintained.

Section A6 of Appenoix A of Regulatory Guide 1.88 specifies that records of periodic checks, inspections, and calibrations performed to verify surveillance requirements are being met be maintained for 5 years.

There are no specific requirements for either the QA or QC organization's to be come.involved in the surveillance testing of the reactor.trip breakers. Either organization may become involved if SCE management desires involvement. The QA organization, for example, may review the reactor'trip breaker surveillance records as part of its Plant Operations surveillance activity, which isan on going, year-round, sampling type' of.a program.

The NRC staff has since examined the procedure S023-II-11.161.(Surveillance Requirement - Reactor Breakers Undervoltage and Shunt Trip Device Circuit Test) revised through Temporary change Notice 2 dated April 12, 1983,,and concludes that the licensee has reviewed the procedure again for adequacy and satisfac torily resolved the comments.identified above and contained in NRC.Inspection Report No. 50-361/83-13.

The licensee has developed an enhanced surveillance interval criteria for the conduct of reactor trip breaker surveillance and described same in their April 1983 Reactor Trip Breaker report. The licensee described a program which provides for increasing the surveillance interval based on repeated acceptable.

reactor trip breaker time response and operation in addition to' shortening the interval whenever unacceptable surveillance data is obtained. Discussions with licensee personnel indicate that three tests of each breaker will be performed at each surveillance and the time response.of the breaker will be conducted during each surveillance test.

The licensee further indicates' in the April 15, 1983 report that the reactor trip breaker maintenance frequency will be once every 12 months unless sur veillance testing data indicates that maintenance should be performed more often than every 12 months.. The staff has reviewed the above and concludes that the new surveillance interval criteria are acceptable.

The staff has reviewed the proposed surveillance program and compared it with the recommendations of Regulatory Guide 1.33 and ANSI N18.7-72. The NRC staff concludes that the.Quality Assurance/control audit and inspection activities of. surveillance are in conformance with regulatory requirements. The staff concludes that surveillance records were adequately stored, filed and retrievable.

Based on the above, the staff finds that reasonable assurance exists that the licensee has taken or will take sufficient action to provide for reliable opera tion of the reactor trip breakers. The NRC staff will follow the closeout action regarding the implementation of the enhanced surveillance and response time testing program to be applied to the reactor trip breakers.. The findings of.this effort will be documented in a Regional Office inspection report.

San Onofre RTB SER III-11 777 7,.-..""'--

III.B.4.b. Surveillance History and Records The Unit 2 operating license was issued on February 16, 1982. The surveillance testing using Procedure S023-II-11.161.commenced on February 25, 1982, using Revision'2 to'the procedure. The following list represents the history of surveillance on Units 2 and 3 from that date. The list was supplied in a large-part-via a memorandum to A. E. ChaffeeNRC) from.!. B. Ray (SCE)., dated March.13, 1983.

February 25, 1982 -

Unit 2 surveillance completed.satisfactorily.

March 25, 1982 -. Four nonconforming Unit 2 circuit breakers were identified due to UV device failure. Disposition is discussed in review of maint enance history based on NCR S023-P-152. Unit 2 was in Mode'5 at this time.

The NCR/was not evaluated for reportability.

April 4, 1982 - Unit 2 surveillance was conducted satisfactorily following the disposition of NCR S023-P-152. 'All work on the breakers in March was done per the direction of the GE vendor representative.

May 4,: 1982 -

Unit'2,surveillance completed.,satisfactorily.

June 4, 1982 - Unit 2 surveillance completed satisfactorily. NOTE:

At this time, since no problems had been' identified by three months of independent UV and shunt testing, the licensee decided (based partly on vendor (CE) recommendati.ons) to increase the surveillance interval to 18 months.

July 6, 1982 - Unit 2 surveillance was performed on TCB5, only, due to replacement pursuant to NCR S023-P-431. the replacement was done because of an open UV coil observed during maintenance. Unit 2 was in Mode 5 at this time.

'July 12, 1982 - Unit 2.

surveillance of TCB1, 2,3, 5,'8, and 9 following Work Order 10245.' The reason for this work is discussed in maintenance history. NCR S023-P-511 was written on TCB4 and 7 reflecting failure of UV trip function. The GE vendor representative was called in to resolve the trip function failure.

July 13, 1982 - Unit 2 surveillance of TCB4 and 7 following Work Order 10305 which resulted from NCR 5023-P-511.

'July 21 1982 - Unit 2 surveillance of TCB1 through.8 again pursuant to Work Order 10245.

The UV device was tested three times.

August 23, 1982 - Unit 3 surveillance identified no problems.

October 30,.1982 Unit 3 surveillance identified no problems.

March 1,.1983 - Unit 3 surveillance identified a failure of the TCB4 UV trip device. This surveillance was accomplished in response tc the West inghouse OB-50 breaker problems identified by IE Bulletin 83-01.

San Onofre RTB SER

-11112

March 8, 1983 - Unit 2 surveillance identified failure of the UV trip device on TCBs 1, 4, and 6. NCR 2-163 and LERs 83-19 and 83-23 for Units 2 and 3, respectively, were issued:

III.B.4.c Adequency of Technical Specification Requirements As part of our evaluation of the RTB issue, we reviewed the San Onofre 2 and 3 Technical Specifications to determine if any changes should be made as a result of recent operating experience.with RTBs. We foundthat the present' San Onofre 2 and 3 Technical Specifications on RPS surveillance have 3 perti nent characteristics.

First,.by definition, the reactor trip system response time starts when a parameter exceeds its trip setpoint as measured by instru ment channels, and~continues, through the operation of the coincident logic matrices, and the operation of the reactor trip breakers to interrupt power to the control rods. This. is a comprehensive measurement with values ranging from 400-900 milliseconds.depending on the parameter.

In a fneeting on, 4/12/83, the licensee stated that the end-point of.the measurement is determined by a current transformer downstream of each reactor trip breaker. Second, the Technical Specification Table 3.3-2 "Reactor Protection Instrumentation Response Times" includes the reactor trip breakers but indicates that response time is not applicable. Third, the Table 4.3-1 "Reactor Protection Instrumentation Surveillance Requirements" includes the reactor trip breakers andrequires a channel functional test on a monthly basis during plant operations. By foot note, an independent functional test of the UV trip and shunt trip is required every 18 months and after maintenance/adjustments.. No calibration checks are required.

As a result of our review, we conclude that revision of the Technical Specifica tions is appropriate.

The areas we found that need improvement are (1) response time measurements are needed for. the trip breaker separate from the compre hensive overall RPS response time measurements, (2) periodic surveillance testing is needed of adjustments and of key RTB characteristics that could indicate degradation,.and (3) more 'frequent testing is needed to independently confirm the functional capability of the UV trip device and the shunt trip device.

During the meeting on 4/12/83, as documented by the SCE submittal dated 4/15/83, the licensee stated that they intended to periodically perform response time measurements of the reactor trip breakers separate from the protective instru mentation and coincident logic matrices. SCE stated that they believed this measurement to be a leading indicator of degradation and UV device failure.

When, a breaker is properly 'lubricated and adjusted, tests have shown response times.of 60-63 milliseconds-for the trip breaker actuated by the UV trip with a voltage suppression diode installed. The licensee indicated that a value such as 70 milliseconds would be used to trigger an engineering evaluation.

We concur with reasonableness of these values and agree with the licensee that 100 milliseconds is an appropriate limit for the purposes of the Technical Specifications.

During the meeting on 4/12/83 the licensee stated that they intended to period ically.check, and readjust as appropriate, the torque required to trip the breaker and the pickup.voltage of the UV trip device.

When the trip shaft bearings and the trip latch roller bearing had been revitalized, torque valves of 1.0 San Onofre RTB SER 111-13 77n

to 1.1 in-lbs. were typical.

The manufacturer's limit of. 1.5 in-lbs. is an appropriate,limit for the purposes of Technical Specifications. The recommended pickup voltage for this applicati'o n is 106 VDC with tolerances of +4 VDC and 2 VDC, which yields allowable valves of 104-11OVDC. 'The measurement of torque requirements and-pickup voltage are considered necessary calibration checks.

The licensee has proposed to conduct the respohse time' measurements, torque measurements, and pickup voltage meas'urements on a periodic basis that will be initially monthly for some items and every 4 months for other. If no "failures" are experienced,'this would go to',every 6 months and then to an annual basis.

The licensee has not yet fully explained..the criteria.for relaxing the frequency of these activities. We believe it is prudent.to have the licensee-discuss the results of his program.with us before going to less than a semi-annual frequency.

Further, we believe that'the UV trip and the shunt trip should be independently fbnctionally tested on at least a semi-annual basis.'

By letter dated April. 22, 1983, the licensee committed'to conduct the additional surveillance testing described'above and has committed to propose appropriate Technical Specifications changes to reflect these 'actions within 30 days of.

plant restart. Because the licensee has made commitments that cover each of the areas that our review found to be in need of' strengthening,. we conclude that this part of our evaluation is acceptably resolved..Since all initial tests and measurements will be performed prior to plant restart and periodic surveillance actions will.in fact.be taken,.we do not believe it necessary to

'delay restart of the plant for the completion of the' change's to the Technical Specifications. On this basis 'we conclude that the Technical Specifications are acceptable for plant restart.

III.B.5 Reporting of Failures The licensee's'Technical Specifications, Section 6, paragraphs 6.9.1.12 and 13 list the types. of events which shall be reported to the NRC Regional Office Administrator.

The licensee implements the reporting requirements in Section 40 of the San Onofre 2 and 3 QA topical report by requiring that nonconformance be reviewed for reportability. Nonconformances are identified by either the traditional Nonconformance Report or'a Limiting Condition for Operation Action Requirement (LCOAR) Report.

The LCOARs are generated by the plant operators when a Technical Specification action statement is entered during plant operations.

These are reviewed for reportability by the shift technical assistant staff andmay be the-source of a station incident report. 'The LCOARs are forwarded to the configuration con trol and compliance staff for the final determination of reportability. In mid-1982, the licensee observed that several LCOARs had not been reviewed for reportability. Subsequent reviews identified several reportable events, at which time LERs were issued. Corrective actions taken by the licensee have been effective in.assuring, that reportability reviews are accomplished, on the LCOARs and station incident reports, in a timely manner.

Nonconformance reports (NCRs) are used to identify, and document resolution of, nonconforming conditions. The QA topical report was revised, in November 1982, San Onofre RTB SER I1l-14

to require reportability reviews of NCRs. On June 24, 1982, the-licensee's Technical Manager assigned a group of engineers' the responsibility of reviewing NCRs for reportability.

The' NCR form was revised to include a signature block for a positive or negative reportability determination.

The licensee has also impTemented measures to assure that NCRs are promptly routed to the quality assurance, station technical, and configuration control and compliance (CC&C) staffs for validation and reportability review.

Discussions with station technical. and CC&C personnel identified two concerns.

a. Familiarity with Technical Specification Reporting.Criteria.

Personnel in theCC&C staff, responsible for the final determination of reportability, were fully knowledgeable ofthe.Technical Specification reporting criteria. However, discussions with station technical person nel.indicated that these persons had only a.general familiarity with the reporting criteria in the conduct of their reviews. Furthermore, the station technical persons indicated that they had-not received any formal training, in the form of lectures or required reading, on the reporting criteria. The licensee stated that action would be taken to assure that appropriate personnel were. trained and knowledgeable of the Technical Specification reporting criteria.

b. Assurance that All NCRs Have Been Evaluated for Reportability.

On about July 1, 1982, the licensee began annotating the NCR forms with a section for determining reportability. Before that time, it is apparent that the NCRs were not formally evaluated for reportability and so docu mented. The licensee agreed to review for reportability all NCRs issued up to the time the.form was revised.

On March 25, 1982,:the licensee performed a surveillance of the Unit 2 reactor breaker undervoltage and shunt trip devices and documented that TCBs 4, 6, 7, and 8 were nonconforming. Non-conformance Report S023 P-152 was written on March 25, 1982, documenting the nonconforming condition.

Discussions with the station technical reviewing engineer established that this NCR was not reviewed for reportability. This event was not reported to the NRC.

On July 12, 1982,.the licensee performed a surveillance of the Unit 2 reactor breaker undervoltage and shunt trip devices and documented that TCB4 and 7 would not trip when their undervoltage devices were deenergized.

This condition was documented by NCR No. S023 P-511, written on July 12, 1982.. A block for reportability determination was annotated on the NCR.

However, apparently a reportability determination was not performed since the block was not checked. Discussions with the reviewing engineer indicated that the NCR was not reviewed for reportability. This event was not reported to the NRC.

This failure to report the failure of certain reactor trip breakers to open upon deenergizing the undervoltage devices, observed on March 25 San Onofre RTB SER III-15

.7"7 77777S-

and July 12, 1982, is.

an apparent violation of the reporting criteria of Technical Specification.paragraph 6.9.1.12.i'.

The staff examined the circumstances surrounding the failure of the Unit 3 trip breaker' on March 1, 1983, and the failure.of the three RTBs on March 8,. 1983,' and evaluated these.events for compliance with the reportability criteria of theTechnical Specification. With-regard to the single RTB breaker failure.on March 1, the licensee concluded that the failure represented an isolated instance and therefore did not report this failure. The.staff concurs %vith.the licensee's action.

With regard to the'failure of.the three RTBs which occurred on March 8, and'were'reported to the NRC on March 10,, 1983, the staff evaluated the licensee's process for reportability determination. This evaluation found that the licensee exercised prope'r judgement and complied with administrative and, programmatic requirements relative to the reportability determination. and the timing.of the report iprovided to the' NRC.

In their April 15, 1983 Reactor Trip Breakers Report the licensee

describes the,history of the review methodology:used to determine' reportability of nonconforming conditions (Section IV.5 and IV.6). The licensee,acknowledges that, due to procedural inadequacies, reportable conditions could, and, in fact, in the case of the earlier RTB failures, did go unreported. As part of the licensee's proposed' improved program for handling nonconformance reports, a comprehensive review of NCR's prepared prior to November 1982 was conducted to identify conditions that should have been reported but.were not. The licnesee is.also revising procedures to strengthen reportability determination by providing clarif ication'and reference material to support reportability determination.

This 'action is currently underway and will be-the subject of training provided to responsible individuals to ensure their familiarity with the new procedures.

The NRC staff will review the implementation of the new procedures and the training associated with the new procedures. The results of this review will be documented in a Regional Office inspection report.

Based on our above-described evaluation of the April 15, 1983 report, and our evaluation of the licensee's reporting of the March 1983 RTB failures, and our evaluation of the improved procedures that were implemented in November 1982, 'we conclude that there-is reasonable assurance that non conforming conditions requiring a report to the NRC will continue to be promptly recognized and reported as required.

III.B.6.

Compliance with IE Bulletins and Circulars During the March 12-15 inspection, the NRC inspector reviewed the licensee's actions in response to IE Bulletin 79-09. The inspector determined that, in response to the Bulletin, the licensee had stated that a preventive maintenance program would be developed in accordance with the Bulletin's requirements prior to plant operation. NRC Inspection Report No. 50-361/81-07 verified that a preventive maintenance procedure, MPES 008, "Undervoltage Tripping Device of GE AK 2-25 Circuit Breakers in the Reactor Trip Switchgear," was developed to meet the inspection requirements of the Bulletin.

San Onofre RTB SER 111-16

The inspector reviewed Revision 1 of MPES 008 and discussed its implementation with cognizant licensee personnel.

The inspector determined that this revision of the procedure implemented many of the recommendations of General Electric Service Letter No. 175 (CPPD)9.3. (This letter was a requirement of.paragraph 3.c of IE Bulletin 79-09.)

However, the procedure was deficient 'in that it did not require checking for excessive clearance between the undervoltage trip device armature and "rivet."

Licensee personnel were unable to explain this omission.

Also, the Bulletin required that the frequency of inspection. of the breakers be increased until the service experience with the breakers demonstrated.that the normal (annual) maintenance interval-was adequate. Similarly, the vendor, Combustion Engineering, recommended on May 27, 1981, that preventive maint enance in accordance with the manufacturer's recommendations should be performed once.every refueling interval unless periodic testing indicatesthat a-more frequent interval is required (emphasis added). The inspector determined.that the licensee had established that trip breaker maintenance would be performed at:each refueling, and had performed Maintenance Procedure MPES 008, Revision 1, in March7-May 1981.' The inspector discussed with maintenance department repre sentatives 'the: licensee's method for incorporating service experience, testing experience, and IE Bulletin requirements into preventive maintenance schedules.

These representatives stated that at Units 2 and 3, key maintenance planners were expected to revise maintenance schedules on the basis of their knowledge and experience with the equipment. The inspector reviewed Maintenance Proce dure S023-I-1.1, "Scheduling of Preventive Maintenance," and concluded that this expectation was not formally delegated. The.inspector concluded that in this case the' licensee's procedures.for revising preventive maintenance sched ules had been inadequate. Consequently, the development of a trip breaker pre ventive maintenance program, as required by IE Bulletin 79-09, was incomplete.

10 CFR 50.34 and the licensee's approved Quality Assurance Program, Chapter 5-C, paragraphs 2.0 and 3.0, require that the preventive maintenance frequency be adjusted to reflect the equipment supplier's suggested schedules and the main tenance history of the equipment. The.licensee's Maintenance Program Pro cedures 50123-M-5, Revision 2, "General Maintenance Order," paragraph 6.3.2, and S0123-M-4, "Preventive Maintenance Program," paragraph 6.2, reiterate this policy. The licensee's failure to revise its preventive maintenance pro gram to reflect the licensee's testing experience and equipment.supplier's suggested schedules, as well as IE Bulletin 79-09 requirements is an apparent violation.

The licensee has reviewed all IE Bulletins and has committed to review all IE Circulars and Notices to determine whether any other deficiencies exist in the preventive maintenance program. This review is scheduled to be completed by April 30, 1983 and the licensee has agreed to effect appropriate corrective action to resolve any deficiencies identified pursuant to that review. The NRC staff will examine the results of this review program, and the corrective action for identified deficiencies, during a future inspection.

The NRC staff has taken enforcement action, in NRC Inspection Report No. 50-361/

83-13, in response to the licensee's failure to reflect IE Bulletin 79-09 requirements in their preventive maintenance procedure for the reactor trip San Onofre RTB SER 111-17

breakers..

Region V will evaluate the licensee'-s response and examine the corrective actions.during a future inspection.

Based' upon the above commitments the NRC staff concludes that there is reasonable assurance that, when.the above, corrective action is completed, the requirements -of Bulletins, Circulars, Notices,,equipment supplies. suggested schedules,.and testing' exper ience will,'be reflected in the licensee's preventive maintenance program procedures.

III.B.7.

Post-Trip/Restart Reviews A review of plant response must' be conducted to determine the cause of any reactor trip. This review shall also determine whether all plant protective features operated'properly. Assuming multiple failures of the Plant.Protective System are.necessary to cause an ATWS event,'thi:s review should'be capable of identifying an ATWS 'event and prevent restart of the plant until appropriate corrective actions have been taken.

Prior to the recent breaker failures at Salem, Operating. Instruction 5023-0-11, "Startup and Shutdown Chart Removal and Identification" (Revision 1), iden tified that a'review of shutdown/startup and trip/transient cha-rts by station engineering personnel could be conducted.

The procedure did.not require completion of this engineering review prior to restart of the-plant.

The shift supervisor had responsibility for authorizing restart of the plant based upon his assessment of plant 'status and readiness.

This authorization was not a formally documented decision.

Subsequent to the recent.Salem event, Operating Instruction S023-0-11 has been revised (Revision 2 dated March 25, 1983) to include a formal post-trip review through the use of a checklist. The objectives of the post-trip review are:

a.

To provide a method for completing post-trip review documentation and ensuring senior level personnel review prior to authorizing reentry into operating Mode 2 (ke> 0.99)., and eff

b.

To describe in detail the review requirements of that post-trip review as fol ows:

(1) Determine the cause of the trip, and implement any required'corrective actions.

(2) Verify if the reactor protective system functioned properly, and implement any required corrective actions.

(3) Verify if ESF systems functioned properly, and implement any required corrective actions.

(4) Verify that all automatic and operator actions have been reviewed and, if any-off-normal occurrences are identified, implement any required corrective actions.

The revised procedure, in combination with the plant startup procedure, assures completion of a post-trip review and of appropriate corrective actions prior San Onofre RTB SER 111-18

to reentry into Mode 2 and return to power. The completion of corrective actions identified above are required only for actions necessary to ensure proper operation of systems important to safety. Corrective actions could be identified which would not prohibit restart.

If the cause of the trip is ;determined during the post-trip review, authori-.

zation by the shift supervisor, and the plant superintendent (or, in his absence, the station operations manager) is required prior, to entry into Mode 2 and return to power. If

.,the cause of the trip cannot be determined, authorization by. the station manager or his superior is also required prior to restart Our review revealed a lack.of clarity with :regard to whether both the shift super visor:and the plant superintendent authorizations were required in every case.

Operating Instruction S023-5-1.3.1, "Plant Startup from Hot Standby to Minimum, Load" is the plant operating procedure ihich provides the steps required to change the plant status' from Mode 3 tp Mode 2. As part of this procedure the shift supervisor must verify that the,post-tripreview (Operating Instruction S023-'0-11) has been satisfactorily completed prior to entering.Mode 2.

The staff finds that the present Post Trip/Restart Review procedure requires further. revision in certain critical areas.

1. The procedure does not provide guidance as to how to evaluate performance of plant systems during a transient situation. Instead it describes how to document a review.

2. The procedure does not giv.e guidance to ensure that the personnel will perform both an independent and inquisitive evaluation. We believe the post-trip review should be conducted by a qualified individual (or group) who was not involved in the control room activities and should be approached

.,as a fresh, inquisitive determination of the cause of the trip and evalua tion of plant systems performance.

3. The procedure does not include an adequate "default clause."

The proce dure should require that if all significant aspects of the transient are not well understood within a period of time, such as 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br />, the plant would be held.in hot or cold Shutdown and a more thorough investigation undertaken.

4.'

The procedure does not clearly indicate who has the final authority to authorize plant restart. This authority should be explicitly defined with no ambiguities.

5. The procedure does not provide for followup evaluation of the post-trip review. This procedure should explicitly provide for followup review by the plant independent safety engineering group within 30 days of restart from a trip/transient.

By letter dated April 22, 1983 the licensee committed to re-review the post trip procedures and to advise the NRC staff within 45 days of appropriate revision. The review will encompass the above items and will include review by the Nuclear Safety Group. Therefore, we conclude that this procedure is acceptable for restart.

San Onofre RTB SER 111-19

III.B.8. Results of Previous Evaluations of Licensee Administrative Controls III.B.8.a. SALP The systematic assessment of licensee performance (SALP) is an integratedNRC staff effort to collect the available observations on an annual basis and evaluate licensee performance based on these observations with te.objectives of improving the NRC regulatory program and licensee performance.

The last annual SALP assessment for San Onofre 2 and 3 covered the. period July 1, 1981 through June 20, 1982.

During this period, Unit 3 had not yet received its operating license and. Unit 2 was basically still performing post.core load preoperational testing.

The results of the SALP board findings.were sent to the licensee in a letter dated December 30, 1982.

Based on the -board's assess ment of licensee performance on Uni.t 2 and..3, all functional areas were found to be Category 1 or Category.2 which -indicates overall performance of licensed activities to be acceptable and directed toward safe faci'lit operation. How ever, in the functional area of plant maintenance, two areas of programmatic oversight and weakness were, identified.

These 'are:

(1)

Administrative controls covering the control-of vendor supplied information were not established to assure that the information was appropriately coordinated, controlled and 'evaluated for into technical manuals and main tenance procedures.

(2)

Administrative controls had not-been established to assure that equipment operability and functional tests,.required by Technical Specifications or the ASME B&PV Code Section. XI, were adequately. specified, performed, and documented.

It appears that, although the licensee had initiated actions to improve performance in these areas, the problems with the reactor trip breakers were due in part to those identified concerns.

III.B.8.b. INPO The Institute of Nuclear Power Operations (INPO) conducted an evaluation of San Onofre Unit 1 during the period of November 30 through December 7, 1981.

An evaluation of.Units 2 and 3 has not been conducted, but has been scheduled.

The maintenance-related findings of INPO during that evaluation were:

(1)

Periodic retraining was not being conducted for maintenance personnel.

(2)

The maintenance history system was not adequate to allow meaningful evalua tion of equipment performance, identification of persistent or recurring maintenance problems, identification of the cause of failure, or timely retrieval of information from the plant records repository.

(3)

The preventive maintenance program was in need of expansion to include major pieces of equipment-in the NSSS support systems.

San Onofre RTB SER 1.1-20

The licensee indicated that a computer based system was being established to schedule and document preventive' maintenance, as a long term resolution for portions of these issues.

While there are separateprocedures for Unit 1 and Units 2 and 3 maintenance activities, the concerns identified in the Unit 1 INPO Report, in our judgement, appear to be appropriate for activities at Unit 2 and3.

III.B.8.c.

Licensee Actions Addressing Previous Evaluation Findings The licensee has committed,' in the April 15, 1983 submittal, to establish a comprehensive configuration control program to support plant operations, to be implemented following the plant start-up phase. This program will include administrative controls for the control and management'of vendor-supplied infor mation.

Until that program is: established and implemented the licensee.will.

continue-to utilize the administrative procedures that controlled.vendor sup plied documentation during the initial phases of the plant. The development and implementation will be examined by theNRC staff.

The staff has examined the administrative controls,.established and imiplemented by the,licensee, to assure that'equipment operability and functional tests, required by the Technical Specifications and the ASME B&PV Code,Section XI, are performed as required. 'The staff has concluded that the licensee has implemented adequate control over these activities.

The licensee identified, in their submittal of April 15, 1983, that certain reactor trip breakers had previously been 'declared operable, following main tenance, based upon.the performance of a functional test instead of surveil lance test, testing the independent functions of the UV and shunt trip devices.

In their April 15, 1983 submittal, the licensee-committed'to revising procedures to strengthen operability assessments." A NRC Resident Inspector examined the procedure revisidns and concluded that the licensee has adequately provided administrative controls to assure the proper performance of operability assess ment following maintenance.

Additional commitments were detailed by the licensee in his April 15, 1983 submittal (Reactor Trip Breakers). These are:

a.

Develop and implement, by July 1, 1983, training.for cognizant personnel emphasizing use of available maintenance procedures in lieu of Technical Manual references and to review the need to identify all pertinent vendor-supplied information for work order planning' where plant procedures are not available to conduct specific maintenance activities.

b. Develop and implement an operational phase system which will permit easier retrieval of individual component histories, including both maintenance and surveillance history.

c. Review the preventive maintenance program to:

establish an operating phase feedback program for preventive maintenance San Onofre RTB SER 111-21

establish baseline conditions and adequate preventive.maintenance intervals, and implement administrative controls to ensure that Dreventive'mainten ance is performed, as appropriate, for replacement components.

The staff has reviewed the above commitments for adequacy and concludes -that, when the above commitments are implemented, there is reasohable assurance that

.preventive maintenance, operability assessment, and personnel knowledge level will be acceptably assured. The NRC staff will continue to review the develop ment and implementation of the above administrative control systems.

III.C. Evaluation of.Capability to Mitigate ATWS Timely and appropriate operator response will mitigate the consequences of PWR Anticipated Transients Without Scram (ATWS) events (i.e., loss of normal feed water, loss of offsite power, and stuck open 'relief valve). Due to the relia bility of.the reactor protective system (RPS), ATWS,events are considered unlikely. Nevertheless, the operator may be required to provide timely response to compensate -for multiple RPS failures.

For the operator to respond quickly, the control room design, the plant emergency operating procedures and operator training must support..early recognition of an ATWS event and must.provide the necessary controls and displays, information and guidance to manually initiate a reactor trip, a turbine trip, emergency boration,.emergency feedwater, and deehergize the control element assemblies. The following is an evaluation of each of these subjects.

III.C.1 Control Room Design For the operator to mitigate an ATWS event the control room must be designed so that reliable information is available (1) to determine that an automatic plant protection system (PPS) reactor trip demand is present, and (2)

'to confirm that sufficient control element assemblies fully insert into the core. The con trol room must also contain the means for the operator to manually initiate a reactor trip, a main turbine trip, emergency boration, and emergency feedwater flow.

Reactor trip demand indications within the San Onofre 2 and 3 control room (Figure 7) include:

(a) 'An overhead hardwired annunciator with red backlighting entitled "RPS Trip Path Activated" (Item 13 of Figure 7).

(b) Four side-by-side plant protective system operator modules.

Each module shows plant protective system channel trips. Illumination of two of four channel.trip lights of any individual parameter constitutes a demand indication (Item 1 of Figure 7).

(c) The core protection calculator indicators can be used as an anticipatory demand and/or a demand indication (Item 14 of Figure 7).

Based on the availability of the above, the staff concludes that there is satis factory reactor trip demand indication provided in the control room.

San Onofre RTB SER 111-22

RPS Remote Oi-erator Module 2 or more channels -indicate trip 2A Depress RXgtrip-pushbuttons 0a 2..

Depresa RX trip pushbuttons (D

(

Ranctor irip1 hkr rtunzs pnarl verify oil 8 Tlls ouen 12 (D

it 15 aExcore linear power recorders verify power <61 F.P.

CEA Bot.tomed Indication M

verify all GEA bottom loes -on 0

nmuiop-cinto

-1 b e - e n e r i z e CaC 1 5 1 t Panel-DMCCM MG Power Supply o

Desk 7

Initiate EFAS !62

'1Initiate EFAS I & 2 IT 2

Initiate emergency boration

-Pumps L

2A CVCS Valves & Blend Control C+

a EmerB* boration path W

~Gravity feed valves for boration

(

Manually trip turbine Verity feedwater flow to S/Cs (MFWand/or AFW)

Annunciator panel 50A31 Par Safety Vv out. Tent.K 50A49 CEDMCS $us Undervoltage 4 ~

Boronometer - verity RCS between

(

an--l----.

1750 and 2250 ppmB before stopp Annum-Ing emergency boration

4.

0.

Confirmatory reactor trip indications as shown on Figure 7 include:

(a), Reactor power (ex-core nuclear instruments) displayed on hardwired trend recorders (Item 4 of Figure 7).

(b) A control element assembly (CEA) status panel which provides 91 (one each) reed switch rod bottom lights. (Item 5 of Figure 7).

(c) A reactor trip status panel which provides red and green (closed and open) light indication of each reactor trip breaker position (Item 3 of Figure 7).

(d) A CRT display of all axial rod positions obtained from stacked reed switches (Item 15 of Figure 7).

The staff concludes that there is sufficient accessible confirmatory reactor.

trip indication provided in the control room.

Threeindependent means by which the operator can manually trip the reactor from the San Onofre 2 and 3 control room'(Figure 7) include:

(a) Two manual reactor trip pushbuttons on the PPS panel (Item 2A of Figure 7)

(b) Two manual reactor trip pushbuttons on the Steam Generator panel.

(Item 2B of.Figure 7).

(c) The operatorcan deenergize the CEAs independent of the reactor trip breaker positions by deeneergizing buses B15 and 816 (Item 6A and 68 of Figure 7).which deenergizes the RPS motor generator sets.

The staff concludes 'that the control room contains sufficent means to manually trip the reactor if required.

Other immediate control room operator actions requied to mitigate an ATWS event (Figure 7) include:

(a) Manually tripping the turbine (Item 9.of Figure 7)

(b) Initiation of emergency feedwater using pushbuttons (Items 7A and 7B of Figure 7).

Verify feedwater flow to the steam generators (Item 10 of Figure 7).

(c) Emergency boration using all-available charging pumps (Items 8A, B, C, D and 12 of Figure 7).

The staff concludes that there are adequate controls in the control room for the operator to carry out the required immediate operator actions for an ATWS event.

The staff further concludes that there are sufficient and satisfactory indica tions and controls contained within the control room which are appropriately located so that the operator can mitigate an ATWS event if required.

San Onofre RTB SER I-24

III.C.2 Reactor Trip and ATWS Procedures On March 11,-1983, the NRC issued IEB 83-04 concerning failure of the under voltage trip function of GE AK-2'.reactor trip breakers. This bulletin request that all operating reactor facilities (except those with Westinghouse DB-50 breakers) perform tests on undervoltage devices, review the maintenance program on the breakers,, notify all licensed operators of the failure-to-trip event at Salem and the testing failures at San Onofre, and finally, to report to the NRC the various results of the program described.

Emergency Operating Procedures are provided to' give the operators written guidence. to mitigate ATWS events.

The NRC staff interviewed San.Onofre 2 and 3 operators on.April 5, 1983,.and has reviewed the San Onofre Nuclear Gener:at ing Station Emergency Operatin Instruction S023-3.5-1, Emergency Plant Shutdown, Revision 9, which is the procedural guidance :provided for normal reactor trips and ATWS 'events to resolve the following issues:

a.

Can,the operator(s) identify or recognize an ATWS 'event based on.knowledge and instructions withi.n the EOP?

b. Are the steps within the procedure technically correct such that if followed the consequences of an ATWS event'will be mitigated?

c. Are the steps within the procedure clear to avoid 'misinterpretation of the procedure?

d. Are there appropriate caution statements and/or notes which give firm guid ance whenever an operator is instructed to bypass a plant interlock or otherwise defeat an automatic plant protection function.

The operator's role in responding to an ATWS is to first recognize that an ATWS has occurred and then take action to manually shut down the -reactor and stabi lize all systems.

Revision 9 of Emergency Operating Instruction S023-3-5.1 does address recognizing ATWS events in the Emergency Plant Shutdown Item 1.0 Symptoms.

This-includes items 1.1 Alarms, 1.2 Indications, and 1.3 Key Parameters. Both sub-items 1.1.1 (alarms) "RPS.Trip Path Activated" and 1.2.1 (Indications) "Reactor Trip Breakers Open," are reactortrip indications observed by operators in the control room. A reactor trip signal from the plant protection system causes both the undervoltage coil to deenergize and the shunt trip coil to energize, each of which independently actuates, the trip bar and'tripping mechanism. Paragraph 3.1.2.1*(Immediate Operator Action on ATWS):

"Deenergize Load Centers B15 and B16."! This step is performed from the control room and cuts off the power to the control element drive motor generator sets..

If all reactor trip breakers had not previously opened, thus deenergizing the CEDMs, removing power from the MG set would deenergize the rod mechanisms and allow the rods to fall into the core. Also, Item 3.0 "IMMEDIATE OPERATOR ACTION", provides instructions to. push all four manual reactor trip push buttons (upon demand or confirmatory trip indication), to manually initiate emergency feedwater activation signals, to initiate emergency boration and to manually trip the turbine. The ATWS section of the procedure is written assuming two failures.

San Onofre RTB SER 111-25

a. Failure of the undervoltage device to open the reactor trip breakers.

b. Failure of the shunt trip coil to open the reactor trip breakers.

A loss of station power would cause a loss of power to the MG sets and result in the control rods dropping into the core. Power.to the undervoltage and shunt trip'devices is supplied by Class IE-125 VDC power the loss of which, while credible, is highly unlikely.

The prescribed actions are technically correct and if they are followed the consequences of an ATWS event will be mitigated. Furthermore the steps within the procedures are achievable by the operator(s).

The instructions must have an adequate technical basis to provide confidence in, their appropriateness, and, they must-be written so that the operator(s) can understand and implement -them in an environment of high stress. This includes immediate actions that must be committed to memory so that they qan be performed without actual reference the procedure.

The.staff'concludes that the.emergency operating procedures provide the operators with means to identify or recognize. an ATWS event based on knowledge and instructions in the EOPs.

The stepswithin the procedures conform to the guidance of NUREG-0460. The procedures are clearly written and include appro priate caution statements.and instruction on plant interlocks. Based on these findings the staff concludes that Revision 9 of S023-3-5.1 Emergency Operating Instruction provides adequate guidance to the operations.to mitigate an ATWS event.

III.C.3. Operator Training/Knowledge In performing an analysis of training needs it is assumed that ATWS is a low probability event, but the benefit of proper operator response to an ATWS event is very great. Due to the high benefit, ATWS training should be incorporated into operator initial, replacement, and requalification training programs. To respond properly to an ATWS event the operator must have the required knowledge and skills to recognize an ATWS event, and to respond in timely manner with the required immediate.operator actions.

On April 5, 1983, the NRC staff interviewed San Onofre Unit 2 and 3 operators.

The operators described demand and confirmatory reactor trip indications. The operators stated that the "RPS Trip Path Activated", annunciator is.a demand annunciator. The operators also stated that highly reliable reactor'trip demand indications include indication from the plant protective system operator modules and the core protection calculator(s).

Operators also stated that highly reliable confirmatory reactor trip indications include reactor power indication, the CEA Status Panel, and the Reactor Trip Status Panel.

They also stated that confirmatory reactor trip backup indication is provided by the CRT display of all axial rod positions.

Based on this review the staff concludes that the operators have a good under standing of Plant Protective System reactor trip demand and confirmatory indications.

San Onofre RTB SER 111-26

Operators described under what conditions they would activate all four manual scram pushbuttons as. required by procedure (5023-3-5.1, Step 3.1).

3.1 If any NSSS parameter is rapidly approalching a Reactor Trip setpoint.

or if two or more RPS channels monitoring the same parameter reach a trip setpoint, then push all four manual reactor trip pushbuttons.

The operators stated that they would manually initiate an anticipatory reactor trip, if they had attained sufficient pretrip annunciators or other indications with the plants in a transient condition such that they expect an automatic trip to occur. The operators stated that in such a case they would not challenge a safety system (i.e., the RPS).

The operators also stated that if the core protection calculators indicate a decreasing margin' to thermal limits (i.e.,

departure of nucleate boiling ratio and local power density) that they would' provide a manual anticipatory reactor trip prior to these margins 'reaching zero.

Operators also stated that the procedure (step 3.1) requires that all four manual reactor trip pushbuttons be depressed on any reactor trip demand irrespective of confirmatory reactor trip indication.

The staff concludes that the operators have had effective training and will actuate.a manual reactor'trip either before, or ina timely fashion after, an automatic trip should haveoccured.

Station operators were asked whether they would manually trip the main turbine (as required by 'procedure). during an ATWS. This action would separate the reactor from its normal heat sink (while operating at power). The operators were'able to describe the technical bases which makes tripping the turbine a desirable action. Tripping the turbine causes secondary plant pressure to increase, primary plant temperature to increase, thus adding negative reactivity causing reactor power to decrease. The operators were able to describe the other immediate operator actions required during an ATWS event.

The staff concludes that the operators knowledge of this procedure 'is sufficient and that the appropriate steps have been memorized and are understood.

Following the issuance of IE Bulletin 83-01, on-shift training sessions were

-conducted for' each operating shift. This included reviewing the ATWS operations included in EOI S023-3-5.1 "Emergency Plant Shutdown."

In addition, each licensed operator was required to review the IE Bulletin referenced above and IE Bulletin 83-04. After the revision of EOI 5023-3-5.1 on March 25, 1983, all licensedreactor operators were required to review and acknowledge, in writing, this review of the revision to the.procedure. In addition, the five operating shifts will receive, as part of the requalification training program, formal classroom training and have an apportunity to discuss the revised procedure. This will occur over a 5-week period.

Based on the interviews with of the licensed operators, it was determined that their knowledge in identifying a reactor trip demand, in providing anticipatory and/or manual backup reactor trip actuation (as required by procedure), and in the immediate steps required to be performed by the operators was sufficient.

The staff concludes that the operations staff has been adaquately trained to mitigate the consequences of an ATWS event.

San Onofre RTB SER 111-27

III.D.

Licensee Conclusions Regarding Causes of RTB Failure In their submittal of April 15, 1983, SCE stated that they had reached the following conclusions:

1. The reactor trip breaker undervoltage trip device is not required for the San Onofre Units 2 and 3 reactor protection system. (RPS) to perform its design basis protective function;.failure of the UV trip device does not affect the ability of the San, Qnofre Units 2 and 3 RPS to perform its'.

protective.system function due to the presence of the shunt trip.

2. From the results of the investigation of four RTBs, it is concluded that the cause of the unreliable operation of the undervoltage trip device for the reactor tripbreakers was a combination.of the small design margin in the force provided by the UV trip device and the-following:

Degraded lubricant on the trip shaft bearings *or latches.

Incorrect setting of the UV device pickup voltage.

Excessive clearance in UV device 'armature hinge area.

3. Despite the small design margin in the force provided by the UV trip device, the UV trip feature of the RTB can be made to operate reliably when enhanced maintenance and survillance techniques are used.

4. The high incidence of equipment difficulties experienced during the start up phase hindered early identification of the RTB problem. Based on this and the conclusion that the UV trip device failures were caused by factors which were not previously clearly known (i.e., temperature effect on UV coil settings and quantitative values required for armature to rivet clear ance), it is concluded that the likelihood of the RTB malfunctions experienced may have been reduced but would not.have been precluded if administrative weaknesses had not existed in SCE programs.

5. A procedure which addresses ATWS was in place at San Onofre Units 2'and 3 prior to the Salem RTB.trip failure. This procedure, S023-3-5.1 "Emer gency Plant Shutdown," was revised on March 25, 1983 to improve operator response to an ATWS event by requiring immediate manual reactor trip and then manual turbine trip when an automatic trip set point is being rapidly approached, regardless of whether the reactor has tripped automatically.

The San Onofre Units 2 and 3 control room is already configured such that the instrumentation and controls to detect and mitigate ATWS are easily accessible to the operators.

6. The existing SCE operator training/retraining program contains sufficient provisions for promptly.notifying operators of important procedural changes and for additional reinforcement through follow-on training and retraining.

7. Potential improvements to SCE administrative procedures and implementa tion have been identified in a number of areas and are either completed or in progress.

San Onofre RTB SER 111-28

III.E. Staff Evaluation and Independent Conclusions III.E.1. Potential Gonsequences of Failure of UV Trips On February 25, 1983, NRC issued Bulletin 83-01 because of the ATWS operating event.at the Salem plant. The Bulletin required PWR licensees with Westihg house type DB-50 breakers in the reactor protection systems to take certain actions, including testing the breakers. The San Onofre 2 and 3 units do not use DB-50 breakers and were both in cold shutdown at the time. Nonetheless, the licensee decided it would be prudent to test the GE AK-2 reactor trip' breakers. On March.1, 1983, for one out of eight breakers at Unit 3, the UV device was unable to trip the breaker; all other UV trips were.successful; and all shunt trips were.successful. On March 8, 1983, for three of the eight

'breakers on Unit 2, the UV device was unable to trip the breaker; all others were successful.

The Unit,2.breakers that failed were.TCB1, 4, and 6 (See Figure 1).

This occurrence at this plant is quite different from the Salem event. First, the Salem event was an actual reactor operating event; the failures at San Onfore were discovered during testing with the reactor shut down. Second, at Salem, all UV trip devices failed resulting in total loss of the automatic safety function; at San Onofre, five UV trip devices were operable and all eight shunt trip devices were operable.

To evaluate the potential consequences of. the San Onofre UV failures, we considered the full power situation. If a plant transient were then to occur and one postulates the same three UV failures (TCB1, 4, and 6) by themselves (i.e., ignoring the shunt trips), breakers TCB2, 3, 5, 7, 8 would 'function in response to the UV device (see Figure 1).

These breakers are sufficient to cause all the control rods to.fall into the core. Further, when the shunt trips are considered, all eight trip breakers would'function. In this design configuration since the shunt trips are included, the failure of any or all UV trip devices has no consequences on the outcome of an anticipated transient.

An ATWS event would not occur.. We therefore conclude that there are nc direct safety consequences of the UV failures during an anticipated transient.

III.E.2. Summary of Contributory.Causes Based on our evaluation of the licensee's tests and evaluations, the staff concludes that in conjunction with the small design margin between the force produced by the UV device and the force required to trip, there are three major causes contributing-to the failure of the reactor trip breakers to open.

These are:

(1) Inadequate lubrication of the trip latch roller bearing and trip shaft bearings.

(2) Improper setting of the undervoltage 'trip device pick-up voltage.

(3) Out of tolerance (excessive) clearances in the undervoltage device armature pivot area between the armature and rivet.

San Onofre RTB SER 111-29

The staff's conclusion above that inadequate bearing lubrication is a -major contributing factor

>to the RTB failures is based on the following tests con ducted on the Unit 2.RTBs.

(1) During the period from.March 12 to 17, 1983, SCE tested two of the four failed breakers, TCB1 and TCB6 from Unit 2. These tests were conducted at the San Onofre site and are discussed in Section III.A.1 of this report.

In the as-found condition, both of.these breakers were found to require excessive trip shaft torque (the recommended' upper limit on torque is 1.5 inch-pounds, per IEB 79-09) and were observed to be erratic and slow to trip. In some trials, failure to trip was observed. After-the application of a grease revitalizing'compound (CRC-5-56) to the.trip shaft bearings and the latch roller bearing, the trip shaft torque ranged from 1.0 to 1.56 inch-pounds, and the two breakers were observed to trip rapidly and positively.

(2) During the period from March 26 to April 1, 1983, SCE tested failed breaker TCB4 from Unit 2 at their Electric Test Laboratory in Alhambra, California.

This. testing is discussed in Section III.A.2. of thi.s report. In the as found condition, the torque required to trip the breaker was observed to tbe in the range of 1.56 to.2.0 inch-pounds.

Trip testing of this breaker.

in the as-found condition showed inconsistent operation, with some failures to trip. After cleaning and relubrication of the trip shaft and latch roller bearings, the trip shaft torque was found to range'.from 1.00 to 1.44 inch-pounds,.and the trip time using the UV coil was found to be in the'range of 65.4 to 70.3 msec(with the surge protection diode installed).

(3) During the period from April 6 to 10, 1983, nine Unit 2 breakers were maintained using the revised preventive maintenance procedure, S023-I-4.66.

Six of the nine breakers were initially found to have trip shaft torque values in excess of 1.5 inch-pounds. Following maintenance, which included revitalizing the trip shaft and latch roller bearings with CRC-5-56, as well as adjusting the UV coil pickup voltage and armature-to restraining rivet clearance, the trip shaft torque for all 9 RTBs was found to be in the range of 1.0-1.26 inch-pounds, and all RTBs had trip times less than 70 msec.'

Since the application of CRC-5-56 to the trip mechanism bearings in the above tests resulted in trip shaft torque being reduced to within the recommended upper limit, and also resulted in trip times being reduced to 70 msec or less, we conclude that the bearing lubrication in the as-found condition was inadequate and that'this was a major-factor contributing to the observed RTB failures.

The last time that we know that the Unit 2 RTBs were maintained was July 14-21, 1982.

This' "overhaul" may have included lubrication, but the specifics of this work were not documended. Since grease degrades with time, we cannot be sure whether the inadequate lubrication that existed in March 1983 was the result of inadequate lubrication during'the July 1982 maintenance or degradation of lubrication which was adequate in July 1982. Therefore, we conservatively conclude that the March 1983 RTB failures were due to inadequate lubrication.

San Onofre RTB SER 11I-30 7

7 F,-77 4W

-1

The staff's conclusion above that improper setting of the UV trip.device pickup, voltage is a major contributing factor to the RTB failures is based on the results of the following tests.conducted on the-Unit 2 RTBs.

(1) The three Unit 2 RTBs that failed (TCB1, TCB4, and TCB6) had pickup voltages in the range of 93.7 to 103.7 volts dc, all below the recommended range of 104-110 vdc.

(2) When TCB2, a Unit 2 RTB that passed the March 8, 1983 surveillance test was examined, its as-found pickup voltage was determined to be 106.0 108.1 vdc. 'However, when its pickup voltage was' reduced to 100 vdc, it exhibited unacceptable operation -

i.e., it was slow to trip.

(3) When failed breaker TCB4 was tested in the laboratory, increasing the as-found pickup voltage of 93.7 vdc'to.104.9 vdc resulted in~ improved, acceptable performance', with a maximum.trip time 'of 75.7 msec.

(4) During the period from April 6 to 10, 1983, nine.Unit 2 breakers were maintained using the revised preventive maintenance procedure, S023-1-4.66.

Seven of the nine breakers were initially found to have UV device pickup voltages that.were less thanthe minimum setting.of 104 vdc.

Following maintenance, which included adjusting the UV coil pickup 'voltage to 106 vdc, as well as revitalizing the trip mechanism lubrication and adjusting the armature-to-restraining rivet clearance, all 9 RTBs had trip times less than 70 msec.

(5) The temperature 'of the coil at the time of setting the pickup voltage was found to be of importance. The recommended pickup voltage of 106 vdc was found to be appropriate only if the UV coil is at ambient temperature (i.e., has been deenergized for 30 minutes or more).

Setting the pickup voltage to 106 vdc on a hot UV coil (one that has been energized for an hour) was found to be equivalent to setting it to approximately 85'vdc at ambient temperature. Since this effect was not known to be so important prior to these tests being performed, the low as-found pickup voltages at San Onofre could be due to setting the pickup voltage on coils that were above ambient temperature.

The staff's conclusion above that excessive clearance betweenthe UV armature and a restraining rivet in the device is not based on trip tests conducted on RTB UV devices.- Rather, it is based on (1) the manufacturer's recommended range of clearances, and (2) the licensee's analysis of the-mechanics of the 1UV device, which shows that excessive clearance between the armature and the rivet will' result in reduced throw of the armature, thereby reducing the tripping force. When the 9 Unit 2 RTBs were maintained during the period from April 6 to 10, 1983, the armature-to-rivet clearance was found to be out of the required range in six of the breakers.. Following maintenance, wherein the armature-to-rivet clearance was set within the required limits, and the RTBs were properly lubricated and the pickup voltage was set properly, all 9 RTBs had trip times less than 70 msec.

San Onofre RTB SER III-31

III. E.2.a. Design Considerations We do not agree with the licensee's conclusion that the UV trip device is not required. As we discuss in Section II.A of this report, we conclude that the UV device is' necessary to meet GDC 23. However, we find the actions proposed by the licensee to test.and maintain the UV device to be acceptable.

The-details of the breaker latch.ing mechanism and the function of the tripping devices were discussed in Sections II.B. A summary follows. The.shunt coil assembly is designed for a nominal 125 viic application to energize and actuates the trip.bar in.a counterclockwise direction. The undervoltage coil assembly is designed for a nominal 125 vdc application and is normally energized. Upon a loss of voltage the device actuates to rotate the trip bar in a counter clockwise directions.

The reactor trip signals are generated in the reactor protectioh system through six logic matrices.. 'The reactor trip signals are in turn transmitted to both the undervoltage and shunt trip devices on each RTB. -Each of the four trip path initiation relays (K relays.) sends an.actuation signal to. both the under voltage and shunt trip devices on a pair of.RTBs. The pair of RTBs receive dc power from the same vital bus that provides ac power. for their associated initiation.relay. The reactor trip breaker design is such that failure of either the undervoltage trip device or the shunt trip device does not eliminate

..the safety function of the reactor trip breaker. The trip circuit breakers will complete their protective actipn of interrupting power'to the control rods using either the undervoltage or shunt trip devices..The RTB undervoltage device and the shunt trip device complement each other in that the undervoltage device trips the breaker upon loss of control voltage while the shunt trip device trips the breaker upon application of control voltage. This diverse means of tripping the breaker is not only acceptable but is considered desirable by the staff.

The design issue that must be addressed is the.margin associated with the under voltage trip device. The UV device has a much smaller force margin to trip the RTB than has the shunt device. Nevertheless, with upgraded maintenance and surveillance testing the UV device should function reliably. Age-related degradation of the breaker could reduce this margin.. Preventive maintenance will reduce or eliminate age-related mechanical degradation of the breaker.

Thus the establishment of an appropriate preventive maintenance program at the proper interval would be the basis for the conlcusion by the staff that there are no design deficiencies. In addition a surveillance program would provide adequate assurance that (1) a single failure will be promptly detected and (2) maintenance is promptly initiated. Successful surveillance test results will.verify that the periodic preventive maintenance is effective. As discussed in.Section III.B.3, III.B.4, and IV of this report, the maintenance and surveil lance programs, when upgraded per the commitments made by the licensee, will provide the appropriate assurance.

III.E.2.b. Maintenance In summary, based upon a review of the San Onofre 2 and 3 maintenance require ments and the maintenance history, the staff concludes that:

San Onofre RTB SER 111-32 7_7 7-

,~..

-w7

(1) The established provisions prescribing control over maintenance activities were not fully implemented in that:

(a) The program did not require prompt evaluation of malfunctions identified during maintenance.

(b) The program did not include details. regarding assignments and instructions for -implementing the program objectives.

(c) Corrective maintenance performed was not adequately prescribed prior to performing, or documented after completing, the work.

(d)- Preventive maintenance frequencies were not adjusted to reflect.

observed adverse service experience because the responsbilities for such adjustment was.not specifically delegated.

(e) On certain occasions, the licensee's maintenance procedures and the RTB vendor's, technical manual were not used in the performance of maintenance.

(f) On certain occasions, the licensee accepted setpoint adjustments of the UV device pickup voltage that deviated substantially from the criteria in the licensee's' procedures and the vendor's 1979 'Service Advice Letter (attached to NRC's.IE Bulletin 79-09), on the basis of verbal recommendations from the vendor's service technicion.

(2) The activities.of vendor representatives 'were not controlled as prescribed by TI-16, "Vendor Monitoring."

Specifically, procedures specifying the details of adjustments, methods of cleaning and lubrication to be performed were not prepared or documented.

These itemized maintenance deficiencies have been addressed by the licensee.

In the April 1983 Reactor TripBreakers report (Section IV.B.6), the licensee describes changes that have been made and will-be made to procedures relating to evaluation of nonconforming conditions identified during maintenance.

The staff has reviewed the procedures which have been completed and determined that the appropriate reporting requirements have now been. included in these procedures. The changes described appear to resolve the concern identified in this area and provide reasonable assurance that maintenance related non conforming conditions will be effectively handled. This area will be reviewed by the staff after full implementation and the findings documented in a Regional Office inspection report.

The'staff has examined the licensee's statement regarding programmatic controls of vendor supplied information in the April 15, 1983 report. The staff con cludes that this program.contains sufficient controls over vendor supplied information if effectively implemented. The staff has determined that the licensee has provided specific guidance to the licensee's maintenance organi zation regarding the full use of vendor supplied documentation on site for the planning and accomplishment of maintenance activities. The staff has also determined that an improved procedure for administrative controls for the utilization of vendor supplied information is in draft form and is in the near San Onofre RTB SER III-33

term. approval process. The licensee has held meetings with responsbile super visors and the procedure writers to ensure the correct handling of-vendor supplied information: A letter dated April 8, 1983 has been issued by the licensee defining the responsibilities regarding the retrieval and use of vendor supplied information relating to maintenance procedures. Based on the above and the fact 'that the-control of vendor/NRC data was not the major cause of the incorrect maintenance action on the RTBs, there is reasonable assurance that the control of vendor information and technical manuals can be and will be accomplished in an acceptable manner.

The staff has reviewed the-revision to the UV device maintenance procedure, MPES 008. Additionally it has reviewed the-maintenance action taken on all.

RTBs which established baseline data and restored all RTBs to a fully operable status in accordance with the. requirements of the,vehdor manual *and IEB,79-09.

Based on the.above, there is reasonable assurance that the RTBs are full-y operable and that the:required records of maintenance performed now existfor future use.

In their April 15, 1983 ReactorTrip.Breakers Report the.licensee acknowledges the potential for inadequate control of vendor supplied services. -In response to this situation the licensee.reviewed all (about 20,000) work.orders to

'identify those cases where a vendor performed or directed work without using a licensee procedure. Forty-one such cases-were identified and are being reviewed to determine the appropriate corrective actions.

The staff.has determined that the licensee has commenced, and is in the process of conducting increased training in verbatim compliance and in supervision of vendor work to ensure programmatic controls of vendor services are effective.

The licensee expects this training to be completed by May 31, 1983. This action when fully implemented will provide reasonable assurance that future

-activities performed by or under the direction of a vendor representative will be accomplished with due regard to quality and safety.

In-summary, we conclude that the licensee's actions to date regarding main tenance provide sufficient and acceptable bases for plant restart. The licen see's actions in the above areas will continue to be reviewed by the NRC staff and the findings documented in a Regional Office inspection report.

III.E.2.c Surveillance The NRC staff has examined the procedures S023-II-11.161 (Surveillance Requirement -

Reactor Breakers Undervoltage and Shunt Trip Device Circuit Test) revised through Temporary Change Notice 2 dated April 12, 1983, and concludes that the licensee has reviewed the procedure again for adequacy and has satis factorily resolved the comments identified in Section III.B.4 of this report and in NRC Inspection Report No. 50-361/83-13.

The licensee has developed an enhanced surveillance interval criteria for the conduct of reactor trip breaker surveillance and described a program which provides for increasing the surveillance interval based on repeated acceptable reactor trip breaker time response and operation in addition to shortening the interval whenever unacceptable surveillance data is obtained. Discussions San Onofre RTB SER 111-34

with licensee personnel indicate that three tests of each breaker will be conducted during each surveillante test.

The licensee further indicates in the April 15, 1983 report that the reactor trip breaker maintenance frequency will be once every 12.months unless sur veillance testing data indicates that maintenance should be performed more often than every 12 months. The staff has reviewed the above and concludes that the new' surveillance interval criteria are acceptable.

The staff has reviewed the proposed surveillance program and compared it with the recommendations of Regulatory Guide'1.33 and ANSI N18.7-72. The NRC staff concludes that the Quality Assurance/control audit and inspection activities of surveillance are in conformance with regulatory requirements. The staff con

'cludes that surveillance records were adequately stored, filed, and are retrievable.

Based on the above, the staff finds that reasonable assurance exists that the licensee has taken or will take sufficient action to provide for reliable.

operation of the reactor trip breakers. The 'NRC staff will follow the closeout, action regarding the implementation of the enhanced surveillance and response time testing program to be applied to the reactor trip breakers. 'The findings of this effort will be documented ina Regional Office inspection report.

III.E.3. Implications for Other Plant Systems and Components The issue of this evaluation is failure of UV trip devices that are integral parts of AK-.2-25 reactor trip circuit breakers. The licensee has provided documentation that states that the only application of AK-2 breakers at'.San Onofre is the reactor trip breakers.

Further, the licensee has stated that there is a high confidence level that there are no circuit breakers of other types with integral UV.trip devices in service at this station. He has under taken a search'to confirm this statement and has agreed toinform us of any deviations. Based upon this information provided by the licensee, we conclude that the failures of the UV devices in the reactor trip breakers have no implications on other. systems at San Onofre 2 and 3.

San Onofre RTB SER 111-35

'-7*"-'

I-

IV. Proposed Corrective Actions and-Basis for Restart In response to the identified reactor trip breaker UV device failures, the licensee conducted a comprehensive program to review potentialcauses and specify corrective action for any deficiencies so identified. These actions resulted in the submittal, by Southern California Edison, of a Reactor Trip Breaker Report, dated.April 15, 1983, to -the Office of Nuclear Reactor Regul ation.

The staff has reviewed the report and has made the following findings:

IV.A. Control.of Hardware While the initial procurement of 16 RTB's resulted in no apparent significant problems, SCE identified several problems-regarding the procurement of spare RTBs. The problems involve:

1.

Items received without certifications.

2.

Items found to be nonconforming.

3. Certification received for nonconforming items.

4.

Items received again without certification.

5.

Items found to still be nonconforming.

6.

Audit results showing:

a.

Incomplete ordering information.

b. Supplier not qualified for activities. required.

(This resulted in subtier supplier having noQA program required by procurement documents.)

These problems were apparently attributable to the more complicated procurement process involving a new supplier and a new manufactuer (GE had discontinued providing this type of breaker).

SCE-resolved these problems in accordance with its QA-program. committed to in Chapter 17 of the San Onofre 2 and 3 FSAR, with nonconforming software (certification) and hardware (the RTBs) being treated in accordance with SCE's nonconformance control system and with unacceptable-audit findings being treated in accordance with SCE's corrective

-action system. The breakers-that were-procured as spares have been rejected by SCE. SCE-is taking action to procure spare RTBs to replace those which were rejected. This is acceptable to the staff beca'pe SCE's nonconformance control system and corrective action system are functional.

IV.B.

Control of Vendor Information and Personnel SCE also identified problems regarding the procurement and control of vendor servicinglof RTBs. :These problems-involved:

1. Not following the procedure for vendor indoctrination.

2. Not assuring adequate control of vendor representative's activities and qualifications.

San Onofre RTB SER IV-1

-S 7T

These problems have resulted in.

SCE conducting training of its personnel to further emphasize compliance with procedures. SCE is also revising its proce dures to emphasize the need to adequately control vendor representatives including qualifications and to assure their proper.indoctrination. Addition ally, SCE reviewed all safety-related work orders since Unit 2 fuel load (February.16, 1982) to identify those cases where work performed or directed by a vendor was not accomplished in accordance with SCE procedures.' Fo.rty-one such cases were identified and are being individually reviewed, by the licensee, to assess necessary corrective actions.

The staff had previously identified weakness in the.licensee'sprogram for controlling venddr supplied information. The licensee has committed to develop and implement a comprehensive configuration control program, including provision for complete control of vendor information, to support the operational phase of the facility.

Based-upon these actions and commitments the staff concludes that, when the above commitments are completed, the licensee's action in this area will be acceptable; in the interim, existing procedures are adequate-to provide a basis for plant restart.

IV.C. Maintenance Procedures and QA/QC Requirements SCE identified several problems associated with the RTB maintenance procedure and related QA/QC requirements. These problems involve:

1. The RTB maintenance procedure written to implement IE Bulletin 79-09 did not reference the bulletin. This contributed to the deletion of some important information.

2;.

Unclear preventive maintenance scheduling for RTBs.

3.

RTB preventive maintenance interval greater than that recommended by the supplier.

4.

All RTBs did not recieve an initial, baseline preventive maintenance.

5. Lack of an experience feedback program.

6. Inadequate reporting of overdue maintenance activities.

7. Lack of controls to assure that replacement items have appropriate

maintenance prior to use.

8. Inadequate maintenance planning in that work documents did not reference the applicable maintenance procedure and/or vendor supplied information.

9. Not following the procedure or technical manual and inadequate inspection documentation.

10.

Excessive reliance on the vendor representatives and incomplete vendor supplied information.

San Onofre RTB SER IV-2 T,~

SCE has resolved each of these problems as they specifically relate to the RTBs.

Where appropriate, SCE has taken (or has committed to take) corrective action to determine whether. such problems have occurred elsewhere and to preclude recurrence. The actions involve such thi'ngs as review of documents, personnel training', procedure revision, and developnient of a new procedure to control vendor services.

Prior to any plant activities that depend upon proper operation of the reactor trip breakers to achieve plant shutdown, the licensee has agreed to complete the following:

1. All reactor trip breakers will be inspected and maintenance will be performed to remove the causes of failure.

2. The operability of each undervoltage trip device will be demonstrated and all the reactor trip breakers' will be baselined.

3.

Appropriate testing to: assure the operability of the reactor protection system will be completed.

4. Corrective actions associated with maintenance procedures and QA/QC requirements as, discussed in Section IV.B.3 of the licensee's April 15,'

1983 submittal on Reactor Trip Breakers will be completed...

5. 'An enhanced maintenance program as described in Section IV.

D.

3.9 of the licensee's April 15, 1983 submittal on Reactor Trip Breakers will be implemented.

Based upon the above actions and commitments, the staff concludes that, when the above commitments are completed, the Ticensee's action in this area is acceptable.

IV.D.

Surveillance Procedures and Technical Specifications The licensee has agreed to the following:

1 An enhanced surveillance program as described in Section IV.D.3.b of the licensee's April 15, 1983 submittal on ReactorTrip Breakers will be implemented. The enhanced surveillance program involves three UV opening time tests before and after each circuit breaker maintenance and at monthly intervals thereafter. The interval will be incrementally increased (up to six months in accordance with supplier'recommendations) as test results show no UV trip failure or degradation.

2.

The results of the surveillance program wil be incorporated into the maintenance program and the NRC staff will be notified of such actions.

3. Technical Specifications will be proposed within 30 days of plant restart that define surveillance testing of RTBs.

Based upon the above licensee actions and commitments the staff concludes that, the above commitments are acceptable and provide a basis for plant restart.

San Onofre RTB SER IV-3 7 '1"

IV.E. Long Term Corrective Actions In the report of April 15, 1983, SCE.has committed to take the following actions-sub.sequent to restart of San Onofre Units 2 'and 3.

1. Complete the development of and implement a comprehensive configuration control program including better control of vendor information.,

2. Develop additional training for planners.

3.

Implement the operational phase maintenance history and records program.

4.

Improve guidance regarding functional testing and surveillance testing.

5. Develop and implement a program for independent review and evaluation of repetitive nonconforming conditions.

6. Develop and implement a procedure to control vendor service during the operations phase.

7. Review the preventive maintenance program to:

a. establish baseline conditions and adequate preventive maintenance intervals

b. establish an operating phase experience feedback program for preventive maintenance

c. assure that appropriate preventive maintenance is performed for replacement components.

These corrective actions are, acceptable to the staff since they will provide additional assurance of acceptable quality.

IV.F. Operator Training There are no proposed corrective actions required prior to plant restart in the area of operator training The staff considers that the present training, including the ATWS training in progress, has provided satisfactory operator knowledge and skill to mitigate an' ATWS event.

IV.G.

Conclusions Regarding Adequacy of Program Based on the above conclusions reached by the staff regarding the causes of the RTB failures, and the measures that have been and will be taken by the licensee to remedy these problems, the staff concludes that continued operation of San Onofre Units 2 and 3 will not jeopardize public health and safety.

San Onofre RTB SER IV-4

Appendix A Principal NRC Staff Reviewers H. Rood Project Manager J. T. Beard Operating Reactor Assessment A. Chaffee Resident Inspection V. DeLiso Procedures Review R. Eckenrode Human Factors Engineering D. Kirsch Regional Inspection M. Martin Operator Training R. Ramirez Human Factors Engineering J. Spraul Quality Assurance R. Wright Equipment Qualification A-1

ML13309B266

Text

SUMMARY

SUMMARY

Navigation menu

Search