ML20073G835
| ML20073G835 | |
| Person / Time | |
|---|---|
| Site: | San Onofre  |
| Issue date: | 04/30/1983 |
| From: | SAN DIEGO GAS & ELECTRIC CO., SOUTHERN CALIFORNIA EDISON CO. |
| To: | |
| Shared Package | |
| ML13309B251 | List: |
| References | |
| REF-SSINS-6820 IEB-83-01, IEB-83-1, NUDOCS 8304180399 | |
| Download: ML20073G835 (73) | |
Text
.. . .
O San Onofre Nuclear Generating Station Units 2 and 3 Unit 2 Docket No. 50-361 Unit 3 Docket No. 50-362 REACTOR TRIP BREAKERS April 1983 O
l l
gE southern caittornia Edison company
\ > San Diego Gas & Electric Company Cityof Anaheim City of Riverside okk!OKO
. -_______-.-___.m.__=_ _
_ .__ _m_ _ _ _ . _ . _ _ _ _ . . _ . . . _ _ .
5 Southern California Edison Company y AO.SOE800 f
2244 WAl. NUT GROVE AVENUE ROBERT DIETCH T ELEs*M O N E l
vice poes.es=, as3 572-di44
- April 15,1983 Mr. H. R. Denton Office of Nuclear Reactor Regulation
! U.S. Nuclear Regulatory Commission Washington, D. C. 20555 I Gentlemen:
l
Subject:
Docket Nos. 50-361 and 50-362 San Onofre Nuclear Generating Station Units 2 and 3 l
SCE met with the NRC staff on April 12,1983 in Bethesda, Maryland to review the technical aspects of the March 1 and 8,1983 surveillance test failures of the San Onofre Units 2 and 3 reactor trip breaker (RTB) i; undervoltage trip devices. By letter dated April 13, 1983, SCE trans-mitted a report relative to the technical aspects of the RTB discussed during the April 12, 1983 meeting and committed to provide additional O information regarding the progrannatic aspects of this issue.
Consistent with this commitment, enclosed please find sixty three (63) copies of the Reactor Trip Breaker report for San Onofre Units 2 and 3.
4 This report provides infonnation on both the technical and programmatic aspects relative to the RTB's. The comparable technical sections of this report are essentially unchanged from the information submitted i on April 13,1983; however, to the extent that these two reports differ, the report transmitted by this letter supersedes the report previously transmitted on April 13, 1983.
l Also enclosed is a copy of the handouts which were used during the i April 12,1983 meeti ng.
Please contact me if you have any questions or comments.
I Very truly yours, l
cc: Mr. John Martin, Regional Administrator NRC Region V O
i I
s ENCLOSURE T0
( SCE TO NRC LETTER DATED APRIL 15, 1983 FROM R. DIETCH TO H. R. DENTON The following paragraphs of the enclosed April 15, 1983 Reactor Trip Breaker Report contain changes other than editorial changes from the technical portion of the RTB report provided by SCE's letter dated April 13, 1983:
III.B.2.d The fourth sentence was corrected to reflect the fact that when the breaker is installed in the " test" position (not " racked out" as indicated previously), the diode is in the UV coil circuit.
IV.D.3.b.6)
Change provides clarification of the intent of the maintenance procedure.
IV.D.4 Paragraph was reworded to clarify intent.
O V.A.3 V
The third paragraph was reviset. to be consistent with discussion provided in Section IV.D.3.b.6).
VI.A.4 Paragraph was reworded to clarify description of shunt coil per-formance.
O
4/15/83 T SAN ONOFRE UNITS 2 AND 3 RETURN TO POWER REPORT REACTOR TRIP BREAKERS OUTLINE I. EXECUTIVE
SUMMARY
II. SCOPE AND PURPOSE i
III. INTRODUCTORY AND BACKGROUND INFORMATION A. Reactor Protection System
- 1. Design Criteria
- 2. Design Description B. Reactor Trip Breakers
- 1. Design Criteria
- 2. Design Descriptinn C. RTB Initial Procurement and Testing
- 1. Procurement History
- 2. Acceptance Tests - Vendor
- 3 Acceptance Tests - Startup
- 4. Startup and Surveillance Tests O D. Man:h 1 and 8,1983 RTB Surveillance Test
- 1. Description of Tests
- 2. Test Results IV. INVESTIGATIVE PROGRAMS A. RTB Investigative Tests
- 1. Description of Tests
- 2. Results of Investigative Tests B. Evaluation of Licensee Administrative Procedures
- 1. Control of Vendor Information, Technical Manuals
- a. Vendor-Supplied Documentation
- b. Vendor-Supplied Services
- c. Vendor Recommendations on RTB
- 2. Control of Hardware Configuration
- a. Spare Part Programmatic Controls
- b. Reactor Trip Switchgear Spare Parts
- 3. Maintenance Program
- a. Procedures and QA/QC Requirements
- b. Maintenance History and Records
- c. Vendor Maintenance Activities
- o i
4/15/83
,O 4. Surveillance Program
- a. Technical Specification Requirements
- b. Procedures and QA/QC Requirements
- c. Surveillance History and Records
- 5. Reporting of Failures
- a. Technical Specification Requirements
- b. Reporting of March 1 and 8,1983 Surveillance Results
- 6. Nonconformance Reports (NCRs)
- 7. Compliance With IE Bulletins and Circulars
- a. Initial Reviews
- b. Follow-up Procedures
- 8. Post-Trip / Restart Reviews C. Evaluation of Capability to Mitigate ATWS
- 1. Procedures for Mitigating ATWS
- 2. Control Room Layout / Design
- 3. Operator Training / Knowledge D. Findings / Conclusions Regarding RTB Failure V. PROPOSED CORRECTIVE ACTIONS ;
O A. Short-Term Corrective Action Leading to Restart
- 1. Control of Hardware, Vendor Information, Vendor Personnel
- 2. Maintenance Procedures and QA/QC Requirements
- 3. Surveillance Procedures
- 4. Technical Specification
- 5. Operator Training B. Long-Term Corrective Actions Subsequent to Restart VI. CONCLUSIONS l
l O
ii l .
4/15/83
/G I. EXECUTIVE SlHMARY As a result of failure of reactor trip circuit breakers (RTBs) to function at Salem 1 (IE Bulletin 83-01), Southern California Edison Company (SCE) perfomed its 18 month surveillance test at San Onofre Units 2 and 3, independently testing undervoltage (UV) and shunt trip functions of the RTB. This was done even though the RTBs are of a different design than Salem and were not required by the bulletin to be tested. The surveillance test was performed in early March 1983 when San Onofre Units 2 and 3 were both shut down and in different stages of their respective startup test programs. Four of the total of 18 RTBs tested failed to trip fosivwing actuation of their UY devices. All 18 tripped following actuation of the shunt devices.
SCE msponded promptly to the failure of the UV devices to trip the RTB by conducting a comprehensive investigation which is discussed in this report. The purposes of this investigation, and resulting findings and corrective action, are to ensure that the UY devices operate reliably and to ensure that the SCE startup and operating programs include any lessons learned from the failure of the UV devices to operate reliably. However, it is important that the following facts be considered in reviewing the experience with RTB UV devices at San Onofre Units 2 and 3:
o The UV device is one of two diverse methods used to trip the RTBs.
Unlike other designs, the UV device is not required to function for m the reactor protection system to complete its design basis protective action at San Onofre Units 2 and 3. Reactor trip is initiated by
[v) opening the RTBs automatically or manually, and both signals actuate both UY (de-energize to operate) and shunt (energize to operate) trip devices. During our investigation the shunt device always functioned to trip the RTB, and, therefore, the RTBs at San Onofre Units 2 and 3 have perfomed so as to meet their design basis function.
o Failure of UV devices to function, and work performed in response to these failures, have occurred during startup testing at San Onofre Units 2 and 3. As a result, the initial failures in March and July 1982 were interpreted as problems with setup and adjustment which are typical of startup testing. The failures during surveillance testing in March 1983 were recognized by the operating phase surveillance program as symptomatic of other potential problems and as requiring in-depth investigation.
The comprehensive SCE investigation discussed in this report has identified factors contributing directly to the failure of the UV device. The short term corrective actions required to obtain satisfactory bmaker perfomance have been identified and implemented.
The UV device has a much smaller force margin to trip the RTB than has the shunt device. Nevertheless, with upgraded maintenance and surveillance testing the UV device will function reliably. The shunt device reliability for tripping the RTB has always been very high in the
() San Onofre Units 2 and 3 design, and operation of this device alone fully V satisfies the reactor protection system design basis.
1 -1
4/15/83 II. SCOPE AND PURPOSE This report provides a summary of the comprehensive review perfomed by the Southern California Edison Company (SCE) subsequent to discovery of the reactor trip breaker undervoltage trip mechanism operability problem on San Onofre Units 2 and 3. The purpose of the report is to describe the cormctive actions that have been implemented as a result of that review.
Following brief discussions of the reactor protection system (RPS), the reactor trip bmakers (RTBs), and surveillance tests identifying the operability problem, the point is made that the UV trip device is not required to function in order for the reactor protection system to perfonn and complete its design basis protective action. The report then describes SCE's investigative program, including the specific breaker tests that were performed on the test bench (NRC witnessed) and subsequent testing conducted by SCE at its Electrical Test Laboratory.
Discussions are provided summarizing SCE's evaluation of administrative procedures, including; control of vendor data, control of hardware configuration, maintenance and surveillance programs, reporting requirements, nonconfomance reports and compliance with IE Bulletins and Circulars. The report also discusses evaluation of post-trip and restart reviews as well as capabilities to mitigate the consequences of Anticipated Transients Without Scram (ATWS).
x Based on the RTB investigative test results, SCE has identified enhanced maintenance and surveillance procedures for the RTB. Based on the evaluation of administrative procedures, SCE has also identified both short- and long-term corrective actions which it believes are appropriate to strengthen its administrative system.
The report concludes that with the implementation of the enhanced procedures for maintaining the RTBs, San Onofre Units 2 and 3 can safely resume operation and startup testing.
O II-l
4/15/83
/"s III. INTRODUCTORY AND BACKGROUND INFORMATION NRC IE Bulletin No. R 31, Failure of Reactor Trip Breakers (RTBs)
(Westinghouse DB-50) to Open On Automatic Trip Signal, issued on February 25, 1983, discussed the February 25, 1983 failure of the DB-50 RTBs to open automatically upon receipt of a valid trip signal at Salem Unit 1. Tne reactor was manually tripped from the control room about 30 seconds later, and the event was successfully teminated without core damage.
The Salem Unit 1 reactor protection system is designed to automatically open the RTBs using only the undervoltage trip device; manual actuation of the RTB uses both the UV and shunt trip devices. The failure of the RTBs to trip automatically was attributed to sticking of the undervoltage trip device.
In contrast with Salem Unit 1, the San Onofre Units 2 and 3 RPS provides both automatic and manual signals to actuate both the UV and shunt trip devices of each of the RTBs and the design configuration utilizes eight channelized RTBs. This section of the report discusses the design criteria and function of the RPS, the design and function of the RTB as part of the overall reactor protection action, and the role of the undervoltage trip device as part of the RTB.
Even though IE Bulletin No. 83-01 did not require testing of the RTBs at n San Onofre Units 2 and 3, the UV trip devices on the RTBs at San Onofre Units 2 and 3 were independently tested on March 8 and 1,1983,
('] respectively. The results of these tests are discussed in Section III.D of this report.
- 1. Design Criteria The design bases for the reactor protection system are presented in FSAR Section 7.2.1.2 and are summarized as follows:
The RPS is designed to ensure adequate protection of the fuel, fuel cladding, and RCS pressure boundary during anticipated operational occurrences. In addition, the system is designed to assist the ESFAS in limiting the consequences of accident conditions.
- a. The system is designed in compliance with the applicable NRC General Design Criteria of 10 CFR 50 Appendix A, IEEE 279-1971, and IEEE 338-1971, and is consistent with the recommendations of Regulatory Guide 1.53 and Regulatory Guide 1.22.
- b. The system is designed to alert the operator when any monitored plant parameter is approaching a condition that would initiate protective action.
f) c. The system is designed so that spurious protective action will v not be initiated during nomal operation of the plant.
III-l
4/15/83
- d. Four measurement channels are provided for each plant parameter monitored by the protection system, with the exception of the 4
control rod positions.
! e. The four measurement channels are independent and isolated from each other utilizing: separate sensors monitoring the channel parameters; separate wire trays or conduits for the channel i interconnecting cabling; separate channels for the RPS cabinet mounted signal processing equipment; and separate instrument ac j power buses backed up by separate batteries for each channel.
- f. The four measurement channels provide trip signals to six 4 independent logic matrices, arranged to effect a two-out-of-four ;
coincidence logic, each having outputs to four independent trip paths for each actuation signal.
- g. When one of the four measurement channels is taken out of service, the protection system logic can be changed to a two-out-of-three coincidence logic for actuation of plant
- protective action.
- h. After initiation, manual reset of the actuation output signal is possible following the clearing of its input signals.
- 1. System functions requiring operator attention or action during routine plant operations are displayed and/or controlled at the operators main control board.
- j. Annunciation is provided at the main control board of all operations at the RPS cabinet that could affect the function of
- the system.
- 2. Design Description Figure III. A.2-1 is a simplified functional diagram of the reactor
)rotection system. As shown the RPS is a four channel system and can
)e divided into several areas defined as: -measurement channels, bistables, logic matrices, logic matrix relays, trip paths and trip circuit breakers. FSAR Section 7.2 provides a detailed discussion of .
each of these areas. The following is a summary description of the RPS operation.
Measurement channels consist of sensors and signal conditioning equipment whose purpose is to convert the parameters being measured (pressure, temperature, etc.) into signals usable to the RPS bistables or calculators. These signals are provided in the fom of analog voltages.
Signals from the measurement channels are sent to voltage comparator circuits (bistables) where the input signal is compared to predetermined set points. Whenever the measurement channel signal reaches the set point, the bistable output will de-energize O associated bistable relays.
III-2
4/15/83 INPUTS FROM NSSS CH 1 2 3 4 5.6 7 8 9 -N 12 3 4 5 6 7 8 9-N 1 2 3 4 5 6 7 8 9 -N 1 2 3 4 5 6 7 8 9 -N
.m
} BISTABLES l CHANNELS A l CHANNELS B l CHANNELS C l l CHANNELS D v . . . .
. - + , f -
- + . + + +
A RICES l A B LOGIC l) l A4 LOGIC l) l A-D LOGIC h ! B4 LOGIC h l,B-D LOGIC h l,C-D LOGIC h
, , AB4 , , AC4 _ , AD4 , , BC4 , ,
804 _ ,
CD4 A AB1 AB2 AB3 AC1 AC2 AC3 AD1 02 AD3 BC1 BC2 BC3 BD1 BD2 803 CD1 CO2 CD3 TO 120 Vac TO 120 Vac TO 120 Vac TO 120 Vac VITA L INSTR UMENT VITAL INSTR UMENT VITA L INSTR UMENT VITAL INSTRUMENT BUS No.1 A g BUS No.18 BUS No.1C BUS No.1D AB1 AB2 AB3 g g AB4 AC1 Z ZAC2 AC3 1 1 AC4 AD1 % ZAD2 AD3 2 1 AD4 BC1 Z ZBC2 BC3 Z Z BC4 TRIP PATHS 1 B01 1802 803 1 Z BD4 TRIP CD1 1 CD2 CD3 1 1 CD4 CIRCUIT BREAKER CONTROL R E LA YS K1l ll lK2 K3l ll lK4 (INITI ATION RELAYS) 480 Vac 3d 480 Vac-3d BUS No.1 BUS No.2 N o.1 d) MAIN BREAKERS MOTOR CIRCUlT No.2[)
GENERATORS No.1 N o.2 240 Vac-3/
MSYNCHRONIZER )
+125 Vdc BUS No.2] _
+125 Vdc BUS No.3
} 1E hh BUS TIE 11" '
\ MANUAL --
- K2 K3 K3 TR IP N o.1 p.- _. _-
-.;-___ _2- _ - _ _ _ - _ - _ . - --
-g _g.g
/uv-
~ DT-- F) o ~
U T o 3
.,. , g-y-- U
--/
+125 Vac BUS No.1
. ) .
_L- L.- 44 p +125 Vdc BUS No. 4 MANUAL TRIP No,2 h- .
b K1 TRIP CIRCUlT BREAKERS q- -- - -
A-
-- -.-_---i -
- - .Qo -
-/_ - ,1) ")s uv/--/ T. 0 JU_V IS I
8 hM- - .
U,V
-[ST
> CEDN1 POWER
'E
. S PPL Y 1 P E S PP b h h h h ' CONTROL CEDM COILS s CEOM COILS CEDM COILS ,
' ~h lLS ELEMENT DRIVE
'T (b Figure Ill.A.2-1 SIMPLIFIED FUNCTIONAL DIAGRAM OF THE REACTOR PROTECTION SYSTEM
4/15/83 Tripping of a bistable results in a channel trip which is (p) characterized by the de-energization of three bistable trip relays.
Contacts from the bistable relays of the same parameter in the four protective channels am arranged into six logic AND gates, designated AB, AC, AD, BC, BD, and CD, which represent all possible two-out-of-four combinations. To form an AND gate circuit, the bistable trip relay contacts of two like protective measurement channels are connected in parallel (e.g., one from A and one from B). This process is continued until all combinations have been fomed.
Since there is more than one parameter than can initiate a reactor trip, the parallel pairs of bistable trip relay contacts for each monitored parameter are connected in series (Logit OR) to form six logic matrices. The six matrices are designated AB, AC, AD, BC, BD, and CD.
This logic requires two or more bistables monitoring the same parameter to be in a tripped condition before a reactor trip can be generated.
Each logic matrix is connected in series with a set of four logic matrix relays. Each logic matrix relay is associated with a trip path. The trip path is made up of six series contacts (cne from each logic matrix). The contacts are also in series with a trip circuit breaker control relay (initiation relay).
For each actuation signal, the above logic causes the de-energizing of the four trip path initiation relays whenever any one of the logic matrices is de-energized.
The trip path initiation relays then transmit trip signals to the RTBs. The design criteria and function of the RTB using these actuation signals is described in the next section.
B. Reactor Trip Breakers 1
- 1. Design Criteria The reactor trip switchgear cabinet assembly, including the reactor trip breakers, is specified in. San Onofre Nuclear Generating Station (SONGS) 2 and 3 FSAR Table 3.2-1 as:
Quality Class 1 (safety related)
Seismic Category I (design basis earthquake)
Electrical Class 1E It should be noted that the seismic qualification test plan does not require the breaker closing cin:uit to function during a DBE; however, it must not interfere in any way with the trip function of the breaker.
O III-3 l
4/15/83 7m The reactor trip signals are generated in the reactor protection
( system through the six logic matrices as described in
') Section III.A.2. The reactor trip signals are in turn transmitted to both the undervoltage and shunt trip devices on each RTB. Each of the four trip path initiation relays (K relays) sends an actuation signal to both the UY and shunt trip devices on a pair of RTBs. The pair of RTBs receive de power from the same vital bus that provides ac power for their associated initiation relay. This can be seen schematically in Figure III. A.2-1 where the K1 relay provides the actuation signal to RTB numbers 1 and 5.
The nractor trip breaker design is such that failure of either the undervoltage trip device or the shunt trip device does not eliminate the safety function of the reactor trip breaker. The trip circuit breakers will complete their protective action of interrupting power to the control rods using either the UY or shunt trip devices.
The RTB undervoltage device and the shunt trip device complement each other in that the undervoltage device trips the breaker upon loss of control voltage while the shunt trip device trips the breaker upon application of control voltage (see Figure III.B.1-1).
- 2. Design Description The details of the breaker latching mechanism and the function of the tripping devices are shown pictorially in the following sketches:
V)
?
- a. Breaker Assembly, Trip Bar and Latch Figures III.B.2-1 through 8 provide simplified representations of the AK2-25 latch mechanism operation.
" tripped" and "not tripped" positions. The device requires 125 Vdc and is energized to trip by actuating the tripper bar in a counterclockwise direction.
- c. Undervoltage Coil Assembly Figure III.B.2-10 depicts the AK2-25 undervoltage trip device in both the " tripped" and "not tripped" positions. This device requires 125 Vdc and is normally energized. Upon a loss of voltage the device actuates to rotate the tripper bar in a counterclockwise direction.
- d. Existing Design Modifications The RTBs in use at SONGS Units 2 and 3 have one additional modification to the foregoing design description. Combustion Engineering (CE) recommended a design change which SCE j iraplemented to place a diode in parallel with the UY coil to III-4
s \
)
na '
4/15/83 30 OTO ee. ..e . ..e es s e ee ne s e ... e.......
n Q zy .u u A V ................................
..... e - ..._. _ . ..
to 3 L S i- * ***a****** *a OO OTB "*"~M*'**
o S2 ."e
- '..e e #.
g il5T
-,,' .M yg y ' * * ' * * ' * " "
- f.L ! RES '"& g.g RE.S *g gpjggg, un L oi 2000 A ;f 7 . W it g T rip m..,.,. f am 1
- ' -v 21.32 2L3 % ' -IUl 11S V D-C .N a p oTB OTS ac 777,l,"2 3s ma
-* st M Mil-1 MT I-l
[a] ='=k V-O L '
- {p3 ---
..To S HT. r.
,)
ss
( .
4' 2L w< UNIT 2 PANE L
)e$ BON h S2 078 3'44 UNIT 3 PANEL.
' BREAKER. C'L OSINb u .
- APPROVED r ,-Tzs a rr+
% to Y
P
)
REFERENCE OTB oTB OTB OTS "a ,C,.2 0-02C M- DESIGN b M. -
~!
OTS oTS oTB c18
'$ 10J CUTG01HG TERMINAL St.DCW sts a a.3 3 /3 L33
^ " ^
f f f JK
- PL AN T' CcM POWR 15,2_. T_ e51, s2 .L 51 . ,, 31 1, s1 ,51 .L st L AIR. CIRCulT BREAnta.t.cov-roooA.
A Tb .' LEMERGENCY TRIP PUSHBUTToN AT
$e
$r b
,, sr
{b a
OTB OTB SWITCHGF.AE TD HAvE A CLtMt Pt.A6 Tic n oTB g OTB 9 0T8 0 078 g OTB 9 0T0 o cTB y "*' tutas y g,
ruc m.a, wp PeoTcCTwc Ccvtg wiTa novisious roe Lociuus waru w Rc scus.
vn. vv "* ## M#M 2Lia oT a o18 MZ -
PC MM tJUCLEAR SAFETY HELATED u M 73 e.py asor og Z
l
,_ _us. ( otsc.nenom f.57; *;,
- e,a,=,.,a 37, ,,v. o c.c.no,, r,
.,",,a...
... m. ..
man av: ,u- -~- 6 s -, 2 u.. . m m. v .; : . . . w - c ' c "' '* ELEMENTARY WimMG DIAGRAM s/ nc. .. c.
t.
.p./a . . g1 EYED 4 "I^cT> 52teAJ :7 ; e., eju . , ".n,ov au:.- ;
(-
- .A.:n.u iu n:it.:s ca a e,.i.n ki ., .
rr k nc.v ntot eno er an :ga :-- - -- REACTOR TCip CIRCUIT BREAWER gf, ,
'.g- , LF ALAD TO AeC D ad. F
--L .v It m ) DC ~f. .J a er y e -)
, - . .e g . .. ..,.2. . . . . , .
g =,, . 5,., c,cc . ;,. .:. css.s cio l - o.r; g - t ec; ..nn:rg s , _,
n..o
- n.: aw- .. . ,. .
nw.:.. .a a
, p ;- ----
i.>.,
V .I
. .. l
.ec.: ..: r-
.1. e
.y,
-- - - c c,.u : 9. w a .. a,.w.c.y
- - , . . _ . , ,, . . n,, ,u,,, mes.=oe:.
,,u.n ,.y c.-. J,a j- g
. .. .... ... .,.:=. .. ,.
... v. . .
/.,,_3, - . , ,.. .
_ _: er_:P,
- =
s tr< nco on- v<c i N' . - " * ~,"e"i e -- 3.. v".cn'~.."r'v"...w a m cecc
.na -~ l un e r w es. n er s n ,-, .
Figure Ill.B.1-1
4/15/83 Figura Ill.B.2-1 GE AK-2 25 CIRCUlT BREAKER UDPER STUD BREAKER BASE STATIONARY CONTACTS MOVABLE C0tlTACT 4
\ }
INSULATED LINKS
\
s \
\ \
-~\
STOP "A" m t. \ \
, @ 6\
_ /_ _ _
s _ L _i _.
OUTPUT CRANK \ \
i g
\ \' \
\
\\
CLOSING SPRIf1G _\ _,1_\
\
i %,
PIVOT
\ M '\'s 1
y h d' \> s l c d}i ] - IQ \ \ ~ ~i L *
, \ (TRIPLATCH P:V0T PIN
\\
\l I '
\-
STOP "B" (B'LOCKIfiG CAM) +
r- i LOWER STUD-'
i l I CLOSING SOLEN 0ID ARftATURE
- \
\
{g CLOSING SOLEN 0ID
! l THE CIRCUIT BREAKER f1ECHANIS?1 HAS BEEN SI:1PLIFIED CONSIDERABLY i 1 -
IN ORDER TO CLARIFY ITS OPERATION.
i "A" RE:RESEITS A PORTI0fl 0F THE !!ECHAtlISt THAT PREVE!iTS I
i f STOP THE LI!1K FR0!1!!OVIrlG BEYOTID Tl;E POINT I:lDICATCD.
I f STOP "B" IS A BLOCKING CA.'1 THAT PREVENTS THE tiECHAT11SI1 1 i FRO:: l:0VIflG AND CLOSING THE COlTACTS UNTIL THE CLOSING SPRING IS CHARGED.
O { THE TRIP LATCH LOCKS THE !!ECHA:lI5!1 IN THE CLOSED CONTACT POSITION U!!TIL RELEASED BY THE TRIP PUSHBUTT0ft, SHU;1T TRIP DEVICE, OR THE UNDERVOLTAGE TRIPPING DEVICE. IT IS SHO'lM ON THIS SKETCH AS A O!CTORIAL REPRESE'lTATION OF ITS CUNCTION.
4/15/83 Figure Ill.B.2-2 GE AK-2 25 CIRCUIT BREAKER Condition 1 UPPER STUD 110VABLE BREAKER BASE x
C0rlTACT STATI0flARY C0tiTACTS t t ,
INSULATED LINKS STOP "A" \ _1 \
' UD
,[ .
ML\
s-CLOSING - [ , OUTPUT CRANK \ \
O SOLEN 0ID / \ \
( / ARl%TURE ,
\ \\.
\
4 L
=r
. w ,I,eT W\. w 7 p}
s -
w PIVOT PIN" TRIP LATCH k \ LOMER STUD d-
, ) -BLOCKING CAft
[t i l CLOSING C0f4DITION 1 l SOLEN 0ID l
g THE CIRCUIT IS OPEN AND THE OPERATING I SEOUENCE IS ABOUT TO START.
g 1 I g ) i THE SPRING LOAD KEEDS THE CONTACTS OPEN.
~t I l
l p /W W I l
l l
4/15/83 Figure Ill.B.2-3 m
i GE AK 2-25 CIRCUlT BREAKER Condition 2 i
UPPER STUD BREAKER BASE MOVABLE STATIONARY CONTACT CONTACTS CLOSI!G SOLEN 0ID ARitATURE 1
s t
CLOSli;G SPRING INSULATED 4r j LINKS i STOP "A"
-]-
)
\
1 \ s
\ ;\. dv .
i I #
\ \
- OUTPUT CRANK
', \
~ ~
I \
PIVOT 7
, ,, - . g r.,[~? ~Y' >
l TRIP LATCH PIVOT PIN i,
/
LOWER STUD i1 - BLOCKING CAM l
l i sC'_0 SING I
\
kl l )*
SOLENOID THE CLOSING SOLEN 0ID IS ENERGIZED AND THE ARitATURE EXTEtlDS. ENERGY IS STORED IN THE i . CLOSING SPRING BUT THE 3LOCKI:!G CAf1 PREVENTS I THE OUTFUT CRAtis CRCit MOVI:G.
, g l
\
\\
\\
- O
'c<
\ ~
4/15/83 Figure Ill.B.2-4 GE AK.2-25 CIRCUlT BREAKER Condition 3 UPPER STUD -
BREAKER BASE-STATI0fiARY9 C0ilTACTS t10VABLE CONTACT ,h
-CLOSIt!G SOLEN 0ID ...,' l 't 'l ,
arf 1ATURE (*** 's <
\ g
\
s 8
I
[
) ' \
r i CLOSING SPRING INSULATED LINK s
\ ', l
{
N STOP "A"
.___________--__y
--lr -
,_ ../ .m.,. q_ k.
A_
i p j' ,, , ' '. '. ; W '
) .:.. ". :
'i..l....~..:.~.:. ................-..--.h..]
~
,! \, ',
s ',
/.. ,
/ ,\
', 's (
) g', . ,' f ' 'g .
, - 0UTPUT CRANK ', ,',
' y'
/,
,/ \, \, k -
f[
, 5 '
i ,
, ,/ , ___ : CRANK PIVOT ,
\ t i
i l %.\.- ,
s s
/ ,,
,1 l
1/ /
-d ,,' d' PIVOT PI:1 L RIPT LATCH BLOCKIt!G LO'.lER STUD CA?1 4'
, { _
$ CLOSIrlG CONDITION 3 k 'j i SOLEN 0ID THE BLOCKIt!G CAi1 RELEASES THE OUTPUT CRAflK AtlD fg.l i THE SPRING TENSION CAUSES THE CRANK TO ROTATE l ABOUT ITS PIVOT AND IT DRIVES THE MOVABLE CONTACTS
{ AGAltlST THE STATIONARY C0rlTACTS.
~ '
THE TRIP LATC'! HOLDS THE "ECHANIS!1 IN POSITION AND THE CIRCUIT DET'XEN THE UPPER AND LOWER STUDS O IS C0f1PLETED.
f
- . . . . . - . - - - , - - , , . ~ - _ . . _ , . , , ,
4/15/83 Figure Ill.B.2-5 O)
(, GE AK-2-25 CIRCUIT BREAKER Condition 4 UPPER STUD BREAXER BASE STATIONARY -
CONTACTS IV fiOVABLE CLCSI:.'G SOLENCID
- r.. .n . . R:.
CONTACT \ 3 g g ,
INSULATED f-CLOSIt:G LINKS l SPRING ,
~ -- -
- 0UTPUT
. - CRANK
\\ .
e , --- P IVOT - -
I l I
PIVOT PIN
-TRIP LATCH k LO11ER STUD
\ \ \ ., q. .
) i i _
E ', t
\
CONDITION 4 s \
i THE CLOSING SOLEtt0ID IS DE-ENERGIZED AND g
THE ARitATURE REI'.AI:!S I:1 ITS POSITION DUE TO THE OVER-CETITEP TOGGLE ACTION.
t I
\
l I
{_ ! CLOS!!;G SOLEN 0ID 1
4/15/83 Figure Ill.B.2 6 O
Q GE AK.2 25 CIRCUlT BREAKER Condition 5 UPPER STUD BREAKER BASE STATIONARY CONTACTS tiOVABLE CONTACT ,$.
, . 1, ,
I
! r I i I
i l r J~.{
! INSULATED LINKS i i 8
i i
s STOP "A" l ,
k ..__.._,.........____.....u_.__._.__r--
l p', ,' , . 7 ' ' '
q' '
i
,'yi:,';
/,.4
..\..~...-..._.--_....._.__._._..,...
O i #2..
,.; i ': !
, i .
i, .,
, i ,' OUTPUT CRAtlK ,
8
' ^ '
i ,- / -
. i .
' ^< l l i i ,
' s' i '. CLOSING , I
' :. l 3 I i jSPRING ,[
' s PIVOT )*2 [
l i - . _y . . .
- l~~ s,9 . I l . PIVOT PIN l
. LOSING l TRIP LATCH f l ;. LOWER STUDJ g
N '
BLOCKING CAM 4, 1 1 I _
Fg l 1 C0flDITION 5 g I g ) THE TRIP LATCH IS TRIPPED BY THE TRIP PUSHBUTTON, THE SHUNT TRIP DEVICE, OR THE UNDERVOLTAGE TRIPPING DEVICE. THE OVER-CENTER I {
g TOGGLE ACTION COLLAPSES AFID THE MECHANISM IS NOU FREE TO MOVE.
' i I THE SPRING TENSION CAUSES THE OUTPUT CRANK TO ROTATE ABOUT ITS I i I i _ PIVOT, PULLING THE INSULATED LINKS AND CAUSING THE CONTACTS TO j i OPEN.
I THE SYSTEli UILL RE 1AIN IN THIS DE-ENERGIZED POSITION UNTIL THE CLOSIN f} .
TRIPPING CONDITION IS RECTIFIED AND THE CLOSING SOLEN 0ID IS SOLENOID ENERGIZED.
4/15/83 Figure Ill.B.2-7 GE AK-2-25 CIRCUlT BREAKER TRIP LATCH : ,/
1 ROLLER PIVOT g
TRIP SHAFT u TRIP LATCH TRIP LATCH -
ROLLER i
THE TRIP LATCH AND TRIP LATCH ROLLER ARE IN THIS POSITION WHEN THE CLOSING SOLEN 0ID IS NOT ENERGIZED AND THE TRIP LATCH IS IN THE RESET POSITION. THERE IS A GAP BETWEEN THE ROLLER AND THE LATCH.
WHEN THE CLOSING SOLEN 0ID IS ENERGIZED, THE TRIP LATCH ROLLER IN ITS HOLDER TURNS COUNTER CLQCKHISE, CLOSES THE GAP, AND IS RESTRAINED BY THE TRIP LATCH.
1 THE TRIP LATCH AND ROLLER REMAIN IN THIS POSITION UNTIL THE BREAKER IS TRIPPED.
...:\
'W'4 s
'Q .
i
.) );,t...
L, .,., - ...
WHEN ANY OF THE TRIPS ARE ACTIVATED, THE TR P SHAFT ROTATES IN A COUNTER CLOCKUISE DIRECTION AND THE TRIP ROLLER ROLLS ALONG THE CONTACTING SURFACE OF THE TRIP LATCH UNTIL THE TRIP LATCH SWINGS OUT OF THE WAY, LEAVING THE ROLLER AND ITS HOLDER TO ROTATE FREELY ABOUT ITS PIVOT. THIS CAUSES THE OVER-CENTER TOGGLE ACTION IN THE MECHANISM TO COLLAPSE AND THE CONTACTS OPEN.
~
O THE ROLLER RETURNS TO ITS ORIGINAL POSITION WHEN THE C0:1 TACTS OPEli AND THE MECHANISM IS DE-ENERGIZED. THE TRIP LATCH WILL REMAIN IN THE POSITIOlI INDICATED UNTIL THE CAUSE FOR TRIPPING'IS RECTIFIED AND THE TRIP LATCH IS RESET.
4/15/83 Figure Ill.B.24 V TRIP SHAFT SKETCH P1ANUAL TRIP PADDLE UNDERVOLTAGE TRIP PADDLE Ns SHUNT TRIP PADDLE TRIP SHAFT
% s, TRIP SHAFT BEARING
( TRIP LATCH p/
/
f
\
TRIP LATCH ROLLER i
l TRIP SHAFT SKETCH fl0TE: THIS SKETCH DOES NOT PORTRAY THE ACTUAL SHAFT OR THE LOCATION OF THE VARIOUS TRIPPING PADDLES WITH RESPECT TO EACH OTHER OR TO THE SUPPORTING BEARINGS. IT IS A PICTORIAL REPRESENTATION OF THE TRIP SHAFT AND ITS FUNCTI0ris.
i O
l
4/15/83 Figure Ill.B.2-9
{m s SHUNT TRIP DEVICE i
s _
cm.
'J s- D "
.- 'lh* [(_.
.e: en t
@2 1
) "
l' e
- fI 3 P: i * ! 'N'
.g.
e 8
fl0T TRIPPED O
W x
r -
, 3*"
L -,,
r 4- _,
U
, @l ;d
/y-mm.sw m
<. e-i ! h =,
.
- i l
ea
%s(..l...g ,
a rc TRIPPED
- 1. Nut 6. Screws 11. Armature Arm
- 2. Frame 7. Magnet 12. Trip Paddle t
p 3. Spring 8. Coil Clamp
- 13. Mechanism Frame Trip Shaft Clamp
\ 4. Rivet 9. 14.
- 5. Wei;ht l 10. Armature
4/15/83 Figure Ill.B.2-10 UNDERVOLTAGE TRIPPING DEVICE g],h 2 7
, M ,
[
- .L A ;6 i IS 0
'f M ::
! O y ,k O 1 13
- r
^
W l
[ N h f
- l!mMk .
9 i
~
II
- l l lh t i I ; I i 4 's
- 4, :o o ~
TRIPPED 20
' ~
I/y 1. Mounting Screw
- 2. rrame i
w is 3. Armature I 4. Spring
,f 5. Ehd7 Ring 2
7' 6. Adjusting Screw g%'t gY 7. L~ Mar Nut i . .l
'6
- 8. BusW l l b. / Jl 9. Clamp
? 2 10. Magnet f2'.
15
' ; 7 o M J is. 13. Rivet
- 1y
- 14. Adjusting Screw m :: 15. L" War Wire
" N J
(- :3
- 16. Mounting Nut
- 17. Meehn'um Frame r s 18." Trip Paddle Clamps e
- 19. Trip Paddle
/- l lQl 12 7 W h 20. Adjusting Screw m%
' :: 9
?
Il I
I 9 J O ^ ' !l> l -
~ v j ,
- O l
N 9 M0T TRIPPED
4/15/83 provide a discharge path for the collapsing field and prevent contact damage to the RPS "K" relay. This diode is shown in Figure III.B.1-1. It should be noted that the diode is not installed on the bmaker, but in the switchgear cabinet assembly, so that when the breaker is installed or in the " test" position, the diode is in the UV coil circuit; but when the breaker is mmoved from the cabinet, it is not. The requirement for simulating this diode when preventive maintenance and testing is perfomed is discussed under test results in Section IV.A of this report. This diode also results in a slower collapse time for the UV coil magnetic field, causing the UV trip device operating time to increase by about 30 msec. This change may also result in a slight loss of UV device tripping force by reducing the kinetic energy of the UV device amature. However, when the enhanced breaker maintenance is performed as described in this report, the overall RPS response time remains within acceptable limits.
C. RTB Initial Procurement and Testing
- 1. Procurement History The San Onofre Units 2 and 3 reactor trip switchgear (RTSG) systems were procured as a part of a general procurement package intended for several CE plants. A procurement history of this generic package and h)
L/
the documents that confim compliance with SCE's project specifications are described below. This documentation is for the RTSG that included the original AK2-25-2 RTBs, and is available for audit. Spare parts procurement would have to verify compliance with these specifications as well as certify complete functional equivalency.
- a. A Request for Quotation (RFQ) was issued by CE for a generic order of four (4) RTSG systems which invoked CE General Specification 00000-ICE-3008 Rev. 01 and CE general RTSG design drawi ngs.
- b. Unit Electric Control, Inc. (UEC) among other vendors, provided a response to CE Request for Quotation which was received during the first quarter 1975.
- c. CE perfomed a Bid Evaluation of the bidders that responded to the CE RFQ recommending Unit Electric Control, Inc. as the preferred vendor for the subject equipment.
- d. CE issued a Master Purchase Order to Unit Electric Control invoking CE General Specification 00000-ICE-3008 Rev. 01.
- e. A supplemental quotation request was issued by CE to Unit Electric Control, Inc. modifying the requirements for packaging, shipping, receiving, storage, and handling (i.e., the ANSI O N45.2.2-1972 requirements now imposed on the vendor).
III-5
I 4/15/83 s
- f. Unit Electric Control, Inc. issued a msponse to the supplemental request and implemented the requirements of same.
- g. Purchase Order Release No.1 against the Master Purchase Order for San Onofre Units 2 and 3 was issued by CE to UEC. This release order invoked CE Project Specification 1370-ICE-3008 Rev.
01 in addition to the general requirements and requirements for seismic / environmental qualification.
- h. Supplement 1 to the Master Purchase Order was issued by CE mquiring deletion of coil status indication lights (which were deleted to enhance breaker control circuit reliability) and changing the RTSG heaters from 120 Vac to 240 Vac.
- i. The following Request for Approval or Review (RAR), Technical Change Request (TCR), Deviation of Contract Request (DCR) submittals were made by Unit Electric Control against the SCE Unit 2 and 3 RTSG contracts.
- 1) RARs 1 through 16 were submitted by UEC during the time period of October 1975 to April 1976. These RAR submittals pertained to Inspection and Test Plans, Drawings, IMQP, Qualification Plan and Results, Tech Manual, and Function Test Procedures and Results. All RARs were approved by CE
,- for compliance with CE specifications.
- /3 U 2) TCRs 1 through 4 were submitted by UEC in October 1975. TCRs 1 through 3 were not approved by CE and the requested changes l
were not accomplished. TCR 4 was approved and allowed a j change in the CE specified painting requirements.
- 3) DCRs 1 through 3 were submitted by UEC during the time period l of October 1975 to February 1976. DCR 1 was disapproved by l CE and the requested change was not allowed. DCR 2 allowed the deletion of terminal block cover wire numbering. DCR 3 allowed UEC not to submit progress reports on RARs. DCRs 2 and 3 were approved by CE.
- j. A conditional Certificate of Equipment was issued in December 1975 which formed the basis for shipment of the RTSGs for both San Onofre Units 2 and 3. An unconditional Certificate of Equipment was subsequently issued in June 1976 to include CE review of the as-certified drawings, instruction manual, and test data,
- k. The following lists the as-shipped breaker configuration for San Onofre Units 2 and 3 by breaker serial number and location.
Unit 2 Breaker Serial No. Location p 256A4002-656-3 TCB-1 Q
l 256A4002-656-15 TCB-2 1
256A4002-656-24 TCB-3 III-6
4/15/83 Unit 2 Breaker Serial No. Location 256A4002-656-29 TCB-4 256A4002-656-1 TCB-5 256A4002-656-8 TCB-6 '
256A4002-656-26 TCB-7 256A4002-656-33 TCB-8 256M002-656-18 TCB-9 Unit 3 Breaker Serial No. Location 256A4002-656-5 TCB-1 256A4002-656-14 TCB-2 256A4002-656-25 TCB-3 256A4002-656-17 TCB-4 256A4002-656-2 TCB-5 256A4002-656-7 TCB-6 256A4002-656-34 TCB-7 256A4002-656-45 TCB-8 256A4002-656-12 TCB-9 The above are the original AK2-25-2 breakers, shipped with the RTSG.
- 2. Acceptance Tests - Vendor Reactor trip switchgear acceptance tests requimd and perfomed by O the RTSG vendor were as follows:
- a. As required by CE General Specification No. 00000-ICE-3008, Section 5.10.3, bench tests were perfomed by the vendor of the San Onofre RTSG prior to installing equipment into tha RTSG cabinet. This testing included:
- 1) Tests to check for defects in mechanical assembly
- 2) Tests to check for defects in electrical wiring
- 3) Checks for grounds, shorts or open circuits
- 4) Actuation checks to ensure:
a) Proper closing operation b) Proper tripping operation c) Proper status indication as defined by breaker position d) Proper functioning of auxiliary relay contacts The above testing was satisfactorily completed and documented,
- b. In addition to the individual equipment bench tests described in tne previous paragraph, the RTSG system underwent additional testing as required by CE General Specification No.
00000-ICE-3008, Section 5.10. This testing included:
- 1) Hi-potential testing O 2) Point-to-point continuity testing
- 3) System operational testing (this procedure is incorporated in the CE Instruction Manual for the reactor trip circuit bmaker switchgear for San Onofre Units 2 and 3).
III-7
4/15/83 (d
a c. Quality Assurance inspection of workmanship as required by Combustion Engineering Quality Control Specification (00000-WQC-11.1, Rev. C) was perfomed.
The above testing and inspection were satisfactorily completed, a Certificate of Equipment was issued by CE, and the RTSG was delivered to SCE.
The following summary of the RTSG equipment qualification data and references reflects testing and analyses perfomed prior to delivery of the RTSG to SCE. The follow-up seismic study by Wyle Laboratories-(Item f) confimed the adequacy of the previous seismic qualification.
- a. The RTSG was seismically qualified to the requirements of IEEE-344, Seismic Qualification of Safety Related Electric Equipment for Nuclear Power Generating Station,1971.
- b. The RTSG environmental test was documented in accordance with the requirements of IEEE-323, Trial Use Standard, General Guide for Qualifying Safety Related Electrical Equipment for Nuclear Power Generating Station,1971.
- c. San Onofre Units 2 and 3 reactor trip circuit breaker switchgear was seismically and environmentally tested / analyzed by Wyle Laboratories, Huntsville, Alabama.
/
b d. The results of the testing / analysis were approved by CE following the requirements of Combustion Engineering " Sellers Procedure for Submitting Approval Review (RAR)" WD-WI-6 Appendix.
- e. Test results were defined in CEN(94)-S Data Sheet No. 3 for the RTSG seismic test documentation and in CEN(95)-S Data Sheet No. 3 for the RTSG Environmental Test documentation as referenced in Section 3.10 and 3.11 of the SONGS FSAR for NSSS equipment.
- f. Additional analysis infomation was provided to SCE by Wyle Laboratories Analysis Report No. 26321 Section 6.11 as response to the NRC Seismic Qualification Review Team (SQRT) general concerns for San Onofre Units 2 and 3.
The RT8s are intended to be used in the reactor trip switchgear cabinets Which are located in a non-harsh radiation and temperature environment. Consequently, environmental testing includes only the upper limit of the normal room temperature and does not test or analyze for radiation effects. The RTB are in an area that is expected to be less than 104 rads total 40-year dose, so their components are considered qualified for this service. Room temperature is regulated by the plant HVAC. Any potential effect of temperature or radiation on maintainable parts (such as lubricant) will be detectable through the enhanced surveillance program described later in this report.
d III-8
4/15/83 3.1.::ceptance Tests - Startup I
As described in the San Onofre Units 2 and 3 FSAR, the startup test program includes mquirements for prerequisite and preoperational
' testing of equipment including the reactor trip switchgear. The .
4 preoperational testing is discussed in Section III.C.4 of this report. The prerequisite testing of reactor trip breakers was as follows: ,
- a. All type AK2-25-2 bmakers (18 total) were successfully tested with startup generic test GT-400-05. This was a generic air circuit breaker test, and contained the following elements:
- 1) Meggering of insulation resistance to >50 M ohms
- 2) Recording of X, Y, and trip coil resistances.
3)
Recording)of
(<100 Vdc minimum voltages to trip (<70 Vdc) and close
- 4) Verification of correct functional operation of auxiliary and i test switches.
! 5) Verification of correct mechanical adjustments of contact wipe and latch adjustment.
- 6) Visual inspection of are chutes, barriers, control switches and wiring, racking and locking mechanism, all switches and wiring, and hardware.
- 7) Closing solenoid resistance was recorded.
. Although not specifically required by GT-400-05, it was SCE's practice to record undervoltage device pickup and dropout voltages during this test.
- b. Equivalent prerequisite testing was perfomed on the four breakers which were procured as spares and subsequently installed as mplacements on Unit 3.
The fifth spare breaker is untested and has never been installed.
- 4. Startup and Surveillance Tests The acceptance tests discussed above have been performed prior to placing RTBs (including RTBs initially supplied as spares) into service. In addition, the following preoperational, surveillance, and maintenance tests were perfomed prior to the March 1 and March 8,1983 surveillance tests:
- a. Pmoperational Testing RTBs were tested as part of the plant protection system (PPS)
- preoperational test, PE-357-01, and the PPS response time test, PE-358-01, on both San Onofre Units 2 and 3. All RTBs installed at the time of the tests responded properly. These tests, however, trip the breakers on combined UV and shunt device
'O operation (as would occur during RPS protective action) and thus would not reveal UV problems.
III-9 1
-a,, , - , ,- -,eran-n.,n. -
,,--r, rw e ,-- - n e n g,m - ,, n w,n,nw,-.w,,. em,m- ,,,y, - - -,ns v. .,, ,, ,--- -,-cm,,,,, v-----,,- - ,
l 4/15/83 O 1) The PPS preoperational test is a comprehensive test of the entire system through RTB operation and Engineered Safety Features actuation (pump starts, etc.). The RTB test in 4
Section 8.10.2 of PE-357-01 is a functional trip test which looks for bmaker operation only, and does not time breaker opening or otherwise detail its operation. The Unit 2 preoperational test was perfomed on May 30, 1981, and the Unit 3 preoperational test on August 25, 1982. No RTB failures occurred.
- 2) The PPS response time test was performed on both Units 2 and 3 to meet the requirements of FSAR 14.2.12.72S. This test recorded electronic response times from the sensor to the RTBs or ESFAS subgroup relays, as appropriate. Mechanical response times of pumps and valves were measured in other r
tests and added on to these results to yield total ESFAS response times. In the case of RTBs, the time from the de-energization of the matrix relays to the opening of the 1 -RTBs was measund as a subcomponent of total RPS response time. The acceptance criterion was <120 ms. The longest time (last RTB to open) was recorded in each case; Unit 2 was tested in November 1981 with the longest time being 60 milliseconc's, and Unit 3 was tested in August 1982, l recording a maximum time of 100 milliseconds. No breaker failures occurred during these tests,
- b. Surveilla.nce Testing Surveillance testing of reactor trip breakers is perfomed in
. accordance with S023-II-1.1, " Plant Protection System - Channel Function Test," S023-II-3.1, -3.2, -3.3, -3.4, " Plant Protection i System Channel Response Time Tests," and S023-II-11.161, " Reactor i Breaker Undervoltage and Shunt Device Circuit Test." These tests are perfomed on the RTBs installed in the switchgear cabinets, and on replacement RTBs as discussed below:
- 1) The PPS Channel Functional Test is a comprehensive test of the protection system from sensor output through RTB
, operation and Engineered Safety Feature actuation similar to the PPS preoperational test discussed previously in this 4 mport. This test is perfomed at least once per month in accordance with the Technical Specifications, and for the RTBs, is a functional trip test using combined undervoltage and shunt trip operation. As operation of the shunt or UY device satisfies protection system design criteria, this test
, verifies RTB function in accordance with design mquirements. The RTBs have never failed to trip during this test.
- 2) The PPS Channel Response Time Tests measure electronic response times from the sensor to the RTBs or ESFAS subgroup O relays, as appropriate. Mechanical response times of pumps and valves were measumd in other tests and added on to these
. III-10
4/15/83 results to yield total ESFAS response times. These four test procedures (one per Channel) are essentially identical to the preoperational response time test discussed previously in this report. As stated previously, no RTB failures were experienced during response time testing.
- 3) The RTB Undervoltage and Shunt Device Circuit Test separately tests the UV and shunt device trip functions. This test is required at least once every 18 months and following maintenance or adjustment of the RTBs, in accordance with the Technical Specifications. On Unit 2, this test was initially perfomed monthly and after any repair or replacement of RTBs. Following correction of UV trip failures on four Unit 2 RTBs in early 1982 and successful completion of two more monthly tests on each RTB, SCE reduced the frequency of this test to that specified by CE and the Technical Specifications. Two additional Unit 2 test failures were observed in July 1982 following UV coil adjustment. An RTB functional test was performed to verify function in accordance with the criteria for plant protection system design. The separate UV and shunt device surveillance was not completed before resuming the rod drop wasts, and LER 82-176 was submitted to document this occurrence. The surveillance test was successfully completed in July 1982 (Unit 2) and in August 1982 (Unit 3). An additional O surveillance test was perfomed on the Unit 3 RTB in October 1982. There were no further UV device. surveillance tests until the March 1 and 8,1983 surveillance tests discussed later in this report.
- c. Maintenance Testing Maintenance, including preventive maintenance (PM), corrective maintenance, and testing as part of maintenance is performed in accordance with work orders. The maintenance relating to reactor trip breakers is discussed below:
- 1) The preventive maintenance program was developed from vendor, NSSS, and Engineer-Constructor (CE and Bechtel) experience and recommendations. WEG-025 is a generic overhaul procedure for 600 volt and below air circuit breakers. Work in accordance with this procedure has not been performed on the RTBs. WES-008 is a specific maintenance procedure for the RTBs developed in response to IE Bulletin 79-09. This procedure addresses overall lubrication and other general maintenance items as well as specific steps pertaining to UV design in accordance with the guidance in IE Bulletin 79-09.
WES-008 was performed on the RTBs originally installed in Unit 2 in March through May 1981 but has not been perfomed on the RTBs originally installed in Unit 3 or on RTBs supplied as spares. Preventative maintenance planning had O scheduled this to be done at the first refueling on the basis III-11
- - ._ _ - .- -_ _ . = _
t 4/15/83 that the RTBs, as delivered, would not require PM before that time. Maintenance procedure S023-I-4.36 was developed from WES-008 for use at first refueling but has been cancelled
- because, through an administrative error, S023-I-4.36 was initially issued with key infomation from WES-008 deleted.
1 This infomation has been added to a new procedum (S023-I-4.66) which also incorporates the reconnendations in Sections IV.D.2 and V.A of this report. Although there is no Technical Specification requirement to perform a baseline PM for all equipment, it is SCE's practice to do this and it was
'1 done for the Unit 2 RTBs. For the Unit 3 and spare RTBs, however, SCE relied on the delivered condition of the RTBs .
and did not specify PM until the first refueling. l
- 2) Corrective maintenance on the RTBs was perfomed in response to problems documented in Nonconformance Reports (NCRs); a work order was then written against the NCR to correct the problem. For the Unit 2 RTB undervoltage trip failures in 4
early 1982, the NCR was dispositioned to correct the problem
- j. per vendor direction, based on a perception that failure of the UV device to function reliably made SCE procedure WES-008 suspect (it had been used earlier for PM on these RTBs) and that vendor assistance was needed to ensure that no unusual problems existed with these safety related component.s. The work order specified that the undervoltage O coil pickup voltages be reset to vendor technical manual (5023-944-352) criteria by SCE electricians under vendor (General Electric) supervision. For the July 1982 RTB adjustment to facilitate breaker closing, SCE electricians reset the undervoltage coil adjustments on all Unit 2 RTBs to vendor technical manual criteria without vendor assistance.
When two RTBs subsequently failed the July 1982 undervoltage surveillance tests, CE recomended that a vendor service
[ representative check the UV coil adjustments for all Unit 2 RTBs. The UV coil adjustments were then reset (some outside the vendor technical manual limits) by the vendor ;
representative to obtain suitable RTB (close and UV trip) operation, and the NCR was dispositioned to accept these settings as-is, based on the perception that the settings detemined by the vendor representative were the best technical infomation available. For the Unit 3 and spare breakers, a work order had been written to perfom both WEG-025 and WES-008 maintenance procedures, but was then cancelled in favor of work orders to adjust per vendor technical direction consistent with Unit 2. Following these activities, the surveillance test was repeated on all RTBs for Units 2 and 3 with satisfactory results.
Further discussion of the above maintenance process is provided later in this report.
O l
i l III-12
- 4/15/83 D. March 1 and 8,1983 RTB Surveillance Tests
, 1. Description of Tests The NRC issued IE Bulletin No. 83-01 " Failure of Reactor Trip Breakers (Westinghouse DB-50) To Open On Automatic Trip Signal" on February 25, 1983.
Even though IE Bulletin No. 83-01 did not require testing of the General Electric RTBs installed at San Onofre Units 2 and 3, surveillance testing was perfomed in accordance with station procedure S023-II-11.161 " Reactor Breaker Undervoltage and Shunt Device Circuit Test." This procedure tests the UV device and shunt trip device independently as described in Section III.C.4 of this report.
i
- 2. Test Results Testing was perfomed on all Unit 3 RTBs on March 1,1983. One breaker, RTB 4, type AK2-25-2 serial No. 256A4002-656-17, failed to trip on its undervoltage test. It performed properly on shunt trip.
NCR-3-243 was written to document the nonconformance. Since this was viewed as an isolated failure and the CEDM cabinets were not energized, the failure was not considered to be reportable pursuant to the Technical Specifications at this time. Testing on Unit 2 was delayed by the presence of a failed power supply in the PPS. After O the sower supply was repaired, testing of Unit 2 RTBs commenced on Marc 1 8, 1983. This testing revealed three RTBs (all type AK2-25-2) which did not trip on undervoltage:
RTB Serial No.
l 1 256A4002-656-3 4 256A4002-656-24 6 256A4002-656-8 All breakers successfully passed their shunt trip test. NCR-2-163
!. was written to document these failures and the NRC was notified of the surveillance failures.
1 i
O III-13
. . . . _ ~ _ _ _ _ . . _ . . _ . _ _ _ _ . _ . . . . _ _ _ _ _ _ _ _ _ _ _ _ _ . _ _ _ _ . _ _ _ _ _ . . _ - . _ _ _ _ _ . _ _ _
4/15/83 l
IV. INVESTIGATIVE PROGRAMS As a result of the Mamh 1 and 8,1983 surveillance failures of the GE Type AK2-25 RTB at San Onofre Units 2 and 3, SCE conducted an investigation to detemine the cause and ramifications associated with the surveillance failures. This section of the report discusses SCE's efforts in the following specific areas to determine the root cause of the problem and to identify specific corrective actions and follow-up
- activities for resolution of RTB concerns and related programmatic deficiencies
A. RTB Investigative Tests
- l. Description of Tests Following the reactor trip breaker surveillance tests of March 1 and 8,1983 in nesponse to IE Bulletin 83-01, SCE initiated further
!- investigation and testing of San Onofre Units 2 and 3 reactor trip cimuit breakers. This further investigation and testing was perfomed in three parts: a) initial investigative tests Mamh 12 to 17, 1983 at San Onofre; b) in-depth testing March 26 to April 1,.1983 at the SCE Electrical Test Laboratory in Alhambra; c) test results of j nine RTBs at San Onofre Unit 2 obtained during baseline preventative maintenance; and d) independent testing by Franklin Reseamh Center I
(requested by NRC). A discussion of investigative test parts a, b, i
and c follows; part d will be provided directly to the NRC by Franklin Research Center.
I a. Initial Investigative Tests (March 12 to 17,1983) l.
- 1) Following an introductory meeting with representatives of NRC
, and Franklin Research Center, SCE initiated an investigative l plan for detemining problems with RTBs. A procedurc was i developed and made available for review and comments by NRC l and .eranklin Reseamh Center. Comments were provided by the l above organizations for SCE consideration and were factored 1
into the final investigative procedure.
- 2) A General Electric Company field service engineer who had previously provided vendor assistance on RTBs was called in.
In addition, Combustion Engineering provided contact with the General Electric Company to request the assistance of a factory expert in the investigation of the RTBs. The General Electric Company provided Mr. Max B. Fornwalt, Senior Project l
Engineer, who arrived on Sunday afternoon, March 13, 1983.
It should be noted that Mr. Fornwalt authored the General
! Electric Service Letter that subsequently became part of IE <
- 3) Following the assessment of comments, a final investigative i procedure was prepared. Work orders were generated to j implement the investigation. Reactor trip breaker TCB2 l- (functioning breaker) was tested to verify the adequacy of the investigative procedure. Reactor trip breakers TCBl and IV-1 1
l 4/15/83 O TCB6 (two of the three malfunctioning Unit 2 breakers) were O then tested with the same procedure. Each of these breakers was visually inspected and tested for undervoltage trip while still installed in the switchgear cubicle, then removed and tested for trip shaft torque, undervoltage trip response time, and undervoltage device pickup voltage. The RTB lubricant was revitalized as necessary and changes were made in the UV pickup voltage levels.
- 4) The observations of the initial investigative tests are as follows:
a) 'Two bmakers were missing the locking wire for undervoltage coil pickup voltage adjustment. One of these (malfunctioning breaker TCB1) had a lower than nominal pickup voltage (101 Vdc versus 106 Vdc) and the other (functioning breaker TCB2) a higher than nominal pickup voltage (107 Vdc versus 106 Vdc).
b) The as-found trip shaft torques all exceeded the 1.50 pound-inches specified by IE Bulletin 79-09. The as-found UV devicetorque trip) thanwasTCB1 slightly higher in TCB2 (malfunctioning UV(functioning) device trip 1
but, as noted above, TCB2 also had a higher than nominal pickup voltage. TCB2 also exhibited erratic behavior (slow trip) when its pickup voltage was lowered to O 100 Vdc.
c) Both malfunctioning breakers tripped satisfactorily on repeated undervoltage trip tests with as-found or lower undervoltage pickup adjustments when the bearing lubricant was revitalized to reduce trip shaft torques to less than the 1.5 pound-inches specified by IE Bulletin 79-09.
d) The shunt trip device successfully tripped the breaker if the undervoltage device failed to do so.
i e) The GE factory representative stated that he could find no evidence of improper handling or mechanical damage to the reactor trip breakers examined. Further, all mechanical adjustments were satisfactory, with the
, exception of undervoltage device pickup voltage and a minor increase needed in the TCB6 overtravel adjustment (made after successful testing of the breaker).
< 5) Pmliminary conclusions of the initial investigative tests were as follows:
a) The major contributing factor to improper reactor trip breaker operation on undervoltage was due to insufficient or degraded lubricant in the trip shaft and latch roller
. bearings.
1-l IV-2
4/15/83 b) The secondary contributing factor was the undervoltage
' device amature pickup voltage being set below the recommended 106 Ydc.
- b. In-Depth Tests (March 26 to April 1,1983)
- 1) Based on the results of the initial investigative tests discussed in Section IV.A.l.a. a more detailed inspection was performed on one of the reactor trip breakers to obtain as much quantitative infomation as possible with regard to the dynamic operation of this breaker under various conditions.
The breaker subjected to this in-depth test and inspection was reactor trip breaker TCB-4 from Unit 2, the third of the three breakers which malfunctioned during surveillance testing in early March 1983. The other two malfunctioning breakers (TCBl and TCB6) had been previously tested and readjusted as discussed in Initial Investigative Tests,Section IV.A.1.a of this report.
- 2) The work was perfomed by SCE at its Electrical Test Laboratory with the aid of a General Electric service representative. The breaker was tested on a bench and when operational tests were made, the breaker was secured to the bench by the same breaker flanges that support the breaker when installed in the cabinet. High speed photography was used to assess breaker trip perfomance. A magnetic sy oscillograph was also used to record the following parameters as required for dynamic tests on the breaker:
o Breaker main contacts (3) o Shunt trip coil current o Shunt trip coil voltage o Closing coil current o Undervoltage trip device current o Undervoltage trip device voltage o Auxiliary "b" switch contact
- 3) The bmaker was visually inspected and tested in the as-received condition and baseline measurements were obtained for all parameters, including undervoltage coil pickup and dropout yoltage, trip shaft torque, trip response time, l undervoltage coil amature air gap, and all electrical component resistances. Adjustments were varied and tests repeated to detemine optimum settings and limitations. The f investigation also included inspecting, cleaning, and l revitalizing the trip shaft and latch roller bearings, and l
cleaning and adjusting the undervoltage device. UV device l pickup voltage was also investigated.
- 4) The results of the in-depth investigative tests were as follows:
b a) As-received trip shaft torque was greater than the 1.5 inch-pound limit and as-received undervoltage device IV-3
~. -.
4/15/83 pickup voltage lower than the 106 Vdc limit specified by IE Bulletin 79-09. Cleaning and revitalizing the trip i shaft and roller bearings reduced the trip shaft torques to less than 1.5 inch-pound and successful UV operation was obtained. This confims the preliminary conclusions of the initial investigative tests discussed in Section IV.A.1.a of this report, b) Considerable variation of the UV device pickup voltage setting will result from variations in the UV device coil temperature during pickup voltage adjustment. A minimum of 30 minutes is required for the UV device coil to reach a stable themal state.
c) Ae optimum adjustment for undervoltage device armature i.,:kup voltage is 106 +2 Vdc at a " cold" UV device coil tmperature of 700 to H50F.
d) The diode installed across the UV device coil for surge l protection of the PPS relays delays the breaker response ,
time (nominal 30 millisecond difference) although it
{ remains within allowable values. This diode is installed in the reactor trip breaker cubicle. wiring and is not -
j present on a removed breaker. Therefore, a diode is
- required during reactor trip breaker bench testing.
e) Excessive clearance between the UV device amature magnet i and restraining rivet reduced the effective throw of the amature by pemitting it to move up against the rivet l
rather than rotating. The as-found clearance of 0.018 inch exceeded the 0.001 to 0.010 range recently recomended by the manufacturer; a somewhat narrower '
range (0.003 to 0.006 inch) will provide improved 4
performance and is consistent with the GE factory range of 0.001 to 0.010 inch, f) As-received condition of the trip latch roller bearing (rough operation and excessive clearance) may have resulted in variation of trip shaft torques with roller position, but did not affect trip reliability when pickup voltage and trip shaft torque were within desired range.
! g) There is ample design margin in the shunt trip device.
Operation of the breaker with the shunt trip was satisfactory down to approximately 30 Vdc; the voltage available is nominally in excess of 130 Vdc.
h) The undervoltage response time of the breaker is faster and more consistent for a well lubricated breaker than one with degraded lubricant.
!O IV-4
i l
-4/15/83 1
l 5) The conclusions of the in-depth investigative tests were as d
follows:
a) Two major contributing factors to improper RTB operation on undervoltage are degraded lubricant and undervoltage device amature pickup voltage adjustment; this confims the preliminary conclusions of the initial investigative tests discussed in Section IV.A.1.a above.
1 b) A third major contributing factor to improper RTB operation on undervoltage is amature/ magnet / rivet clearance in the UV device.
c) A minor coniributing factor to breaker response time variation could be damage to the trip latch roller i
beari ng.
2 d) The undervoltage device amature pickup voltage should be adjusted to 106 +2 Vdc at a " cold" UV coil temperature of
~
700 to 850F.
e) The UV device dropout voltage should be measured (after
- pickup voltage adjustment) with the coil energized a minimum of 30 minutes.
l f) The RTB response time should be measured on clean, lubricated, and properly adjusted breakers to establish a l, baseline. Further response time tests during breaker l
surveillance would detect any degradation of breaker performance from the baseline, thus identifying potential incipient failures.
l g) The RTB response time on the bench should be measured with a test diode connected across the UV device coil (testing in the cubicle would have the installed diode in the circuit).
h) With preventative maintenance as dictated by the results of UV response time tests during enhanced surveillance, there is adequate design margin in the UV devices; no I changes are needed to the undervoltage trip design.
- 1) There is ample design margin in the shunt trip device; no changes to the shunt trip design are needed.
- c. During the period of April 6 through April 10, 1983 preventative maintenance proct. Jure S023-I-4.66 was implemented to maintain the breakers and establish the baseline preventative maintenance data for Unit 2 reactor trip breakers.
O l
l IV-5
[
4/15/83
- 1) As-Found Values a) On six of the nine breakers, the undervoltage device amature to rivet clearance was not within the required range.
b) Seven of the nine undervoltage device pickup voltages were less than the required 104 to 108 volts.
c) 'Six of the nine trip shaft torque values exceed the required 1.5 inch-pounds.
- 2) Final Post-Maintenance Baseline Values a) All trip torque values were less than 1.26 inch-pounds.
(Most were in the range of 1.0 to 1.1 inch-pounds.)
b) All RTBs trip times were less than 70 msec.
c) No other bearing problems were found. It is therefore concluded that the bad bearing found on TCB-4 during the in-depth investigation was an isolated occurrence, d) All TCBs operated satisfactorily in all respects following completion of the preventative maintenance.
- 2. Results of Investigative Tests The conclusions of the investigative tests performed by SCE am discussed in Sections IV. A.1.a and IV. A.1.b of this report.
Pertinent recommendations based on these conclusions are provided in Secticns IV.D, V.A.1, and V.A.2 of this report.
I i
J O
IV-6
l 4/15/83 '
B.- Evaluation of Licensee Administrative Procedures
- 1. Control of Vendor Information, Technical Manuals
- a. Vendor-Supplied Documentation Documentation supplied by vendors during the several phases of plant design, procurement, construction, and startup is controlled by procedures established to support the needs, and to suit the circumstances, of the organizations involved in these activities. As startup testing nears completion, procedures to control vendor-supplied documentation during the operating phase are being implemented. This transition is reflected in the following discussion.
- 1) The principal control point for vendor-supplied documentation mquimd to be furnished with equipment for San Onofre Units 2 and 3 has been the Bechtel Power Corporation Drawing and Document Center (DDC). Here, essentially all technical manuals are received, logged, and identified with the equipment involved. Also, a detemination is made concerning the need for a technical review to be perfomed.
In the case of technical manuals Mr equipment furnished by Combustion Engineering (CE), no technical review is required n
v by Bechtel since that review has already been done by CE during their design and procurement cycle. This was the case, for example, for the General Electric technical manuals furnished with the RTBs supplied by CE.
- 2) From the control point established at the Bechtel DDC, documentation required to be furnished is distributed for use within Bechtel and to the SCE Corporate Documentation Management (CDM) Center. The CDM Center has facilities at the station and at the general offices. The CDM Center uses a standard distribution matrix for this vendor-supplied documentation and responds to individual document requests from users. Thus, vendor technical manuals are made available to organizations such as those perfoming maintenance at the station.
- 3) Revisions to vendor-supplied documentation, when issued, have been controlled and processed in the same manner as for the original issue. However, relevant technical infomation may come from various sources in forms other than revisions to vendor-supplied documentation. In the case of the RTBs supplied by CE, for example, information similar to that in the technical manual came from the NRC, CE, and GE at various times after the technical manual was received by CE from GE in 1976.
- 4) Prior to plant operation, vendor-supplied information other than that required to be furnished with the equipment, and other than revisions thereto, has been received in various IV-7
l 4/15/83 l 4
foms by one or more of the organizations involved in construction and startup, including the operating '
organization. Procedures exist for routing and processing much of this infomation. For example, the SCE Nuclear Safety and Independent Safety Er.gineering Groups route NRC IE bulletins and circulars, infomation notices, NSSS vendor i
technical bulletins, IWO Significant Operating Experience Reports and Significant Event Reports, and a wide variety of i utility reports such as selected LERs and Nuclear Operations and Maintenance Infomation Service Reports to the organizations concerned with their contents. Responses are frequehtly requested in this routing and they are tracked to
, completion. Other infomation not captured by this process (e.g., infomation received directly by QA, station management, project management, etc.) is also routed to the organizations concerned.
. The Bechtel DDC Vendor Print Log and the SCE CDM Center provide reference sources where this other vendor-supplied infomation can be identified and retrieved. In the case of
- the RTBs, for example, the undervoltage device maintenance procedure responsive to IE Bulletin 79-09, WES-008, was
, prepared by Bechtel using the GE manual furnished by CE and the infomation provided in the bulletin.
- 5) Since it is necessary that this other vendor-supplied 4
O infomation be obtained and utilized along with the technical manual, and any revisions thereto, in order to perform work correctly, the SCE administrative program emphasizes use of procedures specific to the tasks being perfomed. The 2
process of developing these procedures involves review of all i relevant vendor-supplied infomation, not only that in the i original technical manual. This infomation is captured and routed as described above.
4 Where work is being perfomed following a procedure which is not sufficiently specific for the task, or following a work order (e.g., investigation and correction of an unanticipated failure), it is necessary for the work planning to include review of all relevant vendor-supplied information. In the
, case of the RTBs, although this information was available to the work planning process, it was not used. This reflects a
- breakdown in the work planning, not in the control of
! vendor-supplied documentation or other infomation. (Section IV.B.3 of. this report further discusses maintenance work planni ng. )
- 6) With the completion of the plant startup phase, a new, comprehensive configuration control program is being
^
implemented by SCE to support plant operation. This program has been under development for the past year. When fully 1 implemented, it will assume responsibility for control and management of all vendor-supplied infomation received thus i far and that which will be received in the future. Until it IV-8 i --
_...._.__,_.-.__.~_.m._ , . _ . . _ _ _ . _ _ _ _ _ _ __ _ _ . . _ _ _ . . _ _ - _ . _ . _ _ _ _ _ _
4/15/83 is fully implemented the administrative procedures which have controlled vendor-supplied documentation during the initial phases of the plant will continue to be used.
- b. Vendor-Supplied Services As 'iscussed in Section III.C.4 of this report, SCE utilized vendor assistat. e for resolution of reactor trip breaker problems. The programatic aspects of such vendor-supplied l
l services are discussed below:
l
- 1) Detailed implementing procedures (TI-16) exist to establish a l
unifom method for the procurement and the monitoring of vendor services. Use of vendor services is typical during the initial startup phase of the plant, when experience with
! particular equipment is limited. These procedures provide
! for the indoctrination and familiarization of vendor representatives with site procedures and assurance that vendor field work is perfomed in accordance with SCE I requirements.
) 2) In the case of RTB vendor services. SCE control of the work
! was inadequate because the existing procedural controls for vendor-perfomed work as discussed above were not fully implemented. As a result, programatic protection was compromised in two areas:
a) Fulfilling the requirements (including those of IE Bulletin 79-09 as embodied) in SCE maintenance procedures, and b) Ensuring proper documentation of work performed.
- 3) Following identification of the above-described instances of i
inadequate programmatic control of vendor-supplied services j
(i.e., for the RTBs), SCE reviewed all safety-related work orders since Unit 2 fuel load to determine the extent of'this
~
situation. Approximately 20,000 work orders were reviewed to identify those cases where the vendor performed work or directed work not to an SCE procedure. Forty-one cases were identified and are being individually reviewed. Each case will be assessed and appropriate corrective action taken.
$ 4) Additional training in verbatim compliance and in supervision 3 of vendor work is being made to ensure programmatic controls of vendor services are effective. Work orders used to document and control vendor services are discussed in Section IV.B.3 of this report.
l c. Vendor Recommendations on RTB Vendor recomendations concerning the reactor trip breakers have
> been provided in several foms.
IV-9
- - - - - - .- - - . - - - - - , -ve,vew,,--r.---,.--m- ,en-r,-,~.,-n,- ,e -wwwm e,v.w-a,- - . ,,----,-n--,mw ~ --,-----v---, ,g .,-m-,---v-------------,,-,e-y~ - - , -
4/15/83
- 1) The vendor technical manual embodies the initial vendor O. recomendations for adjustment and maintenance of the RTB (e.g., 12-month PM).
- Bulletin 79-09 provided additional vendor recommendations for adjustment and maintenance of the RTB undervoltage device, '
including that for PM frequency based on subsequent maintenance experience. Incorporation of IE Bulletin 79-09 requirements is discussed in Section IV.B.7 of this report.
maintenance in accordance with the manufacturer's i recomendations are per refueling interval unless periodic l testing indicates that a more frequent interval is required j (i.e. ,18 months). This recommendation is in conflict with the above RTB breaker vendor recommendations of 12 months.
- 4) Further informal vendor recommendations were provided when vendor representatives were called in to assist SCE with RTB difficulties experienced during plant startup. The vendor representative reconnendations as regards RTB adjustment settings in some cases conflicted with the technical manual requirements, as discussed in Section III.C.4 of this report.
- 2. Control of Hardware Configuration
- a. Spare Part Programmatic Controls Existing programmatic controls utilized for procurement of spare RTBs and installation of RTB spare parts are as follows:
- 1) In accordance with the purchase order requirements, "SCE/CE Master Agreement" dated August 22, 1977, CE supplies manufacturer's documents for:
- a. Certificate of equipment
- b. Certificate of conformance/ compliance -
- c. Test results
- 2) SCE performs documentation review and receipt inspection as follows:
Visual Inspection: Inspected externally for physical damage and cleanliness. In addition, SCE performs receipt inspection as specified per the purchase order request provisions.
Documentation review: Certify parts design, procurement actions and special processes, control of inspection, testing and test equipment, identification and control of equipment O and records thereto as well as all other CE commitments are in accordance with SCE/CE Master Agreement.
IV-10
A 4/15/83 O 3) CE provides a Certificate of Equipment and Records List or equivalent fom on spare parts on all safety-related items.
If any changes are requimd that deviate from the original design, they document and provide the change via CE Standard Technical Change Request (TCR) Fom. The TCR is noted on the Records Check List. CE specifies in their purchase order the' necessary mquirements to their vendors with whatever requirements am necessary. If any changes are made, they are reviewed by CE's Engineering and TCRs are generated as required.
- 4) SCE QA perfoms periodic audits /soun:e inspections of CE to verify that CE applies their QA program to suMier vendors.
- 5) Receiving inspection activities are planned, perfonned, and documented in accordance with written procedures and in accordance with the Topical Quality Assurance Manual (Reference TQAM Chapter 4A and QAP N10.02). Said items are appropriately inspected and accepted prior to installation or use of the item.
A number of QA procedums, startup test instructions, engineering and construction procedures as well as station procedures for
! each area exist for procurement of spare and replacement parts.
l These procedums include the requirements to refer to NRC rules and regulations comitted to by SCE.
- b. Reactor Trip Switchgear Spare Parts
! This discussion outlines the RTB procurement-related activities I and associated pmblems actually encountemd. After the original
[ 18 RTBs were manufactured, GE discontinued providing this type of l bmaker. However, Satin American Corporation purchased all of
- GE's remaining stock and continues to supply spare parts / breakers and services to CE/SCE. Five spare GE breakers were procured for San Onofre Units 2 and 3 from MIDAN Electronics, Inc., who purchased the bmakers from Satin American. Satin American
- modified and tested the breakers. The documentation was then
! sent to CE who reviewed the data and certified the breakers.
l After the RTBs were received at SCE on June 24, 1982, an NCR was issued to identify lack of certifications. The certifications
- were subsequently received and the NCR was closed on July 2, 1982. During the June 28, 1982 receiving inspection process, it was discovered that the five RTBs had wrong undervoltage coils
, (480 Vac instead of 125 Vde). These five breakers were subsequently returned to Satin American on July 8,1982 for:
- 1) Replacc:nent of the UV coils
- 2) Installation of the adjusting hardware on the UV coil armature
- 3) Placement of a metal cover over the close button.
When the RTBs were subsequently returned, the again lacked the necessary certifications and on July 30, 1982 SCE Nonconfomance f Reports were generated during the receipt inspection process.
. IV-ll
I
~
4/15/83 '
The breakers were conditionally released for preoperational testing and installation in Unit 3 pending receipt of l documentation. " Restraint to Prepare to Load Fuel", was assigned to this NCR, which is a Mode 6 restraint. During preinstallation testing, it was detemined that the undervoltage devices were still found to have missing setting adjustment
- hardware. Accordingly, additional WCRs were issued on August 10, i 1982. It should be noted that this equipment deficiency could not have been routinely detemined during receiving inspection without partial disassembly. Receiving inspection was perfomed -
in accordance with the purchase order requirements which included visual inspection and review of Certificates of Confomance from Midan, and Equipment and Test Results for adequacy.
CE acquired proper UV coil assemblies from Midan Electronics and SCE issued work orders to replace the defective UV coil assemblies. The coil replacement work was performed under the
- cognizance of SCE and CE. The GE representative subsequently adjusted the breakers onsit
- nd the breakers were successfully
, retested on August 20,1982 nrough August 25, 1982 in accordance ,
with the GE technical manual for the RTB. The defective UV coil -
assemblies were returned to Midan Electronics and certifications from Midan for the new UV coil assemblies were received on August 27,.1982. All of this was accomplished prior to Unit 3 entry
- into Mode 6 which was on November 15, 1982.
J. A source audit of CE and Midan Electronics by SCE. QA during the week of April 5,1983 detemined the following with regard to the spare bmakers.
- 1) Ordering information for the spare breakers was equivalent to the ordering infomation for the original breakers, except the undervoltage device was not clearly specified. Nomally,
. a detailed list of subcomponents within a spare assembly is t
not required.
- 2) Midan Electronics was qualified from a quality assurance aspect to supply only off the shelf electronic components and not complex assemblies such as circuit breakers which required control of subtier supplier activities. The control of subtier supplier activities was and is the responsibility of the NSSS vendor's QA program.
- 3) As a result of item 2 above Midan Electronics did not pass on l' any quality assurance program requirements to Satin American in the procurement documents.
I
- 4) Certifications provided by Midan Electronics and Satin
. American are not quickly auditable due to items 2 and 3 above.
The same audit did verify that appropriate procurement controls O were adequately provided for the original 18 breakers by CE and the reactor trip switchgear supplier, Unit Electric Control and IV-12
. .= .= . - - . - - ..
t 9
4/15/83 that the switchgear assembly was environmentally and seismically qualified in accordance with procurement requirements.
Corrective action is being requested from CE m1ating to the audit findings in accordance with the SCE Quality Assurance Program.
The configuration of the spare RTBs will be documented to SCE's i
sati sfaction. If this cannot be done, then the spare breakers will be rejected (including all parts / subassemblies used as spares) and replacement RTBs will be procumd.
It should be noted that four of the five subject spare breakers were installed in Unit 3 during pre-critical rod drop testing;
- however, no known failures of installed spare bmakers has occurred. (The fifth spare bmaker has been used for replacement parts.) Environmental / seismic adequacy of the spare RTBs was based on the' prototype testing of the original RTBs and this requires further review to ensure that differences in the spare i bmakers do not affect this qualification.
- 3. Maintenance Program
- a. Procedums and QA/QC Requirements j In compliance with Criterion V of 10 CFR 50, Appendix B, the SCE QA program requims that activities affecting quality be prescribed by and accomplished in accordance with documented instructions, procedures, or drawings of a type appropriate to the circunstances. In the case of the RTBs, this involved both pre-established maintenance procedures specific to the task involved (e.g., WES-008) and work orders which were written to
- implement nonconfomance report dispositions not covered by pre-established maintenance procedures. (The preparation of these nonconfomance reports is discussed further in ,
4 Section IV.B.6. ) l l
In compliance with Criteria V, X and XVIII of 10 CFR 50, Appendix B, the SCE QA program requires that acceptance criteria be established and that inspection and audits be performed to verify compliance with requirements for activities affecting quality. In the case of the RTBs, the implementation of procedures and work orders was subjected to QA program verification as required.
i The following discussion addresses use of procedures and work l orders, including in the case of the RTBs:
l 1) Use of Maintenance Procedures to Perfom Preventative i Maintenance Procedures to perfom PM were initially. written by Bechtel for the construction completion and startup phases of the
~
project. In the' case of the RTBs, these were MPES-008 and ,
1 IV-13
_ , . _ _ _ . . - ~ . _ _ . . _ , _ _ _ . _ _ _ . _ . _ . _ . _ _ . . _ . _ _ _ . . _ _ .
4/15/83 MPEG-025. As discussed in Section IV.B.l.a., these procedures were based on vendor-supplied information, regulatory requirements, and experience with similar equipment. Recommended intervals for PM were included in the maintenance scheduling system. As startup nears completion, these PM procedures and PM intervals are being revised, as necessary, and included in the operations phase PM program.
In the case of the RTBs, the process for development, use, and transfer to the operations phase of the PM procedures and intervals experienced the following problems:
a) Although MPES-008 was written specifically to implement IE Bulletin 79-09, the procedure did not itself identify the bulletin as a source or reference document. As a result, the importance of its use was not consistently recognized and an error was made initially when it was revised for use in the operations phase resulting in deletion of some important infomation. (The deletion of this infomation was later recognized when the procedure was reviewed in response to IE Bulletin 83-01 and prior to its use.)
b) The PM interval was established as " refueling." Often this is interpreted b be a maximum of 18 months, but in this case the PM scheduling system would not call for p/
s_ this work to be done until actual refueling occurred which could be much longer than 18 months from the initial vendor or field PM. All " refueling" PM intervals are being reviewed to identify those which should be revised to ensure against excessive time durations.
c) The PM interval in the GE technical manual is recommended as 12 months. Using the PM program developed by Bechtel, a " refueling" interval was being scheduled based on NSSS vendor (CE) recomendations to perform PM in accordance with the manufacturer's recommended once per refueling interval unless periodic testing indicates that a more frequent interval is required. A comprehensive program to review all PM intervals has been implemented and is expected to be completed in about 6 months. By April 30, 1983, a review against all specific regulatory requirements and commitments will be complete. The initial results of this effort indicate no significant problems exist.
d) An initial, baseline PM was done in accordance with MPES-008 for the RTB undervoltage devices on Unit 2 but was not done for Unit 3 or spare breakers. However, during July 1982, SCE obtained the services of a vendor representative to perform what was believed to be a (q) complete and adequate PM of all Unit 2 and Unit 3 RTBs then installed. All PM records are being reviewed to establish adequate baseline conditions for the scheduling IV-14
l 4/15/83
, of subsequent PM intervals. This will be completed in about 1 month.
f e) Feedback of experience to revise procedures to correct pert:eived deficiencies, and to mvise PM intervals based on observed conditions, was not implemented appropriately. The primary reason for this was that the
- construction completion and startup phases of the project i were considered abnomal with respect to this sort of feedback process. That is, a large amount of data am generated during this period which are unique to the startup and initial operating periods. An experience feedback program is being implemented for the operating phase which will utilize data from nonconformances and cornctive maintenance as input to the PM program.
f) Review of the RTB maintenance _ has indicated that improved reporting of overdue PMs is mquired. Since the RTB PM was scheduled for refueling, it was not yet overdue as i far as the scheduling system was concerned. However, the status of all scheduled PMs is being reviewed, and an improved program for reporting and evaluating overdue items is being implemented for the operating phase.
- 2) PM of Replacement Components In response to problems with the Unit 2 RTBs during startup testing, Unit 3 RTBs, which had not received PM, were
< transferred and installed in Unit 2, which had received PM in
' accordance with MPES-008. A similar potential exists when spare components which have not had PM are installed from the wanhouse. Administrative controls to ensure that PM is done for replacement components, where appropriate, prior to their use will be implemented.
- 3) Corrective Maintenance Utilizing Work Orders Corrective maintenance is often perfomed in response to nonconfomance report dispositions. Development of these dispositions is discussed in Section IV.B.6. Work orders are utilized to control this maintenance, and existing maintenance procedums may be referenced or other direction specified such as to follow instructions in a technical manual or of a vendor representative. All of these options were used in various RTB corrective maintenance work orders.
In all cases, documentation of work done and inspection are requimd to verify that the requirements of the work order are met.
In the case of the RTBs, the use of corrective maintenance work orders experienced the following problems:
, a) In some cases the applicable maintenance procedure was not mferenced when work was done on the undervoltage i IV-15
. , _._. _ . _ . . _ , _ . ~ _ _ _ _ . _ _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __
l 4/15/83 devices. Rether, the technical manual, absent additional vendor-supplied infomation, or vendor representative direction were referenced. As discussed in Section IV.B.1.a. vendor-supplied infomation must be identified and considered in work planning, not only the technical manual. Maintenance planning personnel did not recognize the existence of additional infomation in this case.
(Control of work at vendor representative direction is discussed in Section IV.B.3.c.) Training is being i developed for all cognizant personnel to emphasize use of-available procedures in lieu of technical manual refemnces and to review the need to identify all pertinent vendor-supplied infomation for work order planning where procedures are not available. This effort -
is expected to be in place by July .1,1983.
b) As discussed in a) above, the applicable maintenance procedure for work on the undervoltage devices did not
, itself identify the IE bulletin as a source or reference document. Accordingly, maintenance planning personnel did not recognize its importance with respect to implementing IE bulletin requirements and believed that the best possible response to problems with the undervoltage devices was use of the technical manual or i reference to vendor representative direction. Review of i maintenance procedures discussed in this section above
- will include reference to regulatory requirements and
- comitments where appropriate.
c) Investigation has disclosed that in some cases the referenced procedure or technical manual was not followed in detail. This occurred when direction was provided by
- the vendor representative in March and July 1982. Al so, the work perfomed and the inspections conducted were not adequately documented. Since this is clearly contrary to requirements of the SCE QA program and procedures, additional training of personnel and supervisory followup
! of future work is required and will be provided.
An extensive review of corrective maintenance work orders has shown that these problems am not common. They occurred particularly in the case of the RTBs during startup testing because of reliance on vendor representative dimction and incomplete vendor-supplied i infomation. In this case, the vendor was relied upon I precisely because of the recognized critical importance l of the RTB function.
- b. Maintenance History and Records Individual history files for each RTB have been developed and O pmvided separately to the NRC. These files required considerable effort to develop because the operational phase maintenance history system has not yet been fully implemented,
! IV-16
1 4/15/83 and the data were contained in the construction completion and startup phase records which were established to meet the needs of those phases of the project. Emphasis is being placed on prompt implementation of the operational phase system which will pemit easier retrieval of individual component histories for evaluation
, and assessment.
- c. Vendor Maintenance Activities As discussed in Section IV.B.3.a the GE representative was called in by SCE, as recommended by CE, in response to problems with the undervoltage trip devices in March and July 1982 and again in March 1983. Anticipating the need to carefully control such services during startup testing, a detailed implementing procedure (TI-16) exists to establish a unifonn method for the procurement and control of vendor services. This procedure pmvfdes for the fomal-indoctrination and familiarization of vendor representatives with site procedures and assurance that vendor field work is perfomed in accordance with procedures.
Use of vendor representatives is especially common during the startup phase when numemus setup and adjustment problems are encountered and when site experience with the equipment is limited.
In the case of the RTBs, the procedum for control of vendor r services experienced the following pmblems:
o The procedural requirements for vendor indoctrination were not followed. In addition, vendor representative qualifications to do the work were not evaluated and his i knowledge of requirements beyond the technical manual was not l detemined. As discussed above, training is being conducted
( to emphasize procedural compliance. Procedures will be l
revised to better document vendor representative indoctrination and to assess his qualifications.
o Problems discussed above with corrective maintenance work oniers applied to those that used vendor representative services. Approximately 20,000 work orders were reviewed to identify protential similar examples. Of these, 41 uses of vendor representative services are being carefully reviewed to ensure against any other cases in which such services could have resulted in failure to implement important j requimments such as IE Bulletin 79-09.
In summary, the SCE program recognizes the importance of careful control of vendor services. In the case of the RTBs, the significant importance of problems with the UV devices was recognized, but excessive reliance was placed on the vendor representative to ensure that all maintenance requirements were met to ensure m11able operation following corrective O maintenance. SCE personnel who obtained these services in July 1982, at CE recomendation, believed that a complete and IV-17 l
4/15/83 m; adequate PM and adjustment of all installed RTBs had been accomplished, although this was not adequately controlled or (U documented.
Because SCE recognizes the importance of careful control of vendor services, a procedure to control vendor services during the operating phase will be developed.
- 4. Surveillance Program
- a. Technical Specification Requirements Surveillance testing of the reactor trip breakers is perfomed in accordance with Technical Specification 3/4.3.3.1. This testing consists of a Channel Functional Test (perfomed monthly) and a Response Time Test (perfomed at refueling intervals as part of the overall response time of each RPS function per Technical Specification definition 1.25). At least once every 18 months and following maintenance or adjustment of the reactor trip breakers, independent testing of the undervoltage and shunt trips is required. These are Standard Technical Specification Requi rements. Reporting requirements are discussed in Section IV.B.5 of this report. Recommended changes will be discussed in Sections IV.D.2, V. A, and V.B of this report.
- b. frocedures and QA/QC Requirements
( 1
'v' 1) Reactor trip breaker surveillance testing is perfomed in accordance with procedures S023-II-1.1 (plant protection system testing), S023-II-3.1, -3.2, -3.3, -3.4 (plant protection system response time testing), and S023-II-11.161 (which tests independent undervoltage and shunt ttip actuation), as discussed in Section III.C.4 of this report.
Proposed changes to S023-II-11.161 to improve serviceability and to address detailed operability requirements of the UV device, based on the results of RTB investigative tests, are discussed in Sections IV.D.2, V.A, and V.B of this report.
- 2) The programmatic controls for perfoming surveillance testing are as follows:
a) S023-XV-3.0 " Technical Specification Program Implementation" identifies frequency of test, type of test, mode requirements, responsible department, and the implementing procedures for all equipment identified in the technical specification. With regard to the reactor trip breakers, this procedure identifies monthly, refueling, and after maintenance channel functional tests utilizing S023-I1-1.1 and S023-II-11.161. (Response time testing of RTBs is perfomed as part of the PPS response time test for each PPS channel.)
O
'y) b) S023-0-23, Equipment Status Control, paragraph 6.3.1.6.8, requires identification of operability testing IV-18
4/15/83 (q
v requirements for equipment being cleared for maintenance or modifications.
c) S023-0-23 references S023-0-24 " Redundant and Operability Testing Requirements" which provides more specific direction for operability testing prior to returning equipment to service. Basic categories include a) pump and valve ISI to detemine operability and b) the use of technical specification surveillance tests.
d) For the RTBs, the appropriate post-maintenance test following adjustment of the undervoltage device is S023-11-11.161, " Reactor Breaker Undervoltage and Shunt Service Circuit Test." A review was perfomed to determine if S023-II-ll.161 was used as the post-maintenance test following all RTB maintenance activities. In one instance, a channel functional test was used in lieu of S023-II-ll.161. The channel functional test did not independently test the undervoltage coil following a coil adjustment (refer to III.C.4 and IV.B.5). This was reported to NRC I&E in LER-82-176. All other maintenance activities were subjected to the correct post-maintenance test.
The cause of the above inappropriate use of a functional test is not programmatic; i.e. , adequate overall n; procedure requirements exist to define the requirements (d for post-maintenance testing. Rather the problem is specific methodologies for the detennination of post-maintenance testing requirements. In all cases where a technical specification surveillance requirement exists, the surveillance is usually the most appropriate operability test to use. When neither an appropriate surveillance test or procedural test is available, it is necessary for the work-planning process to include appropriate post-maintenance functional testing. Our review of this area indicates more definitive requirements are needed. Consequently, a task force has been established to develop guidelines for testing requirements. The task force work will be complete and training implemented by July 1,1983.
- c. Surveillance History and Records A summary of RTB surveillance and maintenance history is provided in Sections III.C.4 and III.D of this report. A surveillance record is maintained via completed work order files. The maintenance history and records system being implemented for the operational phase as discussed in Section IV.B.3.b will facilitate retrieval of individual component surveillance histories for evaluation and assessment.
O V
IV-19
4/15/83
- 5. Reporting of Failures b
V a. Technical Specification Requirements Reporting requirements are, in part, specified in Section 6.9 of the Technical Specifications.
- 1) The pertinent portions of Technical Specification 6.9 for surveillance testing of reactor trip breakers are as follows:
" Prompt Notification With Written Followup" "6.9.1.12 The types of events listed below shall be reported within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> by telephone and confinned by telegraph, mailgram, or facsimile transmission to the Regional Administrator of the Regional Office or his designate no later than the first working day following the event, with a written followup report within 14 days. The written followup report shall include, as a minimum, a completed copy of a licensee event report form. Infonnation provided on the licensee event report fonn shall be supplemented, as needed, by additional narrative material to provide complete explanation of the circumstances surrounding the event."
"b. Operation of the unit or affected systems when any parameter or operation subject to a limiting condition for operation is less conservative than O( ,/ the least con::ervative aspect of the Limiting Ccndition for Operation established in the Technical Specifications."
"c. Failure or malfunction of one or more components which prevents or could prevent, by itself, the fulfillment of the functional requirements of system (s) used to cope with accidents analyzed in the SAR."
"1. Perfonnance of structures, systems, or components l that requires remedial action or corrective measures
! to prevent operation in a manner less conservative l than assumed in the accident analyses in the safety analysis report or Technical Specifications bases; or discovery during unit life of conditions not
! specifically considered in the safety analysis l
report or Technical Specifications that require remedial action or corrective measures to prevent the existence or development of an unsafe condition."
" Thirty Day Written Reports" "6.9.1.13 The types of events listed below shall be the o subject of written reports to the NRC Regional Administrator within thirty days of occurrence of the event. The written Q report shall include, as a minimum, a completed copy of a IV-20
4/15/83 n licensee event report fom. Infonnation provided on the licensee event report fonn shall be supplemented, as needed, (V )
by additional narrative material to provide complete explanation of the circumstances surrounding the event."
"a. Reactor protection system or engineered safety feature instrument settings which are found to be less conservative than those established by the Technical Specifications but which do not prevent the fulfillment of the functional requirem'nts of affected systems."
"b. Conditions leading to operation in a degraded mode permitted by a Limiting Condition for Operation or plant shutdown required by a Limiting Condition for Operation."
"c. Observed inadequacies in the implementation of administrative or procedural controls which threaten to cause reduction of degree of redundancy provided in reactor protection systems or engineered safety feature systems."
- 2) Technical Specification 3/4.3.1, " Reactor Protective Instrumentation," requires the reactor trip breakers to be OPERABLE in any mode (except refueling) when the CEDM
(] cabinets are energized. Table 4.3-1 note 12 requires that V the monthly Channel Functional Test used, in part, to demonstrate operability of the RTBs include a test of independent undervoltage and shunt trips at least once per 18 months and following maintenance or adjustment of the RTBs; in other words, a valid test of independent UV and shunt trips is required by the Technical Specifications for an RTB to be considered operable. (However, the UV trip is not required for the RTB to meet its design basis assumed in the accident analyses in the FSAR as discussed in Sections III.A and B of this report.)
l 3) The four reactor trip breaker undervoltage trip failures i
l during surveillance testing in early 1982 were not reportable under Technical Specifications 6.9.1.12b or c, or 6.9.1.13a, b or c since the plant was not in a mode requiring OPERABLE RTBs pursuant to Technical Specification 3/4.3.1 nor were the
! discussed above. SCE did report these failures pursuant to Technical Specification 6.9.1.1 21 via LER 82-175; however, as i discussed in Sections III.A and B of this report, the UV device is not required to function for the SONGS 2/3 Reactor Protection System design basis assumed in the accident analyses in the FSAR (and hence in the Technical Specification design bases).
) 4) The return to service of three RTBs in July 1982, without a I
surveillance test of the UV device pursuant to Technical J
l IV-21 l
i 1 4/15/83 Specification 3/4.3.1 (Table 4.3-1, note 12) following adjustment of the UV coils, was reportable under Technical
- i O Specification 6.9.1.12b since. the RTBs were required to be OPERABLE (i.e., satisfy the pertinent surveillance i'
requirements) and the minimum channels operable for RTBs is four (i.e. , eight channelized RTBs). SCE reported these failures via LER 82-176.
- 5) The four reactor trip breaker undervoltage trip failures during surveillance testing on March 1,1983 (Unit 3) and March 8,1983 (Unit 2) were not reportable under Technical Specifications 6.1.1.12b or c, or 6.9.1.13a, b or c since the plants were not in a mode requiring OPERABLE RTBs pursuant to Technical Specification 3/4.3.1, nor were the UV devices required by the FSAR accident analyses as discussed above.
SCE reported these failures pursuant to Technical Specification 6.9.1.12i via LERs83-019 and 83-023; however, as discussed in Sections III.A and B of this report, the UV device is not required to function for. the SONGS 2/3 Reactor Protection Tystem design basis assumed in the FSAR (and hence in the Technical Specification design bases).
- 6) An additional report pursuant to Technical Specification 6.9.1.13c was made via LER 83-025 concerning inadequacies in RTB maintenance activities identified following the March 1 and 8,1983 surveillance tests. (Programmatic. aspects of maintenance are further discussed in Secton IV.B.3 of this report.)
- b. Reporting of March 1 and 8,1983 Surveillance Results SCE conducted surveillance testing of the GE Type AK2-25 RTBs
! installed at San Onofre Units 2 and 3 on March 1 and 8,1983, F
even though IE Bulletin 83-01 only required testing of the Westinghouse DB-50 breakers.
Testing was perfomed on all nine Unit 3 RTBs on March 1,1983.
One RTB failed to trip on its undervoltage test; however, the RTB tripped properly during the shunt trip. NCR-3-243 was written to document the nonconfomance. Since failure of the undervoltage i trip for one RTB was considered an isolated failure, and the CEDM cabinets were not energized, the condition was not considered prompt reportable at that time.
Testing on Unit 2 was delayed by a failed power supply in the PPS i- until Man:h 8,1983. The Unit 2 surveillance testing revealed three RTBs that did not trip on undervoltage.. All RTBs successfully tripped during the shunt trip. NCR-2-163 was written to document these failures. As a result of requimd reviews of both NCRs, a prompt telephone notification was made on March 10,1983 followed by a prompt written notification on March 11,1983 and a subsequent 14-day followup report on i March 24,1983 (LER 83-019 for Unit 2 and LER 83-023 for j- d Unit 3). Actual reportability of these occurrences under the
}
IV-22 ,
1
4/15/83 Technical Specifications is discussed in Section IV.B.S.a of this i mport.
g
- 6. ' Nonconfomance Reports (NCRs)
^
NCRs' are used for identification and control of nonconforming items at SONGS. The NCR program in effect in March 1982, when problems with the RTB UV trip device were first experienced, consisted of the following elements:
, a. Discovery of the nonconfoming item or condition, notification of l Quality Assurance-(QA) supervision, and initiation of the NCR, including complete description, by the discoverer or QA NCR staff.
- b. Verification of the nonconfomance by QA NCR staff.
- c. Segregation'and/or tagging of the nonconfoming item.
- d. Technical review to detemine the action needed to correct the
, nonconformance (disposition), documentation of the rationale for the disposition, and notification of responsible organization.
- e. Determination of the cause of the nonconformance and action necessary to prevent recurrence by the cognizant ' organization,
- f. Verification by QA NCR staff that disposition actions have been completed and tags removed.
- g. Closum of the NCR by QA supervision, including designation of any further action warranted.
i l h. The SCE QA organization evaluates on a quarterly basis significant tmnds in NCRs by area of responsibility and reports its findings to management of various departments.
l In April 1982, on a trial basis, the NCR process was modified to record operability assessment, notification of operations, and reportability detemination on the NCR fom. Procedure changes were held in abeyance pending evaluation of trial-use. In November 1982, procedums were revised to require assessment of the nonconfomance for operability, detemination of reportability, and notification of operations upon discovery of a nonconfomance.
A detailed review of the seven NCRs associated with the SONGS 2 and 3 installed reactor trip breaker failures was conducted to evaluate the effectiveness of the NCR program, including implementation, in controlling nonconfomances. The review identified several areas that need strengthening in program implementation and program procedums.
Generally, the NCR program investigation revealed the following:
- a. Several NCRs closed prior to verification that " corrective action to prevent recurrence" was complete because the QA procedure only IV-23
4/15/83 mquired verification that the action had been initiated and not (m) v, necessarily completed.
- b. During the trial NCR program NCRs were routinely transmitted to Equipment Control via standard attached distribution for copy; however, the immediate " notification to Equipment Control" block on the NCR fom was occasionally left blank. During the trial-use period, when the NCRs were written, the implementing procedures did not specify the group or individual rr sponsible to notify Equipment Control. This has been corrected with the November 1982 revision to the Topical QA Program.
- c. NCR disposition statemer.ts by SCE technical personnel sometimes did not fully document the technical rationale.
- d. Reportability and operability assessments on NCRs during the March through July 1982 time frame were not always documented as being perfomed by the technical staff nor subsequently formally reviewed with the Configuration and Compliance group. The basic reason for the inconsistency in the assessment documentation was that the implementing procedures were in a trial-use program which was not made manadatory in the Topical QA Manual until the November 1982 revision.
In addition, these implementing proceduras for the trial-use program would have required previous strengthening in order to
(~' have :r.itigated the inconsistency regarding reportability and
( operability assessments of NCRs.
el Although the SCE QA organization has a fomal program to quarterly review and publish trend results on the number of NCRs increasing (or decreasing) in the different responsible organizations, no fomal program existed within SCE to evaluate NCPs fer repetitive failures of components or systems. It should be noted that an infomal site process for obtaining repetitive failures of components or systems, utilizing the master tracking computer system, was Tn place and has been used frequently since Unit 2 fuel load.
- f. During the reactor trip breaker investigation, it was noted that three hold tags were still attached to the Unit 3 RTB equipment.
This resulted in a review of all closed NCRs (approximately 1,000) to determine hold-tag status. This review disclosed four additional hold tags on miscellaneous systems which were inappropriately hung. The NRC and SCE QA representatives on March 12,1983 noted that one of the hold tags on the RTB on Unit 3 could have interfered with the breaker operations.
However, there was no hold tag on TCB4, the only breaker that failed during the March 1,1983 surveillance testing.
Q)
IV-24
4/15/83 To make the NCR program more effective in controlling nonconfomances, the following cormctive actions will be taken:
- a. Revise procedures to require technical review of apparent causes and/or actions to prevent recurrence prior to closing NCR.
Initiate followup on cormctive actions to prevent recurrence from NCRs issued subsequent to Unit 2 fuel load. To be completed by April 30, 1983.
- b. Review NCRs initiated subsequent to receipt of the operating license and prior to November 1982 for reportability. Make required reports. Initial review of NCRs was completed on April 11,1983. Approximately 5 percent of the NCRs require additional infomation to complete the reportability detemination. Final detemination on all NCRs will be completed by April 30, 1983,
- c. QA organization, as part of NCR validation, has reemphasized the procedural requirements that all elements of description of the nonconfomance are included. This was completed and documented on April 1,1983.
- d. As a followup to action a. above, train personnel authorized to approve NCR dispositions in the required content of disposition statements, detemination of the cause and action necessary to prevent recurrence (to be completed by April 30, 1983). Ongoing training will also be perfomed for action f. below.
- e. Revise procedures to stmngthen operability assessments and reportability deteminations by providing clarification that the operability assessment relates to Technical Specification operability and by providing reference material to support mportability determinations. To be completed by April 30, 1983.
- f. Implement a methodology for identifying and evaluating repetitive nonconfoming conditions of components or systems. This methodology will be developed by the SCE Nuclear Safety Group described in Section I.B.l.2 of the. responses to NUREG 0660/0737 of the San Onofre Units 2 and 3 FSAR (to be completed by May 31, 1983). In addition, the site QA groups on April ll, 1983 began fomal transmittal to responsible groups of a M drized listing from the master tracking system of repeur.' t entries by component identification designators for evaluatiw. and appropriate followup action.
- g. The Nuclear Quality Control group verified that hold tags had been removed for previously closed NCRs. To prevent recurrence of this problem, an NCR tag log is now utilized to control issuance and mmoval of all NCR tags. The QA procedures were revised November 12, 1982 to strengthen the hold tag removal program, and training is ongoing. The revised procedures include q a new NCR fom, with appropriate blocks used to indicate the g hanging and removal of hold tags.
I V-2 ',
0 4/15/83
- 7. Compliance with IE Bulletins and Circulars The review and response to IE bulletins and circulars is conducted in accordance with SCE procedunts. The sequence of events documenting the initial review of IE Bulletin (IEB) 79-09 is described below.
- a. SCE received IEB 79-09 on April 20, 1979 and formally requested review and recommendations from CE and Bechtel on May 7,1979.
- b. Bechtel responded on May 22, 1979 indicating that there were no GE AK-2 circuit breakers in their area of responsibility.
- c. CE responded on June 8,1979 indicating that GE AK-2 circuit breakers are used only in the reactor trip switchgear. CE recommended SCE follow the GE recommendations in the IEB.
- d. SCE responded to NRC on June 13, 1979. This response outlined two courses of action: a) if a rule regarding ATWS design changes was approved and issued, SCE would modify the reactor trip switchgear design to eliminate the AK-2 breakers; and b) in the meantime, preventive maintenance would be perfomed on the existing AK-2 breakers in accordance with item 3 of IEB 79-09.
- e. As followup to this response commitment, SCE on June 14, 1979 initiated action in accordance with its IEB response.
SCE QA issued a nonconfomance report (S023-F-216) on July 27, O f.
1979 to track the development of the required maintenance procedure in the startup maintenance program.
- g. Verification that preventive maintenance procedure MPES-008 had been developed to address the IEB 79-09 concerns was made by SCE QA on June 6,1980 and this item was closed,
- h. An additional level of tracking was applied using a QA NRC Action Item Request (AIR) issued on October 26, 1979. This NRC AIR was closed on July 1,1981 by referencing maintenance procedure MPES-008.
- 1. NRC Inspection Report 50-361/81-07 documented the NRC Regional inspection of the SCE review and documentation of IEB 79-09 and confimed that this item was closed.
SCE procedures for review and documentation of IE circulars (IEC) were also followed for IEC 81-12. Since IECs require no fomal response to NRC, SCE review addresses the existing design and s determines whether further action is recommended. IEC 81-12 was reviewed by the Independent Safety Evaluation Group (ISEG) and it was concluded that surveillance procedure S023-II-11.161 for 18-month surveillance of the reactor trip circuit breakers adequately addressed the requirement to independently test the undervoltage and q shunt trip relays.
C' IV-26
4/15/83 It has already been indicated that, at least in the case of the RTBs, h implementation of IE Bulletin 79-09 through the preventive
[b maintenance program was incomplete. Therefore, a review of all IE bulletins, circulars, and notices will be conducted to detemine what other, if any, deficiencies exist in the PM program. This review will be completed by April 30, 1983. Corrective actions as a result of this review will be taken as appropriate.
- 8. Post-Trip / Restart Reviews Prior to the Salem incident, Operating Instruction S023-0-ll, "Startup and Shutdown Chart Removal and Identification" (Revision 1),
identified that a review of shutdown /startup and trip / transient charts by Station Engineering could be conducted. It did not require completion of this engineering review prior to restart of the plant.
In accordance with a separate Operating Instruction, the Shift Supervisor was assigned responsibility for authorizing restart of the plant based upon his assessment of plant status and readiness. This authorization was not a fonnal documented decision. Subsequent to the recent Salem incident, Operating Instruction S023-0-11 has been revised (Revision 2) to include a fomal post-trip review through the use of a checklist. The objectives of this fomalized post-trip review are as follows:
- a. To provide a method for completing post-trip review documentation and ensuring senior level personnel review prior to authorizing reentry into Mode 2, and
- b. To describe in detail the review requirements of that post-trip review as follows:
- 1) Detemine the cause of the trip, and implement any required corrective actions.
- 2) Verify that the reactor protective system functioned properly, and implement any required corrective actions.
- 3) Verify that ESF systems functioned properly, and implement any required corrective actions.
- 4) Verify that all automatic and operator actions have been reviewed and, if any off-normal occurrences are identified, implement any required corrective actions.
The revised procedure requires completion of the post-trip review and of appropriate corrective actions prior to reentry into Mode 2 and return to power. The completion of corrective actions specified above involves those actions necessary to ensure proper operation of systems important to safety. Corrective actions could be identified which would not be restart limiting.
If the cause of the trip was detemined during the post-trip review, authorization by the Shift Technical Advisor, the Shift Supervisor, O'* and the Plant Superintendent (or, in his absence, the Station IV-27
4/15/83 s
Operations Manager) is required prior to entry into Mode 2 and return to power. If the cause of the trip cannot be determined, O- authorization by the Station Manager (or, in his absence, his !
superior) is also required.
i a
f I I O
i i
?
I 1
l O
IV-28
- - - - - , . - - , - , . . + , . , , , . , , - , , , . - , . - - - . . , , . , , . , , , - , . , - - . , _ _ , - - - - , - - - - - - - , , , ~ - - , - - - - ~ . - - - - - - . ~ - - , - , - , . ,
i 4/15/83 l
C. Evaluation of Capability to Mitigate ATWS
- 1. Procedums for Mitigating ATWS Prior to the March 1 and Man:h 8,1983, events regarding the reactor trip breaker undervoltage trip function, Emergency Operating Instruction (E0I) S023-3-5.1 " Emergency Plant Shutdown" (Revision 8)
(this procedure is for any plant trip situation and includes the plant's ATWS procedural steps) included an instruction in the first step of the "Immediate Operator Action" section mquiring operators to verify that all reactor trip breakers are open and reactor power is decreasing. - If the mactor is not tripped, the procedure then requires that all four manual reactor trip pushbuttons be actuated.
In addition, the following specific ATWS actions are called for:
- a. De-energizing load centers B15 and B16. This interrupts power to the control element drive mechanism motor generator sets and, therefore, removes power from the CEDMs regardless of RTB position.
- b. Manually initiate emergency feedwater actuation signals.
- c. Initiate emergency boration.
Revision 9 of E0I S023-3-5.1 was issued on March 25,1983, and implemented additional ATWS steps that require the initiation of a C,)' "'"" ' ' '"' ' "'"" '""' ' ""*" '" '"' "" '
set point is rapidly being approached or has been reached regardless of whether the reactor has tripped automatically.
- 2. Control Room Layout / Design The control room layout drawing (Figure IV.C.2-1) identifies control board locations of indications and instrumentation which can be used to identify whether an ATWS event is occurring and what controls are available to aid in the mitigation or prevention of such an event.
All locations depicted on Figure IV.C.2-1 am numbered relative to the sequence in which they are contained in the "Immediate Operator Action" section of Revision 9 of E0I S023-3-5.1, " Emergency Plant Shutdown."
Figure IV.C.2-1, as drawn, shows only the San Onofre Unit 2 side of the control room. The "same hand" arrangement of the control room indications and controls identified in this drawing are located as shown in either the Unit 2 or Unit 3 control room ama.
As shown on Figure IV.C.2-1, the indications and instrumentation which can be used by operators to detect and initiate ATWS are ,
l easily accessible to operators.
- 3. Operator Training / Knowledge Following the issuance of IE Bulletin 83-01, on-shift training ,
"essions were conducted for each operating shift reviewing the ATWS IV-29 s --%.. . ,-ma w---, , , , . _ _
,,_m,. .. n.,.,--,,.m.,-~,_..--,,-w.,,,,,,m..__, -,,,.--y,.%-_.--,_,,,,,,.-.c. _,,.-
__ - .- . -=._ ..
4/15/83 operations included in E0I S023-3-5.1 " Emergency Plant Shutdown." In
, O addition, eac' licensed operator was required to review the IE Bulletin reftrenced above and IE Bulletin 83-04, when it was issued.
After the revision of E0I S023-3-5.1 on March 25, 1983, all licensed reactor operators were required to review and acknowledge, in writing, this review of the revision to the procedure. In addition, the five operating shifts will receive, as part of the requalification training program, fonnal classroom training and discussion relative to the revised procedure. This will occur over a 5-week period, which started the week of March 28, 1983.
O 4
lO l
l IV-30 L
A /' CN f \
( l }
\, \us I, m,' /
h RpS Remo t e Ope r nt o r 's Ho.f ul c-2 or more cliannels indicate trip i 2A Depress Ry t rip pushbuttons
. . . . . . . . . . . _ _ . . . 2 Depress Rg trip pushbuttons (t) h_ . . - - - - - - --
s h Renct or t rip l.k r statu e p:uir l verify all 8 1 cms opru g ,'
, N s,
's, s
h Excore linear power recorders-verify power <61 F.p.
,' ,' 's, \
s, h CEAverify Bottomed Indication -
' ' all CEA bottom lites on
/ H /, .' 's, '. ~ ~ ~ ~ ~ " - -nnuncinto;
, f 5h 's, i Dc-energize MCC Bis e e 's ' panel CI DM M/G Power Supply
. ---------- De-energize MCC B16
, e Initiate EFAS I&2
! t 8
! s ' Initiate EFAS 1 & 2 WIT 2 l l h Initiate emergency boration:
. I - Pumps
! p h - CVCS Valves & Blend Control I
h Emerg. boration path f
6 '
, h Gravity feed wahrs for boraticn
', 9 Hanually trip turbine
, s s
's Verify feedwater flow to S/cs (MFWand/or AFW)
, 10
's .s 1I Annunciator panel -
's s 'N 50A31 par Safety Vv out. Terig. Mi
's ~, ' 50A49 CEDMCS Bus Undervoltage JLesk . .. & ..-- U' .- ----
h boronometer - verify RCS between
_ EancL ____________... --. 1750 and 2250 ppmB before stopp-ing emergency boration Figure IV.C.21 CONTROL ROOM LAYOUT / DESIGN ATWS INSTRUMENTATION / CONTROLS a
j N w
t.rl N
CD w
l 4
4/15/83 I s D. Findings / Conclusions Regarding RTB Failum
- 1. The reactor trip bmaker undervoltage trip device is not required for i the San Onofre Units 2 and 3 reactor protection system (RPS) to perfom its design basis protective function; failum of the UY trip device does not affect the ability of the San Onofre Units 2 and 3 4
RPS to perfom its protective system function due to the presence of the shunt trip.
i
- 2. From the results of the investigation of four RTBs discussed in Section IV.A of this report, it is concluded that the cause of the unm11able operation of the undervoltage trip device for the reactor trip breakers was a combination of the small design margin in the l
force provided by the UV trip device and the following: ,
o Degraded lubricant on the trip shaft bearings or latches.
5 o Incormet setting of the UV device pickup voltage.
o Excessive clearance in UY device amature hinge area.
- 3. Despite the small design margin in the force provided by the UV trip device, the UV trip feature of the RTB can be made to operate reliably when enhanced maintenance and surveillance techniques are
, used. This conclusion is based on the results of the tests and inspections of three RTBs at San Onofre, the results of in-depth testing of one RTB at the SCE Electrical Test Laboratory, and the results of the review of RTB maintenance and surveillance history.
- The enhanced maintenance and surveillance program, based on these
- results, is as follows:
- a. Maintenance i
Initial baseline data will be taken and the enhanced maintenance procedures for RTBs will include the following:
l 1) Trip shaft torques, UV device pickup voltages, and UV opening times should be detemined for the breakers in their as-tound condition:
a) Measure and record the pickup voltage of the UV device "
with its coil surface temperature within 70 to 850F.
Also, measure the pickup and dropout voltages after the UY device coil has been energized at 130 volts dc for 30 minutes. Ambient and coil surface temperatures should also be measured and recorded.
b) Measure the opening time of the circuit breaker with full loss of voltage to the UV device. This time is to be measured from the loss of de control voltage and to the l breaker main contact part. Three operations and measurements are to be made using an oscillograph recording device, with a test diode connected across the i UV device coil.
- 1Y-31 t --- - - - - - - - _ - -- - - _ _ - - - . . -
. - . - - - . - . - . - - . . = _ - . . . .
4/15/83 c) Measure and record the trip shaft torque which is O required to trip the circuit bmaker. This measurement is to be made three times using a specially made tool to ftt on the trip shaft.
, 2) Prior to the current plant restart, the RTB should be i
disassembled to the point where the operating mechanism can i
be removed. The trip latch roller should be inspected for ,
damage and all bearings and bearing points should be checked for mechanical wear and freedom of rotation. Clean all parts with isopropyl alcohol. Apply CRC 5-56 to all bearings and bearing points. All excess CRC 5-56 should be removed to pmvent the collection of dust and contaminants.
j 3) Remove the UY device to pemit inspection and to check the
! adjustment of the clearance between the UV device amature and the rivet (0.003 to 0.006 inch). The mechanism portion is to be cleaned with isopropyl alcohol and then apply CRC 5-56 to all pivot points. Excess CRC 5-56 should be removed.
i 4) After reassembly of the circuit breaker, the mechanical adjustments should be checked and adjustments made as required.
- 5) ' Repeat the circuit breaker perfomance measurements:
l
' a) Check and adjust the pickup voltage of the UV device when its coil surface temperature is within 70 to 850F This should be set to 106 +2 Vdc. Also, measure the
! pickup voltage after the UY device coil has been energized at 130 volts dc for 30 minutes.
b) Measure the opening time of the circuit breaker with full j loss of voltage to the UV device. This time is to be i measumd from the loss of de control voltage and to the l bmaker main contact part. The criteria for the main t
contact part shall be approximately 70 milliseconds or less, subject to confimation by baseline testing of all l breakers. Three operations and measurements are to be made.
c) Measure and record the trip shaft force to trip the cin:uit breakers. This force shall not exceed 1.5 pounds-inches. This measurement is to be made three times using a specifically made tool to fit on the trip shaft,
- b. Surveillance Enhanced surveillance for RTBs will be made to monitor the performance of the circuit bmakers for indications of degrading n UV trip operation, in order to determine the need for (and to g pemit perfomance of) restorative maintenance before UV trip IV-32
_ ~ . _ _ _ . _ . _ . _ _ _ , . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ . _ -
- - . _ ~ . -
4/15/83 mliability is affected. The enhanced surveillance for RTBs includes the following:
- 1) _ After the circuit breaker maintenance is completed, a surveillance test should be made. This .should include three UV opening time tests. The opening times for the three tests should not exceed the allowable limits shown in Item 5b under maintenance, above.
- 2) Repeat the surveillance test four more times at approximately 30-day intervals (i.e. , monthly).
l 3) After four sequential months of UV trip function surveillance
- testing without UV trip failure or degradation, the
. surveillance testing interval may be increased to at least once per 62 days (i.e., bimonthly).
- 4) After eight more sequential months of successful UV trip testing, the surveillance testing interval may be increased to at least one per 92 days (i.e. , quarterly).
- 5) After 12 more sequential months of successful UV trip testings, the surveillance testing interval may be increased to at least once per 184 days (i.e., semiannually).
- 6) If the average of the three UV trip times or the scatter of the thme trip times during each test increases significantly l- from the baseline surveillance tests, or exceeds the l baseline-adjusted limits, an evaluation will be perfomed to detemine if the maintenance procedures should be repeated and new surveillance tests initiated.
! 7) Surveillance and maintenance frequency should continue to be adjusted on the basis of surveillance and maintenance test i results. The minimum RTB maintenance frequency should be once every 12 months (i.e. , annual PM). Unit 3 surveillance (and maintenance) should be initiated at the frequency in effect for Unit 2 at that time.
- 4. The high incidence of equipment difficulties experienced during the startup phase hindered early identification of the RTB problem.
Based on this and the conclusion that the UV trip device failures were caused by factors which were not previously clearly known (i.e.,
temperature effect on UV coil settings and quantitative values required for amature to rivet clearance), it is concluded that the likelihood of the RTB malfunctions experienced may have been reduced but would not have been precluded if administrative weaknesses had not existed in SCE programs.
S. A procedure which addresses ATWS was in place at San Onofre Units 2 and 3 prior to the Salem RTB trip failure. This procedure, S023-3-5.1 " Emergency Plant Shutdown," has been revised to improve O operator response to an ATW5 event by requiring immediate manual reactor trip and then manual turbine trip when an automatic trip set IV-33
4/15/83 point is being rapidly approached, regardless of whether the reactor
[m U) '
has tripped automatically. The San Onofre Units 2 and 3 control room is already configured such that the instrumentation and controls to detect and mitigate ATWS are easily accessible to the operators.
- 6. The existing SCE operator training / retraining program contains sufficient provisions for promptly notifying operators of important procedural training andchanges and ]for additional reinforcement through follow-on retraining
- 7. Potential improvements to SCE administrative procedures and implementation have been identified in the following areas as a result of the reviews discussed in Section IV.B of this report (and are either completed or in progress as discussed in Sections V.A and V.B of this report):
- a. Provide additional training for verbatim compliance and supervision of vendor work consistent with SCE procedures and develop a procedure for use during the operating phase.
- b. Assess the extent of safety related vendor work which did not utilize SCE procedures. (The approximately 20,000 safety related work orders since Unit 2 fuel load have been reviewed and 41 such cases identified for further review and potential corrective action.)
p V
- c. Provide additional training in use of existing SCE procedures to control vendor-supplied infomation and continue implementation of the comprehensive configuration control program to manage all vendor-supplied infomation,
- d. No improvements have been identified for SCE administrative procedures or implementation for spare parts procurement; however, during a source audit of the RTB vendor by SCE, it was detemined that certifications for the five AK2-25 RTB spares could not be satisfactorily established due to lack of fomal quality assurance program requirements in vendor procurement documentation relating to subvendors. Consequently, the configuration of the spare RTBs will be documented to SCE satisfaction or the spare RTBs (including spare parts /
subassemblies used in other RTBs) will be rejected and replacements procured.
- e. Review the preventative maintenance program to establish adequate baseline conditions and adequate PM intervals. (A review against all specific regulatory requirements and commitments will be complete by April 30, 1983.)
- f. Establish an operating phase experience feedback program for PM.
- g. Provide additional training in use of available SCE maintenance
, procedures rather than technical manuals or other vendor data in j
i (O) maintenance and work planning.
IV-34
4/15/83
- h. Review maintenance mcords to verify that correct post-maintenance testing has been perfomed.
- i. Develop more definitive guidelines for post-maintenance test requirements.
J. Implement changes to NCR program discussed in Section IV.B.6 for tracking actions to prevent recurrence, and for clarification of operability and reportability assessment action; implement changes to identify and evaluate recurring non-confomances.
- k. Review all IE bulletins, circulars, and notices to verify that these documents have been referenced and included in procedures as mquired.
- 1. The fomal post-trip restart procedure as discussed in Section IV.B.8 has been implemented.
- m. Implement administrative controls to ensure that PMs are perfomed for replacement components, where appropriate, prior to their use.
O lO IV-35 l
4/15/83 V. PROPOSED CORRECTIVE ACTIONS U Short-tem and long-tem corrective actions are summarized below based on discussions pmvided elsewhere in this report.
A. Short-Tem Corrective Action Leading to Restart
- 1. Control of Hardware, Vendor Information, Vendor Personnel
- a. Control of Hardware As discussed in Section IV.B.2.b, spare RTB; were provided by CE subsequent to delivery of the original set of 18. These spares involved different CE subvendors and they were manufactured and modified at different locations than the original set. The SCE investigation has identified a number of questions which require resolution. Although none of the spare breakers were involved in the surveillance test failures of March 1 and 8,1983, they will not be used in safety-related applications until these questions are satisfactorily resolved and documented. Only RTB from the original set of 18, which have been verified to confom to all applicable requirements, will be utilized in the meantime,
- b. As discussed in Section IV.B.3.a. , the SCE investigation has identified that failure to explicitly reference IE Bulletin 79-09 in the procedure originally written to provide for its 7 implementation contributed to deleting important information when (V the procedure was converted for eventual use in the operating phase. Accordingly, as discussed in Sections IV.B.3.a. and IV.B.7.b., a thorough review is being made te verify that other applicable IE bulletins, circulars and notices have been referenced and included in procedures where required.
- c. Control of Vendor Services As discussed in Section IV.B.l.b., an adequate administrative program exists for control of vendor services. In the case of the RTB, this program was not fully implemented. Forty-one similar cases of use of vendor services are being reviewed to ensure adequate control was provided in these cases. In addition, as discussed in Section IV.B.3.c., procedures are being strengthened to provide added assurance that control and training is being conducted for cognizant personnel in proper implementation of the program.
- 2. Maintenance Procedures and QA/QC Requirements l Cormctive actions associated with maintenance procedures and QA/QC requirements are discussed in Section IV.B.3.a. These corrective l actions respond to problems with maintenance procedures which are also discussed in Section III.C.4.c and include training in verbatim l n compliance and control of vendor services, review of existing work I
V-1
4/15/83 orders for control of vendors, and review of IE bulletins, circulars, and notices for PM program requirements.
())
'w Corrective actions associated with the development of nonconfonnance reports which lead to implementation of corrective maintenance are discussed in Section IV.B.6.
A completely revised RTB maintenance procedure incorporating the experience and findings of the SCE investigation has been developed by SCE and reviewed by CE and GE. All comments have been resolved and this procedure has been implemented on all RTBs. Based on this, and performance of the revised surveillance testing discussed in the following section, SCE is confident that the RTBs can be returned to operable status. For ongoing and future maintenance work on safety-related equipment, the implementation of QA/QC requirements will emphasize compliance with adequately detailed procedural steps.
- 3. Surveillance Procedures As discussed in Section IV.D.1, surveillance testing of the RTB will incorporate conclusions identified during the investigative tests conducted by SCE. The details of this surveillance testing will be as described in Section IV.D.3.b of this report.
If at any time during this surveillance program, degradation or failure of the undervoltage device occurs, the failure will be p analyzed, corrected, and used to enhance the preventative maintenance procedure to prevent recurrence. Additionally, the surveillance V testing program will be re-zeroed to the beginning of the interval phase in progress to ensure successful correction of the problem prior to extending the test interval.
Unit 3 surveillance and maintenance will be initiated at the same frequency in effect for Unit 2.
The monthly PPS testing under S023-II-1.1, which tests the RTB shunt and undervoltage t<*ips together, will be unchanged.
- 4. Technical Specification SCE will review the need for Technical Specification changes after the appropriate maintenance and surveillance frequencies have been identified.
- 5. Operator Training As discussed in Section IV.C. of this report, the Emergency Plant Shutdewn Procedure for San Onofre Units 2 and 3, which contains operating instructions for any plant trip situation, including ATWS procedural steps, has been modified to require the initiation of a manual reactor trip and to manually trip the turbine when an g automatic reactor trip set point is rapidly being approached or has b
Y-2
4/15/83 m been reached regardless of whether the reactor has tripped automatically.
All licensed reactor operators have acknowledged in writing the review of the revision to this procedure. Additionally the five opeMting shifts will receive, as part of the requalification training program, fomal classroom training and discussion relative to the revised procedure. This will occur over a 5-week period which started the week of March 25, 1983.
SCE is confident that the procedural revision along with the review of the revised Emergency Plant Shutdown Procedure, which is being reinforced through the operator requalification training program, will enable operators to detect and mitigate ATWS.
B. Long-Term Corrective Actions Subsequent to Restart
- 1. As discussed in Section IV.B.l.a., a comprehensive configuration control program is being implemented to support the operational phase. This program includes provision for complete control of vendor infonnation and has been under development for approximately 1 year. It will provide for resolution of conflicts in information similar to those identified in Section IV.B.l.c.
- 2. As discussed in Section IV.B.3.a, training is being developed for all cognizant personnel to emphasize use of available maintenance S procedures in lieu of technical manual references and to review the (d need to identify all pertinent vendor-supplied information for work order planning where procedures are not available. This effort is expected to be in place by July 1,1983.
Implementation of the operational phase maintenance history and 3.
records program is discussed in Section IV.B.3.b. This will provide for easier retrieval of this infonnation, including surveillance testing experience as discussed in Section IV.B.4.c.
4 As discussed in Sections III.C.4.b.(3) and IV.B.4.b.(2)(d), one instance was identified by the SCE investigation in which a functional test was specified instead of the surveillance test following undervoltage trip device maintenance. The functional test did not independently verify operability of the undervoltage trip device. Improved guidance will be developed and provided to cognizant groups to avoid recurrence of this error.
- 5. As discussed in Sections IV.B.6.b., a program to provide for independent review and evaluation of repetitive nonconforming conditions will be developed and implemented.
- 6. As discussed in Section IV.B.3.c, a procedure to control vendor services during the operating phase will be developed.
O V-3
.- .= .. - - . - _ _ . . - .-.
l 4/15/83
- 7. As discussed in Section IV.B.7, the PM program will be reviewed to:
O 1) establish baseline conditions and adequate PM intervals;
- 2) establish an operating phase experience feedback program for PM; and 3) implement administrative controls to ensure that PMs are .
performed for replacement components where approprwte. !
l i
i O
O V-4
4/15/83 VI. CONCLUSIONS O A. Conclusions Resulting from the SCE Evaluation of Reactor Trip Breaker Performance
- 1. From the standpoint of the reactor protection system design, the UV coils are not required to operate in order for the breakers to trip under any design basis condition. In addition, the conservative reactor protection system design at San Onofre provides added assurance (compared to similar vintage plants of other designs) that 4 an ATWS event will not occur.
- 2. Exhaustive breaker testing in SCE's laboratories was performed on one of the breakers that exhibited unsatisfactory performance. SCE was able to do this evaluation because of its experience and capabilities for testing of this nature that are largely unique to SCE. The
, experienced staff and facilities for such testing do not exist in most utilities throughout the country. This testing identified three i key areas where additional attention needs to be devoted during maintenance and surveillance. These are:
, a. Lubrication of key components
- b. Performance of UV coil adjustments at a known temperature
- c. Maintenance of the UV armature to rivet clearance within recomended close tolerances.
j Based on the results of the investigative testing, it is clearly
! understood how to adjust and maintain the breakers to provide a high 3
degree of reliability for the UY trip function.
- 3. The proposed maintenance and surveillance programs will provide this high degree of reliability. The initial conservative frequency will be adjusted based on actual experience.
- 4. Based upon the above, and the fact that the shunt coil nevertheless tripped the breaker when the UV coil failed, there is a very high degree of assurance that the RTB will operate if called upon to do so. Therefore, the health and safety of the public are assured and any and all restrictions limiting operation of San Onofre 2 and 3 associated with the RTB should be removed.
B. Conclusions Resulting from the SCE Evaluation of Administrative Processes Detailed conclusions have been presented in Section IV.B of this report.
In general it has been concluded that:
- 1. Based on the evidence that the UV trip device failures were caused by factors which were not previously clearly known (i.e., temperature effect on UV coil settings and quantitative values required for armature to rivet clearance), SCE concludes that the likelihood of O the RTB malfunctions experienced may have been reduced but would not i
}
VI-1
4/15/83 have been precluded if administrative weaknesses had not existed in SCE programs.
- 2. Investigations completed to date, and short-term corrective actions already implemented, provide assurance that San Onofre Units 2 and 3 may safely resume their startup programs without undue risk to the health and safety of the public.
- 3. Long-tenn corrective actions will incorporate lessons learned from the RTB investigation into appropriate areas of the SCE administrative program.
- 4. The SCE operating organization has received adequate training and guidance in ATWS situations to ensure that such events would be appropriately handled in the unlikely event one should occur.
i l
lO VI-2