ML11242A081

MEng-SEng Report

ML11242A081
Person / Time
Site:	San Onofre
Issue date:	07/17/2008
From:	Southern California Edison Co
To:	Office of Information Services
References
FOIA/PA-2011-0157
Download: ML11242A081 (34)

Text

ISOUTHFMCAI IFORtNIA 7/17/2008

-2:19:33 PM EDISON SONGS -MEna-SEnq.

Report: ASSESSMENT OF THE IMPACT OF THE DEGRADED CONNECTIONS BETWEEN BREAKER 2D201 AND BATTERY 2B008 ON THE FUNCTIONALITY-OPERABILITY OF THE BATTERY 2B008 Executive Summary: During a weekly surveillance test of battery 2B008 and normal charger 2B002 carried out on March 25, 2008, two degraded connections (one per polarity) were discovered between the battery and breaker 2D20 I.Subsequent investigations yielded the following conclusions:

I. The probable cause for the degraded connections is the under-torquing of the connection bolts during the replacement of the breaker (2D201) carried out on 3/17/2004.

2. There is reasonable assurance that the degraded connections did not compromise the capability of the battery to perform its design-function until the end of the normal charger test on 3/21/-08.

From 3/17/08 thru 3/21/08 the swing-charger was in operation while the normal charger was out of service for testing. This conclusion is supported by plant data which indicates the connection resistance likely changed state upon completion of the 3/21/08 normal charger testing. This conclusion is supported by test data previous to this date which includes weekly battery surveillances, two successful Integrated ESF surveillance tests, two successful LOVS surveillance tests, as well as data from the four day period when the swing charger supplied 112 amps to the 2D2 bus through the degraded connections (from 3/17/08 to 3/21/08).3. Although there is no plant data to evaluate the condition of the connections during the period from 3/21/08 until the failed surveillance test on 3/25/08, it is likely that the degraded connections would have been capable of performing their design function from 3/21/08 until the degraded condition of the connection was discovered on 3/25/08. This conclusion is based on the fact a very thin layer of oxidation across the joint and/or a small gap would have been readily breached by the 120 VDC applied to it during a real event. The degraded connections may have impacted battery capability for coping with Station Blackout (SBO) design basis. Due to the lack of plant data, SCE conservatively considers the battery to have been nonfunctional sometime during this period.4. Seismic events would not have prevented the battery from meeting its design basis as a result of the degraded connections.

File: Degradation of DC connection D201 I of 34 7/17/2008

-2:19:33 PM JDISrAOFN(AFR An IOt$3 IMURA4'10f4l1.

C "paný

Background:

On 03/25/2008 a weekly surveillance voltage test on 125 volt battery 2B008 failed its acceptance criteria.

The test consists of verifying the battery voltage is at least 129 VDC when measured at the battery terminals.

During the test, the measured voltage was approximately 121 VDC. During subsequent troubleshooting activities, degraded connections (one per polarity) were found on the battery side of the 2D201 breaker which connects the 2B008 battery to the 2D2 bus. Specifically, the bolts connecting the terminals of the 2D201 breaker to the aluminum buses leading to the battery were not fully tight. The battery terminal voltage recovered to 131 VDC when a voltage probe handled by the electricians was touched to one of the under-torqued connections during troubleshooting.

This report evaluates the impact of the under-torqued connections on the associated DC system, and the ability of the 213008 battery to perform its credited safety functions during the period from 3/17/04 to 3/25/08. This report is not intended to address human factors, which are discussed in a separate RCE.Definition of Terms: LOVS: Loss of Voltage Signal ESF: Engineered Safety Features SIAS: Safety Injection Actuation Signal DGVSS: Degraded Grid Voltage with SIAS Signal SBO Station Blackout ADC DC amperes VDC DC volts Factual Information:

1. The surveillance test that failed was carried out on 03/25/08 (MO #080301117).

The measured value was 121 VDC. The required minimum is 129 VDC.2. Previous surveillance tests were carried successfully on 3/18/08 and every week before that. A complete table of weekly surveillance tests is included in Attachment 2.3. Two Integrated ESF Surveillance Tests and two LOVS Surveillance tests were successfully completed during the four year period between 3/17/04 when the 2D201 breaker was replaced, and 3/25/08 when the weekly surveillance test was failed.These tests are designed to closely mimic the actual accident loading conditions on the battery bus. Successful completion of these tests provides a measure of confidence that the battery was capable of performing its design function.

During approximately the first 15 seconds of these tests, the battery charger power is removed, so the battery is required to supply all required power to the bus through the impacted connection.

The data below shows how the voltage decreased on the 125 VDC supply at switchgear 2A06 during the period of time when the battery charger was off. Graphs File. Degradation of DC connection D201 2 of 34 l SM!11111-RN CAI tFUR~NIA 7/17/2008

-2:19:33 PM EDISON An EVIS N TNA 47t0'4. COmpmn)obtained during the tests are included (Figures 1-3). To see where the voltage drop is measured, see Attachment 6.Data is included from two ESF tests prior the 2D201 breaker replacement and two tests after the replacement.

The data shows there was no significant change in voltage drop after the breaker was replaced.

This supports a determination that the connection at the 2D201 breaker was sufficient to allow the battery to perform its safety function.Data is also included from one LOVS test prior to the 2D20I breaker replacement and two LOVS tests after the replacement.

The data shows one case with a slight increase in voltage drop (0.7 to 2.3 VDC). This occurred after the breaker was replaced.

The measurement is affected by inverter-noise and has an uncertainty of approximately

+/-I volt, rendering the data inconclusive.

It is judged that there would be sufficient margin in the battery design to accommodate a I to 2 volt change in voltage drop, based on the fact that the battery capacity test in 2008 determined a capacity near 100% while the calculation is based on a capacity of 80%. In the table below, all tests prior to the change of breaker 2D201 on March 2004 are noted as "TIGHT". Those after the breaker replacement are noted as "NOT TIGHT" (Refer to figure in Attachment 6).Charger-off Connection's MO Date Test VDC Drop Time (sec) Condition 06122943 11/29/07 ESF 11.04 13.7 NOT TIGHT 05081829 1/6/06 ESF 10.97 13.4 NOT TIGHT 03031098 2/11/04 ESF 11.42 14.5 TIGHT 01061449 5/24/02 ESF 11.26 14.6 TIGHT 08010611 1/9/08 LOVS 12.32 15.0 NOT TIGHT 06011546 1/21/06 LOVS 10.70 14.8 NOT TIGHT 03031098 2/20/04 LOVS 9.98 15.8 TIGHT 4. The breaker 2D201 was replaced on 03/17/04.5. The last time the bolts on the impacted connection were disturbed was during the breaker replacement on 03/17/04 [8].6. The affected bolts were tightened to their specified torque during the repair activities following the failed surveillance test of 03/25/08, and according to the maintenance personal involved, each bolt was tightened using a torque-wrench.

7. Each bolt is fastened with two flat washers and a lock-washer.

8. Interview with the two craftsman that tightened the bolts after the problem was discovered revealed the following facts: a. Each bolt was touched by hand while wearing insulated rubber gloves. Only one bolt was found completely loose.b. Two of four bolts on each connection were snug enough that they couldn't be moved without the application of torque with a wrench. The remaining bolts could be barely moved by hand.File: Degradation of DC connection D201 3 of 34 SOMI11RN CAI IFIONIA 7/17/2008

-2:19:33 PM EDISON c. No evidence of sparking or pitting of bolts, washers and buses (restricted visual access only permitted a limited view of the connection).

d. All eight nuts were fully threaded in their bolts.e. The condition of the grease in the contact area was not inspected.

f. Did not count number of turns applied on the nuts to effectively apply the specified torque. The ratcheting torque-wrench requires a number of strokes to move the nut a full turn.g. Bench tests previous to the interview show that it takes a full turn of the nut to flatten a lock-washer.

h. When asked what is the typical manner they will use to tighten the bolts, one of them said:-First by hand till the lock-washer is engaged,-Then with a regular wrench,-Finally with a torque-wrench.

9. Phone interview with the electrician that applied the Fluke probe to the degraded connection, revealed the following facts: a. Several minutes after the weekly surveillance test failed (battery voltage measured below acceptable limit), and following a phone call, he reached the charger room to perform troubleshooting tests.b. After taking a couple of readings on the charger side with satisfactory results, he touched the side of the head of one bolt of the degraded connection.

c. He applied little pressure with the probe and cannot say for certain that the bolt moved, although he thinks he perceived a very minor movement.d. At the time the probe touched the head of the bolt, the sound from the charger or inverter changed, apparently indicating the loading changed.e. Then he returned to the battery and now the voltage readings were satisfactory.

f. Looking at the busses and bolts with the panels removed, he could not see any signs of discoloration or other symptom related to high temperature on the busses and/or fasteners.

g. At about the same time an operator came to say that there was a Trouble Alarm on 2D2. The alarm was gone immediately (likely caused by a momentary ground imbalance when the battery was reconnected to the circuit).10. Breaker 2D201 has a magnetic trip circuit, rendering it insensitive to temperature rise of its terminals due to abnormal heat generated at the degraded connection (if any)(MO #04021613).

11. Swing charger 2B022 continuously fed 112 ADC thru the degraded connection between 3/17/2008 and 3/21/2008 (MO #08011467000).

12. The joint in question is designed for 1,200 continuous amperes of current (Reg. Dwg.S023-302-5A-2).

This is about four times the level of current this joint must carry continuously during an accident.File: Degradation of DC connection D201 4 of 34 54 SURIMtR CA1IFORNIA 7/17/2008

-2:19:33 PM J SO ON An 01(X0$% p. Y.R% 4110% 41.13. A Ductor test on the re-torqued connections upstream of 2D201 carried out on 7/15/08 (NMO #800117881), show acceptable contact resistance of no more than about 10 Analysis: The following questions are addressed in this section: 1. Most probable cause(s) for the degraded connection.

2. Was the degradation the result of a single event, or of a developing nature?3. Did the degraded connection impact the ability of the DC System to perform its credited safety functions?

4. How would have a seismic event affect the integrity of the affected connections?

1. Most probable cause(s) for the degraded connection A bolted connection can become loose over time by a number of mechanisms.

The table in Attachment I identifies the most probable mechanisms for looseness, and their relevance to the present situation:

As shown therein, SCE concludes that the most probable cause is that the bolts were not fully tightened during the breaker replacement of March 2004, as required by the relevant procedures.

2. Was the degradation the result of a single event, or of a developing nature As explained in the previous section, the root-cause for the final onset of the degraded connection can be said to be due to a single event: the incorrect bolting activity on March 2004. However, due to the original oversight, there also was a potential for deterioration of the connection, given proper conditions; for instance, the drying out of the conductive grease applied between the busses during the breaker replacement.

As explained below, a connection may lose pressure and still have a very low resistance.

However, a connection which is not properly tightened may provide an opportunity for oxidation of the aluminum surfaces, suddenly increasing the contacts resistance to low voltage, which appears to be the case in the incident addressed herein.The following provides the most probable explanation for the final failure of the weekly surveillance test: During the March 2004 breaker replacement activity the electricians applied to the affected bolts enough torque to put some compression on the lock-washers, without fully tightening them. In doing so, they might have used a regular wrench, based on the fact at File: Degradation of DC connection D201 5 of 34 7/17/2008

-2:19:33 PM J EDISON An frll,.S least two bolts per connection were found tight enough they could not be moved by hand (see item 8 under "Factual Information", above). The incomplete activity would have left the connections with a pressure less than required by the pertinent procedure.

If a proper surface preparation was made, and anti-oxidant grease applied, this activity would have resulted in good contact (low contact resistance).

In Figure 4, the anti-oxidant grease is visible on the edge of the bus joints. Also, as indicated in Item 12 under "Factual Information", the contact surface is designed with a current-carrying capacity of about four times the required under an accident, providing ample margin to carry the required current, even if torquing of all or some of the bolts did not meet existing procedures.

Between 3/17/08 and 3/21/08, the swing charger was placed 'in-service on 2D2 which caused about 112 amps to be supplied through the degraded connections.

This resulted in increased heating of the connections for about 4 days, followed by a cooling of the connections when the swing charger was removed from service. Increased heating can accelerate the rate of oxidation on an under-torqued joint allowing the formation of a very thin layer of oxidation on the aluminum surfaces (about 50-120 angstroms thick [5]).Increased heating can also accelerate the drying of the conductive grease, which can increase joint resistance.

The normal charger output voltage is about 131 VDC and the.open circuit battery voltage is 120 VDC, so a developing layer of oxide or dried grease at the breaker connections would be exposed to a maximum of II VDC. This voltage may not be high enough to puncture a developing surface layer of oxidation or grease. However, if a real safety event were to occur, the entire 120V DC would be applied to the oxide layer. This potential is sufficient to break the thin layer of insulation and/or small gap recreating the conducting joint. Contact theory indicates that there is a threshold voltage, called the"fritting" voltage, associated with the breakdown of surface oxides or other contaminants between conducting surfaces.

Below the fritting voltage, the surface materials act as an.electrical insulator preventing current flow. Above the fritting voltage, electrical stress causes an avalanche breakdown of the surface insulator and conductivity is restored [4]. The fritting voltage necessary to break contact resistance can vary from several volts to several hundred volts depending on the thickness of the surface buildup, the pressure between the surfaces, the materials involved and other environmental factors. From relay applications, contact surfaces with less than 24 volts across them are known to be susceptible to high contact resistance from surface oxidation or contamination.

Based on calculations and instrument tolerances, the joint resistance until the end of the swing charger test of 3/17-21/08 is calculated to be between 0 to 8 mQ. The calculations are included in Attachment

3. The acceptable joint resistance value of 5.4 mQ (Attachment

4) is statistically within the range 0-8 mQ. A change in the total resistance of the circuit (attributable to the degraded connection) can be established as happening after 3/21/08.Using "as-left" data for the charger voltage and voltage measurements taken on the battery terminals, and conservatively assuming (based on the fact the battery was fully charged) a charging current of no more than 0. 1 ADC, the calculation for the resistance of the degraded connection yields a value in the range of 0-5.7 Q at the time the regular charger took over from the swing charger (See Attachment 3). Statistically, this resistance is orders of magnitude higher than the connection resistance of 3/17/08.File: Degradation of DC connection D201 6 of 34 7/17/2008

-2:19:33 PM I EDISON~~An FOISN ' % (If 4 M) .pan The change that happened with the reversal of current from the swing charger to the normal charger can be explained as follows: Following the 4-day operation of the swing-charger (3/17/08 thru 3/21/08), the joint degraded.

We believe the current flow thru the connection dried the grease on the contact surfaces and allowed the formation of an aluminum oxide layer. This layer of oxidation created enough resistance to withstand the very low potential differential between the charger and the battery. As explained above, this oxidation layer, and its corresponding fritting voltage would have been breached by the full battery voltage during a real event. A second possible cause (or in addition to the previous one), is that a small gap was created due to the geometry change following cool-down of the unloaded joint). The interview with the electrician that restored the circuit integrity by just lightly touching one bolt of the connection demonstrated that the oxidation layer or/and gap was indeed very weak/small.

3. Did the degraded connection impact the ability of the DC System to perform its credited safety functions?

The safety function of the 2B008 Battery is to provide power to the loads on the 2D2 Bus during three types of accident scenarios:

Safety Injection Actuation Signal with Loss of Voltage Signal (SIAS/LQVS), Degraded Grid Voltage with SIAS Signal (DGVSS) and Station Blackout (SBO).The design-basis bus loading during the various applicable scenarios is: CONNECTION LOAD DURING SIAS/LOVS or DGVSS (normal charger on): 0-1 MIN = 474 ADC 1-90 MIN = 0 ADC CONNECTION LOAD DURING SIAS/LOVS or DGVSS (swine charmer on): 0-1 MIN = 474 ADC 1-90 MIN = 89 minutes of 193 ADC with one minute spike of 312 ADC SBO: 0-1 MIN = 341 ADC 1-30 MIN = 193 ADC with one minute spike of 312 ADC 30-240 MIN = 155 ADC with one minute spike of 239 ADC Evaluation of SIAS/LOVS and DGVSS scenarios (with normal charger in service): During a SIAS/LOVS event or a DGVSS scenario, the battery is designed to provide adequate voltage to the 2D2 Bus loads for 90 minutes. Technical Specifications require periodic verification of the ability to supply the bus loads for 90 minutes by performing a Service Test on the battery.File: Degradation of DC connection D201 7 of 34 SOMMIl" (AI 7/17/2008

-2:19:33 PM I EDISON An if~(~JJrR1NVfA.4ff%41.-

CMwnpv'During these scenarios, offsite power is lost and it takes approximately 10 seconds for Diesel Generator 2G003 to restore power to the 4KV system which will restore AC power to the battery charger. Once AC power is restored, the charger takes approximately 5 seconds to develop a DC output current due to the soft-start circuitry in the charger. Based on this, the 2B002 battery charger will be unable to support the 2D2 bus loads for approximately 15 seconds during SIAS/LOVS and DGVSS accident scenarios.

During this 15 second period, the 2B008 battery would be required to supply the 2D2 bus loads through the 2D201 breaker with the degraded connections.

During this 15 second period, the maximum design loading through the degraded connection is 474 amps for the first second and 340 amps for the remaining 14 seconds. Following this 15 second period, the battery charger will be restored to service. The charger is designed to carry 325 amps of load. The charger current supplied to the 2D2 bus will not pass through the degraded connections on 2D201 because the charger is connected to the 2D2 bus through a different breaker. All bus loading, up to 325 amps, will be supplied by the charger rather than the battery, once the charger recovers AC power.During the period from 15 seconds to 60 seconds, the maximum bus loading is 338 amps.Since the charger would assume 325 amps, only 13 amps would be supplied by the battery through the degraded connection.

During the period from 1 minute to 90 minutes, the maximum bus loading is 245 amps so all of the current would be supplied by the charger and no current would be supplied by the battery through the degraded connection.

Based on this, the 2D2 DC System can perform its safety function if the degraded connections on the 2D201 breaker can carry the following current profile without adverse impacts: 464 amps for 1 second, followed by 340 amps for 14 seconds, followed by 13 amps for 45 seconds. Based on the short duration of this loading, it is concluded the degraded connection were capable of carrying the accident loading without significant degradation.

The connections passed all weekly surveillance tests (Ref Attachment

2) and four current-carrying tests [2] during the 4 year period of interest.

The Integrated ESF tests and LOVS tests are designed to closely mimic the accident loading conditions on the bus This provides confidence that the degraded connections did not impact the ability of the battery to supply adequate voltage to the 2D2 loads during accident conditions..

From the list of tests indicated above, traces from three of the tests are presented herein:* Figure 1 shows a trace from an ESF test carried out on 2/11/2004.

• Figure 2 shows a trace from an ESF test carried out on 11/29/2007, almost three years after the 2D002 breaker replacement." Figure 3 shows a trace from an LOVS test carried out on 1/28/2008, just a few weeks before the discovery of the degraded condition of the connections.

All three tests show time durations

-when the battery charger was de-energized for approximately 15 seconds, and the degraded connections were conducting all of the 2D2 bus loads under simulated accident conditions.

Item 3 in the Factual Information Section of this report contains a table which summarizes the results of the Integrated ESF Tests and LOVs File: Degradation of DC connection D201 8 of 34 7/17/2008

-2:19:33PM Ji ED ISON tests. The data supports a determination that adequate voltage is being maintained on 2D2 when accident loading is being supplied through the degraded connections.

Satisfactory weekly surveillance tests from the time of the breaker replacement on 3/17/04 until the one on 3/18/08 (Ref Attachment

2) indicate the affected connections, although not fully torqued during the maintenance activity of March 2004, were able to carry charging current without any problem until the last satisfactory weekly battery surveillance test, on 3/18/08. This indicates that the battery remained fully charged during this 4 year period.Evaluation of SIAS/LOVS and DGVSS scenarios (with swing charger in service): There was a 4-day period between 3/17/08 and 3/21/08 when the 2D2 bus was being supplied by swing charger 2B022 rather than the normal charger 2B002. The swing charger is connected at the battery terminals, so its output current would have been supplied to the 2D2 bus through the degraded connections on the 2D201 breaker. In this alignment, if an accident were to occur, the degraded connections would have to carry the entire bus load for the full duration of the event because the charger current goes through the degraded connections.

The load profile is 474 amps for 1 minute, followed by 193 amps for 89 minutes with a 1-minute spike to 312 amps during the second period (due to a consideration of random loads). It is noted that during the 4-day period between 3/17/08 and 3/21/08 when the swing charger was in-service, it was supplying approximately 112 amps through the degraded connections continuously.

Subsequent inspections of the connections during re-torquing on 3/25/08 and subsequent ductor testing performed on 7/16/08, revealed no evidence of significant degradation of the connections.

Although the average accident loading is about twice as high as the 112 amp loading carried over the 4-day period, the accident loading lasts only 90 minutes. Based on the above, it is concluded that the degraded connections were capable of carrying the accident loading without significant degradation.

Evaluation of SBO Scenario: During an SBO event, the battery is designed to provide adequate voltage to the 2D2 Bus loads for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. Verification testing for this scenario is not required by Tech Specs.Analyses are performed to ensure the battery has adequate capacity to supply the 2D2 loads for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />.During this scenario, the battery charger (normal or swing) is not expected to be reenergized until the end of the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> period so the battery would have to supply the bus loads for a 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> period. During this scenario the battery must supply the SBO load profile current (itemized above) through the degraded connection.

Due to the 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> duration of this event, there is some possibility that heating of the connections would occur. It is concluded that this would not result in tripping of the 2D201 breaker within the four hour period given it has a magnetic trip not directly affected by the temperature at its terminals.

File: Degradation of DC connection D201 9 of 34 7/17/2008

-2:19:33 PM IEDISON A. FM$(JV P~rFK%4j10%4.

,.pýIt is noted that during the 4-day period between 3/17/08 and 3/21/08 when the swing charger was in-service, it was supplying approximately 112 amps through the degraded connections continuously.

Subsequent inspections of the connections during re-torquing on 3/25/08 and subsequent ductor testing performed on 7/16/08, revealed no evidence of significant degradation of the connections.

During an SBO condition the average current through the degraded connections would be approximately 161 amps. Although this is about 45% higher than the current carried during the 4-day period, the SBO current lasts only for 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br />. Based on the above, it is concluded that the degraded connections were capable of carrying the SBO accident loading without significant degradation.

File: Degradation of DC connection D201 10 of 34"5 7/17/2008

-2:19:33 PM f. q .M. FPI J U)VIHFRN (AIFOM"iA Ani 11111% AS f1.%A1t0S.4L Ciifflri F'r W~fs :,. t-e34*0 ;LX09 .0,S kEE5T F Ki-~I 2E~I4oaN3Ic'2I.5rI ZjLJ.20~.

314 M off, .. uwd.f -s 154 .~ ft -r wt.w IF~ -= Ia,-Ii 6t00.-I I 11 1 MC.A- , k0 1430.0- J 300.0- M~C-M0.0-0.-1 X., rG-, 1 fli.j£I -2 2rAW PYfi[3- 2AfVA Ma2 QL-4 2M607 PD24~ -$ 2A1W4 P016 6O ?AUCe L.:'dS PESFr 1. ACWNi PD18[]2-2 2A~fli I PI 14 RA.1 AC131335 11;-S ZAM4.1 SIV5 START H2-6 2AO61E UAI DWSr P1 02 7 1 _ :.s SUrIJ SJ-0 ZA13618 UAT CfMsr M 31 EXI) -4x H G J-' 7. 1i, 3-7 SP.'BE A2ACf.4 w~ rp.:Cý& vs HA21.-(C S i47 Hi I2EK-C- --4 1E4 I &2EH 16 4 2H.32-40T S B3MCE) %1X.TS 64. SPR 2Na.0--m -x-LASCCC J54i20 IGPDDX LO.W.,~ ISXO64.00 000 UbO.M 17.ZQ0 172C? 1 74OI~r MAN0 1%.-7 133 iZ I E 22.3 ST8.I Fin: P.!.. PC: ~ O -e Figure 1 -ESF test on affected circuit on 2/11/2004 before the breaker replacement.

Test duration of the test about 15 seconds.File: Degradation of DC connection D201 i11 of 34 7/17/2008

-2:19:33 PM 4.in " Matb FFT I SMtII4IRS (Al IOI1iA EDISON An ftQ.0f J%1:J#r%.41(0AL-C* pl Fie tcisL .~t MM1*I 1-3 2ADNG P1375 ~4 2 AXA E4 ERO 6 20* a:wu setvei A i V Ik 11 ,ffi AWL 61.,V 2A61 RAT _____PW 2AD61-5T-test-PD.D12rrG3SC1 JIM=7*D~.

I:23 a ___________

ar.IS ... "dig 6=6 5ý.. d. T. VI.. fF-2Dj 200.a-0.0-*3CA.-400.0-.-....aRC I I W~ V-4 WAA ?0.0C 112.00 K~oce 16.000 YsX 3Cx0 62.13W KfOM ft0.=1-0 JM1.W.3 V501i L-3 2AGV.1 P916 L-3 ZAGVA PW S~L-4 IAMW0 P525 L~2AMI P524 0 20 AW RFSF-B 1. ZMA2 P1 2[12.2 JAMf I P'134 U 23 2.AL.22 W1135[]- AMI3J SIAS SrAaR H- 2M0CGS OAT O$PST Ph 3 4 072 P 32 Me1012 14.342 36EI T432 ,.1-5 2806219 071'4 2EP6faý2.32 S6 28%43 MCC ID VXTS z32~ ~ *4~I I is se;;n-ýi-i Lý -12.- &c2 & lened X .:e..Oa Figure 2 -ESF test on affected circuit on 11/29/2007 after breaker replacement.

It is important to note no significant changes in the test results between this test after breaker replacement and the one in Figure 1, before breaker replacement.

Test duration about 15 seconds.File: Degradation of DC connection D201 12of 34 7/17/2008

-2:19:33 PM 4.M ,n Haf f FT l IEDISON Fir. kxel :A "m n~uecurmr Seic~r~TIE~iJ-~ ~'LI fl..L4JflS~Afl.3LI a:a L.~,-LJ-jQV!S.sz1rL 00 I. -;AV 3 rGRtK I i L3609 .Ov oys I'U 4 R: I:1I r CA'£0.0-2X6£. LZA ETW.T 2..2AR63 SW STAR-~~ ~ 4 2XJIC43Lr#:

gPE£2 C 2.ACS04 LC1 FESE--1 2AOW4 PD13 Li-4 2MscT P011 122AC1 L P1 14 J-. 242ACASS I.WT OHW£T P%J- 1AW8PT CFF5T F%a 20AE I V4 3-5 250 is MY]2 IRKc- r:-U)42" t'CC 0) !g7.1J0A.0 U2.100 1S.G0B 34.3A .40.6 92AW0 1'U'6 -47D W.600 S4td ALLA ALL -Ajoo-0 nD DC 22.15X J 5.000 2? Xa~3~2 __.9 s- C I :r... t lu-ýi:!A 1<8> 1 CaOD£140X

~ £..a~'Figure 3 -LOVS test on affected circuit on 1/21/2008 performed a few weeks before the degraded condition of the connection was discovered.

The test results were satisfactory.

Test duration about 15 seconds.File." Degradation of DC connection D201 13 of 34 7/17/2008

-2:19:33 PM I mtI ,UihF%"I~

IFORNIA EDISN I % t.1.1.P J Y~%rM 4ToI, * .flIPral Figure 4: One bolt on the affected connection.

All the aforementioned are in accordance with the assumption based on existing knowledge (as shown graphically in Figure 5), that once a clean contact was established on March 2004, losing pressure did not immediately result in an increase of contact resistance.

This only happened after the 4-day period of swing-charger current flowing thru the impacted joint.File: Degradation of DC connection D201 14 of 34'- ,

7/17/2008

-2:19:33 PM I SOUHF " (Al lIOCtNIA EDISON An fOI.ON 1.%1h#%ATC)10.iL 4L.Mpan)Figure 5 [4] -Contact theory says that contact points are actually cold welds.If the high contact force is decreased, the contact resistance remains constant and does not increase until a much lower force is reached.4. How would have a seismic event affect the inte2ritv of the affected connections A seismic event would have no adverse impact on the integrity of the connection

[9].First of all, from inspection of Vendor's drawings and Bill of Material, the breaker is attached by dedicated bolts and channels to the panel. The connections to the buses upstream and downstream are not designed for support purposes, but to provide electrical continuity to the rest of the circuit. From an electrical point of view, all the bolts belonging to the two affected connections are fully threaded into their nuts and some of them quite tight (from interview with the craftsman that tightened the bolts -Item 8 under "Factual Evidence" above). This means that there was little or no increase in risk that a seismic event would separate the buses breaking the physical continuity of the circuit. Second, the violent shaking of the connection would have help in breaching the very thin layer of oxidation between the busses further reducing (perhaps to zero) the fritting voltage of the connection.

Once a DC current starts, it is well known that current flow will be maintained over small gaps due to the lack of zero-current crossing inherent to DC circuits.

This is particularly true in an inductive circuit, such as the one being fed by 2B008.The following statement results from seismic calculations performed specifically to address the situation on hand [9]: File: Degradation of DC connection D201 15 of 34 NI 7/17/2008

-2:19:33 PM J EDSOVHC$1ONA An ISO JOiT~ftK%A704

). -(Unpn~"The connections will not lose contact because there will be very little differential movement between the breaker tabs and the bus bars. Since the bolts have sufficient strength to withstand the seismic loads, lateral movements will be limited to about 1/16 inch which is the difference between the 3/8 inch diameter holes and the 5/16 inch bolt diameters.

As shown in the figures (Figure 6 below), the breaker tabs are sandwiched top and bottom by the bus bars. Due to the out-of-plane flexibility of the bus bars and having at least two tight bolts per connection, the bus bars will move with the tabs vertically and thus, existing gaps would not be expected to become greater in a seismic event." Figure 6: Physical support of breaker 2D201

Conclusions:

1. The cause for the degraded bus connections between 2D201 and 2B008 was insufficient torque applied to the eight bolts (four per polarity) holding the buses and breaker terminals together, during 2D201 replacement in March 2004.2. Reasonable assurance is provided by surveillance tests and data from the normal-charger test period that the connections were able to perform their design function for the entire period starting with the replacement of 2D201 until the last successful surveillance test a week before the failed one.3. Based on the physics of aluminum-to-aluminum contact and fritting voltage, it is likely that the 120 DC-volts circuit would have exceeded the fritting voltage during a real event allowing the connections to carry current without abnormal voltage drop from 3/21/08 up to the time of the failed surveillance test.4. As shown by calculations

[9], a seismic event would have no deleterious effect on the physical integrity of the breaker or in the capability of the connections discussed herein to perform their design-duty.

File: Degradation of DC connection D201 16 of 34

$M ll SO II'% r'.AII}f(M'IfA 7/17/2008

-2:19:33 PM J EDISON A-, N;*" I%,YAMK% I It0% I L- t mriin, 5. For a LOVS/SIAS or DGVSS event with normal charger 2B002 in service, there is high confidence that the 2B008 battery would have supplied the required power to the 2D2 loads, even with the degraded connections at the 2D201 breaker. This is based on the following facts: a) The 120 VDC available at the battery would be expected to be above the fritting voltage for the connection so it would punch through any surface buildup of oxidation.

b) Once DC current begins to flow, it tends to be self sustaining as there is no zero crossing of the current & voltage, c) The battery would need to support the loads for only 15 seconds until the EDG starts, and then 2B002 charger recovers, so there would not be any significant heat buildup at the connection that would impact the connection or the breaker d) Two successful ESF tests and two successful LOVS tests were performed between 2004 and 2008. This corroborates the conclusions above.6. For a LOVS/SIAS or DGVSS event with the swing charger 2B022 in service, there is reasonable confidence that the 2B008 battery would have supplied the required power to the 2D2 loads, even with the degraded connections at the 2D201 breaker. This is based on the following facts: a) The 120 VDC available at the battery would be expected to be above the fritting voltage for the connection so it would punch through any surface buildup of oxide or contaminants, b) Once DC current begins to flow, it tends to be self sustaining as there is no zero crossing of the current & voltage, c) The battery would need to support the loads for 90 minutes, however during a recent (3/2008) normal charger test, the swing charger supplied 112 ADC thru the under-torqued connections for approximately four days.7. For an SBO event, the degraded connection would have to support the 2D2 bus loads for a 4 hour4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> period and there is some possibility that heating of the connections would occur. But, this would not result in tripping of the 2D201 breaker within the four hour period given it has a magnetic trip not directly affected by the temperature at its terminals.

Calculations from data obtained during the swing charger test of 3/17/08 to 3/21/08 indicate the resistance before the current reversal was no more than 0-8 mf2. For an 8 mfl resistance, a current of 170 ADC (SBO average current) will produce a loss in the connection of about 250 watts -not enough to cause any deleterious heating over a four hour period.The calculations in Attachment 3 and 4, independent from all the above, show the resistance of the joint prior the end of the 4-day normal charger test is statistically within the maximum allowable value, i.e.: the connection would have allowed the battery and swing charger to deliver their charger/load as required.File: Degradation of DC connection D201 17 of 34

.S.I. F CMIFORNIA 7/17/2008

-2:19:33 PM EDISON A, f 1 ArMRS 4T 0.'4. Compan9

References:

I. Attached Timeline. (Attachment 5)2. Graphs obtained during the ESF and LOVS tests noted in Item 3 of the section titled"Factual Information", below.3. Engineering Report: Bolting Techniques and Practices Aluminum / Copper, by Columbia MFG, LLC, Astoria, Oregon.4. Creating Reliable Electrical Connections, by Norman Shackman, P.E., Maintenance World, Posted on Internet on 8-25-03.5. Conductive Grease and Electrical Contact Lubricant "NO-OX-ID A-Special Electric Grade, The Sanchem Company.6. SONGS -Calculation No. E4C-0I 7.1 Rev. 3, sheet # 83.7. MO #07042009000 (battery charger 2B002 test).8. MO #03100406 (2D201 replacement).

9. SONGS Seismic study of degraded connections

-July 2008.10. Ductor test on re-torqued connections performed on 7/15/08 (NMO #800117881).

1I. Notification 200055175 (As found test of 2EYD2 computer readout for 2D bus voltage).File: Degradation of DC connection D201 18 of 34 7/17/2008

-2:19:33 PM I SOU1141R CAI IFC)WdIA EDRISON A- tamo Ci ~rim.%4ou

)ATTACHMENTS File: Degradation of DC connection D201 19 of 34) 7/17/2008

-2:19:33 PM J VJVIIIPR%.

CAI IFOHNIA IEDISON A F .FMS0 4! N4110V41 CLmr*.n, ATTACHMENT 1 Potential Cause(s) for the Marginal Connection Found on 3/25/08 Cause Relevance External surfaces of bolts found not corroded Corrosion of the bolts' threads(1)

Over-tightening (taking the bolt material The looseness found on some of the bolts is beyond its yield point) more than it would have been found if the bolts were torqued over their yield point (1).Vibration Equipment is not subject to severe and continuous vibration.

There is no visible indication of deformation Deformation of the bolted buses (1)Continuous lateral movement of the bolt No such wearing of the buses/bolts/washers wearing the bus and/or washer was found on their outside surface (1), Most probable cause.Possible scenario:

Maintenance personnel tightened the bolts with the common wrench, Under-tightening but forgot to finalize the torquing activity with the torque wrench. This conclusion is based on the interview with the craftsmen (see Item 8 under "Factual Information," above.-Note (1): The degraded connections have not been disassembled and inspected.

File: Degradation of DC connection D201 20 of 34 7/17/2008

-2:19:33 PM I %OUHFR% CAl IICWNIA EDISON A. f AOr)% iTFir.7 %4rr m C ompa ATTACHMENT 2 Weekly Battery (2B008) Test Data -2004-2008 SpGr Cat A Limits Battery Met?Date Volts (>= 1.202)06/10/08 06/03/08 05/27/08 05/20/08 05/13/08 05/06/08 04/29/08 04/22/08 04/15/08 04/08/08 04/01/08 03125/08 03/18/08 03/11/08 03/04/08 02/26/08 02/19/08 02/12/08 02/05/08 01/29/08 01/22/08 01/15/08 01/08/08 01/01/08 12/25/07 12/18/07 12/11/07 12/04/07 11/27/07 11/20/07 11/13/07 11/06/07 10/30/07 10/23/07 10/16/07 10/09/07 10/02/07 131.76 131.75 131.75 131.87 131.46 131.80 131.86 131.86 131.74 131.88 131.52 131.85 131.73 131.60 131.94 131.91 131.85 131.84 131.82 131.93 131.76 131.40 131.22 131.29 131.37 131.50 131.35 131.62 131.45 131.61 131.69 131.37 131.31 131.23 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Pilot Cell A Corr SpGr 1.229 1.225 1.232 1.224 1.234 1.236 1.234 1.225 1.237 1.223 1.237 1.240 1.225 1.226 1.225 1.226 1.225 1.227 1.226 1.227 1.234 1.227 1.226 1.229 1.226 1.230 1.227 1.227 1.225 1.227 1.229 1.229 1.226 1.229 Pilot Cell B Corr SpGr 1.224 1.218 1.224 1.218 1.225 1.223 1.225 1.223 1.231 1.225 1.225 1.224 1.219 1.219 1.217 1.220 1.217 1.220 1.209 1.218 1.228 1.221 1.221 1.224 1.223 1.223 1.221 1.221 1.220 1.221 1.223 1.224 1.225 1.217 MO 08060144 08052367 08051932 08051182 08050425 08042805 08042026 08041266 08040720 08040112 08031922 08031473 08030752 08030184 08022321 08021808 08020977 08020457 08012307 08011542 08011162 08010585 08010076 07121685 07121328 07120824 07120307 07111436 07112430 07110810 07110397 07101717 07101477 07101022 07100642 07100110 07091415 Remarks MO not imaged MO not imaged 2D201 Event MO not imaged File: Degradation of DC connection D201 21 of 34 7/17/2008

-2:19:33 PM J l 114FRIN CAI IFORNIA IEDISON 09/25/07 09/18/07 09/11/07 09/04/07 08/28/07 08/21/07 08/14/07 08/07/07 07/31/07 07/24/07 07/17/07 07/10/07 07/03/07 06/26/07 06/19/07 06/12/07 06/05/07 05/29/07 05/22/07 05/15/07 05/08/07 05/01/07 04/24/07 04/17/07 04/10/07 04/03/07 03/27/07 03/20/07 03/13/07 03/06/07 02/27/07 02/20/07 02/13/07 02/06/07 01/30/07 01/23/07 01/16/07 01/09/07 01/02/07 12/26/06 12/19/06 12/12/06 12/05/06 11/28/06 11/21/06 11/14/06 11/07/06 10/31/06 10/24106 10/17/06 131.27 131.45 131.33 131.41 131.36 131.48 131.44 131.41 131.42 132.04 131.99 131.99 132.00 131.87 131.86 131.81 131.80 131.74 131.81 131.49 131.58 131.77 131.57 131.60 131.67 131.66 131.66 131.69 131.64 131.65 131.60 131.18 132.05 131.54 131.96 131.06 131.14 131.21 131.15 131.16 131.12 131.13 131.21 131.23 131.30 131.27 131.27 131.29 131.28 131.31 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 1.229 1.228 1.229 1.230 1.231 1.229 1.220 1.223 1.223 1.212 1.223 1.224 1.223 1.223 1.224 1.223 1.222 1.222 1.222 1.220 1.210 1.215 1.219 1.221 1.215 1.217 1.215 1.210 1.212 1.212 1.220 1.219 1.209 1.228 1.214 1.221 1.219 1.218 1.218 1.218 1.222 1.218 1.217 1.223 1.230 1.222 1.223 1.223 1.223 1.223 1.226 1.223 1.223 1.223 1.226 1.225 1.222 1.218 1.218 1.214 1.220 1.220 1.222 1.219 1.223 1.222 1.222 1.217 1.213 1.215 1.208 1.213 1.218 1.217 1.209 1.212 1.211 1.213 1.210 1.211 1.218 1.209 1.217 1.225 1,218 1.216 1.213 1.212 1.213 1.213 1.215 1,215 1.214 1.222 1.217 1T218 1.217 1.216 1.216 1.217 07091049 07090784 07090267 07081640 07081180 07080810 07080393 07080101 07071141 07070851 07070424 07070128 07061477 07061065 07060693 07060287 07051854 07051514 07051093 07050550 07050145 07041721 07041298 07040841 07040377 07033435 07032955 07032416 07030537 07022492 07021801 07021348 07020750 07011942 07011363 07011107 07010592 070100097 06121575 06121303 06150848 06120337 06111608 06111257 06110903 06110429 06110184 06101425 06100933 06100741 File: Degradation of DC connection D201 22 of 34 7/17/2008

-2:19:33 PM 5J SMI111 RN~ LAI 1101041A IEDISON Mn FjjXj 1. rpp!%4T0 11. -Compa')10/10/06 10/03106 09/26/06 09/16/06 09/12/06 09/05/08 08/29/06 08/22/06 08/15/06 08/08/06 08/01/06 07/25/06 07/18/06 07/11/06 07/04/06 06/27/06 06/20/06 06/13/06 06/06/06 05/30/06 05/23/06 05/16/06 05/09/06 05/02/06 04/25/06 04/18/06 04/11/06 04/04/06 03/28/06 03/21/06 03/14/06 03/07/06 02/28/06 02/21/06 02/14/06 02/07/06 01/31/06 01/24/06 01/17/06 01/10/06 01/03/06 12/27/05 12/20/05 12/13/05 12/06/05 11/29/05 11/22/05 11/15/05 11/08/05 11/01/05 131.28 131.27 132.22 131.79 131.91 131.60 131.59 131.58 131.59 131.68 131.76 131.66 131.61 131.64 131.63 131.59 131.49 131.57 131.53 131.29 131.47 131.45 131.48 131.46 131.34 131.36 131.41 131.37 131.35 131.24 131.44 132.56 131.44 131.65 131.66 131.67 131.74 131.58 131.71 131.54 131.56 131.63 131.63 131.59 131.64 131.72 131.62 131.68 131.71 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 1.218 1.218 1.222 1.218 1.226 1.222 1.225 1.220 1,229 1.224 1.223 1.224 1.219 1.226 1.224 1.223 1.223 1.224 1.222 1.223 1.226 1.228 1.218 1.227 1.228 1.226 1.227 1.227 1.220 1.229 1.233 1.220 1.238 1.220 1.221 1.218 1.219 1.220 1.221 1.218 1.220 1.219 1.220 1.222 1.220 1.221 1.219 1.221 1.222 1.214 1.214 1.216 1.216 1.219 1.217 1.217 1.217 1.218 1.219 1.219 1.219 1.218 1.220 1.221 1.219 1.219 1.219 1.221 1.222 1.221 1.225 1.217 1.223 1.222 1.222 1.223 1.224 1.218 1.223 1.225 1.223 1.230 1.217 1.218 1.217 1.216 1.217 1.218 1.215 1.217 1.217 1.218 1.219 1.219 1.218 1.217 1.218 1.218 06100111 06091284 06090865 06090189 06090189 06081588 06081261 06080908 06080586 06080107 06071235 06070854 06070606 06070160 06062026 06061477 06060926 06060596 06060174 06051947 06051339 06050784 06050300 06041634 06041324 06040718 06040424 06032246 06031753 06031102 06030667 06021785 06021408 06020927 06020532 06020011 06011826 06011303 06010560 06010091 05121705 05121316 05120898 05120406 05111528 05111259 05110868 05110596 05110066 05102804 MO not imaged File: Degradation of DC connection D201 23 of 34-d 7/17/2008

-2:19:33 PM O.J SXI4111% CAI IFORNIA IEDISON A. F11SO ON 'TNOW44l -C mp~ft 10125/05 10/18/05 10/11/05 10/04/05 09/27/05 09/20/05 09/13/05 09/06/05 08/30/05 08/23/05 08/16/05 08/09/05 08/02/05 07/26/05 07/19/05 07/12/05 07105/05 06128/05 06/21105 06/14/05 06107/05 05/31/05 05/24/05 05/17/05 05/10/05 05/03/05 04/26/05 04/19/08 04/12/05 04/05105 03/29105 03/22/05 03/15/05 03/08/05 03101/05 02/22/05 02/15/05 02/08/05 02/01/05 01/25/05 01/18/05 01/11/05 01/04/05 12/28/04 12/21/04 12/14/04 12/07/04 11/30104 11/23/04 11/16/04 131.67 131.67 131.71 131.67 131.67 131.70 131.70 131.71 131.82 131.78 131.73 131.79 131.79 131.87 131.66 131.68 131.63 131.59 131.50 131.61 131.61 131.63 131.63 131.76 131.63 131.58 131.62 131.56 131.57 131.54 131.61 131.56 131.52 131.60 131.56 131.59 131.52 131.54 131.55 131.63 131.67 131.64 131.42 131.76 131.76 131.63 131.57 131.49 131.45 131.56 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 1.221 1.223 1.221 1.218 1.220 1.219 1.223 1.219 1,221 1.221 1.220 1.220 1.220 1.220 1.220 1.217 1.217 1.218 1.217 1,219 1.220 1.218 1.222 1.214 1.220 1.220 1.217 1.221 1.219 1.217 1.218 1.219 1.217 1.222 1.219 1.220 1.219 1.220 1.220 1.218 1.218 1.219 1.219 1.221 1.220 1.220 1 221 1.222 1.222 1.220 1.219 1.219 1.218 1.217 1.218 1.217 1.219 1.218 1.217 1.216 1.217 1.218 1.219 1.218 1.217 1.217 1.216 1.216 1.217 1.218 1.216 1.216 1.218 1.214 1.217 1.217 1.218 1.216 1.217 1.215 1.217 1.216 1.215 1.216 1.216 1.217 1.216 1.216 1.216 1.219 1.216 1.217 1.216 1.218 1.215 1.215 1.216 1.217 1.218 1.215 05102429 05100637 05100194 05091712 05091222 05090683 05090306 05082109 05081531 05081047 05080627 05080126 05071506 05070985 05070626 05070246 05062441 05061796 05061234 05060690 05052313 05052099 05051459 05050915 05050250 05041899 05041362 05040766 05040467 05032192 05031618 05031095 05030661 05030091 05022033 05021455 05020661 05020116 05012154 05011488 05011156 05010189 04123087 04122690 04122402 04120542 04112002 04111735 04111198 04110704 File: Degradation of DC connection D201 24 of 34*".'

7/17/2008

-2:19:33 PM J OtJIHF'% C;Al IFORNIA IEDISON AnM IXO 1% Tf.#?%.I UO.4L -(umpan 11/09/04 11/02/04 10/26/04 10/19/04 10/12/04 10/05/04 09/28/04 09/21/04 09/14/04 09/07/04 08/31/04 08/24/04 08/17/04 08/10/04 08/03/04 07/27104 07/20/04 07/13/04 07/06/04 06/29/04 06/22/04 06/15/04 06/08/04 06/01/04 05/25/04 05/18/04 05/11/04 05/04/04 04/27/04 04/20104 04/13/04 04/06/04 03/07/04 03/30/04 131.51 131.62 131.63 131.66 131.60 132.00 131.83 131.78 131.92 131.89 131.81 131.80 131.84 131.87 131.83 131.85 131.89 131.86 131.82 131.72 131.77 131.76 131.76 131.75 131.70 131.79 131.74 131.90 131.31 132.28 131.93 131.87 131.48 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 1.222 1.220 1.216 1.215 1.219 1.217 1.217 1.217 1.219 1.216 1.220 1.220 1.217 1.221 1.218 1.220 1.218 1.219 1.219 1.216 1.215 1.218 1.217 1.218 1.218 1.218 1.218 1.213 1.224 1.227 1.225 1.222 1.220 1.226 1.227 1.230 1.216 1.218 1.221 1.219 1.220 1.219 1.219 1.220 1.219 1.216 1.215 1.216 1.214 1.218 1.216 1.219 1.215 1.219 1.217 1.222 1.222 1.218 1.223 1.221 1.219 1.218 1.219 1.220 1.219 1.220 1.219 1.220 1.221 1.219 1.220 1.218 1.221 1.216 1.229 1.225 1.221 04110222 04102097 04101499 04100924 04100488 04091788 04091336 04090929 04090272 04082120 04081618 04081084 04080658 04080244 04072035 04071416 04070933 04070422 04062326 04061791 04061223 04060698 40600810 04052007 04051494 04050904 04050260 04042120 04041552 04041001 04040483 04032496 2D201 Breaker Replacement 03/23/04 131.25 Yes 03/16/04 131.28 Yes 1.224 04031814 1.227 04031252 1.225 04030721 03/09104 03/02104 02/24/04 02/17/04 02/10/04 02/03/04 01/27/04 01/20/04 01/13/04 01/06/04 131.57 131.10 131.35 131.32 131.57 131.39 131.38 131.63 131.50 131.42 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 1.228 1.221 1.218 1.227 1.222 1.227 1.224 1.226 1.226 1.227 04030178 04021740 04021295 04020785 04020282 04011819 04011327 04010936 04010242 03121961 File: Degradation of DC connection D201 25 of 34 (.

SSOUII4FB CAIIF}CWNIA 7/17/2008

-2:19:33 PM J EDRI(SON An EISO)N I%rTNN4rr'n%4, C ,nywn)ATTACHMENT 3 Calculation of joint resistance of degraded connections during the normal charger test of 3/17-21/08 1). Calculation of contact resistance while the swing charger is working: BATTERY -SWING I I213008 -CHARGER NORMAL 2Bi -22 CHARGER LI-2B2 2B002 >tIC...........

SH UN T ......................

DEGRADED CONNECTION BREAKER IL 2D201 to 125 VDC bus DCload5 On 3/17/08, when the normal charger (2B002) was removed from service and the swing charger (2B022) was placed in service, the swing charger voltmeter read 131.52 (+/- 0.15)VDC and 112 ADC. Based on plant computer data, the 2D2 bus voltage under these conditions was between 130.7 and 131.8 VDC. Based on this information, the 2D201 connection resistance at this time would be 0 to 8 mf2.IL = 112 ADC (from MO #08011467)

Vsc = 131.52 +/- 0.15 VDC = 131.67-131.37 VDC V2D1) 130.7-131.8 VDC (RTime reading of 132 VDC and calibration data from Notification

1. 200055175).

Rmax = (Vscmax -V2Dmin)/IL

= (131.67 -130.7)/112

= 8 mfl Rmin = (Vscmin --V2Dmax)/IL File: Degradation of DC connection D201 26 of 34 7/17/2008

-2:19:33 PM SOUI11FRN CM IICRNIA Ij EDISON A0 t. M % )%TfX,4t 10%41. -* ýHowever, V2D can only be as high as the swing-charger's voltage, otherwise current would flow in the "wrong" direction, thus: Rmin = (131.37 -131.37)/112

=0 m(Q 2) Calculation of contact resistance when the normal charger resumed operation:

BATTERY -2B008 NORMAL CHARGER 2B002 DEGRADED CONNECTION t Ic SWING CHARGER S2B022 S...........

125 VDC bus I= IL + IC IL DC load On 3/21/08, when the normal charger (2B002) was returned to service, the 2D2 bus voltage was 130.7-131.8 VDC. The battery terminal voltage under these conditions was 131.38 (+/- 0.15) VDC. Assuming a typical float current of 0.1 amps into the battery, the corresponding connection resistance at this time would be equal to 0-5.7 Q. Statistically, this resistance is orders of magnitude higher than the connection resistance on 3/17/08.IC=Vb=V2D=0.1 ADC 131.38 +/- 0.15 VDC = 131.53-131.23 VDC 130.7-131.8 VDC Rmax = (V2Dr-ax Vbmin)/IC

= (131.8 -131.23)/0.1

= 5.7 fl Rmin = (V2Dmin -Vbmax)/IC File: Degradation of DC connection D201 27 of 34 7/17/2008

-2:19:33 PM JSOU)I)F N CAl IORNI*lA A~ ft HIM^ I%TAft41IC)\AL

-Compan)However, V2D can only be as high as the -charger's voltage, otherwise current would flow in the "wrong" direction, thus: Rmin = (131.53 -131.53)/0.1

= 3) Calculation of VD2 based on measured value As shown on the schematic below, the value of VD2 is measured on a computer, after it is affected by the error of the resistance divider, analog-digital converter, and truncated at the computer.

Thus, if the value displayed in the computer for VD2 is 132 VDC, then the true value at the bus is as shown below.VD2displayed Digital truncation In order to measure the relation between the true value of VD2 and the value displayed on the computer, a careful "as-found" test performed on 7/15/08 (Ref. 11) yielded the following.

VD2displayed

= 132 VDC 4 true value of VD2 = 130.8-131.7 VDC File: Degradation of DC connection D201 28 of 34 7/17/2008

-2:19:33 PM I SOI)IHFR%

CAiFORNIA Anl EI.7SO%' J%r,'Rft~OV4L

-C.ompan 7 ATTACHMENT 4 Calculation of voltage values during analyzed accidents and maximum acceptable resistance in degraded connection t3 EU Fdk &1w rwrt Fwnmt Io* Qata Wndo H* _-n a U tj tn eprJIN qCASE, ,-- ý. .. ." ..-9 x M" fir 8OUNDING CASE F TO 2 1202 a 1 CONMXTO'3 LaT4j-srzE-SI alwooo li.4 LI1 F-N.mm2_W4IYTE TME PEFW&1I 1 1-20 1 20.99 5 i -. T u Nu~fl21 TVW PEF 0.90 IPANR~fAOId 0.1 1 .3 V ....-., 1-30 m6 M04.11 I IJ mm to." cm.00 lb.lb.110.106 U t0-U., 71j 22 24,; Po4wlaJln 25 P497(Z3nl7 2?_.211 332 2 , .0 ... .14 "C.9 C --~llO 54 DO v? I I vp g C-O ' 6000 GA I 12.0 4q 41 0 0090" n oo91~ .il.l 7.44DE-44 a DNCAE 2020 1-RrK. Coa...lr 0GF~t!195tnd.

0.0054 Q2 46 (6000206 W-4.1C7030 W____ __47 1 ____________________

t_-______-_

7 49 .-I 0~~Ad'-o 1. I-Ol*. ~.l so _ _ ___ I____________

S4 --iýM -TabIC 0.3. 36~: ~ ..1~ ~-.0, ~h8 8~T-H TseS8-tJMZVSIQI-1 FTaJIC .4.18-VLI.

z~~ReadV~RatI~ ~ ~ .Ic~~nccrri..

AZZ1F169-LOSIJM.105II~iCcwocftElce-LOYV-S;CMP 5:2 fPM File: Degradation of DC connection D201 29of 34 7/17/2008

-2:19:33 PM CMIIONNIA t rEDISON AAtl~.9lS0.'V

.%

CUM PSM.EDj zile Edt kAev. jnoet Prin.t lad Daa AIM. Hep VTIK 1",-t q, ;,-,- h-t --9 X a2 tge "% 43j &9.. ; -'0, _.. ..o. .Lr -- ý! ýArial-50

• Bit ffum U6-0 wi ~ t ~~ ~ * ~ '- I I a e)375% *U D5 -A .H I J K L M _ N 0 P A S .T I .U V" 17 64 6 AV 0458 6.322 15.81 15.61 15.81 ISS 157 5.10 5.10 50 50 .5 2-1125 108205 10401 1M 408 12 AVG r .572 0oD46 2.64 244 244 264 2.64 2-12 2.12 2.12 22 2.12 21.42i 105.54 M102 107.73 is0m 23 AVO r 572 90,575 076 0.76 0.76 0.76 876 035 (135 0.35 025 6.35 2L-345 M722 T26 10.5 20 574-p 1 2 AV r. r__U-_ 200 2 200 208 200 226 226 2.26 226 22 10540 I107.65 23 1 4l__ 124 ba 26.80 0.00 0. 00 0.004 104 620 0.00 0.0 0.00 P4'12L3731 65M 10w 1.36 24 15 ____ 1.57 0.297 31.10 6.30 6.30 6.30 6.30 5--.32 165 1.89 1059 165 P49q a 5 76.29 106.72 103.41 25 340 12 AVG3 1.572 0.6705 26.530 6.30 1 6.30- 6.30 6.30 161.64 422 1 4.22 422 1 4.22 P145 21373 69.58 104.35 1014 I 26 i Be 0t .0371 0.0052 128646 1286 2846 504" 50.66 0.67 647 0.67 0.47 0.47 2Y1002 108.93 112.91 W09.66 27 1152 3~7 .02C 0.0297 41071 60 0.5 1.61 1.01 124 O023 0.03 O023 0.03 2MS4706 10830 173.55 110.30 28 44 2AVG 0-203 0.00m 1.39 3 3 0.74 0.74 0.07 001 0.01 a01 0.01 27007 10959 113.6 (10.31 25 BIG 22 AVG 1.572 1.2029 131 13 0.6 0.66 0.66 1.08 t.58 0.75 0.75 0.75 2L-71 108.01 11158 109.53 30 24 Q2 AVG 1T972 0-6305 0.08 0 .08 0.08 0.0502 0.08 05 0.08 040S 0.08 12L-345 105D.54 113.51 15026 31 32 1 peel vo~ag associated with motor stariNg.Mwit tomis r 105.414 W.,e 0.0 CortoWro 5600 6 "1 [,1 '. "Feeders 4 OUT

.00 37 VPC .113 3-..mpet Cables C-rtoeoor G.(J 38 ToWalCc-trorz 62.00 W9 1.,e e

'-. '. ,e200E -05 12,11" 41 Total Connectons Resistance 7.440E-04 D 40;.4 20201 Areakei Connecto's Resistance 0.0054 n 4 3-Jumper Cables Resisrance 3.830E-04 3 8 3 p 0 46 g ~~(1008 =201t 11CIt 3101 ____ ___47 7 49. .- .., oral,, ........t... , , 4T tA,Jdad t(r C il. Ft. .C '.t-1Minimum Inet.e Yolltare 05 .414i.2I T ~H Ttd.3.0-8008 g1.8S~a~8 A'-0osz~io ATlS8.3-61TVN-1Tablee8AS42VGd

/1T4_8-3R1 Iii _- _ý.4 Ready Aiuaotjl 4 4 ~ 1 adat-o'DC-r...j (D>A Fies-Lctus Otsfl~tcrosoftEntced-Seu W/MM0NetAMpIt 5: 30PM File: Degradation of DC connection D201 30 of 34 7/17/2008

-2:19:33 PM%(lMll I RC A1FORtNJA.J EDISON Mn 1.10 I% tf4110".4L-I ATTACHMENT 5 Breaker 2D201 and Battery 2B008 Sequence of Events 05/02/2002 MO 01061449 completed 5/27/02 connect monitoring equipment (2A04, Train B ESF test)02/11/2004 MO 03031098 completed 9/30/04 Support various Operations procedures.

Completed the DG tests "Train B" 2A06. (ESF)02/20/2004 MO 03031098 Support various Operations procedures.

Completed the DG tests "Train B" 2A06. (LOVS)02/27/2004 MO 04021613 retested new breaker for long time trip on 2/20/04 per ECP 031200986-9.

On 2/10/04 obtained new breaker and trip tested per SO123-1-4.7.03/17/2004 MO 03100406 completed changing the Short Time Delay Settings and installing new breaker 2D201 per Calc E4C-109. (ECP 001000280-05, ECN A20785)01/06/2006 MO 05081829 completed 3/28/06, the DG tests "Train B" 2A06. (ESF)01/21/2006 MO 06011546 support Operation's procedure for 2G003 DG. (LOVS)02/25/2006 MO 05011306 completed 5-year Battery Performance Test (21)201 breaker operated per WAR 2- R41)2BAT)07/15/2006 MO 06041451 completed quarterly battery surveillance on B008 battery.10/03/2006 MO 06070800 completed quarterly battery surveillance on B008 battery.12/28/2006 MO 06100206 completed quarterly battery surveillance on B008 battery.01/30/2007 MO 06070902 Annual Battery Performance test after reaching 85% service life in October 2006 and performance of Thermography (2D201 breaker operated per WAR 2-0601303)

File: Degradation of DC connection D201 31 of 34 n-7/17/2008

-2:19:33 PM MiEDISO.. n f j %~ r.rif %4 10%4 A .-C wap.03/27/2007 06/13/2007 09/17/2007 11/29/2007 12/08/2007 12/27/2007 12 ?.8 '2:, 0 '0 01/09/2008 01/302008 02/19/2008 02/22/2008 02/25/2008 02/27/2008 03/03/2008 03/11/2008 MO 07010094 completed quarterly battery surveillance on B008 battery.MO 07033365 completed quarterly battery surveillance on B008 battery.MO 07060901 completed quarterly battery surveillance on B008 battery.MO 06122943 completed the ESF tests "Train B" 2A06.MO 07091048 completed quarterly battery surveillance on B008 battery.MO 07020582 Perform pre-weekly quarterly battery performance test.,i( ) 7 l:(T, l [) It. .'_ ") LW 1'. ..2,! ,. C dLa 2c... 2H!102. D lIi-SlY n i 6lv (-3A5 7 ". .l) ( j C ! ' -, d l MO 08010611 support for ESF tests (LOVS)MO 07020585 Weekly lE battery inspection completed

-sat.MO 08020977 Weekly I E battery inspection completed

-sat.MO 08011261 Perform pre-weekly quarterly battery performance test.MO 07120476 Connect 2/3B00X to 2D206 (2B008). This temp ECP to connect battery 2/3BOOX to bus 2D2 to support 2B008 battery bank performance testing. (Temp ECP 070900092-3)

(2D201 operated per MO with In-test)\1() U ),LL I i2.l t'olIlIl2 t.iCI v I(.I t; .oC lwct i .n i 2( h,.i! gýr (o" k w ';, k MO 08011292 3 1-day post test weekly/quarterly inspection MO 07020581 Annual Battery Performance test after reaching 85% service life in October 2006 and performance of Thermography.

(2D201 operated per WAR 2-0702901 via an In-test)MO 07020584 Perform Physical inspection of battery MO 08010255 completed quarterly battery surveillance on B008 battery.MO 08030184 Weekly I E battery inspection completed

-sat File: Degradation of DC connection D201 32 of 34 7/17/2008

-2:19:33 PM EDISON An F 01SON I N T M %4 710 N4 1.

• Umps.03/17/2008 MO 08030095 31-day post test weekly/quarterly inspection 03/21/2008 MO 06100929 B002 Battery Charger performance test and thermography inspection.

(21)201I not operated per WAR 2-0702504)

MO 08011467 2B002, Bus D2 Normal Charger, placed in service and 2B022, Swing charger removed from service by soft transfer operation. (Operations)

MO 07042009 Charger Performance Test, perform Operability readings on 2B008 while on the Swing Charger 2B022.03/25/2008 MO 08031473 completed weekly battery surveillance which discovered low voltage on battery 2D2.Operations declared 2D2 battery inoperable.

Electricians discovered eight loose bolts on the battery side of breaker 2D201.ACE 080301117 generated to determine cause of loose bolts on bus bars of 2D201.MO 08031721 Tighten and torque the loose bus bar bolts.MO 08031738 Perform weekly/quarterly inspection for declaring batteries operable 03/28/2008 MO 07101740 perform annual energization testing on swing charger 2B022.(2D201 not operated per WAR 2-0702931

)04/01/2008 MO 08031922 Weekly 1 E battery inspection completed

-sat.04/02/2008 MO 07101741 Swing Battery Charger 2B022 Perform 12-hour Full Load Test-sat 04/10/2008 OE 26600 Operating Experience generated on loose Bus Bars bolts causing low battery voltage.05/19/2008 MO 08030494 completed quarterly battery surveillance on B008 battery.NOTE: all dates are MO completed dates unless noted rev.08 File: Degradation of DC connection D201 33 of 34 7/17/2008

-2:19:33 PM 2B008 battery SM/' z W% (A IFORMA A M EDISON Aft Iff~hO i%lr4fli4.K

%A1U U- .pl ATTACHMENT 6} Other 2D2P 1 loads DEGRADED -)CONNECTIONS

'N 21B002 Charger 2D2 bus'N'N I) ')-~I Other 2D2 loads I ) I)I -A IA I I 2A06 switchgear Voltage monitor File. Degradation of DC connection D201 34 of 34