ML11243A059

From kanterella
Jump to navigation Jump to search
Songs System Health Report
ML11243A059
Person / Time
Site: San Onofre  Southern California Edison icon.png
Issue date: 04/16/2010
From:
Southern California Edison Co
To:
Office of Information Services
References
FOIA/PA-2011-0157
Download: ML11243A059 (29)
v • d • e


Text

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1 st Quarter - 2010 EDGS - Emergency Diesel Generators Previous Four Quarters 2009 2009 2009 QTR-1 QTR-2 QTR-3 QTR-4 Unit 2 Unit 3 vy System Engineer: I(b)(6) Date: 4/16/2010 Supervisor Approval: I Date: 4/16/2010 Plant Health Committee Date: 9/21/2009 l oBa re Overall Rating: RED The performance of the U2 and U3 EDG systems has shown signs of improvement over the last quarter. However Unit 2 system vulnerabilities still exist since reliability improvements have not been implemented due to outage extension.

Since the performance of both Units has been poor overall, the Road to Green summary will be applied to both units.

Overall Rating: RED It is expected the overall system health for the EDO System will change from RED to YELLOW in July 2010. Actions Unit 3 required to change system health to YELLOW are:

-Incorporate reliability improvements on one train of EDG. There has been significant effort to develop plans and produce design changes, however no reliability improvements have been implemented. Installing improvements on one train shows commitment to ensuring long term reliability. See problems for EDG Reliability Improvements It is expected the overall system health for the EDG System will change from YELLOW to BLUE in December 2010.

Actions required to change system health to BLUE are:

- Install EDG reliability improvements in all EDGs Formal Performance Monitoring and Trending Project Plan Actions Complete It is expected the overall system health for the EDG System will change from BLUE to GREEN in January 2012.

Actions required to change system health to GREEN are:

- U2/3Systems in a(1)

- U2/3 INPO Unavailabity above 2.5% (4 year Maintenance Plan Implemented) 133

SONGS System HEALTH REPORT 6 N' San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 Road to Ge

- Remaining Generators replaced (U2 Train A/B, U3 Train A)

See Problem List and Improvement Plan for details 134

SONGS System HEALTH REPORT

~7<$~fr~L San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 Relabllty

  • Lost Generation G G Maintenance Rule Status F (a)(1) / (a)(2) Status G Y U2 is still meeting all performance criteria for a(2) status U3 is in a(1) status with corrective actions plans in place. U3 will be expected to go back to a(2) inthe 2nd quarter of 2011. This will provide sufficient time for Corrective Actions to be implemented and performance to be trended.
  • MRFFs B K0 U2 Train A: 0 Failures in last 36 months U2 Train B: 2 Failures in last 36 months U2 load-run demands shows 0/20, 0/50, 2/100.

U3 Train A: 4 Failures in last 36 months U3 Train B: 3 Failures in last 36 months U3 load-run demands shows 2/20, 3/50, 7/100.

Train Performance Criteria: 4 MR Unit Performance Criteria 3/20 4/50 6/100

  • RMRFFs G G No Repeat Maintenance Rule Functional Failures for U2.

No Repeat Maintenance Rule Functional Failures for U3.

NOTE: The 4009 indicated a RMRFF. This was later rescinded when the cause of the inital power supply failure was attributed to maintenance practices and the refurbishment of annunciator power supplies, and the most recent failure was attributed to a latent manufacturing defect in the power supply capacitor.

  • Unavailability B Y U2 Train A: 2.5% Unavailability, U2 Train B: .16% Unavailability U3 Train A: 4.21% Unavailability U3 Train B: 3.99% Unavailability MR Peformance Criteria is 8%
  • Plant Level Events G G There have been no plant level events for U2 during the past quarter There have been no plant level events for U3 during the past quarter Functional Materal Condition
  • Walkdown Results I B I B I 135

-j

(N~ ~1/2~* SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010

" Open Corrective Maintenance Tasks G G There are no open corrective maintenance orders for U2 There is one open corrective maintenace order for U3 800437071 Replace the leaking flexible hose MY689 on the EDG compressor (COI 3B)

Owner: Welders Due Date: 3/30/2010

  • Open Elective Maintenance Tasks At the time of this report there are 32 elective S maintenance orders for U2 assigned to Test Techs, Electrical, Machinist, I&C, Welders, B&C, Composites
  • At the time of this report there are 36 elective

"°* i maintenance orders for U2 assigned to Test Techs, A, Electrical, Machinist, I&C, Welders, B&C, Composites, FIN, Painters and HVAC, 0 PM/PdM Results and Feedback G G 0 Open Operator Workarounds G G 0 Open Operational Decision Making G

  • Open Operator Burdens G G The lighting conditions of the EDG rooms is poor. This has been documented on multiple notification. The following NECPs will improve room lighting:

NECP 800449989E NECP 800449991 F, NECP 800449993C NECP 800449995c

  • Open Control Room Deficiencies G G e Open Annunciator Compensatory Actions G G 136 I

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 Design Basis and Conflguradon Health 0 Open Degraded & Nonconforming Conditions B Y There are two open DNCs for U2. These have to do with replacing the radiator fan safety screen 800315855 Owner: Welders Due Date: 8/19/2010 800314710 Owner Welders Due Date: 8/19/2010 There are ten open DNC orders for U3.

800324626 (List) - Inspect Nose Cone Mounting Assy Owner: HVAC Due Date: 5/13/2010 800348694 - Replace CAP sounding tube gasket 3MT036 Owner: Machinist Due Date: 6/21/2010 800350029 - Late IST on relief valve S3.EDGS.3PSV5921 A Owner: B&C Due Date: 4/19/2011 800350035 - Late IST on relief valve S3.EDGS.3PSV5921C Owner: B&C Due Date: 6/28/201C 800350038 - Late IST on relief valve S3.EDGS.3PSV5921D Owner: B&C Due Date: 9/26/2010 800350060 - Late IST on relief valve S3.EDGS.3PSV5950A Owner B&C Due Date: 5/11/2010 800350062 - Late IST on relief valve S3.EDGS.3PSV5950B Owner B&C Due Date: 5/11/2010 800362495 - Repair/Replace Nicked wire #119 at Al Owner Test Techs Due Date: 5/10/2010 800439571 - Alarm horn inaudible Owner FIN Due Date: 6/22/2010 137

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 0 Open Engineering Modification Requests (EMRs) Y Y The following modifications are being evaluated by the EDG Recovery Team. Expected presentation date to the URT will be 2010.

200791540 Install vibration transducers on EDG radiator fans 200791550 Install vibration transducers on EDGs 200684915 Permanent stairs on storage tank-safety concern 20075622780 - Decrease mesh size of EDG vent screen 200818234C Install new lube oil gauge 200278150 Lube oil pump mods 200000319 Replace EDG vibration monitor ( obsolescence) 200390637-8U-Add pressure gauge to monitor piston cooling oil pressure 200390637-12E Provide thermowells to monitor radiator supply/return water temp.

200390637-14D DLpgrade to FDX-II digital pyrometer TI 5953B 200390637-15-Clnstall air box pressure gauge for turbo charger monitoring 200390637-10 ] Develop process to monitor air quality,moisture in air receiver 800073355-6012 L Install infrared windows L160/161 panels 0 Temp ECPs Installed G G

  • Open Doc Assignments Y Y There are many NECPs open the the U2/3 EDGs. Most will be worked during the 2010 scheduled AOTs. The ones that are not being tracked as reliability improvements or as ACE/RCE corrective actions are listed in System Problem List Item 10.

Oprating Experience

" Equipment Performance and Information G G Exchange (EPIX)

  • 1_0CFA Part 21 issues G G 138

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 0 INPO OEs G G OE30844 Emergency Diesel Generator Declared Inoperable due to Engine Coolant in Engine Cylinder After a review of EDG cylinder head walkdown and discussion with Columbia it was determined SONGS has the same vintage (diamond-3) but of a different year which is not susceptible to head cracking.

OE30672 Oil Leak on Emergency Diesel Generator Required Emergency Shutdown SONGS has the same check valves, after visual inspection no check valve caps are currently backed off.

  • NRC Info Notices G G

" Benchmarking G G Atteneded the Winter 2010 EMD Owners Group The following Information Bulletins were sent to the EMD Owners group IB 10-13 EDG Lighting, Painting, Exhaust IB 10-15 Annunciator Power Supply transistor screw intrusion

___IB 10-21 MDR Relays AR Backlog

" Open RCE (Root Cause Evaluations) B B RCE 200704617 Unit 3 Downpower Associated with Emergency Diesel Generators evaluation was completed this quarter. The corrective actions will be expected to complete in the 3rd quarter of 2011

  • Open ACE (Apparent Cause Evaluations) B Y There is one open cause evaluation 200446662 on EDG speed push button failure. Expected closure date of ACE is 12/31/2011, which coordinates replacement of DRUs for each generator.

There are two open cause evaluations 200695875 is for 140 drop per minte leak on U3 train B.

The expected closure date of the ACE is 8/20/2010 after all flexible hoses have been replaced.

200457220 is for Power Supply Failure on U3 Train A EDG. The expected close date of the ACE is 5/31/2010 when the annunciator power RMO has been changed to eliminate the refurbishment option.

  • Open ACE/RCE Required Corrective Actions '. Z There are 'a significant number of corrective actions associated with the RCE on the Unit 3 Downpower that l**

S stremainopen. Due to this event in the 4P09 this indicator

  • is red until the majority of the corrective actions have closed. To see the lists of Corrective Actions see

.. System Problems Lists Items 1-4.

139

SONGS System HEALTH REPORT

\'~ San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 Addlifonal Significant Issues

  • Chronic problems or adverse trends The EDG chronic problems associated with Power Supplies, Flex Lines and Tubing remain system vunerabilites for all EDGs. The reliability improvements have not been implemented due to extended refueling outage. Actions are in place to monitor and/or replace all problem components during EDG Allowed Outage Times R. (AOTs). Actions are tracked in problem lists.
  • Reactivity Management Events G___ G_
  • Unplanned Shutdown LCOs G G No challenges to U2 during the first quarter of 2010 No challenges to U3 during the first quarter of 2010
  • Margin G Y IMSPI Margin to White for U2 is 4 Failures
  • __ _ MSPI margin to White for U3 is 2 Failures
  • Assessments / Audits I Inspections Y Y Directed Assessment Assignments are being tracked under order 800073355.

SPV findings are being tracked under notification 200390637. INPO Assist visit actions are being tracked under notification 200498890.

Parameter Trending

  • Start Air Consumption G G
  • Fuel Filter D/P G G

" Exhaust Temps G B For U3 Train A the exhaust temperatures between cylinders 3 and 6 exceeded 200F. For results of this evaluation see notification 200873728.

" Engine Jacket Water Delta Temp G G

" Housekeeping/ Material Condition Y Y The Housekeeping for the EDGs is still Yellow due to order to clean the engines online not being implemented by Machinists U2 Train A: 800432351 Due Date 8/31/2010 U2 Train B: 800424427 Due Date 1/27/2010 U3 Train A: 800424503 Due Date 7/5/2011 U3 Train B: 800422977 Due Date 6/22/2010 140

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010

  • Leaks Y Y Leaks are yellow for both units due to having many open IODs on minor leaks and no leak monitoring plan.

Leak plan will be developed by Notification 200696825 Task 2 Owner Systems Engineering Due Date 5/1/2010

  • Monthly Surveillance Parameter Monitoring G G
  • Cooling Water Chemistry G G e EDG Fast Start Time G 0 Fast-start data trending per Tech Spec Basis 3.8.1 - No issues this quarter.
  • Fuel Oil Analysis G G
  • Fuel Oil Transfer System Performance G G
  • Lube Oil Analysis: Spectrochemical G B Higher than normal Silicon Level in engine oil for U3 Train B. Vendor guidance is to collect additional samples (200716420).
  • Lube Oil Analysis: Direct Reading Ferroscan G G No negative trends.
  • Mitigating System Perfo'rmance Indicator G B U2: 100 % Margin to White U3: 62% Margin to white
  • INPO Performance Indicator "  : B' ' -' 3% INPO 3 year Average of 3.36%. Plan to reach 2.5%

I___,_,.________ documented in Improvement Plan Problem Statement Problem Owner: )

On 6/1/09 during the monthly surveillance test run of Unit 2 Train A Diesel Generator 2G002, while attempting to manually raise generator load, the output megawatts experienced a 25% unexpected load drop from approximately 2.37 MW to 1.27 MW.

(200446662)

Corrective Action

1. During the next scheduled maintenance window for 2G002, visually inspect the entire length of the control wires for the Gov Raise signal and Gov homing signal, wherever these wires share the same cable bundle or pass through the same bulkhead fitting inside the 2L160 control panel. Replace or repair any wires which show signs of damage. Order 800335472 due date 6/9/2010.
2. Need to obtain a supply of 15 new DRUs for the next planned PM windows on 2L1_60, 2L1 61, 3L160 and 3L161 in 2012. Order 800335473 due date 12/31/2011.
3. For 2L1_60, 2L1 61, 3L1_60 and 3L1 61, change the DRU PM from an A8 interval to an A6 interval and eliminate refurbishment option. Order 800337128 due date 9/30/2010.

2 Problem Statement Problem Owner: i( 6 The Unit 3, Train-A, Emergency Diesel Generator tripped approximately 48 seconds after an idle-mode start was initiated and Operations declared the EDG inoperable. The control operator in the EDG room reported that the EDG speed increased abnormally slow. He also reported that low lube oil pressure alarms were received on the local EDG annunciator panel at the time of the trip. (200457220)

Corrective Action

1. For each of the 4 EDG control panels (2L160, 2L161, 3L160 and 3L161) create a new RMO to perform voltage checks at the power supply terminals of the EDG speed switch to ensure there are no excessive voltage spikes. (800344787 Complete)
2. Revise the 8 year RMO on the EDG annunciator power supplies to eliminate the refurbishment option and require that the power supplies be replaced with newly manufactured power supplies. (200457220 Task 7 5-31-2010)
3. Need to purchase 8 new annunciator power supplies for use on the 4 EDGs during the next AOT. (800345012 Complete) 141

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 3 Problem Statement Problem Owner: 1(b)(6)

On 12/1112009 Unit 3 Train B Emergency Diesel Generator (EDG) was declared inoperable when a technician grounded an energized wire in the local EDG junction box, opening a fuse and causing alarms in the Control Room due to an unidentified ground.

Technical Specification (TS) required starting the alternate train EDG (Train A) within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. On 12/12/09, operators were unsuccessful in their attempt to start Unit 3 Train A EDG when a capacitor failed in one of the two EDG (Train A) annunciator power supplies. As a result Unit 3 entered a TS required shutdown and an Unusual Event was declared for a loss of EDG trains. The Unit 3 shutdown from 96% power ended on 12/12/09 at 40% power when Train B EDG became operable. This downpower event was self revealing.

Corrective Action

1. Implement an Engineering Change Package (NECP) to design the EDG control circuitry to make the speed switch circuitry less susceptible to noise spikes from the annunciator power supplies.

U2 Train A - 800473163 U2 Train B - 800485046 U3 Train A - 800485047 U3 Train B - 800485049 Due Date 6/28/2010

2. Develop an Engineering Change Package (ECP) for the EDG controls to ensure that a failure of the local annunciator power supplies cannot impact the ability of the EDG to start, run, load. Due date 8/16/2010 (800455726)
3. Perform PM on ground monitor 50-DCF1 in Control Panel 2L1-60. Due Date 8/31/2010 (800463729)
4. Perform PM on ground monitor 50-DCF1 in Control Panel 2L161. Due Date 8/31/2010 (800463731)
5. Establish 2 Year PM for EDG ground monitor 50-CF1 in each Emergency Diesel Generator including the taking of voltages from the power supplies (terminal to terminal & terminal to ground). Due Date 8/10/2010 (800455483)
6. Identify and create Maintenance Orders to replace the obsolete lube oil temperature switches for each Emergency Diesel Generator. Complete
7. Implement monthly voltage checks on the speed switches of all four (4) EDGs to provide early indication of degrading power supply performance. Due Date 4/30/2010 (800453931)
8. Evaluate the current overhead lighting in the EDG room to improve the room brightness and ease of replacement. Due Date 6/12/2010 (800453937)
  • List contains EDG Component Improvements which are not being tracked by other EDG Initiatives 4 Problem Statement Problem Owner: (b)(6)

On December 7, 2009, during a monthly surveillance run on Unit 3 Train B Emergency Diesel Generator (EDG) 3G003, a flex hose in the lube oil system for developed a leak of 140 drops per minute, which caused the diesel to be declared inoperable. The cause of the leak was external corrosion pitting by chlorides. The hose was replaced and 3G003 was declared operable on December 10. The remaining susceptible hoses on all four EDGs are scheduled to be replaced in their respective BOWs in 2010.

Corrective Action

1. Ensure that status of the flex hose replacements is tracked in the EDG System Health Report. Complete 200695875-10.

Other Corrective Actions to replace flex hoses are tracked in Steel Braided Flexible Hose Replacement (Problem 6 in System Health Report)"

142

  • SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 5 Problem Statement Problem Owner: 1(b)(6)

During EDG troubleshooting it takes a minimum of two days to setup labview to perfom EDG monitoring. The Diesel has Technical Specification 3.8.1 which requires two Emergency Diesel Generators to be Operable in Modes 1 thru 4. With one Emergency Diesel Generator Inoperable, the Unit enters a 14 day shutdown LCO. Taking two days to set-up labview impacts the 14 day shutdown LCO, because no troubleshooting can be done during those two days. It would be better to have permanenl connections which could be setup more quickly. This modlification will allow system electrical component performance to be monitored, and quickly identify problems during troubleshooting.

Corrective Action U2 Train A Develop an NECP to install Data Acquistion System (DAS) for U2 Train A. Complete (800368840)

Install DAS in U2 Train A EDG. Due Date 8/20/2010 (800448810, 800448813)

U2 Train B Develop an NECP to install DAS for U2 Train B. Complete (8004400006)

Install DAS in U2 Train B EDG. Due Date 4/23/2010 (800429752, 800400008)

U3 Train A Develop an NECP to install DAS for U3 Train A. Complete (800399854)

Install DAS in U3 Train A EDG. Due Date 8/20/2010 (800400010, 800484838)

U3 Train B Develop an NECP to install DAS for U3 Train B. Due Date 4/23/2010 (800399856)

Install DAS in U2 Train B EDG. Due Date 6/25/2010 (800400011, 800485005) 6 Problem Statement Problem Owner: 1- 1 SONGS currently has 321 stainless steel braided flexible lines on the Fuel Oil and Lube Oil piping. Some of the lube oil piping has been replaced with Hasteloy material to protect against pitting corrosion due to the SONGS salt air environment. This notebook entry will track replacement of the rest of the 321 SS flexible connections with Hasteloy. This will remove the susecptibility of the lube oil and fuel oil flex hoses to pitting corrosion in the salt air environment (Notification 200695875 Task 0010).

Corrective Action

1. U2 Train A (8/20/2010)

Fuel Oil Flex Line Replacement (800039831)

Lube Oil Flex Line Replacement (800076056)

2. U2 Train B (4/23/2010)

Fuel Oil Flex Line Replacement (800075970)

Lube Oil replaced during installation of lube pump oil modification

3. U3 Train A (5/14/2010)

Fuel Oil Flex Line Replacement (800075973)

Lube Oil (800438656)

4. U3 Train B (6/21/2009)

Fuel Oil Flex Line Replacement (800075973)

Lube Oil Flex Line Replacement (800438656) excluding MY709, MY710 and MY719"

  • Replaced during December of 2009 143

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station

.. ".. EDGS 1 st Quarter - 2010 7 Problem Statement Problem Owner: i(b)(6)

The copper tubing in the EDG are known to have fretting failures due to rubbing from engine vibration. Additionally, the the flared fittings are known to cause failures due to the fitting not being a precision fitting. The tubing that is most susceptible are the fuel lines and the air start lines, because they tubing runs have had failures. However, water and oil copper tubing lines are also susceptible to the same failures. This modification will reduce the susceptibility the engine has to vibration induced failures, and leaking at flared fittings.

Corrective Action

1. Issue NECP to allow the replacement of Copper Tubing. Complete (800418793)
2. U2 Train A 800418712 - 8/20/2010 800432819 - 8/20/2010
3. U2 Train B 800422126 - 4/23/2010 800425531 - 4/23/2010
4. U3 Train A 800451970 - 5/14/2010 800457971 - 5/14/2010
5. U3 Train B 800451955 - 6/25/2010 800451956 - 6/25/2010 8 Problem Statement Problem Owner: I The EDG speed switch has a contact that closes when the engine reaches 150 rpm. When this contact closes it disables the air start solenoid which retracts the air start motors. With this present condition, Ifripple voltages caused by a degraded power supply are present the speed switch contacts will close preventing the EDG from starting. This condition produces a silent failure in which there is no indication of system failure. This modification will provide indication to local and control room operators that the EDG will not start in an accident condition in order to ensure compensatory measures are taken in a timely manner.

Corrective Action

1. U2 Train A NECP 800443797 Complete NMO 800445608 - 8/20/2010
2. U2 Train B NECP 800445522 Complete NMO 800445522 - 4/23/2010
3. U3 Train A NECP 800458463 Complete NMO 800459856 - 5/14/2010
4. U3 Train B NECP 800463807 Complete NMO 800475605 - 6/25/2010 144

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station rlPb O r EDGS 1st Quarter -2010 9 Problem Statement Problem Owner:(b)(6)

Each Fuel Oil Storage Tank has a high and low and low-low level switchs that provide hight alarm, low alarm, low-low alarm and low-low pump trip. Each fuel oil storage tank has two transfer pumps, one normal and one in auto mode are start and stopped by one level instrument in the day tank. Standby pumps have only manual control. Curently both pumps trip on low-low level of the fuel oil storage tank as sensed by a common level transmitter, There exists a potenial of losing both pumps in the fuel oil storage tanks form one level switch low-low malfunction. This modification will remove the possibility of the EDG storage tank level indicator failure causing the transfer pumps to become inoperable.

Corrective Action

1. Issue NECPs to jumper out the low-low level failure to prevent level instrument from causing both pumps in the Fuel Oil Storage tank from causing the EDG to be Inoperable. Complete (800071683,800071684)
2. U2 Train A 800260928 800380334 800453573 - 8/20/2010
3. U2 Train B 800380460 800260924 800450202 - 4/23/2010
4. U3 Train A 800380594 800380335 800458717 - 5/14/2010
5. U3 Train B 800380596 800302571 800458718 - 6/25/2010 145

SONGS System HEALTH REPORT

'San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 10 Problem Statement Problem Owner: :(b)(6) j There are a number of NECPs with are not associated with EDG reliability improvements or ACE/RCE Corrective Actions. The NECPs listed below are issued and awaiting implementation or work in progress (WIP).

Corrective Action U2 Train A 800071094- Issued 001200682-10 U3 DG 4kv Trn B alarm chgs NMO 800075839 Due Date 1/3/2011 800251283- Issued Remove Spare Alarm Contact for 2G002 NMO 800303047 Due Date: 1/2/2012 800449989 Upgrade lighting in EDQ Bldg for 2G002.

Due Date: 6/17/2010 800453513 - WIPE Replacement of Unit 2 EDG's 2G002 & 2G00 Due Date: 1/3/2012 U2 Train B 800251280- Issued Remove Spare Alarm Contact for 2G003 NMO 800303041 Rejected 800449991 .- WIP Upgrade lighting in EDG Bldg for 2G003.

Due Date: 05/28/2010 U3 Train A 800126250- Issued 080300355 Mod to U3 DGFO Flow inst NMO 800265183 Due Date: 6/21/2010 800251281 - Issued Remove Spare Alarm Contact for 3G002 NMO 800303042 Due Date: 7/5/2011 800449993 - WIP Upgrade lighting in EDG Bldg for 3G002.

Due Date: 4/29/2010 800493394 - WIP NECP for U3 EDG Intake Bellows Due Date: 6/2/2010 U3 Train B 800251282 - Issued Remove Spare Alarm Contact for 3G003 NMO 800303048 Due Date: 1/2/2012 800449995 - WIP Upgrade lighting in EDG Bldg for 3G003.

Due Date: 4/29/2010 146

(if SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 800454397- WIP Replacement of EDG 3G002 Due Date: 3/31/2011 Lon Rag Plan Perform evaluation for the feasability of replacing EDG relay start logic with a digital control upgrade Evaluation (Item 1.3 Project Plan)

I Owner: Design Engineering Due Date 10/1/2010 Develop a plan to Perform replacement of EDG Generators with New Ideal Generators (Items 1.8 Project Plan)

Owner: EDG Recovery Team Due Date: 7/1/2010 Create a formal Life Cycle Management Plan with focus on electrical and mechanical portions of the system.

Owner: Systems Engineering Due Date: 12/31/2010 Develop a strategy to address MDR Relays in the EDG starting logic.

Owner Maintenance Engineering

Due Date: 10/1/2010 Imroeen Pa (Oerl Rain of RE or YELW Major Problems Resolutions
1. Over the past three years the reliability and unavailability of 1.

the Emergency Diesel Generators (EDG) has been Red to Yellow (Completion by July 2010) unsatisfactory. There have been multiple failures of differing components in the mechanical and electrical systems. In U2 Train B (All Due Dates 4/23/2010) addition to cycling the organization, these failures have led to 800429752, 800400008 Install Data Acquisition System (DAS) increasing unplanned equipment unavailability. Both of these Owner: Construction indicators (reliability and unavailability) have been driven by 800075970 Install EDG Fuel Oil Flex Lines Owner: Machinist an inadequate performance monitoring program. With an 800422126, 800425531 Replace EDG copper tubing with adequate EDG performance monitoring program failures Stainless Steel Owner: Machinist could be avoided and unplanned unavailability would be 800445608 Modify EDG Circuitry to annunciate if speed switch minor. The following actions listed will improve the equipment in the incorrect position Owner: Test Techs reliability, address the high unavailability and implement a 800380460, 800450202 Modify EDG Transfer Pump circuitry to performance monitoring program. The completion of actions prevent level switch failure from causing the transfer pumps to will be tied directly to overall system health.

  • become inoperable Owner: Electrical Improvement Plan documented in Notification 200603808 U3 Train A (All Due Dates 5/14/2010) 800400010, 800484383 Install Data Acquisition System (DAS)

Owner: Construction 800075973, 800438656 Install EDG Fuel/Lube Oil Flex Lines Owner: Machinist 800451970, 800457971 Replace EDG copper tubing with Stainless Steel Owner: Machinist 800459856 Modify EDG Circuitry to annunciate if speed switch in the incorrect position Owner: Test Techs 800380594, 800458717 Modify EDG Transfer Pump circuitry to prevent level switch failure from causing the transfer pumps to become inoperable Owner: Electrical U3 Train B (All Due Date 6/25/2010) 800400011, 800485005 Install Data Acquisition System (DAS)

Owner: Construction 800075973, 800438656 Install EDG Fuel/Lube Oil Flex Lines Owner: Machinist 147

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station EDGS 1st Quarter - 2010 Improvement Pla Major Problems Resolutions 800451956, 800451955 Replace EDG copper tubing with Stainless Steel Owner: Machinist 800380596, 800458718 Modify EDG Circuitry to annunciate if speed switch in the incorrect position Owner: Test Techs Yellow to Blue (Completion by December 2010)

U2 Train A (All Due Dates 8/20/2010) 800448810, 800448813 Install Data Acquisition System (DAS)

Owner: Construction 800039831, 800076056 Install EDG Fuel Oil Flex Lines Owner:

Machinist 800418712, 800432819 Replace EDG copper tubing with Stainless Steel Owner: Machinist 800445608 Modify EDG Circuitry to annunciate ifspeed switch in the incorrect position Owner: Test Techs 800260928, 800453573 Modify EDG Transfer Pump circuitry to prevent level switch failure from causing the transfer pumps to become inoperable Owner: Electrical Develop a plan and implement actions to improve EDG Unavailability to less than 2.5% within 4 years which contains generator replacement Owner: EDO Recovery Team Due Date: 7/1/2010 Capture planned EDG unavailability in EDG Allowed Outage Time (AOT) schedule Owner: EDG Recovery Team Due Date: Complete Implement EDO Diagnostic Monitoring (Engine Analysis)

Owner: EDG Recovery Team Due Date: 7/1/2010 Develop an EDG performance monitoring program to trend system performance to identify degrading conditions Owner: EDG Recover Team Due Date: 6/1/2010 148

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 DCPS - DC Power Systems Previous Four Quarters 20 2009 2009 2009 2009 QTR-1 QTR-2 OTR-3 QTR-4 Unit 2 y 1B Unit 3 Y System Engineer: (b)(6) Date: 4/14/2010 SuDervisor ADproval: Date: 4/23/2010 Plant Health Committee iDate: 10/31/2002 Overall Health Trend Rating (G,B,Y,R)

Unit 2 B Unit 3 B

Road toGre Overall Rating: BLUE The U2 DC System ranking is "BLUE". Since the plant can take credit for the cross-tie capability to 2B009, the Unit 2 completion of 2B007 is not considered a constraint to return the system to "Blue".

The U2 DC System will return to "GREEN" when 2B007, 2B016 and battery chargers 2B001, 28002, 2B003 and 2B004 have been replaced.

Battery 2B007 is scheduled for on-line replacement in May 2010 as originally planned.

Battery 2B016 & battery chargers 2B015 were replaced in U2C16 refueling outage.

Battery chargers 2B001, 2B002, 28003 and 28004 are scheduled for on-line replacement in 2010. For component replacement schedule refer to SPL or LRP.

Individual battery rankings are:

28007 is RED (to be replaced online in 2010) 2B008, 28009 and 2B010 are GREEN 2B011 is BLUE 2B012 is GREEN 2B016 is GREEN 28019 is GREEN Swyd batteries are GREEN Individual battery charger rankings are:

28001 is YELLOW (to be replaced online in 2010) 28002 is YELLOW (to be replaced online in 2010) 28003 is YELLOW(to be replaced online in 2010) 28004 is GREEN 28005 & 2B025 are GREEN 2B006(A) is YELLOW (to be replaced in U2C17) 2B015E, 2B015C and 2801 5W are GREEN 2B018E & 21018W are BLUE 157

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 RodtoJ'* Green

'J"*

Overall Rating: BLUE The U3 DC System ranking is "BLUE".

The U3 DC System will return to "GREEN" when battery 3B016, battery chargers 3B015 and battery chargers 3B001, 38002, 38003 and 38004 have been replaced.

Battery 38016 &battery chargers 38015 will be replaced in U3C16 refueling outage.

Battery chargers 38001, 38002, 38003 and 38004 are scheduled for on-line replacement in 2010.

For component replacement schedule refer to SPL or LRP.

Individual battery rankings are:

38007 and 3B008 are GREEN 3B009 and 3B010 are GREEN 38011 is BLUE 3B012 is GREEN 3B016 is YELLOW (to be replaced in U3C16) 3B019 is GREEN Individual battery charger rankings are:

38001 is YELLOW (to be replaced online in 2010) 38002 is YELLOW (to be replaced online in 2010) 3B003 is YELLOW (to be replaced online in 2010) 38004 is YELLOW (to be replaced online in 2010) 38005 is YELLOW (38005 to be replaced and 3B025 to be added in U3C 16) 38006(A) is YELLOW (to be replaced in U3C17) 38015 is YELLOW (38015 to be replaced with 3B015E, 3B015C &3B015W in U3C16) 3B018E is BLUE 38018W is BLUE 158

-1

> \>~ SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 Rellabilty 0 Lost Generation G G U2 &3: Zero EFPD and Zero Events.

Maintenance Rule Status

  • (a)(1) / (a)(2) Status B B U2 &3: Meeting all Performance criteria limits.

" RMRFFs G G U2 &3: No RMRFF in 36 months.

  • Unavailability G G 2-MR-DC-01 annual limit is 3.3 hrs -

Total Yr UA hrs 0.0.

2-MR-DC-02 annual limit is 3.3 hrs -

Total Yr UA hrs 0.6.

2-MR-DC-03 annual limit is 16.3 hrs -

Total Yr UA hrs 0.7.

2-MR-DC-04 annual limit is 272.2 hrs -

Total Yr UA hrs 0.0.

2-MR-DC-05 annual limit is 16.3 hrs -

Total Yr UA hrs 0.0.

3-MR-DC-01 annual limit is 5.3 hrs -

Total Yr UA hrs 0.0.

3-MR-DC-02 annual limit is 5.3 hrs -

Total Yr UA hrs 0.0.

3-MR-DC-03 annual limit is 26.3 hrs -

Total Yr UA hrs 0.0.

3-MR-DC-04 annual limit is 438.0 hrs -

Total Yr UA hrs 0.0.

3-MR-DC-05 annual limit is 26.3 hrs -

Total Yr UA hrs 0.0.

  • Plant Level Events G G U2 &3: Meeting PLPC.

Functlonal Material Condition

  • Walkdown Results Y Y U2 &3 batteries and battery chargers need to be replaced (see SLP or LRP for details).
  • Open Corrective Maintenance Tasks G G U2: 0 CMTs.

U3:2 CMTs.

  • Open Elective Maintenance Tasks B Y U2:9 EMTs.

U3: 27 EMTs.

  • PM/PdM Results and Feedback G G None this quarter

" Open Operator Workarounds G G None this quarter

  • Open Operational D5ecision Making G G None this quarter

" Open Operator Burdens G G None this quarter

  • Open Control Roomn Deficiencies G G None this quarter
  • Open Annunciator Compensatory Actions G G None this quarter Design Basis and Configuration Health
  • Open Degraded & Nonconforming Conditions G G None this quarter
  • Open Engineering Modification Requests (EMRs) G G None this quarter
  • Temp ECPs Installed G G None this quarter None this quarter
  • Open Doc Assignments G G None this quarter Operating Experience
  • Equipment Performance and Information G G None this quarter Exchange (EPIX) 159

N~ ~ SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010

  • 1 0CFR Part 21 issues G G None this quarter
  • INPO OEs B B For details refer to LRP 9 NRC Info Notices G G None this quarter
  • Benchmarking G G None this quarter AR Baddog 0 Open RCE (Root Cause Evaluations) G G None this quarter 9Open ACE (Apparent Cause Evaluations) G G None this quarter
  • Open ACE/RCE Required Corrective Actions G G None this quarter Addhtional Signfficant Issues
  • Chronic problems or adverse trends Y Y U2 & 3 battery and battery charger aging issues (refer to SPL or LRP).

0 Reactivity Management Events NA NA

  • Unplanned Shutdown LCOs G G None this quarter

" Margin B G 2B007 <3% battery margin

" Assessments / Audits / Inspections G G None this quarter

" Other G G None this quarter

' U. * - - . *' S Parametler Trendilng System PKA ___

  • 1E 125 VDC Battery Bank B007 G 2B007 is RED due to low margin and battery end-of-life.

Scheduled replacement window:

5/31/10 - 7/1/10 3B007 is GREEN due to high margin and battery

_______, _ replacement in 2009

  • 1E 125 VDC Battery Bank B008 G G 2B008 is GREEN due to high margin and battery replacement in 2009 3B008 is GREEN due to high margin and battery replacement in 2009
  • 1E 125 VDC Battery Bank B009 G G 2B009 is GREEN due to high margin and battery replacement in 2005 3B009 is GREEN due to high margin and battery replacement in 2005
  • 1E 125 VDC Battery Bank B010 G G 2B010 is GREEN due to high margin and battery replacement in 2005 3B010 is GREEN due to high margin and battery replacement in 2005 S1 E 125 VDC Charger B001 Y Y 2B001 is YELLOW due to end-of-life.

3B001 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

S1 E 125 VDC Charger B002 Y Y 2B002 is YELLOW due to end-of-life.

3B002 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

1E 125 VDC Charger B003 Y Y 2B003 is YELLOW due to end-of-life.

3B003 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

S1 E 125 VDC Charger B004 Y Y 2B004 is YELLOW due to end-of-life.

3B004 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

160

SONGS System HEALTH REPORT W10ý San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010

  • 1E 125 VDC Distribution D1 B B 2D1 is BLUE meeting management expectations 3D1 is BLUE meeting management expectations
  • 1 E 125 VDC Distribution D2 B B 2D2 is BLUE meeting management expectations 3D2 is BLUE meeting management expectations
  • 1 E 125 VDC Distribution D3 B B 2D3 is BLUE meeting management expectations 3D3 is BLUE meeting management expectations
  • 1 E 125 VDC Distribution D4 B B 2D4 is BLUE meeting management expectations 3D4 is BLUE meeting management expectations Parameter Trending Sysftm NKA

" Non-IE 125 VDC Battery Bank B01 1 B B 2B01 1 is BLUE meeting management expectations.

3B01 1 is BLUE meeting management expectations.

Due to end-of-life, battery replacement may be required in the future (see SPLor LRP).

" Non-1 E 125 VDC Charger B005 G Y 2B005 is GREEN due to new installation in U2C1 6 outage.

3B005 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

" Class 1 E 125 VDC Charger B025 G NA 2B025 is GREEN due to new installation in U2C16 outage.

Note: The Class 1E charger feeds a Non-i E bus to provide power from a diesel generator backed power source.

3B025 will be installed in U3C16 outage.

" Non-lE 125 VDC Distribution D5 B B 2D5 is BLUE meeting management expectations.

3D5 is BLUE meeting management expectations Parameter Trending System NJA

" Non-lE 250 VDC Battery Bank B012 G G 2B012 is GREEN due to high margin and battery replacement in U2C15.

3B012 is GREEN due to high margin and battery replacement in U3C1 5.

" Non-1E 250 VDC Battery Bank B016 G Y 2B01 6 is GREEN due to high margin and battery replacement in U2C16.

3B016 is YELLOW based on low margin and end-of-life.

B016 output -120 kVA instead of the 150 kVA rating of the other UPS components. For battery replacement in U3C15 see SPL or LRP.

" Non-lE 250 VDC Battery Bank B019 G G 2BO19 is GREEN due to high margin and battery installation in 2002.

3B019 is GREEN due to high margin and battery installation in 2003.

" Non-lE 250 VDC Charger B006 Y Y 2B006 is.YELLOW due to end-of-life.

3B006 is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

" Non-iE 250 VDC Charger B006A Y Y 2B006A is YELLOW due to end-of-life.

3B006A is YELLOW due to end-of-life.

For charger replacement see SPL or LRP.

  • Non-iE 250 VDC Charger B015E, B015C, B015W G 2B015E, 2B015C, 2B015W are GREEN due to high margin and charger replacement In U2C16.

" Non-lE 250 VDC Charger B015 Y 3B015 is YELLOW due to end-of-life.

For charger replacement in U3C16 see SPL or LRP.

" Non-lE 250 VDC Charger B018E B B 2B018E is BLUE due to external corrosion.

3B018E is BLUE due to external corrosion.

HVAC was installed to eliminate or mitigate the effect of the marine environment.

161

/

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 Non-lE 250 VDC Charger B018W B B 2B018W is BLUE due to external corrosion.

3B018W is BLUE due to external corrosion.

HVAC was installed to eliminate or mitigate the effect of the marine environment.

  • Non-I E 250 VDC Distribution D6 B B 2D6 is BLUE due to end-of-life and external corrosion.

306 is BLUE due to end-of-life and external corrosion.

  • Non-iE 250 VDC Distribution Y012 G Y 2D8 Distribution is GREEN due to replacement in U2C16.

3D8 Distribution is YELLOW due to end-of-life.

The Non 1 E UPS has become obsolete and is subject to accelerated degradation of inverters, battery chargers and batteries. For component replacement see SPL or LRP.

  • Non-iE 250 VDC Distribution D7 B B 2D7 is BLUE due to external corrosion.

3D7 is BLUE due to external corrosion.

HVAC was installed to eliminate or mitigate the effect of the marine environment.

Syte Prbe List 1 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance PKA - Class 1E Battery Replacement 2B007, 2B008, 3B007 & 3B008 (D1, D2).

Replace batteries based on the latest performance history and battery sizing margins.

Current Class 1 E battery margin for 28007 is < 3%.

Preferred industry Class 1 E battery margin after 20 years of operation is 5 to 10%, which can be met with 1800 AH batteries.

Batteries 2B008, 3B007 & 3B008 were replaced in 2009.

Corrective Action Replace battery 2B007 in 2010. Confirmed Replacement Window:

2B007:5/31/10 - 7/1/10 NN 200176280- NECP 800074405- NMO 800 074 405 Starting in 2009, batteries 2B008, 3B007 & 3B008 will receive a modified performance discharge test every 30-months instead of a separate service and performance test (2B007 MPT will start in 2010).

Note that spare battery 2/3B00X will be used to replace 28007 and starting July 2010 spare battery 2/3800X is no longer available as a substitute for other batteries.

2 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance PKA - Class 1E Battery Replacement 2B009, 2B010, 3B009 & 3B010 (D3, D4).

Batteries 2B009, 2B010, 3B009 & 3B010 were replaced in 2005.

Corrective Action Starting in 2009, batteries 2B009, 2B010, 3B009 & 3B010 will receive a modified performance discharge test every 30-months instead of a separate service and performance test.

162

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 3 Problem Statement Problem Owner: Design Engineering Maintenance Engineering, Maintenance PKA - Class 1E Battery Charger Replacement 28001, 2B002, 28003, 28004, 3B001, 38002, 3B003 & 3B004 (D1, D2, D3, D4).

The existing Class 1 E battery chargers 2(3)BOO1, 2(3)B002, 2(3)B003, 2(3)B004 are obsolete and need to be replaced before the scheduled PCB replacement activities on the existing chargers. Implementation: No later than 2010 The eight Train-Aligned replacement chargers will be the same as the recently added 400A swing chargers except:

The current limit will be adjusted down so that the chargers limit themselves to the output that would be delivered by a 300A rated charger, and there is no need for a two-section charger configuration because special physical separation breakers are not required.

Corrective Action The Class 1E battery chargers are Critical-A equipment and are Identified for replacement in 2010 prior to the RMO expiration date of the PCBs. The plan was for Maintenance to replace the chargers in 2010 using Material Code 02728616 in lieu of PCB refurbishment as a Level of Effort (LOE) project subject to EMR approval (NN 200232109 & 200172525) utilizing maintenance capital overhaul accounting.

Replace Class 1 E chargers in confirmed installation work window (2010) with mat code 02728616 using capital overhaul accounting:

Unit 2 NN 200232109 (2) replaced by NN 200446150 (U2):

Batt Charger Work Window (installation) NECP generation and Charger delivery in 2009 28001: July 2010 NECP 800 311 863 - NMO 800 311 872 2B002: May 2010 NECP 800 311 864 - NMO 800 311 874 2B004: April 2010 Completed NECP 800 311 866 - NMO 800 311 878 2B003: June 2010 NECP 800 311 865 - NMO 800 311 876 Unit 3 NN 200172525 (U3) replaced by NN 200446150 (U2):

Batt Charger Work Window (installation) NECP generation and Charger delivery in 2009O1DOLJO 3B001: July 2010 NECP 800 311 867 - NMO 800 311 879 3B004: August 2010 NECP 800 311 868 - NMO 800 311 862 3B003: September 2010 NEOP 800 311 870 - NMO 800 311 965 3B002: October 2010 NECP 800 311 871 - NMO 800 311 966 The Notifications and Orders for charger testing and RMO-PCB due dates are:

28001 Charger PCB replacement September 2010. NN 200165530 - 28001 Existing Charger PT - OR 800172509 - 05/26/2010.

28002 Charger PCB replacement April 2010. NN 200023000 - 28002 Existing Charger PT - OR 800066786 - 02/22/2010.

2B003 Charger PCB replacement June 2010. NN 200023550 - 28003 Existing Charger PT - OR 800067340 - 03/08/2010.

28004 Charger PCB replacement May 2010. NN 200152212 - 28004 Existing Charger PT - OR 800166150 - 07/01/2010.

3B001 Charger PCB replacement August 2010. NN 200015724 - 3B001 Existing Charger PT - OR 800048421 - 12/22/2010.

38002 Charger PCB replacement February 2010. NN 200017724 - 38002 Existing Charger PT - OR 800050834 - 12/15/2010.

38003 Charger PCB replacement August 2010. NN 200014165 - 38003 Existing Charger PT - OR 800046631 - 12/01/2010.

38004 Charger PCB replacement October 2010. NN 200159140 - 3B004 Existing Charger PT - OR 800169713 - 07/21/2010.

163

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 4 Problem Statement Problem Owner: Design engineering, Maintenance Engineering, Maintenance NKA - Non-1 E Battery Performance Test/Replacement 2B011 & 3B011 (D5)

Battery 31011 has reached the End-of-Service-Life (February 2007).

Continued operation is justfied because battery is performing at 105% capacity without signs of unacceptable degradation.

Replacement may be required in the next 5 years when battery capacity drops to <95%. Anticipated Implementation: 2010 thru 2013 Battery 2B011 will reach the End-of-Service-Life (June 2013).

Continued operation is justfied because battery is performing at 100% capacity without signs of unacceptable degradation.

Replacement may be required after 2013 when battery capacity drops to <95%. Anticipated Implementation: 2013 thru 2018 Corrective Action Baltedes are monitored closlely to detect degradation and capacity loss.

Battery 38011 has reached the End-of-Service-Life (February 2007) and will receive a performance test in each refueling outage.

Battery 2B011 has not reached 85% End-of-Service-Life (June 2010) and will receive a performance test every second refueling outage.

5 Problem Statement Problem Owner: Design engineering, Maintenance Engineering, Maintenance NKA - Non-i E Battery Charger Replacement 28005 & 3B005 (D5)

Existing non-Class 1E battery chargers 2(3)B005 are obsolete.

2B005 was replaced in U2C16 outage. 38005 will be replaced in U3C16 outage.

Battery chargers 2(3)B005 are scheduled to be replaced to avoid any potential problems in taking corrective actions due to lack of spare parts.

As part of the 2Y005 inverter replacement project it was decided to use two new 400 amp chargers (one Class 1E and one Non-1 E).

Corrective Action The Notifications and Orders for charger replacement and testing are:

NN 200090636 - 2Y005 Inverter and 28005 battery charger replacement - NECP 800135812 - Super 0: 800134233 - NPS 800135802 NN 200187365 - 3Y005 Inverter and 3B005 batery charger replacement - NECP 800072036 - Order 800159974 - Super 0:

800072035 6 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance NJA - Non-1E Battery Replacement 2B012 and 3B012 (D6) 2B012 battery bank was replaced in U2C15 refueling outage because of the multiple failures of individual cells and the age of the battery.

3B012 battery bank was replaced in U3C15 refueling outage because of the multiple failures of individual cells and the age of the battery.

New battery racks are required to support implementation of 2B016 & 3B016 in U2C16 & U3C16. Reconfiguration of 2B012 was completed in U2C16 outage. Reconfiguration of 3B012 is scheduled for U3C16 outage.

Corrective Action Installation of 2-tier racks to make room for the larger capacity batteries (28016 & 3B016) to be responsive to the predicted UPS load growth in the future will be performed in Cycle 16 refueling outages.

NN 200046082 - 2(3)B012 Battery Rack Replacement - U2C16 & U3C16.

164

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 7 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance NJA - Non-1 E Battery Charger Replacement 2B006(A) & 3B006(A) (D6)

Battery chargers 2(3)BO06A and 2(3)B006 are 27 years old and need to be replaced. The marine environment in the turbine building and the voltage drop limitations of these chargers require consideration for new and larger battery chargers (300-500 amps).

Anticipated imlementation: No later than 2010/2011.

The 200 amp rated 2(3)BO06A normal charger and 100 amp 2(3)B006 stand-by chargers have no internal heaters and for ventilation purposes, both the lower front and the rear covers have perforations which expose the internals to the outside environment (cold, damp and salty air). The outside covers have been painted on regular basis, but the inside panels are rusted and the copper components have tarnished. Every 2 years, these chargers are subject to PM to maintain them in a working state.

Since the normal charger B006A goes in current limit and D6 bus voltage drops from nominal 275 volts to approximately 250 volts whenever the DC emergency turbine lube oil pump starts (starting current being 450 amps approx.), larger size chargers should be considered for upgrade for both normal and stand-by. Improvement in PM program will provide limted assistance to extend the life of equipment. The chargers are obsolete and vendor support will be minimal. 2(3)B006 chargers are now showing either signs of drift or component failures.

Corrective Action Replace existing obsolete normal charger 2(3)BO06A and stand-by charger 2(3)B006 with new latest technology model equipped with required control and monitor functions. Due to the voltage drop limitations identified above larger battery chargers (300-500 amps) are recommended.

NN 200152186 - 2(3)B006 charger replacement 8 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance NJA - Non-1E UPS Battery Replacement 2B016 & 3B016 (Y012)

The Non 1 E UPS has become obsolete and is subject to accelerated degradation of inverters, battery chargers and batteries (insufficient battery capacity to support 150 kVA rating of UPS inverter). Existing batteries cannot function at or near the 150 kVA rating of the UPS components. Currently, the UPS is fully loaded and future load growth is not possible without an increase in battery capacity. The UPS power output is administratively limited by the existing 90 minute AH capacity of batteries 2B016 and 3B016 which allow the system to operate only at about 120 kVA.

In 2008, it was decided to include the B016 battery reolacement (EEB 060900047-4) in the Y01 2 inverter replacement project (AR 020300093-2) including the use of 2-tier racks for 28012 and 38012 (EEB 060800539-5).

Corrective Action Replace the Non-1 E UPS batteries 28016 and 38016 so they can function at or near its 150 kVA rating:

2Y012 UPS Inverter and 28016 Battery replacement project was completed during this quarter in 2010 (U2C16).

The Notifications and Orders are:

NN 2001155451/0-800129814 2Y012 UPS inverter, 2B016 battery and 28015 battery charger replacement project was completed in U2C16.

NN 200115452/0-800128908, SO-800167910 3Y012 UPS inverter, 3B016 battery and 2B015 battery charger replacement project - Scheduled for U3C16 NN 200155920 - WNU-00042: Replace 2Y012, 28016 & 2B015 NN 200019604 - 28016 Battery Performance Test 165

SONGS System HEALTH REPORT (t San Onofre Nuclear Generation Station

- DCPS 1 st Quarter - 2010 9 Problem Statement Problem Owner: Design Engineering, Maintenance Engineering, Maintenance NJA - Non-1E Battery Charger Replacement 2B015 & 3B015 (Y012)

Battery Charger 2B015 & 3B015 are obsolete and need to be replaced. Recommended Implementation: C1 6 Refueling outages.

The charger vendor, Ametek Solidstate Controls Inc. (Ametek SCI) recommends that SCR rated voltage be increased form original 500 volts to the latest 800 volts, so that the additional safety margin of the SCR voltage will reduce possibilities of transient voltages exceeding the rating of the device. The vendor also recommends that the current CVT capacitors be replaced with Ametek SCI part number 03--020139-00. The unique construction of the Ametek part allows for high transient voltages without concern of failure.

EEB 020300093-28: Replace Inverter Y012, Charger B015, Rework HVAC, Remove Y010 and connect T01 1& T069 for annual Maintenance.

The equipment is more than 20 years old and exposed to cold and moist condition in turbine building room 101at Elev. 7'.

Improvement in PM program will provide limted assistance to extend the life of equipment. BOIS chargers are now showing either signs of drift or component failures.

Corrective Action Replace existing obsolete charger B015 with new latest technology model equipped with required control and monitor functions.

The Notifications and Orders are:

NN 2001155451/0-800129814 2Y012 UPS inverter, 2B016 battery and 2B015 battery charger replacement project - Completed in U2C16 (2010).

N N 200115452/ 0-800128908, SO-800167910 3Y012 UPS inverter, 3B016 battery and 2B015 battery charger replacement project - Scheduled for U3C16 NN 200155920- 2(3) B015 charger replacement (refer to Y012 UPS Inverter replacement took place in 2010)

OR 800072279 - 2B015 charger replacement was completed in U2C16 outage by implementing 3 individual chargers: 2B015E, 2B015C & 2B015W.

The Non 1E LIPS is presently used for Security loads and various Non 1E UPS instrument Buses. During replacement of B015 temp. modification and compensatory security steps will be considered. The new models will be installed at the existing locations.

10 Problem Statement Problem Owner: Maintenance Engineering, Maintenance NJA - Non-1 E Battery B019 (D7)

The D7 electrical battery and charger rooms have experienced ash intrusion due to absence of door weather stripping (ash from Hondo fire).

The 5-year battery performance discharge test is required in Cycle 16.

Corrective Action The Notifications and Orders for battery maintenance and testing are:

NN 200003177 - Door weather stripping (AR 071001231: Replace filters and door weather stripping) - to be scheduled.

NN 200102303 - 2B019 battery performance test - OR 800140834 10/19/2009 - scheduled for U2C16.

NN 200227790 - 38019 battery performance test - to be scheduled for U3C1 6.

166

SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 11 Problem Statement Problem Owner: Maintenance Engineering, Maintenance NJA - Non-1E Battery Charger B018 (D7)

The D7 electrical charger room had experienced extensive corrosion during first yearin service. Marine air intrusion into charger cabinet caused internal damage and ash intrusion was observed to absence of door weather stripping (ash from Hondo fire).

Corrective Action The Notifications and Orders are:

NN 200003177 - Door weather stripping (AR 071001231: Replace filters and door weather stripping) - to be scheduled.

Charger overhaul, cleanup and sheet metal replacement was completed in C15. Also, HVAC was installed to prevent marine air from damaging components inside charger room.

12 Problem Statement Problem Owner: Maintenance Engineering, Maintenance Non-i E Battery Charging Current Measurements:

Permanent test jacks (and shunts as applicable) are required to enable measurement of battery float charging current reading less than 2 amps for:

2D5-BO11, 2D6-B012, 2D7-B019, 2Y012-B016, ADP1 -BA1, ADP3-BA2, 3D5-B011, 3D6-B012, 3D7-B019 and 3Y012-B016.

The implementation of float charging current measurement capability on non-Class 1 E batteries is highly desired to establish a consistent methodology for all batteries in the plant to proactively reduce human error (EEB 051200026-2).

Corrective Action The Notifications and Orders are:

NN 200004155 - U2 Scope for ECPs in AR 051200026- N-PHI & N-EMR NN 200004156 - U3 Scope for ECPs in AR 051200026 - N-PHI & N-EMR The following equipment requires permanent test jacks (and shunts as applicable):

2D5-B01o - Part of Y005 replacement ECP - scheduled for U2C1 6 3D5-B01 1 - Part of Y005 replacement ECP - scheduled for U3C1 6 2D6-8012 - Part of Y01 2 replacement ECP - scheduled for U2C1 6 3D6-B012 - Part of Y01 2 replacement ECP - scheduled for U3C1 6 2Y012-B016 - Part of Y012 replacement ECP - scheduled for U2C16 3Y012-B016 - Part of Y012 replacement ECP - scheduled for U3C16 2D7-B019 - to be scheduled 3D7-B019 - to be scheduled ADP1 -BA1 - to be scheduled ADP3-BA2 - to be scheduled 167

(~Ny\ SONGS System HEALTH REPORT San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 13 Problem Statement Problem Owner: Maintenance Engineering, Maintenance Evaluate and initiate actions resulting from OEs and regulatory observations such as CDBI.

Corrective Action Notification 200087477 - Orders 800135545, 800135739 & 800136560:

This notification documents a CDBI observation that SONGS uses the LOCA/LOOP load profile for the battery service test, which was interpreted by the NRC as not meeting TS SR 3.8.4.7 requirements. For completed and planned corrective actions refer to the details in the Notification 200087477.

Other OEs associated with DC systems:

OE26600 "Low Battery Voltage due to Loose Breaker to Bus Bar Connection Bolts at SONGS, dated 3/25/2008.

Lon Rag Plan~

1. Maintenance Rule Performance Continue to maintain and operate the system without FFs. MR DC-01, DC-02, DC-03, DC-04, DC-05, DC-06 & DC-07 exceedance criterion is 2 FFs in a 36-month period. For this reporting quarter there were no MR Functional Failures.
2. Class 1E Battery Replacement and DC Reconfiguration Project AR 001000280 evaluated options to enable online Class 1 E battery testing and maintenance (including battery replacement) to eliminate the scope of battery work during outages by allowing cross connection of Channel A and C (or Channel B and D) batteries during Modes 1 through 4. Currently, all Class 1E batteries can be tested online.

New Technical Specification surveillance frequencies and new site procedures are to be used for any routine maintenance and testing activities of the Class 1E DC systems. In conjunction with modifications (that are still in progress) Class 1 E 125 VDC system maintenance work can be performed online using 1800 amp-hour rated batteries.

Batteries 2B009, 28010, 3B009 and 31010 were replaced in 2005. Batteries 3B008, 3B007 and 2B008 were replaced in 2009.

Battery 2B007 is scheduled for replacement 5/31/10 - 7/1/10 - NECP 800074405. Spare battery 2/3B00X will be used to replace 2B007.

3. Class 1 E Battery End-of Service-Life:

ClasslE batteries may be replaced "ONLINE" at 85% of the End-of-Service-Life (ESL=20 yrs) or extended by annual performance testing.

Bus-Battery ID Rated AH Capacity In-Service 85% of ESL Remarks 2D1-B007 U2-1260 08/93 08/10 125V Class 1E - Subsystem A 3D1-B007 U3-1 800 06/09 06/26 2D2-B008 U2-1800 09/09 09/26 125V Class 1E - Subsystem B 3D2-B008 U3-1800 05/09 05/26 2D3-B009 U2-1 800 05/05 05/22 125V Class 1E - Subsystem C 3D3-B009 U3-1 800 02/05 02/22 2D4-BO10 U2-1 800 07/05 07/22 125V Class 1E - Subsystem D 3D4-8010 U3-1 800 03/05 03/22 2/3BOOX-, 2/3-1800 06/04 06/21 125V Class 1E - Spare (Subsystem A, B, C, or D)

4. Non-Class 1 E Battery End-of-Service-Life:

Non-iE batteries may be replaced at the End-of-Service-Life (100% of ESL=20 yrs) or justified for continued operation.

Bus-Battery ID Rated AH Capacity In-Service 100% of ESL - Unit Cycle Remarks 2(3)D5-B01 1 2380 06/93 (02/87) 06/13 (02/07) - C18 (C14) 125V Non-lE 2(3)D6-B012 2057 C15 (C15) 08/23 (08/24) - C24 (C25) 250V Non-i E 2(3)D7-B019 4290 06/02 (02/03) 06/22 (02/23) - C23 (C23) 250V Non-i E 2(3)YO12-BO16 2057 07/93 (11/93) 07/13 (11/13) - C19 (C19) 250V Non-i E ADP1-BA1 1070 05/01 05/21- NA SCE-Swyd 125V Non-1E ADP3-BA2 1038 11/99 11/19 - NA SDG&E-Swyd 125V Non-lE

  • Battery 3D5-B1011 will be assessed for replacement when performance drops below 95%.
5. Non-i E Battery Charging Current Measurements:

168

SONGS System HEALTH REPORT

'San Onofre Nuclear Generation Station DCPS 1st Quarter - 2010 Permanent test jacks (and shunts as applicable) are required to enable measurement of battery float charging current reading less than 2 amps for:

2D5-B011, 2D6-B012, 2D7-B019, 2Y012-B016. ADP1 -BA1, ADP3-BA2, 3D5-B011, 3D6-B012, 3D7-B019 and 3Y012-B016.

The implementation of float charging current measurement capability on non-Class 1 E batteries is highly desired to establish a consistent methodology for all batteries in the plant to proactively reduce human error (EEB 051200026-2). For details refer to SPL #12.

6. 250 VDC Battery Improvement Items:

Batteries 2D6-2B012 and 3D6-3B012 were replaced in the Cycle 15 refueling outages, due to the battery bank's age and the history of cell failures.

Batteries 26016 and 38016 (Non-1 E UPS) will be replaced by larger capacity batteries in the Cycle 16 refueling outages, so they can support the 150 kVA rating of the other components. Currently, the UPS is fully loaded and future load growth is not possible without an increase in battery capacity. The UPS power output is administratively limited by the existing 90 minute AH capacity of batteries 2B016 and 38016 which allow the system to operate only at about 110 kVA. To make room for installation of the higher capacity Non-1 E UPS battery the four existing rows of single tier battery racks for batteries 2B012 and 38012 need to be replaced with two rows of new 2-tier racks. Battery 28016 was replaced in U2C16 outage. Battery 3B016 will be replaced in U3C16 outage.

7. Class 1 E Battery Charger Replacement The existing Class 1E battery chargers 2(3)B001, 2(3)B002, 2(3)B003, 2(3)B004 are obsolete and need to be replaced before the scheduled PCB replacement activities on the existing chargers. The printed circuit boards (PCBs) of the existing battery chargers were replaced in 2004/2005 and are due for replacement in 2010/2011. All Class 1 E battery chargers are currently scheduled to be reolaced in 2010.

For details refer to SPL #3.

8. Non-Class 1 E Battery Charger Replacement Existing non-Class 1E battery chargers 2(3)B005, 2(3)B006, 2(3)B015 are obsolete. SE recommends that battery chargers 2(3)B005, 2(3)B006, 2(3)B015 be replaced no later than 2010/2011 to avoid any potential problems in taking corrective actions due to lack of spare parts.

A new Class 1 E charger 2B025 was added to feed bus 2D5 to provide power from a diesel generator backed power source and 2B005 &

2B015 were replaced in the U2C16 outage.

A new Class 1 E charger 3B025 will be added to feed bus 3D5 to provide power from a diesel generator backed power source and 36005 & 36015 will be replaced in the U3C16 outage.

For details refer to SPL #5, 7 & 9.

, N 169 e \

Retrieved from "https://kanterella.com/w/index.php?title=ML11243A059&oldid=2547632"