ML072060369

Follow-up Summary of Close-Out of July 4, 2007 Unusual Event Classification

ML072060369
Person / Time
Site:	Ginna
Issue date:	07/05/2007
From:	John Carlin Constellation Energy Group, Ginna
To:	Document Control Desk, Office of Nuclear Reactor Regulation
References
CR-2007-004819
Download: ML072060369 (7)

Text

John Carlin R.E. Ginna Nuclear Power Plant, LLC Site Vice President 1503 Lake Road Ontario, New York 14519-9364 585.771.5200 585.771.3943 Fax John.Carlin @constellation.com 0Constellation Energy Generation Group July 5, 2007 U. S. Nuclear Regulatory Commission Washington, DC 20555 ATTENTION: Document Control Desk

SUBJECT:

R.E. Ginna Nuclear Power Plant Docket No. 50-244 Follow-up Summary of Close-Out of July 4, 2007 Unusual Event Classification In accordance with NUREG 0654, FEMA-REP-1, Rev. 1, enclosed as attachment 1 is a summary of close-out conditions for the July 4, 2007 Unusual Event classification. This written summary report is required within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of close-out of the Unusual Event.

As verbally requested by NRC staff, enclosed as attachment 2 is the initial technical assessment of the loss of annunciator event as prepared by the issue response. team that performed the troubleshooting and repairs. THis issue has been entered into the corrective action program as CR-2007-004819. An equipment root cause analysis will be performed to fully understand the event, including extent of cause and extent of condition.

If you should have any questions regarding this submittal, please contact Robert Randall at (585) 771-5219.

Attachment 1: July 4, 2007 Unusual Event Close-out Summary Attachment 2: Initial Technical Assessment of Loss of Annunciator Event cc: S. J. Collins, NRC D. V. Pickett, NRC Resident Inspector, NRC P.D. Eddy, NYSDPS A.00J

/ o0/9/9

ATTACHMENT (1)

July 4, 2007 Unusual Event Close-out Summary

ATTACHMENT 1 July 4, 2007 Unusual Event Close-out Summary On July 4, 2007 at 0158 hours0.00183 days <br />0.0439 hours <br />2.612434e-4 weeks <br />6.0119e-5 months <br /> EDT, with the plant at approximately 98.5% power, a loss of the control room annunciator panels A, B, C, and D occurred. At that time the control room had just commenced a load reduction to repair a turbine control valve.

The load reduction was terminated at the time of the annunciator panel failures. As a result of the loss of the annunciator panels an unusual event was declared at 0214 in accordance with plant procedure EPIP 1-0, "Ginna Station Event Evaluation and Classification". This was declared under Emergency Action Level (EAL) criteria 7.3.1 (Unplanned loss of annunciators or indications on any of the following Control Room Panels for greater than 15 minutes (A, AA, B, C, D, E, F, G) AND increased surveillance is required for safe plant operation). At approximately 0220 hours0.00255 days <br />0.0611 hours <br />3.637566e-4 weeks <br />8.371e-5 months <br /> EDT offsite notifications were made in accordance with plant procedure EPIP 1-5, "Notifications".

The Technical Support Center (TSC) was activated in accordance with Ginna procedures to provide support for the increased surveillance, repair and recovery efforts.

At approximately 1543 hours0.0179 days <br />0.429 hours <br />0.00255 weeks <br />5.871115e-4 months <br /> EDT on July 4, 2007 the control room annunciator panels were repaired and tested and the unusual event was terminated. The cause was determined to be a failure of a card in one annunciator rack that caused a protective shutdown of the power supplies that are paralleled together, with consequential failures of two other cards.

This issue has been entered into the corrective action program as CR-2007-004819.

An equipment root cause analysis will be performed to fully understand the event, including extent of cause and extent of condition.

ATTACHMENT (2)

Initial Technical Assessment of Loss of Annunciator Event

Attachment 2 Initial Technical Assessment of Loss of Annunciator Event July 4, 2007 This document records the initial investigation relative to the sequence of events as understood during Issue Response Team assessment, response, and evaluation of equipment conditions during initial event assessment and recovery. A root cause assessment will be performed to ensure all data is collected and reviewed to develop a final report of the event.

Symptoms:

1. AA-1 7 alarmed and horn quit before being acknowledged

2. AA panel lights on and flash, but no horn

3. No lights or horn on A, B, C or D panels

4. Center & Right MCB panels maintained all capability, lights & horn sounds Critical Configuration Parameters Initially Investigated:

1. The A, B, C, D and AA annunciators share the same horn initiate relay.

2. The A, B, C and D share the same 125Vdc power feed separate from the AA annunciator.

3. The A, B, C and D annunciators share the same flasher circuit card.

4. The AA annunciator has a separate flasher card.

5. The main control board center and right section annunciators were fully functional throughout the event, and have their own flasher cards (1 for the center and 1 for the right)

Trouble shooting:

1. Voltage measured 133Vdc at supply breaker (XTMA) feed to annunciator panels A, B, C and D. This verified appropriate power supply to the overall annunciator system.

2. Voltage measured 133Vdc at each annunciator chassis power supply input in A, B, C and D

3. LED's on A, B, C and D power supplies were not lit. The LED's are indication of power out from the power supplies. The power supplies step down 125VDC to -

28VDC, +12VDC, -12VDC, and 125 VDC. (28V-drive annunciator lights, 12V drive relay coils and logic, 125VDC supplies inputs)

4. Temporarily opened slide links TVN-8 and TVN-1 1 to isolate alarm acknowledge, first out and test circuitry from the annunciator system. (No change in symptoms).

5. Requested Operations to initiate annunciator test to visually verify alarm test, alarm first out and alarm acknowledge relays not failed. This resulted in no change to the system, A-D still did not have power or annunciate. Restored slide links.

6. Cycled breaker (XTMA) 125Vdc to A, B, C and D power supplies. The LEDs on power supplies temporarily lit and extinguished. The A, B, C and D annunciator lights that were previously locked in came in temporarily lit and then extinguished.

7. Opened DC breaker (XTMA) then pulled A, B, C, and D annunciator common flasher card then closed DC breaker (XTMA). The flasher card is common to these four banks, so was a suspected common cause point. This resulted, in input cards D6 (Steam Generator Lo-Lo Level Loop B) and D7 (Reactor Coolant Loop A&B Low Flow) failing (smoke was produced as overheated components failed). Both D6 and D7 input cards are in close physical proximity to one

another. The DC breaker was opened to remove power to the D6 and D7 cards, removed these cards.

8. Drawings were reviewed and additional voltage. and ground readings were taken to ensure that any other common cause failure modes had been removed prior to proceeding with troubleshooting to minimize the potential for additional component failures.

9. Closed DC breaker (XTMA) to restore power to A, B, C and D annunciators with the D6, D7 input cards and flasher card removed. The A, B, C and D annunciator lights restored. I

10. Operations simulated an annunciator actuation and extinguish signal. Operators observed the annunciator light lit and extinguish as expected. However, no audible or flash due to the flasher card still being removed.

11. Bench-tested old flasher card and new flasher card (both tested SAT). Flasher card for A, B, C and D annunciators replaced, and flasher card function verified.

Testing was done to simulate alarm signals to verify alarms would come in, flash, audible alarm, and reset actions worked as designed.

12. Visually the D6 card input resistors could be seen to have overheated internally resulting in overheating the adjacent D7 card.

13. Measured resistance across input resistors R36 and R2 of the D6 card. R36 resistor measured 40ohms (rated 56K). R1 resistor measured open (rated 6.8K).

This verified D6 card as the cause of the system failure.

14. Replaced D6 and D7 card with new cards. As each card was installed, functionality of all portions of the AA through D panels was verified.

15. Testing of the "AA" panel indicated that the horn driver was still not functioning.

Investigation found a blown fuse on the flasher card for the "AXA panel. The flasher circuit shares the "common" with the A-D power supplies.

Fault mechanism:

This event was initiated by an alarm in the "AA" panel. Window AA-1 7 lit and flashed as expected. However, 'the horn started but quit before the alarm was acknowledged by the control room operators.

Cards D6 (Steam Generator Lo-Lo Level Loop B) and D7 (Reactor Coolant Loop A&B Low Flow) were identified as having failed, as evidenced by smoke and burn marks on the boards. D6 failed and overheated, which caused D7 to fail.

An existing, but.previously undetected degradation of the D6 annunciator input card voltage divider resistor network R36/R2 overburdened the parallel power supplies for panels "A" through "D" resulting in the oscillator circuits in each chassis' power supply shutting down and causing loss of all functions of the "A" through "D" panels (see power supply discussion below). When AA-17 alarmed, the common flasher card circuit coupled with the loss of the "A" - "D" annunciator power supplies overburdened the "AA" flasher card, blowing the fuse.

The "AA" annunciator panel drives the flasher card in parallel with the flasher card in the "A" annunciator panel. The flasher card in the "A" panel performs the flasher function for the "A", "B", "C" and "D" panels on the left main control board. The "E" through "L" panels have 2 additional flasher cards and are located on the center and right MCB and were not impacted during this event. One function of the flasher card is to drive the horn in the control room any time a new alarm is received at the annunciator. A blown fuse on the "AA" flasher card resulted in the horn not functioning when any of the "AA"

alarms came in. The "AA" flasher card failure only impacted the "AA" audible function, but maintained ability for windows to light and flash.

Power Supply Discussion:

In accordance with the RiS vendor manual, the power supplies take 125VDC input and transistors Q1 and Q2 oscillate on and off to switch the DC input to an AC voltage which is fed through a transformer and then rectified back to DC voltage. When the power supply sees an over current condition, transistors Q1 and Q2 stop oscillating. With no switched AC, the transformer does not couple any voltage and power supply output, goes to zero. With the power supplies shut down the power supply LED's extinguish.

The initiating event described above resulted in such an over current seen by all of the paralleled power supplies in the A-D chassis, so all of those power supplies shut down.

Input Card D-6 Failure Discussion:

During troubleshooting, the power was cycled to the annunciators, per the annunciator maintenance procedure M-94. Upon cycling the 125VDC breaker to the A, B, C and D annunciators, the annunciators initially operated until the oscillator circuit shut down the power supplies again.

After initially cycling the power, the flasher card was removed from the "A" panel.

Power was restored and input card D-6 failed. Upon returning power to the D6 card, with R36 degraded resistance, R2 opened due to over current. When R36/R2 failed, nearby components on the circuit board and the adjacent circuit board (D-7) overheated.

The cause of the failed resistors on the D-6 card is attributed to heating and aging. The D-6 point being monitored is a normally closed contact so current has been drawn through the circuit near-continuously for almost 40 years. A carbon resistor will change value as it ages due to moisture absorption and operating temperature. The change in resistance produces more heat over time. The board itself behind R36 had degradation due to the heating which is evidence of long term overheating of the resistor. Over time, R36 continued to degrade, I

causing eventual thermal runaway, overheating and failure.

Functional Restoration:

The input cards D-6 and D-7 were replaced. The "A" flasher module was replaced. The "AA" flasher card was replaced.

Following restoration, power supply outputs were verified to be maintaining 133 VDC, with an AC ripple addition of less than 0.4 volts AC, which is the expected value.

Functional testing of the annunciators was accomplished by simulating an alarm on an input signal. Also, all annunciators were tested using the test and acknowledge pushbuttons.

The vendor of the cards has been engaged and will be performing additional testing of the failed boards to rule out other potential causes of the failures. The root cause investigation that will follow will include an extent of condition review and corrective actions to evaluate further component degradation in the annunciator system.