05000341/LER-2004-004
| Docket Number12 04 2004 2004 - 004 - 00 01 25 2005 | |
| Event date: | 12-04-2004 |
|---|---|
| Report date: | 01-25-2005 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| Initial Reporting | |
| ENS 41243 | 10 CFR 50.72(b)(2)(iv)(B), RPS System Actuation |
| 3412004004R00 - NRC Website | |
Initial Plant Conditions:
Mode� 1 Reactor Power� 60 percent
Description of the Event
On December 4, 2004, at 0417 hours0.00483 days <br />0.116 hours <br />6.894841e-4 weeks <br />1.586685e-4 months <br /> EST, during startup from the tenth refueling outage, the reactor scrammed from 60% power as a result of the actuation of an Automatic Voltage Regulator (AVR) [RG] trip relay. The AVR experienced an interruption of the communications link between the digital AVR and the power converter thyristors of the generator exciter [TL]. The AVR trip resulted in a main generator [TB] trip and subsequent main turbine [TA] trip. A reactor scram occurred as designed from the turbine control valve fast closure signal. The reactor protection system (RPS) [JD] performed as expected, and all rods were fully inserted into the core.
Reactor water level reached a low of 136 inches above top of active fuel and recovered to normal automatically without operator intervention. Subsequent to the event, the main steam isolation valves (MSIVs) remained open and reactor water level was maintained in the normal band of 173 to 214 inches. Reactor water was supplied by the condensate [SD] and reactor feedwater systems [SJ], and the resultant reactor steam was sent to the condenser [SG] via the turbine bypass lines. Pressure control was maintained by the turbine bypass valves. Reactor dome pressure peaked at about 1010 psig. With reactor pressure maintained below the Safety Relief Valve (SRV) setpoints, none of the SRVs lifted. Reactor water Level 3 isolations [JM] occurred as expected. These included isolation Group 4 (Residual Heat Removal Shutdown Cooling and Head Spray), Group 13 (Drywell Sumps), and Group 15 (Traversing In-core Probe System) isolations.
A 4-hour notification of this event was made to the NRC in accordance with 10 CFR 50.72(b)(2)(iv)(B) at 0507 hours0.00587 days <br />0.141 hours <br />8.382936e-4 weeks <br />1.929135e-4 months <br /> EST on December 4, 2004 (EN 41243).
The AVR is part of the main generator excitation system that maintains generator output voltage under varying conditions of load within set tolerances. This is achieved by using a closed loop static excitation system consisting of a 5200 KVA excitation transformer, a rectifier cubicle (power converter thyristor unit), an excitation control cubicle (AVR unit), and a field suppression cubicle. The static excitation system controls the main generator output voltage and reactive power flow by direct variation of the generator excitation current in
- response to feedback signals from the generator output terminals. Excitation power is obtained fronfthe generator output and fed through the excitation transformer to the rectifier cubicle. The rectifier cubicle supplies excitation to the main generator field through the direct current (DC) field breaker in the field suppression cubicle.
The controls for the static excitation system consist of two electronic processing units located in the excitation control cubicle and microprocessor based thyristor triggering units, located in each of three forced air cooled thyristor converter bridges (rectifier cubicles). Each processing unit contains the software for generator output voltage control (AVR), generator field current (manual voltage regulator), excitation system monitoring and protective functions, and a programmable logic controller, providing a dual channel design for complete processing redundancy. An integral part of the AVR configuration is a local area network (LAN), which communicates between the display panel and modules in both voltage regulator racks, the rectifier converter electronic control modules and the Field Breaker/Suppression Circuits. The LAN network is configured with coaxial cabling connections between each interface/termination point and operates using an `ARCnet' signal token passing communication system. The excitation system can provide full rated excitation with only two of the three converter thyristor bridges in service. This allows for continuous service if one bridge fails during normal operation.
Under normal conditions, the excitation control starts up with one channel selected in AVR mode. If a failure occurs in the operating channel, a transfer to the remaining AVR channel occurs automatically. If the second AVR channel fails, a transfer to the manual regulator should occur automatically.
An investigation of the unit trip was initiated immediately after the event. Extensive troubleshooting was performed. The vendor representative was included in these activities, and an engineering consultant performed an independent evaluation of electromagnetic interference issues of the ARCnet communications network that initiated the AVR trip.
A temporary modification had been installed in a module of each thyristor converter unit in order to correct a condition that was discovered during troubleshooting associated with the September 3, 2004 AVR scram. Three thyristor converter units were replaced during the November 2004 refueling outage to eliminate the temporary modification. This involved the replacement of several sub-component boards in each converter unit, including each of three ARCnet coupler communication boards of a later design than the original ARCnet coupler communication boards. It was thought that the newer boards would enhance the robustness of the ARCnet communication system, the failure of which was the most probable cause of the September 3, 2004 scram. The three new ARCnet coupler communication circuit boards were expected to provide improved interrupt handling and superior immunity to ARCnet communication errors. However, about 4-1/2 hours after synchronization of the generator during plant restart, an AVR ARCnet communication related alarm was received, shortly followed by a blocking and isolation of one of the three thyristor converter units. A subsequent failure in a second converter unit resulted in an AVR trip which resulted in a main generator trip and subsequent main turbine trip and reactor scram.
The plant was restarted on December 5;2004. Required circuit board replacements, the addition of noise suppression devices, AVR software changes, and associated testing were completed, and the unit was synchronized on December 8, 2004.
Cause of the Event
The apparent cause was an incompatibility of new ARCnet communication circuit boards with the original system design. A temporary modification had been installed in a module of each thyristor converter unit in order to
- correct a condition that was discovered during troubleshooting associated with the September 3, 2004 AVR scram. Three thyristor converter units were replaced during the November 2004 refueling outage to eliminate that temporary modification. All three thyristor converter units were replaced at that time, including each of three ARCnet coupler communication boards in those units. Those boards were of a later design than those installed in the previous thyristor converter unit, and it was thought that the newer boards would enhance the robustness of the ARCnet communication system. However, the configuration of the system was not formally controlled in accordance with the site policies and procedures, and a formal equivalency evaluation was not performed on the thyristor converter replacement. The three new ARCnet coupler communication boards were determined to be 'incompatible with the original system design. Ostensibly,.the three new ARCnet coupler communication circuit boards were designed to provide improved interrupt handling and superior immunity to ARCnet communication errors. Nevertheless, communication problems did occur which resulted in the blocking and isolation of one of the three thyristor converter units. A subsequent failure in a second converter unit resulted in an AVR trip, as designed. The AVR trip resulted in a main generator trip with subsequent main turbine trip and reactor scram.
Analysis of the Event
The AVR has no safety related function. The AVR, Generator, and Turbine trips functioned as designed. The reactor scrammed as designed from the turbine control valve fast closure signal. The plant response to the turbine trip was as expected and was enveloped by the more severe turbine trip without bypass transient described in the UFSAR. There was no challenge to the integrity of the reactor coolant system or the main steam system. The lowest reactor water level during the transient was measured to be 136 inches above top of active fuel which is below the reactor water Level 3 isolation trip setpoint. Reactor water Level 3 isolations occurred as expected.
These included isolation Group 4 (Residual Heat Removal Shutdown Cooling and Head Spray), Group 13 (Drywell Sumps), and Group 15 (Traversing In-core Probe System) isolations. The highest reactor pressure received was about 1010 psig which is below the safety relief valve setpoints; 5 each at 1135, 1145, and 1155 psig. Subsequent to the unit trip reactor pressure was adequately controlled using the main turbine bypass valves, and reactor water level was controlled using the condensate and feedwater systems.
Therefore, since the generator, turbine and reactor protection systems performed as designed, and since plant response was enveloped by the UFSAR transient analyses, there was no undue risk to the health and safety of the public as a result of this event.
Corrective Actions
ARCnet coupler communication (0672) circuit boards were replaced with the original circuit boards.
Ferrite beads were added to the ARCnet coaxial signal cables to filter out high frequency noise and reduce susceptibility to internally and externally generated electronic noise. This modification reduced measured noise levels on the shield of the ARCnet communication system by about 50%.
Software changes were made to incorporate a 2-second time delay to eliminate unnecessary channel changeover or excitation trips due to communication interruption. A single ARCnet communication alarm was replaced by six separate alarms to better identify the origin of a communication problem.
This event has been documented in the Fermi 2 corrective action program, CARD 04-26443. The compatibility of all of the interconnected AVR systems subcomponents is being evaluated under this CARD. Fermi 2 programs are also being evaluated to determine whether changes should be made that could preclude a problem of this type in the future. Any further corrective actions identified as a result of these evaluations will be tracked and implemented commensurate with the established processes and priorities of the corrective action program.
Additional Information
A. Failed Components:
Component: ARCnet coupler communication boards Function: ARCnet communication system Manufacturer: ABB Brown Boveri Model Number: 0672 Failure Cause: Incompatible replacement ARCnet coupler communication board B. Previous LERs on Similar Problems:
failure or failures of the original Channel 1 processor (Type PP C322) module and/or the combination input/output data module (Type UA C326) located in the plant relay room. The current event was associated with the replacement of the three SCR Controller units located in the vicinity of the SCRs, or more specifically, the apparent incompatibility in replacement ARCnet Coupler communication boards (Type 0672) within those units. Although both of these events involve the ARCnet communication subsystem of the AVR, the current event was not a repeat of the earlier event since the current event was caused by incompatible replacement parts and involved different components than those involved in the original failure.