05000317/LER-2004-001
| Calvert Cliffs Nuclear Power Plant, Unit 1 | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| Initial Reporting | |
| ENS 40601 | 10 CFR 50.72(b)(2)(iv)(B), RPS System Actuation |
| 3172004001R00 - NRC Website | |
I. DESCRIPTION OF EVENT
Calvert Cliffs Unit 1 tripped from 100 percent power on March 20, 2004 after Bus 1Y09 was grounded during scheduled maintenance. Specifically, while replacing the 500 kV Bus Voltage Chart Recorder (1ER-101), in Control Room Cabinet 1C29, a power lead feeding the recorder became pinched between the recorder and its case. As installation continued, the insulation on one of the power leads was cut, exposing the live power lead, and a short to case was experienced. This led to a loss of 11 Digital Feedwater Automatic Bus Transfer bus, which in turn resulted in the deenergization of the 11 Main and 12 Backup Central Processing Units for the digital feedwater system. Per digital feedwater system design, the 11 Feedwater Regulating Valve (FRV) positioner selector solenoid, 1-SV-1111B, electrically shifted to the "A" position.
However due to mechanical binding of the solenoid valve, the positioner remained stuck in the "B" position. This resulted in a loss of signal to the 11 FRV, which immediately began closing. The valve shut, and 11 Steam Generator water level lowered quickly, resulting in the automatic reactor trip. As a result of 11 FRV closure, both 11 and 12 Steam Generator Feed Water Pumps tripped on high discharge pressure resulting in a loss of normal heat removal. As steam generator levels lowered to the Auxiliary Feedwater Actuation System setpoint, the system actuated and feed to the steam generators was re-initiated.
Immediately after the trip, the turbine bypass valves (TBVs) opened as a result of the quick-open signal. They shut when the quick-open signal was clear. However, they did not reopen to modulate TBV flow. The TBV controller was placed in Manual but the valves still remained closed. Since the TBVs were not operating, the atmospheric dump valve (ADV) controller was placed in Manual and modulated to control the cool down.
Unit 1 was restarted and paralleled to the grid on March 22, 2004.
II. CAUSE OF EVENT
The initiating fault which led to the plant trip occurred as a short circuit from the line terminal of the 500 kV Bus Voltage Recorder at panel 1C29. This component is powered by the 1Y09 Circuit 68 which is supplied by phase C of the 1Y09 bus (120 VAC ungrounded 3-phase bus). This fault caused a voltage shift in reference to ground potential of the A and B phases of the 1Y09 bus.
During the time that the ground fault was present, these phases experienced a 208 VAC potential as referenced to plant ground. This voltage exceeded the design ratings for a metal oxide varistor (varistor) which was installed in the FRV Dixson (1ZI-1111A) indicator causing it to fail approximately 20 minutes later. The Dixson indicator is powered by the Digital Feedwater Automatic Bus Transfer bus which is normally fed from the B phase of the 1Y09 bus. Indications suggest that the varistor failure manifested itself as a short circuit between the line and the case ground of the indicator. Observations made by operators at the time of this failure suggested that the short circuit was intermittent at first and that extreme heat (electrical fault odor) was being developed within the Dixson indicator. The varistors have been removed from the Dixson indicators.
The FRV positioner selector solenoid valve (1-SV-1111 B) failed to re-position to the de-energized state when alternating current (AC) power was removed from the coil. During the post-trip investigation, jarring of this component resulted in its re-positioning. The solenoid valve was replaced with an equivalent model during the forced outage. The failed solenoid valve was sent to an off site organization for a detailed failure analysis. The analysis determined that the lubricant used in the solenoid valve assembly dried out in service causing an adhesive bond which caused the solenoid valve to stick. The lubricant should not have been in the area where it was discovered. Most likely, the lubricant was inadvertently introduced in the relatively tight clearance area between the solenoid core and the solenoid guide tube during assembly at the manufacturer's facility. This component failure is unrelated to the 1Y09 bus transients but contributed to the trip.
Background:
Dual positioners were added to the digital feedwater system in the 2002 Unit 1 Refueling Outage (RFO) and the 2003 Unit 2 RFO. Intended to reduce the probability of a single failure tripping a unit, this modification also installed Dixson position indicators for the Unit 1 and Unit 2 FRV and FRV bypass valves. While preparing the Design Input Requirements Package for the dual positioner modification, the design engineer contacted the vendor to determine compatibility between the Dixson indicators and Calvert Cliffs Nuclear Power Plant's (CCNPP's) ungrounded electrical system. However, it was not specified that CCNPP uses a 3-phase ungrounded system and the potential for creating up to 208 VAC to ground was not conveyed to the vendor. The vendor assumed CCNPP used a single-phase ungrounded AC system.
The position indicators were delivered to CCNPP as ordered. However, a vendor design modification had installed varistors on the power supply input cards to provide AC input power supply surge protection. The modified Dixson indicators were acceptable for use in a single- phase ungrounded AC system but were not acceptable for use in a 3-phase ungrounded system.
When the position indicators were installed and energized during the 2003 Unit 2 RFO, the varistors failed. Line-to-ground voltage was approximately 160 VAC due to the pre-existing ground resistance imbalance on other phases. The varistors in the Dixson position indicators were not rated for this voltage. Subsequently, the varistors were removed from the Dixson position indicator circuit cards for Unit 2.
After discovering the varistors during the Unit 2 testing the following actions were taken:
1. � Specific directions were included in the design input requirements for activities adding or changing instruments on Y05/09/10 buses. The following statement was included to identify the additional requirement for ungrounded 3-phase systems; "Equipment must be suitable for operation from an ungrounded power source. If surge suppression is supplied, it must be rated for continuous operation at or above 208 Volts from line to ground.
2. Training was provided to all Instrumentation and Control design engineers. This training reviewed the basic principles of ungrounded 3-phase electrical systems, and steps to be taken by design engineers to ensure future modifications evaluate the ground-faulted power supply requirements.
3. In 2003, all modifications were reviewed to determine the extent of this varistor issue. One other modification which had not been issued for installation was identified that added the same Dixson indicators. This modification was reviewed to ensure the varistor issue was addressed.
4. A modification was proposed to install isolation transformers on the Feedwater Control and the Radiation Monitoring Systems. The modification was not installed based on the subsequent decision to install a temporary modification to re-power the Dixson indicators with a single-phase source. However, the temporary modification was not installed based on the unlikely probability of a ground on 1Y09 or 1Y10 and because the most likely failure scenario would result in the digital feedwater system shifting to manual control. It was concluded that the risk involved in the installation of a modification outweighed the anticipated increase in digital feedwater system reliability.
A memo describing the potential failure and effects was written by the system engineer following the decision to not install either modification. The groups involved understood that work affecting 1Y09 or 1Y10 would be controlled by Operations. The communications between Plant Engineering, Operations, and Work Management on compensatory measures to manage work that could affect 1Y09 or 1Y10 were verbal and ineffective.
Knowledge of any work restrictions on these buses was inconsistent among Maintenance supervisors.
On March 20, 2004, the recorder motor replacement job was informally briefed by the technicians. Interviews with the technicians indicated that the brief was not as thorough as it should have been and roles and responsibilities were not assigned. It was not recognized that this work was to be conducted in a trip sensitive area. Plant procedures define the Control Room as a trip sensitive area, however the Control Room has never been posted as such. Plant procedures require that all work in trip sensitive areas be briefed by a supervisor and that participants are to be made aware of the trip hazard.
The technicians then proceeded to the Control Room to replace the chart drive motor. The recorder was removed and the chart drive motor was replaced. Upon reinstallation, the technicians were not able to get the bottom screws to install. This was due to pinching the power connector and wires between the cabinet and the recorder. The technicians were not properly using self-checking or peer checks due to not having clearly defined roles assigned from the pre- job brief. The recorder was moved slightly and the technicians were able to install the screws.
Upon tightening the last screw, they heard a loud pop and saw a flash. The technicians realized that the recorder power leads had been damaged and were grounded. They immediately informed Control Room personnel and, after researching the prints, were directed by the Control Room to open the recorder feeder breaker at 1Y09. At about this time the reactor tripped automatically. The breaker was then opened, removing the ground from 1Y09.
Root causes were identified as follows:
1. Maintenance activities that affected the ungrounded 3-phase 120 VAC system, with known deficient conditions in the plant were not adequately managed.
2. Maintenance work practices, including pre job briefs, and supervisory oversight, did not meet management expectations for work inside of trip sensitive areas.
3. The FRV positioner selector solenoid valve (1-SV-1111B) failed to re-position to the de- energized state when AC power was removed from the coil.
Contributing Causes were identified as follows:
1. During the procurement process for the FRV dual-positioner modification, not all critical design characteristics were provided to the vendor.
2. The communications between Plant Engineering, Operations, and Work Management regarding compensatory measures to manage work that could affect 1Y09 or 1Y10 were verbal and ineffective.
3. The current Maintenance Planning process does not preclude working Rover or Reference Maintenance Orders that have an engineering issue, parts issue etc. ("assist") open.
III. ANALYSIS OF EVENT
The 11 FRV shut leading to an automatic reactor trip due to low steam generator level. All other parameters were normal for the trip and all alarms that were received during the transient were expected. There were no actual nuclear safety consequences incurred from this event.
Combined Core Damage Probability was calculated as 5.6E-6.
This event resulted in an automatic actuation of the Reactor Protective System and is, therefore, reportable in accordance with 10 CFR 50.73(a)(2)(iv)(A). Immediate notification of this event (Event Number 40601) was made on March 20, 2004 in accordance with 10 CFR 50.72(b)(2)(iv)(B).
IV. CORRECTIVE ACTIONS
A. The varistors connected to ground have been removed from the Dixson indicators so they are no longer a potential problem for this system. There are no other varistors installed in the same configuration as that of the Dixson indicators.
B. The failed solenoid valve (1SV-1111B) was replaced with an equivalent solenoid valve from a different manufacturer. A detailed failure analysis was completed. The analysis determined that the lubricant used in the solenoid valve assembly dried out in service causing an adhesive bond which caused the solenoid valve to stick. The lubricant should not have been in the area where it was discovered. Most likely, the lubricant was inadvertently introduced in the relatively tight clearance area between the solenoid core and the solenoid guide tube during assembly at the manufacturer's facility. This component failure is unrelated to the 1Y09 bus transients but contributed to the trip.
C. � Establish a process to manage short- and long-term plant conditions or deficiencies that may cause increased risks to the operating unit during maintenance, and include it in a procedure. Ensure the process includes communication among all site business units involved.
D. � Identify processes that place conditions on conducting plant work but that do not get incorporated into the Work Management Process and develop a method for including them into Work Management Processes.
E. � Formalize the minimum pre job briefing requirements for any maintenance activity to include:
a. Assignment of roles and responsibilities (lead and peer tech).
b. Discussion of the error-prevention tools to be used including who will be using them.
F. � Post signage that designates the Control Room as a trip sensitive area, subject to the work controls of the procedure.
G. � Develop a procedure for on-line testing of the FRV positioner selector solenoid valves.
H. � Develop monitoring plans for trip sensitive components of the feedwater controls, steam dump and bypass controls, and decay heat removal functions. Monitoring plans should be sensitive to latent type failures. Monitoring plan will include trip critical functions, supporting components, failure modes, and methods of monitoring for identified failure modes.
I. � Add an administrative barrier to the procurement process by revising the "Long Description" block to add a statement that achieves the following; "This item is powered from a 3-phase ungrounded AC Electrical System, and as such may be subjected to continuous 208 VAC line to ground on ground faults." Add this statement to all material items for instruments fed from the affected buses as identified in Engineering Standard.
J. � Populate the Equipment Data Base "Special Indicators Screen" with a new indicator; "3-Phase Ungrounded Load.
K. Change the process for Rover and Reference Maintenance Orders to ensure they do not work until assist codes are cleared.
V. ADDITIONAL INFORMATION
A. Component Failures
IEEE 803 IEEE 805 Component EDS Function System ID Feedwater Reg Valve Position Indicator ZI JK 1ZI-1111A Feedwater Reg Valve Positioner Selector PSV JK Solenoid Valve 1SV-1111B
B. Previous Occurrences
A review of Calvert Cliffs' events over the past several years was performed. No previous occurrences were identified involving a reactor trip due to low steam generator level caused by a fault during scheduled maintenance. The following similar events were identified.
1. Licensee Event Report 318/2004-01 At 15:26 on January 23, 2004, Calvert Cliffs Unit 2 tripped from 100 percent power, initiated by the Reactor Protective System due to low steam generator water level caused by an erroneous over speed trip signal on the steam generator feed pump.
The erroneous trip signal occurred because of a degraded digital speed monitor supply voltage caused by corrosion of an inline fuse and the fuse holder.
The TBVs and ADVs opened as designed, but the quick-open signal did not clear due to a failure of a relay in the reactor regulating circuit.
The corroded fuse, fuse holder, and failed relay were replaced.
During the January 23, 2004 event, an error that occurred at an interface between two vendor designs was identified as a cause for the failed relay. The relay vendor supplied a component that was not rated for the application. However, it was determined that corrective actions were not needed to address this cause due to subsequent improvements in vendor oversight of design modifications.
The March 20, 2004 event is similar in that a component (varistor) was supplied that was not rated for the system. However, the vendor supplied components were installed in the 2002 RFO which precedes the corrective actions established to address the cause of the January 23, 2004 event.
2. � Licensee Event Report 318/2003-03 On May 28, 2003, Calvert Cliffs Unit 2 experienced an automatic reactor trip from 100 percent power. The automatic reactor trip was initiated by the Reactor Protective System due to the high pressurizer pressure condition that resulted from the rapid loss of load. The rapid loss of load occurred when the main turbine governor valves shut unexpectedly during planned troubleshooting on the main turbine controls in a Turbine Auxiliaries Electro-Hydraulic Control Cabinet. A short-circuit created during the troubleshooting induced a loss-of-voltage to the valve position limiter causing the governor valves to shut unexpectedly. The short-circuit was caused by incorrect use of test equipment during planned troubleshooting; therefore, the root cause of the trip lies in Human Performance in the area of Work Practices. Also, contributing causes in the area of verbal communications and procedure clarity were identified.
Corrective actions included awareness training on the event, its causes and recommendations, procedure changes, and also initial and continuing training on appropriate work practices when using test equipment.
Corrective actions from the May 28, 2003 event included reinforcing Management's expectations for supervisory oversight during troubleshooting activities. However, the corrective actions did not address other maintenance activities. Broadening the corrective actions from the May 28, 2003 event to address scheduled maintenance activities may have prevented the subject event of March 20, 2004.