05000318/LER-2003-003

LER-2003-003,

Calvert Cliffs Nuclear Power Plant
Event date:
Report date:
Initial Reporting
3182003003R00 - NRC Website

I. DESCRIPTION OF EVENT

On May 28, 2003, at 11:34 AM during planned troubleshooting on the Unit 2 Main Turbine Automatic Electro-Hydraulic Controllers in Turbine Auxiliaries Electro-Hydraulic Cabinet 2T11, a short circuit created from the misuse of test equipment resulted in an automatic reactor trip. The short circuit induced the loss of voltage to the valve position limiter (VPL) and caused the Main Turbine governor valves to shut unexpectedly. This resulted in a Reactor Protective System (RPS) initiated automatic reactor trip due to the high pressurizer pressure condition from the rapid loss of load.

The troubleshooting effort in cabinet 2T11 was being performed to determine the cause of Governor Valve No. 1 oscillations first discovered during power escalation at approximately 85 percent following the recent Unit 2 Refueling Outage. The oscillations on the Governor Valve No. 1 were found to be present when the Main Turbine Controls operating mode was in Operator Automatic, but the valve oscillations cleared when the operating mode was changed to Operator Manual. Seven previous troubleshooting efforts and one Rover Maintenance action (replacement of a mixing amplifier card) had been performed prior to the eighth and final troubleshooting effort on May 28, 2003.

During the final troubleshooting effort, noise was determined to be present on the VPL card on Pin 11. A change to the troubleshooting plan was processed to further trace the source of the noise. The final checks led to the digital-to-analog converter card. However, during the process of moving the test leads, while connected to the 10 Volt reference signal on this card, an automatic reactor trip occurred. At this point, the troubleshooting effort was suspended and all personnel exited the area. A Root Cause Analysis (RCA) team was immediately assembled to determine the cause of the automatic reactor trip and a Significant Incident Finding Team was assembled to manage the investigation. The team determined the underlying cause of the trip and Unit 2 was restarted May 29, 2003. This event is only applicable to Unit 2 because the troubleshooting activities were specific to Unit 2.

II. CAUSE OF EVENT

A review of the Sequence of Events printouts indicated that a turbine overspeed alarm was the first alarm to come in. The alarm reset in 0.047 seconds. This alarm setpoint comes in at 110 percent turbine operating speed. However, based on review of the turbine supervisory data log, the turbine was rolling at normal speed with no excursions and there was no actual turbine overspeed. Therefore, the only way this alarm could come in would have to be from the loss of a reference power supply in 2T11. The RCA team developed a troubleshooting plan to replicate the activities that were being performed in cabinet 2T11 just prior to the reactor trip. The RCA team's troubleshooting effort determined that the trip was caused when a short circuit was introduced while checking a digital-to-analog converter card. The short circuit was created due to the Bayonet Neill Concelman (BNC) connector shorting during insertion into the digital multi- meter (DMM). This grounded the power supply, effectively zeroing the VPL output. With the VPL signal at 0, the control signal to the governor valves was then limited to 0 VDC, which caused the governor valves to shut. As the governor valves closed, reactor pressure rose to the RPS trip setpoints. The RCA team's troubleshooting efforts confirmed that the sequence of events was identical to those that caused the actual trip.

Throughout the first seven troubleshooting plans, the testing progressed in a methodical manner using either the front 2T11 panel installed test connection and test selector switch or taking readings from the back circuit card pins of selected automatic control circuit cards. During these initial seven troubleshooting plans, the technicians followed routinely performed work practices with the test leads in a de-energized state whenever they were manipulated. The turbine controls were in manual through this entire troubleshooting process.

However, during the eighth and final troubleshooting plan on May 28th, the plan called for recording the DC voltage and millivolt AC component, and observing the AC component using an oscilloscope. The oscilloscope was an added test device since previous attempts to select an appropriately sized filter failed to eliminate the noise.

The change/off-normal condition allowed the swapping of the BNC cable from the DMM to oscilloscope and vice versa while the other end of the cable was connected to 2T11. This condition resulted in a energized test lead manipulation. The plan included a total of 44 test points from which to collect data. While collecting data at the last test point (-10 VDC reference power supply), the BNC cable center pin touched the banana-to-BNC adapter plug shield. The intermittent short circuit removed —10 VDC reference power to the circuit and caused the VPL to fail to 0 volts, thus driving the governor valves shut. This improper testing technique lead to the event due to the manipulation of energized test leads.

During the eighth and final troubleshooting plan on May 28th, the team identified additional data points needed to effectively trace the noise problem. Therefore, the troubleshooting plan required a change. The troubleshooting plan procedure requires review of all changes to an approved plan by appropriate plant staff before rework begins. However, the procedure does not clarify whom the appropriate plant staff members are in the review and approval process for changes.

The change was approved with the understanding that "high impedance" type test equipment would be used to minimize the effects on existing circuits. In addition, it was not communicated that some of the additional points crossed into the "Manual" circuit, thus significantly increasing plant trip risk. The supervisor reviewed the change and schematic drawings, but did not recognize the intrusion into the "Manual" circuit, and thus did not challenge the scope change.

The root cause of this event is human performance in that the technicians manipulated the test equipment leads in a manner that allowed energized leads to be plugged into the DMM. Although the technicians applied several event free tools learned from both industry and site events, they did not recognize they were injecting a new failure mechanism when plugging an energized BNC cable into the DMM.

A contributing cause was determined to be inadequate supervisory oversight, and a lack of clear communication and questioning attitude between team members when the troubleshooting scope was changed to include circuits that could affect the Manual control.

Additionally, a contributing factor was that the troubleshooting procedure is not clear on what approvals are required when scope changes occur during troubleshooting activities.

Another contributing factor was that the integrated risk management procedure allowed the troubleshooting of a plant trip potential evolution to be screened as Nuclear Medium Risk. The responsible maintenance group supervisor correctly completed the required determination of risk, however, a High Risk activity is determined only if peer checks are not practical. Peer checks were practical in this troubleshooting effort, therefore, the high-risk management tools, including a Management Oversight Board and contingency plans, were not an available barrier to prevent the subject event.

III. � ANALYSIS OF EVENT The Unit 2 main turbine governor valves shut unexpectedly leading to an automatic reactor trip due to high pressurizer pressure. All other parameters were normal for the trip and all alarms that were received during the transient were expected. There were no actual nuclear safety consequences incurred from this event.

This event resulted in automatic actuation of the RPS and is, therefore, reportable in accordance with 10 CFR 50.73 (a)(2)(iv)(A). Immediate notification of this event (Event Number 39885) was made on May 28, 2003 in accordance with 10 CFR 50.72 (b)(2)(iv)(B).

No actual nuclear safety consequences were incurred from this event; however, plant equipment malfunctions that occurred following the trip were as follows:

1. All control element assemblies (CEAs) inserted fully (as indicated by all amber indicating lights coming on) as required, however, six CEAs did not indicate 0" on the CEA position display system. Issue reports were initiated as required.

2. The primary relief valves (RVs) did not lift as a result of the trip; however, one of the primary RVs (RV-201) did leak when the quench tank was vented. Previously issued engineering on venting the quench tank in a slow and controlled matter would have precluded the RV leaking and the associated delay in starting up the unit. However, Operations was apparently unaware of the engineering directions on venting the quench tank. Issue reports were initiated as required.

IV. CORRECTIVE ACTIONS

A. Maintenance supervision briefed the appropriate maintenance personnel on the root cause of the trip so as to preclude a similar transient.

B. Awareness training to plant personnel on the causes of the event and on the corrective actions being taken will be provided. Training shall particularly discuss the lack of communications and its importance in relaying critical information, and include the failures in the use of the troubleshooting procedure for the risk level evaluation process, the approvals required for changes made to the troubleshooting form, and the process to follow to implement the scope addition.

C. The troubleshooting procedure will be revised to clearly define the scope addition process for troubleshooting. Plant personnel will be trained on the revised procedure.

D. The Electrical and Controls Training Program will be verified to incorporate the correct usage of test leads.

E. Questions will be added to the Maintenance Annual Skills and Knowledge Assessment to reinforce the correct usage of test leads.

F. The integrated risk management procedure will be changed so that the determination for Nuclear Medium and High Risk is not solely based upon the practicality of peer checks.

G. Management's expectations for supervisory oversight in Maintenance will be reinforced, particularly concentrating on a Supervisor's role in remaining in a disengaged, hands-off fashion to encompass the `big picture.

A. Component Failures

IEEE 803 IEEE 805 Component or System EllS Funct System ID Main Turbine Governor Valve 65 V TA

B. Previous Similar Events:

A review of Calvert Cliffs' events over the past several years was performed. One similar event was identified involving human performance errors during troubleshooting that resulted in a reactor trip.

Specifically, Licensee Event Report No. 318/96-001 "Automatic Plant Trip Due to Partial Loss of Off- Site Power" documents the Unit 2 reactor trip on February 27, 1996 that occurred while personnel were troubleshooting the cause for 500 kV breakers 552-41 and 43 tripping.

The causal analysis describes a plant trip due to human performance errors. Specifically, a worker chose to deviate from a troubleshooting plan and cycled a breaker in the switchyard. This led to a partial loss of off-site power and an automatic reactor trip. The casual analysis determined that the troubleshooting team did not work together to fully assess the potential risks of cycling the breaker.

Corrective actions taken to address the 1996 event included development of a plant procedure to control switchyard maintenance. However, the troubleshooting procedure was not revised to control changes for CCNPP personnel performing all troubleshooting activities. Revising the troubleshooting procedure may have prevented the subject event.