05000440/LER-2003-003
| Docket Number | |
| Event date: | |
|---|---|
| Report date: | |
| Initial Reporting | |
| 4402003003R01 - NRC Website | |
I. � DESCRIPTION OF OCCURRENCE:
At 1206, August 21, 2003, with the reactor in MODE 1 at approximately 13 percent power, the monthly "Diesel Generator Start and Load" surveillance was being performed and the starting output voltage was found to be approximately 4534 volts. This value is outside of the Technical Specification (TS) acceptance criteria band of 3900 to 4400 volts AC.
A root cause investigation traced the initiating event of this condition to a reverse power trip of the Division 1 diesel generator [DG] and its output breaker [BKR] that occurred at 1813, August 14, 2003.
On August 14, 2003, the Perry Nuclear Power Plant was recovering from a loss of offsite power condition (Perry LER 2003-002). The Division 1, Class 1E (nuclear safety related), 4160 volt electrical bus [ED] was being powered from the Division 1 diesel generator. As part of this recovery, the diesel generator was being paralleled with the Division 1 preferred power supply, the offsite source [JX], in preparation for transferring the safety bus from the diesel generator to the offsite source per the system operating instruction (S01).
The offsite source was at a higher than normal voltage. The Division 1 diesel generator was prepared for paralleling with the offsite source. During the restoration, the Division 1 diesel generator output voltage was matched to the preferred source voltage. The syncroscope [SYN] was adjusted to rotate "slowly in the clockwise direction" as directed in the SOI, and then the preferred source breaker was closed at approximately 12 o'clock on the syncroscope. The diesel generator began to immediately lose load as the breaker was dosed, which is expected due to the diesel generator speed and voltage control reverting from the isochronous to the droop mode of operation.
At 1813, August 14, 2003, the Division 1 diesel generator and its output breaker tripped on reverse power, and the Division 1 electrical bus remained energized by the offsite source.
This diesel generator reverse power trip occurred prior to the completion of the SOI steps that would have shutdown the diesel. As such, the diesel generator shutdown step that has the operator adjust the voltage regulator to between 4100 and 4200 volts prior to shutdown was marked "not applicable' by the control room staff as it could not be performed with the diesel generator already shutdown. The diesel generator trip was reset and the diesel generator was placed in standby readiness at 1835, August 14, 2003.
At 1152, August 21, 2003, the Division 1 diesel generator was started for the performance of the °Diesel Generator Start and Load" monthly surveillance. At 1206, it was found that the starting voltage stabilized at approximately 4534 volts, which Is above the maximum allowed 4400 volts, and the diesel was declared inoperable. During the shutdown to standby readiness of the diesel generator, the step of the 801 that instructs the operator to adjust the voltage regulator [RG] between 4100 and 4200 volts was performed by the control room staff, which restored the Division 1 diesel generator to the desired voltage band. The diesel generator was stopped at 1227 and shutdown to standby readiness at 1246, per the SOI.
The system engineer recognized that there was a connection between the reverse power trip on August 14 and the starting voltage being high on August 21. Computer traces of the reverse power trip were obtained.
These traces indicated that the bus voltage was at this high level when the diesel generator tripped on reverse power on August 14, 2003. Since the Division 1 diesel generator voltage was not properly restored on August 14, 2003, the Division 1 DG voltage remained high outside of TS limits and unrecognized by the plant operators until its discovery on August 21, 2003.
DESCRIPTION OF OCCURRENCE: (Continued) hour seventeen minutes for the performance of a post-shutdown engine roll (8/16/2003). The plant was in mode 4 at the time of the post-shutdown roll and plant personnel did not know that the Division 1 diesel generator was inoperable.
TS 3.8.2, AC Sources-Shutdown, is applicable in MODE 4 and specifies that one diesel generator capable of supplying one division of the Division 1 or 2 onsite Class lE AC electrical power distribution subsystem(s) is required. If the aforementioned cannot be met, required action B.4 states to initiate action to restore a required DG to operable status with a COMPLETION TIME of "Immediately." Since it was unknown that the Division 1 diesel generator was inoperable, the Division 2 diesel generator was declared operable at the completion of the post-shutdown engine roll instead of "Immediately." The Division 3 diesel generator was operable during this time.
On 8/20/03 the plant changed MODES from 4 to 2 and 2 to 1 making TS 3.8.1, AC Sources-Operating, applicable. TS 3.8.1 requires that two qualified circuits between the offsite transmission network and the onsite Class lE AC Electrical Power Distribution system; and three diesel generators be operable when in MODES 1, 2, and 3. TS 3.8.1 does not have an exception to TS 3.0.4 to allow a MODE change with an inoperable diesel generator.
TS 3.0.4 states that when an LCO Is not met, entry into a MODE or other specified condition in the Applicability shall not be made except when the associated ACTIONS to be entered permit continued operation in the MODE or other specified condition in the Applicability for an unlimited period of time. Since Division 1 diesel generator was inoperable, the plant made the following MODE changes in violation of TS 3.0.4: from MODE 4 to MODE 2 at 0150 on 08/20/2003 at 0 percent power; and from MODE 2 to MODE 1 at 2146, on 08/20/2003 at approximately 8 percent power.
II. CAUSE OF EVENT
Three root causes were identified for this event:
The "Precautions & Limitations" (P&L) section of the diesel generator SOI does not contain the diesel generator high voltage limit value. As a result, operators reviewing the "Precautions & Limitations" prior to manipulation of the diesel generator to restore offsite power were unaware of this limit.
Expectations for the review of the operating parameters in the "System Operations" section of the SOI during preparations for and during system operation are not reinforced.
The Post Scram Restart Report, a process intended to evaluate plant response to the scram, failed to address the operability of the diesel generator after it was paralleled to the offsite source at high voltage, tripped on reverse power, and was placed in standby readiness with procedural steps not performed. As a consequence, the restart readiness review, which uses the input of the Post Scram Restart Report for identification of such issues, did not identify operability of the Division 1 diesel generator as a restraint to mode change.
Contributing causes were identified as:
Supervisory attention to the consequences of high offsite voltage was not fully focused due to attention to other priority issues during high workload conditions. Operational limits were not questioned.
Diesel generator governor design routinely challenges successful operation when restoring offsite sources. Prompt operator action and specific procedural guidance are essential to avoid trips.
The diesel generator SOI does not contain procedural steps for avoidance of reverse power trips. Instead, general guidance is presented as a "Note" instead of a `Caution" prior to the step that parallels the generator to the offsite source. As a result, strict procedural compliance may result in reverse power trips.
Simulator training did not provide accurate diesel generator response for restoration of offsite sources.
Reduced attentiveness to determining and tracking operability of systems, during high activity periods, led to the failure of declaring the diesel generator inoperable following the reverse power trip.
Operator mindset that a reverse power trip (RPT) does not have an impact on operability.
Failure to effectively engage Technical Support Center (TSC) and Responsible System Engineer (RSE) resources in the reverse power trip and restoration of the diesel generator to operable status.
III.SAFETY ANALYSIS To assess the impact of the inoperable condition of the Division 1 diesel generator from August 14, 2003 through August 21, 2003 a Probabilistic Safety Assessment (PSA) Risk Impact was performed.
Although the Division 1 diesel generator was inoperable from 1813, on August 14, until 1624, August 21, 2003, an engineering evaluation was performed and concluded the diesel generator would meet its design function.
Since the diesel generator was fully expected to meet its design function, there was no significant increase in risk during the time it was Inoperable.
Additionally, if the diesel generator had started in this condition, the operator would have recognized the voltage was outside the operating band and adjusted it in band. Therefore, there would have been negligible effect on the equipment supplied due to the temporary high voltage condition. The contribution due to the failure mechanism of the diesel generator to fulfill its function (to run for its mission time) would be several orders of magnitude below the existing equipment failure probabilities. This is because operation of the diesel is directed by procedure, the operators are trained to the procedure, a Perry plant operator is dispatched to the diesel generator and monitors the operating limits, the control room operators walk down the panels periodically and verify parameters on operating equipment, and ample time would exist to identify an over voltage condition while still maintaining a negligible effect on connected equipment.
IV.CORRECTIVE ACTIONS:
Immediate Actions Taken The Division 1 diesel generator voltage was adjusted, per the SOI, during the shutdown following the failed surveillance test. Subsequent testing using monthly "Diesel Generator Start and Load" surveillance verified operability of the Division 1 diesel generator.
The system operating instructions were changed to provide a precaution and limitation and a note (in applicable sections), to ensure that Division 1, 2, and 3 diesel generators are not declared operable until their voltage settings are adjusted correctly during the shutdown steps of the SOI.
Preventive Actions The following actions In aggregate represent the Preventative Actions for this event:
Revised DG 801 to add DG voltage limitations to the Precautions and Limitations section and to the steps of the procedures related to voltage.
Expectations will be established for pm-evolution preparation that in addition to P&Ls that 801 Section 5 system operating parameters be included in the preparation review.
A strengthened method to review and document equipment response issues following plant scram and other significant transients will be implemented including a debriefing of emergency response organizations as well as control room personnel that were part of the organizational response.
Remedial Actions The system operating instructions for Division 1, 2, and 3 diesel generators were revised to provide guidance to recover a diesel generator from a tripped condition and to ensure that the diesel generator voltage is adjusted within limits prior to restoring the diesel generator to an operable status.
The system operating instructions for Division 1, 2, and 3 diesel generators were revised to provide the correct sequence and specific steps to adjust the diesel generator speed to prevent a reverse power trip situation and verify grid voltage within acceptable band prior to paralleling evolutions. Other sections of the system operating instructions for the Division 1 and 2 Diesel Generator System and Division 3 Diesel Generator were reviewed for similar corrections.
A process designed to strengthen command and control skills of control room supervisory personnel when exposed to high stress, high activity conditions is being developed. Included in this process is the use of emergency response organizations, prioritization of issues, supervisory oversight and attention to system operability.
A lessons learned discussion of this event was conducted with plant operation section control room personnel, the diesel generator responsible system engineers and the plant engineering section engineers .
A diesel generator design to reduce or eliminate the tendency of the division 1 and 2 diesel generators to reverse power trip when paralleling with offsite sources Is being developed and will be implemented if determined appropriate.
The simulator design database has been modified to incorporate diesel generator characteristics identified during this event.
Additional training was provided to operators on the diesel generator governor and generator response during paralleling operations. The actual load shedding characteristics of the diesel generator when going from lsochronous to Droop voltage control and the need to rapidly go to raise on the governor control switch in this condition needs further discussion. This was done in the simulator with the operators practicing the paralleling operation.
V. PREVIOUS SIMILAR EVENTS
A review of plant operating experience did not reveal any past events where diesel generator output voltage was left out of Technical Specification allowable band as a result of a reverse power trip.
A review of Perry's corrective action program from January 1, 2000 through September 9, 2003 revealed one condition report documenting a 2001 reverse power trip of the Division 1 diesel generator (condition report # 01-1099, Division 1 Diesel Generator Reverse Power Trip). This condition occurred during a post maintenance run used to vent the governor hydraulics after a new governor actuator was installed on the diesel generator. A corrective action for that event provided for the control room operators to review the investigation for sensitivity of governor control issues.
Additionally, it was identified that another reverse power trip of the Division 1 diesel generator occurred during testing in refueling outage 9 (RFO9). This occurrence was not logged in a condition report, but it was documented in the responsible system engineer's RFO9 logbook. This reverse power trip occurred during tuning and testing of the diesel generator governor when it was inoperable for maintenance.
DG operability was not an issue in either case since both trips occurred while the DG was Inoperable for maintenance under conditions where DG voltage had been within the acceptable band prior to the trip.
A similar condition was documented in Perry Licensee Event Report (LER) 2001-005-01 (CR 01-4350). This documented that as a result of normal surveillance testing, the high pressure core spray (HPCS) diesel [DG] was found to have been improperly shutdown during the recovery from a Level 2 reactor protection system actuation and subsequent HPCS diesel start. The cause of this event was attributed to the operator over adjusting the governor prior to opening the generator output breaker. The over-adjustment resulted in the diesel generator failing to meet Technical Specification 3.8.1 surveillance allowable value for rated speed and frequency start time. The system operating Instruction for the Division 3 diesel generator was changed to include steps to verify the diesel generator speed prior to stopping the diesel.
VI. Component Failure There were no structures, systems, or components that were inoperable at the start of this event and contributed to the event.
Energy Industry Identification System (EIIS) codes are identified in the text as [XX].