05000339/LER-2010-001
| North Anna Power Station, Unit 2 | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| 3392010001R00 - NRC Website | |
1.0 DESCRIPTION OF THE EVENT On April 27, 2010 at 1637 hours0.0189 days <br />0.455 hours <br />0.00271 weeks <br />6.228785e-4 months <br />, with Unit 2 operating in Mode 1 at 74% power during recovery from a refueling outage, a generator lockout occurred while testing a new digital automatic voltage regulator (AVR) (El IS System — TL, Component — RG). The generator lockout initiated a turbine trip and subsequent reactor trip.�
- A new main generator automatic voltage regulator manufactured by Turbine Control Service Associates (TCSA) was installed in North Anna Unit 2 by Design Change Package (DCP)07-010 during the Spring 2010 refueling outage. The new voltage regulator required the limiting and protective functions (minimum excitation, over-excitation, volts per hertz, etc.) to be tuned. This required introducing small step changes at specified power levels and verifying the response matches the prediction. If the actual response does not match the prediction, then the applicable voltage regulator variables are changed and tuning repeated until the actual response matches the predicted response. The step changes are introduced through TCSA tuning software on a laptop that is connected to the Digital Generator Controller (DGC).
Minimum Excitation Limiter (MEL) tuning at approximately 30% power had been completed with minimal changes to the applicable voltage regulator set points and Unit 2 power level was raised to 74%. At 74% power, the laptop was again connected to the DGC, the required step change was entered into the tuning software, the 'GO' button was pushed to start the step change, and the unit tripped on exceeding the Minimum Excitation Protection (MEP) set point.
Troubleshooting was performed following the reactor trip. It was determined that the MEL tuning software uses a 5-point discrete curve for the MEL in the test tuning mode. The normal operating MEL curve, based on the equation of a circle, is not used. The troubleshooting revealed that the tuning software used for MEL tuning, upon initiation, temporarily reconfigures the MEP set point calculation to utilize the 5-point discrete curve.
The x-y values used for 5-point discrete curve were default values that permitted the generator to be operated under-excited only up to 593 mega-watts (MW). North Anna 2 is rated greater than 1000 MW. When the tuning software was initiated, the actual operation at approximately 750 MW exceeded the generic set point which resulted in a unit trip on under-excitation.
Following the reactor trip, all three Auxiliary Feedwater (AFW) Pumps (El IS System — BA, Component — P) received an automatic start signal due to low/low level in the steam generators (EllS Component — SG). Shortly after the automatic start of the Turbine Driven AFW (TDAFW) pump, the Control Room received annunciator alarm (F-D8). This was an AFW Pump Trouble or Lube Oil Trouble alarm. An operator was dispatched and reported that oil was spraying from the pump bearings and reservoir. Abnormal Procedure O-AP-23, "Oil or Hazardous Substance Spill Response," was entered. Steam generator water level was subsequently restored above 23% in all three steam generators. The TDAFW Pump had operated for thirty-one minutes and was subsequently secured.
After the TDAFW Pump was secured, oil stopped spraying and reservoir pressure began decreasing. The TDAFW pump was declared inoperable and quarantined for cause investigation. A 72 hour8.333333e-4 days <br />0.02 hours <br />1.190476e-4 weeks <br />2.7396e-5 months <br /> action, Technical Specification 3.7.5, was entered for one AFW Pump inoperable.
A four-hour and eight-hour report was made to the NRC on April 27, 2010 at 1844 hours0.0213 days <br />0.512 hours <br />0.00305 weeks <br />7.01642e-4 months <br />, due to a Reactor Protection System actuation in accordance with 10 CFR 50.72(b)(2)(iv)(B) and an Engineered Safety Feature (ElIS System — JE) actuation of the AFW System in accordance with 10 CFR 50.72(b)(3)(iv)(A). This event is reportable per 10 CFR 50.73(a)(2)(iv)(A) for the automatic actuation of the Reactor Protection System and the AFW System.
2.0 SIGNIFICANT SAFETY CONSEQUENCES AND IMPLICATIONS No significant safety consequences resulted from this event since all Engineered Safety Feature equipment initially responded as designed. Steam Generator inventory was restored to the normal operating level. The health and safety of the public were not affected by this event.
The TDAFW Pump performed its design function for the duration it was needed to do so.
Subsequent troubleshooting determined the oil spray was found to be caused by an over pressurization of the lube oil system. However, as the oil level decreased in the lube oil reservoir, more air volume would be available for system expansion and the oil leakage would diminish. Once the vent on the sight glass was uncovered, the oil system would become vented and the oil leakage would stop and would be self corrected. At this level there is sufficient volume of oil in the reservoir to provide adequate cooling and lubrication.
3.0 CAUSE A root cause evaluation (RCE) of the event was performed. The direct cause of the generator lock-out protective relay actuation was incorrect MEP set points derived from the voltage regulator MEL tuning software. The MEL acts by modifying the AVR's output to keep from exceeding the main generator's lower operating limit.
The root cause of the generator lockout protective relay actuation was determined to be inadequate guidance for software validation for non-safety related equipment that can impact power generation. Specifically, Dominion Nuclear Administrative Procedure (DNAP) 0306 "Software Quality Assurance Program" does not provide requirements for an effective review of computer software for non-safety related power block digital equipment.
In addition, DNAP-0306 exempts software written to perform calculations that must be validated prior to each use (e.g., to hand calculations in accordance with the applicable Design Control Program "Calculation" process). As a result, computer software calculations for power generation equipment classified as non-safety did not receive adequate testing or simulation prior to use on an operating plant system. Also, DNAP-0306 does not require a design verification report to validate the software design basis of computer software for non-safety related power block digital equipment. As a result, an effective review was not performed on the vendor's operating software.
A contributing cause to the event was failure to follow administrative procedural requirements due to ineffective change management of an engineering risk assessment procedure. Procedure CM-AA-RSK-1001, Engineering Risk Assessment was not initiated or used in the design change process which contributed to an inadequate risk assessment of the design change process. In addition, personnel failed to validate computer operating software design parameters and vendor technical documents contributed to a faulty mental model on the operation of the MEL tuning program which contributed to the inadequate review of the vendor's software parameters.
The cause of the oil leakage from the TDAFW pump bearings and reservoir was determined to be inadequate ventilation for the lube oil reservoir. The inadequate ventilation of the lube oil reservoir was due to Engineering Transmittal ET-ME-93-009 removing the vented cap on the lube oil reservoir to reduce water intrusion. This ET relied on the existing deflector style seals to provide a vent path for the lube oil reservoir. A 1995 item equivalency replaced these seals with a zero leakage seal not realizing that seals provide a vent path. This issue did not become apparent until additional leak paths were removed during maintenance activities during the Spring Unit 2 refueling outage.
4.0 IMMEDIATE CORRECTIVE ACTION(S) Control Room personnel responded to the event in accordance with emergency procedure 2-E-0, Reactor Trip or Safety Injection. Control Room personnel stabilized the plant using ES-0.1, Reactor Trip Recovery. All Engineered Safety Feature equipment initially responded as designed. The TDAFW pump was secured due to developing a lube oil leak. The TDAFW pump was declared inoperable and a 72-hour limiting action of Technical Specification 3.7.5 was entered.
5.0 ADDITIONAL CORRECTIVE ACTIONS The software setting errors in the MEL tuning utility for the North Anna Unit 2 AVR were corrected. Testing on the new AVR was successfully completed.
Technical documents and source material on the vendor's reference website were updated.
As an interim corrective action, a required reading was initiated to ensure the following requirements are implemented in DCPs (approved or awaiting approval) with non-safety related computer software affecting power generation and /or reactivity. Requirements to include at a minimum:
- Utilize CM-AA-RSK-1001, Engineering Risk Assessment
- Utilize DNAP-0306 Level I software classification requirements for software testing and code validation of non-safety related power block equipment operating software affecting power generation and/or reactivity.
- Validate computer software design basis for operating and testing parameters.
- Software test plan incorporating all system operating and testing parameters.
- Simulation of operating software on a simulator prior to use on plant equipment.
- Requirements to re-run software testing when adding or removing software parameters or features.
Maintenance was performed on the TDAFW pump to re-establish a vent path for the lube oil reservoir.
6.0 ACTIONS TO PREVENT RECURRENCE Actions to prevent recurrence from the RCE are being tracked in the Central Reporting System to completion.
7.0 SIMILAR EVENTS None.
8.0 ADDITIONAL INFORMATION Unit 1 was operating in Mode 1 at 100% power at the time of this event and remained at approximately 100% power for the duration of the event.
The Unit 1 TDAFW pump and both the Unit 1 and 2 motor driven AFW pumps have an installed vent cap in its lube oil reservoir. No problems have been reported due to over pressurization on these lube oil systems.
Component Information Affected System: Voltage Regulator Component Manufacturer: Siemens Power Generation, Inc Sub-Contractor: Turbine Control Service Associates, Inc �