IR 05000440/2003009
| ML032880107 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 10/10/2003 |
| From: | Reynolds S Division Reactor Projects III |
| To: | Kanda W FirstEnergy Nuclear Operating Co |
| Shared Package | |
| ML032880283 | List: |
| References | |
| IR-03-009 | |
| Download: ML032880107 (28) | |
Text
ber 10, 2003
SUBJECT:
PERRY NUCLEAR POWER PLANT NRC SPECIAL INSPECTION REPORT 05000440/2003009
Dear Mr. Kanda:
On September 11, 2003, the U.S. Nuclear Regulatory Commission (NRC) completed a special team inspection at your Perry Nuclear Power Plant. The enclosed report documents the inspection findings which were discussed with Mr. T. Rausch and other members of your staff on September 11, 2003.
On August 14, 2003, the Perry plant scrammed as a result of a loss of offsite power (LOOP).
During the event, the plant protective systems successfully shutdown the plant and the emergency diesel generators (EDGs) successfully started and provided power to the emergency busses. During the recovery from the LOOP, two anomalous conditions occurred which merited further review by the NRC. The first involved air binding of the Division 1 waterleg pump which was designed to keep the discharge lines of the residual heat removal (RHR) A and low pressure core spray (LPCS) systems full. As a result of the air binding, both the RHR A and LPCS systems became inoperable until the pump was vented. The second condition was discovered on August 21, 2003, when the Division 1 EDG failed a surveillance due to high output voltage. The Special Inspection evaluated the facts, potential significance, and your resolution of these issues. The inspectors conducted the inspection in accordance with Inspection Procedure 93812.
The Perry staffs root cause evaluation, extent of condition review, and past operability determination for the waterleg pump air binding were not complete at the time of the inspection.
Further work was ongoing, including a determination of the gas accumulation rate in the feedwater leakage control system piping which your root cause team had identified as the most probable cause of the waterleg pump air binding. The inspectors will conduct further reviews of the Perry staffs ongoing actions including the timeliness of these actions. The inspectors concluded that with the successful venting of the feedwater leakage control system piping on September 10, 2003, you had established a reasonable basis for system operability at that time. The inspectors also concluded that subsequent venting, at a periodicity to be determined by gas accumulation rate data, was an adequate compensatory action to maintain system operability. Therefore, the inspectors concluded that a safety issue did not exist at the end of the inspection. Although the root cause evaluation for the EDG issue was also not complete at the time of the inspection, you had made sufficient progress in determining the cause of the EDG reverse power trip and subsequent failure to recognize that the EDG was not shutdown in a manner to establish operability. As a result, procedural changes were implemented to the EDG operating procedure to provide assurance that a similar shutdown would not result in a declaration of operability until a proper shutdown occurred. The inspection report documents one self-revealed finding of very low safety significance (Green). The finding was determined to involve a violation of NRC requirements. However, because of its very low safety significance and because it had been entered into your corrective action program, the NRC is treating this finding as a Non-Cited Violation in accordance with Section VI.A.1 of the NRCs Enforcement Policy.
If you contest the subject or severity of a Non-Cited Violation, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with a copy to the Regional Administrator, U.S. Nuclear Regulatory Commission - Region III, 801 Warrenville Road, Lisle, IL 60532-4351; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the Resident Inspector Office at the Perry Nuclear Power Plant.
In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC's document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/ RA /
Steven A. Reynolds, Acting Director Division of Reactor Projects Docket No. 50-440 License No. NPF-58
Enclosure:
Inspection Report 05000440/2003009 w/Attachments: A. Supplemental Information B. Figure See Attached Distribution
DOCUMENT NAME: G:/perr/ML032880107.wpd *See previous concurrence To receive a copy of this document, indicate in the box:"C" = Copy without enclosure "E"= Copy with enclosure"N"= No copy OFFICE RIII RIII RIII RIII NAME MRing for MRing* SReynolds RLerch/trn*
DATE 10/09/03 10/09/03 10/10/03 10/ /03 OFFICIAL RECORD COPY
REGION III==
Docket No: 50-440 License No: NPF-58 Report No: 05000440/2003009 Licensee: FirstEnergy Nuclear Operating Company (FENOC)
Facility: Perry Nuclear Power Plant, Unit 1 Location: P.O. Box 97 A200 Perry, OH 44081 Dates: August 27 through September 11, 2003 Inspector : R. Powell, Senior Resident Inspector J. Ellegood, Resident Inspector T. Steadham, Resident Inspector, Fermi Approved by: Mark A. Ring, Chief Branch 1 Division of Reactor Projects Enclosure
SUMMARY OF FINDINGS
IR 05000440/2003009; 08/27/2003 - 09/11/2003; Perry Nuclear Power Plant. Special
Inspection for anomalies at Perry following the August 14, 2003, loss of offsite power (LOOP).
This Special Inspection examined the facts and circumstances surrounding two anomalous conditions related to the August 14, 2003, LOOP event. The first involved the discovery during the LOOP event that the Division 1 waterleg pump was air-bound and required venting. As a result, the A loop of the residual heat removal (RHR) system and the low pressure core spray (LPCS) system also had to be filled and vented prior to restoring the systems to service. The second anomalous condition was discovered about a week after the LOOP event when the Division 1 emergency diesel generator (EDG) exceeded the high voltage criteria during a monthly surveillance test. This EDG had tripped off simultaneously with the restoration of offsite power on August 14. The inspection was conducted by Resident Inspectors from Perry and Fermi. This inspection identified one Green Non-Cited Violation (NCV). The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review.
The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
NRC-Identified
Finding
Cornerstone: Mitigating Systems
- Green.
A self-revealed violation of Technical Specification (TS) 5.4 occurred on August 21, 2003, when the Division 1 emergency diesel generator (EDG) failed its surveillance due to high output voltage. Technical Specification 5.4 required maintenance and implementation of procedures required by Regulatory Guide 1.33. Regulatory Guide 1.33 required procedures for EDG operation.
Licensee procedures did not provide direction to perform proper EDG restoration following an automatic EDG trip.
The finding was greater than minor because it could reasonably be viewed as a precursor to a significant event and was associated with the mitigating system cornerstone attribute of equipment reliability. The finding affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences.
Specifically, the EDG was relied upon to provide emergency power to safety systems in the event of a LOOP. The finding is of very low safety significance because no damage to equipment occurred and operators would have been able to restore proper EDG output voltage. As such, no loss of safety function would have occurred. (Section 4OA3.8)
EXECUTIVE SUMMARY Background At approximately 4:10 p.m. on August 14, 2003, the Perry Nuclear Power Plant scrammed from 100 percent power during a LOOP. The LOOP was part of a larger electrical blackout, now called Blackout 2003, which encompassed portions of Northern Ohio, Western Michigan, portions of Canada, parts of New York, Pennsylvania and New Jersey. Plant systems operated as designed immediately after the plant scram. Operators confirmed that the reactor shutdown and that the plants EDGs started and connected to the emergency busses. At approximately 4:20 p.m. the plant declared an Unusual Event and remained there until 7:52 p.m. on August 15, 2003. During the Unusual Event two anomalous conditions occurred that warranted further NRC review. The first concerned air binding of the RHR A and LPCS waterleg pump which resulted in a low discharge pressure alarm on the pumps at 4:10 p.m. The other condition concerned a reverse power trip of the Division 1 EDG while restoring offsite power to the Division 1 emergency bus at 6:14 p.m. On its next surveillance, the EDG failed because of high output voltage.
Division 1 Waterleg Pump Power to all four waterleg pumps was momentarily interrupted by the LOOP until the EDGs started and loaded their respective electrical busses. Although all four waterleg pumps restarted, the RHR A and LPCS pump low discharge pressure alarms were received and did not clear. Both alarms remained locked in because the common waterleg pump for the RHR A and LPCS systems was not supplying adequate discharge pressure. At approximately 9:45 p.m. the licensee vented an abnormal amount of gas from the waterleg pump. Following venting, control room personnel successfully started the pump and verified both low pressure alarms cleared at 9:52 p.m. Subsequently, both LPCS and RHR A were successfully used in restoring plant conditions.
The licensees root cause team developed a theory concerning gas accumulation in the vertical section of feedwater leakage control system piping due to the waterleg pump stripping gasses from solution. The licensee theorized that gasses collected in the vertical piping and remained pressurized at approximately 44 psig. When the pump lost power, the pressure dropped, thus allowing the gas bubble to expand and void a large portion of the waterleg pump piping.
The licensee validated the theory on September 10 when a substantial quantity of gas was vented from the piping. The licensee determined that the quantity of gas vented would have been more than sufficient to re-bind the pump had another LOOP occurred. Thus this system and therefore LPCS and RHR A were inoperable for a yet to be determined period of time between August 14 and September 10. Further, the waterleg pump was likely inoperable for some period of time prior to the August 14 LOOP. As a result, both LPCS and RHR A were also likely inoperable.
At the end of this inspection, the licensees root cause investigation, extent of condition review, and past operability review, were not complete. The licensee was in the process of developing a testing methodology to obtain additional gas accumulation rate data in order to better ascertain past system operability. The inspectors concurred with the licensees determination that following the venting of the feedwater leakage control system piping on September 10, the LPCS/RHR A waterleg pump was demonstrated to be fully operable and thus no immediate safety issue existed. The inspectors also concurred that with the proposed venting periodicity, gas accumulation would not challenge system operability. As the licensee systematically vented the piping to obtain the accumulation rate data, system operability was being assured.
Further review of issues concerning past operability, the licensees initial operability determination, feedwater leakage control system design, and the adequacy of system venting procedures will be addressed following completion of the licensees reviews. An Unresolved Item (URI) was opened in Section 4OA3.6 of this report.
Division 1 EDG As part of the licensees recovery plan from the LOOP, the licensee coordinated with grid operators to quickly restore a source of offsite power to permit transfer of electric loads from the EDGs to offsite power. At 6:12 p.m. the licensee closed the preferred power source breaker for the Division 1 emergency bus and shortly thereafter, the Division 1 EDG tripped on reverse power. The licensee placed the EDG in standby using the applicable procedure; however, the performance of a procedure step to adjust generator output voltage could not be performed since the EDG had tripped. Subsequent reviews by the licensee failed to identify the ramifications of the inability to perform this step. On August 21 the EDG exceeded the high voltage criteria during its surveillance test and was declared inoperable. Later that day, the generator successfully passed the surveillance. An NCV associated with procedure adequacy was identified and is discussed in Section 4OA3.8 of this report.
REPORT DETAILS
OTHER ACTIVITIES
4OA3 Event Follow Up
.1 Sequence of Events - LPCS/RHR A Waterleg Pump
August 14, 2003 4:10 p.m.
The Perry Nuclear Power Plant experienced a LOOP with subsequent reactor scram.
4:10 p.m.
Power was lost to all four waterleg pumps.
4:10 p.m.
EDGs started and powered the respective busses.
4:10 p.m.
All four waterleg pumps restarted after power was restored from the EDGs.
4:10 p.m.
The following annunciators were received: RHR PUMP A DISCHARGE PRESSURE HI/LO, LPCS PUMP DISCHARGE PRESSURE LO, RHR PUMP B DISCHARGE PRESSURE HI/LO.
4:11 p.m.
RHR PUMP B DISCHARGE PRESSURE HI/LO alarm cleared.
5:36 p.m.
A team was dispatched to verify that the LPCS/RHR A waterleg pump breaker was closed.
5:51 p.m.
The LPCS/RHR A waterleg pump was still in service with a suction pressure of 9 psig and a discharge pressure of 7 psig.
5:57 p.m.
The LPCS/RHR A waterleg pump was secured and then restarted from the Control Room. No alarms cleared as a result of this action.
7:30 p.m.
The LPCS/RHR A waterleg pump was secured.
7:58 p.m.
A team was dispatched to perform a local walkdown of the LPCS/RHR A waterleg pump. No signs of leakage were noted after inspecting the LPCS/RHR A valve room, RHR A heat exchanger room, LPCS pump room, and the RHR A pump room. The coupling guard was removed from the pump and nothing unusual was noted. Both the pump and motor were found to be warm to the touch.
9:45 p.m A team of operations and engineering personnel was dispatched to vent the LPCS/RHR A waterleg pump. After the pump casing vent valve was opened, air was vented for about the first 3 seconds. After approximately 8 seconds of venting, an air-free stream of water was seen from the vent valve and the vent was closed.
9:52 p.m The LPCS/RHR A waterleg pump was started from the control room and both the RHR PUMP A DISCHARGE PRESSURE HI/LO and LPCS PUMP DISCHARGE PRESSURE LO alarms cleared.
.2 Review of Waterleg Pump Operability Evaluation
a. Inspection Scope
The inspectors reviewed the technical adequacy of the licensees immediate investigation and operability basis as documented in Condition Report (CR) 03-04764 dated August 18, 2003.
b. Findings
The LPCS/RHR A waterleg pump provides keep-fill pressure for both the LPCS and RHR A systems and is one of two pumps that provide a source of water for the feedwater leakage control system. The keep-fill function reduces the injection time for both LPCS and RHR A following a loss-of-coolant accident and prevents water hammer events subsequent to LPCS or RHR A pump starts. The feedwater leakage control system limits offsite release following a design basis loss-of-coolant accident by providing a water seal. Thus, the waterleg pump is required to maintain operability of three systems: LPCS, RHR A, and feedwater leakage control system.
Power to all four waterleg pumps was momentarily interrupted on August 14 due to the LOOP. The EDGs started and loaded the respective electrical busses. Although all four waterleg pumps restarted, the RHR A and LPCS pump low discharge pressure alarms were received and did not clear. Both alarms remained locked in because the common waterleg pump for the RHR A and LPCS systems was not supplying adequate discharge pressure. At approximately 9:45 p.m. the licensee vented an abnormal amount of gas from the pump. Following venting, control room personnel successfully started the pump and verified both low pressure alarms cleared at 9:52 p.m. Both LPCS and RHR A were successfully used in restoring plant conditions after the low pressure alarms were cleared.
Following the pump venting, the licensee determined the system to be fully operable because:
- (1) the waterleg pump had been performing acceptably since the pump had been vented;
- (2) a complete system fill and vent had been performed on the waterleg pump piping such that the possibility of re-binding the pump was small; and
- (3) engineering judgement. After reviewing the licensees initial immediate investigation and operability basis, the inspectors were unable to identify sufficient evidence of a continuing operability issue.
Throughout the Special Inspection, the inspectors challenged the licensee to reconsider waterleg pump operability as new data was acquired. Throughout, the licensee presented arguments to support system operability based on the facts available at the time. Regardless of the cause of the pump binding, the inspectors recognized that a proven method - venting of the pump casing - existed to recover the pump should the pump have lost power while the issue was still under evaluation.
The inspectors also reviewed the licensees basis for declaring the other three waterleg pumps operable. No issues of significance were identified due, in part, to the normal operation of the pumps before, during, and after the LOOP and, more significantly, the differences in system configurations.
On September 9, after consultation with an independent pump expert, the licensees root cause team evaluation began to focus on gas accumulation in the feedwater leakage control system piping. As the licensee evaluated the normal operation of the waterleg pump as a gas source (stripping gasses from solution) the licensee also recognized that the feedwater leakage control system vent valve was not included in the system fill and vent procedure. The vent valve was physically located approximately 65 feet above the waterleg pump and was the system high point vent. The feedwater leakage control system piping provided a location for gasses to collect outside the pump since it is a vertical run of piping in the pump discharge line (see figure in Attachment B in the back of this report). The licensee theorized that gasses collected in this vertical section of pipe and remained pressurized at approximately 44 psig. When the pump lost power, the pressure dropped, thus allowing the accumulated gas to expand and void the waterleg pump piping up to the LPCS suction piping and the check valves between LPCS and RHR A.
The licensee theorized that when the pump was vented on August 14 only some of the gas was removed from the piping - enough to fill the suction piping and the pump casing with water, while potentially, the remainder of the system remained voided. In other words, the feedwater leakage control system piping would still contain a gas bubble since that section of pipe was not vented and was at a higher elevation than the pumps vent valve. The process of venting the pump removed enough gas from the system to re-prime the pump and allowed the pump to be restarted. The inspectors questioned the licensee about system operability since, if the theory was correct, gas would remain trapped in the feedwater leakage control system piping and pump operation would generate additional gas.
The licensee planned to vent the feedwater leakage control system piping the morning of September 10 but postponed the venting in order to complete a water hammer evaluation concerning starting and stopping of the LPCS/RHR A waterleg pump. After discussions with the inspectors on the priority assigned to the water hammer evaluation, the evaluation was completed and venting was rescheduled for the evening of September 10. The inspectors witnessed the venting and observed a significant quantity of gas vented from the feedwater leakage control system piping.
The licensees engineering personnel determined that the feedwater leakage control system piping had a volume of approximately 1.65 ft3 and that a 44 psig gas bubble occupying 40 percent of that volume would expand enough to essentially void the entire LPCS/RHR A waterleg pump piping, up to the check valves, when the pressure dropped to 7 psig. When the feedwater leakage control system piping was vented on September 10 the rotometer connected to the vent measured a flow of 0.4 scfm of water-free gas passing through the vent line for 18 minutes and 45 seconds. This quantity of vented gas was more than sufficient to re-bind the pump had another LOOP occurred prior to the September 10 venting. Therefore, the inspectors concluded that the system likely became inoperable sometime between August 14 and September 10.
At the end of this inspection, the licensee was developing plans to obtain additional gas accumulation rate data in order to better ascertain past system operability. Additionally, the licensee had not completed the root cause evaluation which should address both LPCS and RHR A system operability and availability. The inspectors noted that with the successful venting of the feedwater leakage control system piping on September 10, the licensee established a reasonable basis for operability of the waterleg pump. The inspectors also concluded that subsequent venting, at a periodicity to be determined by gas accumulation rate data, was an adequate compensatory action to maintain system operability. Therefore, the inspectors concluded that a safety issue did not exist at the end of the inspection.
.3 Review of LPCS/RHR A Waterleg Pump Operability Prior to the August 14, 2003,
LOOP (93812)
a. Inspection Scope
The inspectors reviewed the results of the venting performed on September 10 and conducted interviews of plant personnel to determine the operability of the LPCS and RHR A systems prior to the LOOP.
b. Findings
The September 10 venting evolution established that gasses were collecting in the dead-leg vertical run of feedwater leakage control system piping. The inspectors agreed with the licensees theory that gas expansion from the feedwater leakage control system piping was the most plausible explanation of why the waterleg pump failed to perform satisfactorily after the LOOP. Because this pocket of trapped gasses existed prior to the LOOP, it was also reasonable to conclude that the waterleg pump, and hence, both the LPCS and RHR A systems, were inoperable for some period of time prior to the LOOP.
The licensee determined that the feedwater leakage control system piping was filled with water on May 9 when water was seen coming from the open feedwater leakage control system outboard header vent valve, 1N27F786 (the system high point vent),while the waterleg pump was operating. As such, it is reasonable to conclude that sometime between May 9 and August 14 the feedwater leakage control system piping coming from the LPCS/RHR A waterleg pump discharge line accumulated a sufficient quantity of gasses to air-bind the waterleg pump prior to August 14 if power was interrupted to the pump. Additionally, the condition would not have been unique to the current operating cycle and, therefore, operability during previous operating cycles would have been challenged as well.
At the end of this inspection, the exact dates of system operability could not be determined. The licensee informed the inspectors that they would continue venting the feedwater leakage control system piping periodically in order to quantify the gas accumulation rates. Once this information is known, a more accurate determination of the waterleg pump past operability can be made.
Pending the determination of the gas accumulation rate and completion of the licensees root cause evaluation, the inspectors consider past operability issues to be an Unresolved Item (URI 05000440/2003009-01).
.4 Review of Waterleg Pump System Differences
a. Inspection Scope
The inspectors reviewed drawings, walked down system piping, and interviewed plant personnel to ascertain any key similarities or differences between the waterleg pump systems. This information was reviewed to determine if any common mode failure mechanism could exist such that the operability of the other waterleg pumps could be in question.
b. Findings
The inspectors determined that only two of the waterleg pumps had dead-leg runs of piping where gasses could accumulate: the LPCS/RHR A waterleg pump and the RHR B waterleg pump. Both dead-legs were piping for the feedwater leakage control system.
The discharge piping from the LPCS/RHR A waterleg pump initially ran vertically out of the pump (all waterleg pumps were end suction, vertical discharge type pumps) and then teed off to a horizontal section of pipe. The horizontal section of piping teed off to a vertical run of approximately 65 feet. Thus, a collection point for any gas leaving the pump would have been in the vertical rise of the feedwater leakage control system.
The key difference with the RHR B waterleg pump was the piping orientation. The pumps discharge piping was vertical where the feedwater leakage control system piping teed off horizontally. Thus, any gasses leaving the pumps discharge would have been swept up and past the feedwater leakage control system piping. It was less likely that an appreciable amount of gas would accumulate in this feedwater leakage control system piping. Therefore, the inspectors concluded that the air binding that occurred after the LOOP was limited to the LPCS/RHR A waterleg pump because of the differences between the waterleg system piping.
.5 Review of Human and Equipment Performance - LPCS/RHR A Waterleg Pump
a. Inspection Scope
The inspectors reviewed waterleg pump performance information to identify human or equipment performance issues that could have contributed to the LPCS/RHR A waterleg pump failure after the LOOP. The inspectors reviewed vendor technical manuals, system drawings, and other documents, and interviewed various plant personnel who were involved with the event.
b. Findings
The inspectors determined that the LPCS/RHR A waterleg pump performed as designed in accordance with the pump vendors technical manual. Because the pump was not a self-priming pump, it could not re-prime absent operator intervention. Once the operators re-established the prime as a result of venting the pump casing, the pump resumed satisfactory operation.
The inspectors reviewed the piping diagrams and performed system walkdowns. The system design appeared to be susceptible to gas accumulation in the feedwater leakage control system supply line. However, the inspectors noted that the system was equipped with a vent valve that, if used, would have prevented this problem.
The inspectors reviewed the operator response to the waterleg pump failure and determined that personnel responded appropriately. Specifically, the pump was secured, piping was walked down to look for a system break, and the pump was vented which ultimately allowed the pump to restart. With events in progress, specifically LOOP recovery, the inspectors concluded operators gave the issue appropriate priority at the time of discovery. The inspectors noted the control room staff discussed options available if the waterleg pump could not be recovered. However, the inspectors also recognized that human error contributed to the failure since the procedure and practice used to fill and vent the system did not include the feedwater leakage control system high point vent. This procedure had been used for many years, thus opportunity existed to detect the omission prior to the LOOP.
The inspectors did not identify any other human performance or equipment performance issues that contributed to this event, with the exception of the inadequate interface between feedwater leakage control system design and system venting procedures.
.6 Review of Adequacy of Procedures - LPCS/RHR A Waterleg Pump
a. Inspection Scope
The inspectors reviewed various plant procedures to determine if any procedural inadequacies contributed to the cause of the LPCS/RHR A waterleg pump failure after the LOOP.
b. Findings
The inspectors reviewed the licensee's procedure for performing a complete system fill and vent, SOI-E12, "Residual Heat Removal System," Rev. 14. The inspectors concluded that the feedwater leakage control system outboard header vent valve, 1N27F786, was not included in the licensee's fill and vent procedures. However, this review was not complete at the end of the inspection. Therefore, the inspectors considered procedure adequacy to be an element of URI 05000440/2003009-01.
.7 Evaluation of Root Cause Efforts - LPCS/RHR A Waterleg Pump
a. Inspection Scope
The inspectors monitored the licensees root cause efforts. The inspectors interviewed licensee personnel and observed data gathering efforts such as system venting operations.
b. Findings
At the end of this inspection, the licensees root cause investigation, extent of condition review, and past operability review, were not complete. However, the licensee had determined that gas accumulation in the feedwater leakage control system piping was the most probable cause of the air binding of the waterleg. On September 10 the licensee vented the feedwater leakage control system and essentially validated the gas accumulation theory. While doing so, the licensee also negated the previous operability basis. At the end of this inspection, the licensee was in the process of developing a testing methodology to obtain additional gas accumulation rate data in order to better ascertain when the system became inoperable.
Throughout the inspection, the inspectors discussed the status of the root cause review with senior plant management. The inspectors were concerned that greater progress had not been made with respect to the root cause determination. Although the licensee indicated that the root cause efforts were progressing on a schedule commensurate with the safety significance of the event, licensee management concurred that the formation of the root cause team should have been more timely. The inspectors noted that once the root cause team was formed, the licensee performed the root cause evaluation with technically competent personnel in a careful and thorough manner. However, the inspectors did note that the licensee was slow in requesting vendor and independent expert assistance. The inspectors noted that while many of the investigative actions taken by the licensee required specific plant conditions or involved planned safety system unavailability; the venting of the feedwater leakage control system did not. Therefore, the feedwater leakage control system venting should have been more expeditiously pursued.
.8 Sequence of Events - EDG
August 14, 2003 4:10:25 p.m. Turbine trip; reactor scram.
4:10:28 p.m. Undervoltage condition on vital busses EH11, EH12 and EH13; Division 1 and 2 EDG start signals.
4:10:29 p.m. Division 3 EDG start signal.
4:10:35 p.m. Division 1 and 2 EDG breakers closed.
4:10:37 p.m. Division 3 EDG breaker closed.
6:12 p.m.
Division 1 EDG tripped on reverse power while paralleling with offsite power.
6:25 p.m.
Shift Manager authorized NA (not applicable) to procedural step to adjust EDG voltage.
6:35 p.m.
Division 1 EDG restored to standby.
August 21, 2003 11:52 a.m.
Division 1 EDG started for monthly surveillance.
12:06 p.m.
Division 1 EDG failed surveillance due to voltage high at 4500 V (acceptance criteria 3900 to 4400 V); declared EDG inoperable.
12:26 p.m.
Division 1 EDG shutdown per operating instruction, including voltage adjustment.
1:57 p.m.
Division 1 EDG started for monthly surveillance.
4:24 p.m.
Division 1 EDG declared operable following surveillance.
.9 Review of EDG Reverse Power Trip and Subsequent Failed Surveillance
a. Inspection Scope
The inspectors reviewed the sequence of events surrounding the August 14 reverse power trip of the Division 1 EDG through its successful surveillance on August 21. In order to understand the event, the inspectors reviewed operator logs, CRs, operating procedures, and computer data. This inspection also included interviews with licensee personnel directly involved with the reverse power trip, as well as those involved with plant restoration and recovery. The inspectors observed a simulator re-enactment of the reverse power event. The inspection reviewed procedure adequacy, system design, human performance, and equipment performance, to determine their contribution to the trip, and subsequent failed surveillance. Finally, the inspectors reviewed the potential for damage to equipment that could occur due to an EDG reverse power or operation of the emergency bus at an abnormally high voltage.
b. Findings
The inspectors concluded that the equipment operated as designed and that the design was consistent with industry standards. In addition, the inspectors concluded that the operators of the EDG followed applicable procedures. However, the procedures used neither adequately addressed adding load to the EDG following breaker closure, nor restoration of the EDG to a proper standby lineup following an automatic shutdown. In addition, the inspectors concluded that the licensee had several opportunities to identify the improperly adjusted voltage prior to the next routine surveillance. Based on this review, the inspectors concluded that the following NCV was warranted.
Introduction:
A self-revealed violation of TS 5.4 occurred on August 21 when the Division 1 EDG failed its surveillance due to a high output voltage.
Description:
On August 14, immediately after paralleling the Division 1 EDG with the grid, the generator tripped on reverse power. Since the plant was recovering from a LOOP, the licensee quickly placed the EDG in a standby condition and declared it operable. As written, the procedure to place the EDG in standby requires the operator to adjust EDG output voltage to between 4100 and 4200 V in order to establish the proper voltage on the next start. Since this adjustment can only be performed with the EDG operating and the reverse power caused the EDG to shutdown, the operator obtained two senior reactor operator concurrences to omit that step and continue with the procedure. Once the operator had completed the rest of the procedure, the shift manager declared the EDG operable. Neither the procedure used for EDG operation nor the alarm response procedure contained steps to address adjustment of EDG parameters following an automatic shutdown. On August 15, the licensee performed a post-run EDG roll. No other activities were performed on the EDG until the failed surveillance on August 21. With the EDG inoperable, the licensee left Mode 4 and entered Mode 2 on August 20. On August 21 the licensee started the EDG for routine surveillance and declared it inoperable when the voltage was 4500 V versus the acceptance criteria of 3900 to 4400 V.
The inspectors concluded that several opportunities existed for the licensee to detect the improper adjustment of the generator voltage prior to performance of the surveillance.
The operators had the first opportunity during EDG shutdown had they recognized that the voltage at shutdown was out of specification high and would require adjustment prior to declaration of EDG operability. In addition, when a step in a procedure was not performed, the licensee typically wrote a CR. Since the plant computer was not working due to the LOOP, a CR was not generated for this specific exception to a procedural step. A CR was written to document the reverse power trip, but it was written a day later by an individual performing documentation reviews. The final opportunity to identify the high voltage was the routine review of EDG data conducted by engineering personnel.
While the licensee performed a review, they failed to identify the high value of the generator as-left voltage. The inspectors reviewed the computer data for the EDG shutdown and concluded that the high voltage was readily apparent on the computer charts.
Analysis:
The inspectors determined that failing to adjust EDG output voltage to the proper value prior to declaring the EDG operable represented a performance deficiency necessitating a significance evaluation. The inspectors applied the criteria in IMC 0612, Power Reactor Inspection Reports, Appendix B, Issue Disposition Screening, and concluded that the finding was greater than minor because it could reasonably be viewed as a precursor to a significant event and was associated with the mitigating system cornerstone attribute of equipment reliability. The finding affected the cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the EDG was relied upon to provide emergency power to safety systems in the event of LOOP. To that end, Technical Specifications were written to establish acceptance criteria to assure that the EDG could supply these loads without damaging either the EDG or the attached loads.
In other circumstances, the EDG may have been left in a condition where it could have damaged EDG loads or caused protective functions to disconnect the EDG from the bus.
In this case, the voltages supplied to the bus did not result in any damage. The finding is of very low safety significance because no damage to equipment occurred and operators would have readily been able to restore proper EDG output voltage. As such, no loss of safety function would have occurred.
Enforcement:
The performance deficiency associated with this event was the failure to implement and maintain procedures for EDG operation. Technical Specification 5.4 required maintenance and implementation of procedures required by Regulatory Guide 1.33. Regulatory Guide 1.33 required procedures for EDG operation. The licensee developed SOI-R43 Division 1 and 2 Diesel Generator System for generator operation and ARI-H13-P887-1 for alarm response. Neither procedure provided direction to perform proper EDG restoration following an automatic EDG trip. The licensee entered this finding into the corrective action program (CR 03-04912). Because of the very low safety significance and because the issue has been entered into the licensees corrective action program, it is being treated as an NCV, consistent with Section VI.A.1 of the NRC Enforcement Policy (NCV 05000440/2003009-02).
.10 Extent of Condition Review - EDG
a. Inspection Scope
The inspectors reviewed the licensees progress on the root cause and discussed planned activities with system engineers and plant management. The inspectors reviewed documentation that provided a chronology of events, including graphical representations of key EDG parameters. The inspectors reviewed the operating procedures in use for EDG operation and surveillance.
b. Findings
The inspectors concluded that the licensees extent of condition review was not complete at the conclusion of the inspection. However, the licensee had made considerable progress in determining the cause of the EDG reverse power trip and subsequent failure to recognize the EDG was not shutdown in a manner to establish operability. In addition, the licensee had implemented procedure changes to their EDG operating procedure to provide assurance that a similar shutdown would not result in a declaration of operability until a proper shutdown occurred. Based on their progress, the inspectors concluded that the extent of condition would be sufficiently broad to prevent recurrence.
4OA6 Meetings
Exit Meeting The inspectors presented the inspection results to Mr. T. Rausch and other members of licensee management at the conclusion of the inspection on September 11. The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee:
- W. Kanda, Vice President-Nuclear
- V. Higaki, Manager, Regulatory Affairs
- J. Lausberg, Supervisor, Compliance
- T. Rausch, General Manager, Nuclear Power Plant Department
- A. Rabenold, Acting Operations Superintendent, Plant Operations
- A. Pusateri, Lead HVAC/Diesel Generator System Engineer, Plant Engineering
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
- 05000440/2003009-01 URI Review of Low Pressure Core Spray/Residual Heat Removal A Waterleg Pump Operability Prior to the August 14, 2003, Loss of Offsite Power
- 05000440/2003009-02 NCV Failure to Implement and Follow Procedures for Diesel Generator Operation