IR 05000285/2005012
| ML052020042 | |
| Person / Time | |
|---|---|
| Site: | Fort Calhoun  |
| Issue date: | 07/20/2005 |
| From: | Graves D NRC/RGN-IV/DRP/RPB-E |
| To: | Ridenoure R Omaha Public Power District |
| References | |
| IR-05-012 | |
| Download: ML052020042 (18) | |
Text
July 20, 2005
SUBJECT:
FORT CALHOUN STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000285/2005012
Dear Mr. Ridenoure:
On June 22, 2005, the NRC completed a supplemental inspection at your Fort Calhoun Station.
The enclosed report documents the inspection findings, which were discussed with you and other members of your staff.
The NRC issued a White inspection finding and associated Notice of Violation in NRC Inspection Report 05000285/2005010, dated April 15, 2005. This finding involved the failure to promptly identify and correct a condition adverse to quality resulting in Emergency Diesel Generator 2 being inoperable for a period of approximately 28 days, a violation of plant Technical Specifications. The inspection finding was assessed using the significance determination process and was characterized as White, a finding with low to moderate increased importance to safety, which required this additional NRC inspection This supplemental inspection was conducted to provide assurance that the root and contributing causes of the White inspection finding are understood and to provide assurance that the corrective actions are sufficient to address the root and contributing causes and prevent recurrence of the problems. Detailed observations, assessments, and conclusions of the inspection are presented in the enclosed inspection report.
The inspection concluded that the root causes of the finding were adequately defined and understood, and the corrective actions resulting from the evaluations appropriately addressed the identified causes.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Omaha Public Power District
- 2 -
Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/
David N. Graves, Chief Project Branch E Division of Reactor Projects Docket: 50-285 License: DPR-40
Enclosure:
NRC Inspection Report 05000285/2005012
REGION IV==
Docket:
50-285 License:
DPR-40 Report:
05000285/2005012 Licensee:
Omaha Public Power District Facility:
Fort Calhoun Station Location:
Fort Calhoun Station FC-2-4 Adm.
P.O. Box 399, Highway 75 - North of Fort Calhoun Fort Calhoun, Nebraska Dates:
June 13-17, 2005 Inspectors:
R. Azua, Acting Senior Project Engineer Approved By:
D. Graves, Chief, Project Branch E Division of Reactor Projects
Enclosure
SUMMARY OF FINDINGS
IR 05000285/2005012; 06/13-17/2005; Omaha Public Power District; Fort Calhoun Station.
Supplemental Inspection for one White finding in the Mitigating Systems cornerstone.
The inspection was conducted by a senior project engineer. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for which the significance determination process does not apply are indicated by the severity level of the applicable violation. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described at its Reactor Oversight Process website at http://www.nrc.gov/NRR/OVERSIGHT/index.html.
Inspector Identified Findings
Cornerstone: Mitigating Systems
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluation associated with the inoperability of Emergency Diesel Generator 2. This performance issue was previously characterized as having low to moderate risk significance (white) in NRC Inspection Report 05000285/2005010. During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspectors determined that the licensee performed a comprehensive evaluation of the inoperable diesel. The licensees evaluation identified the primary root causes of the performance issue to be premature aging of emergency diesel generator fuses and a lack of formality and rigor by the operators in responding to computer generated alarms. The licensee has taken corrective actions to address both root causes as well as other issues identified as contributing causes.
Given the licensees acceptable performance in addressing the inoperable emergency diesel generator, the white finding associated with this issue will only be considered in assessing plant performance for a total of four quarters in accordance with the guidance in IMC 0305,
Operating Reactor Assessment Program. Implementation of the licensees corrective actions will be reviewed during a future inspection.
REPORT DETAILS
INSPECTION SCOPE
This supplemental inspection was performed by the NRC, in accordance with Inspection Procedure 95001, Inspection for One or Two White inputs in a Strategic Performance Area, for failing to promptly identify and correct a failure of Fuse 2FU in the Emergency Diesel Generator 2 (DG-2) excitation circuit. This failure resulted in DG-2 being inoperable for a period of 28 days from July 21 to August 18, 2004, exceeding the Technical Specification 2.7 limit of 7 days when the reactor coolant system temperature was greater than 300o F. A finding of low to moderate safety significance (White) and the associated Notice of Violation were documented in a letter to Omaha Public Power District on April 15, 2005.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.
Determination as to who identified the issue and under what conditions On October 19, 2004, while reviewing detailed plant computer data related to the operation of DG-2, the licensee discovered that DG-2 had been inoperable for 28 days beginning on July 21, 2004, and extending through August 18, 2004. The inoperability of DG-2 was the result of an open fuse condition affecting the generators voltage output. Data obtained from the plants computer system indicates that the condition occurred as the operators were performing engine unloading and shutdown during completion of the monthly surveillance test on July 21, 2004.
b. Determination of how long the issue existed and prior opportunities for identification
As indicated above, the licensee discovered that DG-2 had been inoperable for 28 days beginning on July 21, 2004, and extending through August 18, 2004.
On August 18, 2004, during the initial acceleration of DG-2 as part of a monthly surveillance test, the licensee noted that the generator output voltage was indicating approximately 2200 Vac instead of the expected 4200 Vac. The licensee halted surveillance activities and secured the diesel. Subsequent troubleshooting revealed that Fuse 2FU in the generator excitation bridge rectifier circuit had failed open. The licensee initially believed that the failure originally occurred during the August 18 surveillance test. However, when the licensee tried to establish the actual time the fuse failed for the purpose of developing performance indicator data for emergency ac power system unavailability, they identified that the fuse in question had actually failed during the July 21, 2004, surveillance test. This conclusion was arrived at following the review of detailed plant computer data.
The licensee performed a root cause analysis and identified a number of opportunities whereby the failed fuse condition could have or should have been identified.
1. On July 21, 2004, following the required, one-hour, fully loaded run, DG-2 was
unloaded in preparation for shutdown and conclusion of the test. At this time, two computer alarms were received approximately one minute apart, A DG-LOW FREQUENCY ALARM and DG-LOW VOLTAGE ALARM. Since these alarms are normally expected during the performance of this surveillance test, the operators acknowledged these alarms as expected alarms out of habit and did not evaluate whether they had alarmed sooner than expected.
2. On July 21, 2004, following the opening of the output breaker and prior to
shutting down DG-2, Surveillance Procedure OP-ST-DG-0002, Diesel Generator 2 Check, directs the operators to record the watt-hour reading. It was noted by the operator that the meter was not displaying a reading. The operators, control room shift supervisor, shift manager, and cognizant system engineer were unaware that the failure of the meter to provide indication was due to a design feature which removed power to the meter if diesel generator output voltage dropped below a specific value. Subsequently, following a review in the control room, the licensee incorrectly determined that the meter had failed and, since the watt-hour meter reading is obtained only for the purpose of recording and crediting the power generated by DG-2 when paralleled to the bus and is not required for diesel generator operability, the licensee declared the meter to be inoperable and proceeded.
3. Following the performance of the test, the shift technical advisor and shift
manager are required by procedure to evaluate and review the test results within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> of test completion. This review consisted of ensuring that all the test acceptance criteria were met and that all test data was attached. The test criteria for this test consisted of:
- All systems/components actuated and performed as required.
- The diesel generator carried continuous load for alt least one hour
- Fuel oil transfer pumps operated as required As part of the surveillance test effort, a Diesel Generator, DG-2, Group Trend Report is generated. The purpose of this report is to assess whether the diesel generator maintained adequate load during the period of time required by the test. However, the report also contained data that indicated the failure of the fuse, i.e., indicated the voltage drop from approximately 4230VAC to 2175VAC.
The licensee determined that, since the report contained information that could have identified the failed fuse condition, this review was considered a missed opportunity.
c.
Determination of the plant-specific risk consequences and compliance concerns associated with the issue In Condition Report 200403634 the licensee satisfactorily identified all of the issues that were directly and indirectly involved with the failure of Fuse 2FU and subsequent
inoperability of DG-2. Each issue was researched and inspected. The interelationships between systems, components, and structures were evaluated.
The licensees risk assessment was reviewed when the finding was evaluated through the significance determination process. The NRC determined the licensees results to be valid and were in agreement with the final conclusion that the finding was more than minor and that the finding had low to moderate safety significance because DG-2 was unavailable to respond upon demand for loss of off-site power and would have been unable to perform its mitigating system function.
02.02 Root Cause and Extent of Condition Evaluation a.
Evaluation of methods used to identify root causes and contributing causes The licensee utilized Fault Tree Analysis, Event and Causal Factor Analysis, field walkdowns, document reviews, and personnel interviews. These methodologies are generally accepted as standard methods and were adequately utilized for this finding.
The inspector concluded that the licensee effectively utilized accepted root cause determination methods and adequately identified the root and contributing causes for this finding.
b.
Level of detail of the root cause evaluation The licensee was thorough in their analysis of the equipment failures and human performance errors.
The licensees root cause investigation identified two major causes for the 28-day inoperability of DG-2.
1. The root cause for the open fuse condition was determined to be the result of
premature aging. The licensee indicated that the failed fuse had experienced accelerated degradation due to past overexcitation events. Due to the paired fuse configuration that exists in this portion of the voltage regulation system, overcurrent conditions that result in a single blown fuse also place the paired fuse in a near blown condition. This condition can lead to early failure due to heating effects (surface cracking) present on the fuse element surface.
considered to be the lack of formality and rigor in validating computer alarms which occur during the performance of surveillance testing The following items were identified as root cause contributors:
- Lack of a specific Fort Calhoun Station policy or guidance document concerning the need to replace fuses in sets once one fuse within the set has been blown.
- The absence of a control panel alarm would indicate that output voltage had dropped below required levels for the condition of the diesel generator system, the lack of which likely provided a false sense of security that plant conditions remained normal.
- The plant computer alarm display currently provides indications on a wide variety of computer alarms as well as information concerning changes in state of a wide variety of monitored equipment, such as valve and breaker positions. The licensee determined that the number of alarms and change-of-state flags that are presented to the operator during the course of a shift could present a challenge for maintaining appropriate attention levels due to information overload.
- Initial engineering assessment activities related to determining the point in time at which fuse failure had occurred did not consider all the available historical information from the previous test performed in July.
The inspector noted that the level of detail of the root cause evaluation was adequate, with some exceptions.
First, the root cause analysis identified the failure by the operator, on July 21, 2004, to note the drop in voltage on the control panel voltmeter as a missed opportunity to detect the fuse failure. The inspector found that no further discussion or recommended actions were documented regarding this observation. Subsequent interviews with operations personnel and reviews of licensee actions to address the overall issue showed that, even though it was not documented, this issue was discussed at length and that corrective actions were put into place as a result. The failure by the licensee to document this information placed any planned corrective action outside the licensees tracking system, which may have resulted in no action being taken. In this case, operator training on this particular aspect was recommended and added to planned operator performance measures.
Second, Action Item 3 of the root cause analysis, which addressed the extent of condition regarding failed fuses, called for a generic review of all safety-related components that may have experienced an overcurrent condition resulting in a blown fuse. The action called for reviewing any identified conditions to determine if pair set fuses needed to be replaced. It was noted that the documented actions by the assigned staff member did not seem to address all of the recommended actions. After interviewing the individual in question, he was able to recreate the actions taken which indeed did address the recommended actions.
These observations show the need to ensure that information is properly documented to prevent the inadvertent omission of recommended actions and subsequent completed actions.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensee reviewed their corrective action program and did not identify any prior occurrences identical to this event. The licensee, however, reviewed other condition reports where a fuse failure was involved even though they did not match the event in question.
The licensee also reviewed operating experiences from several different events throughout the industry in addition to reviewing 10 CFR Part 21 reports. The licensee contacted fuse vendors and also reviewed a number of NUREG documents related to fuse failure and fuse aging.
d.
Consideration of potential common causes, extent of condition, and extent of cause of the problem Extent of Condition is defined as the extent to which the actual condition exists with other plant processes, equipment, or human performance.
The licensee conducted a systematic review of fuses in other plant systems, in addition to the diesel generator system.
The diesel generator fuse review included an evaluation to determine if the cyclic load aging could be present in any other fuses in the diesel generator system. There are four fuses in each diesel generator that are potentially subjected to the same aging mechanism. Two are 5-kV fuses on the primary of the 4160 V/240 V single phase transformer used to supply ac power for the excitation system. Two fuses are installed on the secondary of that same 4160 V/240 V single phase transformer and are used to protect the SCR/Rectifier circuit that supplies the generator dc field current. One of these fuses on DG-2 is the fuse that failed.
There are 17 fuses on each diesel generator that are energized during the time from diesel generator field flash until the generator reaches rated voltage. Fourteen of these fuses are generator instrument potential transformer circuits that provide voltage signals for metering, protection, and control for each diesel generator. Three of the fuses are 5-kV fuses on the primary of the 4160 V/480 V three-phase transformer that can be used as an alternate power source for the fuel oil transfer pumps. These fuses have not been subjected to the cyclic load aging, but are subjected to cyclic voltage changes. The licensee plans further investigation to ensure that this aging mechanism does not have an adverse aging affect.
With regard to other plant systems, the licensee considered that cyclic aging may be occurring on other fuses in the plant. Thus the licensee conducted a review of other plant systems. The licensee did not identify any occasions where fuses in other plant systems had experienced cyclic aging.
Extent of Cause is defined as the extent to which the root causes of an identified problem have impacted other plant processes, equipment, or human performance.
The licensee evaluated the extent of cause of the reactor operator's lack of formality and rigor in validating computer alarms which occurred during the performance of surveillance testing of DG-2. The licensee determined that, with regard to the computer alarms, the cause was unique in nature and limited to the control room. The emergency operations facility receives the same computer alarms; however, there is no response requirement for personnel in that facility. In addition, the licensee identified that a contributing cause to the lack of formality and rigor in validating computer alarms by the operators was the fact that the computer currently provides indications on a wide variety of computer alarms as well as information concerning changes in state of a wide variety of monitored equipment, such as valve and breaker positions. The licensee determined that the number of alarms and changes-of-state flags that are presented to the operator during the course of a shift could present a challenge for maintaining appropriate attention levels due to information overload. As a result, the licensee, as part of their corrective actions, will be evaluating which emergency response facility computer system alarms can be eliminated.
02.03 Corrective Actions a.
Appropriateness of corrective actions The inspector reviewed the licensees immediate and long-term corrective actions.
1. Revise appropriate station procedures concerning fuse replacement to require
that fuses be replaced in sets. Training to be provided to electrical maintenance, planning, and system engineering personnel through the procedure change process.
2. Review other fuses or fuse sets associated with the operation of the diesel
generator system to determine if other fuses could be susceptible to similar premature aging effects as noted in the root cause analysis. Provide replacement work requests to be scheduled to coincide with scheduled diesel generator testing if applicable.
3. Review of fuse failures at the Fort Calhoun Station and other industry data bases
to determine if the Fort Calhoun Station fuse replacement program warrants updating in addition to replacing fuses in sets.
4. Revise Procedure SO-O-1, Conduct of Operations, and ARP-1, Annunciator
Response Procedure, to require reactor operators to acknowledge all computer annunciators during normal operation.
5. Revise Procedure SO-O-1 and ARP-1 to have computer alarm response match
annunciator tile alarm response
6. Include discussion and simulator scenarios of this event in Operations
initial/requalification training, stressing the importance of verifying system response, validating annunciators, and alarm response requirements.
7. Evaluate possibility of modifying diesel generator alarms in such a manner that
they would be tied to applicable engine speed so that they would alarm when a true abnormal condition exists.
8. Initiate an engineering change to create low voltage and frequency annunciator
tiles to provide additional indication of abnormal diesel generator condition.
9. Initiate an engineering change to eliminate emergency response facility
computer system alarms that do not indicate a degraded condition, affect safety-related system status or parameter, or affect continued operation.
10.
Add appropriate steps to the diesel generator operation procedure to verify correct voltage is present prior to depressing the stop pushbuttons when shutting down the diesel generators.
11.
Add steps to appropriate diesel generator surveillance tests to provide printouts of data from the plant data server for system engineering review during the normal review process for the surveillance tests.
12.
Provide training for the engineering division at a generic continuing training session.
13.
Diesel generator performance is acceptable for three surveillance periods.
14.
Through observation, measure whether or not Corrective Action Items 4, 5, and 6 meet the expectations of the manager of Operations.
The inspector determined that the licensees proposed corrective actions were appropriate to address the root causes identified to prevent recurrence. For corrective actions that had already been completed, the inspector reviewed the licensees efforts.
No problems were noted.
b.
Prioritization of corrective actions The inspector concluded that the corrective actions were properly prioritized. Actions of an immediate nature were given the highest priority and accomplished on an acceptable schedule. Actions to resolve program, training, and procedure weaknesses were established. A completion date and a responsible manager were assigned for each corrective action, and these were tracked through the corrective action system.
c.
Establishment of schedule for implementing and completing the corrective actions The inspector determined that, by the end of the inspection, the licensee had completed all but two of the corrective actions for Condition Report 200403634 (Action Items 13 and 14). The inspector reviewed a sample of the 12 corrective actions and concluded that they had been implemented successfully.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees root cause analysis and recommended corrective actions were reviewed and approved by the Plant Review Committee. Each recommended corrective action was assigned a member of licensee management for responsibility, and completion of these actions will be tracked and trended through the licensees corrective action program. One of the licensees corrective actions specifically calls for the licensee to evaluate the effectiveness of some of the other corrective actions. Additionally, the licensee's corrective action program requires effectiveness reviews for Level 1, 2, or 3 condition reports. This issue was a Level 2 condition report.
OTHER ACTIVITIES
40A3 Event Followup (Closed) Licensee Event Report 05000285/2004002-00: Inoperable Diesel Generator for 28 Days Due to Blown Fuse During Shutdown.
On October 19, 2004, while reviewing plant computer data related to the operation of DG-2, it was discovered that DG-2 had become inoperable for 28 days from July 21 to August 18, 2004. The inoperability was the result of a failed fuse affecting DG-2 voltage output exceeding the Technical Specification 2.7 limit of 7 days when the reactor coolant system temperature was greater than 300o F. A finding of low to moderate safety significance (White) and the associated Notice of Violation were documented in NRC Inspection Report 05000285/2005010, Section 1R15. The licensee event report was reviewed by the inspectors and no new findings were identified. The licensee documented the issue in Condition Report 200403634. This licensee event report is closed.
40A5 Other (Closed) VIO 05000285/2005010-01: DG-2 Inoperable in Excess of Technical Specifications due to Failed Fuse.
As noted in the above Section 02, Evaluation of Inspection Requirements, the inspector reviewed the licensees root cause analysis developed for Condition Report 200403634. As indicated, the inspector found that it adequately identified the
apparent root causes of the finding in question and determined that the licensees recommended corrective actions were appropriate to preclude recurrence of this event.
This violation is closed.
40A6 Management Meetings
Exit Meeting Summary
The inspectors presented the inspection results to Mr. David Bannister, Plant Manager, Fort Calhoun Station, and other members of licensee management via a conference call on June 22, 2005. The licensee acknowledged the information presented.
The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. No proprietary information was identified.
Upon completion of the exit meeting, a Regulatory Performance Meeting was conducted as prescribed in NRC Manual Chapter 0305 to discuss the event, root causes, and corrective actions. The significance of the finding was reviewed and the performance issues, underlying causes, and corrective actions were discussed and understood.
ATTACHMENT:
SUPPLEMENTAL INFORMATION
Attachment
SUPPLEMENTAL INFORMATION
PARTIAL LIST OF PERSONS CONTACTED
Licensee
- D. Bannister, Plant Manager
- M. Bare, System Engineer, System Engineering
- G. Cavanaugh, Supervisor, Station Licensing
- M. Core, Manager, System Engineering
- D. Dryden, Station Licensing
- S. Goodell, Manager, Operations
- H. Faulhaber, Division Manager, Engineering
- J. Herman, Manager, Nuclear Licensing
- T. Pilmairo, Manager, Corrective Action Group
- R. Roning, System Engineer, System Engineering
- S. Swearngin, Supervisor, Reliability Engineering
- J. Tills, Manager, Maintenance
- D. Trausch, Manager, Quality Assurance
- J. Zagata, Maintenance Rule Engineer, Reliability Engineering
NRC
- J. Hanna, Senior Resident Inspector
- L. Willoughby, Resident Inspector
LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED
Closed
LER
Inoperable Diesel Generator for 28 Days Due to Blown
Fuse During Shutdown (Section 4OA3). 05000285/2005010-01
Emergency DG-2 Inoperable in Excess of Technical
Specifications due to Failed Fuse (Section 4OA5).
LIST OF DOCUMENTS REVIEWED
RIVER BEND STATION CONDITION REPORTS (CRs)
Number
Topic
CR-200403634
Inoperability of DG-2 Diesel Generator During Engine
Shutdown
Attachment
CR-200203542
Fuse Failure in Reactor Power Calibration and Indication
Panel Assembly AI-31A-AW11
CR-200402186
Control Power Fuse Failure for Group 3 of Bank 1 of the
Pressurizer Proportional and Backup Heater Banks
CR-200402456
Fuse on Diesel Fire Pump FP-1B Battery Charger was
found to be blown
CR-200402879
Emergency Diesel Generator DG-2 Output Voltage Read
200 Vac Instead of Expected 4160 Vac During
Surveillance Test OP-ST-DG-0002
CR-200404060
Questions Raised by NRC Resident Inspectors Following
Issuance of the Root Cause Analysis for CR-200403634
CR-200402518
Watt/Hour Meter WH/D2 Had No Indication
MISCELLANEOUS
Document
Description
Revision
OI-DG-1
Diesel Generator 1 (Operating Instruction)
OI-DG-2
Diesel Generator 2
OP-ST-DG-001
Diesel Generator 1 Check
OP-ST-DG-002
Diesel Generator 2 Check
OP-ST-ESF-001
Diesel Auto Start Initiating Circuit Check
OP-ST-ESF-002
Diesel Generator No.1 and No.2 Auto Operation
MM-ST-DG-0001
Diesel Generator DG-1 Inspection
MM-ST-DG-0002
Diesel Generator DG-2 Inspection
SO-O-1
Conduct of Operations
Master Checklist for Plant Startup
ARP-1
Annunciator Response Procedure
NOD-QP-19
Cause Analysis Program
EM-RR-EX-0601
Replacement of Fuses / Safety Related
Attachment
OMA-25505
Exelon Power Labs - Failure Analysis of Fuses for
(09/22/2004)
Diesel Generator No.2 - Shawmut Amptrap
Fuse A24X100 Type (4)