05000346/LER-2006-001
| Docket Numbersequential Rev | |
| Event date: | 01-20-2006 |
|---|---|
| Report date: | 10-06-2006 |
| Reporting criterion: | 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications |
| 3462006001R02 - NRC Website | |
12. LICENSEE CONTACT FOR THIS LER
FACILITY NAME TELEPHONE NUMBER (Include Area Code) Gerald M. Wolf, Staff Engineer, Regulatory Compliance (419) 321-8001
13. COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT
MANU- REPORTABLE MANU- REPORTABLECAUSE SYSTEM COMPONENT CAUSE SYSTEM COMPONENTFACTURER TO EPIX FACTURER TO EPIX 14. SUPPLEMENTAL REPORT EXPECTED 15. EXPECTED MONTH DAY YEAR
- YES (If yes, complete EXPECTED SUBMISSION DATE).1 @ NO DATE
SUBMISSION
ABSTRACT (Limit to 1400 spaces, i.e., approximately 15 single-spaced typewritten lines On January 20, 2006, while preparing Emergency Diesel Generator (EDG) 2 for testing, an investigation was performed of a tapping noise heard following overhaul maintenance the prior week. When the top deck on the EDG engine was opened, damage was discovered to a cylinder valve bridge. The damage was determined to be due to an improperly torqued lash adjustment screw lock nut, which was caused by inadequate verification of the torquing. The observed damage would have prevented the EDG from operating for its duty time of one week as specified in the Safety Analysis Report. Because the EDG was inoperable from the start of the six year preventive maintenance overhaul on January 8, 2006, until repairs were completed on January 23, 2006, or more than the seven days permitted by the Technical Specifications, this event is being reported in accordance with 10 CFR 50.73(a)(2)(i)(B) as a condition prohibited by the Technical Specifications. The diesel engine maintenance procedure will be revised to incorporate concurrent verification for the step that requires torquing of the lash adjustment screw lock nuts and add additional signoffs for steps that require a torquing operation to be performed. Because a structured problem solving technique was not used when the tapping sound was first heard, training in problem solving techniques has been given to applicable personnel.
DESCRIPTION OF OCCURRENCE:
System Description:
The onsite electric power system [EB] at the Davis-Besse Nuclear Power Station (DBNPS) is powered by two separate and independent Emergency Diesel Generators (EDGs) [EK-DG]. These EDGs provide onsite standby electrical power sources to supply their respective 4160 Volt Essential Buses [EB-BU]. Each of the EDGs are equipped with a twenty cylinder model 645-E4 diesel engine manufactured by Electro-Motive Division, and are nominally rated for 2600 kilowatts continuous electric service. A minimum of 36,000 gallons of fuel oil is stored on site for each of the EDGs to permit operation for at least seven days following a Loss of Offsite Power. Additionally, one non-class lE diesel generator can be aligned to power either 4160 Volt essential bus in the event of a station blackout.
DBNPS Technical Specification (TS) Limiting Condition for Operation (LCO) 3.8.1.1.b requires two separate and independent EDGs to be Operable while the plant is operating in Modes 1, 2, 3, and 4.
With one EDG inoperable, the Operability of the remaining A.C. sources must be demonstrated by performing Surveillance Requirement 4.8.1.1.1.a (verification of correct breaker alignments and indicated power availability) within one hour and at least once per 8 hours9.259259e-5 days <br />0.00222 hours <br />1.322751e-5 weeks <br />3.044e-6 months <br /> thereafter and by performing Surveillance Requirement 4.8.1.1.2.a.4 (verification that the Operable EDG starts and accelerates up to 900 rpm) within 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br />. Two EDGs must be restored to Operable status within 7 days or the plant must be in Hot Standby (Mode 3) within the next 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> and in Cold Shutdown (Mode 5) within the following 30 hours3.472222e-4 days <br />0.00833 hours <br />4.960317e-5 weeks <br />1.1415e-5 months <br />.
Event Description:
On January 8, 2006, at 1505 hours0.0174 days <br />0.418 hours <br />0.00249 weeks <br />5.726525e-4 months <br />, with the plant at power operation (Mode 1), EDG 2 was removed from service to perform six year preventive maintenance activities. The main purpose of the EDG 2 outage was to perform an overhaul of the engine to replace the head to liner gaskets. Based on inspection criteria, replacement of the valve bridges was determined to be required. During the process of engine reassembly, the exhaust valve lash is set for each of the cylinders. Once the lash is set, a lock nut is tightened and torqued to 80 foot-pounds to ensure the lash adjustment screw remains in position. The entire six year preventive maintenance activity was governed by procedure DB-MM- 09320, Revision 10, "Emergency and Station Blackout Diesel Engine Maintenance.
After completion of the maintenance activities, a three hour test run of EDG 2 was performed. During this test run, a tapping noise was identified in the area of cylinders 4 and 5 at approximately 2000 hours0.0231 days <br />0.556 hours <br />0.00331 weeks <br />7.61e-4 months <br /> on January 13, 2006. Discussions between the vendor representative and FENOC personnel present as well as examination of test data concluded that the tapping noise was due to a lash adjustment being off and that the cylinders were operating normally. Follow-up documentation was initiated to further investigate the noise at a later time. EDG 2 was declared operable on January 14, 2006 at 0000 hours0 days <br />0 hours <br />0 weeks <br />0 months <br />.
On January 20, 2006, at 1557 hours0.018 days <br />0.433 hours <br />0.00257 weeks <br />5.924385e-4 months <br />, the air start motors were tagged out for barring of the EDG prior to running the engine for post-maintenance testing after change out of a fuel filter. The top deck on EDG 2 was opened to investigate the tapping noise heard on January 13, and damage was discovered to the cylinder 4 left valve bridge. The left rocker arm of cylinder 4 was found with the lash adjustment screw lock nut missing from the top of the rocker arm. The missing lash adjustment screw lock nut was DESCRIPTION OF OCCURRENCE: (continued) eventually located in the oil trough. The extent of damage to EDG 2 was isolated to cylinder 4. Repairs to the EDG 2 engine were initiated. On January 23, 2006, after completion of maintenance and testing activities, EDG 2 was declared operable at 0642 hours0.00743 days <br />0.178 hours <br />0.00106 weeks <br />2.44281e-4 months <br />.
As a result of the damage discovered to the cylinder 4 left valve bridge of EDG 2, the EDG would not have been able to supply power to its respective 4160 Volt Essential Bus for the seven days specified in the Safety Analysis Report. This condition existed from January 14, 2006, when EDG 2 was declared operable following preventive maintenance, until January 23, 2006, at 0642 hours0.00743 days <br />0.178 hours <br />0.00106 weeks <br />2.44281e-4 months <br /> when repairs to the engine were completed and EDG 2 declared operable. This timeframe, in conjunction with the approximately 5.4 days EDG 2 was inoperable for preventive maintenance, resulted in EDG 2 being incapable of performing its required safety function for a total of approximately 14.7 days.
Because EDG 2 was incapable of performing its required safety function for longer than the 7 days permitted by TS LCO 3.8.1.1 Action b while the plant remained in Mode 1, the plant was operated in a condition prohibited by the Technical Specifications, which is reportable per 10 CFR 50.73(a)(2)(i)(B).
During this timeframe, both EDG 1 (except for approximately 3 minutes as explained later under the Analysis of Occurrence Section) and the non-Class lE diesel generator were Operable.
APPARENT CAUSE OF OCCURRENCE:
The most probable cause of the damage to EDG 2 is the failure to torque the left lash adjustment screw lock nuts due to an inadequate level of verification. Miscommunications between the lead mechanic and the vendor representative are more than likely the initiator. Both the lead mechanic and the vendor representative are aware that the lash adjustment screw lock nuts are required to be torqued, but the vendor representative normally only tightens the lash adjustment screw lock nuts with a hand wrench.
The lash adjustment screw lock nuts are normally torqued after all cylinders have had their lash set.
The miscommunication may have occurred upon completion of the lash adjustment by the vendor representative communicating to the lead mechanic. The lead mechanic did not adequately verify whether the torquing of the lash adjustment screw lock nuts was completed as part of the lash adjustment, and signed off for the torquing of the lash adjustment screw lock nuts as complete when the torquing had not been performed.
The basis for declaring a probable cause is due to not being able to prove conclusively whether the lash adjustment screw lock nuts were properly torqued, or that a component failure occurred. The overall evidence indicates that the lock nuts were not torqued even though all but one of the lash adjustment screw lock nuts were tight. Subsequent analysis and inspection of the damaged EDG 2 components by the vendor identified no component manufacturing defects that contributed to this event.
Three contributing causes were identified relating to this event. The first is that the lead mechanic is expected to sign off the procedure for the work performed regardless of who performs the maintenance.
The expectation is that the lead mechanic provides direction to those working on the engine, ensures the work is completed correctly, and signs the procedure to document the work performed. The diesel engine is a V-20 that has work platforms on either side of the engine, which are blocked from direct view of each other. While this dual platform configuration allows for simultaneous maintenance activities, it also increases the difficulty of one person verifying the completion of the work activities on both platforms.
APPARENT CAUSE OF OCCURRENCE: (continued) The second contributing cause is the structure of the procedure used to perform the six year preventive maintenance activity on the diesel engine. Procedure DB-MM-09320, Revision 10, "Emergency and Station Blackout Diesel Engine Maintenance," includes 33 enclosures of which 13 are not directly related to maintenance on the engine cylinders. The procedure enclosures are not ordered in specific sequence to facilitate ease of use. Some of the enclosures have only one signoff for each cylinder even though the enclosure includes several steps.
The third contributing cause is the failure to use proper place keeping in a procedure designated as "Step-By-Step" level of use. While procedure DB-MM-09320 had proper place keeping in a number of the enclosures, this proper place keeping was not used consistently.
An evaluation was performed regarding the conclusion that none of the possible causes of the tapping noise identified as part of post-maintenance testing on January 13, 2006, warranted immediate investigation. Had this tapping noise been addressed without delay, the repairs to EDG 2 may have been completed within the TS allowed outage time of seven days. The cause for not fully investigating the issue prior to declaring EDG 2 operable was determined to be that a structured problem solving and decision making technique was not used when evaluating the tapping sound. The collective opinion of those involved was that the tapping sound was related to an exhaust valve lash being slightly out of adjustment, and similar diesel engine tapping sounds are not unusual. The group consensus was that the condition would not affect the ability of the equipment to perform its function, and that adjustment could be made at a later date. The noise level adjacent to the operating diesel engine was very high (to the point that normal discussions were difficult), and the group did not assemble to a better environment (less noise) to discuss and brainstorm the concern.
ANALYSIS OF OCCURRENCE:
The damage to the EDG 2 engine's left valve bridge was due to a loss of compression between the head and rocker arm, caused by the lash adjustment screw backing out to the fully retracted position.
The valve bridge was then free to lift off of the head, which eventually resulted in the damage to the bronze spring retainer. As the valve bridge cycled up and down, the lip of the bronze spring retainer broke off, allowing the retainer to slide up the center post. This allowed the center post to strike the top of the head, resulting in elongation of the spherical depression that held the bronze spring retainer. The left rocker arm was damaged where it contacted the valve bridge that was found skewed from its normal position.
A damaged valve bridge will result in a higher rotation rate of the associated exhaust valves due to the loss of the hydraulic snubbing effect of the lash adjusters. The higher rotational rate of the exhaust valves will result in wearing of the exhaust valve stem in the area of the valve keeper. The accelerated wear of the exhaust valve stem could result in failure of the valve stem, and the exhaust valve would then drop into the engine cylinder.
The amount of time that it would take to cause a failure of the exhaust valve stem due to the accelerated rotational rate is unknown. There is one known case in the nuclear industry where an exhaust valve dropped into an engine cylinder of similar design. The failure of the exhaust valve occurred 19 hours2.199074e-4 days <br />0.00528 hours <br />3.141534e-5 weeks <br />7.2295e-6 months <br /> into a 24-hour test run following a maintenance outage, resulting in catastrophic ANALYSIS OF OCCURRENCE: (continued) failure of the engine. The cause of this exhaust valve failure was suspected to be failure to torque the lash adjustment screw lock nut.
During the time EDG 2 was inoperable due to the loose/missing lash adjustment screw lock nut, EDG 1 was also declared inoperable on January 18, 2006, from 2259 to 2302 hours0.0266 days <br />0.639 hours <br />0.00381 weeks <br />8.75911e-4 months <br /> while swapping the operating Component Cooling Water (CCW) Pumps [CC-P] on train 1. When CCW Pump 1 was started with CCW Pump 3 operating, EDG 1 was considered inoperable due to the potential overload condition that would exist if a Loss of Coolant Accident were to occur coincident with a Loss of Offsite Power. However, based upon engineering judgment, no loss of safety function existed during this brief period of time (3 minutes) since EDG 2 remained functional and exhibited no abnormal operating characteristics when it was shutdown following testing on January 13, 2006, and returned to service on January 14, 2006. Except for these 3 minutes, EDG 1 remained operable during the time EDG 2 was inoperable for the planned maintenance outage as well as due to the loose/missing lash adjustment screw lock nut.
During the time EDG 2 was inoperable, Containment Spray (CS) Pump 1 [BE-P] was inoperable from January 17, 2006, at 0233 hours0.0027 days <br />0.0647 hours <br />3.852513e-4 weeks <br />8.86565e-5 months <br /> until January 20, 2006, at 0539 hours0.00624 days <br />0.15 hours <br />8.912037e-4 weeks <br />2.050895e-4 months <br /> for a planned maintenance outage. DBNPS TS LCO 3.6.2.1 requires two independent containment spray systems to be Operable while the plant is operating in Modes 1, 2, 3, and 4. With one containment spray system inoperable, the inoperable spray system must be restored to Operable status within 7 days. However, while CS Pump 1 was inoperable for maintenance, redundant CS Pump 2 was also inoperable because it had no emergency power source (EDG 2). Because the plant remained in Mode 1 with both CS Pumps inoperable, this represents an additional example of operation in a condition prohibited by the Technical Specifications.
During the time EDG 2 was inoperable, Station Vent Radiation Monitor [IL-MON] RE4598AA was inoperable on January 16, 2006, from 1234 hours0.0143 days <br />0.343 hours <br />0.00204 weeks <br />4.69537e-4 months <br /> to 1524 hours0.0176 days <br />0.423 hours <br />0.00252 weeks <br />5.79882e-4 months <br /> for maintenance, and again on January 17, 2006, from 1055 hours0.0122 days <br />0.293 hours <br />0.00174 weeks <br />4.014275e-4 months <br /> to 1201 hours0.0139 days <br />0.334 hours <br />0.00199 weeks <br />4.569805e-4 months <br /> for a filter change. DBNPS TS LCO 3.7.6.1 requires two independent control room emergency ventilation systems [VI] to be Operable while the plant is operating in Modes 1, 2, 3, and 4. With both channels of Station Vent Normal Range Radiation Monitoring instrumentation inoperable, the control room normal ventilation system [VI] must be isolated and at least one control room emergency ventilation system train be placed in operation within one hour. While RE4598AA was inoperable for maintenance and for a filter change, redundant Station Vent Radiation Monitor RE4598BA was also inoperable because it had no emergency power source.
Because the plant remained in Mode 1 with both Station Vent Radiation Monitors inoperable and the control room normal ventilation system in service, this represents an additional example of operation in a condition prohibited by the Technical Specifications.
Based upon the availability of train 1 equipment to respond to any actual event during the time EDG 2 was inoperable due to the loose/missing lash adjustment screw lock nut except for the brief periods of time described above involving trains, independent of the EDGs, this event had minimal safety significance. A more detailed risk analysis was also conducted to determine the total incremental conditional core damage probability (ICCDP) for this event, which included evaluation of internal events, internal flooding, interfacing system leakage, seismic events and fires. The ICCDP for this event was determined to be approximately 6.8 E-07, confirming that this event was of very low safety significance.
CORRECTIVE ACTIONS:
On January 20, 2006, upon discovery of the damage to EDG 2 cylinder 4 left valve bridge, repairs to the engine were initiated. All other adjusting screw lock nuts were checked and found to be tightened to at least the required torque value. The other 19 cylinders were inspected and no damage was found.
EDG 2 was declared operable on January 23, 2006, at 0642 hours0.00743 days <br />0.178 hours <br />0.00106 weeks <br />2.44281e-4 months <br /> after completion of maintenance and testing activities.
Procedure DB-MM-09320, "Emergency and Station Blackout Diesel Engine Maintenance," will be revised to incorporate concurrent verification signoffs for the step that requires torquing of the lash adjustment screw lock nuts and any other operation for which concurrent verification is appropriate.
Additional signoffs will also be added to the procedure for steps that require a torquing operation to be performed, which will allow the individual performing the torquing to initial that the task was completed.
The procedure and associated enclosures will be reorganized to accommodate field work sequencing, and the procedure will be separated into individual procedures for each preventive maintenance activity (2-year, 4-year, 6-year, and 12-year preventive maintenance activities). The procedures associated with the 2-year preventive maintenance activities have been developed, approved, and implemented in order to support scheduled Station Blackout Diesel Generator and EDG 2 outages in October and November of 2006. The remainder of the procedures for the 4-year, 6-year, and 12-year preventive maintenance activities will be implemented prior to January 19, 2007, in order to support the EDG 1 outage currently scheduled for the week of January 22, 2007.
This event was reviewed with mechanical maintenance personnel on the lessons learned. This review focused on the expectation of using the approved place keeping method when performing a procedure that is designated "Step-By-Step" level of use. In addition, this review included the consequences related to this event to the station and personnel.
Training has been presented to improve worker understanding and performance in the area of problem solving techniques. The target audience for this training was engineering personnel, maintenance supervisors and above, and operations supervisors and above, who are typically involved in these types of problem solving issues.
The latest revision to procedure NOP-ER-3001, "Problem Solving and Decision Making," which has been implemented at other FirstEnergy Nuclear Operating Company facilities, has been implemented at the DBNPS. This revision (Revision 02), which was not previously implemented at the DBNPS due to the impending Fourteenth Refueling Outage, provides a simplified, yet structured troubleshooting technique that can be used for equipment deficiencies that do not rise to the level of requiring a formal problem solving team.
FAILURE DATA:
There have been no Licensee Event Reports submitted by FENOC for the DBNPS in the last three years reporting an event associated with inoperability of an Emergency Diesel Generator due to damaged engine components or a maintenance activity.
FAILURE DATA: (continued) An event was documented on a Condition Report in 2003 where inadequate written Preventive Maintenance instructions led to improper torquing of the EDG 2 piston cooling oil pipe flange bolts for one cylinder. Contributing to this 2003 event was the lack of sufficient written detail regarding the torquing of these piston cooling oil pipe flange bolts in procedure DB-MM-09320, "Emergency and Station Blackout Diesel Engine Maintenance." Corrective Actions were completed to revise the procedure and Preventive Maintenance instructions for these piston cooling oil pipe flange bolts.
As described earlier, the cause of this January 2006 event was not a lack of adequate instructions or detail, but miscommunication, a lack of verifications, and the overall structure of the procedure. As a result, a review of a sample of similar mechanical maintenance procedures was performed, and the level of verification of torquing operations was determined to be adequate to prevent an event similar to the one described in this LER. Therefore, the corrective actions taken for the 2003 event, which were due to different causes, could not have been expected to prevent this occurrence. The past occurrences do not indicate a generic or broader scope issue at the DBNPS.
Energy Industry Identification System (EIIS) codes are identified in the text as [XX].
NP-33-06-001-02� CRs 06-00154, 06-00583