IR 05000315/2008009

IR 05000315-08-009, 05000316-08-009; 09/22/2008 - 12/15/2008; D.C. Cook Nuclear Power Plant, Units 1 and 2; Special Inspection Team Review of September 20, 2008, Turbine and Fire Protection System Event

ML090260032
Person / Time
Site:	Cook
Issue date:	01/23/2009
From:	Pederson C Division Reactor Projects III
To:	Jensen J Indiana Michigan Power Co, Nuclear Generation Group
References
IR-08-009
Download: ML090260032 (47)

Text

UNITED STATES ary 23, 2009

SUBJECT:

D.C. COOK NUCLEAR POWER PLANT, UNITS 1 AND 2, NRC SPECIAL INSPECTION TEAM REPORT 05000315/2008009; 05000316/2008009

Dear Mr. Jensen:

On December 15, 2008, the U.S. Nuclear Regulatory Commission (NRC) completed a Special Inspection at your Donald C. Cook Nuclear Power Plant to evaluate the facts and circumstances surrounding the turbine high vibration and fire protection system failure on September 20, 2008.

The enclosed report documents the inspection findings, which were discussed on December 15, 2008, with Mr. L. Weber and members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed plant personnel.

Based on the risk and deterministic criteria specified in Management Directive 8.3,

"NRC Incident Investigation Program," and Inspection Procedure 71153, "Event Follow-up," and due to the equipment performance problems, which occurred, a Special Inspection was initiated in accordance with Inspection Procedure 93812, "Special Inspection." The Special Inspection evaluated the facts and circumstances surrounding the event, as well as the actions taken by your staff in response to the unexpected equipment conditions. The inspection focus areas are detailed in the Special Inspection Charter (Attachment 3).

Based on the results of this special inspection, one NRC-identified finding of very low safety significance (Green) was identified.

If you contest the finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission - Region III, 2443 Warrenville Road, Suite 210, Lisle, IL 60532-4352; the Director, Office of Enforcement, U.S. Nuclear Regulatory Commission, Washington, DC 20555-0001; and the Resident Inspector at the Donald C. Cook Nuclear Power Plant.

In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records System (PARS)

component of NRC's document system (ADAMS), accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html.

Sincerely,

/RA Gary Shear, Acting For/

Cynthia D. Pederson, Director Division of Reactor Projects Docket Nos. 50-315; 50-316 License Nos. DPR-58; DPR-74 Enclosure: Inspection Report 05000315/2008009; 05000316/2008009 w/Attachment 1: Supplemental Information w/Attachment 2: Sequence of Events w/Attachment 3: Special Inspection Team Charter cc w/encl: L. Weber, Site Vice President J. Gebbie, Plant Manager G. White, Michigan Public Service Commission Michigan Department of Environmental Quality Planning Manager, Emergency Management and Homeland Security Division, Michigan State Police Department T. Strong, State Liaison Officer

SUMMARY OF FINDINGS

IR 05000315/2008009, 05000316/2008009; 09/22/2008 - 12/15/2008; D.C. Cook Nuclear

Power Plant, Units 1 and 2; Special Inspection Team review of September 20, 2008, turbine and fire protection system event.

This report covers the inspection by the Special Inspection Team into the Cook Unit 1 Turbine and Fire Protection System events of September 20, 2008. One Green finding was identified.

The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006.

NRC-Identified

and Self-Revealed Findings

Cornerstone: Mitigating Systems

Green.

A finding of very low safety significance was identified by the team for the failure to have appropriate procedures for control room operator actions. Specifically, a control room annunciator response procedure for a fire protection alarm panel failed to provide appropriate guidance for diagnosing a fire protection system failure as evidenced by the simultaneous operation of all three fire pumps. The licensee entered the issue into their corrective action program and planned to revise the procedure.

The finding was determined to be more than minor because the failure to provide adequate procedural guidance contributed towards operators failing to recognize that a fire protection system pipe break had occurred. The issue was of very low safety significance because there was sufficient pumping capacity to maintain system pressure for a substantive period of time. (Section 4OA3.6.b(1))

REPORT DETAILS

4OA3 Special Inspection

Background - September 20, 2008, Event:

At approximately 2005 hours0.0232 days <br />0.557 hours <br />0.00332 weeks <br />7.629025e-4 months <br /> eastern daylight savings time, on September 20, 2008, the Unit 1 main turbine at the D.C. Cook Nuclear Power Plant failed. Control room turbine vibration monitors indicated severe vibrations. Control room operators tripped the reactor, entered the reactor trip response emergency operating procedure, and tripped open the main condenser vacuum breakers to stop main turbine rotation. Severe ground and building vibrations were experienced at and near the plant. Part of the main generator cooling system, which uses hydrogen, failed and hydrogen was released. As a result, a fire occurred originating from below the turbine and main generator. Several fire protection sprinkler systems automatically actuated. In addition, operators actuated a turbine water spray system to assist in controlling the fire. The on-site fire brigade responded to the fire. Off-site fire protection assistance was requested and provided assistance on-site. An estimated maximum of 41,000 gallons of water was used to control and extinguish the fire. The fire was extinguished within a half-hour.

Unknown to the control room operators and fire brigade personnel, a section of fire protection system yard loop piping had failed early in the event resulting in the loss of at least 544,000 gallons water from the north fire water storage tank. Although only one fire pump was necessary to supply flow for the fire protection systems, which had actuated, all three fire pumps (one electric and two diesel driven pumps) had started as a result of the break in order to maintain system pressure. System pressure was maintained until water in the north fire water storage tank ran out. Control room operators received a low fire protection header pressure alarm at 2125 hours0.0246 days <br />0.59 hours <br />0.00351 weeks <br />8.085625e-4 months <br /> and directed fire brigade personnel to investigate. Fire brigade personnel discovered all three fire pumps were operating without water and determined that the aligned fire water storage tank was empty. After consulting with the control room operators, fire brigade personnel shut down all three fire pumps. Both diesel driven pumps showed evidence of overheating - one of which was damaged beyond repair. Operations and fire brigade personnel did not recognize that there was a pipe break until they attempted to refill the fire protection system using the electric fire pump aligned to the south fire water storage tank. The pipe break was isolated at 2210 hours0.0256 days <br />0.614 hours <br />0.00365 weeks <br />8.40905e-4 months <br /> allowing recovery of the fire protection system.

.1 Sequence of Events (Refer to Attachment 2 for detailed sequence of events)

The following is an edited version of the Detailed Sequence of Events found as 2 at the end of this report.

D.C. Cook Turbine Generator and Fire Protection Timeline September 20 - 22, 2008 9/20/08 20:04

• Unit 2 experiences a large floor vibration, loud rumbling noise, and multiple control room alarms.

9/20/08 20:05

• Unit 1 receives numerous control room alarms on the secondary side with severe vibrations and rumbling from the turbine deck.

• Operators note all vibration points on Main Turbine supervisory panel were red which indicates high-high vibrations.

• Operators manually trip reactor and enter E-0, Reactor Trip and Safety Injection.

• Operators immediately trip open main condenser vacuum breakers to stop main turbine rotation.

• Initial reports from the field indicate that the main generator is on fire.

9/20/08 20:05 Times and Detailed Sequence Unknown

• Fire occurs in the Unit 1 main generator area.

• Eventually 40 sprinkler/sprays appear to have actuated. The licensee estimated that the maximum demand from sprinkler systems and hose streams was 975 gallons per minute (gpm); within the capacity of a single fire pump.

9/20/08 20:05

• Security Officer in Ballistic Resistant Enclosure west of turbine building, directly after the vibration and shaking stopped, observed a large amount of water coming from near the side of the turbine building.

9/20/08 20:05 Exact Times Unknown due to lack of system parameter recording/monitoring:

(Apprx)

• Fire protection header pressures drops to 140 pounds per square inch gauge (psig). The electric fire pump starts. Both control rooms receive an Electric Fire Pump Running alarm.

• Fire protection header pressure drops to 134 psig. Both control rooms receive a Fire Water System Header Pressure Low alarm.

• Fire protection header pressure drops to 130 psig. The east diesel fire pump starts. Both control rooms receive an East Diesel Fire Pump Running alarm.

• Fire protection header pressure drops to 120 psig. The west diesel fire pump starts. Both control rooms receive a West Diesel Fire Pump Running alarm.

• The Fire Brigade is notified of fire in Unit 1 by firefighters in the security office.

9/20/08 20:06

• Operations initiates process that sends out a Page to all onsite and (Apprx) offsite Fire Brigade personnel. Equipment and communications issues result in a paging error and lost phone contacts.

9/20/08 20:07

• Security Officer in Ballistic Resistant Enclosure D-6 on the West side of the Turbine Building, reported by radio, "extreme flooding" in his area to "any unit" 9/20/08 20:07

• Security Officer in Ballistic Resistant Enclosure D-6 on the West side of (Apprx) the Turbine Building reported the observed water to Secondary Alarm Station (SAS) by phone.

9/20/08 20:10

• Control room received "Fire PP House Subpanel Alarm," annunciator due (Apprx) to a "North Tank Level Low" annunciator lit on the pump house subpanel.

Alarm indicates only that the level in the tank has dropped to a volume of 568,323 to 579,705 gallons (Technical Requirements Manual (TRM)-

required volume), which may be expected upon a fire system demand.

• Procedure directs the Control Room to send an operator out to the pump house to check which Subpanel Alarm is lit. This action was placed on a

'To-Do' List and given low priority due to it being an expected alarm. No operators were sent to the pump house until after the fire pumps were secured due to lack of water.

9/20/08 20:18

• Shift Manager (SM) has declared an Unusual Event based on a fire in the Protected Area not extinguished within 15 minutes and Toxic or Flammable gas release affecting plant operation.

9/20/08 20:18

• SM requested that security perform the accountability procedure.

(Apprx) Miscommunication results in action not being accomplished.

9/20/08 20:20

• While performing procedure PMP-2080-EPP-100, Emergency Response, (Apprx) Attachment 7, Notification of On-Site Personnel, miscommunications resulted in a number of plant managers and the NRC resident inspectors not being informed of the event.

9/20/08 21:02

• During manning of the Technical Support Center (TSC); the incoming TSC Security Director was informed that the SM had requested Accountability to be performed, but that it had not yet been initiated.

9/20/08 21:20

• Fire protection header pressure again decreases to less than 134 psig due to loss of water from the north fire water tank. Both control rooms received a System Fire Header Pressure Low alarm 9/20/08 21:20

• Fire Brigade receives a call from Unit 1 that the Fire Protection (FP)system had zero water pressure.

• Fire protection supervisor entered pump house, discovered all three pumps running, and determined that there was no water going into or out of pumps based on local pressure readings. Another fire brigade member determined that the north water tank was at zero water level.

Fire protection supervisor contacted the Unit 1 control room and shut down all three pumps.

9/20/08 21:28

• Operations dispatch auxiliary equipment operators to realign pumps to the South tank.

9/20/08 21:36

• Shift Manager informs Fire Brigade personnel that the fire header is re-pressurized using the electric fire pump on the South Tank.

9/20/08 21:38

• Fire Brigade personnel report that they are still losing thousands of gallons per minute from the South fire tank and that the electric fire pump needs to be secured and isolated.

9/20/08 21:47

• Fire Brigade personnel report collapse of asphalt and large amount of water coming up on the west side of the plant 9/20/08 22:10

• Operations isolates FP header leak by closing FP-161, FP-101, and FP-107 9/20/08 22:23

• All personnel are accounted for (Accountability complete)9/20/08 23:47

• Completed NRC event notification (EN) #44507.

9/21/08 1:30

• Fire Header Pressure is at 140 psig and stable. Teams dispatched to fire protection pump house to place the electric fire pump in service (full starting party of Operations, Maintenance, and Engineering.)

9/21/08 1:55

• Fire Protection header is 154 psi from the pegging pump.

• Operators start the electric fire pump to demonstrate availability. Fire Protection header pressure rose to 170 psi after electric fire pump was started.

• The starting party concurs that pump operation is acceptable.

9/21/08 2:29

• The South Fire Water Storage Tank (FWST) level is going down at

~1000gpm.

9/21/08 2:50

• The licensee determines that the 1000 gpm "leak" from the South tank was due to the recirculation line up going to the North FWST. System re-aligned.

9/21/08 13:41

• Unit 1 has entered mode 4.

9/22/08 3:27

• Unit 1 has entered mode 5.

9/22/08 9:40

• The North Fire Protection Water Storage Tank is at 597,200 gallons and Operable.

• Exit TRM 8.7.5 Condition B

.2 Licensees Event Response Following the Reactor Trip and Main Turbine Generator Fire

a. Inspection Scope

The team evaluated the licensee's initial response to the event, focusing on event classification and notifications. This evaluation included a review of the control room operators' use of emergency operating procedures and Emergency Plan implementing procedures, and identification of degraded plant conditions. The team interviewed plant personnel and reviewed applicable portions of the Emergency Plan and Emergency Plan implementing procedures, emergency operating procedures, emergency response organization logs, control room logs, plant process computer data, and corrective action program documents.

b. Findings and Observations

Discussion On September 20, 2008, at 2005 hours0.0232 days <br />0.557 hours <br />0.00332 weeks <br />7.629025e-4 months <br />, the Unit 1 main turbine generator experienced high vibrations that caused plant operators to manually trip the reactor and break main condenser vacuum to quickly stop the turbine from rotating.

The plant response to the reactor trip was as expected with safety-related systems operating as designed. Control room operators effectively controlled and stabilized plant parameters following the reactor trip. Refer to Section 4OA3.6 of this report for a more detailed discussion of operator performance during the event. The turbine driven auxiliary feedwater pump started automatically due to low-low levels in at least two steam generators; an expected result for a reactor trip from full power. The licensee made the appropriate 4-hour notification required by 10 CFR 50.72(b)(2)(iv)(B) due to the reactor protection system actuation and 8-hour notification required by 10 CFR 50.72(b)(3)(iv)(A) due to the auxiliary feedwater system actuation.

A hydrogen fire resulted within the first few minutes of the event when hydrogen apparently escaped from one of the generator bushings located below the main generator. Portions of the turbine generator fire suppression system automatically actuated and control room operators manually actuated the main turbine lube oil fire water spray portion of the system. The plant fire brigade responded. The local fire department was also called and responded to the site. The fire was extinguished by the fire suppression systems and plant fire brigade around 2030 hours0.0235 days <br />0.564 hours <br />0.00336 weeks <br />7.72415e-4 months <br />.

At 2018 hours0.0234 days <br />0.561 hours <br />0.00334 weeks <br />7.67849e-4 months <br />, the licensee declared a Notification of Unusual Event (NOUE), the lowest of four emergency classifications, for a fire in the plant protected area lasting more than 15 minutes and a flammable gas release that affected plant operation. The licensee made the appropriate notifications to the State of Michigan, local agencies, and the NRC Headquarters Operations Center as required by 10 CFR 50.72(a)(3). However, notification calls to senior plant management and to the NRC resident inspectors following declaration of the NOUE were not made.

The team had three observations based on its review of the licensee's response to the event:

1) During review of the emergency classification and event notifications, the team noted that in addition to the two emergency action levels identified by the licensee when the NOUE was declared (i.e., H-4, fire in the plant protected area lasting more than 15 minutes and H-5, flammable gas release that affected plant operation),emergency action level N-5, main turbine failure, was also applicable.

2) Based on interviews, the team determined that the shift manager had requested plant security to implement an accountability of all plant personnel on site shortly after he declared the NOUE. However, there was apparently a breakdown in communication between operations and plant security personnel and the accountability process was not implemented. When the Technical Support Center (TSC) was activated, the TSC Security Director learned that the accountability process had not been implemented and he directed it to be completed.

The accountability process was implemented at 2102 hours0.0243 days <br />0.584 hours <br />0.00348 weeks <br />7.99811e-4 months <br /> and was completed at 2223 hours0.0257 days <br />0.618 hours <br />0.00368 weeks <br />8.458515e-4 months <br />. Although there was no completion time specified for accountability of plant personnel in the licensees Emergency Plan implementing procedure, 1 hour1.157407e-5 days <br />2.777778e-4 hours <br />1.653439e-6 weeks <br />3.805e-7 months <br /> 22 minutes did not meet licensee managements expectation for timeliness.

3) The main generator tripped automatically immediately after the main turbine trip due to actuation of the generator overall differential auxiliary relays. Following a normal reactor trip and main turbine trip, there is a 30-second time delay before the main generator trips. The licensee was investigating the cause for the immediate main generator trip at the conclusion of this inspection. Although it was abnormal, the early main generator trip had no adverse effect on the plant or on the operators response to the event.

No findings of significance were identified.

.3 Impacts on Unit 1

a. Inspection Scope

The team reviewed the licensees plans and actions for assessing the impact that the event had on Unit 1 systems. The review included discussions with various site engineers, review of action plans for evaluating the condition of systems and components in the Unit 1 turbine building, review of in-process results from the licensees reviews, and methods for tracking identified deficiencies. In addition, in a effort to quantify the magnitude of the vibration, the team discussed the response of the sites seismic monitoring equipment with the licensee.

b. Findings and Observations

The licensees action plan was broken down by disciplines, e.g., structural integrity (civil engineering), piping integrity, and electrical. The licensee divided the turbine building into areas using the pre-existing structural column designation. Each area was reviewed using a team approach by the licensee. The licensee used its corrective action program to track identified deficiencies. The licensee had not completed all reviews at the time this inspection was completed. The team concluded that the licensees actions to assess the impact of the event on Unit 1 were appropriate.

The operating basis earthquake for D.C. Cook is 0.10g acting coincidentally with a maximum vertical ground acceleration of 0.067g. Likewise, the safe shut down earthquake is a horizontal ground acceleration of 0.20g coincident with a vertical ground acceleration of 0.134g. The licensee has two types of monitoring equipment. One is a recorder with a trip set point to turn the unit on. The other seismic monitoring equipment are peak recording accelerographs (tape plates/scratch gauges), which are passive devices without set points. None of the seismic monitoring devices are located in the turbine building. The set points for starting the recording devices are 0.02g in either the horizontal or vertical direction. The licensee indicated that the recording devices did not start during the September 20, 2008 event, indicating that the vibrations did not exceed the set point of 0.02g and therefore never approached either the operational or safe shutdown seismic levels.

No findings of significance were identified.

.4 Licensees Evaluation of Event

a. Inspection Scope

To assess the licensees evaluation of the events of September 20, 2008, the team reviewed the Action Request (AR) documents listed at the end of the report and held discussions with licensee personnel. In addition, the team reviewed the root cause evaluation associated with the failed fire protection system piping and the evaluation of the various communications issues that arose during the event.

b. Findings and Observations

1) Root Cause Evaluation:

The team observed the licensees Corrective Action Review Board (CARB)meeting addressing the root cause evaluation of the fire protection system issues. Following CARBs acceptance of the root cause analysis, with comments, the team provided the following feedback based on its assessment of the root cause analysis.

a) The team concurred with the CARBs assessment that the root cause team had identified the primary mechanisms that led to the fire protection line failure.

b) Some statements in the root cause analysis were not supported by factual information. For example, the root cause analysis stated that the unknown 8-inch piping running above the fire protection piping placed a significant bending moment on the failed coupling. However, there was no physical evidence nor discussion presented in the report to substantiate the bending moment.

c) The licensees extent of condition review for piping interference was narrowly focused by considering only potential interferences impacting fire protection lines. The extent of condition did not consider the population of potential interferences of non-fire protection piping.

d) While issues associated with the failed fire protection piping, e.g.,

procedural deficiencies and performance issues, were included in the original scope of the root cause analysis, they were not evaluated in the analysis. All of the performance and procedural issues were moved out of the root cause analysis to an apparent cause evaluation.

2) Investigative Techniques:

The team followed the licensees activities associated with initial inspection of the failed fire protection system coupling, excavation of the washed-out area, removal of the coupling and attached piping, and failure analysis of the broken coupling. Based on its reviews, the team had the following observations.

a) The licensee took a diligent and cautious approach to the initial inspection of the failed coupling and excavation of the washed-out area to allow access to and removal of the failed fire protection piping. The licensees approach was directed towards personnel safety and preserving the as-found condition of the failed coupling.

b) Prior to removing the failed section of fire protection system piping from the washed-out area, the licensee did not take as-found measurements to locate either the fire protection piping or the unknown 8-inch pipe relative to a fixed point. While this omission may not have changed the outcome of the root cause analysis, such measurements may have provided additional information relative to the assumed interference between the fire protection pipe and the unknown 8-inch pipe described in the root cause analysis.

c) The licensee had not considered interviewing the security officer who was in the proximity of the ruptured fire protection line until the team asked if they had talked to the individual. The officer provided critical information relative to when the line ruptured, which was otherwise not available because of a lack of fire protection system parameter monitoring.

3) Main Turbine Failure Root Cause

Analysis:

The root cause analysis for the turbine issues was not complete at the end of the SIT inspection. The inspectors followed the licensee's activities associated with the initial inspection of the turbine components and their plans for vendor evaluation and repair of damaged components. The licensee took a diligent and cautious approach, focused on personnel safety and preserving the as-found condition of the damaged turbine casings, rotors and bearings. Evaluation of the licensee's completed root cause analysis will occur under the normal baseline inspection program.

No findings of significance were identified.

.5 Significance of the Fire Protection System Malfunctions

a. Inspection Scope

The team evaluated the significance of the fire protection system malfunctions. The evaluation included performing walk-downs of the plant, review of engineering analyses, review of operator procedures, review of operator logs, and interviews of engineering and fire brigade personnel.

b. Findings and Observations

The fire associated with the turbine event was controlled by installed fire suppression systems. The fire in the Unit 1 main generator area was extinguished within a half-hour.

Although the pipe break in the yard loop piping occurred early in the event, all three fire protection pumps operated to maintain water pressure for the fire protection system until the water supply ran out. Fire protection water header pressure was maintained until 75 minutes after the turbine failure occurred. However, even if the fire protection system had failed completely initially, the team concluded that the fire and associated damage would have been limited to the Unit 1 turbine building area. The teams conclusion was based on 1) Unit 2 and nearby safety-related fire areas were separated from the Unit 1 turbine building area with rated three-hour fire barriers; and 2) any significant quantities of oil released would have collected in the Unit 1 turbine building basement, away from other fire areas, due to floor openings.

The pipe break in the fire protection system resulted in a temporary loss of water-based fire protection capability. Specifically, water-based fire protection capability was lost for approximately 4-1/2 hours until the fire protection yard loop piping break had been isolated and fire protection water system pressure had been restored. During the majority of this time, fire trucks and fire protection personnel from Lake Township were on-site. The west diesel driven fire pump and the electric driven fire pump were formally declared operable approximately 15 and 17 hours1.967593e-4 days <br />0.00472 hours <br />2.810847e-5 weeks <br />6.4685e-6 months <br /> later, respectively. The east diesel driven fire pump was damaged due to the lack of water supply and resultant overheating. Although the east diesel driven fire pump remained unavailable, 200 percent fire water supply capacity existed for TRM-required water-based fire protection systems. Only one fire pump was necessary to provide supply for TRM-required water-based fire protection systems. In addition, the team noted that a number of safety-related areas, such as the control room cable vaults and safety-related switchgear rooms, relied upon carbon dioxide suppression systems instead of water-based systems for fire suppression. Based on the above information, the team concluded that the actual safety significance associated with the fire protection piping break was very low.

The licensee performed calculations for the risk significance associated with the turbine failure event and determined that the conditional core damage probability associated with the event was less than 1 x 10-6, which was of very low safety significance. The licensees analysis was based on the loss of heat removal capability from the Unit 1 main condenser, loss of Unit 1 main feedwater, and loss of the Unit 1 plant air compressor - all of which had been impacted by the event. The team reviewed the licensees draft calculation (documented by, PRA-STUDY-053, Post 9-20-08 Main Turbine Failure Risk Assessment, dated September 2008) and did not identify any concerns.

.6 Operator Performance in Response to the Event

a. Inspection Scope

The team evaluated operator performance in response to the event, including awareness of and response to plant conditions, shutdown activities, response to fires, and fire protection system status. This evaluation included a review of the control room operators' use of emergency operating procedures, identification of degraded plant conditions, initial actions to mitigate the event, and actions to restore the fire protection water system following draining of the in-service north fire water storage tank. The team interviewed plant personnel and reviewed applicable portions of the Technical Requirements Manual (TRM), emergency operating procedures, annunciator response procedures, plant drawings, control room logs, plant process computer data, and corrective action program documents.

b. Findings and Observations

Discussion On September 20, 2008, at 2005 hours0.0232 days <br />0.557 hours <br />0.00332 weeks <br />7.629025e-4 months <br />, very high vibrations of the Unit 1 main turbine generator caused plant operators to manually trip the reactor and break main condenser vacuum to quickly stop the turbine.

Control room operators effectively controlled and stabilized plant parameters following the reactor trip. The plant response to the reactor trip was as expected with all safety-related systems operating as designed, and no unexpected safety related system actuations identified. By reviewing the plant process computer alarm printout and control room logs, the team noted that operators promptly assessed the abnormal condition and manually tripped the reactor. Excessive noise, building vibration, and multiple alarms, alerted operators to the impending main turbine failure. Breaking main condenser vacuum resulted in the loss of the ability to use the main condenser steam dump valves. Also, in response to excessive plant cooldown following the reactor trip, operators closed the main steam isolation valves. Operators attributed the excessive plant cooldown primarily to the inability to close the bypass header steam lead drain valves (1-DRV-403 and 1-DRV-404). A 250 Volt direct current control power circuit,1-CRCD-7, tripped during the event. This prevented isolation of numerous extraction steam lead drains due to the loss of control power. Decay heat removal was achieved using steam generator atmospheric relief valves.

A hydrogen fire resulted within the first few minutes of the event when a seal on one of the main generator bushings failed and hydrogen was released. Portions of the turbine generator fire suppression system automatically actuated and control room operators manually actuated the main turbine lube oil fire water spray portion of the system. The plant fire brigade responded. The local fire department was also called and responded to the site. The fire was extinguished by the fire suppression systems and plant fire brigade around 2030 hours0.0235 days <br />0.564 hours <br />0.00336 weeks <br />7.72415e-4 months <br />.

While the operators took prompt effective actions to place the Unit 1 reactor in a safe shutdown condition, none of the on-shift operators recognized the significance of all three fire pumps running because of procedural and training deficiencies. Specifically the operators did not recognize that there were no fire scenarios which would require more than two fire pumps to be running and therefore didnt recognize that three pumps running was indicative of a system failure. See item 1) below for additional details.

At approximately 2125 hours0.0246 days <br />0.59 hours <br />0.00351 weeks <br />8.085625e-4 months <br />, a fire system low pressure alarm alerted operators to a problem with the system. Operators stopped the sites three fire pumps. Unbeknownst to the operators, the site fire protection header had ruptured and one of the sites two nominal 585,000 gallon fire protection storage tanks had been pumped dry. By 2309 hours0.0267 days <br />0.641 hours <br />0.00382 weeks <br />8.785745e-4 months <br />, operators had isolated the rupture and established alternate fire water supplies.

All three fire pumps overheated because they ran dry for some time. The two diesel engines are supplied cooling by the fire water supply. By approximately 0200 hours0.00231 days <br />0.0556 hours <br />3.306878e-4 weeks <br />7.61e-5 months <br /> on September 21, the motor-driven fire pump was restored to an operable status. One of the two diesel-driven fire pumps was restored to operable status by mid-day on September 21, 2008. The other diesel driven fire pump was damaged and had to be replaced. Refer to Section 4OA.3.8 of this inspection report for a more detailed discussion of the fire protection system issues revealed by this event.

The licensee inspected Unit 2 to ascertain whether there was any damage as a result of the vibrations associated with the Unit 1 main turbine failure. No significant damage was identified affecting the operation of Unit 2.

The team identified one finding and three observations based on its review of the licensee's response to the event:

1) Failure to Provide Adequate Operator Response Procedures

Introduction:

A finding of very low safety significance was indentified by the team for the failure to have appropriate procedures for control room operator actions. Specifically, a control room annunciator response procedure for a fire protection alarm panel failed to provide appropriate guidance for diagnosing a fire protection system pipe failure. The licensee entered the issue into their corrective action program and planned to revise the procedure.

Description:

Procedure 1-OHP-4024-101, Annunciator #101 Response: Plant Fire System, revision 19, provided guidance to the operators for responding to alarms (referred to as drops) on the number 101 annunciator panel in the Unit 1 control room.

The drop 11 section of procedure 1-OHP-4024-101 provided guidance for responding to the Water Fire System Pressure Low annunciator. The drop 11 section identified operation of a water-based fire suppression system and a break in fire protection system water piping as the two probable causes for the annunciator. The drop 11 section directed operators to check for other fire system panel annunciators, locate reason for water demand, and stop fire pumps when water was no longer required. The drop 11 section further directed operators to ensure all fire pumps were running and ensure that there was sufficient level in the water tank lined up to the fire pumps.

The team noted that for the majority of potential water demands on the fire protection system, one fire pump was sufficient. Only one scenario involving the Unit 1 main transformer deluge system required two fire pumps. The procedure directive to ensure all fire pumps were running, in conjunction with operators understanding of system operation, led operators to believe that having all three fire pumps operate in response to a fire protection system demand was an expected response. This belief masked the fact that a significant fire protection system pipe break had occurred during the September 20, 2008 event. Control room operators assigned a low priority to sending an operator to the fire pump house, in part, due to this belief. Although the drop 11 section identified a break in fire protection system water piping as one of the probable causes for low fire header pressure, the procedure did not provide any guidance for diagnosing whether a break in the fire protection water system had occurred.

Annunciator drops 34, 44, and 54 of procedure 1 OHP-4024-101 provided guidance for responding to the electric fire pump running and the East and West Diesel fire pump running annunciators. Drops 44 and 54 responses also referenced procedure 12-OHP-4021-066-00, Fire Protection System (Water) Operation, revision 19, for assessing proper diesel operation. The guidance for all three drops was essentially the same; dispatch an operator to the fire pump house to monitor operation of the pumps and drivers. The team noted that while the individual drop sections and referenced procedures provided guidance on assessing operating parameters of the electric and diesel pump drivers, none of the procedure sections addressed operation of the fire protection system nor did any of the procedure sections specify monitoring the water tank level to ensure an adequate water supply to the pumps. The team noted that there is no automatic exchange of water supply from the one tank to the other under any circumstances.

License condition 2.C(4) for the Unit 1 operating license required the licensee to implement and maintain in effect, all provisions of the approved fire protection program as described in the Updated Safety Analysis Report (USAR). Section 9.8.1 of the USAR stated that the Fire Protection Program Manual comprised part of the approved fire protection program. Section 2.4 of the Fire Protection Program Manual stated that the fire protection quality assurance program was specified in the Quality Assurance Program Description (QAPD). Section 7.a of the QAPD stated that the licensee complied with quality assurance guidance documents listed in Table 1, Regulatory and Safety Guides/ANSI Standards, of the QAPD. Item 23 of the QAPD Table 1 listed the June 14, 1977, Nuclear Plant Fire Protection Functional Responsibilities, Administrative Controls, and Quality Assurance (FRACQA), as one of the standards. The FRACQA required that fire fighting procedures be established for actions to be taken by control room operators upon receipt of alarms on control room annunciator panels. Based on the above requirements, the team determined that the annunciator procedures for the fire alarm panels and procedures referenced by annunciator procedures for the fire alarm panels were required, by license condition, to be established and maintained. The team considered procedures 1-OHP-4024-101 and 12-OHP-4021-066-001 to not be appropriate to the circumstances in that 1) procedural guidance to ensure all fire pumps were running contributed towards operators believing that having all fire pumps running was a normal part of system operation, 2) guidance to diagnose fire protection system pipe breaks was lacking, and 3) guidance to monitor fire water tanks levels to ensure adequate water supply was also lacking.

Analysis:

The team determined that the failure to have procedures appropriate to the circumstances for control room operator actions upon receipt of alarms on control room annunciator panels was a performance deficiency. The finding was determined to be more than minor because the failure to have procedures appropriate to the circumstances for control room operator actions upon receipt of alarms on control room annunciator panels was associated with the Mitigating Systems cornerstone attribute Procedure Quality and affected the cornerstone objective of ensuring the reliability and availability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to provide adequate procedural guidance contributed towards operators, in response to the September 20, 2008, event, failing to recognize that operation of all three fire pumps was abnormal and that a fire protection system pipe break had occurred.

In accordance with IMC 0609, Significance Determination Process, 0609.04, Phase 1 - Initial Screening and Characterization of Findings, Table 3b the team determined the finding degraded the fire protection defense-in-depth strategies. Therefore, screening under IMC 0609, Appendix F, Fire Protection Significance Determination Process, was required. The team noted that the finding affected the availability and reliability of water-based suppression systems and water-based fire brigade activities. IMC 0609, Appendix F, does not effectively address findings involving potential loss of fire protection water supply nor does it address findings that affect performance of the fire brigade. As such, NRC management review using IMC 0609, Appendix M, Significance Determination Process Using Qualitative Criteria, was required. The September 20, 2008 event approximated a worst case scenario in that a large (i.e., 12 inch) fire protection system piping break had occurred coincident with significant water-based fire suppression system demands. The September 20, 2008, event demonstrated that there was sufficient pumping capacity to maintain fire header pressure in excess of one hour which permitted substantive time for a fire to be controlled and for off-site fire protection resources to be made available thereby limiting the significance of the procedural inadequacies. Based on the above qualitative analysis, the team, with the concurrence of NRC management, determined that the finding was of very low safety significance (Green).

This finding has no cross-cutting aspect associated with it because the time frame in which the procedure should have been modified was approximately 13 years ago and no opportunities to identify the discrepancy were identified between then and now.

Enforcement:

As of September 20, 2008, procedures 1-OHP-4024-101 and 12OHP-4021-066-001 were not appropriate to the circumstances in that 1) procedural guidance to ensure all fire pumps were running contributed towards operators believing that having all fire pumps running was a normal part of system operation, 2) guidance to diagnose fire protection system pipe breaks was lacking, and 3) guidance to monitor fire water tanks levels to ensure adequate water supply was also lacking. Although these procedures were required to be established and maintained as part of the fire protection program, the specific requirement (10 CFR Part 50, Appendix B, Criterion V, Procedures) to have procedures appropriate to the circumstances was not applicable because the fire protection system was a non-safety related system. Therefore, no violation of regulatory requirements occurred. This issue was considered a finding of very low safety significance. The licensee entered this issue into their corrective action program as AR Number 00839618, Fire Header Low Pressure Annunciator Response, 20081002 and planned to revise the procedure. (FIN 05000315/2008009-01; 05000316/2008009-01)2) Operator Delay in Reducing Auxiliary Feedwater System Flow During review of plant data following the event, the team noted that the excessive plant cooldown was also attributed, in part, to delay in reducing auxiliary feedwater system flow to the steam generators and delay in transferring auxiliary steam loads from Unit 1 to Unit 2. Normally, operators should not have to close main steam isolation valves to address excessive plant cooldown following a reactor trip. Considering the challenges presented to plant operators during this event, the above delays were understandable.

The team concluded that operators appropriately prioritized response actions during the early stages of the event and that the delay did not represent a performance deficiency.

Closing main steam isolation valves had no additional plant impact because the main condenser, the normal plant heat sink, was already lost since operators had broken the main condenser vacuum to stop the main turbine from rotating.

3) Failure to Communicate Pipe Break to Control Room Within the first several minutes of the event, a security officer identified a large flow of water coming up from the ground just outside the west side of the turbine building indicating a rupture of a subterranean pipe. The officer first attempted to contact any security office by radio. When this failed he contacted the secondary alarm station by telephone. This significant information was never communicated to the control room personnel.

4) Failure to Follow Procedure for Switching Tanks for Fire Protection System

Description:

As part of the restoration of the fire protection system, following draining of the north water tank, operators switched the fire water suction source to the south water tank. The pressure relief path for the fire pumps was back to the fire water tanks and was normally aligned to the fire water tank from which suction was being drawn. The pressure relief path alignment was performed by locally operating manual valves at the fire water tanks. While operations personnel successfully switched the suction source for the fire pumps to the south fire water tank, they did not switch the pressure relief flow path from the north tank to the south tank. After the electric fire pump was started, fire brigade personnel noted that level was dropping in the south fire water tank and that the level was increasing in the north fire water tank. Operations personnel realized that the recirculation flowpaths had not been realigned and corrected the recirculation flowpath valve alignment.

16, Switching from North Fire Protection Water Storage Tank to South Fire Protection Water Storage Tank, of procedure 12-OHP-4021-066-001 provided procedural guidance for switching the fire water to the south tank. The precautions and limitations section of the procedure attachment specifically discussed the potential for transferring a large volume of water from one storage tank to the other due to the recirculation flowpath. The procedure attachment provided specific steps for opening the recirculation flowpath to the south fire water tank and isolating the recirculation flowpath to the north fire water tank. The procedure attachment also required independent verification of these steps. The failure to correctly implement procedure 12-OHP-4021-066-001 was a performance deficiency. However, the failure to correctly implement the procedure was quickly recognized and corrected. In addition, the failure did not result in any loss of fire protection water supply. As such, the team concluded that the issue was of minor safety significance and did not warrant a finding as described in IMC 0612, Power Reactor Inspection Reports, issued December 4, 2008.

.7 Equipment Quarantine

a. Inspection Scope

The team observed the licensee activities to preserve information, i.e., quarantine activities, relative to the failed fire protection piping coupling during initial inspection activities, excavation activities, and extraction activities of the failed piping from the wash-out area.

b. Findings and Observations

The team concluded that the licensee appropriately balanced efforts to preserve physical information with personnel safety in handling the broken fire protection system coupling.

The broken coupling was located under a large concrete pad that was resting against the turbine building wall on one side and sandy soil on the other. Observation of the coupling was limited to what could be seen without going under the concrete pad. The licensee used appropriate precautions in lifting the pad to allow access to the coupling.

Following removal of the pad, the licensee used a vacuum truck to remove the loose sandy soil from the wash-out area using supports to ensure the 8 inch unknown pipe did not shift during excavation. The area was designated as a confined access area for personnel safety and controlling access to the area.

The team did note, as documented in Section 4.b.2.b) of this report, that the licensee did not take as-found measurements/dimensions to accurately locate the various pipes with respect to each other.

No findings of significance were identified.

.8 Fire Protection System Performance

a. Inspection Scope

The team reviewed the fire protection system design and the systems performance during the September 20, 2008, event to assess whether the system operated as designed. In reviewing the fire protection system design, the team reviewed system drawings (piping and electrical) and system descriptions and held discussions with fire brigade members and system engineers.

b. Findings and Observations

The team concluded that the fire protection system functioned as designed in response to the fires associated with the turbine event and the ruptured 12 inch fire protection piping. The team did not identify deficiencies related to the design of the fire protection piping section that failed. The design and installation of the failed piping section was consistent with specifications of the National Fire Protection Association (NFPA) 24, Outside Protection that were in effect at the time of construction. While the team concluded that the system was adequately designed, installed, and functioned as designed, the team had a number of observations.

1) Lack of Low-Low Fire Water Tank Alarm, Pump Trip

Description:

Each fire water tank had a low level alarm set point of 568,323 to 579,705 gallons. The set points were established to ensure that 565,000 gallons of usable water was maintained in each of the fire water tanks as required by the Technical Requirements Manual. The alarms provided annunciation in the fire pump house which, in turn, provided a general Fire Pump House Subpanel Alarm alarm in the control room. For any significant water demand upon the fire protection system, a low level alarm for the tank in use would be expected. As such, the low level alarms did not provide useful information to operators that there was an unexpected condition or that there was insufficient water to maintain pump net positive suction head (NPSH). The systems design did not provide an alarm to notify operators of a low water level condition which could affect operability of the pump. The team did not identify a performance deficiency relating to this observation because there are no regulatory requirements for such an alarm.

In addition, the fire pumps did not have a low suction pressure pump trip to ensure that the pumps would not be damaged due to low water level conditions. The team noted that NFPA 20-1980, Standard for Installation of Centrifugal Fire Pumps, did not explicitly address automatic shutdown of fire pumps due to low suction pressure. The only automatic pump shutdown discussed by NFPA 20-1980, as not being bypassed when a starting cause existed, was engine emergency overspeed. As such, the NFPA codes appeared to prohibit shutdown of fire pumps for reasons other than local manual shutdown. The licensee did have the option to deviate from the NFPA code in this regard, install a pump trip upon low suction pressure, and document the deviation.

Given the limited nature of the water supply, an evaluation of this condition would have been appropriate. The damage to a fire protection pump during this event would likely have been avoided if such a trip had been installed. The team did not identify a performance deficiency relating to this observation because there are no regulatory requirements for such a pump trip.

2) Limited Fire Protection System Information Available in Control Room Information available to control room operators concerning the operational status of the fire protection system was limited. For the water-based suppression systems, the fire protection alarm panel in the control room was generally limited to providing information that the system had actuated, which fire pumps were running, and if there was an alarm relating to individual fire pump controllers. Additionally, the control room alarm panel provided fire water header pressure indication, a low fire water header pressure alarm, and a fire pump house subpanel alarm. However, the low water tank level alarms, along with tank level indications, were only located in the fire pump house, which was not routinely, manned. A low water tank level alarm coming in at the fire pump house subpanel would have only resulted in the fire pump house subpanel alarm annunciating in the control room with no information concerning the cause of the alarm. As such, control room operators did not have any readily available information concerning water tank levels for the fire protection system. The team did not identify a performance deficiency relating to this observation because there are no regulatory requirements for such control room indications.

3) Lack of Permanent Recording of Fire Protection Alarms

Description:

The licensees fire alarm signaling system lacked the capability to permanently record fire protection system alarms, which came in. In addition, the controllers for the fire protection pumps lacked the capability to permanently record fire header pressures. The team noted that the lack of a permanent recording capability hindered the evaluation of the event. Specifically, timing information regarding the actuation of the fire pumps and turbine building suppression systems would have been helpful for verifying that the fire pumps had started sequentially (as required by NFPA 20) and did not cause a pressure surge contributing to the break. Pump sequencing was subsequently checked by verifying calibration of the pressure switches and timing devices for automatic pump starts. In addition, having recorded pressure information would have been helpful for determining that water hammers caused by actuation of fire suppression systems did not contribute to the break. The licensee subsequently performed engineering analyses to show that such water hammers could not have resulted in sufficiently high pressure transients at the break location to contribute to the break. The team noted that the NRC has not required that licensees maintain a permanent recording capability and that the lack of such capability would not directly impact fire protection capability. As such, the team concluded that the lack of such permanent recording capability at the D.C. Cook Nuclear Plant did not represent a performance deficiency.

4) System Design Pressure During review of the fire protection system piping failure, the team questioned the design operating pressure of the fire protection system. The piping specification documents for the underground fire protection piping indicated that the design operating pressure of the system was 150 pounds per square inch gauge (psig). The team determined that the system normal operating pressure around 170 psig, with pressure as high as 175 psig being normal during pump operation.

The licensee reviewed the design operating pressure of each component in the fire protection system and concluded that all components were rated at greater than or equal to 175 psig. The licensee then developed a white paper assessing the condition and concluded, based on a review of individual components and piping design pressures, that the design pressure for the system should be 175 psig rather than the previously stated 150 psig.

The team concluded that the licensees corrective actions, documented in AR Number 00840308, Fire Water System Design/Operating Pressures, were reasonable and appropriate. The team determined that the discrepancy between the specified design pressure and operating pressure was of minor significance and did not warrant a documented finding as described in IMC 0612.

No significant findings were identified.

.9 Licensees Corrective Actions to Restore Fire Protection System Operability

a. Inspection Scope

The team reviewed the actions taken by the licensee to compensate for the loss of the fire protection water system to ensure the continued acceptability of the system to support operation of Unit 2. The review included discussions with the fire protection system engineer and fire brigade members and evaluations of the compensatory measures in place.

b. Findings and Observations

The team concluded that the actions taken to restore the system to an operable condition and the compensatory actions taken to address the failed section of the fire protection system were adequate to support continued operation of Unit 2.

1) Lack of Fill and Vent Procedure After the fire protection system piping rupture was located and isolated, the licensee refilled the water system piping using water from the south fire water tank. Because the licensee did not have a procedure providing guidance for filling and venting the system, fire brigade personnel relied on historical knowledge to initially vent the system on September 21, 2008, to ensure that the system did not have significant pockets of air trapped. The brigade members historical knowledge dated back to the early 1990s when the system took a suction from the lake (instead of water storage tanks) and the system had to be treated periodically to control marine growth in the piping. Although the historical knowledge included some of the high point vents, not all of the numerous high point and isolated piping vents were included.

When the team brought this issue to the licensees attention, the licensee developed a work instruction which accounted for all of the high point and isolated piping vents.

The team reviewed the work instruction, on a sample basis, and verified that necessary high point and isolated piping vents were included in the instruction. The licensee re-performed venting of the fire protection system based on the developed work instruction. The team witnessed portions of the venting and noted that although small amounts of air were entrained in the system, no significant amounts of trapped air were identified. In addition, the licensee entered the issue into their corrective action program (AR Number 00840233, Venting of Fire Protection System, lacks guidance) to develop a formal procedure for filling and venting the fire protection system. The failure to have a procedure for a fire protection maintenance activity such as filling and venting the system was a performance deficiency. However, because part of the fire protection system would essentially be vented to air when in use, the team concluded that the presence of limited quantities of air in the fire protection system downstream of the pumps would not present an operability concern. As such, the team determined that the issue was of minor safety significance and did not warrant a documented finding as described in IMC 0612.

2) Off-site Fire Protection Assistance Shortly after the turbine event occurred, the licensee requested assistance from the off-site fire department (Lake Township). Fire trucks from off-site arrived within 15 minutes of assistance being requested. Lake Township firefighters assisted plant fire brigade personnel with fire fighting activities. Lake Township fire trucks remained on-site for the majority of time that the fire protection system water header was depressurized due to the break. The team did not identify any concerns with respect to assistance provided by the Lake Township Fire Department nor with the licensees procedures for bringing off-site fire protection assistance on-site.

3) Compensatory Actions for Loss of Hydrant In response to the fire protection system pipe break, the licensee isolated a portion of the fire protection system yard loop piping to allow restoration of the rest of the yard loop piping. The isolation resulted in one outside hydrant not being operational. The licensee promptly established a back-up hose line from a nearby outside hydrant as a compensatory measure consistent with Technical Requirements Manual (TRM)requirements. The team inspected the back-up hose line and did not identify any concerns.

4) Temporary Lines from City Water System Established In response to the fire protection system pipe break and resultant loss of fire protection system pumping capability, the licensee established a back-up water supply within approximately 90 minutes after loss of fire header pressure from a municipal fire hydrant located outside the owner controlled area. Although the connection had considerably less water flow capability than a single fire pump, the connection satisfied TRM requirements for a back-up water supply. The team inspected the back-up water supply connection and did not have any concerns.

4OA6 Management Meetings

.1 Exit Meeting Summary

On December 15, 2008, the Team Lead presented the inspection results to Mr. L. Weber and other members of the licensee staff. The licensee acknowledged the issues presented. The team confirmed that none of the potential report input discussed was considered proprietary.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee

D. Baker, Mechanical/Civil/HVAC/EFIN Supervisor

R. Crane, Learning Organization Manager

J. Gebbie, Plant Manager

R. Gray, Fire Protection Program Manager

J. Jensen, Site Support Services Vice President

K. Horsh, Fire Protection System Manager

C. Hutchinson, Emergency Preparedness Manager

E. Lee, Fire Protection Supervisor

S. Lies, Plant Engineering Director

P. Mangan, Design & Modifications Manager

D. MacDougall, Fire Protection Engineer

R. Niedzielski, Regulatory Affairs Specialist

R. Peltz, Fire Protection Compliance Coordinator

J. Petro, Senior Counsel

P. Schoepf, Regulatory Compliance Manager

H. Torberg, Site Protection Manager

L. Weber, Site Vice President

J. Zwolinski, Regulatory Affairs Manager

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened/Closed

05000315/2008009-01 FIN Failure to Provide Adequate Operator Response Procedures for Fire Protection System Operation Section 6.b.1)

Attachment 1