IR 05000397/2009010

IR 05000397-09-010 on 08/17/09 - 10/20/09 for Columbia Generating Station

ML093280158
Person / Time
Site:	Columbia
Issue date:	11/23/2009
From:	Chamberlain D NRC/RGN-IV/DRP
To:	Parrish J Energy Northwest
References
IR-09-010
Download: ML093280158 (28)

Text

UNITED STATES NUC LE AR RE G UL AT O RY C O M M I S S I O N R E GI ON I V 612 EAST LAMAR BLVD , SU I TE 400 AR LI N GTON , TEXAS 76011-4125 November 23, 2009 Mr. J. Chief Executive Officer Energy Northwest P.O. Box 968, Mail Drop 1023 Richland, WA 99352-0968 Subject: COLUMBIA GENERATING STATION - NRC SPECIAL INSPECTION REPORT 05000397/2009010

Dear Mr. Parrish:

On October 20, 2009, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Columbia Generating Station to evaluate the facts surrounding a fire in the turbine building that occurred on the nonsafety-related 6900 Vac electrical bus between the normal transformer and Breakers SH-5 and SH-6, that resulted in an automatic turbine trip and subsequent reactor scram. The enclosed report documents the inspection findings, which were discussed on October 20, 2009, with Mr. G. Cullen, Regulatory Programs Manager, and other members of your staff.

The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.

The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

On August 5, 2009, a fault in the 6900 Vac electrical bus resulted in a main generator differential lockout which in turn caused a turbine trip, which resulted in an automatic reactor scram. As a result of the fault on the electrical distribution system, the turbine building filled with smoke. Operators declared a Notification of Unusual Event based on toxic gas in the turbine building. The reactor scram was complicated by an unexpected response in the turbine bypass valve control system. The turbine bypass valves remained open and allowed pressure and temperature to drop until the operators closed the inboard main steam isolation valves to limit the transient. This resulted in the reactor pressure vessel experiencing a 106°F cooldown in approximately 6 minutes and 30 seconds.

Based upon the risk and deterministic criteria specified in NRC Management Directive 8.3, NRC Incident Investigation Program, including a potential generic concern, along with associated moderate risk with the operators isolating the main steam isolation valves, the NRC initiated a special inspection in accordance with Inspection Procedure 93812, Special Inspection. The basis for initiating the special inspection and the focus areas for review are detailed in the Special Inspection Charter (Attachment 2). The determination that the inspection would be conducted was made by the NRC on August 10, 2009, and the onsite inspection started on August 17, 2009.

Energy Northwest 2 This report documents two self-revealing findings of very low safety significance (Green). If you disagree with the characterization of any finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, Region IV, and the NRC Resident Inspector at the Columbia Generating Station. The information you provide will be considered in accordance with Inspection Manual Chapter 0305.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, and its enclosure, will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRCs document system (ADAMS).

ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Dwight D. Chamberlain, Director Division of Reactor Projects Docket: 50-397 License: NPF-21

Enclosure:

NRC Inspection Report 05000397/2009010 w/Attachments:

Attachment 1: Supplemental Information Attachment 2: Special Inspection Charter

REGION IV==

Docket: 50-397 License: NPF-21 Report: 05000397/2009010 Licensee: Energy Northwest Facility: Columbia Generating Station Location: Richland, Washington Dates: August 17 through October 20, 2009 Inspectors: J. Dixon, Senior Resident Inspector, South Texas Project M. Runyan, Senior Reactor Analyst G. Tutak, Reactor Inspector, Engineering Branch 2 Approved By: W. Walker, Chief, Project Branch A Division of Reactor Projects-1- Enclosure

SUMMARY OF FINDINGS

IR 05000397/2009010; 08/17/09 - 10/20/09; Columbia Generating Station; Special Inspection into the fire in the turbine building that occurred on the non safety-related 6900 Vac bus and resulted in a reactor scram.

This report covered 1 week of onsite inspection and in office review through October 20, 2009.

One resident inspector and one regional inspector performed the inspection. Two Green self-revealing findings were identified. The significance of most findings is indicated by their color (Green, White, Yellow, or Red) using Inspection Manual Chapter 0609, Significance Determination Process. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 4, dated December 2006.

Cornerstone: Mitigating Systems

Green.

The inspectors reviewed two examples of a self-revealing finding for the failure to follow Procedure SWP-DES-01, Plant Modification & Configuration Control, Revision 11, for the modifications to the digital electrohydraulic control system and the reactor feedwater pumps. The first example occurred when the licensee installed a new digital electrohydraulic control system with an incorrect pressure setpoint due to an erroneous calculation in the plant design change. The licensee determined that this pressure setpoint was too low for expected pressures under all potential conditions, including transients. This resulted in the turbine bypass valves remaining open and causing the reactor pressure vessel to exceed the cooldown safety limit of 100°F per hour. The second example occurred when the licensee installed a new reactor feedwater level control system which raised and staggered the suction pressure setpoints between the pumps, and the time delay between the pumps was not staggered. The licensees investigation into the reactor feedwater trips determined that the speed setpoint that the level control system allowed the reactor feedwater pumps to achieve was too high.

The inspectors determined that the finding was more than minor because it affected the Mitigating Systems Cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using the Significance Determination Process from Inspection Manual Chapter 0609, the inspectors determined that a Phase 3 analysis was required. Based on the senior reactor analysts significance determination process Phase 3 analysis, this finding was determined to have very low safety significance. This finding had a human performance crosscutting aspect associated with the work practices component in that the personnel associated with the technical review did not use human error prevention techniques commensurate with the assigned task H.4(a) (Section 2.1).

Cornerstone: Initiating Events

Green.

The inspectors reviewed a self-revealing finding for the failure to follow Procedure PPM 1.5.13, Preventive Maintenance Optimization Living Program,

Revision 16, for not evaluating the scope changes for the preventive maintenance inspections on the non segregated high voltage buses. The preventive maintenance work orders included visual inspection, cleaning, torque verification of the rigid and flexible bus connections, and high potential testing of the bus to ground. The inspectors reviewed completed work orders and determined that for all the work orders performed from 2001 through 2005 that the steps to check the torque verification of the bus connections and the high potential testing were inappropriately marked as not applicable. For the 2009 work orders, the inspectors found that the steps for the torque verification and the high potential testing were deleted. In doing so, the licensee was no longer using industry operating experience that was determined to be applicable to the station and had changed the scope of the work orders by not performing these steps; performance of these steps could have prevented the August 5, 2009, 6900 Vac bus failure.

The inspectors determined that the finding was more than minor because it impacted the human performance attribute of the Initiating Events Cornerstone objective to limit the likelihood of those events that upset plant stability and challenge critical safety functions during power operations. Using the Significance Determination Process Phase 1 worksheets from Inspection Manual Chapter 0609, the inspectors determined that the finding was of very low safety significance because it did not contribute to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. This finding also had a human performance crosscutting aspect associated with the decision making component in that personnel performing the preventive maintenance work orders failed to use conservative assumptions and in doing so changed the scope of the work inappropriately H.1(b) (Section 2.3).

Licensee-Identified Violations

None.

REPORT DETAILS

1.0 Special Inspection Scope The NRC conducted a special inspection at Columbia Generating Station to better understand the circumstances surrounding the catastrophic fault in the 6900 Vac electrical distribution system. On August 5, 2009, at 7:49 a.m., a fault in the non safety-related 6900 Vac electrical bus resulted in a main generator differential lockout which, in turn, caused a turbine trip. As a consequence of the turbine trip and reactor power above 30 percent, an automatic reactor scram occurred. As a result of the fault on the electrical distribution system, the turbine building began filling with smoke. A short time later, the licensee declared a Notice of Unusual Event based on toxic gas in the turbine building. The reactor scram was complicated by an unexpected response in the turbine bypass valve control system. The turbine bypass valves remained open and allowed pressure and temperature to drop until operators closed the inboard main steam isolation valves to limit the transient. This resulted in the reactor pressure vessel experiencing a 106°F cooldown in approximately 6 minutes and 30 seconds. In accordance with NRC Management Directive 8.3, it was determined that this event had sufficient risk significance to warrant a special inspection.

The team conducted the inspection in accordance with NRC Inspection Procedure 93812, Special Inspection, and the inspection charter. The special inspection team reviewed procedures, corrective action documents, operator logs, design documentation, maintenance records, and plant computer system data. The team interviewed various licensee personnel regarding the event, and reviewed the licensees troubleshooting plan, root cause analysis report, apparent cause evaluations, past failure records, extent of condition evaluation, immediate and long term corrective actions, and industry operating experience.

1.1 Event Summary On August 5, 2009, while operating at 100 percent rated thermal power, at 7:49 a.m., the main generator experienced a differential current lockout. This resulted in a main generator overall differential lockout trip, which resulted in a turbine trip and a fast transfer from the output of the main generator to the start up transformer (normal breakers opened and start up breakers closed). As a result of the turbine trip, and reactor power above 30 percent, an automatic reactor scram occurred.

Following the reactor scram, two control rods did not initially indicate full in. At 7:50 a.m.

reactor pressure reached a peak value of roughly 1075 psig and resulted in the turbine digital electrohydraulic control system pressure controller failing to manual with a setpoint ramping to 0 psig. This resulted in the turbine bypass valves going full open and the demand signal calling for full open to remain locked in, since the controller switched to manual (an unexpected response). The turbine bypass valves were controlled in automatic, only the pressure control signal that feeds into the turbine bypass valve system switched to manual. Further complicating the event was the unexpected trip of both reactor feedwater turbine-driven pumps on low suction pressure.

The trips resulted from the new reactor feedwater speed control system operating differently than the previous system; the new system responded faster. This system had just been implemented during Refueling Outage R19. This unexpected feedwater response, which performed as designed, resulted in the condensate system exceeding its design pressure. Peak condensate system pressure was measured at 844 psig with a design limit of 775 psig. Also, at this time, the reactor operator turned the reactor mode control switch to shutdown.

At 7:51 a.m., the control room received a report from the turbine building watch of a fire in the electrical bus duct above Switchgear SM-3. This resulted from a fault on the non-segregated 6900 Vac N2/X bus which supplies power from the normal auxiliary transformer to Switchgear SH-5 and SH-6. The bus catastrophically failed, melting all three phases of the conductor, in the location of a flexible bus connection link. The faulted bus section was above Switchgear SM-3. Operators contacted the fire brigade to respond. At 7:53 a.m., operators closed the inboard main steam isolation valves to halt the reactor vessel depressurization and cooldown. Once the final main steam isolation valve went closed, the lowest pressure measured was 393 psig. This ultimately resulted in the reactor pressure vessel experiencing a 106°F cooldown in approximately 6-1/2 minutes, as determined by the reactor recirculation water temperatures. This exceeded the technical specification limit of 100°F per hour. Subsequently, operators controlled reactor vessel pressure manually with the safety relief valves in the band of 500 to 600 psig, and controlled reactor vessel pressure with the condensate booster pumps.

At 7:54 a.m., all rods were verified fully inserted. Operators called the Hanford Fire Department for assistance due to heavy smoke in the turbine building.

At 8:01 a.m., the shift manager made the initial determination that the fire did not meet the criteria for the declaration of a Notification of Unusual Event, since the fire did not meet the requirement for being adjacent to a safe shutdown building. At 8:11 a.m., the fire brigade leader declared that the fire was out and that the smoke was cleared. This report was based on no identified hot spots using thermography cameras to penetrate the smoke. At 8:12 a.m., the shift manager determined that the smoke was potentially toxic and declared a Notice of Unusual Event for toxic smoke. All required notifications to state and local representatives were made in a timely manner. At 9:29 a.m., the turbine building was reported as smoke free, contained normal oxygen levels (21 percent), and contained no explosive gasses (0 percent lower explosive limit). At 9:49 a.m., the shift manager determined that no emergency existed and exited the emergency operating procedures. At 10:06 a.m., the shift manger exited the Notification of Unusual Event. The Hanford Fire Department and the fire brigade were then released.

Operators initiated a plant cooldown to cold shutdown using the safety relief valves to control reactor vessel pressure and condensate booster pumps to control reactor vessel water level. No other significant events occurred during the remainder of the shutdown.

1.2 Licensee Response The inspectors evaluated the licensees implementation of procedures (abnormal, alarm, emergency, and normal operations), training, and technical specifications; reviewed

plant managements control and decision making actions including notifications; and reviewed the troubleshooting and investigating activities that occurred following the fault on the 6900 Vac electrical bus on August 5, 2009. The inspectors reviewed corrective action documents, procedures, technical specifications, plant computer data, and operations logs. The inspectors performed system walkdowns and interviewed engineering, maintenance, and operations personnel.

The inspectors concluded that the operators performed as expected. While the operators complicated the event by closing the inboard main steam isolation valves, the action was necessary to minimize the cooldown that was occurring as a result of the unexpected operation of the digital electrohydraulic system in controlling the turbine bypass valves and causing them to remain open. The inspectors concluded that it was unlikely that operators could have anticipated this plant response because the simulator training for this modification failed to identify any discrepancies, due to a modeling practice of failing the transmitter to the specific high indicated value and not an over-ranged value. However, the operations were trained on turbine bypass valve failures to close. The licensee determined that changes to the response procedure are warranted to more clearly define operator action to prevent exceeding a 100°F per hour cooldown rate. The licensee documented the lessons learned on the performance of the operations personnel and potential improvements in training and procedures in apparent cause Action Request 202477.

The licensee also determined that several issues related to previously implemented modifications resulted in complicating the event. First, the modification to the digital electrohydraulic control system had an inappropriately low reactor vessel pressure over-range setpoint, at which the system determined that the parameter was too high to be valid. The second inadequate modification was to the reactor feedwater control system. The system responded faster than anticipated and, as a result, it caused both reactor feedwater pumps to trip on low suction pressure rather than provide for a time delay after the first pump tripped to allow the second pump time to recover to prevent a loss of reactor feedwater. Additionally, the licensee also determined that the process used for changing the work scope of preventive maintenance work orders and for the flagging of operating experience in plant procedures was less than adequate.

The licensee has entered these issues into their corrective action program for resolution.

Rather than one root cause evaluation for the event, the licensee separated each item for resolution. The 6900 Vac electrical bus fault and plant trip was addressed by the root cause evaluation in Action Request 202384. The issues with the digital electrohydraulic control system modification were addressed by the apparent cause evaluation in Action Request 202385. The issues with the reactor feedwater control system modification were addressed by the apparent cause evaluation in Action Request 202716. The inappropriate changing of the work scope and flagging of operating experience were addressed by apparent cause Action Request 202519.

1.3 Root Cause Evaluation The licensees root cause evaluation determined that the fault was located on the non safety-related non segregated 6900 Vac N2/X bus. The bus catastrophically failed,

melting all three phases of conductor, in the location of a flexible link. The faulted bus section was above medium voltage Switchgear SM-3 and damage from the slag produced by the fault included another high voltage bus and other cables in the area of Switchgear SH-5 and SH-6. The licensee inspected the damage and noted that approximately 3 feet of bus and duct was destroyed in the event. Photographs revealed that a circular pattern of material remained. Therefore, the licensee concluded that the fault likely originated on the center flexible link of the bus connection. However, the licensee concluded that the root cause of failure was indeterminate, because the catastrophic failure of the bus destroyed any evidence that would have provided an indication as to the cause.

The licensee determined that the most probable cause of the event was the relaxation of a bolted connection on the flexible link from repeated thermal cycles over time. The relaxation of the bolted connection resulted in degradation and overheating of the connection. The insulation continued to degrade to the point where a short occurred between two phases of the bus. The short destroyed the bus and also melted the surrounding bus enclosure. The melted aluminum and copper splattered nearby switchgear cabinets but did not cause any internal damage. A possible contributor to the non-segregated bus failure was attributed to running the bus near its original rating of 2500 amps. The nominal bus loading at 90 percent of capacity was closer to the rating than the other four buses, and, therefore, could have caused higher temperatures and connection degradation. The licensee performed a bus uprate analysis in 1994 and increased the rating to 2806 amps to account for the worst case loading of 2610 amps.

As part of the root cause, the licensee reviewed the analysis and determined that the updated value was based on non-conservative data. The factory test included 29 thermocouple test points, but the analysis that upgraded the bus rating used an extrapolation based on favorable data by disregarding 11 of the 29 locations where the temperature rise exceeded its allowable limit. They also found that the 600 MCM flexible connections were the weak points, so they implemented a corrective action to upgrade the connections to 800 MCM making the aluminum bus bar the limiting factor. The licensee recalculated the rating using 800 MCM connections and all 29 data points and determined that the actual rating was 2643 amps. This rating exceeded the worst case loading and allowed for a higher margin when the bus carried its normal load.

The inspectors determined that the licensees root cause evaluation was thorough, in-depth, and self-critical. The root cause evaluation reviewed the event from multiple standpoints to determine weaknesses in design, procedures, training, perception, and management. The root cause evaluation was also thorough in the extent of condition and extent of cause reviews. The corrective actions listed were thorough, in-depth, and focused to prevent recurrence.

2.0 Contributing Causes to 6900 Vac Bus Fault and Complicated Reactor Scram As part of the review for each of the items listed below, the inspectors considered the following factors: equipment failures, human factor and procedural issues, quality assurance issues, radiological issues, security issues, and safety culture issues.

2.1 Digital Electrohydraulic Control System Equipment Failures and Modification Implementation

a. Inspection Scope

The inspectors reviewed the previous 5 years of digital electrohydraulic control system failures and the corrective actions associated with those failures to better understand the health of the system. The inspectors also reviewed the design change package associated with the complete replacement of the system in Refueling Outage R18. The inspectors verified that when changes, tests, or experiments were made, that evaluations were performed in accordance with 10 CFR 50.59 and that licensee personnel had appropriately concluded that the change, test, or experiment can be accomplished without obtaining a license amendment. The inspectors also verified that safety issues related to the changes, tests, or experiments were resolved. The inspectors reviewed corrective action documents, procedures, work orders, design change packages, operator logs, technical specifications, the Updated Final Safety Analysis Report (UFSAR), the apparent cause evaluation. The inspectors also reviewed a sample of past and outstanding work orders to determine whether any deficiencies significantly affected the system function. The inspectors also reviewed similar documents associated with the turbine bypass valve control system to understand the communications between the two systems.

b. Findings and Observations

As part of the 5 year review of the digital electrohydraulic control system, two separate categories emerged, those events that occurred prior to the Refueling Outage R18 modification and those that occurred after. Prior to the modification, the system experienced four events, two circuit card failures, an alignment pin issue on an electrohydraulic operator, and poor overall system performance that resulted in cyclic reactor pressure and steam flow at low power during a plant start up. Of those events, the two circuit card failures and the cyclic reactor pressure and steam flow events were corrected by the installation of the new digital electrohydraulic control system. The new system design included additional hardware and software redundancy that could have prevented those three events. The alignment pin issue on the electrohydraulic operator was determined to be a human performance error during maintenance. After the modification the system experienced four events, one failed check valve, two quadvoter solenoid issues, and the incorrect pressure transmitter over-range setpoint. The failed check valve event was a maintenance issue and one of the quadvoter solenoid issues was determined to be a human performance error that resulted in a failed fitting. Each of these events were unrelated to the installation of the new system. The other quadvoter solenoid issue and the incorrect pressure transmitter over-range setpoint were determined to be design deficiencies of the new digital electrohydraulic control system modification. Overall, the inspectors concluded that the licensee was being proactive and thorough in resolving the issues with the electrohydraulic system.

Introduction.

The inspectors reviewed two examples of a Green self-revealing finding for the failure to follow Procedure SWP-DES-01, Plant Modification & Configuration

Control, Revision 11, for the modification to the digital electrohydraulic control system and the reactor feedwater pumps.

Description.

The first example occurred during June of 2007, when the licensee installed a new digital electrohydraulic control system per plant Design Change 4934, Replace the DEH Pressure Control/Turbine Control and Trip System, dated January 5, 2007. Per the design change, turbine throttle pressure transmitters were replaced with a new narrower range transmitter, and as part of the change, the over-range setpoint, the value at which the digital electrohydraulic control system considered the input invalid, was set to 1089 psig. The licensees investigation into the August 5, 2009, turbine trip, as part of apparent cause Action Request 202385, revealed that this value was too low for expected pressures under all potential conditions, including transients. During this reactor scram from 100 percent power the maximum pressure reached was 1095 psig, and as a result, the system responded as if the pressure transmitters had failed, and consequently the digital electrohydraulic control system transferred the pressure control system of the turbine bypass valves from automatic to manual. This resulted in the turbine bypass valves remaining open and caused the reactor pressure to decrease to around 400 psig. Consequently, the reactor pressure vessel experienced a 106°F cooldown in approximately 6-1/2 minutes, which exceeded the technical specification maximum cooldown rate of 100°F per hour. The licensee failed to follow Procedure SWP-DES-01, Plant Modifications & Configuration Control, Revision 11, Step 3.3.3.a.2 which stated, in part, Independent reviewer(s) to ensure technical accuracy and compliance with the design bases and procedures, and to ensure the adequacy of the design by use of calculational methods, design reviews, or qualification. Contrary to this, the plant design change for the digital electrohydraulic control system did not receive an adequate review of the calculational setpoints that were affected, in particular the control system throttle pressure transmitter over-range setpoint. The value listed was 1089 psig, with no justification for the selection, where as the correct value should have been 1113 psig, based on the maximum expected pressure increase during a transient.

The second example also occurred during June of 2007, when the licensee installed a new reactor feedwater level control system per plant design change 4337, Preventing Level 3 Trip After Reactor Scram, dated June 14, 2007, and 4661, Upgrade RFP Trip Logic and Stagger RFW Low Suction Pressure Trips, dated July 12, 2007. For a more detailed discussion see Section 2.2. As part of the design change, the suction pressure setpoints were raised and staggered between the pumps, and the time delay between the pumps was not staggered. The time delay was included in the different suction pressure trip setpoints that then started a 4 second delay to allow the pump suction pressure to recover before the pump tripped, but both delays could be counting simultaneously. Additionally, all the changes implemented to the level control system resulted in the system responding faster than before. Therefore, following the reactor scram on August 5, 2009, the setpoint-setdown logic in the reactor feedwater level control system lowered the setpoint as designed, which in turn demanded near 100 percent speed from the reactor feedwater pumps. The speed of the feedwater pumps rose to greater than 125 percent of rated feedwater flow and resulted in the suction pressure to the reactor feedwater pumps lowering to below the low suction pressure setpoint within 1 second of each other. Consequently, 4 seconds later, after

the time delay timed out, both reactor feedwater pumps tripped, within a 1/2 second of each other. The licensees investigation into the reactor feedwater pump trips, as part of apparent cause Action Request 202716, determined that the speed setpoint that the level control system allowed the reactor feedwater pumps to achieve was too high.

Rather than 125 percent of rated flow it should have been limited to 110 percent of rated flow. The licensee also staggered the time delay setpoint of the reactor feedwater pumps in addition to the low suction pressure setpoints. As part of the plant design change process the licensee failed to follow Procedure SWP-DES-01, Plant Modifications & Configuration Control, Revision 11, Step 3.3.3.a.2 which states, in part, Independent reviewer(s) to ensure technical accuracy and compliance with the design bases and procedures, and to ensure the adequacy of the design by use of calculational methods, design reviews, or qualification. Contrary to this, the plant design change for the reactor feedwater level control system did not receive an adequate review of the calculational setpoints that were affected, in particular the reactor feedwater maximum speed setpoint following a reactor scram. The value listed was 125 percent of rated flow where as the correct value should have been 110 percent.

The inspectors determined that a significant contributor to the finding was that the licensee did not use human error prevention techniques commensurate with the task to ensure incorrect setpoints were not used. In both cases a detailed setpoint review and appropriate simulator evaluation should have identified the setpoint errors. In both cases the simulator failed to identify the concern due to modeling assumptions that resulted in the simulator not performing like the plant. In the case of the digital electrohydraulic control system, the simulator was modeled to treat the over-range pressure value by using the highest indicated value, not a greater value. In the case of the reactor feedwater level control system the simulator still responded as it did before the level control system design change.

Analysis.

The inspectors determined that the finding was more than minor because it affected the Mitigating Systems Cornerstone objective of ensuring the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Using the Significance Determination Process Phase 1 worksheets from Inspection Manual Chapter 0609, the inspectors determined that since the finding represented a loss of a system safety function, loss of the power conversion system, the inspectors used Appendix A to determine the risk of the finding. Since the circumstances of the finding depart from the guidance provided for Phase 2, per Appendix A, a Phase 3 analysis was completed. The most dominant sequences in the Phase 3 analysis resulted from the main steam valves failing to remain open. Mitigating this effect was the main steam safety valves and manually reopening the main steam isolation valves. The finding was determined to be of very low safety significance (Green), because the Phase 3 screening by the senior reactor analyst concluded that the delta core damage frequency was 3.2E-07 per year. This finding had a human performance crosscutting aspect associated with the work practices component in that the personnel associated with the technical review did not use human error prevention techniques commensurate with the assigned task H.4(a).

Enforcement.

The performance deficiency did not involve a violation of regulatory requirements because the digital electrohydraulic control system and the reactor

feedwater level control system are not safety-related. The licensee entered this issue into their corrective action program as Action Requests 202385 and 202716. Because this finding was not a violation of regulatory requirements and has very low safety significance, it is identified as FIN 05000397/2009010-01, Inadequate Technical Review of Design Change Packages.

2.2 Reactor Feedwater Turbine-Driven Pump Modification Implementation

a. Inspection Scope

The inspectors reviewed the design change package associated with the reactor feedwater speed control system replacement in Refueling Outage R19. The inspectors verified that when changes, tests, or experiments were made, that evaluations were performed in accordance with 10 CFR 50.59 and that licensee personnel had appropriately concluded that the change, test, or experiment could be accomplished without obtaining a license amendment. The inspectors also verified that safety issues related to the changes, tests, or experiments were resolved. The inspectors reviewed corrective action documents, procedures, work orders, design change packages, operator logs, technical specifications, UFSAR, the apparent cause evaluation, and other documents to understand the operation and control functions of the system.

b. Findings and Observations

Two separate design modifications were installed in the main feedwater system. The first modification was installed in Refueling Outage R18 and was associated with the trip logic for the reactor feedwater pumps. It installed higher setpoints and time delays for a trip on low suction pressure. The new trip logic was such that the low suction trip setpoints were 331 psig and 291 psig staggered for time delay between tripping pumps with a 4 second delay on each. Therefore, when the pump reached the low setpoint the pressure must remain below the setpoint for 4 seconds until the pump tripped. However, both time delays could have counted simultaneously. The second modification was installed in Refueling Outage R19 and it replaced the reactor feedwater speed control system. As a result of this event, the licensee determined that the new installed system responded faster than the previous system.

These changes resulted in the condensate system exceeding its design pressure rating of 775 psig. After the turbine trip and reactor scram, reactor vessel water level dropped off scale low (expected). This caused a 100 percent demand from the reactor level control system to the reactor feedwater speed control system. Since this was a turbine trip there was sufficient steam still in the system to allow the reactor feedwater pumps to reach 100 percent speed demand. As a result of the rapid increase in speed, at 100 percent speed demand, the reactor feedwater pump suction pressure dropped to below 200 psig. Pressure decreased so rapidly at the pump suction, due to the rapid speed increase, that the pressure drop from 331 to 291 psig occurred in 0.5 seconds. As a result, when reactor feedwater Pump A tripped, reactor feedwater Pump B could not recover pressure before its time delay was satisfied. Consequently, reactor feedwater Pump B tripped 0.5 seconds after reactor feedwater Pump A. Since the reactor feedwater speed control system was at 100 percent demand, the flowrate of the

condensate system increased. Thus. when the reactor feedwater pumps tripped the condensate pumps were dead-headed at increased flow rates, which resulted in the over-pressure transient.

The reactor feedwater pump suction pressure remained at 846 psig for approximately 8 to 10 seconds. The condensate system booster pumps discharge pressure reached approximately 814 psig. Since the design pressure for this system was 775 psig, the licensee documented this event in Action Request 202388, performed an initial engineering evaluation, and performed an apparent cause evaluation to further understand the circumstances surrounding the event. The licensee concluded that ASME Section XI allowed an overpressure condition of 10 percent based upon the installed relief valve capacity. This equated to 852 psig. Therefore, the licensee concluded that no further structural evaluation was required since the peak pressure recorded was 846 psig. The licensee also performed a system walkdown for any visual signs of damage and no anomalies were identified.

The licensee planned additional modifications are to prevent recurrence

(1) limit reactor feedwater flow demand to 110 percent post scram if two reactor feedwater pumps were running,

(2) stagger the suction pressure time delays to allow the second pump to recover pressure after the first pump tripped,

(3) limit the demand from the level control system following a reactor scram if two reactor feedwater pumps were running, and (4)adjust the low suction pressure trip setpoints to increase the margin between the operating pressure and the trip setpoint at the maximum demand. The inspectors reviewed the licensees apparent cause evaluation, and planned and implemented corrective actions and determined that the licensees actions were acceptable. The inspectors also reviewed the licensees engineering evaluation and the applicable ASME Codes and determined the licensees use of the codes seemed appropriate and allowable.

2.3 6900 Vac Non-segregated Bus Bar Catastrophic Failure

a. Inspection Scope

The inspectors reviewed the technical specifications, UFSAR, corrective action documents, preventive maintenance documents, work orders, the root cause analysis, and other documents to understand the events leading up to the catastrophic failure of the flexible bus bar connection. This review included assessing the adequacy of the licensees root cause analysis and corrective actions, the outage activities that were performed within the past 5 years for any high voltage connections, the compatibility of bus materials, applicable vendor information, and the technical basis behind the bus design. These activities were considered for their impact on the reliability of the system.

The inspectors also reviewed a sample of past and outstanding work orders to determine whether any deficiencies significantly affected the system function.

b. Findings and Observations

The plant has a total of five non safety-related non-segregated buses: two 6900 Vac buses and three 4160 Vac buses, located on the 471 foot elevation of the

turbine building. The non-segregated bus bars were custom designed and supplied by Delta-Star, which is now known as Delta Unibus. The bus bars are completely enclosed in aluminum ducts, with all three phases in the same duct (non-segregated). The bars were a combination of aluminum square hollow tubing and copper connection plates and were mounted on porcelain insulators with spring washers. The bars were assembled together by a series of rigid and flexible connections. The rigid connections consisted of solid copper links, and the flexible connections were 600 MCM braided cable and were used for connections where an expansion joint was required or to connect bus sections that did not properly align together. Both the rigid and flexible connections were bolted to the bus with stainless steel material. All joints and contact surfaces were silver-plated and installed with a layer of zinc chromate grease to minimize the potential for corrosion to occur and were insulated with polyvinyl chloride tape. Duxseal, a chlorinated wax-based putty which included asbestos, was used over the bolted connections to act as a stress relief and eliminate any sharp discontinuities before taping. To complete the installation, a flame-retardant electrical tape was wrapped around the polyvinyl chloride tape. These insulation materials were not optimum for the bus configuration for several reasons. The polyvinyl chloride plasticizers could have evaporated and form films on the bus, which are corrosive in the presence of moisture, and can form hydrogen chloride fumes in the event of a fire.

The licensees apparent cause evaluation in Action Request 202519 that concerned marking steps as not applicable, concluded that there was less than adequate procedural guidance in Procedure SWP-PRO-01, Description and Use of Procedures and Instructions, Revision 9, for step signoffs or data entry in the area of preventive maintenance. Another aspect of the apparent cause evaluation was the lack of guidance on marking steps not applicable in Maintenance Instruction MI-1.8, Conduct of Maintenance, Revision 36. The licensee determined that a contributing cause was a lack of awareness within the maintenance craft supervision of the requirement that a scheduled maintenance system action request was required if the preventive maintenance scope changed. As a result, the licensee determined that this condition could extend across the entire preventive maintenance program because of the lack of procedural guidance on marking steps as not applicable in place of step signoffs or data entry. Therefore, the licensee reviewed a sample of completed critical component preventive maintenance work orders to determine if inappropriate scope changes had occurred. The inspectors evaluated the scope, adequacy, and timeliness of the licensees corrective measures that were planned in response to this issue. The inspectors concluded that the planned actions by the licensee were appropriate to address the identified issue, to prevent recurrence, and were consistent with the safety significance of the issue. These corrective actions included revising Procedure SWP-PRO-01 to require a completed action request prior to marking a preventive maintenance step as not applicable, revising Maintenance Instruction MI-1.8 to address marking steps as not applicable in work instructions - including the requirement to ensure action requests are completed prior to marking a step as not applicable, coaching the maintenance department craft supervision on scope changes and markings steps as not applicable, and revising the work order revision sheet to require an action request if the revision changes the scope.

Introduction.

The inspectors reviewed a Green self-revealing finding for the failure to follow Procedure PPM 1.5.13, Preventive Maintenance Optimization Living Program, Revision 16, for not evaluating the scope changes for the preventive maintenance inspections on the non-segregated high voltage buses.

Description.

In December of 2000, the licensee evaluated NRC Information Notice 2000-14, Non-Vital Bus Fault Leads to Fire and Loss of Offsite Power, and determined that parts of the Information Notice were applicable to the facility. As a result, the licensee created preventive maintenance work orders to inspect the five non-segregated 4160 Vac and 6900 Vac buses. The preventive maintenance work orders included visual inspection, cleaning, torque verification of the rigid and flexible bus connections, and high potential testing of the bus to ground. The work orders were scheduled to be completed every four years with the first inspection to be performed in the 2001 refueling outage. The inspectors reviewed completed work orders from 2001 through 2009 and determined that for the one performed in 2001 and all five performed in 2005, the licensee marked the steps to check the torque verification of the bus connections and the high potential testing as not applicable. For the five work orders completed in 2009, the inspectors note that the steps for the torque verification and the high potential testing were deleted. In doing so the licensee was no longer using industry operating experience that was determined to be applicable to the station and had changed the scope of the work orders by not performing these steps.

The preventive maintenance work order procedures did not have any information identifying them as the response to a significant industry operating experience or the NRC Information Notice. Consequently, the technicians performing the work orders marked the steps as not applicable to save time. Without identification that these steps were a response to operating experience, they failed to ask additional questions. For the 2009 work orders, the system engineer determined that since the steps had been marked as not applicable for such a long period of time that the step was not critical; and, therefore, could be removed. This resulted in the licensee failing to follow Procedure PPM 1.5.13, Preventive Maintenance Optimization Living Program, Revision 16, Step 3.12, which directed the licensee to initiate a scheduled maintenance system action request for changes that affected preventive maintenance, (e.g., due date changes, deferrals, frequency changes, cancellations, etc.) and provide thorough justification for the request. By not performing this evaluation, the licensee failed to appropriately assess the effect of not performing preventive maintenance on the non-segregated buses as required, and subsequently, a fault on the 6900 Vac N2/X non-segregated bus occurred on August 5, 2009, which caused a turbine trip and complicated reactor scram. The licensees root cause analysis determined that the bus connection failure was likely caused by a loose flexible connection that overheated. The immediate corrective actions were to replace all flexible connections on the faulted bus with larger cables, perform boroscopic inspections of all bus connections, and install thermographic windows at each connection location. The licensee also revised the procedure for allowing steps to be marked as not applicable.

Additionally, the inspectors determined that a significant contributor to the finding was that the licensee did not make conservative assumptions in deciding to use personal judgment to allow steps to be marked as not applicable instead of using the approved

process as described in Procedure PPM 1.5.13. A detailed review and formal evaluation of the preventive maintenance scope changes would have prompted the licensee to analyze the deficient non segregated bus maintenance.

Analysis.

The failure to ensure the appropriate evaluations were performed when making scope changes to preventive maintenance items is a performance deficiency.

This finding was more than minor because it impacted the human performance attribute of the Initiating Events Cornerstone objective to limit the likelihood of those events that upset plant stability and challenge critical safety functions during power operations.

Using the Significance Determination Process Phase 1 worksheets from Inspection Manual Chapter 0609, the inspectors determined that the finding was of very low safety significance (Green) because it did not contribute to both the likelihood of a reactor trip and the likelihood that mitigation equipment or functions would not be available. This finding also had a human performance crosscutting aspect associated with the decision making component in that personnel performing the preventive maintenance work orders failed to use conservative assumptions and in doing so changed the scope of the work inappropriately H.1(b).

Enforcement.

The performance deficiency did not involve a violation of regulatory requirements because the 6900 Vac non-segregated bus bars are not safety-related.

The licensee entered this issue into their corrective action program as Action Request 202384. Because this finding does not involve a violation of regulatory requirements and has very low safety significance, it is identified as FIN 05000397/2009010-02, Failure to Perform Inspections Resulted in Bus Failure and Reactor Scram.

2.4 Assessment of Licensees Evaluation of Industry Operating Experience

a. Inspection Scope

The inspectors reviewed the licensees corrective action documents related to industry operating experience for non-segregated bus bars and the digital electrohydraulic control system. This review included assessing the adequacy of the licensees actions, as a result of the operating experience, and how it impacted the operation of the plant.

The inspectors also evaluated the effectiveness and timeliness of the licensees actions.

b. Findings and Observations

No findings of significance were identified. However, the inspectors did have several observations. The inspectors noted that there was of applicable industry information available to the licensee that identified the need to perform preventive maintenance on the high voltage non-segregated buses. The licensees industry operating experience Program reviewed NRC Information Notice 2000-14, Non-Vital Bus Fault Leads to Fire and Loss of Offsite Power at Diablo Canyon in problem evaluation request 200-1702.

The licensee initiated the non-segregated bus preventive maintenance work orders for the 4160 Vac and 6900 Vac non-segregated buses to prevent a similar event from occurring. However, the licensee failed to flag the steps or work order as critical to implementation of operating experience. Additionally, for subsequent related operating experience, the licensee closed out the problem evaluation requests by stating that the

preventive maintenance work orders created in response to the NRC Information Notice was sufficient. The inspectors reviewed several industry operating experience events that were screened out by the licensee. These examples were deemed not applicable, because the licensee already had preventive maintenance work orders in place to inspect the buses. The licensee only reviewed that the work orders were completed, but did not verify what was actually performed. The licensee missed at least one opportunity to identify that the steps were not being completed as part of these reviews.

Furthermore, the work orders did not identify that the steps were in response to operating experience, which contributed to the craftsmen not understanding their importance. Consequently, in early 2009, the licensee removed the critical steps that implemented the operating experience. The inspectors conclusion was that the licensee was both effective and timely with the creation of the preventive maintenance work orders, but failed to ensure the actions were accomplished. This resulted in the response ultimately being neither effective nor timely to prevent a similar occurrence.

The inspectors reviewed available operating experience on the digital electrohydraulic control system and determined that the licensees resolution of the issues was proactive.

The inspectors determined that the licensee addressed the issues with the digital electrohydraulic control system in a timely manner and commensurate with the risk. The majority of the operating experience was internal operating experience that was effectively monitored and resulted in the licensees decision to replace the entire system during Refueling Outage R18.

2.5 Assessment of Licensees Maintenance and Testing Program for Safety and Non Safety-Related Bus Connections

a. Inspection Scope

The inspectors reviewed the technical specifications, UFSAR, corrective action documents, preventive maintenance documents, work orders, and other documents to understand the implementation of the licensees maintenance and testing programs in relationship to the safety and non safety-related bus bars and connections. This review included assessing the adequacy of the licensees maintenance and testing program as it related to preventive and corrective maintenance and how rigorously the licensee was implementing the provisions of the programs.

b. Findings and Observations

No findings of significance were identified. However, the inspectors did have several observations. The licensees program for maintenance and testing of the safety-related bus connections and cables was in accordance with standard industry practices. The safety-related buses were not configured in the same fashion as the bus that failed. The safety-related buses use insulation jacketed cables rather than non-segregated bus bars. The inspectors reviewed the preventive maintenance work orders for the safety-related buses and had no findings. The inspectors concluded that the licensees preventive maintenance program for the safety-related buses was effective and being implemented consistent with industry standards.

For the non safety-related bus connections, the inspectors concluded that if the licensee had implemented the preventive maintenance program as it was proceduralized it could have identified any long term degradation mechanisms before significant damage would have occurred. However, the licensee failed to implement the program as it was written.

The licensee routinely marked steps as not applicable and in effect, changed the intent of the program. Corrective measures that the licensee had put in place included upgrading some of the flexible and rigid connections, upgrading some of the flexible cables to a higher current capacity, and eliminating some of the rigid connections, installing thermographic windows at the connection locations, and performing boroscope inspections of all remaining connections to determine an as-found reference point.

Additionally, the licensee changed the procedures to clarify when to mark steps as not applicable, and better identify steps implemented as a result of operating experience.

4OA6 Meetings

Exit Meeting Summary

On August 20, 2009, the inspectors debriefed to Mr. D. Atkinson, Vice President, Operations, and other members of the licensee staff. The licensee acknowledged the issues presented and agreed to a future exit date pending the completion of the root cause analysis.

On October 20, 2009, the inspectors presented the inspection results to Mr. G. Cullen, Regulatory Programs Manager, and other members of the licensee staff via a telephonic exit. The licensee acknowledged the issues presented. The inspectors asked the licensee whether any materials examined during the inspection should be considered proprietary. Proprietary information was identified and was handled in accordance with licensee requirements. No proprietary material is included in this report.

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

J. Arbuckle, Preventative Maintenance Program Manager

D. Atkinson, Vice President, Operations Support

B. Boyum, Assistant Engineering Manager

G. Cullen, Regulatory Programs Manager

S. Dallas, System Engineer

K. Dittmer, Design Engineering Manager

E. Dumlao, System Engineer

S. Gambhir, Vice President, Technical Services

M. Humphreys, Licensing Supervisor

J. Powers, Maintenance Programs Supervisor

R. Prewett, Operations Support Manager

F. Schill, Licensing Engineer

A. Torres, Corrective Action Program Lead

J. Weers, System Engineer, digital electo-hydraulic system

NRC Personnel

R. Cohen, Senior Resident Inspector,

M. Hayes, Resident Inspector

LIST OF ITEMS OPENED, CLOSED, AND DISCUSSED

Opened and Closed

Inadequate Technical Review of Design Change Packages

05000397/2009010-01 FIN (Section 2.1)

Failure to Perform Inspections Resulted in Bus Failure and

05000397/2009010-02 FIN Reactor Scram (Section 2.3)