IR 05000483/2004009
| ML043430011 | |
| Person / Time | |
|---|---|
| Site: | Callaway  |
| Issue date: | 12/07/2004 |
| From: | Graves D NRC/RGN-IV/DRP/RPB-B |
| To: | Randolph G Union Electric Co |
| References | |
| IR-04-009 | |
| Download: ML043430011 (15) | |
Text
ber 7, 2004
SUBJECT:
CALLAWAY PLANT - NRC SUPPLEMENTAL INSPECTION REPORT 05000483/2004009
Dear Mr. Randolph:
On November 8, 2004, the NRC completed a supplemental inspection at your Callaway Plant.
The enclosed report documents the inspection findings, which were discussed with Mr. Keith Young and other members of your staff.
As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams per 7000 critical hours crossing the threshold from Green (very low risk significance) to White (low to moderate risk significance). This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes, and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.
The NRC concluded that your staff performed thorough evaluations for each of the three reactor trips and performed a thorough and broad-based self-assessment to identify any performance and process issues that should be addressed as a result of the performance indicator crossing the threshold from Green to White.
In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRC's document system (ADAMS). ADAMS is accessible from the NRC website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Union Electric Company -2-Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/
David N. Graves, Chief Project Branch B Division of Reactor Projects Docket: 50-483 License: NPF-30
Enclosure:
NRC Inspection Report 05000483/2004009 w/attachment: Supplemental Information
REGION IV==
Docket: 50-483 License: NPF-30 Report: 05000483/2004009 Licensee: Union Electric Company Facility: Callaway Plant Location: Junction Highway CC and Highway O Fulton, Missouri Dates: November 1-8, 2004 Inspector: M. S. Peck, Senior Resident Inspector Approved By: D. N. Graves, Chief, Project Branch B Enclosure
SUMMARY OF FINDINGS IR 05000483/2004009; 11/01 - 11/08/2004; Callaway Plant; Supplemental Inspection IP 95001 for a White performance indicator in the initiating events cornerstone.
This inspection was conducted by the senior resident inspector. No findings of significance were identified. The significance of most findings is indicated by their color (Green, White, Yellow, Red) using IMC 0609, Significance Determination Process. Findings for which the significance determination process does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process,"
Revision 3, dated July 2000.
Inspector-Identified and Self-Revealing Findings Cornerstone: Initiating Events The NRC conducted a supplemental inspection to assess the licensees evaluation of conditions associated with a White performance indicator in the initiating events cornerstone. Three unplanned reactor trips resulted in the unplanned scrams per 7,000 critical hours performance indicator to cross the threshold from Green to White during the second quarter of 2004. The inspector concluded that the licensees problem identification, root cause, extent-of-condition evaluations, and corrective actions for the three reactor trips were adequate. Two of the reactor trips were caused by main generator supervisory relay failures. The third reactor trip was caused by a reactor operator's failure to follow the power ascension procedure. Several of the root causes contributing to the third reactor trip have been long-standing station problems. The inspector identified weaknesses in the licensees root cause determination and corrective actions related to the third reactor trip. The inspector did not identify any common attributes linking the three reactor trips from a risk perspective.
Enclosure
REPORT DETAILS 01 INSPECTION SCOPE The Nuclear Regulatory Commission conducted a supplemental inspection at the Callaway Plant to assess the effectiveness of the licensees evaluation and corrective actions following a White performance indicator in the initiating events cornerstone. Three unplanned reactor trips resulted in the unplanned scrams per 7,000 critical hours performance indicator to cross the threshold from Green to White during the second quarter of calendar year 2004.
The first reactor trip occurred from full power on January 27 after a contact on a main generator protection relay shorted and caused the main generator output breakers to open. The generator trip resulted in the reactor trip. A second reactor trip occurred from full power on February 3 after a main generator protection relay timer failed in conjunction with a switchyard overcurrent condition. The combination of the failed timer and overcurrent condition resulted in the main generator output breakers opening. The third reactor trip occurred due to an operator error during power ascension on February 15. The operator error exacerbated a steam generator (SG) level transient. The level transient resulted in a feedwater isolation and main turbine trip. The loss of feedwater flow following the isolation resulted in the reactor trip from low SG level.
02 EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a. Determination of who identified the issue and under what conditions The three reactor trips were self-revealing events. The January 27 and February 3 reactor trips occurred from full power during normal plant operations. Both trips were the result of failed components in the main generator protection circuitry. The February 15 reactor trip occurred from about 20 percent power while licensee personnel were performing a plant startup. This trip was the result of an operator error during power ascension activities following main generator synchronization.
b. Determination of how long the issues existed, and prior opportunities for identification The January 27 reactor trip was caused by a failed contact on the 321 G "distance" relay.
The relay was designed to sense remote electrical faults and protect the stator windings from exceeding thermal limits. The licensee verified the functionality of the relay contact during the previous refueling outage. This condition may have existed since the November 2002 reactor restart from completion of the refueling outage.
The February 3 reactor trip occurred after the "dead machine" main generator protection relay timer failed in conjunction with a switchyard overcurrent condition. The "dead machine" circuit was designed to protect the main generator from damage caused by inadvertently energizing the windings when the unit is off line. The relay scheme causes the generator output breakers to open if terminal voltage is sensed (voltage supervision) in combination with any phase high current (29,600 amperes). The voltage supervision Enclosure
-2-portion of the circuit included a timer module. The timer module failed at some point prior to the reactor trip. When the timer module failed, the voltage supervision portion of the circuit locked in. The overcurrent relay activated during routine high voltage switching at the time of the trip. The licensee verified the functionality of the timer module during the previous refueling outage. After the trip, inspection revealed that the timer module had severe heat-related degradation, which caused two voltage supervision contacts to stick closed. The module failure occurred after the November 2002 reactor restart from the last refueling outage.
The February 15 reactor trip resulted from an operator error during a reactor startup. The reactor tripped from low SG water level 28 minutes after operations personnel synchronized the main turbine generator to the grid. SG level oscillations began immediately after the generator output breakers were closed. The level oscillations were caused by a combination of SG shrinkage induced by decreasing feedwater temperatures and a high rate of load increase. The operator had not aligned extraction steam to the feedwater heaters before beginning the power ascension. Feedwater temperatures dropped from 327EF to 240EF during the first 15 minutes of power ascension due to the lack of extraction heating steam. Plant procedures allowed plant operation below 25 percent generator load without feedwater heating in service. A high rate of turbine load increase contributed to the magnitude of the level oscillations. The operator raised plant load about 120 MWe during the 17-minute transient. This power rate increase was equivalent to about 35 percent load change per hour. The operator controlled SG levels with the bypass feedwater regulating valves during power ascension.
In an attempt to dampen the level oscillations, the operator opened the main feedwater regulating valves about 15 minutes after the SG transient began. The magnitude of the level oscillations increased dramatically after the main regulating valves were opened. SG levels reached the high level trip setpoint about 4 minutes later, resulting in a feedwater isolation, a feedwater pump trip, and a main turbine trip. SG levels quickly dropped to the low level reactor trip setpoint. The direct cause of the reactor trip was the operators action to open the main feedwater regulating valves before the plant was stable. Callaway General Operating Procedure OTG-ZZ-00003, "Plant Startup Hot Zero Power to 30%
Power," Revision 27, required the operator to stabilize turbine load at greater than 240 megawatts before transferring feedwater control to the main regulating valves. The turbine load was unstable at about 140 megawatts when the operator opened the main feedwater regulating valves.
c. Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with these issues The inspector reviewed the licensees probabilistic risk assessment for each reactor trip:
The January 27 reactor trip occurred while the main feedwater system was available and resulted in very low risk significance with a conditional core damage probability of 1.28x10-7. The inspectors did not identify any compliance concerns associated with this event.
Enclosure
-3-The February 3 reactor trip occurred with the main feedwater system available and resulted in very low risk significance with a conditional core damage probability of 9.2x10-7.
The risk was slightly elevated because the turbine-driven auxiliary feedwater pump failed to run approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> into the event. The inspector did not identify any compliance concerns associated with this event.
The February 15 reactor trip occurred with the main feedwater system available and resulted in very low risk significance with a conditional core damage probability of 9.2x10-7.
The risk was slightly elevated because the turbine-driven auxiliary feedwater pump failed to run due to an overspeed condition about 15 minutes into the event. The inspector previously identified a self-revealing Green finding and noncited violation of Technical Specification 5.4.1, "Procedures," after the operator failed to follow the plant power ascension procedure (NCV 50-483/2004002-01) and a self-revealing Green finding following the unplanned loss of the turbine-driven auxiliary feedwater pump during the subsequent plant transient (FIN 50-483/2004002-02). Both of these findings were discussed in Section 1R14 of Callaway Plant Integrated Inspection Report 05000483/2004002.
02.02 Root Cause and Extent-of-Condition Evaluation a. Evaluation of methods used to identify root causes and contributing causes The licensee used a structured fault tree analysis and the Electrical Power Research Institute Systems and Equipment Troubleshooting Guide to evaluate the January 27 and February 3 generator protection relay failures. The licensee did not retain the root cause analysis for the inspector to review. However, the inspector concluded that the licensee correctly identified the root and contributing causes of the two reactor trips.
The licensee used an event and causal factor method to establish a timeline for the February 15 reactor trip. The licensee followed up with a barrier analysis methodology to determine the root and contributing causes for the February 15 reactor trip. The inspector reviewed the licenseess root cause evaluation.
b. Level of detail of the root cause evaluation The inspector concluded that the licensees root cause evaluations were of sufficient detail to support the identified root and contributing causes for the January 27 and February 3 reactor trips.
The inspector concluded that the licensees direct cause determination of the February 15 reactor trip was weak. The licensee concluded that the direct cause of the trip was the lack of operator experience and unfamiliarity with secondary side plant startups. The licensee concluded that the operators lack of experience directly lead to the failure to align extraction steam to feedwater heating prior to main turbine-generator synchronization. The inspector concluded that the operator's actions were consistent with station written procedures and policies. While alignment of extraction steam prior to turbine roll was station management's expectation, the inspector concluded that this expectation was not Enclosure
-4-communicated to the operator nor was it incorporated into station written polices and procedures. While the lack of feedwater heating resulted in the SG level oscillations, these oscillations did not become divergent until the operator opened the main feedwater regulation valves.
The inspector concluded that the direct cause of the event was the operators actions to prematurely open the main feedwater regulation valves. While the level transient was caused by the lack of feedwater heating and exacerbated by the high load rate increases, the SG level oscillations were not divergent until the main feedwater regulation valves were opened. The licensees root cause evaluation did not fully consider the underlying causes contributing to the operating crews decision not to stop the power ascension when the unexplained SG level oscillations began.
The licensee identified the following root causes for the February 15 reactor trip:
- The policy regarding prejob briefs was not strict enough and allowed interpretation which resulted in varying degrees of quality of briefs.
- Operations supervisory oversight and standards reinforcement needed improvement.
- Training course work needed to be improved in the areas of operating secondary plant equipment and situational awareness of indications.
- General operating procedures were cumbersome and difficult to follow.
c. Consideration of prior occurrences of the problem and knowledge of prior operating experience The inspector concluded that the licensee properly considered prior operating experience when evaluating causes of the January 27 and February 3 reactor trips.
The licensee considered past operating experience in the evaluation of the February 15 reactor trip. In 2002, an independent assessment identified significant deficiencies in the use of human performance tools at the Callaway Plant. This assessment concluded that the lack of fully implemented tools, specifically prejob briefings and procedures, resulted in errors and unsafe work practices. The assessment stated that, while workers and supervisors received training on these tools, the tools were not embraced as useful throughout the station (Callaway Action Request (CAR) 200204836). This assessment also referenced a number of previous Callaway operational events, including a 2001 reactor trip and an unexpected reactivity change resulting from poor operations procedures (CAR 200204811). The licensee had also previously identified that operations had not placed the appropriate priority on monitoring and implementing changing industry standards, resulting in the loss of the ability to reinforce and maintain high standards (CAR 20020409). The licensees corrective actions addressing these past assessment issues were not timely. For example, the licensees corrective action plan to improve power ascension procedure quality was not scheduled to be completed until 2008.
Enclosure
-5-d. Consideration of potential common causes and extent of condition of the problem The inspector concluded that a common cause did not exist for the January 27 and February 3 reactor trips. While both trips resulted from failed main generator protection relays, the failed mechanisms were not related. The root causes identified by the licensee for the February 15 reactor trip have been long-standing station problems. In addition, an unplanned safety injection occurred on February 11 during plant heatup activities. The licensee determined the root causes of the safety injection were the same as the root causes of the February 15 reactor trip.
02.03 Corrective Actions a. Appropriateness of corrective actions The licensee took prompt corrective actions to repair the equipment failures related to the January 27 and February 3 reactor trips. The licensees immediate corrective actions included identifying and inspecting other similar relays installed in the plant and revising preventive maintenance procedures to enhance relay inspection activities. Long-term corrective actions included a review of all single-point relay failures which could result in a reactor trip. The licensee implemented a design change, Request for Resolution 023345A, to evaluate removal of single-point reactor trip vulnerability from plant relay schemes. The licensee identified 101 relays that have the potential to cause a reactor trip. The licensee plans to perform Modification MP-04-1016 during the next refueling outage to eliminate these potential sources of a single-point trip in the protective relaying scheme.
The licensee concluded that conditions leading to the February 15 reactor trip revealed a significant human performance concern. The licensees corrective actions following the February 15 reactor trip included:
- Strengthened prejob brief expectations for operations
- Evaluated strengthening the station prejob brief policy
- Informal reenforcement of operations standards
- Enhanced supervisory oversight for infrequent evolutions
- Enhanced observation and coaching of operations personnel
- Enhanced operations training for secondary plant manipulations
- Revision of the power ascension procedure to require extraction steam alignment prior to rolling the turbine Enclosure
-6-b. Prioritization of corrective actions The inspector concluded that the licensee properly prioritized the corrective actions following the January 27 and February 3 reactor trips. Both failed relay schemes were removed from the main generator protection logic. The licensee also performed a single-point relay assessment to identify any other potential signal protective device failures that could result in a reactor trip.
Many of the corrective actions which resulted from the February 15 reactor trip were not scheduled to be completed until the end of the calendar year. While actions associated with verbal reenforcement of operations policies and expectations have been completed, the licensee has not incorporated all of these changes into the station administrative control programs. The reliance on informal corrective action has been a recurring problem at the plant. For example, the operator's decision to roll the turbine before aligning the feedwater heater started the SG oscilations and contributed to the reactor trip. The station's expectation for aligning feedwater heating was informal and had not been incorporated into written administrative controls. Similarly, the licensee has not incorporated several of the postevent corrective actions into written operating procedures and policies.
The inspector concluded that the licensees corrective actions to strengthen prejob brief expectations for operations had not been effectively implemented. At the time of the reactor trip, the licensee had a definitive formal written prejob brief policy (UEND-PRE/POSTJOB-01, Revision 2). The policy clearly defined when prejob briefs were required, the level of detail of the brief, and individuals required to attend the brief.
In April 2004, the licensee deleted this prebrief policy and replaced it with a worker safety policy describing the minimum prejob brief requirements from the Department of Labor, Occupational Safety & Health Administration for electric power generation, transmission, and distribution (29 CFR 1910.269). The station prejob brief policy was revised again in September 2004. The inspector concluded that the revised policy met minimum Occupational Safety & Health Administration requirements, but did not clearly define when briefs were required, allowing interpretation and resulting in varying degrees of brief quality. In addition, the NRC identified a subsequent finding where poor prejob briefs contributed to an operational effort resulting in plant workers exposure to an unplanned high radiation area (NCV 50-483/2004003-03, Section 1R23 of Callaway Plant Integrated Inspection Report 05000483/2004003). The licensee currently has an open corrective action document to further review the potential need to enhance the prejob brief guidance.
c. Establishment of a schedule for implementing and completing the corrective actions The inspector verified that the licensees corrective action program identified assigned individuals, completion dates, and reference numbers to ensure that individual corrective actions would be completed in accordance with their priority.
Enclosure
-7-d. Establishment of quantitative or qualitative measures of success for determining the effectiveness of corrective actions to prevent recurrence The licensee did not establish specific success or corrective action effectiveness measures following the January 27 and February 3 reactor trips, although the hardware issues were repaired and the circuitry modified such that recurrence was unlikely. The licensee did establish specific criteria for determining the effectiveness of the corrective actions following the February 15 reactor trip.
03 MANAGEMENT MEETINGS Exit Meeting Summary The inspector presented the inspection results to Mr. Keith Young, Manager, Regulatory Affairs, and other members of the plant staff on November 8, who acknowledged the findings. The inspectors confirmed that proprietary information was not provided or examined during the inspection.
Enclosure
ATTACHMENT Persons Contacted R. Barton, Assistant Superintendent, Operations J. Hiller, Engineer, Regulatory Affairs L. Kanuckel, Superintendent, Quality Assurance M. Reidmeyer, Supervisor, Regional Regulatory Affairs D. Waller, Supervisor, System Engineering K. Young, Manager, Regulatory Affairs Documents Reviewed Root Cause Evaluation - S-2003-0134 CAR 200204809, INPO AFI OP.1-1, Operations does not reinforce high standards CAR 200204811, INPO AFI OP.4-2, Deficiencies in the quality of operations procedures CAR 200204836, INPO AFI HU.1-1, Human performance tools not fully implemented CAR 200400812, Dead machine relay outage CAR 200408169, Callaway Plant guidance for pre-job briefs differs from the industry CAR 200401167, Reactor trip due to low steam generator water level CAR 200404996, NRC PI for unplanned scrams went white PRA Evaluation Request Form 04-217 (CAR 200400629)
PRA Evaluation Request Form 04-218 (CAR 200400791)
PRA Evaluation Request Form 04-219 (CAR 200401168)
Procedure APA-ZZ-00500, Corrective Action Program, Revision 35 Request for Resolution 23375A&B, Replace relays with generator protection relay, completed on May 12 Minor Modification Work Request W722981, Defeat the trip of dead machine relay by lifting leads, completed on June 11 Systems and Equipment Troubleshooting Guide, EPRI, Palo Alto, CA LER 50-483/2004-002-00, Reactor trip due to faulty electrical relay LER 50-483/2004-003-00, Reactor trip due to faulty relay A-1 Attachment
-2-LER 50-483/2004-004-00, Safety injection while conducting plant heatup to normal operating pressure and temperature LER 50-483/2004-005-00, Inadequate feedwater heating during plant start-up causes turbine trip and subsequent reactor trip Simple self-assessment of five CARs documenting the events that resulted in Callaways NRC performance indicator on unplanned scrams going white, SA04-PI-S05, November 2, 2004 A-2 Attachment