IR 05000498/2003010
| ML032550265 | |
| Person / Time | |
|---|---|
| Site: | South Texas  |
| Issue date: | 09/11/2003 |
| From: | Johnson W NRC/RGN-IV/DRP/RPB-A |
| To: | Sheppard J South Texas |
| References | |
| IR-03-010 | |
| Download: ML032550265 (24) | |
Text
September 11, 2003
SUBJECT:
SOUTH TEXAS PROJECT ELECTRIC GENERATING STATION - NRC SUPPLEMENTAL INSPECTION REPORT 05000498/2003010 AND 05000499/2003010
Dear Mr. Sheppard:
On August 22, 2003, the NRC completed a supplemental inspection at your South Texas Project Electric Generating Station, Units 1 and 2, facility. The enclosed report documents the inspection findings which were discussed on August 21, 2003, with you and other members of your staff.
As required by the NRC Reactor Oversight Process Action Matrix, this supplemental inspection was performed in accordance with Inspection Procedure 95001. The purpose of the inspection was to examine the causes for and actions taken related to the performance indicator for unplanned scrams per 7000 critical hours crossing the threshold from Green (very low risk significance ) to White (low to moderate risk significance) for Unit 2. This supplemental inspection was conducted to provide assurance that the root causes and contributing causes of the events resulting in the White performance indicator are understood, to independently assess the extent of condition, and to provide assurance that the corrective actions for risk significant performance issues are sufficient to address the root causes and contributing causes and to prevent recurrence. The inspection consisted of selected examination of representative records and interviews with personnel.
The NRC concluded that your staff performed thorough evaluations for each of the three Unit 2 reactor trips in 2002 and performed a thorough and broad based self assessment to identify any performance and process issues that should be addressed as a result of the performance indicator crossing the threshold from Green to White. The inspectors identified one discrepancy concerning a contributing cause identified in your self assessment evaluation that was not adequately addressed by the specified corrective actions. This was corrected during the course of the inspection.
In accordance with 10 CFR 2.790 of the NRCs "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
-2-STP Nuclear Operating Company Should you have any questions concerning this inspection, we will be pleased to discuss them with you.
Sincerely,
/RA/
William D. Johnson, Chief Project Branch A Division of Reactor Projects Dockets: 50-498 50-499 Licenses: NPF-76 NPF-80
Enclosure:
NRC Inspection Report 05000498/2003010 and 05000499/2003010 w/Attachment: Supplemental Information
REGION IV==
Dockets:
50-498 50-499 Licenses:
NPF-76 NPF-80 Report No:
05000498/2003010 05000499/2003010 Licensee:
STP Nuclear Operating Company Facility:
South Texas Project Electric Generating Station, Units 1 and 2 Location:
FM 521 - 8 miles west of Wadsworth Wadsworth, Texas 77483 Date:
August 18 through August 22, 2003 Inspectors:
T. Farnholtz, Senior Project Engineer A. Sanchez, Resident Inspector Approved By:
W. D. Johnson, Chief Project Branch A Division of Reactor Projects
Enclosure
SUMMARY OF FINDINGS
IR05000498/2003010; IR05000499/2003010; 08/18/2003-08/22/2003; South Texas Project
Electric Generating Station; Units 1&2. Supplemental inspection.
Cornerstone: Initiating Events
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluations associated with three unplanned reactor trips of Unit 2 during calender year 2002. The cumulative effect of these trips was that the Performance Indicator (PI) for unplanned scrams per 7000 critical hours crossed the threshold from Green (very low risk significance) to White (low to moderate risk significance). In addition to the evaluations for each of the three trips, the licensee performed a self assessment evaluation to identify any performance and process issues that should be addressed as a result of the PI crossing the threshold from Green to
- White.
During this supplemental inspection, performed in accordance with Inspection Procedure 95001, the inspectors determined that the licensee performed comprehensive evaluations of each of the three events. For each case, specific problems were identified, an adequate root cause evaluation was performed, and corrective actions were taken or planned to prevent recurrence. The self assessment evaluation identified several contributing causes and proposed corrective actions to address these causes. In general, the self assessment evaluation was comprehensive and thorough, however, the inspectors identified one case where a contributing cause was not adequately addressed by any of the proposed corrective actions within the self assessment document. This was associated with the corrective action process that did not in all cases identify the root cause(s) of issues.
Corrective actions for this contributing cause were included in the licensees strategic performance improvement plan and were subsequently added to the self assessment evaluation to fully account for the resolution of the contributing causes.
Report Details
INSPECTION SCOPE
The U.S. Nuclear Regulatory Commission (NRC) performed this supplemental inspection to assess the licensees evaluation associated with a Performance Indicator (PI) that crossed the threshold from Green to White. The PI was for unplanned scrams per 7000 critical hours for Unit 2 and was related to the initiating event cornerstone in the reactor safety strategic performance area. The PI was White for the fourth quarter 2002 and the first quarter 2003.
South Texas Project (STP) Unit 2 experienced three unplanned reactor trips in 2002. The cumulative effect of these trips was to cause the PI to cross the threshold from Green to White.
The inspectors reviewed the licensees actions associated with these three events and conducted interviews of licensee personnel.
The licensee performed a self assessment evaluation to identify any performance and process issues that should be addressed as a result of the PI crossing the threshold from Green to White. The scope of the licensees examination was significantly broader than the scope of this supplemental inspection. The inspectors reviewed this self assessment.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification The licensees self assessment evaluation focused on plant trips and significant downpowers that occurred between January, 2000 and March, 2003. Condition Reports (CRs) associated with 19 reactor trips, significant downpowers, and plant shutdowns were analyzed using the performance-based tool Common Cause Analysis to evaluate possible shared causes.
The inspectors considered the approach taken to identify the broader issues and problems during the self assessment evaluation to be thorough and methodical.
Manual Reactor Trip due to Feedwater Isolation Valve Closure a.
Method of identification On June 14, 2002, while at 100 percent power, Unit 2 commenced feedwater isolation valve (FWIV) operability testing. After successfully testing the 2A and 2B FWIVs, the 2C FWIV was tested. At 4:35 am (CDT) the test button to partially stroke the 2C FWIV was depressed and released. The 2C FWIV continued to travel in the closed direction.
When the Unit Supervisor was informed that there was no feedwater injection into the C steam generator, the Unit Supervisor ordered the reactor to be manually tripped.
This event was self-revealing and there were no indications of the impending failure prior to the event.
b.
Duration of issue and prior opportunities for identification In a concerted effort by the licensee to reduce the number reactor trips, a Trip Reduction Task Force was established in 1998. The Trip Reduction Task Force identified that the FWIV control circuit was susceptible to single point failures. The failure of the 2C FWIV control circuit diodes represent a single point failure. Prior to this failure there were no other failures of this particular type, although the design weakness of a single point failure was identified in 1998.
c.
Risk consequences and compliance issues The licensee performed a probabilistic risk assessment, which determined the conditional core damage probability to be 3.1 X 10 -7. The inspectors identified no risk consequences or compliance issues.
Automatic Reactor Trip due to Steam Generator High Level a.
Method of identification On July 7, 2002, at approximately 11:12 pm, while the reactor was at 100 percent power, Unit 2 experienced an automatic reactor trip due to high level in steam generator 2B. This occurred shortly after a failure of the Unit 2 Channel II Inverter 1202. All four steam generator level narrow range control channels were selected to Channel II. The failure of Inverter 1202, and subsequent loss of the associated distribution panel 1202, caused a loss of all control channels selected to Channel II. This resulted in a complete loss of a steam generator level control signal to the control circuits. The main feedwater regulating valves (MFRV) responded by repositioning to their full open position. The reactor operator immediately took manual control of the steam generator 2C and 2D MFRV controllers. Level control was regained in steam generators 2C and 2D, but the reactor tripped on a high steam generator water level on steam generator 2B.
b.
Duration of issue and prior opportunities for identification The Inverter 1202 failure was instantaneous and produced an immediate effect on Unit 2 operations. The cause for the inverter 1202 failure was a blown fuse caused by a malfunctioning printed circuit board (PCB). A very similar event took place at STP in late 1992, but no action was taken to investigate the cause of the failure. The inspectors believe that the event in 1992 may have been a prior opportunity for problem identification, but because the inverter experienced no more failures until this most recent event, and the fact that the PCB has been replaced since then, the inspectors cannot say that it was a strong opportunity for identification.
c.
Risk consequences and compliance issues The licensee performed a probabilistic risk assessment for the event. The conditional core damage probability (CCDP) for an excessive feedwater initiating event was
calculated to be 1.9 X 10-7, while the CCDP for an excessive feedwater initiating event concurrent with the loss of the instrument channel was calculated to be 5.7 X 10-7. The delta CCDP was determined to be 3.8 X 10-7.
The inspectors identified no significant risk consequences or compliance issues.
Manual Reactor Trip due to Turbine Blade Failure a.
Method of identification On December 15, 2002, control room operators received main turbine bearing vibration high alarms and reports of abnormal noise from the turbine building. A manual reactor trip was initiated and the main turbine was taken off-line. The cause of the alarms and abnormal noise was a failed turbine blade in low pressure turbine 22. This event was self-revealing in that no indications of impending failure were identified.
b.
Duration of issue and prior opportunities for identification The conditions leading to this failure were established during refueling outage 2RE09 (October 2 through December 6, 2002) with the installation of a replacement main generator rotor. Nine days after completion of the refueling outage, the turbine blade failed and resulted in a manual reactor trip.
The licensee performed inspections of the turbine rotor blades every three cycles.
Cracks were identified in the Unit 2 low pressure turbine 23 blades as early as refueling outage 2RE08 (March 7 through April 2, 2001); however a different mechanism was identified as the cause of these cracks. The inspectors reviewed the history of turbine blade inspections in both Unit 1 and Unit 2 and determined that prior opportunities for identification of impending blade failure following refueling outage 2RE09 were not available.
c.
Risk consequences and compliance issues No risk consequences or compliance issues were identified.
02.02 Root Cause and Extent of Condition Evaluation The licensees self assessment evaluation identified one apparent cause and four contributing causes for the problems experienced in the area of station reliability. The apparent cause was identified as:
Station expectations, roles, priorities, and goals are not unified in improving plant reliability.
The contributing causes were:
1)
The corrective action process did not in all cases identify the root causes of issues nor consistently supply effective corrective actions to prevent recurrence.
2)
The modification process was not consistently and effectively implemented, particularly in the areas of design input, design verification, and failure modes and effects analysis.
3)
Appropriate resources have not always been dedicated to identify critical equipment vulnerabilities, develop strategies to prevent their failure, and to implement these strategies.
4)
The licensee has not reinforced and implemented programs or processes to proactively eliminate latent organizational and programmatic deficiencies. This results in a reactive approach to improvement via events.
The inspectors considered these broad ranging observations to be well supported and adequate to address the issues identified in the self assessment evaluation.
Manual Reactor Trip due to Feedwater Isolation Valve Closure a.
Evaluation of method used to identify root causes and contributing causes The licensees efforts to identify a root cause for the event consisted of a systematic approach of inspection, laboratory analysis of failed components, maintenance and operating history review, corrective action program review, and construction and use of an event and causal factor diagram. This analysis yielded one root cause.
The inspectors considered the approach taken to be adequate to provide a good level of confidence that the root cause was accurately identified.
b.
Level of detail of the root cause evaluation The licensees root cause determination was thorough and identified two failed diodes as the root cause. The failed diodes caused the circuit protection fuse to fail open.
Once power was no longer available, the solenoid valve in that circuit de-energized and opened. This allowed the 2C FWIV to fail to the closed position. With the exception of the failed diodes, the control circuit functioned as designed without any issues. The laboratory analyzed the diodes and concluded that the diodes failed due to having been electrically overstressed. After troubleshooting and testing, and reviewing operational and vendor experience, the licensee could not pinpoint an exact cause for the electrically overstressed condition that caused the diodes to fail. The licensee considered the failure to be random.
Based upon a review of the associated condition reports, significant conditions adverse to quality investigation and the event team review report, discussion notes between the
licensee and the vendor, and discussions with licensee personnel, the inspectors determined that the actions taken to establish the root cause were adequate and accurate.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluation included an operational and industry experience review. No prior occurrences of this specific problem at STP were identified. In searching operational experience for the vendor of the specific diodes, the licensee did identify four diode failure events. Of these, two of the failures had no known cause of failure, one was due to age, and the fourth was due to an electrical overload. During a continued review of other operational experience and industry events, numerous failures were found and most causes were random or unknown with the balance due to age (infant mortality or old age).
The inspectors did not possess any information to the contrary and believe that operational and industry experience was considered.
d.
Consideration of potential common causes and extent of condition of the problem The licensees evaluation considered the potential common causes and the extent of the condition associated with the failed diodes in the FWIV control circuit. The evaluation determined that all FWIV control circuits, Unit 1 and Unit 2, were susceptible to this type of single point failure, but because the specific cause for the diode failure was considered to be random, no common cause between the control circuits were identified. The licensee also acknowledged the fact that the control circuits for the MFRVs and the main steam isolation valves (MSIV) for Units 1 and 2 were also susceptible to single point failure, but no common cause element was identified.
The inspectors agreed with the licensees common cause and extent of condition evaluation.
Automatic Reactor Trip due to Steam Generator High Level a.
Evaluation of method used to identify root cause and contributing causes The licensees efforts to identify a root cause for the event consisted of a systematic approach of inspection, extensive troubleshooting in the field, discussions with the vendor, vendor analysis and refurbishment of the component involved, laboratory analysis of failed components, maintenance and operating history review, corrective action program review, and construction and use of an event and causal factor chart.
The analysis identified two root causes and two contributing causes.
The inspectors considered the approach taken to be adequate to provide a good level of confidence that the root causes were accurately identified.
b.
Level of detail of the root cause evaluation The licensees root cause determination was thorough and identified two root causes and two contributing factors. The first root cause was the intermittent failure of a PCB (serial number A0158) used in the inverter itself. The two silicon controller rectifiers (SCR) on the PCB (serial number A0158) were unexplainably gated at the same time.
This simultaneous gating caused an overcurrent through the 1FU fuse on the inverter.
The fuse failed resulting in the loss Inverter 1202.
Since the Inverter 1202 loss on July 7, 2002, which caused the unit to trip, there were two more subsequent failures of this same Inverter 1202. The second instance of the Inverter 1202 failure occurred one month later on August 7, 2002, but due to corrective action taken by the licensee the unit did not trip. The licensees investigation narrowed, and centered on the PCB (serial number A0158), which controls the gating operation on the inverter as the root cause. The PCB (serial number A0158) was removed from service and sent to the vendor for analysis. The results from the analysis were inconclusive. The PCB (serial number A0158) was refurbished, tested, and sent back to the licensee.
The third failure occurred during ongoing diagnostic testing activities in Unit 2, while in refueling outage 2RE09. Due to an unrelated defect discovered in a specific batch of vendor supplied gating PCBs, the refurbished PCB (serial number A0158) had been placed back into Inverter 1202. The licensee decided to use the PCB (serial number A0158) because it has just came back from the vendor and was newly refurbished and tested. The failure of the inverter occurred after this replacement had occurred. The common denominator was the same exact PCB (serial number A0158).
A second root cause was that it was a proceduralized practice to operate with all four steam generator level control inputs aligned to the same channel. This had a decisive effect on the event outcome. Once the inverter was lost, the distribution panel 1202 was also lost. This caused a loss of all steam generator level inputs to the control circuits.
The loss of steam generator level indication resulted in the MFRV to reposition to the full open position. The main feedwater pumps responded by going to maximum speed.
The control room reactor operator could not maintain steam generator levels in all four steam generators. The unit tripped on high steam generator level in the 2B steam generator.
Two contributing causes identified by the licensees root cause evaluation were troubled communication in the control room between the reactor operator and the unit supervisor, and guidance in procedure 0POP04-VA-001, Loss of 120 VAC Class Vital Distribution, Revision 11, supplied no information regarding HI-HI steam generator level manual trip criteria.
Based upon a review of the licensees root cause investigation, condition reports associated with the trip, discussions with plant personnel, and vendor and drawing information, the inspectors determined that the actions taken to establish the root causes and contributing causes were adequate and accurate.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensee performed an industry events and operational experience search concerning loss of inverters due to fuse failure. The licensee discovered that there were relatively few occurrences of fuse failures due to gating PCB problems. Interestingly, one event was recorded at STP in 1992. The event in 1992 was identical to the failure in 2002, except that the 1992 event occurred at shutdown and no event investigation or root cause was performed. There were approximately 16 industry events that involved a loss of an inverter, which affected feedwater flow control that resulted in a reactor trip.
The licensee gained useful and confirmatory information concerning their corrective actions.
The inspectors believe that the licensee effectively researched and considered industry and operational experiences, and prior occurrences of the problems at the plant.
d.
Consideration of potential common causes and extent of condition of the problem The licensees root cause investigation considered potential common causes, but because the root cause was determined to be a specific, faulty, PCB (serial number A1058) on Inverter 1202, no common causes were identified. The extent of condition was reviewed by the licensee and was determined that the problem was specific to Inverter 1202. This decision was mainly due to the fact that no other problems or failures had been seen on any of the other safety related instrument inverters (Unit 1 or Unit 2).
The inspectors agreed with the licensees common cause and extent of condition evaluation.
Manual Reactor Trip due to Turbine Blade Failure a.
Evaluation of method used to identify root causes and contributing causes To evaluate this issue, the licensee used a systematic approach of inspection, laboratory analysis of failed components, and maintenance and operating history review.
Possible root causes were considered and evaluated. Two probable root causes were identified following the blade failure. To confirm the actual root cause, the licensee made repairs to the turbine generator and associated equipment and installed a blade vibration monitoring system and a torsional vibration monitoring system on the turbine generator following the failure on December 15, 2002. The unit was started up on January 22, 2003, and operated at full power to gather data from the vibration
monitoring systems. The unit was shut down two days later to perform corrective actions based on the data obtained.
The inspectors considered the approach taken to be adequate to provide a good level of confidence that the root cause and contributing causes were accurately identified.
b.
Level of detail of the root cause evaluation The licensees root cause determination was thorough and identified the primary root cause as high cycle fatigue. The specific crack initiator was torsional vibration caused by a turbine generator rotor system designed with a natural frequency near 120 Hz and a new generator rotor that changed the turbine generator rotor system closer to 120 Hz.
The Unit 2 main generator rotor was replaced during refueling outage 2RE09. During fabrication, the new rotor was machined incorrectly. The vendor completed the rotor with modifications and recalculated the natural frequency and determined that the rotor was acceptable for use in Unit 2. The licensees evaluation determined that an error was made in the calculation in that the natural frequency in this installation was near 120 Hz. This corresponded with the natural frequency of the L-0 disk and blade assemblies used in the low pressure turbines. The result of this combination was that the disk and blade assemblies were excited during operation which caused high cycle fatigue, crack initiation in the turbine blades, crack progression, and finally blade failure. The torsional vibrations were severe enough to cause blade failure nine days after startup from refueling outage 2RE09.
Based upon a review of the associated condition reports, root cause evaluation, laboratory reports, vendor recommendations, and discussions with licensee personnel, the inspectors determined that the actions taken to establish the root cause were sufficiently detailed and adequately supported the conclusions.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience No prior occurrences of this specific problem was identified and no knowledge of operating experience was available. The licensee concluded that the vendor that manufactured the replacement main generator rotor made an error in fabrication and did not adequately analyze the as-built component to ensure the turbine generator system would operate within acceptable limitations.
d.
Consideration of potential common causes and extent of condition of the problem The licensee performed an engineering evaluation (OPGP04-ZA-0002) to evaluate the continued operation of the Unit 1 main turbine generator given the L-0 blade failures in the Unit 2 turbine generator. This evaluation was performed at a time when the root cause of the blade crack initiation was not known for the Unit 2 blade failure and Unit 1 was still on-line. The licensee had established two possible root causes for the failure
including
- (1) off-normal operating conditions resulting in periodically exceeding the endurance limit of the turbine blades; and
- (2) torsional vibration in the turbine rotor system. The engineering evaluation considered both these possible root causes as they applied to Unit 1. Based on the differences between the Unit 1 and Unit 2 turbine generators and a review of the operating history of Unit 1, it was recommended that Unit 1 continue to operate at 100 percent power with no operational restrictions. In addition, recommendations were made to inspect the Unit 1 turbine blades if a forced outage occurred or at the next scheduled refueling outage (1RE11).
Unit 1 remained on-line until the scheduled refueling outage (March 26, 2003) and turbine blade inspections were performed. A total of five blades were identified as having cracks. These blades and the adjacent attached blades were replaced during the refueling outage. The root cause for these cracks was identified as being different from the cause of the Unit 2 blade failure. During the outage, a torsional vibration monitoring system was installed in the Unit 1 main turbine generator. The data from this system confirmed that the torsional vibrations identified as the root cause of the Unit 2 blade failure did not exist in the Unit 1 turbine generator.
The inspectors considered the licensees extent of condition review to be thorough and complete. There were no common cause issues identified.
02.03 Corrective Actions The licensees self assessment evaluation identified four contributing causes for the problems experienced in the area of station reliability. Three of the four contributing causes were adequately addressed by the corrective actions. The corrective actions specified in the self assessment evaluation did not address the contributing cause that concerned the corrective action process that did not, in all cases, identify the root cause(s) of issues. Upon discovery, the inspectors discussed this deficiency with the licensee. The licensee agreed with the deficiency within the corrective actions that precipitated from the self-assessment evaluation, but did state that the issue of improving the corrective action process to better identify root causes and repeat events was a part of the STP Nuclear Operating Company Strategic Performance Improvement Plan. This plan, which consists of five strategic areas identified by the Senior Management Team (SMT) for overall plant improvement, was reviewed by the inspectors. The licensee did correct and add the appropriate actions to the condition report used to track the corrective actions of the self-assessment evaluation, to fully account for the resolution of the contributing causes.
Manual Reactor Trip due to Feedwater Isolation Valve Closure a.
Appropriateness of corrective actions The licensee took immediate and proper corrective actions to repair the 2C FWIV control circuit and determine the root cause of the failure. The licensee also checked the material condition of the rest of the Unit 2 FWIV control circuits, specifically the
diodes that were involved in the 2C FWIV control circuit failure prior to plant startup.
The failed diodes were sent to a test laboratory for analysis. A review of the maintenance history for 2C FWIV was performed, as well as a review of the Energize to Actuate modification that is expected to eliminate the single point failure potential.
The inspectors determined that the proposed corrective actions were appropriate.
b.
Prioritization of corrective actions The licensees immediate corrective action returned the 2C FWIV to operable status.
The subsequent actions to review the material condition and the testing of all other Unit 2 FWIV control circuits, including the diodes, prior to Unit 2 start up was appropriately prioritized. All other subsequent actions dealt with completing a root cause investigation.
The inspectors reviewed the prioritization of corrective actions and determined that the licensee properly prioritized those actions.
c.
Establishment of schedule for implementing and completing corrective actions The inspectors review of the licensees scheduling of the corrective actions has determined that the actions were performed and are still being performed according to the risk significance of the equipment involved. Actions are being completed within stated scheduled deadlines. Several corrective actions involve permanent plant modifications and are properly scheduled for upcoming Unit 1 and Unit 2 refueling outages.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees corrective actions serve to prevent a reactor trip resulting from a single point failure. The licensees qualitative measurement of the effectiveness of their corrective action is to have no reactor trips due to single point failures in these control circuits.
Automatic Reactor Trip due to Steam Generator High Level a.
Appropriateness of corrective actions The licensee took immediate and proper corrective actions to repair the failed Inverter 1202 and to determine the root cause. After the July 7, 2002, loss of Inverter 1202, which caused the Unit 2 reactor trip, the licensee replaced the blown fuse and performance tested the inverter satisfactorily. The failure could not be duplicated and operation of the Unit was commenced. On August 7, 2002 the same inverter failed again while swapping battery chargers. After this second event, the licensee performed a more through investigation which included the use of fault tree analysis for the failure,
replaced components (among these were the blown fuse and the PCB, serial number A0158), and performance tested the inverter after the replacements. The PCB was sent to a test laboratory for analysis.
Unit 2, and Inverter 1202, operated successfully until its scheduled outage in the Fall of 2002. While Unit 2 was shutdown, the licensee continued with their diagnostic testing on Inverter 1202. An independent anomaly was discovered during testing on the newly replaced gating PCB. This PCB was replaced, again, by the newly refurbished and vendor tested PCB (serial number A0158). Testing continued and while swapping battery chargers, the fuse 1FU failed, which caused the inverter to fail. This PCB (serial number A0158) was removed and replaced. The PCB (serial number A0158) was sent back to the vendor for a more involved, component analysis and requested that the PCB (serial number A0158) not be returned to the licensee. This gating PCB will not be used again by the licensee.
The inspectors determined that the corrective actions were appropriate.
b.
Prioritization of corrective actions The licensees immediate corrective action restored Inverter 1202 to an operable condition, through fuse replacement and performance testing. The licensee recognized that there were two parts to this event: the procedural aspect and the electrical aspect.
The licensee took prompt action in informing and training the operations staff as to the procedural role in the unit trip. The effectiveness of the procedural corrective action was most evident when the same inverter failed one month later, on August 7, 2002, and the unit did not trip as before.
The licensees actions after the first failure were thorough enough given the intermittent nature of the failed gating PCB. The actions after the second failure were much more involved and resulted in reliable plant operation unit the Fall 2002 outage. Continued troubleshooting and diagnostic activities were properly scheduled according to their potential risk impact on plant operation.
The inspectors reviewed the licensees prioritization of the corrective actions and have determined that the licensee properly prioritized those actions.
c.
Establishment of schedule for implementing and completing corrective actions The inspectors review of the licensees scheduling of the corrective actions has determined that the actions were performed in accordance to the component risk and practical implementation. All actions necessary for Unit 2 startup were verified completed in a timely fashion as stated in Licensee Event Report (LER) 2002-03.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence
The licensees corrective actions served to prevent a reactor trip resulting from a generic issue concerning the PCB installed in the instrument channel inverters. The licensees qualitative measurement of the effectiveness of their corrective action is to have no reactor trips due failed PCB in the inverters or improper instrument channel groupings.
Manual Reactor Trip due to Turbine Blade Failure a.
Appropriateness of corrective actions The licensee specified 46 individual actions associated with the turbine blade failure in Condition Report 02-19072. The actions taken to correct the torsional vibration condition in the turbine generator system included machining the outlet face of the turbine disk that supports the L-0 blades in the low pressure turbines. To de-tune other susceptible turbine rows, mass was added to the three jackshafts located between low pressure turbines 21, 22, and 23 and low pressure turbine 21 and the main generator.
The purpose of these modifications was to move the natural frequency of the disc-rotor combination away from 120 Hz.
The inspectors considered these actions to be well considered and documented. The basis for these modifications was data taken from the operating turbine generator and analysis of the as-built configuration.
b.
Prioritization of corrective actions Once the licensee established the root cause of the failure, corrective actions were developed and implemented prior to plant restart. Additional corrective actions are planned to replace the incorrectly manufactured generator rotor in Unit 2 and restore the Unit 1 and Unit 2 turbine generators to their original design. It is the licensees desire to maintain both turbine generators in the same configuration to allow interchangeability of replacement parts.
The inspectors considered the corrective actions to have been appropriately prioritized.
c.
Establishment of schedule for implementing and completing corrective actions All corrective actions were completed prior to restart of the plant. The longer term actions to return the turbine generators to their original design is an economic consideration regarding replacement parts.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence
A blade vibration monitoring system and a torsional vibration monitoring system was installed on the Unit 2 turbine generator to monitor the operational conditions of the equipment. Data from the torsional vibration monitoring system confirm that the root cause had been effectively corrected. The blade vibration monitoring system failed to provide useful data and was therefore not incorporated in the long term.
The inspectors considered the measures to determine the effectiveness of the corrective actions to be adequate.
MANAGEMENT MEETINGS
Exit Meeting Summary
The results of the supplemental inspection were presented to Mr. J. Sheppard, President and CEO, and other members of licensee management and staff on August 21, 2003.
The inspectors asked the licensee representatives whether any materials examined during the inspection should be considered proprietary. Proprietary information was reviewed and returned to the licensee at the end of the inspection.
ATTACHMENT
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- W. Bealefield, Senior Staff Specialist
- J. Cook, Supervisor, Engineering Specifications
- J. Crenshaw, Manager, Systems Engineering
- D. Dayton, Systems Engineer
- R. Harper, Associate Senior Design Engineer
- E. Halpin, Manager, Plant General
- T. Hayward, Unit Supervisor, Operations
- S. Head, Manager, Licensing
- B. Jenewein, Supervisor, Generation Systems Engineering
- T. Jordan, Vice President, Engineering and Technical Services
- W. Jump, Manager, Training
- M. Kanavos, Manager, Mods and Design Basis Engineering
- A. Khosla, Liaison, Co-owner
- D. Leazar, Manager, Fuels and Analysis
- M. McBurnett, Manager, Quality and Licensing
- M. Meier, Manager, Generation Station Support
- G. Parkey, Vice President, Generation
- K. Richards, Director, Outage
- D. Rencurrel, Manager, Operations
- M. Ruvalcaba, Acting Supervising Engineer
- R. Savage, Senior Staff Specialist
- P. Serra, Manager, Plant Protection
- J. Sheppard, President & CEO
- S. Thomas, Manager, Plant Design Engineering
- D. Towler, Manager, Quality
- T. Walker, Manager, Engineering And Support Services and Quality
- R. Wright, System Engineer
NRC Personnel
- J. Cruz, Senior Resident Inspector
G.Guerra, Resident Inspector
Documents Reviewed
Condition Reports (CR):
CR-98-853
CR-99-12878
CR-00-1181
CR-00-10937
CR-01-7068
CR-02-8678
CR-02-9142
CR-02-9755
CR-02-10007
CR-02-10215
CR-02-10917
CR-02-11228
CR-02-11233
CR-02-14153
CR-02-15289
CR-02-17816
CR-02-19072
CR-02-19206
CR-03-962
CR-03-2487
CR-03-4794
Preventative Maintenance (PM):
PM-02000481
PM-02000482
Condition Report Work Order (CRWO):
CRWO-400312
CRWO-417984
CRWO-422453
CRWO-422457
CRWO-410879
CRWO-410880
CRWO-410881
CRWO-410882
Procedures:
0POP03-SP-0008A, SSPS Train A Slave Relay Test (Outputs Blocked), Revision 11
0POP03-SP-0008B, SSPS Train B Slave Relay Test (Outputs Blocked), Revision 12
STP Nuclear Operating Company Strategic Performance Improvement Plan (7/23/2003)
Engineering Evaluations:
OPGP04-ZA-0002
Conditional Core Damage Probability
CR
Condition Report
Feedwater Isolation Valve
LER
Licensee Event Report
MFIV
Main Feedwater Isolation Valve
NRC
Nuclear Regulatory Commission
PCB
Printed Circuit Board
Silicon Controller Rectifiers
SMT
Senior Management Team