IR 05000400/2003009
| ML032930029 | |
| Person / Time | |
|---|---|
| Site: | Harris  |
| Issue date: | 10/17/2003 |
| From: | Fredrickson P NRC/RGN-II/DRP/RPB4 |
| To: | Scarola J Carolina Power & Light Co |
| References | |
| IR-03-009 | |
| Download: ML032930029 (17) | |
Text
October 17, 2003
SUBJECT:
SHEARON HARRIS NUCLEAR POWER PLANT - NRC SUPPLEMENTAL INSPECTION REPORT NO. 05000400/2003009
Dear Mr. Scarola:
By letter dated August 27, 2003, you were informed that the Nuclear Regulatory Commission (NRC) would conduct a supplemental inspection at your Shearon Harris Nuclear Power Plant for a White performance indicator in the initiating events cornerstone. On September 19, the NRC completed this supplemental inspection. The enclosed report documents the inspection results that were discussed with you and other members of your staff on September 19, 2003.
The purpose of this supplemental inspection was to examine your problem identification, root cause and extent-of-condition evaluation, and corrective actions associated with a White performance indicator in the initiating events cornerstone. The White performance indicator involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the second quarter of calendar year 2003. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.
Based on the results of this inspection, the NRC determined that the problem identification, root cause and corrective actions for the White performance indicator were adequate. The inspectors did not find common cause aspects linking the four reactor scrams from a risk perspective. No findings of significance were identified during this inspection.
In accordance with 10CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system
(ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA/
Paul E. Fredrickson, Chief Reactor Projects Branch 4 Division of Reactor Projects Docket No. 50-400 License No. NPF-63
Enclosure:
NRC Inspection Report 05000400/2003009 w/Attachment : Supplemental Information
REGION II==
Docket No:
License No:
50-400 NPF-63 Report No:
05000400/2003009 Licensee:
Carolina Power & Light Company Facility:
Shearon Harris Nuclear Power Plant, Unit 1 Location:
5413 Shearon Harris Road New Hill, NC 27562 Dates:
September 15 -19, 2003 Inspector:
G. MacDonald, Senior Project Engineer Approved by:
Paul E. Fredrickson, Chief Reactor Projects Branch 4 Division of Reactor Projects
SUMMARY OF FINDINGS
IR 05000400/2003-009; 09/15/2003 - 09/19/2003; Shearon Harris Nuclear Power Plant, Unit 1; supplemental inspection IP 95001 for a White performance indicator in the initiating events cornerstone.
This inspection was conducted by a senior project engineer. No findings of significance were identified. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
NRC-Identified and Self-Revealing Findings
Cornerstone: Initiating Events
This supplemental inspection was conducted to assess the licensees evaluation associated with a White performance indicator in the initiating events cornerstone. The White performance indicator involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the second quarter of calendar year 2003. Specifically, the licensee experienced three reactor trips during the first two quarters of 2003 and also one reactor trip in the third quarter of 2003. The first reactor trip, which occurred on May 18, 2003, was an automatic trip from approximately 27 percent reactor power most likely caused by an equipment failure associated with the main turbine generator electrical overspeed protection circuit. The second reactor trip, which occurred on May 20, 2003, was a manual trip from approximately 20 percent reactor power caused by an equipment failure associated with a condensate booster pump. The third reactor trip, which occurred on June 14, 2003, was a manual trip from approximately 100 percent reactor power caused by an equipment failure associated with a main feedwater pump. The fourth reactor trip, which occurred on August 17, 2003, was a manual reactor trip from approximately 100 percent reactor power caused by an equipment failure of a condensate pump.
The licensees problem identification, root cause and extent-of-condition evaluations, and corrective actions for the four reactor trips were adequate. Common cause aspects linking the four reactor trips from a risk perspective were not evident.
Licensee-Identified Violations
None
REPORT DETAILS
INSPECTION SCOPE
The purpose of this supplemental inspection was to assess the licensees evaluation associated with a White performance indicator in the initiating events cornerstone of the reactor safety strategic performance area. The White performance indicator involved crossing the threshold from Green to White for the Unplanned Scrams per 7,000 Critical Hours Performance Indicator in the second quarter of calender year 2003. Specifically, the Unit experienced three reactor trips during the first two quarters of 2003 and one trip during the third quarter of 2003. The first reactor trip, which occurred on May 18, 2003, was an automatic trip from approximately 27 percent reactor power most likely caused by an equipment failure associated with the main turbine generator electrical overspeed protection circuit. The second reactor trip, which occurred on May 20, 2003, was a manual trip from approximately 20 percent reactor power caused by an equipment failure associated with the A condensate booster pump (CBP). The third reactor trip, which occurred on June 14, 2003, was a manual trip from approximately 100 percent reactor power caused by an equipment failure associated with the B main feedwater pump (MFP). The fourth reactor trip, which occurred on August 17, 2003, was a manual reactor trip from approximately 100 percent power caused by an equipment failure of the A condensate pump.
EVALUATION OF INSPECTION REQUIREMENTS 02.01 Problem Identification a.
Determination of who identified the issue and under what conditions The four reactor trips were self revealing events which occurred during the course of normal operational conditions.
The May 18 automatic reactor trip occurred during the process of unit startup. The most likely cause was that the main turbine generator electrical overspeed protection speed probe generated a spurious overspeed signal which automatically tripped the turbine and the reactor was automatically tripped on the turbine-trip-reactor-trip signal.
The May 20 trip occurred during the process of unit startup when the reactor was manually tripped in response to a trip of the operating MFP in response to a failure and trip of the A CBP. The cause of the trip of the A CBP was due to variable speed fluid coupling (VSFC) low oil pressure caused by failure of the internal oil pump (IOP) pinion gear.
The June 14 trip occurred during normal operations when the reactor was manually tripped in response to an automatic trip of the B MFP. The MFP trip was caused by a failure of a bistable comparator card in the MFP flow control system which generated a spurious low-flow trip signal to the B MFP.
The August 17 trip occurred during normal operations when the reactor was manually tripped in response to a loss of the A MFP caused by a failure of the A condensate pump (CP) motor.
b.
Determination of how long the issue existed, and prior opportunities for identification The May 18 automatic reactor trip most likely resulted from a spurious electrical overspeed signal from the main turbine electrical overspeed probe. The licensee was not able to definitively verify the cause of the trip but determined the most likely cause by process of elimination. The turbine electrical speed signal comes from a single probe. Two speed probes are installed but only one is wired into the overspeed circuit.
The other speed probe is an installed spare. The original speed probe was in service until April 26, 2003, when it was found to have no signal during its channel calibration during refueling outage eleven (RFO-11). The previous satisfactory check of the original speed probe was on December 17, 2001 during RFO-10. The original probe was replaced with a new speed probe which was tested and became the spare probe.
The original spare probe was tested and placed in service. This original spare probe was the one which most likely failed in a manner which generated a spurious overspeed signal resulting in the May 18 trip. Following the May 18 trip, the licensee wired the new spare probe into service. The second speed probe failure occurred on May 18, 2003 resulting in the reactor trip. Because the licensee was not able to remove the second failed probe for examination and failure mode analysis due to plant conditions, removal was planned for RFO-12, which is scheduled for November 2004. The inspector concluded that the first speed probe failure on April 26 represented a prior opportunity to identify that the main turbine electrical overspeed protection speed probe was a single-point reactor trip vulnerability.
The May 20 manual reactor trip was the result of the trip of the single operating MFP.
The A MFP tripped in response to an unplanned trip of the A CBP due to low oil pressure in the VSFC caused by IOP pinion gear failure. The VSFC had been refurbished by a non-original equipment manufacturer (OEM) vendor. The refurbishment included replacement of the IOP pinion gear with a non-OEM gear. The licensee determined that the vendor did not perform a run-in of the new gear and the gear failed due to gear overload. During RFO-10, in 2001, a similar failure occurred with the A CBP. Prior to this 2001 failure, the VSFC also had been refurbished by the non-OEM vendor including pinion gear replacement with a non-OEM gear. During the 2001 failure, both the IOP and the auxiliary oil pump were being simultaneously operated by a test procedure which installed jumpers to allow dual pump operation. The 2001 licensee failure evaluation concluded that the gear failed due to overload caused by operating against higher pressure with dual pump operation. This 2001 failure represented a prior opportunity to determine that the IOP design had little margin to failure and that loss of a CBP was a potential single-point vulnerability that could necessitate a manual reactor trip on loss of one MFP.
The June 14 reactor trip was caused by a failure of the Westinghouse NAL2 bistable comparator card in the B MFP flow control system. A transistor on the card failed which generated spurious low-flow alarm and trip signals to the B MFP. The inspector reviewed the vendor manual and verified that no preventive maintenance or card replacement intervals were specified other than the routine calibration. There was no evidence to suggest that the failure was due to aging or lack of maintenance and the inspector concluded that the June 14 trip was the result of a random electrical equipment failure that occurred at or near the time of the trip. Prior opportunities for identification of this transient were therefore limited.
The August 17 reactor trip was the result of a failure of the A CP motor. The A CP motor failed in 1999 (different motor than 2003) and the conclusion was that the failure was due to a large voltage surge, most likely due to a phase-to-phase internal fault.
The evaluation of the initiating event or condition which caused the 1999 motor failure was indeterminate. The 1999 motor was then replaced (by the motor that failed on August 17, 2003). Since the 1999 motor failure cause was indeterminate, the extent of condition was limited to the failed motor itself. There were no recurrence control corrective actions following the 1999 failure so the corrective actions from the 1999 failure would not have prevented the 2003 motor failure. During the evaluation of the 2003 motor failure the licensee determined that the equipment grounding for the A CP was inadequate to protect against lightning surges. The evaluation following the 1999 failure did not examine the adequacy of the equipment grounding for the A CP. The 1999 motor failure was a prior opportunity to have identified the grounding system weakness with the A CP and to recognize that CP motor failure represented a potential single-point reactor trip vulnerability that could necessitate a manual reactor trip on loss of one MFP.
c.
Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues The licensees evaluation assigned a change in core damage frequency of 7.1E-7 to the four reactor trips. The inspectors reviewed the licensees evaluation and assumptions.
No compliance issues were identified.
02.02 Root Cause and Extent-of-Condition Evaluation a.
Evaluation of methods used to identify root causes and contributing causes The licensee used combinations of different methods to identify root and contributing causes for the four reactor trips; interviews, timelines, failure mode analysis, equipment performance analysis, barrier analysis, event and causal factor analysis, change analysis and fault tree analysis. The methods and combinations of methods used to identify root and contributing causes for the four reactor trips were appropriate.
For the May 18 trip evaluation, the licensee utilized iterative failure mode analysis to arrive at potential failure modes and used cause and effects analysis to support and refute the potential failure modes and discern the most likely root and contributing causes.
For the May 20 trip evaluation, the licensee utilized timelines, events and causal factors analysis, fault tree analysis for equipment failures and personnel interviews for human factors including input from vendors and the OEM.
For the June 14 trip evaluation, the licensee utilized timelines, equipment performance analysis, fault tree analysis and interviews to evaluate the root/contributing causes and extent of condition.
For the August 17 trip evaluation, the licensee utilized failure mode analysis, event and causal factors analysis, barrier analysis, and independent expert review of the issue from the vendor and several engineering corporations (Sargent and Lundy and General Electric).
b.
Level of detail of the root cause evaluation For the four reactor trips, the root cause evaluations were of sufficient detail to support the identified root and contributing causes.
For the May 18 trip, fifteen electrical failure modes and fourteen mechanical failure modes were analyzed. The failure evaluation was an iterative elimination analysis. The turbine overspeed annunciation did not actuate on the spurious overspeed signal. The licensees evaluation of the May 18 trip determined that the trip was most likely caused by a main turbine electrical overspeed probe failure, however the licensee had not yet performed a failure analysis of the speed probe which failed on May 18. The licensee did analyze the speed probe which was found failed on April 26, however the two speed probes did not fail in a manner which produced the same results. The licensee determined that the root cause for the May 18 trip was that both electrical overspeed trip circuits were fed from a single speed probe and thus a speed probe failure was a single-point reactor trip vulnerability. The level of detail of the root cause evaluation for the May 18 trip was adequate to support the root and contributing causes but was limited by the fact that the speed probe which failed on May 18 was not available for forensic examination and thus the licensee could not determine how the spurious overspeed signal was generated. The examination of the speed probe which failed on May 18 was scheduled for November 2004, during RFO-12.
For the May 20 trip, the licensee determined that the trip was caused by a trip of the A CBP caused by IOP pinion gear failure. The evaluation of the failed gear determined that the gear failed due to low cycle fatigue from operation at high loads without gear run-in. The IOP pinion gear was a weak link in the VSFC as it had limited design margin even under steady state conditions. Input from the OEM indicated that gear dimensions are critical and that a probable cause might be a mismatched gear. The OEM also indicated that this problem was not uncommon with non-OEM gears. The CBP VSFC had been refurbished earlier including an oil system filtration upgrade which increased the load on the IOP. The IOP pinion gear had also failed during 2001 (RFO-10). Both the 2001 and 2003 pinion gear failures were failures of non-OEM gears. The two CBP VSFCs which are currently in service have not had their pinion gears replaced.
The licensee concluded that the root cause was the VSFC oil system modifications which increased pinion gear load and the failure to perform gear run-in on the new pinion gear. The inspectors review concluded that the root cause of the May 20 gear failure (IOP pinion gear overload), was supported by the licensees evaluation, however no basis was provided to eliminate gear geometry (non-OEM gears) as a potential contributing cause.
For the June 14 trip, the licensee determined that the root cause was a random failure of a bistable comparator card failure caused by a shorted transistor. The inspectors determined that the level of detail for this root cause evaluation was appropriate.
For the August 17 trip, the licensee determined that the A CP motor failed when a lightning voltage surge exceeded the dielectric strength of the motor winding insulation and the equipment grounding system was not effective at protecting the A CP motor from a lightning strike. The inspectors determined that the evaluation of this root cause analysis was sufficiently detailed to support the root and contributing causes.
c.
Consideration of prior occurrences of the problem and knowledge of prior operating experience The root cause evaluations for the four reactor trips did consider prior occurrences of similar problems where applicable.
d.
Consideration of potential common causes and extent of condition of the problem The inspectors review of the 4 reactor trips verified that there were no equipment or system related common causes which could be linked to the failures. All 4 trips were caused by non-safety related equipment failures and all four represented single-point reactor trip vulnerabilities. The inspectors also concluded that the licensee had correctly assessed the extent of condition for the equipment failures which contributed to the four reactor trips.
The licensee performed a common cause analysis of six previous reactor trips: January 2, 2002, July 13, 2002, August 15, 2002, May 18, 2003, May 20, 2003, and June 14, 2003. The analysis, contained within AR 103534103534 did not review the trip of August 17, 2003. The analysis identified two non-equipment / system common causes and two contributing factors. The two common causes were: an over-reliance on the licensees ability to predict non-safety related critical component failures, and that single-point vulnerabilities exist on non-safety related equipment that could result in plant trips. The contributing factors were: the licensee has an over-reliance on vendors, and that the potential consequences of emergent non-safety related equipment issues were not properly assessed. The inspectors reviewed the common cause assessment and verified that the conclusions were valid for the four reactor trips reviewed in this inspection report.
As a management initiative, the licensee performed a single point vulnerability study which utilized the sites operating experience and examined the details and circumstances leading to all the reactor trips which had occurred. The study also determined the causes and equipment which contributed to the trips. This study was very comprehensive and thorough and formed the cornerstone for the plant changes in operating procedures, preventive maintenance and plant modifications to minimize reactor trip vulnerabilities and improve equipment reliability. The inspectors reviewed the proposed changes which included four procedure revisions, nine preventive maintenance revisions and twelve plant modifications and noted that all the single-point vulnerabilities in the four trips reviewed in this inspection report were included in the proposed modifications.
02.03 Corrective Actions a.
Appropriateness of corrective action(s)
The licensee took prompt corrective actions to repair the equipment failures related to the reactor trips. Comprehensive corrective actions to address root and contributing causes, where appropriate, were performed or scheduled to be performed.
To address the May 18 trip, the licensee immediate corrective action was to replace the failed speed probe with a new speed probe which had been in the spare slot. Additional corrective actions planned included: modifications to the main turbine electrical overspeed circuit to remove the single-point vulnerability to speed probe failures, modification to the overspeed relay logic to ensure proper annunciation, revision to applicable operator rounds to periodically monitor turbine electrical speed indications at the main turbine emergency trip cabinet, forensic analysis of the failed speed probe during RFO-12, and review of the turbine protection system for other single-point design vulnerabilities.
To address the May 20 trip, the licensee performed an immediate corrective action to replace the VSFC with the original VSFC which did not have non-OEM gears.
Recurrence control corrective action was to specify that the pinion gear set be run-in in accordance with industry recommendation. Additional corrective actions included the following: 1. evaluating options for modifications to the VSFC to recover operating margin, including modifying the system so that loss of the IOP would auto-start the auxiliary oil pump and not trip the CBP; 2. reviewing the engineering change process; 3.
performing a training needs analysis for operations and technical training; and 4.
coordinating with outage and scheduling to create a secondary recovery coordinator position.
The immediate corrective actions for the June 14 trip was to replace the failed NAL2 bistable comparator card. An additional corrective action was a modification to the MFP control system such that a single failure of a NAL2 comparator card would not cause a reactor trip.
To address the August 17 trip, the licensee replaced the motor as immediate corrective action. As corrective action to prevent recurrence, the licensee installed surge protection on the A CP motor and planned to install cable tray ground conductor protection from the A CP to the switchgear. Additional corrective actions included surge protection for the B CP motor and completion of a validation of the adequacy of all the high/medium voltage motor equipment grounding for lightning protection.
b.
Prioritization of corrective actions Corrective actions for the four reactor trips were properly prioritized.
c.
Establishment of a schedule for implementing and completing the corrective actions The inspector verified that the licensees corrective action program identified assigned individuals, completion dates, and reference numbers to ensure that individual corrective actions would be completed in accordance with their priority.
d.
Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The inspector determined that effectiveness reviews had been scheduled for all 4 trips.
OTHER ACTIVITIES
03.01 Event Followup a.
(Closed) Licensee Event Report (LER) 05000400/2003001-00, Automatic Turbine Trip -
On May 18, 2003, an automatic reactor trip occurred which was most likely caused by a spurious main turbine electrical overspeed signal which automatically tripped the turbine resulting in a turbine trip-reactor trip. The licensee replaced the failed speed probe with a new speed probe which had been installed in the spare slot. The licensee documented the failed equipment in AR 93855. The inspector reviewed the licensees corrective actions delineated in the LER and determined that the actions were adequate.
The corrective actions were either complete or scheduled to be completed in accordance with the licensees corrective action program. No findings of significance were identified by the inspectors review of this LER. This LER is closed.
b.
(Closed) LER 05000400/2003002-00, Manual Reactor Trip Due to Trip of A Condensate Booster Pump.
On May 20, 2003, the A MFP tripped due to a trip of the A CBP which was caused by a failure of the VSFC IOP pinion gear. The licensee replaced the VSFC. The failed equipment was documented in AR 93981. The inspector reviewed the licensees corrective actions delineated in the LER and determined that the actions were adequate.
The corrective actions were either complete or scheduled to be completed in accordance with the licensees corrective action program. No findings of significance were identified by the inspectors review of this LER. This LER is closed.
c.
(Closed) LER 05000400/2003003-00, Manual Reactor Trip Due to Loss of B Main Feedwater Pump.
On June 14, 2003, the B MFP tripped on a spurious low flow signal caused by a failed comparator card in the feedwater flow instrument loop circuit. The licensee replaced the failed card and calibrated the flow instrument loop components. The licensee documented the failed equipment in AR 96156. The inspector reviewed the licensees corrective actions delineated in the LER and determined that the actions were adequate.
The corrective actions were either complete or scheduled to be completed in accordance with the licensees corrective action program. No findings of significance were identified by the inspectors review of the LER. This LER is closed.
d.
(Closed) LER 05000400/2003005-00, Manual Reactor Trip Following a Trip of the A Condensate Pump Motor.
On August 17, 2003, the A CP tripped following an electrical motor short due to a lightning voltage surge that overcame the dielectric strength of the motor winding. The failed equipment was documented in AR 103182103182 The inspector reviewed the licensees corrective actions delineated in the LER and determined that the actions were adequate.
The corrective actions were either complete or scheduled to be completed in accordance with the licensees corrective action program. No findings of significance were identified by the inspectors review of the LER. This LER is closed.
MANAGEMENT MEETINGS
Exit Meeting Summary
The inspector presented the inspection results to Mr. J. Scarola, Vice President - Harris Plant, and other members of licensee management at the conclusion of the inspection on September 19. The inspector confirmed that proprietary information was not provided or examined during the inspection.
SUPPLEMENTAL INFORMATION
KEY POINTS OF CONTACT
Licensee Personnel
- J. Bouchard, Supervisor, Electrical and I&C Systems Engineering
- J. Caves, Licensing Supervisor
- J. Connor, Supervisor, Balance of Plant Systems
- J. Dills, Superintendent Technical Services
- F. Diya, Acting Engineering Manager
- R. Duncan, Director of Site Operations
- W. Gurganious, Manager Nuclear Assessment Section
- G. Miller, Maintenance Manager
- T. Morton, Manager Support Services
- S. OConnor, Superintendent, Design Engineering
- P. Oakley-Lisk, Harris Nuclear Plant Communications
- J. Scarola, Harris Plant Vice President
- M. Wallace, Senior Specialist, Licensing
- E. Wills, Operations Manager
- J. Yadusky, Engineer, Licensing
NRC Personnel
- R. Musser, Senior Resident Inspector, Harris
- P. Fredrickson, Branch Chief, Division of Reactor Projects, Region II
ITEMS OPENED, CLOSED, AND DISCUSSED
Opened
None
Opened and Closed
None
Closed
- 05000400/2003001-00 LER Automatic Turbine Trip - Reactor Trip (Section 03.01).
- 05000400/2003002-00 LER Manual Reactor Trip Due toTrip of A Condensate Booster Pump (Section 03.01).
- 05000400/2003003-00 LER Manual Reactor Trip Due to Loss of B Main Feedwater Pump (Section 03.01)
- 05000400/2003005-00 LER Manual Reactor Trip Following a Trip of the A Condensate Pump Motor (Section 03.01).
Discussed
None