IR 05000280/2002008
| ML021280594 | |
| Person / Time | |
|---|---|
| Site: | Surry  |
| Issue date: | 05/08/2002 |
| From: | Landis K NRC/RGN-II |
| To: | Christian D Virginia Electric & Power Co (VEPCO) |
| References | |
| EA-01-235 IR-02-008 | |
| Download: ML021280594 (17) | |
Text
May 8, 2002 EA-01-235 Virginia Electric and Power Company ATTN: Mr. David A. Christian Sr. Vice President and Chief Nuclear Officer Innsbrook Technical Center - 2SW 5000 Dominion Boulevard Glen Allen, VA 23060-6711 SUBJECT: SURRY NUCLEAR POWER STATION - NRC SUPPLEMENTAL INSPECTION REPORT NOS. 50-280/02-08 AND 50-281/02-08 On April 12, 2002, the NRC completed an inspection at your Surry Power Station, Units 1 and 2. The enclosed report documents the inspection findings which were discussed on April 11, 2002, with Mr. Blount and other members of your staff.
The inspection examined activities conducted under your license as they relate to safety and compliance with the Commissions rules and regulations and with the conditions of your license.
The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel. Specifically, this inspection reviewed activities associated with a White inspection finding on Units 1 and 2 related to degraded emergency diesel generator (EDG) 3 piston wrist pin bearings and a White Safety System Unavailability - Emergency AC Power Performance Indicator on Unit 2 associated with the EDG 3 degradation in combination with EDG 2 output breaker problems. Activities reviewed included your root cause evaluations and associated corrective actions. The White finding and performance indicator was in the mitigating systems cornerstone of the reactor safety strategic performance area.
Based on this inspection, we have concluded that your evaluation of performance deficiencies was adequate and corrective actions were appropriately prioritized and consistent with the identified root cause and contributing factors.
No findings of significance were identified during the inspection.
In accordance with 10 CFR 2.790 of the NRCs Rules of Practice, a copy of this letter and its enclosure will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRCs document system
VEPCO 2 (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
Sincerely,
/RA L. Garner for/
Kerry D. Landis, Chief Reactor Projects Branch 5 Division of Reactor Projects Docket Nos.: 50-280, 50-281 License Nos.: DPR-32, DPR-37
Enclosure:
Supplemental Inspection Report No /02-08 and 50-281/02-08
REGION II==
Docket Nos.: 50-280, 50-281 License Nos.: DPR-32, DPR-37 Report Nos.: 50-280/02-08, 50-281/02-08 Licensee: Virginia Electric and Power Company Facility: Surry Power Station, Units 1 and 2 Location: 5850 Hog Island Road Surry, VA 23883 Dates: April 8 - 12, 2002 Inspector: R. Moore Approved by: K. Landis, Chief, Reactor Projects Branch 5 Division of Reactor Protection Attachment: Supplemental Information
SUMMARY OF FINDINGS Surry Power Station Units 1 and 2 IR 05000280-02-08, IR 05000281-02-08, on 4/8-12/2002; Virginia Electric and Power Co.; Surry Power Station Units 1 & 2. Supplemental Inspection Report for degraded mitigating systems cornerstone.
The inspection was conducted by a regional engineering inspector. The inspection identified no findings of significance. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process, Revision 3, dated July 2000.
Cornerstone: Mitigating Systems This supplemental inspection was performed by the NRC to assess the licensee's evaluation and corrective actions associated with a low to moderate risk significance (White) finding applicable to Units 1 and 2 and a Unit 2 White performance indicator (PI). The White finding and PI are in the mitigating systems cornerstone in the reactor safety strategic performance area. The White finding is described in NRC Final Significance Determination letter dated December 21, 2001, and was associated with the Emergency Diesel Generator (EDG) 3 wrist pin failures. The PI, Safety System Unavailability - Emergency AC Power, crossed the White threshold in the third quarter of calendar year 2001 and remained there through the current quarter. The White PI resulted mainly from the EDG 3 piston wrist pin failures and from EDG 2 output breaker problems. During this supplemental inspection, which was performed in accordance with Inspection Procedure 95001, the inspector determined that the licensee performed an overall adequate evaluation of performance deficiencies related to the EDG 3 piston wrist pin failure and the EDG 2 output breaker problems. The depth of root cause evaluations was adequate. The corrective actions were appropriately prioritized and consistent with the identified root cause and contributing factors and provided reasonable assurance to prevent recurrenc Report Details Inspection Scope This supplemental inspection was performed by the NRC to assess the licensees evaluation and corrective actions associated with a low to moderate risk significance (White) finding applicable to Units 1 and 2 and a Unit 2 White performance indicator (PI). The White finding and PI were in the mitigating systems cornerstone in the reactor safety strategic performance area. The White finding was described in NRC Final Significance Determination letter dated December 21, 2001, and was associated with the Emergency Diesel Generator (EDG) 3 wrist pin failures. The PI, Safety System Unavailability - Emergency AC Power, crossed the White threshold in the third quarter of calendar year 2001 and remained there through the recent quarter. The White PI resulted mainly from the EDG 3 piston wrist pin failures and from EDG 2 output breaker problems. The performance issues were previously described in NRC Inspection Reports Nos. 50-280, 281/01-06 and 50-280, 281/01-07. The inspection involved a review of the licensees evaluation of the White finding and PI with respect to problem identification, root cause analysis and establishment of corrective action The inspectors assessed the adequacy of the licensees root cause evaluation by determining if the root causes and contributing causes were understood, and if the resulting corrective actions were sufficient to address those causes in order to prevent recurrence. This assessment included: a review of the licensees root cause evaluations (RCEs), vendor engineering analyses, plant issue reports which documented the identified deficiencies and other related/referenced documents, and interviews with licensee personnel involved in the root cause evaluations. Evaluation of Inspection Requirements EDG 3 Wrist Pin Failure 2.1.1 Problem Identification Determination of who (i.e., licensee, self-revealing, or NRC) identified the issues and under what conditions The degraded/failed EDG 3 piston wrist pin condition was identified by the licensee during inspection of the wrist pins and documented in Plant Issue S-2001-1168, dated April 23, 2001. The wrist pin inspection was conducted as a result of elevated silver content levels in periodic lube oil analysis which were documented in Plant Issue S-2001-0872, dated March 22, 2001. The inspector concluded that the licensees evaluation adequately determined who and under what conditions the degraded equipment condition was identifie Determination of how long the issues existed, and prior opportunities for identification The inspector reviewed the plant issues and root cause analyses associated with the EDG 3 piston wrist pin issue and discussed development of this issue with the licensee staff involved in the identification and analysis of the condition. An adverse trend of
increasing silver content in the EDG 3 lube oil began in March 2000, following a change in the lube oil manufacturer. Increasing silver content in monthly lube oil samples and exceeding the vendor recommended threshold for silver content in October of 2000 provided previous opportunities to identify the developing condition. The inspector concluded that the licensees evaluation provided an adequate determination of how long the issue existed and prior opportunities for identificatio Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues In the licensees response letter, Risk Assessment of Surry EDG No. 3 Degradation, dated November 26, 2001, to the NRCs Letter of Preliminary Yellow Finding, dated October 11, 2001, the licensee documented their risk assessment and compliance concerns on this issue. The licensee used a current updated Surry probabilistic risk assessment (PRA) model for internal initiating events which credited additional deterministic analyses of the EDGs. This included a more detailed PRA model and updated information not used in the NRCs preliminary Phase III SDP analysis. In particular, consideration was given to the impact of successful monthly two hour surveillance runs for the period that EDG 3 was degrading which increased the probability of offsite power recovery. The result was that the delta core damage frequency increase, due to the EDG 3 degraded condition, was between 1E-5 and 1E- The inspector concluded that the licensees evaluation adequately addressed the plant specific risk consequences and compliance concerns associated with the issues.
2.1.2 Root Cause and Extent of Condition Evaluation Evaluation of methods used to identify root causes and contributing causes The licensee used a combination of structured root cause analysis techniques to evaluate this white finding, including Event and Causal Factor Charting, Interviewing, Walkthrough Task Analysis, and Change Analysis. The licensee performed a Category 2 RCE to address the degraded piston wrist pin condition and extent of condition application to EDG 1 and 2, and a Category 1 RCE to address the failure to identify the adverse trend in EDG lube oil analysis results prior to development of the degraded wrist pin condition. These RCEs were performed and categorized in accordance with Virginia Power Administrative Procedure (VPAP)-1604, Root Cause Evaluation Program, Revision 4. The inspector concluded that the licensee incorporated systemic methodologies in determination of root cause and extent of condition evaluations for the degraded EDG piston wrist pin condition and failure to identify adverse trends in EDG lube oil parameter Level of detail of the root cause evaluation The evaluation of the degraded wrist pin condition included internal and external engineering and laboratory analyses of the degraded components and EDG operating characteristics. The evaluation appropriately considered design and operating characteristics of the EDG and lube oil system. The probable root cause of the degraded wrist pin condition was determined to be inadequate lubrication of the wrist
pins which resulted in excessive wear of the wrist pin bearing components. A contributing factor, based on strong indications from change analysis, was the change in lube oil manufacturer in March 2000. This was documented in Category 2 RCE S-2001-1168, dated December 1, 200 The root cause for the performance deficiency of failing to identify the adverse trend in EDG lube oil analysis reports was determined to be inadequate change management when the licensee changed from a commercial test laboratory to the licensees test laboratory for lube oil analysis. The RCE determined that organizational and process deficiencies contributed to an environment where inadequate treatment of predictive monitoring resulted in a lube oil analysis program that did not effectively address the health of the EDGs. This was documented in Category 1 RCE S-2001-2806, dated January 10, 200 The inspector concluded that the licensees root cause evaluations were thorough and incorporated an appropriate level of detai Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluation included a review of other General Motors electro-motive division (EMD) users for this condition. Another occurrence was identified at Arkansas Nuclear Unit 1 in 1986. The review also noted that Surry did not implement the pre-lubrication modification as did other EMD users, although this was not an apparent contributor to the degraded wrist pin condition. Non-nuclear applications of EMD diesels were also researched. The inspector concluded that the licensees evaluation appropriately included consideration of prior occurrences and operating experience with the General Motors EMD Model 999-20 diesel generato Consideration of potential common causes and extent of condition of the problem The licensees evaluation in the RCE S-2001-1168 addressed potential common cause of the inadequate lubrication/change in lube oil, i.e., applicability to EDG 1 and 2. The evaluation identified similar, less advanced degradation of wrist pins in EDG 1 and significantly less degradation on EDG 2, based on laboratory analysis of similar power pack components on each EDG. The review of previous lube oil analyses results for these EDGs indicated a similar, less severe adverse trend in silver content on EDG 1 and no adverse trend on EDG 2. The inspector concluded that the licensee included in their evaluation appropriate consideration of potential common cause and extent of condition to the other onsite EDGs.
2.1.3 Corrective Actions Appropriateness of corrective actions The licensee took immediate corrective actions to establish the reliability and operability of all station EDGs. These actions included replacement of all power packs (cylinder, cylinder head, piston, rod) for all three EDGs with new power packs and replacement of lube oil in all EDGs with an oil product used by other nuclear EMD EDG owners. The oil
analysis program was upgraded to assure recognition when analyzed parameters exceed pre-set action levels and correct action levels were incorporated into the program software. The manufacturers specified action levels based on assumed uniform wear were modified to address the potential for non-uniform wear resulting in more conservative action level set points. The lube oil analysis testing was contracted to a commercial test laborator The long term corrective actions that resulted from RCE recommendations address organizational and programmatic weaknesses that contributed to the failure to recognize the adverse trend in EDG oil samples and ensure EDG lubrication system design is adequate to minimize any adverse effects on critical EDG components during standb Additionally, maintenance and monitoring efforts were being optimized to ensure EDG availability and reliability. The inspector concluded that appropriate corrective actions were specified for identified root and contributing causes and the status of these corrective actions was monitore b. Prioritization of corrective actions The corrective actions to resolve risk significant concerns were completed as immediate corrective actions by the return of all station EDGs to an operable and reliable statu c. Establishment of schedule for implementing and completing the corrective actions The completion due dates assigned to corrective actions developed by the root cause evaluation process reflected the prioritization considerations. The compliance issues were addressed by corrective actions to improve the licenses capability to identify and correct adverse conditions via the lube oil analysis program. Completion of corrective actions to ensure continued availability of EDGs are completed or scheduled. The root cause evaluations, RCE S-2001-1168 and RCE S-2001-2806, for the degraded piston wrist pin and the inadequate problem identification function of the EDG lube oil analysis program included completion due dates and status for the approved corrective action The inspector concluded that the licensees evaluation and corrective actions include appropriate prioritization with consideration of risk significance and regulatory compliance. Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence Qualitative measures to determine the effectiveness of corrective actions for the degraded wrist pin condition include specific preventive maintenance requirements for EDG wrist pin inspection, comparison of lead wire readings to previous values, and more conservative action levels for identification of adverse trends in silver content to ensure timely detection of potential wear problems. Measures to ensure effectiveness of the inadequate problem identification due to deficient change management include a requirement for effectiveness reviews when significant organizational or process changes occur at the station. The licensees Root Cause Analysis Program, VPAP-1604, requires annual effectiveness reviews for corrective actions for Category 1 and 2 RCEs. Following discussion with the inspector regarding specific action items within the RCE to address effectiveness, the licensee added an action item in each RCE to assess
the effectiveness of corrective actions. The inspector concluded the licensee had established measures for determining the effectiveness of corrective actions to prevent recurrence. EDG 2 Output Breaker Problems 2.2.1 Problem Identification Determination of who (i.e., licensee, self-revealing, or NRC) identified the issues and under what conditions The initial output breaker failure was self revealing when it failed to operate properly during the EDG monthly start exercise test, 2-OPT-EG-001, performed on September 16, 2001. Licensee trouble shooting did not identify a cause and it was concluded that the breaker was functional after successful bench testing. The breaker was returned to service until September 27, 2001, when it was replaced with a spare breaker and additional investigation was performed on the original breaker. Disassembly and inspection of the breaker on October 12, 2001, identified a broken contact on the anti-pump coi Determination of how long the issues existed, and prior opportunities for identification The degraded output breaker was identified on September 16, 2001. The previous successful monthly EDG surveillance which included breaker operation was on August 19, 2001. The length of time the condition existed with respect to unavailability was based on half the time from the last successful monthly test, which was approximately 15 days. The licensees evaluation identified that the investigation after the September 16, 2001, failure was the single opportunity for prior identification. This failure to identify and correct the equipment problem on September 16, 2001, was the subject of a non-cited violation in NRC Inspection Report Nos. 50-280, 281/01-07. The inspector concluded the licensees evaluation adequately addressed how long the condition existed and prior opportunities for identificatio Determination of the plant-specific risk consequences (as applicable) and compliance concerns associated with the issues The licensees evaluation did not specifically address the risk consequences of this breaker failure. Discussion with the licensee indicated the risk associated with the breaker failure and associated EDG unavailability was incorporated in the determination of the white threshold for the Safety System Unavailability - Emergency AC Power Performance Indicator. This performance indicator was White due to the combination of unavailability time for the EDG 3 piston wrist pin failure and the EDG 2 output breaker problems. The inspectors agreed that the risk was appropriately addressed by the performance indicator. The risk consequences of just the EDG 2 output breaker problem was characterized in NRC Inspection Report Nos. 50-280, 281/01-07 Section 4OA7 as having very low safety significanc .2.2 Root Cause and Extent of Condition Evaluation Evaluation of methods used to identify root causes and contributing causes The licensees evaluation included review of breaker history, review of materials and application, and a detailed laboratory analysis of the failed breaker component. No specific root causes were identified and the evaluation concluded this was an isolated case. The component laboratory analysis was performed by an internal test laborator The root cause evaluation was conducted in accordance with the licensees procedure (VPAP-1604) as a Category 2 RCE. The inspector concluded that the evaluation methodology for investigation of root cause was adequat Level of Detail of the Root Cause Evaluation The level of detail of the initial problem investigation of the failed breaker conducted on September 16, 2001, was inadequate, in that, it did not identify the specific breaker component failure and the breaker was returned to service without repair. This was the subject of a non-cited violation in NRC Report Nos. 50-280, 281/01-07. The second cause investigation initiated on October 12, 2001, was of sufficient detail to identify the failed component. This investigation went beyond bench testing to include disassembly of the breaker control unit, where the faulted contact was located. Laboratory analysis was conducted of the faulted component and the design and operating characteristics for the breaker were reviewed. The most probable cause was determined to be related to a manufacturing or construction flaw. Due to repeated cycling over time from normal plant operation, the flaw propagated until the resultant failure. The inspector concluded that the level of detail of the second root cause evaluation was adequat Consideration of prior occurrences of the problem and knowledge of prior operating experience The licensees evaluation included a review of equipment history for ABB 4160 volt and 480 volt breakers at the station. No previous similar failures were identified. An industry operating experience search was performed and no related failure was identified. The breaker vendor and refurbishment vendors were contacted and they stated this type of failure had not been previously experienced. Based on this information the licensee concluded this type of breaker contact failure was an isolated case. The inspector concluded that the licensees evaluation included appropriate consideration of prior operating experience and previous occurrence Consideration of potential common causes and extent of condition of the problem The licensee identified this as an isolated occurrence so no common cause factors were identified. Regarding extent of condition, the licensee noted this type of control device was common for all 4160/480 volt ABB breakers used in the plant. The inspector concluded that the licensees evaluation included adequate consideration of potential common cause and extent of conditio .2.3 Corrective Actions Appropriateness of corrective actions The short term corrective actions were to replace the problem breaker with a successfully tested spare and inspect the remaining EDG output breakers for contact degradation. The long term corrective actions were to upgrade 4160/480 volt breaker maintenance procedures to include internal visual inspection of new and existing control devices. The inspector reviewed the completed work orders for the EDG output breaker inspections and reviewed the maintenance procedure changes to verify the control device internal inspections were incorporated. As the probable cause was identified to be a manufacturing/construction deficiency, the inspector concluded that the licensees corrective actions appropriately addressed the identified caus b. Prioritization of corrective actions The immediate corrective actions to restore the equipment to a reliable condition were complet c. Establishment of schedule for implementing and completing the corrective actions Long term corrective actions to inspect breaker control devices were scheduled in accordance with routine maintenance schedules and replacement of equipment. The inspector concluded the licensees evaluation included appropriate prioritization and schedule establishment for corrective action Establishment of quantitative or qualitative measures of success for determining the effectiveness of the corrective actions to prevent recurrence The licensees evaluation did not initially include specific measures to determine the effectiveness of corrective actions to prevent recurrence. The licensees Root Cause Analysis Program, VPAP-1604, requires periodic effectiveness reviews for Category 1 and 2 RCEs. This issue was evaluated as a licensee designated Category 2 RCE and therefore, progammatically, the effectiveness of corrective actions would be addresse The inspector reviewed an effectiveness review of a previous RCE and concluded that the review adequately addressed this aspect of the corrective action process. Following discussion with the inspector regarding specific action items within the RCE to address effectiveness, the licensee added an action item in the RCE to assess the effectiveness of corrective actions. The inspector concluded the licensees evaluation included appropriate measures to determine the effectiveness of corrective actions. Management Meetings The inspector presented the inspection results to Mr. Richard Blount and other members of the licensees staff and management on April 11, 2002. The inspector inquired whether any proprietary material were examined during the inspection. No proprietary information was identifie SUPPLEMENTAL INFORMATION KEY POINTS OF CONTACT Licensee W. Adams, Engineering Manager R. Allen, Maintenance Manager G. Bischof, Director Nuclear Engineering R. Blount, Site Vice President L. DePrisco, Materials Site Manager A. Farmer, Electrical and I&C Systems Manager B. Foster, Safety & Licensing Manager L. Hartz, Nuclear Engineering Vice President R. MacManus, Nuclear Design Engineering Manager T. Sowers, O&M Manager ITEMS OPENED, CLOSED AND DISCUSSED Non LIST OF DOCUMENTS REVIEWED Plant Issue S-2001-0872, EDG 3 Lube Oil Samples Indicate Elevated Silver Content Greater Than Vendor Threshold of 2 ppm Plant Issue S-2001-1168, Indication Of Piston Wrist Pin Damage on EDG 1, dated 7/16/01 Category 1 Root Cause Evaluation (RCE) 2-2001-2806, Failure To Identify and Correct a Condition Adverse to Quality in a Timely Manner, dated 1/10/02 Category 2 RCE S-2001-1168, EDG 3 Damage on Wrist Pins of Cylinders 12, 15, 16, dated 12/1/01 Category 2 RCE S-2001-2890, EDG 2 Output Breaker, dated January 17, 2002 Virginia Power Administrative Procedure (VPAP)-1604, Root Cause Evaluation Program, Revision 4 VPAP-1601, Corrective Actions, Revision 15 Engine Systems Inc. (ESI) Report No. 87360-FA, Failure Analysis for One Power Assembly, EMD P/N 8470863, For Dominion Virginia Power, Surry Power Station, Revision 0, dated 4/27/01 and Revision 1, dated 5/25/01 Attachment
ESI Report 90342-A, Failure Analysis for Ten Power Assembly, Fork Rod, EMD P/N 8470863 and Power Blade Rod, EMD P/N 8470864, for Dominion Virginia Power, Surry Power Station, EDG #1, Revision 0, dated 8/15/01 Trident Engineering Associates, Inc. Report, EDG Support Services, Surry Power Station, VEPCO Contract No. PR-CU0012-000, Trident Contract No. 918-007, dated 9/26/01 Ricardo Engineering Report, Project No. H0829, Engine Life Prediction for EDG 3 at Surry Nuclear Power Station for Dominion Virginia Power, dated 11/15/01 Nuclear Engineering Services Engineering Laboratory Materials Analysis Report, NESML-Q-480, dated 11/19/01 0-ECM-0301-01, 4160 Volt Breaker Maintenance, Revision 21 0-ECM-0302-03, 480 Volt K-Line Breaker Maintenance, Revision 24 Plant Issue Resolution S-2000-2180-R13, 1-VS-F-58A & 58B Breakers Trip on ESF Test, (corrective action effectiveness review) dated 1/2/02