IR 05000416/2003006

IR 05000416-03-006, on 02/10-13/2003, Grand Gulf Nuclear Station, Entergy Operations, Inc. Design Control, Problem Identification and Resolution

ML031040581
Person / Time
Site:	Grand Gulf
Issue date:	04/11/2003
From:	Johnson W NRC/RGN-IV/DRP
To:	Eaton W Entergy Operations
References
IR-03-006
Download: ML031040581 (32)

Text

ril 11, 2003

SUBJECT:

GRAND GULF NUCLEAR STATION - NRC SPECIAL TEAM INSPECTION REPORT 50-416/03-06

Dear Mr. Eaton:

On February 13, 2003, the U.S. Nuclear Regulatory Commission (NRC) completed the onsite special team inspection at your Grand Gulf Nuclear Station. The enclosed report documents the inspection findings, which were discussed with Mr. Ron Moomaw and other members of your staff on March 13, 2003.

This inspection examined the increased failure rate of Agastat General Purpose relays, which was highlighted by a relay failure in each train of standby service water system on November 17, 2002. The inspection examined the events surrounding the increased failure rate of Agastat General Purpose relays as they relate to safety and compliance with the Commission's rules and regulations and with the conditions of your license. The inspection consisted of examination of procedures and records and interviews with station personnel.

This report documents one finding of very low safety significance (Green) which was determined to involve a violation of NRC requirements. However, because of the very low safety significance and because it was entered into your corrective action program, the NRC is treating this finding as a noncited violation (NCV) consistent with Section VI.A of the NRC Enforcement Policy. If you contest any NCV in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001, with copies to the Regional Administrator, U.S. Nuclear Regulatory Commission, Region IV, 611 Ryan Plaza Drive, Suite 400, Arlington, Texas 76011; the Director, Office of Enforcement, US Nuclear Regulatory Commission, Washington, DC 20555-0001; and the NRC Resident Inspector at the Grand Gulf Nuclear Station.

In accordance with 10 CFR 2.790 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records component of NRCs document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Entergy Operations, Inc. -2-Should you have any questions concerning this inspection, we will be pleased to discuss them with you.

Sincerely,

/RA/

William D. Johnson, Chief Project Branch A Division of Reactor Projects Docket: 50-416 License: NPF-29

Enclosure:

NRC Inspection Report 50-416/03-06

REGION IV==

Docket: 50-416 License: NPF-29 Report: 50-416/03-06 Licensee: Entergy Operations, Inc.

Facility: Grand Gulf Nuclear Station Location: Waterloo Road Port Gibson, Mississippi 39150 Dates: February 10-13, 2003 Inspectors: T. W. Jackson, Team Leader, Resident Inspector, Diablo Canyon J. L. Taylor, Reactor Engineer Approved By: W. D. Johnson, Chief Project Branch A Division of Reactor Projects Attachment 1: Supplemental Information Attachment 2: Special Inspection Team Charter

SUMMARY OF FINDINGS IR 05000416/2003-006; Entergy Operations, Inc., 2/10/03 - 2/13/03; Grand Gulf Nuclear Station; Design Control, Problem Identification and Resolution.

The report covers a special inspection that assessed the licensee response to a higher failure rate of Agastat General Purpose relays. The inspection team was composed of a resident inspector and a reactor inspector. One Green noncited violation was identified. The significance of any findings is indicated by the color (Green, White, Yellow, or Red) assigned using IMC 0609, "Significance Determination Process" (SDP). Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review.

The NRC's program for overseeing the safe operation of commercial power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, dated July 2000.

Inspector Identified and Self-Revealing Findings Cornerstone: Mitigating Systems Green. A noncited violation of Criterion XVI of Appendix B to 10 CFR Part 50 was identified for the failure to adequately identify the cause of relay contact failures-to-transition, a significant condition adverse to quality, and corrective actions to prevent recurrence.

This finding is greater than minor because, if the condition were left uncorrected, it would become a more significant safety concern. Specifically, the failure to understand the failure mechanism behind the failure mode mentioned above would impede the licensee's ability to control that failure mechanism and could lead to additional failures of safety-related equipment to actuate when called upon. The finding was determined to be of very low risk significance since no other failures of this type have been experienced since the discovery of the initial five failures (Section 4.1).

TABLE OF CONTENTS 1.0 Special Inspection Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2.0 Description of Event and Chronology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2.1 Event Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2.2 Preliminary Risk Significance of Event . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.3 Sequence of Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 3.0 Root Causes of Relay Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.1 Agastat GP Relay Service Life . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 3.2 Operating Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Corrective Actions Associated With High Relay Failure Rate . . . . . . . . . . . . . . . 8 3.4 Human Performance and Procedural Aspects . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.0 Extent of Condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.1 Failure Modes of Agastat GP Relays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.2 Impact of Potential Relay Failures on Plant Systems . . . . . . . . . . . . . . . . . . . . 12 4.3 Aging Management of Other Plant Components . . . . . . . . . . . . . . . . . . . . . . . 14 5.0 Exit Meeting Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 ATTACHMENT 1 - Supplemental Information ATTACHMENT 2 - Special Inspection Team Charter

Report Details 1 SPECIAL INSPECTION ACTIVITIES The NRC conducted this special inspection to better understand the circumstances surrounding the increased failure rate for Agastat General Purpose (GP) relays.

Failure of Agastat GP relays was highlighted by two relay failures, discovered within 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> of each other, on opposite trains of the standby service water (SSW) system.

The two failures prompted operators to enter a 12-hour Shutdown Technical Specification Action Statement.

The Special Inspection Team, or team, evaluated the potential safety implications related to the cause of the increased failure rate. The inspectors used NRC Inspection Procedure 93812, "Special Inspection," to conduct the inspection. The team reviewed procedures, logs, corrective action documents, and design and maintenance records for the equipment of concern. The team interviewed key station personnel regarding the SSW event and failure rate of Agastat GP relays and other safety-related components.

The team observed locations where the relays are in-service to determine the impact of environmental conditions. Attachment 2 is the charter for the team, which describes the inspection scope in greater detail.

2 DESCRIPTION OF EVENT AND CHRONOLOGY 2.1 Event Summary On November 17, 2002, with the reactor at the Grand Gulf Nuclear Station at full power, a licensed operator started SSW Train B to support residual heat removal (RHR)

Train C quarterly valve stroke testing. Approximately 45 minutes after starting SSW Train B, operators discovered that Valve 1P41F005B, SSW return to the cooling tower, would not close because normally energized Relay P41R019 had failed to the de-energized state. Operators declared SSW Train B inoperable and entered Technical Specification Action Statement 3.7.1.D. Approximately 6 hours6.944444e-5 days <br />0.00167 hours <br />9.920635e-6 weeks <br />2.283e-6 months <br /> later, operators walked down SSW Train A and found the inlet valve for SSW to RHR heat exchangers, Valve 1P41F014A, open. This valve would not close via the control room handswitch.

Operators declared SSW Train A inoperable, placing the unit in Technical Specification 3.7.1.E, which requires the reactor to be in Mode 3 in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />.

Subsequently, maintenance personnel found normally energized Relay P41R004 had also failed to the de-energized state, causing Valve 1P41F014A to open. Maintenance personnel replaced the two failed relays and operators reset the SSW system to its normal alignment. SSW Train B was declared operable approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> after Technical Specification Action Statement 3.7.1.E was entered, and SSW Train A was declared operable 45 minutes later. Throughout the event, the reactor remained at 100 percent power.

Plant personnel initiated Condition Report (CR) CR-GGN-2002-2426 in response to the relay failures in the SSW system and 11 other Agastat GP relay failures found during Refueling Outage RFO12. The team determined that, over the past 18 months, 15 relays with a service life of greater than 10 years had failed, including the 2 relays that had failed in the SSW system. Two other Agastat GP relays failed during the past

-2-18 months, but both had 7 years and 4 months of service time. The established service life for normally energized Agastat GP relays at Grand Gulf Nuclear Station is 10 years, which was determined through a plant-specific testing program. However, the preventive maintenance program was modified in 1989 and 1990 to permit the subject relays to be replaced between 10 and 12.5 years. Licensee personnel identified 227 normally energized, safety-related Agastat GP relays that had greater than 10 years of service life and were currently installed in the plant. Twenty-seven of these relays had greater than 12 years of service life. Following the SSW system event on November 17, maintenance personnel replaced all normally energized Agastat GP relays in safety-related systems that had been in service for greater than 12 years. At the time of the Special Inspection, maintenance personnel had replaced 34 out of 59 high risk relays and planned to have all 59 relays replaced by the end of February 2003. The remaining relays are planned to be replaced by the end of June 2003, or evaluated to see if they should remain in the relay replacement program.

2.2 Preliminary Risk Significance of Event Following the relay failures in the SSW system, the NRC performed an evaluation of the preliminary risk significance in terms of conditional core damage probability (CCDP).

The CCDP is the probability of core damage over a period of time given a specific plant condition. The CCDP analysis used the observed failure rate of Agastat GP relays with service lives greater than 10 and 12 years, the number of relay failures to the nonconservative state, and the impact of nonconservative relay failures on the SSW system. The analysis assumed that the SSW system was affected by the aged relays for a period of 1 year, and the CCDP was calculated to be on the order of 1 E-6.

NRC Management Directive 8.3, "NRC Incident Investigation Program," requires the consideration of a Special Inspection Team when the estimated CCDP is greater than or equal to 1 E-6. Based on the number of relays with service lives greater than 10 years and the use of the aged relays in other safety-related and risk-significant, nonsafety-related systems, the NRC determined that a Special Inspection Team would assess the cause of the increased relay failure rate, the potential impact of relay failures, and the corrective actions taken.

2.3 Sequence of Events The team developed a detailed sequence of events and organizational response timeline. The timeline included applicable events and actions before, during, and following the relay failures in both trains of the SSW system. The timeline was generated from operator logs, written records, and interviews with members of the licensee's staff. The team's review satisfied the activities associated with Special Inspection Team Charter Element 1, "Develop a complete sequence of events related to the subject relay failures in the SSW system and licensee actions taken in response to the failures."

(All times are given in Central Standard Time)

-3-October 17, 1983 Earlier than anticipated end-of-service-life failures of Agastat GP relays were reported by Grand Gulf Nuclear Station via the provisions of 10 CFR Part 21.

Twelve out of approximately 1700 relays failures were identified during 18-month surveillance tests. These failures precluded the automatic operation of SSW system valves, main steam isolation valves, components in the control room ventilation system, combustible gas control system, reactor core isolation cooling system, RHR system, and the high pressure core spray system. The failures centered around the shrinkage of the relay's plastic casing, preventing the contact arm from changing states.

January 6, 1984 In Letter AECM-84/0024 to the NRC, the service life of normally energized Agastat GP relays was reported to be 4.5 years.

March 21, 1984 The NRC issued Information Notice 84-20 regarding the relay failures identified at Grand Gulf Nuclear Station.

July 5, 1985 In Letter MPGE-85/138, General Electric provided the results of additional analysis which stated normally energized Agastat GP relays to have a service life of 6.6 years if operating temperature environment is below 71EF, and 5.9 years if the operating temperature environment is below 74EF.

June 6, 1989 Based on additional testing, design engineering specified a service life of 10 years for Agastat GP relays located in the control room and upper cable spreading room, 5.8 years for those located in the control building switchgear room, and 6.4 years for those located in the SSW pump house switchgear room.

1989 to 1990 System engineering developed preventive maintenance tasks to replace normally energized Agastat GP relays based on the tested service life of 10 years. A 25 percent grace period was assigned for all relays in the replacement program.

October to November 1990 Approximately 150 Agastat GP relays were replaced during Refueling Outage RFO4.

-4-April to June 1992 Approximately 150 Agastat GP relays were replaced during Refueling Outage RFO5. This was the last bulk replacement of Agastat GP relays.

October 1999 Following Refueling Outage RFO10, many of the Agastat GP relay replacements were performed as an on-line maintenance task. As a result, many of the tasks were deferred from the outage scope, developing a population of relays with greater than 10 years of service life.

September 2002 Licensee personnel reported 16 Agastat GP relay failures during Refueling Outage RFO12.

November 17, 2002 12:34 p.m. Started SSW Train B to support RHR Train C quarterly valve stroke testing.

1:16 p.m. Declared SSW Train B and Division 2 diesel generator inoperable as a result of Valve 1P41F005B not closing on demand. The valve would not close because Relay P41R19 had failed to the de-energized state.

7:15 p.m. Declared SSW Train A inoperable when an operator performed a system walkdown and discovered Valve 1P41F014A open. The valve would not close because Relay P41R004 had failed to the de-energized state.

9:18 p.m. Operators declared SSW Train B operable.

10:02 p.m. Operators declared SSW Train A operable.

November 27, 2002 Maintenance personnel completed the replacement of relays with service life greater than 12 years.

January 30, 2003 Following a reactor scram, maintenance personnel replaced all 12 Agastat GP relays that could only be replaced during an outage.

-5-February 10, 2003 The NRC initiated a Special Inspection to assess the cause, impact, and corrective actions related to the increased failure rate of Agastat GP relays at the Grand Gulf Nuclear Station.

3 ROOT CAUSES OF RELAY FAILURES 3.1 Agastat GP Relay Service Life a. Inspection Scope The team examined the licensee engineers' determination of service life for Agastat GP relays. The team reviewed design documents, tests, and calculations performed by the licensee, General Electric, and third-party vendors. The team also reviewed application of industry testing standards for which the licensee had committed in their Updated Final Safety Analysis Report. Interviews with design and system engineers provided additional insight into the calculated service life for the relays. The team's examination in this area satisfied a portion of the activities associated with Special Inspection Team Charter Elements 2 and 4.

b. Findings Background. During 18-month surveillance tests in August 1983, licensee personnel identified 12 out of 1700 normally energized Agastat GP relays, whose contacts would not transition when the coil was de-energized. The problem was determined to be shrinkage of the plastic relay casing due to thermal aging. As a result of the casing shrinkage, binding occurred between the casing and the relay contact arm, preventing the contacts from transitioning to the safety position when the coil de-energized. The corrective actions for this problem included replacement of relays that were manufactured before August 1977 with later models that did not have the binding problem. On March 21, 1984, the NRC issued Information Notice 84-20, "Service Life of Relays In Safety-Related Systems," in response to the relay failures at Grand Gulf Nuclear Station. The Information Notice discussed the fact that normally energized relays have significantly shorter lives than those relays that are cycled or normally de-energized. It also noted that using relays manufactured after August 1977 may not extend the service life of Agastat GP relays.

In January 1984, the licensee stated in Letter AECM-84/0024 to the NRC that the service life of normally energized Agastat GP relays was 4.5 years. Licensee engineers believed that there was additional margin in the service life of normally energized relays and requested General Electric to provide an analysis that would extend the service life. In Letter MPGE-85/138, dated July 5, 1985, General Electric provided an analysis that would allow the licensee to extend the service life of Agastat GP relays to 6.6 years if the operating temperature environment was 71EF or below, and 5.9 years if the operating temperature environment was 74EF or below. The licensee engineers still felt there was additional margin in the service life of normally energized

-6-Agastat GP relays and sponsored their own tests to establish a service life. In June 1989, the licensee engineers completed Calculation EC-Q1111-88002, "Thermal Life of Agastat Relays," Revision 1, which determined the service life of normally energized Agastat GP relays to be 10 years if they were used in the control room or upper control room. The team evaluated the calculation and found it to be consistent with IEEE 323-1974, "IEEE Standard for Qualifying Class 1E Equipment for Nuclear Power Generating Stations," which the licensee had committed to in their Updated Final Safety Analysis Report.

Through 1989 and 1990, planning and scheduling modified the process whereby they scheduled preventive maintenance tasks. Engineers responsible for various systems and components submitted requests for routine preventive maintenance tasks. The engineer would specify a frequency for the task to occur and the maintenance scheduling system would calculate a due date and a late date for a given task. Initially, the due date was the date that the licensee was targeting to have the maintenance task completed. The late date was the date by which the task had to be completed. Unless the responsible engineer specified a late date, the maintenance scheduling system automatically calculated the late date as 25 percent of the task frequency and added that amount of time to the due date to arrive at a late date. The 25 percent "grace period" arose from the Technical Specification allowed surveillance frequency extension under limiting conditions. Once maintenance task schedules were determined, the responsible engineers were to review the due and late dates. In the case of Agastat GP relay replacement schedules, the task frequency was 10 years. However, engineering personnel did not specify a grace period for the late date. Therefore, the maintenance scheduling system calculated the due date based on 10 years and the late date as 12.5 years, which is 10 years plus 25 percent of 10 years.

Beginning in the year 2000, a population of normally energized Agastat GP relays that exceeded the service life time of 10 years. The team determined that this population appeared as a result of several factors. First, CR-GGN-2002-2426 stated that, prior to Refueling Outage RFO10, the majority of relay replacements occurred during the refueling outage. After RF10, an organizational shift to perform many of the refueling outage tasks as online maintenance resulted in many relay replacements being deferred due to scheduling constraints and other higher priority activities. CR-GGN-2002-2426 and interviews with licensee personnel also revealed that organizational expectations for due and late dates changed from the original intent. Initially, the licensee's expectation was to have routine maintenance activities completed prior to the due date. However, if circumstances arose where the activity could not be completed prior to the due date, then it must be completed before the late date. However, in the late 1990's, the expectation changed and allowed many routine maintenance tasks to be completed between the due and late dates. The expectation change was reflected in the definitions of due and late dates in Procedure 01-S-17-11, "Repetitive Task Program,"

Revision 5, issued in March 1999.

Two other factors contributed to the population of Agastat GP relays that exceeded 10 years of service. Calculation EC-Q1111-88002, "Thermal Life of Agastat Relays,"

calculated the service life of 10 years, but did not test the relays to failure. Therefore, the licensee engineers knew the service life of the relays was at least 10 years, but did

-7-not know how much service life margin was left. The licensee engineers also tracked the number of Agastat GP relay failures and compared them to the limiting failure rate of 1E-6 relay failures per hour given in General Electric Purchase Part Drawing 169C9489. Since there were approximately 1700 Agastat GP relays, and using the limiting failure rate, the licensee engineers calculated that 22 relay failures per cycle was within their design basis. Therefore, as long as the number of relay failures did not exceed 22 per cycle, then there was no indication of problems with the Agastat GP relay preventive maintenance program. The licensee engineers used the premise of additional service life margin in Agastat GP relays and the fact that the number of observed relay failures had not exceeded 22 per cycle as engineering judgment to justify normally energized Agastat GP relays to be replaced between 10 and 12.5 years of service life. However, the licensee did not document the engineering judgment.

Introduction. The inspectors identified a finding that Agastat GP relays were allowed to exceed the design basis life of 10 years without adequate design control measures commensurate with those applied to the original design.

Description. The team determined that 227 normally energized Agastat GP relays were allowed to exceed their service life of 10 years. The licensee engineers stated that their design basis limiting failure rate of normally energized Agastat GP relays was 1E-6 failures/hour and, based on a population of 1700 normally energized relays, the 16 observed relay failures were under the calculated 22 failures per cycle. However, the team observed that Drawing 169C9489, which specified a limiting failure rate of 1E-6 failures/hour, also specified a service life of 10 years. Additionally, the purchase part drawing was a general specification that did not specify whether the relays would be used in normally energized or normally de-energized applications. Since only two failures of Agastat GP relays with a service life below 10 years were observed, the stellar performance of those relays with a service life below 10 years tended to mask the high number of failures experienced with relays having a service life greater than 10 years. The team also observed that the licensee engineers had not documented the engineering judgment or provided design control measures commensurate with those applied in Calculation EC-Q1111-88002 or previous General Electric calculations for Agastat GP relay service life.

Analysis. The finding was determined to be more than minor since the failure to apply proper design control measures, if left uncorrected, would lead to a more significant safety concern. Specifically, the failure to apply adequate design control measures reduced the reliability and availability of safety-related systems due to an increased probability of failure from those Agastat GP relays that had exceeded their service life.

The finding was assessed in accordance with NRC Inspection Manual Chapter 0609,

"Significance Determination Process," Phase 1 and 2. Based on the Phase 2 analysis, a Phase 3 analysis was required. The Phase 3 analysis has not been completed.

Therefore, this finding is identified as an unresolved item (URI 50-416/2003006-001)

pending results of the Phase 3 analysis.

-8-3.2 Operating Experience a. Inspection Scope The team reviewed the use of operating experience related to Agastat GP relays.

Documents reviewed included NRC Information Notices, 10 CFR Part 21 Reports, licensee CRs, and results of industry operating experience searches. This activity satisfied a portion of Special Inspection Team Charter Items 2 and 4.

b. Findings The team reviewed the licensee's response to NRC Information Notice 84-20, "Service Life of Relays in Safety-Related Systems." In 1983, casing to contact arm binding in normally energized Agastat GP relays was attributed to thermal aging. This event was documented in Information Notice 84-20. The information notice discussed the shortened life of Agastat GP relays if they are used in normally energized applications, and it addressed the need for licensees to assess service life and replacement programs accordingly. The licensee engineers determined the service life of normally energized Agastat GP relays following the relay failures in 1983 and implemented a replacement program at that time.

The team evaluated other NRC, licensee, and industry experience with Agastat GP relays to determine if there was any experience applicable to the increased failure due to service life. Relay failures as described in CRs, 10 CFR Part 21 Reports, Information Notices, and industry operating experience searches did not have close correlation with the increased failure rate due to end-of-service life issues at Grand Gulf Nuclear Station.

Furthermore, the licensee engineers had appropriately used operating experience to address potential performance issues in their normally energized Agastat GP relays.

The team did note that licensees who use Agastat GP relays in normally energized applications experience service life limits from 4.5 years to 10 years. It was also noted that, at Grand Gulf Nuclear Station, there were between 4 and 14 Agastat GP relay failures per cycle. Not all failures involved normally energized relays and not all failed relays were included in the relay replacement program.

3.3 Corrective Actions Associated With High Relay Failure Rate a. Inspection Scope The team evaluated immediate and long-term corrective actions that the licensee has taken, or plans to take, in response to the Agastat GP relay failures. Databases, procedures, and CRs were some of the documents used to verify relay replacement and other corrective actions. The team also interviewed licensee staff and observed some of the relays that had been replaced. Operability of the SSW system during the November 17, 2002, event was verified. This activity satisfied Special Inspection Team Charter Items 2 and 6.

-9-b. Findings The team evaluated the operability of the SSW system following the failure of two relays in opposite trains on November 17. Both relays failed to the de-energized state, actuating their respective safety functions and placing the SSW system in a partial safety function alignment. In SSW system Train A, failure of Relay 1P41R004 caused Valves 1P41F014A and 1P41F068A, SSW inlet and outlet valves for RHR Heat Exchanger A, to open. In SSW system Train B, failure of Relay 1P41R019 caused Valve 1P41F006B, SSW Pump B recirculation valve, to open as long as RHR Heat Exchanger B inlet and outlet valves were not full open. The team used the guidance found in Generic Letter 91-18, "Information to Licensees Regarding NRC Inspection Manual Section on Resolution of Degraded and Nonconforming Conditions," Revision 1, to determine the operability of the SSW system. The team determined that, in the event a valid automatic SSW start signal were received, the SSW system would fully align to the proper safety alignment and carry out its mission. The two relay failures would not impede the system from performing its designed function.

The licensee's immediate corrective actions included the replacement of Agastat GP relays that had exceeded the service life of 10 years. Licensee personnel identified normally energized Agastat GP relays that were in the relay replacement program and had more than 10 years of service life. Licensee engineers and maintenance personnel prioritized the relay replacement based on age, risk significance, and accessibility. Of the 227 normally energized, safety-related relays, there were 27 relays that had more than 12 years of service life, and they were replaced by the end of November 2002.

Upon the replacement of those 27 relays, licensee personnel identified 59 relays that could initiate a transient or cause major degradation of safety-related systems if they failed to transition to the de-energized state. At the time of the Special Team Inspection, maintenance personnel had replaced 34 of the 59 relays and planned to have the remaining 25 replaced by the end of February. The remaining 141 relays would be replaced by the end of June 2003. Engineering personnel planned to evaluate some of the 141 relays to determine if they should remain in the relay replacement program. Following a reactor scram on January 30, 2003, maintenance personnel took the opportunity and replaced all 12 relays that could only be replaced when the unit was shutdown, in order to avoid a transient while replacing the relay. The team determined that the relay replacement actions were aggressive, while adequately considering the risk impact of each relay.

The team reviewed aspects of 10 CFR 50.65, "Maintenance Rule," and how they may apply to the Agastat GP relays. In CR-GGN-2002-2325, the engineers considered making the relays a pseudo-system in the Maintenance Rule Program in order to track its failures and apply goal setting criteria. However, the engineers decided to continue tracking relay failures as they occur in each system and continue to track the number and types of failures through an engineering database.

The engineers pursued two corrective actions to address contributing causes for Agastat GP relays in use beyond their service life. They planned to modify the repetitive task program to reflect the expectation that tasks are to be performed before the due date and the grace period would be used under extreme conditions. Corrective

-10-actions regarding the repetitive tasks program are described in Section 3.4. Other corrective actions involve a review (described in Section 4.2) of preventive maintenance tasks dealing with service life replacement of safety-related and risk-significant, nonsafety-related components.

3.4 Human Performance and Procedural Aspects a. Inspection Scope The team reviewed licensee personnel actions before, during, and following the SSW system relay failure event on November 17, 2002. Additionally, the team reviewed applicable procedures related to the repetitive task program and how they impacted the increased failure rate of Agastat GP relays. This activity satisfied a portion of Special Inspection Team Charter Items 2, 3, and 4.

b. Findings The team interviewed operations personnel to discuss the sequence of events on November 17, 2002. The event began as a routine valve stroke test of RHR Train C.

During the test, operators discovered Valve 1P41F005B would not close. Operators conservatively declared SSW Train B inoperable. Operators walked down the remaining operable train of SSW and discovered Valve 1P41F014A open and it would not close from the control room handswitch. Operators conservatively declared SSW Train A inoperable also, placing the unit in a 12-hour Technical Specification Shutdown Statement. Approximately 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> later, operators exited the 12-hour Shutdown Statement when SSW Train B was repaired and returned to operable status.

Subsequently, Train A was returned to operable status approximately 45 minutes later.

The team determined that operators appropriately responded to the two relay failures in the SSW system.

CR-GGN-2002-2426 identified several contributing causes that lead to the use of Agastat GP relays past their service life of 10 years. One contributor was the definitions of due date and late date in Procedure 01-S-17-11, "Repetitive Task Program," Revision 8. The definition of due date in this procedure stated, in part, that the due date was the date upon which a repetitive task meets its criteria to begin. The definition of late date stated, in part, the late date was the latest date by which a task may be completed. These definitions, coupled with a tolerance for exceeding the due date, promoted an organizational expectation for completing repetitive maintenance tasks between the due date and late date. Subsequently, a corrective action in CR-GGN-2002-2426 is to revise the definitions of due and late date in Procedure 01-S-17-11 to support the expectation that repetitive tasks are to be completed before the due date, with the exception of extreme circumstances.

Additionally, the procedure provided for an automatic 25 percent grace period for all repetitive tasks if a late date was not specified. The program administrator plans to change the automatic grace period for repetitive tasks to be a maximum 180 days for tasks that exceed 2 years. The team reviewed the corrective actions related to Procedure 01-S-17-11 and found them to be adequate.

-11-4 EXTENT OF CONDITION 4.1 Failure Modes of Agastat GP Relays a. Inspection Scope The team evaluated databases, CRs, and Maintenance Action Items to determine the types of failure mechanisms, modes, and causes for Agastat GP relay failures. The team also interviewed engineering personnel to receive a full understanding of the relay failures. The team primarily looked at relay failure data back to 1999, with the exception of relay failures that occurred since 1997 and were documented in CRs. This activity satisfied a portion of Special Inspection Team Charter Item 7.

b. Findings The team identified three primary failure mechanisms/modes in the last operating cycle.

The primary failure mechanism was a relay coil "burnt," or open-circuit, condition. This failure mechanism was caused by a thermal aging condition that affected the insulation of the relay coil. When sufficient thermal aging of the coil insulation had occurred, the coil windings would short, resulting in an open circuit in the coil. The open circuit would cause the coil to de-energize and the contacts would transition to the de-energized position. The open-circuit coil condition was often accompanied by a characteristic smell that came from the relay.

A second failure mechanism was an open-circuit bridge rectifier for ac relays. When an open-circuit bridge rectifier occurred, a diode in the bridge circuit exhibited an open fault, causing the coil to de-energize. Of the 18 relays that failed in the 18 months prior to the team inspection, 13 of the failures were a result of open-circuit coils and one was the result of an open-circuit bridge rectifier.

Background. Licensee personnel identified five reported relay failures in the past 18 months where one or more of the contacts in each relay failed to transition. The contacts that failed to transition appear to be random in the affected relays and not the same contact. The relays were replaced and tested on a test bench. However, the failure mechanism could not be repeated on the bench. Since the failure could not be repeated, the engineers determined that the relays had not failed and did not pursue further actions to determine the cause of the system failure.

Introduction. A Green inspector-identified finding was determined in that licensee personnel failed to adequately assess the cause of relay contact failures-to-transition in order to prevent repetition of this significant condition adverse to quality.

Description. The team found that the failure to determine the cause of relay contact failures-to-transition conditions was a significant condition adverse to quality since a safety function would be prevented. In normally energized Agastat GP relays used at the Grand Gulf Nuclear Station, if the relay de-energizes, then the contacts are to transition and initiate certain safety functions. Depending upon the function, a

-12-safety-related system could be significantly degraded or inhibited if the contact fails to transition. Licensee personnel identified the failure-to-transition of single contacts, but failed to provide sufficient followup to understand the cause of this particular failure mechanism. For example, maintenance personnel did not adequately test the relay base or test the relay under its normal operating conditions (i.e., normally energized and heated to its operating temperature). Without a sufficient understanding of this failure mechanism, licensee personnel are not able to provide adequate corrective actions to prevent recurrence or to control the failure mechanism.

Analysis. The finding was determined to be more than minor since the failure to determine the cause of contact failures-to-transition, if left uncorrected, would lead to a more significant safety concern. Specifically, the mechanism causing the contact failures-to-transition may exist in Agastat GP relays still in service and could inhibit a train of safety-related equipment from performing its function. The finding was determined to be of very low safety significance because no further failures of this type have been experienced since the five failures, mentioned above, were discovered.

Enforcement. 10 CFR Part 50, Appendix B, Criterion XVI, "Corrective Actions," states, in part, in the case of significant conditions adverse to quality, the measures shall assure that the cause of the condition is determined and corrective action taken to preclude repetition. In this case, licensee personnel failed to adequately pursue the cause of the contact failures-to-transition, which are considered to be significant conditions adverse to quality due to the potential impact on safety. Because this failure to determine the cause of single relay contact failure-to-transition is of very low safety significance and has been entered into the corrective action system as CR-GGN-2003-0873, this violation is being treated as an NCV, consistent with Section VI.A of the NRC Enforcement Policy: NCV 50-416/03-06-02, Failure to Determine Cause of Single Relay Contact Failure-to-Transition.

4.2 Impact of Potential Relay Failures on Plant Systems a. Inspection Scope The team evaluated the potential impact of those Agastat GP relays that had a service life greater than 10 years. The impact was evaluated for relay failure in either the de-energized or energized state. While the team did not evaluate all 227 relays that were in operation past their service life, the team did look at the 137 most risk-significant relays in 11 systems. Risk significance was determined by loss of safety function; loss of a nonsafety-related, risk-significant function; initiation/impedance of a half-scram; and impact on containment isolation. Relays that provided annunciation functions were not included in the evaluation. This activity satisfied a portion of Special Inspection Team Charter Items 7 and 9.

b. Findings Several systems had a greater susceptibility to Agastat GP relay aging failures due to the location and number of relays that had service lives of greater than 10 years.

Table 1 identifies those systems that had the most potential of being impacted by a relay

-13-failure. The table does not list all of the potential impacts to those systems, but rather the most significant impacts. In those systems, the failure of the relay contacts to transition when the coil de-energizes poses the most impact on safety systems (i.e., the relays were designed to initiate a safety function when the coil de-energizes). Since the primary failure mechanism for relays that exceeded their service life was an open-circuit coil that caused the relay to actuate its safety function, the actual risk is lower than the risk portrayed in Table 1. It should also be noted that manual action is available in the majority of cases if the relays failed to transition.

Table 1 - Systems Impacted by Aged Agastat GP Relays System Number of Potential Impacts Aged Relays Containment and 12 Failure to receive a containment isolation on Drywell Isolation several large pipes - a redundant valve is available Control Room 10 Failure of the control room ventilation to align Ventilation automatically Nuclear Boiler System 31 Would not receive half-isolation of main steam Controls isolation valves Standby Service Water 38 Failure of a SSW pump to start; Failure of a train to align and supply SSW to its loads; Failure of a train to supply individual loads such as diesel generator coolers, Class 1E switchgear room coolers, and RHR heat exchangers The team reviewed the impact of normally energized Agastat GP relays in risk-significant, nonsafety-related systems, but did not find any significant conditions that would impact a mitigating function or initiate a plant transient. The majority of relays used in risk-significant, nonsafety-related systems are normally de-energized.

The team observed the ambient temperature and other environmental conditions for which the normally energized Agastat GP relays were subjected. Operating the relays above their recommended ambient temperature significantly decreases the life of the relays. The team determined that the relays were operated in conditions consistent with the relay's design basis and vendor recommendations.

The team interviewed licensee personnel to determine the depth of the probabilistic risk assessment to analyze relay failures. As with other licensees, Grand Gulf Nuclear Station's probabilistic risk assessment does not model to the relay level, rather it models to the train level. Therefore, both the licensee's and the NRC's probabilistic risk assessment must depend upon expert judgment to provide a reasonable risk analysis for Agastat GP relays used beyond their service life.

-14-4.3 Aging Management of Other Plant Components a. Inspection Scope The team sampled several components that are used in safety-related systems that would also have a service life. The team chose Rosemount transmitters, ASCO solenoids, power supplies, and scram pilot valves. The team interviewed licensee personnel and reviewed engineering documents for this portion of inspection activity.

This activity satisfied a portion of Special Inspection Team Charter Items 5, 6, and 8.

b. Findings The team reviewed the preventive maintenance programs for Rosemount transmitters and scram pilot valves and found that these components were treated similarly as far as maintenance scheduling was concerned. The Rosemount transmitters were included in the Environmental Qualification program. Since the Rosemount transmitters had an end-of-service life, the engineers specified the end-of-service life as the late date for maintenance scheduling. Maintenance scheduling would then back-calculate to determine a due date and further back-calculate to set an execution date.

The scram pilot valves did not have a design basis service life, but the licensee engineers, through their experience, and the experience of others, had determined an end-of-service life. Responsible engineers for the scram pilot valves would also set the late date as the end-of-service life and allow maintenance scheduling to back-calculate the due date and execution date.

By specifying the late date as the end-of-service life for Rosemount transmitters and scram pilot valves, the licensee engineers effectively controlled the possibility of exceeding the service life. The team determined that the same mechanisms that allowed Agastat GP relays to exceed their service life would not affect these two components.

Vendor documentation for the Lambda power supplies did not specify a service life.

However, the responsible engineer had a database to track power supply failures. The engineer had learned from experience that the limiting component in the power supplies was the electrolytic capacitors. Therefore, the engineer set up an effective replacement program for the replacement of electrolytic capacitors in those power supplies.

ASCO solenoid valves were used primarily in drywell, auxiliary building, and drywell isolation valves. Similar to power supplies, the responsible engineer maintained a database of ASCO valve failures. Additionally, the engineer used the air-operated valve testing data to identify degradation in ASCO solenoid valves. The team determined that ASCO solenoid valve failures had been effectively controlled using the testing data to predict solenoid valve degradation.

-15-5 EXIT MEETING SUMMARY On March 13, 2003, the team presented the inspection results to Mr. Ron Moomaw and other members of the licensee staff at a teleconference meeting. While proprietary information was reviewed during this inspection, no proprietary information is included in this report.

ATTACHMENT 1 PARTIAL LIST OF PERSONS CONTACTED Licensee C. Abbott, Quality Assurance Supervisor C. Bottemiller, Manager, Plant Licensing C. Ellsaesser, Manager, Corrective Action and Assessment M. Larson, Senior Licensing Specialist R. Moomaw, Manager, Outage Planning and Scheduling R. Patterson, Senior Reactor Operator, Operations J. Roberts, Director, Nuclear Safety Assurance J. Robertson, Manager, Quality Assurance M. Rohrer, Manager, Maintenance G. Smith, Senior Staff Engineer, Design Engineering T. Thornton, Engineering Supervisor, Design Engineering H. Yeldell, Manager, Design Engineering NRC C. Paulk, Senior Project Engineer T. Hoeg, Senior Resident Inspector D. Loveless, Senior Reactor Analyst ITEMS OPENED, CLOSED, AND DISCUSSED Opened 05000416/0306-01 URI Failure to Apply Adequate Design Control Measures Lead to Increased Agastat Relay Failure Rate (Section 3.1)

Opened and Closed 05000416/0306-02 NCV Failure to Determine Cause of Single Relay Contact Failure-to-Transition (Section 4.1)

LIST OF DOCUMENTS REVIEWED Procedures:

Procedure 01-S-17-11, "Repetitive Task Program," Revision 8

-2-Procedure 04-1-01-P41-1, "System Operating Instruction: Standby Service Water System," Revision 118 Company Procedure MA-111, "Repetitive Task Program," Revision 1 Procedure 07-S-12-70, "Agastat Control Relay Test & Replacement Instructions,"

Revision 6 Company Procedure PL-161, "Zero Tolerance for Unanticipated Equipment Failures,"

Revision 2 Condition Reports:

CR-GGN-2000-0151 CR-GGN-2001-1972 CR-GGN-2002-2422 CR-GGN-2000-1121 CR-GGN-2002-1290 CR-GGN-2002-2426 CR-GGN-2001-0421 CR-GGN-2002-2082 CR-GGN-2001-0423 CR-GGN-2002-2137 CR-GGN-2001-0703 CR-GGN-2002-2325 Letters and Memoranda:

Letter AECM 84/0024, "Response to NRC Questions on Agastat Relays,"

January 10, 1984 Letter AECM 84/0034, "Additional Information on Agastat Relays," January 20, 1984 Letter from D. P. Alexander, Amerace Corporation, to Gerald Lantz, Mississippi Power &

Light Company, "Re: Your telephone record dated 1/16/85; Qualified Life/Service Life EGP Relays," January 30, 1985 Letter MPGE-85/138, "Agastat Relays - Qualified Life Analysis," July 5, 1985 Memorandum GIN 91-06976, "Agastat Relays," December 13, 1991 Memorandum GIN 95-03473, "Failure Rate of Agastat Relays," December 27, 1995 Memorandum GIN 97-02274, "Review of Repetitive Task Module," November 18, 1997 Memorandum PMI 89/02786, "Agastat Relays," June 9, 1989 Memorandum, Brad Edwards to Greg Sparks et. al, Expectations for Preventive Task, December 9, 2002

-3-Maintenance Action Items:

MAI317113 MAI319948 MAI324171 MAI324177

-4-Material Nonconformance Reports:

MNCR 0117-93 MNCR 0121-84 Procurement Evaluation Requests:

ER 94/6233 ER 95/0309 ER 96/0876 Other Documents:

Calculation EC-Q-1111-88002, "Thermal Life of Agastat Relays," Revision 1 Product Specification EGP, "Model EGP Series Power Relays Class 1E," Revision F, Amerace Corporation Work Activity Task 21328, "Agastat Replacement Instructions" Drawing 169C9489, "Purchase Part Relay," Revision 18, General Electric Drawing M-1061A, "P&I Diagram: Standby Service Water System," Revision 55 SWRI Document 04-1738-001, "Nuclear Component Qualification Test Report for the Qualification of Agastat Relays," December 1988, Southwest Research Institute Root Cause Analysis 93-43, "Unexpected Half Scram During Performance of Surveillance 06-OP-1C71-Q-0001," May 11, 1993 Vendor Manual 460001867, "Technical Manual for Lambda Electronics,"

November 14, 1995 LIST OF ACRONYMS USED CCDP Conditional Core Damage Probability CFR Code of Federal Regulations CR Condition Report GP General Purpose NCV Noncited Violation NRC Nuclear Regulatory Commission RHR Residual Heat Removal SDP Significance Determination Process SSW Standby Service Water

ATTACHMENT 2 CHARTER FOR THE SPECIAL INSPECTION TEAM AT THE GRAND GULF NUCLEAR STATION

January 27, 2003 MEMORANDUM TO: Terry W. Jackson, Resident Inspector, Diablo Canyon Power Plant FROM: Arthur T. Howell III, Director, Division of Reactor Projects /RA/

SUBJECT: CHARTER FOR THE SPECIAL INSPECTION TEAM AT THE GRAND GULF NUCLEAR STATION In response to our initial evaluation of a high failure rate of Agastat General Purpose (GP)

relays, including the failure of two relays on November 17, 2002, that placed the plant in a Technical Specification 12-hour shutdown limiting condition for operation due to both trains of standby service water (SSW) being declared inoperable, a Special Inspection Team is being chartered. You are hereby designated as the Special Inspection Team leader.

A. Basis On November 17, 2002, the licensee discovered that two Agastat GP relays had failed in the SSW system; one in Train A and the other in Train B. Since the two relay failures occurred within hours of each other, both trains of SSW were declared inoperable. The plant entered a Technical Specification limiting condition of operation with a required action of plant shutdown in 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br />. Preliminary NRC risk assessments indicated that the conditional core damage probability for the dual relay failures in the SSW system was between 6.0E-7 and 4.6E-6.

The 2 relay failures, coupled with 11 Agastat GP relay failures that were discovered in September 2002, during Refueling Outage RF12, prompted the licensee to initiate Condition Report CR-GGN-2002-02426 in order to better understand the nature and causes of the relay failures and correct the apparent high failure rate of Agastat GP relays. The established service life for normally energized Agastat GP relays at Grand Gulf Nuclear Station is 10 years. This service life was developed by the licensee through a plant-specific testing program. Based on engineering judgment, the licensee modified their preventive maintenance program in 1989 and 1990 to allow a 2 1/2-year grace period, beyond the service life, for replacing the relays. Subsequently, the licensee discovered that the failed relays had been in service beyond their established service life of 10 years. They also determined that 226 safety-related relays were still in use, but beyond their 10-year service life. Of those 226 relays, 27 had exceeded 12 years of service.

A Special Inspection Team will be dispatched to better understand the cause of the Agastat GP relay failures, the extent of the impact of relay failures, and the licensees

-2-actions leading up to and including the event. The team is also tasked with gaining a better understanding of the licensees common mode failure analysis as related to their root cause(s).

B. Scope The team is expected to perform data gathering and fact-finding in order to address the following items:

1. Develop a complete sequence of events related to the subject relay failures in the SSW system and licensee actions taken in response to the failures.

2. Evaluate pertinent industry operating experience and potential precursors to the event, including the effectiveness of any action taken in response to Information Notice 84-20 and development of the relay replacement program.

3. Evaluate the adequacy of the licensee response to Agastat GP relay failures (timeliness of evaluation, notifications, appropriate use of all relevant data, procedure usage, etc.).

4. Review the licensees root cause evaluation determination for independence, completeness, and accuracy, including the risk analysis of the event.

5. Sample the licensees preventive maintenance and aging management programs to independently assess the extent of condition in regard to other components.

6. Review and assess the licensees corrective actions and ensure that they have adequately evaluated and addressed the extent of condition.

7. Evaluate and determine the potential for common-cause failure.

8. Review the event for generic implications.

9. Evaluate the actual and potential failure impacts of those Agastat GP relays with a service life greater than 10 years that are in safety-related systems and those in risk-significant but nonsafety-related systems.

C. Guidance Inspection Procedure 93812, "Special Inspection," provides additional guidance to be used by the Special Inspection Team.

This memorandum designates you as the Special Inspection Team leader. Your duties will be as described in Inspection Procedure 93812. The team composition will consist of yourself and Joseph Taylor, Reactor Inspector, Engineering and Maintenance Branch, Division of Reactor Safety. During performance of the Special Inspection, the designated team member is separated from normal duties and reports directly to you.

-3-The team is to emphasize fact-finding in its review of the circumstances surrounding the event, and it is not the responsibility of the team to examine the regulatory process.

Safety concerns identified that are not directly related to the event should be reported to the Region IV office for appropriate action.

The Team will report to the site, conduct an entrance, and begin inspection on February 10, 2003. Tentatively, the inspection should be completed by the close of business on February 14, 2003. A formal exit will be scheduled following completion of the on-site inspection. A report documenting the results of the inspection will be issued within 30 days of the completion of the inspection. While the team is onsite, you will provide daily status briefings to Region IV management.

This Charter may be modified should the team develop significant new information that warrants review. Should you have any questions concerning this Charter, contact William D. Johnson at (817) 860-8148.

cc via E-mail:

E. Merschoff T. Gwynn A. Howell E. Collins D. Chamberlain C. Marschall W. Johnson D. Powers T. Hoeg J. Taylor C. Paulk D. Loveless S. Morris, OEDO W. Ruland, NRR