ML18067A695

LER 97-007-00:on 970826,discovered Inadequate Testing of DG Sequencer Control Relay Contacts.Caused by Oversight on Part of Personnel Involved in Installation of Facility Change FC-800.Tested 106D-1/XL & 106D-2/XL Relay Contacts

ML18067A695
Person / Time
Site:	Palisades
Issue date:	09/24/1997
From:	Roberts W CONSUMERS ENERGY CO. (FORMERLY CONSUMERS POWER CO.)
To:
Shared Package
ML18067A694	List: ML18067A693 ML18067A695
References
LER-97-007, LER-97-7, NUDOCS 9709300385
Download: ML18067A695 (7)

Text

NRC FORM 366 U.S. NUCLEAR REGULATORY COMMISSION APPROVED BY OMB NO. 3150-0104

.,.. (4195) EXPIRES 4130/98 ESTIM4TED IUU)0j PER RESPONSE TO COMPLY Willi THIS MANDATORY INfORMATIOH COU£CT10H REQUEST: 50.0 HRS. REPORTED LESSONS LEARNED ARE INCORPORATED LICENSEE EVENT REPORT (LER) 11'1"0 THE LICENSING PROCESS AND FED BACK TO INDUSTRY. FORWARD COMMEllTS REGARDING BURDEN ESTIMATE TO THE INfORMATIOH AND RECORDS llNW3EME1'T BRANCH (T.11 F33), U.S. NJCLEAR REGULATORY COMMISSION, WAS>ilNGTON, DC 20S55-0001, AND TO THE PAPERWORK REDUCTlOH PROJECT (31SB.011M, OFFICE OF (See reverse for required number of digits/characters for each block) llNWlEMEllT AND BUOGET, WA9GHGTOH, DC 20503 FACILITY NAME (1) CONSUMERSENERGYCOMPANY DOCKET NUMBER (2) Page (3)

PALISADES NUCLEAR PLANT 05000255 1of6 TITLE (4 )LICENSEE EVENT REPORT 97-007 - INADEQUATE TESTING OF DIESEL GENERATOR SEQUENCER CONTROL RELAY CONTACTS EVENT DATE (5) LER NUMBER (6) REPORT DATE (7) OTHER FACILITIES INVOLVED (8)

I SEQUENTIAL REVISION FACILITY NAME DOCKET NUMBER MONTH DAY YEAR YEAR NUMBER NUMBER MONTH DAY YEAR 05000 FACILITY NAME DOCKET NUMBER 08 26 97 97 - 007 - 00 09 24 97 05000 OPERATING THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR§: (Check one or more) (11)

MODE (9) N 20.2201(b) 20.2203(a)(2)(v) x 50.73(a)(2)(i) 50.73(a)(2)(iii)

POWER 20.2203(a)(1) 20.2203(a)(3)(i) 50.73(a)(2)(ii) 50.73(a)(2)(x)

LEVEL (10) 100 20.2203(a)(2)(i) 20.2203(a)(3)(ii) 50.73(a)(2)(iii) 73.71 li[lfll!! !f&!l1lll~

20.2203(a)(2)(ii) 20.2203(a)(4) 50. 73(a)(2)(iv) OTHER 20.2203(a)(2)(iii) 50.36(c)(1) 50.73(a)(2)(v) Specify in Abstract below or 20.2203(a)(2)(iv) 50.36(c)(2l 50.73(al(2l(viil in NRC Form 366A LICENSEE CONTACT FOR THIS LER (12)

NAME TELEPHONE NUMBER (Include Area Code)

William L. Roberts, Licensing Engineer (616) 764-2976 COMPLETE ONE LINE FOR EACH COMPONENT FAILURE DESCRIBED IN THIS REPORT (13)

CAUSE SYSTEM COMPONENT MANUFACTURER REPORTABLE CAUSE SYSTEM COMPONENT MANUFACTURER REPORTABLE TONPRDS TONPRDS D EPS [EK,RLY]

SUPPLEMENTAL REPORT EXPECTED (14) MONTH. DAY YEAR I YES If yes COMPLETE EXPECTED COMPLETION DATE x I NO EXPECTED SUBMISSION DATE (15)

ABSTRACT (Limit to 1400 spaces, i.e., approximately 15 single-spaced typewritten lines) (16)

On August 26, 1997, at approximately 1200 hours0.0139 days <br />0.333 hours <br />0.00198 weeks <br />4.566e-4 months <br /> with the plant at 100% power, it was determined that contacts on the emergency diesel generator sequencer load control relays 1060-1 /XL and 1060-2/XL (one contact per relay per diesel generator) were not adequately tested as part of the Technical Specification Test Program. The function of these contacts is to provide a block signal to the SIS-X relays . Failure of this contact to open following a safety injection system actuation

.signal, coincident with. a Joss_of..off..,site.power,-would.result-in-simultar-ieously-loading *all- safety system loads onto the diesel generator as soon as the output breaker closed . This large load on the diesel generator would potentially result in the diesel generator breaker rapidly tripping on overcurrent. Based on failure to test these relay contacts in accordance with Technical Specification requirements, both diesel generators were declared inoperable. Testing of the subject relay contacts was successfully completed within the 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> allowed by T.S. 4.0.3. The controlling Technical Specification was exited and the equipment declared operable at approximately 2100 hours0.0243 days <br />0.583 hours <br />0.00347 weeks <br />7.9905e-4 months <br /> on August 26, 1997. Future Technical Specification testing will include testing of these relay contacts.

9709300385 970924 PDR ADOCK 05000255 S PDR

NRC F\JRM 366CI U.S. NUCLEAR REGULATORY COMMISSION

'** 4/95 LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION FACILITY NAME 11 l DOCKET12l 05000255 YEAR I LER NUMBER '6\

SEQUENTIAL NUMBER REVISION NUMBER PAGE 2 OF6 PALISADES NUCLEAR PLANT 97 - 007 - 00 TEXT (If more space 1s required, use add11ional copies of NRC Form 366A) (17)

EVENT DESCRIPTION On August 26, 1997, at approximately 1200 hours0.0139 days <br />0.333 hours <br />0.00198 weeks <br />4.566e-4 months <br /> with the plant at 100% power, it was determined that contacts on the emergency diesel generator (EDG) sequencer load control relays 106D-1/XL and 106D-2/XL (one contact per relay per EDG) were not adequately tested as part of the Technical Specification Test Program. The Technical Specification Test Program does, however, effectively test the 106D-1 /XL and 106D-2/XL relay coils and other associated contacts. The function of the contacts which had not been tested prior to August 26, 1997, is to provide a block signal to the SIS-X relays. Failure of this contact in each EOG's loading circuit to open following a safety injection system actuation signal, coincident with a loss of off-site power, would result in simultaneously loading all safety system loads onto the EOG as soon as the output breaker closed. This large load on the EOG would potentially result in the EOG breaker rapidly tripping on overcurrent. Based on failure to test these relay contacts in accordance with Technical Specification (T.S.) requirements, both EOGs were declared inoperable.

Upon notification, control room personnel entered Technical Specification Limiting Condition for Operation (T.S. LCO) 3.0.3 due to noncompliance with the requirements of Technical Specification 4.7.1 b. Technical Specification 4.7.1 b requires demonstrating the overall automatic operation of the emergency power system when subjected to a simulated simultaneous loss of normal and standby power sources and a simulated SIS signal. Further, control room operators invoked the provisions of Technical Specification 4.0.3 which allows delaying technical specification actions up to 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> to complete the missed surveillance requirement. Testing of the subject relay contacts was successfully completed within the 24 hours2.777778e-4 days <br />0.00667 hours <br />3.968254e-5 weeks <br />9.132e-6 months <br /> allowed by T.S. 4.0 .. 3. Based on this testing, control room personnel exited T.S. LCO 3.0.3 at approximately 1950 hours0.0226 days <br />0.542 hours <br />0.00322 weeks <br />7.41975e-4 months <br /> on August 26, 1997, when EOG 1-1 was declared operable. EOG 1-2 was declared operable at 2057 hours0.0238 days <br />0.571 hours <br />0.0034 weeks <br />7.826885e-4 months <br /> on the same day.

The failure to test these relay contacts was identified as part of a Palisades ongoing program to review Technical Specification Testing as requested by Generic Letter 96-01, "Testing of Safety Related Logic Circuits." This Generic.Letter requested that electrical schematic drawings and logic diagrams for the reactor protection system, EOG load shedding and sequencing, and actuation logic for the engineered safety features be reviewed against plant surveillance test procedures.

This review is to ensure that all portions of the logic circuitry, including the parallel logic, interlocks, bypasses and inhibit circuits, are adequately covered in the surveillance procedures.

To perform this review, Palisades has prepared a data base that identifies each relay contact in the subject logic schemes. A cross reference to the procedure step that tests the contact is then provided. It was during the course of this review that it was determined that one contact on the sequencer control relay for each EOG was not adequately tested.

NRC FORM 366<1 U.S. NUCLEAR REGULATORY COMMISSION

• ' 4/95 LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION FACILITY NAME (1 l DOCKET(2)

I LER NUMBER 16) PAGE YEAR SEQUENTIAL REVISION NUMBER NUMBER 05000255 3 OF 6 PALISADES NUCLEAR PLANT 97 - 007 - 00 TEXT (If more space 1s required, use additional copies of NRC Form 366A) (17)

ANALYSIS OF EVENT The Palisades emergency diesel generators (EOGs) *are designed to 1) provide power promptly to engineered safety features if loss of off-site power occurs and 2) provide power to equipment needed to safely shut down the plant and/or maintain the plant in a safe shutdown condition during the loss of off-site power event. If a design basis accident were to occur coincident with ,,

loss of off-site power, the appropriate emergency equipment would be loaded onto the associated EOG in a series of steps or sequence so as not to overload the EOG with simultaneous loading of all the equipment at once. In this condition, the SIS-X relays operate to block a simultaneous start of all loads. Simultaneous loading of all required post accident equipment does occur if off-site power is available.

The contacts used currently to block the SIS-X relays were added to the design by Facility Change 561 in late 1983. At that time, however, the contacts were not relied upon to block the SIS-X relays for a simultaneous SIS actuation with a loss of off-site power. Rather, the contacts were used to allow an orderly restoration of off-site power following a SIS actuation. At that time, a lockout relay on startup power transformer 1-2 prevented block lo~ding of the safety injection loads on each EOG.

In late 1989, Facility Change 800 was installed. This modification replaced the lockout relay contact in each SIS-X block circuit with non seal-in contacts from bus 1C and 10 undervoltage circuits. These contacts would close following reenergization of the safety buses by the EOGs.

Thus, the 1060 sequencer control relay contacts in the block SIS-X circuit were required to prevent block loading the EOGs. Prior to 1989, these contacts had not been required to perform this function.

While reviewing the electrical system surveillance test procedures, as requested by Generic Letter 96-01, *it was determined that a contact on each sequencer control relay 1060-1 /XL and 1060-2/XL was not adequately tested. Each contact is in its associated EOG circuit to prevent simultaneous JQading_oUhe_.safetyJoads_ontoJhe ..EDG inJhe..evenLota .desi.gn .basis. accident coincident with a loss of off-site power. These contacts are in series with other contacts which at various times in the EOG start sequence prevent the safety loads from being actuated (see Figure 1 ). The test procedure (RT-8C/O) for testing the circuitry simulates a bus undervoltage by pulling the bus Potential Transformer fuses. This action opens the bus undervoltage relay 127-1-X2 or the 127-2-X2 contact in the appropriate block SIS-X logic circuit. As the fuses are not reinstalled until SIS actuation is reset, the 127-1-X2 or 127-2-X2 contact remains open, and the action of the 1060 relay contact in the EOG circuit is not adequately verified. Thus, failure of the 1060 relay contact to open would not be detected during the test. Note that the testing that is in place verifies that all other contacts on the 1060 relays are properly tested.

NRC FORM 366a U.S. NUCLEAR REGULATORY COMMISSION 4/95 LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION FACILITY NAME 11 l OOCKET12\ LER NUMBER I 6\ PAGE 05000255 YEAR I SEQUENTIAL NUMBER REVISION NUMBER 40F6 PALISADES NUCLEAR PLANT 97 - 007 - 00 TEXT (If more space 1s required, use add1t1onal copies of NRC Form 366A) (17)

The electrical test procedures have been revised and reviewed extensively over the years. The last extensive review of these procedures was performed in late July and early August 1995. This review was performed as part of the corrective action that was initiated due to a finding that the containment high pressure trips were inoperable. As documented in LER 95-008-01, this review was performed to verify that procedures used for testing safety related circuits completely test the design functions of the equipment. Further, LER 95-008-01 committed to reevaluate the responses to Industry Experience reports pertaining to inadequate circuit modification and testing.

Specifically, the commitment referred to a review of NRC Information Notices IN-88-83, "Inadequate Testing of Relay Contacts in Safety-Related Circuits," and IN-93-38, "Inadequate Testing of Engineered Safety Features Actuation Systems." These and other information notices taken together were the basis for the NRC issuing Generic Letter 96-01 on January 10, 1996; requiring comprehensive testing reviews.

Discussions with the engineer who, in mid 1995, reviewed those portions of the test containing sequencer control 1060 relays indicates that he missed the fact that these relay contacts were not tested. He stated that the evaluation performed at that time was based on the logic diagrams.

The fact that the 127 undervoltage relays would be open throughout the test, resulting in the subject 1060 relay contacts not being verified, would be difficult to discern as part of a review using logic diagrams.

The current review being performed to meet Generic Letter 96-01 is a more thorough review of the circuitry. This review is at the schematic level. Each relay and control switch contact is being individually identified and cross referenced to a procedure step which verifies the contact function.

This method of evaluating the testing is much more rigorous than reviews previously performed.

SAFETY SIGNIFICANCE The 1060 relay contacts associated with blocking SIS-X relay actuation were immediately tested satisfactorily following identification of the problem. The contacts were shown to properly open upon energization of the relay, and effectively _block the SIS-X relays to prevent EOG block loading. Based on this testing and a review of the history of the subject relays, it was determined that the relay contacts were always capable of performing their required safety function. Thus, failure to test this contact did not adversely affect the safe operation of the plant or the health and safety of the public.

PAST OPERABILITY A review of work order history did not identify any work being performed on these relays following the modification to put one 1060 contact in the block SIS-X circuit of each EOG.

NRC FORM 366:i U.S. NUCLEAR REGULATORY COMMISSION 4/95 LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION FACILITY NAME 11 l DOCKETl2l LER NUMBER 16\ PAGE 05000255 YEAR I SEQUENTIAL NUMBER REVISION NUMBER

• 5 OF 6 PALISADES NUCLEAR PLANT 97 - 007 - 00 TEXT (If more space 1s required, use add1t1onal copies of NRC Form 366A) (17)

Also, since the relays and other adjacent contacts have been tested regularly as part of the technical specification test program, it is highly unlikely that any problem affecting these untested contacts would not have been discovered.

Based on the successful test of the untested relay contacts and past history revealing no documented work on the relays following implementation of FC-800, it is concluded that there was no time in the past where the relay contacts would not have performed their function. Thus, there are no past operability concerns related to the relay contacts not being tested.

CAUSE OF THE EVENT The root cause of this event was an oversight on the part of personnel involved in the installation of Facility Change FC-800. The change to the logic made by that modification was not adequately tested as part of the post-modification test. The personnel involved in installing that modification over relied on existing surveillance testing to verify design functions. It was wrongly believed that Technical Specification test procedures RT-BC and RT-80 would adequately test the modification.

Additional oversights occurred in later reviews of these tests which further contributed to the event. The non-tested condition of the subject 1060-1 /XL and 1060-2/XL relay contacts was a very subtle aspect within the approved test methodology and difficult to detect. The difficulty was due to opened contacts in series with the subject contacts.

We have reviewed this event and its potential implications against the Palisades Post-Modification Test Program. The modification that made the change to the plant that resulted in reliance on these particular contacts was completed in 1989 under Facility Change 800. As identified in LER 95-008, similar problems with post-modification testing were identified on modifications completed in 1993. We have implemented a series of corrective actions intended to address the post-modification test issues. These actions serve to concentrate and focus both engineering and operating expertise into a test authority responsible for the review of post-modification test.procedures. W.e..are__cootinuingJo__evaluate the_eff.e.ctiveoess of these actions.

The process of implementation of the Generic Letter 96-01 actions is, in itself, a learning experience as to how subtle these design testing conditions can be, and how detailed reviews must be to identify them. We will fold the lessons learned from the Generic Letter 96-01 program into our overall post modification testing program as appropriate.

NRC FORM 366'3 U.S. NUCLEAR REGULATORY COMMISSION 4/95 LICENSEE EVENT REPORT (LER)

TEXT CONTINUATION I

FACILITY NAME 111 DOCKET121

• LER NUMBER' 61 PAGE YEAR SEQUENTIAL REVISION NUMBER NUMBER 05000255 6 OF 6 PALISADES NUCLEAR PLANT 97 - 007 - 00 TEXT (If more space 1s required, use additional copies of NRC Form 366A) (17)

CORRECTIVE ACTION

1. Testing of the 106D-1 /XL and 106D-2/XL relay contacts to verify proper operation of these contacts was successfully completed on August 26, 1997, immediately following identification of the test discrepancy.

2. This incident was reviewed with engineering personnel responsible for developing and reviewing tests for safety related logic circuitry as well as the piant test authority personnel responsible for reviewing post-modification testing. This same group of people are also participating in the Generic Letter*96-01 review presently underway and, therefore, are all familiar with the tes.t requirements based on their knowledge of the Generic Letter.

3. The reviews requested by NRC Generic Letter 96-01 will be completed on the current schedule. The commitment to complete the Generic Letter review is provided in a letter to the NRC dated April 17, 1997, and commits to completing the reviews by the end of the 1998 refueling outage.

4. Prior to startup from the 1998 refueling outage, the Technical Specification Test Program will be revised and testing performed to verify that the 106D-1 /XL and 106D-2/XL relay contacts, which input to block actuation of the SIS-X relays, function as designed. The Technical Specification Test Program revision will assure that the subject relay contacts will be tested on a refueling basis thereafter.

ADDITIONAL INFORMATION A related testing issue was reported under LER 95-008.

• Figure 1 In our Refueling Technical Specification Test we simulate a LOOP (coincident with SIS) by pulling the primary PT fuses for the bus (channel) under test. This causes 127-l-X2 5-6 to open. Due to the order of test steps, the PT fuses are not reinstalled until after SIS is reset. Consequently. the function of contact 1060-1/XL 1-2 (block simultaneous starting of all safety loads on DG) was not verified.

Note: All other contacts on relay l 060-1/XL were verified by existing tests. *

• Not-Tested Contact* Cl3L 1060-1/XL Contact opens simultaneously with closure of DG Output Breaker SIS-5 Contact Closes on Safety Injection 3 (5) Test Circuit Contact TX-L 4 (6) Contact opens during select testing 4 Contact opens to drop-out SISX relays 62USIS (prevents lock-in of breaker anti-pump 6 circuits)

Cl3L (TOO 15 SEC)

Xl3 C04L f 62USIS Xl3 19400 s

127-l-X2 Bus undervoltage (auxiliary) relay 6 contacts. Contacts open upon 5 sensed undervoltage condition on 162-153-Xl safety related bus.

6 19400 SISX C04L f Safety Injection block loading relays. During SIS witb offsite SISX power ayailahle. relays pickup Cl3L to simultaneously start safety loads.

SIS-Xl SIS-X3 SIS-XS SIS-X7 Partial of Scheme E-20'J SH. 1