05000220/LER-2010-001

LER-2010-001, Reactor Scram Due to Inadequate Post Maintenance Testing

Docket Number
Event date:	11-10-2010
Report date:	01-07-2011
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(B), System Actuation 10 CFR 50.73(a)(2)(iv)(A), System Actuation
2202010001R00 - NRC Website

A. PRE-EVENT PLANT CONDITIONS:

Prior to the scram, Nine Mile Point Unit 1 (NMP1) was operating at 100 percent power.

B. EVENT:

At 1056 on November 10, 2010, NMP1 scrammed from full power operation due to closure of air-operated, outboard Main Steam Isolation Valves (MSIVs) 01-03 and 01-04. Valves 01-03 and 01-04 closed following receipt of an invalid low-low reactor water level signal during performance of the reactor low-low water level quarterly channel surveillance test.

Nine Mile Point Unit 1 outboard MSIVs 01-03 and 01-04 are air operated valves. Each MSIV is designed with two solenoid operated valves (SOVs) such that when one or both SOV(s) are energized, air is ported to the MSIV and the valve is held open. Each SOV receives an input from the reactor vessel isolation Channel 11 and Channel 12 one-out-of-two taken twice logic. If this logic is met, the SOVs will deenergize causing redirection of the air supply and the MSIVs to close (see attached Figure 1 for SOV and logic configuration). Reactor low-low level is an input signal to the reactor vessel isolation one-out-of-two taken twice logic.

On November 10, 2010, at approximately 1032, maintenance technicians were given permission to perform N1-ISP-036-004, Lo-Lo RPV Level Instrument Trip Channel Test/Calibration, Attachment 1, for isolation Channel 11. In performing Attachment 1, technicians inserted a reactor low-low level signal which deenergized the Channel 11 isolation relay logic. Normally, this would have resulted in generating only half Channel 11 and half Channel 12 isolation signals. However, two preexisting plant conditions resulted in an MSIV closure and the subsequent reactor scram on MSIV position. Specifically, the following conditions existed:

1) SOV-01-04D and SOV-01-03D, the Channel 11 MSIV SOVs, were each found to have misaligned connector pins in their respective Grayboot splice connectors. The connectors are located in the SOV's power circuit which meant that both Channel 11 MSIV SOVs were deenergized. In other words, one of the two SOVs associated with each outboard MSIV was deenergized prior to the surveillance test. Nine Mile Point Nuclear Station, LLC has determined that this condition existed since 2005 when the Grayboot connectors were installed as part of the MSIV SOV replacement effort.

2) Relay 12K74 (General Electric (GE) Model CR305) was found to have a misaligned contact spring and evidence of electrical arcing on the contact that provides input to the Channel 12 reactor vessel isolation logic. Nine Mile Point Nuclear Station, LLC believes that this contact failed open following performance of the previous relay surveillance performed on August 12, 2010.

As stated above, the misaligned connector pins meant that the Channel 11 side MSIV SOVs remained deenergized since the connectors were installed in 2005. The misaligned contact spring in relay 12K74 caused a half Channel 12 isolation signal. In this configuration, only Channel 11 of the one-out-of-two taken twice logic needed to be completed for an MSIV isolation to occur.

During performance of surveillance test N1-ISP-036-004, the induced Channel 11 low-low level completed the Channel 12 isolation signal and deenergized the Channel 12 MSIV SOVs.

Together with the two preexisting conditions, the performance of the surveillance test resulted in closure of the outboard MSIVs and the subsequent scram on MSIV position.

Following the scram, N1-SOP-1, Reactor Scram, and N1-E0P-2, RPV Control, were entered and executed. Reactor vessel pressure was controlled using Emergency Condenser 11 until use of the main heat sink (condenser) was reestablished. The High Pressure Coolant Injection (HPCI) System initiated, as designed, on a low reactor water level signal.

THE EVENT:

Relay 12K74 was found to have a misaligned contact spring and evidence of electrical arcing on the contact that provides input to the Channel 12 reactor vessel isolation logic.

SOV-01-04D and SOV-01-03D, the Channel 11 MSIV SOVs, were each found to have misaligned connector pins in their respective Grayboot splice connectors. However, the MSIVs remained operable in that the two preexisting conditions would not have prevented the MSIVs from closing.

D. DATES AND APPROXIMATE TIMES OF MAJOR OCCURRENCES:

4/25/2005 - The Grayboot splice connectors associated with Channel 11 MSIV solenoid operated valves SOV-01-04D and SOV-01-03D were installed in 2005. Nine Mile Point Nuclear Station, LLC has determined that the connecter pins in the Grayboot connectors have been misaligned since their installation.

8/12/2010 - Relay 12K74 was cycled for surveillance testing on August 12, 2010. The contact associated with the low-low reactor vessel isolation logic remained in an open state since performance of this test.

11/10/2010 - At 1056 on November 10, 2010, NMP1 scrammed from.full power operation due to closure of outboard MSIVs 01-03 and 01-04. Valves 01-03 and 01-04 closed following receipt of an invalid low-low reactor level signal.

E. OTHER SYSTEMS OR SECONDARY FUNCTIONS AFFECTED:

No systems or secondary functions were affected by this condition. This event did not affect Nine Mile Point Unit 2.

F. METHOD OF DISCOVERY:

The reactor scram was self-revealing via multiple control room indications.

G. MAJOR OPERATOR ACTION:

Following the scram, Operations confirmed that all control rods had fully inserted. N1-SOP­ 1, Reactor Scram, and N1-EOP-2, RPV Control, were entered and executed. Plant operators took manual control of RPV level control and stabilized level between 53 and 95 inches. Reactor vessel pressure was controlled by manually initiating Emergency Condenser 11 until the main heat sink (condenser) was reestablished. Emergency Condenser 11 was then secured. The HPCI System initiated, as designed, on a low reactor vessel water level signal. The HPCI System was then secured following restoration of normal water level.

H. SAFETY SYSTEM RESPONSES:

Nine Mile Point Unit 1 outboard MSIVs 01-03 and 01-04 closed as expected with both Channel 11 and 12 SOVs deenergized due to a combination of preexisting plant conditions and performance of the low-low reactor water level surveillance test. All control rods fully inserted as designed on closure of the MSIVs. Partial closure of MSIVs in both main steam lines produces a scram so the reactor is not operated without its main heat sink. Reactor vessel pressure was controlled by manually initiating Emergency Condenser 11 until the main heat sink (condenser) was reestablished by reopening MSIVs 01-03 and 01-04. The HPCI System initiated, as designed, on a low reactor vessel water level signal. At NMP1, HPCI initiation on a reactor low level signal is an expected occurrence due to water level shrinkage following a scram.

II. CAUSE OF EVENT:

The November 10, 2010 scram was the result of the combination of two latent preexisting plant conditions and performance of a quarterly instrument channel surveillance test. The first preexisting condition was misaligned connector pins on the Grayboot splice connectors found in the power circuit of the outboard MSIV Channel 11 SOVs. The cause of the misalignment has been determined to be insufficient rigor in the behaviors and knowledge (training) used in determining the appropriate post maintenance testing (PMT) in 2005 for redundant/complex control circuits.

Following replacement of the SOVs (which included installation of the Grayboot connectors), the PMT identified and performed was cycling the outboard MSIVs per surveillance procedure.

However, since each MSIV is designed with two redundant SOVs, each capable of porting air to or away from the MSIV, the test performed was not adequate to identify the failed Grayboot connection or the deenergized SOV. A contributing cause was inadequate technical guidance in the maintenance procedure and vendor manual regarding inspection and testing of the Grayboot connectors.

The second preexisting condition was a misaligned contact spring in isolation logic Channel 12 relay 12K74 which was installed in April 2005. The cause of the misalignment has been determined to be excess material (plastic) left on the contact spring holding peg during fabrication of the relay's movable contact holder. The excess material allowed the position of the contact spring to change and uneven forces to be applied to the movable contact. Consequently, the contact associated with the low-low level isolation logic remained open since the relay was cycled during the previous surveillance test performed in August, 2010.

The misaligned connector pins and contact spring, together with insertion of a reactor vessel low­ low water level signal for surveillance testing, resulted in an MSIV closure and subsequent reactor scram on MSIV position.

Condition Report 2010-011008 applies to this LER.

III. ANALYSIS OF THE EVENT:

This event is reportable in accordance with 10 CFR 50.73(a)(2)(iv)(A) as an event or condition that resulted in manual or automatic actuation of any of the systems listed in paragraph 10 CFR 50.73(a)(2)(iv)(B). On November 10, 2010, closure of outboard MSIVs 01-03 and 01-04 resulted in a reactor scram and initiation of HPCI.

The SOV Grayboot splice connectors were left in a misaligned position since 2005. However, this misalignment resulted in the subject SOVs being left in a deenergized state (i.e., the required state to port air away from the MSIVs) and therefore would not have prevented the MSIVs from closing.

Relay 12K74 failed in a position such that one path of the Channel 12 reactor vessel isolation logic was completed. Therefore, its failure would not have prevented the MSIVs from closing and performing their safety function.

The November 10, 2010 scram is bounded by the NMP1 Updated Final Safety Analysis Report (UFSAR) analyzed transient delineated in UFSAR Chapter XV, Section A.3.5, Main Steam Isolation Valve Closure, (With Scram). The analyzed event assumes the reactor is at rated power, an MSIV closure occurs and that a scram is initiated automatically by MSIV position. The UFSAR also states that the Electromatic Relief Valve (ERV) set pressures are low enough to prevent lifting of the Reactor Pressure Vessel (RPV) safety valves. No ERVs lifted during this event.

A walkdown of the Main Steam System was performed per procedure and identified no evidence of damage to snubbers or supporting structural members.

Based on the above, it is concluded that the safety significance of this event is low and the event did not pose a threat to the health and safety of the public or plant personnel.

This event does impact the NRC Regulatory Oversight Process (ROP) Index for Unplanned Scrams. Due to this scram, the Unplanned Scram Index value will be 0.8 compared to a Green-to- White threshold value of greater than 3.

IV. CORRECTIVE ACTIONS:

A. ACTION TAKEN TO RETURN AFFECTED SYSTEMS TO PRE-EVENT NORMAL STATUS:

Reactor vessel pressure was controlled using Emergency Condenser 11 until the main heat sink (condenser) was reestablished; the Emergency Condenser was then secured. The HPCI System initiated, as designed, on a low reactor vessel water level signal following the scram.

HPCI was secured following restoration of normal water level.

The Channel 11 Grayboot splice connector misaligned connector pins were repaired and an appropriate PMT performed to assure proper operation of the SOVs. Channel 12 relay 12K74 was replaced and tested to assure proper operation.

B. ACTION TAKEN OR PLANNED TO PREVENT RECURRENCE:

Actions which were taken or are planned to be taken include the following:

Grayboot Splice Connector 1) Immediate actions were to repair the Channel 11 Grayboot splice connectors.

2) Training will be provided to planning personnel to ensure an adequate understanding of complex/redundant circuits for the proper determination of PMT requirements.

3) Procedure/manuals governing Grayboot connector maintenance activities will be revised to include additional post assembly inspections and testing. This will include a continuity check following each Grayboot installation.

General Electric (GE) Relay 12K74 1) Immediate action was to replace Channel 12 relay 12K74 and inspect two other CR305 relays.

2) Four additional CR305 relays from the same lot as the failed relay will be inspected. If additional deficiencies are found, additional corrective actions will be initiated.

A. FAILED COMPONENTS:

Relay 12K74 was found to have a misaligned contact spring and evidence of electrical arcing on the contact that provides input to the Channel 12 reactor vessel isolation logic.

SOV-01-04D and SOV-01-03D, the Channel 11 MSIV SOVs, were each found to have misaligned connector pins in their respective Grayboot splice connectors.

B. PREVIOUS LERs ON SIMILAR EVENTS:

None C. THE ENERGY INDUSTRY IDENTIFICATION SYSTEM (EllS) COMPONENT FUNCTION

IN THIS LER:

COMPONENT� IEEE 803 FUNCTION� IEEE 805 SYSTEM IDENTIFIER� IDENTIFICATION Plant Protection System� -� JC Reactor Vessel Isolation System� JM Emergency Condenser� COND� BL Main Steam Isolation Valve� ISV� SB Relay� RLY� JC Connector� CON� SB Solenoid Operated Valve� FSV� SB

D. SPECIAL COMMENTS:

None