05000388/LER-2012-002, Regarding Manual Scram Due to Loss of the Integrated Control System

Regarding Manual Scram Due to Loss of the Integrated Control System

ML13008A340
Person / Time
Site:	Susquehanna
Issue date:	01/07/2013
From:	Helsel J Susquehanna
To:	Document Control Desk, Office of Nuclear Reactor Regulation
References
PLA-6953 LER 12-002-00
Download: ML13008A340 (6)

LER-2012-002, Regarding Manual Scram Due to Loss of the Integrated Control System

Event date:
Report date:
Reporting criterion:	10 CFR 50.73(a)(2)(iv)(A), System Actuation 10 CFR 50.73(a)(2)(i) 10 CFR 50.73(a)(2)(vii), Common Cause Inoperability 10 CFR 50.73(a)(2)(ii)(A), Seriously Degraded 10 CFR 50.73(a)(2)(ii)(B), Unanalyzed Condition 10 CFR 50.73(a)(2)(viii)(B) 10 CFR 50.73(a)(2)(iii) 10 CFR 50.73(a)(2)(v)(A), Loss of Safety Function - Shutdown the Reactor 10 CFR 50.73(a)(2)(v)(B), Loss of Safety Function - Remove Residual Heat 10 CFR 50.73(a)(2)(i)(A), Completion of TS Shutdown 10 CFR 50.73(a)(2)(v), Loss of Safety Function 10 CFR 50.73(a)(2)(i)(B), Prohibited by Technical Specifications
LER closed by
IR 05000387/2013011 (18 November 2013)
3882012002R00 - NRC Website

text

Jeffrey M. Helsel Nuclear Plant Manager U. S. Nuclear Regulatory Commission Attn: Document Control Desk Mail Stop OP1-17 Washington, DC 20555-0001 PPLSusquehanna,LLC 769 Salem Boulevard Berwick, PA 18603 Tel. 570.542.3510 Fax 570.542.1504 jmhelsel@pplweb.com SUSQUEHANNA STEAM ELECTRIC STATION LICENSEE EVENT REPORT 50-388/2012-002-00 UNIT 2 LICENSE NO. NPF-22 PLA-6953 Docket No 50-388 Attached is Licensee Event Report (LER) 50-388/2012-002-00. The event involved a reactor scram and subsequent automatic initiation of various plant systems due to loss of the Integrated Control System. A subsequent scram signal was received due to low water level during recovery from the initial scram. Both events are being reported in accordance with 10 CPR 50.73(a)(2)(iv)(A).

There were no actual consequences to the health and safety of the public as a result of this event.

No regulatory commitments are associated with this LER.

qif~~~~

Attachment: LER 50-388/2012-002-00 Copy: NRC Region I Mr. P. W. Finney, NRC Sr. Resident Inspector Mr. J. A. Whited, NRC Project Manager Mr. L. J. Winker, PA DEP/BRP

NRC FORM 366 U.S. NUCLEAR REGULATORY COMMISSION APPROVED BY OMB: NO. 3150-0104 EXPIRES: 10/31/2013 10-2010)

, the NRC digits/characters for each block) may not conduct or sponsor, and a person is not required to respond to, the information collection.

p. PAGE Susquehanna Steam Electric Station Unit 2 05000388 1 OF5

4. TITLE Unit 2 Manual Scram Due to Loss of the Integrated Control System

5. EVENT DATE

6. LER NUMBER

7. REPORT DATE

8. OTHER FACILITIES INVOLVED YEAR I SEQUENTIAL I REV FACILITY NAME DOCKET NUMBER MONTH DAY YEAR NUMBER NO.

MONTH DAY YEAR 05000 FACILITY NAME DOCKET NUMBER 11 09 2012 2012

- 002

- 00 01

()'7 2013 05000

9. OPERATING MODE

11. THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR §:(Check all that apply) 1 D 20.2201(b)

D 20.2203(a)(3)(i)

D 50.73(a)(2)(i)(C)

D 50.73(a)(2)(vii)

D 20.22o1 (d)

D 20.2203(a)(3)(ii)

D 50.73(a)(2)(ii)(A)

D 50. 73(a)(2)(viii)(A)

D 20.2203(a)(1)

D 20.2203(a)(4)

D 50.73(a)(2)(ii)(B)

D 50.73(a)(2)(viii)(B) 1 0. POWER LEVEL D 20.2203(a)(2)(i)

D 50.36(c)(1 )(i)(A)

D 50.73(a)(2)(iii)

D 50. 73(a)(2)(ix)(A) 90%

D 20.2203(a)(2)(ii)

D 50.36(c)(1 )(ii)(A)

~ 50.73(a)(2)(iv)(A)

D 50. 73(a)(2)(x)

D 20.2203(a)(2)(iii)

D 50.36(c)(2)

D 50.73(a)(2)(v)(A)

D 73.71(a)(4)

D 20.2203(a)(2)(iv)

D 50.46(a)(3)(ii)

D 50.73(a)(2)(v)(B)

D 73.71(a)(5)

D 20.2203(a)(2)(v)

D 50.73(a)(2)(i)(A)

D 50.73(a)(2)(v)(C) 0 OTHER D 20.2203(a)(2)(vi)

D 50.73(a)(2)(i)(B)

D 50.73(a)(2)(v)(D)

Specify in Abstract below or in Background Information Associated with the ICS System Design

3. PAGE 30F5 The ICS is an implementation of an lnvensys Intelligent Automation (1/A) Series Distributed Control System (DCS). The ICS utilizes six fault-tolerant pairs of Field Control Processors. These processors and their related input/output (I/O) subsystems are used to control reactor recirculation pumps, reactor feed pump turbine speeds, and reactor vessel water level. To accomplish these control functions, the processor pairs must communicate with other processor pairs. This communication takes place over a digital communication network referred to as the "Mesh Control Network" or just the "mesh." The mesh utilizes network cabling and multipart network switches to implement a digital communications network that provides multiple communication paths between any two devices on the network. If there is a communications problem between two network devices, the mesh should automatically establish a different connection utilizing a different path. This is accomplished via a software algorithm called Rapid Spanning Tree Protocol (RSTP), which manages the network traffic, eliminating system loops, minimizing data packet collisions, and providing fast switchover if a fault occurs.

Background Information Associated with Monitoring RPV Level With the ICS NR le,vel indication locked up, operators were uncertain as to the reliability of the hard wired instrumentation on the standby information panel (SIP) and relied on wide range (WR) instrumentation. At normal operating pressure, WR indicates approximately 1 0 inches below the NR instrument. As the plant cools down, these indications converge and then diverge, with WR reading almost 10 inches above NR.

Following RPV stratification, the water level band was changed from 45-54 inches to 13-30 inches in accordance with plant procedures. The second scram signal was received with NR levels greater than 15 inches (i.e., the scram signal occurred on the conservative side of the setpoint). WR indication was reading 23 inches when the scram signal was received.

CAUSE OF THE EVENT

The root cause of the scram was as follows:

ICS C2 Series Switches have a Latent Design Deficiency.

The November 9, 2012 plant scram was directly caused by the lockup of one of the two ICS Core switches (Enterasys C2 Series). Troubleshooting indicated that a core switch fault occurred, which both prevented network communication and prevented transfer of network control to the secondary core switch.

The root cause team was able to identify a history of reliability issues with applications of the C2 switches outside the nuclear industry.

The direct causes of the second scram signal included the following:

ICS indication failed, preventing operators from accurately monitoring level while controlling RCIC in manual The scram occurred at 15 inches instead of the required 13 inches The current prescribed level band is inside the scram setpoint of a Level 3 scram Procedures do not adequately caution operators about narrow range instrumentation being the source of the RPS actuation and the potential impacts of ICS failures

ANALYSIS/SAFETY SIGNIFICANCE

Actual Consequences:

2012 40F5

- 002

- 00 Loss of ICS resulted in operators manually scramming Unit 2. Reactor vessel stratification challenged Operations; however, no Technical Specification limits were exceeded. Reactor Water Cleanup (RWCU)

Letdown [EllS System Code CEJ was not available due its tie to ICS. In accordance with NEI 99-02, this was classified as an Unplanned Scram with Complications due to loss of Feedwater.

Although the scram challenged the operators, the safety consequences of the event were well controlled.

The reactor operator placed the mode switch in shutdown when reactor water level reached +25 inches and lowering. All control rods inserted and both reactor recirculation pumps tripped at -38 inches. Reactor water level lowered to -52 inches causing Level 3 (+13 inches) and Level 2 (-38 inches) isolations. HPCI and RCIC both automatically initiated. HPCI was overridden prior to injection and RCIC was utilized to restore reactor water level to the normal band. All isolations and initiations at this level occurred as expected. No steam relief valves opened. Pressure was controlled via turbine bypass valve operation. All safety systems operated as expected.

The ICS responded as expected with the following exception: A High/Low Level alarm was received when level as indicated at the Standby Information Panel (SIP) was 25 inches. This alarm was expected at 30 inches. The alarm is driven off of selected level, which was impacted by the communication failure.

The indicators at the SIP panel do not go through the mesh network.

The actual consequences associated with the second scram signal involved Operations being required to respond and reset the scram to avoid exacerbating RPV stratification.

Potential Consequences:

Had operators not taken manual control and had safety systems not responded as designed, additional challenges and complications could have arisen requiring the use of additional safety systems and potential entry into additional emergency operating procedures.

The potential consequences for the second scram signal would be an additional plant transient which would complicate recovery actions of operators.

Assessment of the risk associated the initiating event determined that the increase in risk to Unit 2 was less than the NRC IMC 609 Appendix K Green/White Threshold of less than 1 E-06 I COP and less than 1 E-07 ILERP.

All safety systems operated as expected; therefore the potential consequences of this event were mitigated.

CORRECTIVE ACTIONS

Key corrective actions include:

3. PAGE 50F5

1. Unit 2 ICS cores switches were replaced with newer version switches (C5) and loop protection algorithms have been enabled.

2. Unit 1 ICS core switches will be replaced with newer version switches (C5) and loop protection algorithms will be enabled on Unit 1.

3. Procedures will be revised to caution operators about narrow range instrumentation being the source of the RPS actuation and the potential impacts of the ICS failures.

4. Procedures will be revised to ensure that the post-scram level band is outside the expected scram set point.

5. Calibration of scram setpoints will be evaluated with regard to the scram occurring at greater than 13 inches.

PREVIOUS SIMILAR EVENTS

Susquehanna has had three previous scrams associated with ICS. These events were as follows:

LER 387/2010-002-00, 01, and 02- "Automatic Reactor Scrams Occur During Post-Modification Testing of the Digital Feedwater Integrated Control System" LER 388/2011-003 "Unit 2 Scram Due to Main Turbine Trip During ICS Testing" Susquehanna also had two recent LERs that involved a similar cause (switch failures):

LER 388/2012-001-01 -"Two Control Room Floor Cooling Systems Inoperable" One of the causes of this event was a design deficiency in a chiller circulation pump control switch.

LER 387/2012-008 "Loss of One of Two Offsite Power Sources" The causes of this event included foreign material from the manufacturing process that prevented an ammeter switch from closing and design of the protective relay scheme that included a shared metering function.