05000265/LER-2010-002
| Docket Number Sequential Revmonth Day Year Year Month Day Year N/A N/Anumber No. | |
| Event date: | |
|---|---|
| Report date: | |
| Reporting criterion: | 10 CFR 50.73(a)(2)(iv)(A), System Actuation |
| Initial Reporting | |
| 2652010002R00 - NRC Website | |
PLANT AND SYSTEM IDENTIFICATION
General Electric - Boiling Water Reactor, 2957 Megawatts Thermal Rated Core Power Energy Industry Identification System (EIIS) codes are identified in the text as [XX].
EVENT IDENTIFICATION
Unit 2 Manual Reactor Scram Due to 2B Recirculation Pump Trip
A. CONDITION PRIOR TO EVENT
�Unit: 2 Event Date: August 17, 2010 � Event Time: 1414 hours0.0164 days <br />0.393 hours <br />0.00234 weeks <br />5.38027e-4 months <br /> � Reactor Mode: 1 Mode Name: Power Operation� Power Level: 100%
B. DESCRIPTION OF EVENT
On 8/17/2010 at 0401, the reactor recirculation control system (RRCS) received a major failure alarm. The system manager was notified, the Duty Team was activated, and preparations were made to address the issue.
On 8/17/2010 at 0930, the Site Troubleshooting Team asked the ASD Vendor simulator subject matter expert (SME) to run a simulation to ensure that re-synchronizing the 'B' PLC would not impact (or trip) the 2B ASD.
On 8/17/2010 at 1030, a Site Troubleshooting Team conference call was made with the ASD Vendor. The ASD Vendor provided that according to simulator runs, work to recover the ASD communications can be performed online.
It was noted, however, that the simulator runs were performed with the ASD simulator control key switch [33] in the local position. The ASD Vendor was challenged on the identified difference between the key switch position run on the simulator (local) and the actual key switch position to be used during work at the station (remote). When questioned, the ASD Vendor stated that this difference would have no impact (no ASD trip) on the recovery of the 'B' PLC. The Team accepted the ASD Vendor response based upon the available information. The ASD Vendor provided a written response at the Team's request that ASD communications recovery can be conducted safely. The verbal information concerning the ASD control key switch position (remote/local), with a follow-on letter from the ASD Vendor was accepted as fact by the Team without requesting a supporting validation.
On 8/17/2010 at 1115, a Corporate Engineering technical challenge was performed to finalize processes for conducting the ASD communications recovery. The ASD control key switch position difference in the simulator run is discussed briefly, but it was noted that the ASD Vendor was providing a written response that states this event will not affect operation of the 2B ASD.
On 8/17/2010 at 1230 a Duty Team Challenge was performed. Although challenged to do so, the Duty Team did not revisit the ASD re-synchronization work original assumptions or the aggregate impact of degraded components, such as the FRV controller status before proceeding with the ASD work. This challenge concluded that it was acceptable to proceed with the ASD work.
On 8/17/2010, between 1230 and 1411, Instrument Technicians, Operations, and Engineering performed work order activities through work step G.11 which downloaded data and reset the PLC. Step G.12 required Instrument � Technicians to carefully observe the PLC central processing unit [CPU] (CPU) with the flashing light emitting diode [IL] (LED). In 120 seconds, the flashing light was expected to remain steady indicating a successful synchronization.
During step G.12, the 2B ASD tripped. No physical activities or critical steps were in progress at the time. This activity was observed by a Maintenance Supervisor, Operations Field Supervisor, and a member of the Duty Team (Engineering Manager).
Shortly after the 2B ASD trip, Operators noticed that the 2B recirc pump motor [MO] had tripped. At this time the RPV water level was at 34 inches and rising. Operators then transferred the 2B FRV from manual to automatic control per operating guidance. The 2B FRV immediately started ramping close at approximately 0.2% per second. There was no direction to the Operators to use manual control to close the FRV.
The 2B FRV closure rate was insufficient to prevent the reactor level from increasing. The fixed ramp rate of 0.2% per second is a designed program set-point to prevent a rapid RPV water level change during a transfer from manual to automatic FRV control.
On 8/17/2010 at 1414, as the RPV water level increased to 44 inches (the previously set manual scram criteria), Operators were required to insert a manual scram on Unit 2. A reactor scram was unavoidable under this condition.
The digital [DCC] feedwater level control [LC] (DFWLC) system operated as designed. The rising RPV water level was caused by the trip of the 2B recirc pump motor and the 2B FRV controls initially being in the manual mode.
C. CAUSE OF EVENT
The root cause of the recirc pump trip and subsequent reactor scram is management's failure to recognize and effectively challenge critical assumptions used in authorizing work activities for performing the ASD resynchronization, and in determining risk for the 2B FRV. Both process reviews failed to adequately assess and manage risk prior to performing these activities.
The cause of the 2B ASD trip and subsequent 2B recirc pump motor trip was that re-synchronization of the 2B ASD 'B' PLC was performed with the ASD control key switch in the remote position while a communication loss from the 'A' PLC already existed. The re-synchronization caused the ASD logic to trip the 2B ASD and the 2B recirc pump motor per design. Had the ASD key switch been in the local position, the logic in the ASD software would not have caused the 2B ASD to trip. Review of the events leading up to the Unit 2 scram identified several opportunities for the station to identify this issue had assumptions been validated, testing been performed on the simulator, or had independent technical human performance reviews been performed.
On 8/17/2010 a Corporate Engineering technical review was performed to challenge the process for reestablishing redundancy within the 2B ASD and PLC CPU resynchronization activities. During the Corporate challenge, the ASD control key switch position discrepancy between the ASD Vendor simulated run (switch in local position) and the actual plant configuration (switch in remote position) was not specifically identified when differences were discussed between the simulator and the plant. The only difference identified was that the PLC resynchronization attempt would be performed coincident with a loss of communication on the A side (unlike previous resynchronizations), and that the ASD Vendor would send a letter confirming that this posed no risk to the plant.
The ASD Vendor letter did not discuss the differences in switch position, but provided that no additional risks were identified when attempting the 'B' PLC CPU resynchronization on ASD given the loss of communication on the 'A' PLC side. Subsequent to this scram event, simulator runs confirmed that the ASD would trip while the key switch is in the remote position.
� The cause of the increasing reactor water level was due to the rapid steam flow / feedwater flow mismatch when the 2B ASD tripped the 2B recirc pump motor. Water level increases continued since the 2B FRV was in manual, and transferring the 2B FRV to automatic did not allow a sufficient valve closure rate due to a fixed ramp rate for closure (0.2% per second). The "bumpless transfer" fixed ramp rate of 0.2% per second is a designed program set-point to prevent a rapid RPV water level change during transfer to automatic control. Prior to the 2B ASD drive trip, the 2A FRV was in automatic (approximately 55% open) and the 2B FRV was in manual (approximately 62% open) due to ongoing issues with the 2B FRV hunting/oscillating while in the automatic position. Since the 2B FRV closure rate was not fast enough to keep the reactor level from increasing, this required operators to manually insert a reactor scram at 44 inches RPV water level. A reactor scram could not have been avoided under these plant conditions. The DFWLC system operated as designed.
On 4/25/10, the 2B FRV was discovered to be "hunting," i.e., oscillating back and forth around a flow point, whereas the FRV would function as designed, the concern was that the oscillations would eventually cause packing leaks, which would get worse over time. On 5/6/10, an Operational and Technical Decision Making (OTDM) document had been requested from Operations. On 6/11/2010, an OTDM was prepared, reviewed and approved. Included in this OTDM was the following, "Operational impact comes from the fact that IMMEDIATELY following a plant transient, the -NSO has to remember that this valve is in MANUAL and take it to AUTO." This statement was inserted into the OTDM based upon statements made during a prior related Duty Team meeting, but the review and approval process for the OTDM did not require any technical human performance or critical parameter review. In addition, no simulator evaluation was performed regarding the FRV response in different scenarios. On 8/3/2010, a follow-up Adverse Condition Monitoring Plan (ACMP) was prepared, reviewed, and approved. This ACMP contained the same information as the OTDM and added actions to place the 2B FRV in automatic per procedure, Operation of Feedwater Level Control System. It also contained the manual scram criteria of 44 inches RPV water level should RPV water level be unable to be maintained. The ACMP development was another missed opportunity to challenge the 2B FRV transfer from manual to auto under different scenarios.
D. SAFETY ANALYSIS
The safety significance of this event was minimal. The reactor and turbine responded as designed. Operators performed required actions safely and in accordance with procedures and training. There were no complications during the reactor scram and turbine trip, and all systems functioned as required.
The manual shutdown of Quad Cities Unit 2 did not involve any failure of plant equipment before or after the manual scram. The initiator, 'Turbine Trip With Bypass" (TT), includes malfunctions or operator errors that result in manual or automatic shutdown of the turbine with steam bypass to the main condenser operable. The initiator TT contributes 2% to the core damage frequency (CDF) in the plant probabilistic risk assessment (PRA) model. This initiator is not in the Top 10 cut-sets. None of the failures modeled in the Top 10 cut-sets occurred. An examination of the Top 10 Accident Sequences also yielded similar results. Even though a successful scram is part of each of these sequences, none of the postulated failures occurred before or after the scram. The conclusion reached is that the manual shutdown during this event was not risk significant.
_
E. CORRECTIVE ACTIONS
Immediate:
- Adjusted gains and replaced servo valve on the 2B FRV during the Q2F65 outage.
Follow-up:
- Review/revise, as necessary, work management systems to ensure that the importance or impact of subtle problems are recognized, prioritized, and addressed. Integrate the Operations Aggregate Impact Procedure as one of the inputs to the Duty Team. Ensure simulator testing is fully vetted before approving work.
- Review Duty Team Checklists and Work Control Week Challenges for existing degraded conditions that would impact the risk or consequences of this event, and address the need for a simulator challenge.
- Engineering will establish the expectation that ACMP and OTDM assessments require the application of an additional technical human performance review under procedure, Technical Task/Rigor Assessment. The Technical Task/Rigor Assessment procedure will be revised to ensure ACMPs and OTDMs are addressed.
- Revise procedure Main Feedwater Regulator Operation, to include a specific note describing the impact of the of 0.2% flow per second valve closure rate when steam/feed flow is mismatched.
- Revise ASD training to address the consequences of the LOCAL vs. REMOTE switch position.
- Revise training and simulator lesson plans/scenarios for DFWLC; include negative aspects of FRV controls "bumpless transfer.
- Conduct case studies with managers and supervisors to discuss engaged, thinking organizations and station standards, using this event and other significant events as examples to focus the discussions.
F. PREVIOUS OCCURRENCES
The Station Events Database, EPIX, NPRDS, and LERs were reviewed for similar events at Quad Cities Station.
This event was caused by failure to adequately assess and manage risk of ASD resynchronization and FRV controls when transferring from manual to automatic. There were no prior incidents identified involving a reactor scram due to ASD resynchronization issues coupled with FRV control issues at Quad Cities. A scram involving FRV controls occurred in 2002 and the LER is described below.
- Station Events Database - Quad Cities Investigation Report (IR) 102589 - U2 Manual Scram on Increasing Reactor Pressure Vessel Level (4/5/02) — See LER below.
- EPIX/ NPRDS — No similar events identified for Quad Cities.
- LER 05000265/2002-002-00, Manual Scram Due to Reactor Level Transient as a Result of a Digital Feedwater Level Control System Design Error (05/24/02) - A manual scram was inserted on Unit 2 in response to increasing reactor water level. The increase in reactor water level was due to a blown fuse in the DFWLC system, caused by the inadvertent grounding of test leads during an instrument surveillance. Although the DFWLC logic is intended to be able to respond to a blown fuse without causing a level transient, the indicating fuse holder provided a circuit that inhibited recognition of the blown fuse by the DFWLC logic. Corrective actions taken were, however, applied primarily to removal of the indicating fuse holders and actions to minimize the potential for grounding test leads, and since that event did not identify the FRV control "bumpless transfer" vulnerability issue, the corrective actions would not have mitigated the manual scram event as identified in this 2010 LER.
G. COMPONENT FAILURE DATA
This event has been reported to EPIX as Failure Report No. 1064.
� NRC FORM 366A (9-2007) PRINTED ON RECYCLED PAPER