IR 05000254/2023001
| ML23128A101 | |
| Person / Time | |
|---|---|
| Site: | Quad Cities  |
| Issue date: | 05/08/2023 |
| From: | Robert Ruiz NRC/RGN-III/DORS/RPB1 |
| To: | Rhoades D Constellation Energy Generation, Constellation Nuclear |
| References | |
| IR 2023001 | |
| Download: ML23128A101 (1) | |
Text
SUBJECT:
QUAD CITIES NUCLEAR POWER STATION - INTEGRATED INSPECTION REPORT 05000254/2023001 AND 05000265/2023001
Dear David Rhoades:
On March 31, 2023, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at Quad Cities Nuclear Power Station. On April 25, 2023, the NRC inspectors discussed the results of this inspection with Brian Wake, Site Vice President, and other members of your staff.
The results of this inspection are documented in the enclosed report.
One finding of very low safety significance (Green) is documented in this report. This finding involved a violation of NRC requirements. We are treating this violation as a non-cited violation (NCV) consistent with Section 2.3.2 of the Enforcement Policy.
If you contest the violation or the significance or severity of the violation documented in this inspection report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN:
Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region III; the Director, Office of Enforcement; and the NRC Resident Inspector at Quad Cities Nuclear Power Station.
If you disagree with a cross-cutting aspect assignment in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region III; and the NRC Resident Inspector at Quad Cities Nuclear Power Station.
May 8, 2023 This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding.
Sincerely, Robert Ruiz, Chief Reactor Projects Branch 1 Division of Operating Reactor Safety Docket Nos. 05000254 and 05000265 License Nos. DPR-29 and DPR-30
Enclosure:
As stated
Inspection Report
Docket Numbers:
05000254 and 05000265
License Numbers:
Report Numbers:
05000254/2023001 and 05000265/2023001
Enterprise Identifier:
I-2023-001-0066
Licensee:
Constellation Nuclear
Facility:
Quad Cities Nuclear Power Station
Location:
Cordova, IL
Inspection Dates:
January 01, 2023 to March 31, 2023
Inspectors:
Z. Coffman, Resident Inspector
C. Hunt, Senior Resident Inspector
C. Mathews, Illinois Emergency Management Agency
L. Rodriguez, Senior Reactor Inspector
A. Tran, Resident Inspector
Approved By:
Robert Ruiz, Chief
Reactor Projects Branch 1
Division of Operating Reactor Safety
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting an integrated inspection at Quad Cities Nuclear Power Station, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Inadequate Procedure for Feedwater System Failure Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events Green NCV 05000254,05000265/2023001-01 Open/Closed
[H.14] -
Conservative Bias 71152A The inspectors identified a finding of very low safety significance (Green) and an associated non-cited violation (NCV) of Technical Specification 5.4.1, "Procedures," for the licensee's failure to have station procedures appropriate for the circumstances to address a failure of the 2B feedwater regulating valve (FRV) in the full open position. As a result, on November 4, 2022, following a failure of the 2B FRV in the full open position, operators were required to insert a manual trip of the reactor on high water level when they could not arrest the subsequent reactor water level transient. The overfeed condition was due to a latent design vulnerability in the 2B FRV allowing it to be capable of passing greater than 100 percent total feedwater flow at full reactor power.
Additional Tracking Items
Type Issue Number Title Report Section Status LER 05000265/22-003-00 LER 022-003-00 for Quad Cities Nuclear Power Station, Unit 2, Manual Scram Due to Feedwater Regulator Valve Failure Increasing Reactor Water Level 71152A Closed
PLANT STATUS
Unit 1 The unit began the inspection period at full-rated thermal power. On January 20, 2023, the unit began its end-of-cycle coastdown period. The unit shut down on March 26, 2023, for refueling outage Q1R27 and remained in a shutdown condition through the end of the inspection period.
For all other periods, the unit was at full-rated thermal power with the exception of short-term power reductions for control rod sequence exchanges, testing, and as requested by the transmission system operator.
Unit 2 The unit began the inspection period at full-rated thermal power. On January 21, 2023, the unit was down powered to approximately 70 percent to perform maintenance on the 2B feedwater regulating valve. The unit was returned to full-rated thermal power on January 22, 2023, where it remained with the exception of short-term power reductions for control rod sequence exchanges, testing, and as requested by the transmission system operator.
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors performed activities described in IMC 2515, Appendix D, Plant Status, observed risk significant activities, and completed on-site portions of IPs. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
REACTOR SAFETY
71111.01 - Adverse Weather Protection
Impending Severe Weather Sample (IP Section 03.02) (1 Sample)
- (1) The inspectors evaluated the adequacy of the overall preparations to protect risk-significant systems from impending severe weather (high winds, thunderstorms, and tornado watch/warning) on March 31, 2023.
71111.04 - Equipment Alignment
Partial Walkdown Sample (IP Section 03.01) (4 Samples)
The inspectors evaluated system configurations during partial walkdowns of the following systems/trains:
(1)1A residual heat removal (RHR) on February 8, 2023 (2)1C RHR on February 16, 2023
- (3) Unit 1 reactor core isolation cooling (RCIC) on February 27, 2023
- (4) Unit 1 high-pressure coolant injection (HPCI) following planned maintenance window on March 1, 2023
71111.05 - Fire Protection
Fire Area Walkdown and Inspection Sample (IP Section 03.01) (6 Samples)
The inspectors evaluated the implementation of the fire protection program by conducting a walkdown and performing a review to verify program compliance, equipment functionality, material condition, and operational readiness of the following fire areas:
- (1) Fire Zone (FZ) 8.2.7.A, Unit 1 turbine building (TB), elevation 615'-6", hydrogen seal oil area and motor control centers on January 4, 2023
- (2) FZs 6.1.A and 6.1.B, Unit 1 TB, elevation 615'-6", 'A' and 'B' battery charger room on February 2, 2023
- (3) FZs 6.2.A and 6.2.B, Unit 2 TB, elevation 615'-6", 'A' and 'B' battery charger room on February 17, 2023
- (4) FZ 11.1.3, Unit 1 reactor building, elevation 554'-0", HPCI and HPCI access tunnel on February 28, 2023
- (5) FZ 8.2.7.B, Q1R27 hot work in Unit 2 TB, elevation 615'-6" and 608'-6", low-pressure heater bay (east and west) on March 29, 2023
- (6) FZ 8.2.6.A, Unit 1 TB, elevation 595'-0", 'D' heater bay on April 5, 2023
71111.06 - Flood Protection Measures
Flooding Sample (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated internal flooding mitigation protections in the:
Flood Zone 8.2.8.A SWGR [switchgear] area MG [motor generator]
set(s)/elevation 639'/building: TB-U1 on February 2, 2023
71111.11Q - Licensed Operator Requalification Program and Licensed Operator Performance
Licensed Operator Performance in the Actual Plant/Main Control Room (IP Section 03.01) (1 Sample)
- (1) The inspectors observed and evaluated licensed operator performance in the control room during the Unit 1 reactor shutdown for Q1R27 on March 27, 2023.
Licensed Operator Requalification Training/Examinations (IP Section 03.02) (1 Sample)
- (1) The inspectors observed and evaluated licensed operator requalification exercises in the simulator on February 9, 2023.
71111.12 - Maintenance Effectiveness
Maintenance Effectiveness (IP Section 03.01) (1 Sample)
The inspectors evaluated the effectiveness of maintenance to ensure the following structures, systems, and components (SSCs) remain capable of performing their intended function:
- (1) Action Request (AR) 4523962, "Unit 2 EDG [emergency diesel generator] Failed to Start," on September 22, 2022
71111.13 - Maintenance Risk Assessments and Emergent Work Control
Risk Assessment and Management Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the accuracy and completeness of risk assessments for the following planned and emergent work activities to ensure configuration changes and appropriate work controls were addressed:
- (1) Units 1 and 2 elevated risk due to Unit 2 125 Vdc battery charger load test on January 10, 2023
- (2) Unit 2 elevated risk during a planned downpower for feedwater regulating valve repairs on January 20, 2023
- (3) E-2 certification meeting and risk management for work week 3/20/2023 on March 9, 2023
- (4) Q1R27 shutdown safety plan review on March 24, 2023
- (5) Q1R27 shutdown safety reviews and protected equipment walkdowns during week of March, 27, 2023
71111.15 - Operability Determinations and Functionality Assessments
Operability Determination or Functionality Assessment (IP Section 03.01) (6 Samples)
The inspectors evaluated the licensee's justifications and actions associated with the following operability determinations and functionality assessments:
(1)
'B' control room emergency ventilation system freeze seal work on January 3, 2023
- (2) AR 4542921, "Recommend Unit 2 SBLC [standby liquid control] Heat Trace Replacement," on February 5, 2023
- (3) Unit 2 HPCI pump piping connection leaks on February 5, 2023
- (4) AR 4558106, "HPCI Signal Converter Motor Gear Unit Drive Signal Issues," on March 1, 2023
- (5) AR 4554301, "Unexpected 901-8 A-4 Alarm," for the 1/2 emergency diesel generator on March 2, 2023
- (6) Unit 1 HPCI turning gear impact on operability on March 3, 2023
71111.20 - Refueling and Other Outage Activities
Refueling/Other Outage Sample (IP Section 03.01) (1 Partial)
(1)
(Partial) Q1R27 refueling outage beginning March 26, 2023 through March 31, 2023
71111.24 - Testing and Maintenance of Equipment Important to Risk
The inspectors evaluated the following testing and maintenance activities to verify system operability and/or functionality:
Post-Maintenance Testing (PMT) (IP Section 03.01) (4 Samples)
(1)1B RHR service water pump comprehensive test on January 18, 2023
- (4) Unit 1 1A core spray pump breaker replacement on March 23, 2023
Surveillance Testing (IP Section 03.01) (7 Samples)
- (1) Unit 1 EDG monthly load test on January 9, 2023
- (2) QCOS 1400-01, "1B Core Spray System Flow Rate Test," on February 13, 2023
- (3) QCOS 1000-44, "Unit 2 'B' Loop LPCI [low-pressure coolant injection] and Containment Cooling Modes of RHRS [residual heat removal system] Non-Outage Logic Test," on March 9, 2023
- (4) QCIS 1400-01 and QCIS 1400-05, "Unit 2 Core Spray Pump Discharge Flow Trip and Cal Test," on March 9, 2023
- (5) QCOS 6500-09, "Function Test of Unit 1 Second Level Undervoltage," on March 13, 2023
- (6) QCOS 7500-05, "SBGTS [standby gas treatment system] Operability Test," on March 16, 2023
- (7) QCOS 1300-05, "RCIC Pump Operability Test," on March 21, 2023
Inservice Testing (IST) (IP Section 03.01) (3 Samples)
- (1) QCOS 1100-07, "SBLC Pump Flow Rate Test," revision 42, for the 2A SBLC pump on February 27, 2023
- (2) QCOS 1300-05, "RCIC Pump Operability Test," revision 63, on March 7, 2023
- (3) QCOS 1400-05, "Core Spray Pump Flow Rate Test," revision 52, on March 22, 2023
Containment Isolation Valve (CIV) Testing (IP Section 03.01) (1 Sample)
- (1) QCOS 0100-11, "HPCI Steam Supply Local Leak Rate Test MO 1(2)-2301-4 MO 1(2)-2301-5," revision 1, on March 28, 2023
Reactor Coolant System Leakage Detection Testing (IP Section 03.01) (1 Sample)
- (1) AR 4553630, "U2 DWFDS [drywell floor drain sump] Leakage Increasing Trend," on February 14, 2023
71114.06 - Drill Evaluation
Select Emergency Preparedness Drills and/or Training for Observation (IP Section 03.01) (1 Sample)
- (1) The inspectors observed activities related to the technical support center during an emergency preparedness drill on February 7, 2023.
Drill/Training Evolution Observation (IP Section 03.02) (1 Sample)
The inspectors evaluated:
- (1) Focused emergency preparedness drill on January 31,
OTHER ACTIVITIES - BASELINE
===71151 - Performance Indicator Verification The inspectors verified licensee performance indicators submittals listed below:
IE01: Unplanned Scrams per 7000 Critical Hours Sample (IP Section 02.01)===
- (1) Unit 1 (January 1, 2022, through December 31, 2022)
- (2) Unit 2 (January 1, 2022, through December 31, 2022)
IE03: Unplanned Power Changes per 7000 Critical Hours Sample (IP Section 02.02) (2 Samples)
- (1) Unit 1 (January 1, 2022, through December 31, 2022)
- (2) Unit 2 (January 1, 2022, through December 31, 2022)
IE04: Unplanned Scrams with Complications (USwC) Sample (IP Section 02.03) (2 Samples)
- (1) Unit 1 (January 1, 2022, through December 31, 2022)
- (2) Unit 2 (January 1, 2022, through December 31, 2022)
71152A - Annual Follow-up Problem Identification and Resolution Annual Follow-up of Selected Issues (Section 03.03)
- (1) AR 4534873, "Unit 2 Scram on Reactor High Water Level," on March 6, 2023
71153 - Follow-Up of Events and Notices of Enforcement Discretion Event Report (IP section 03.02)
The inspectors evaluated the following licensee event reports (LERs):
- (1) LER 265/2022-003-00, "Manual Scream Due to Feedwater Regulator Valve Failure Increasing Reactor Water Level" (ADAMS Accession No. ML22364A273). The inspection conclusions associated with this LER are documented in this report under the Inspection Results Section 71152A. This LER is closed.
INSPECTION RESULTS
Inadequate Procedure for Feedwater System Failure Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events Green NCV 05000254,05000265/2023001-01 Open/Closed
[H.14] -
Conservative Bias 71152A The inspectors identified a finding of very low safety significance (Green) and an associated non-cited violation (NCV) of Technical Specification 5.4.1, "Procedures," for the licensee's failure to have station procedures appropriate for the circumstances to address a failure of the 2B feedwater regulating valve (FRV) in the full open position. As a result, on November 4, 2022, following a failure of the 2B FRV in the full open position, operators were required to insert a manual trip of the reactor on high water level when they could not arrest the subsequent reactor water level transient. The overfeed condition was due to a latent design vulnerability in the 2B FRV allowing it to be capable of passing greater than 100 percent total feedwater flow at full reactor power.
Description:
On November 4, 2022, with Unit 2 operating at full power, the 2B FRV, HO 2-0624B, failed in the full open position without a change in demand signal from the digital feedwater level control (DFWLC) system and without any component manipulation from control room operators. This failure caused an unplanned rise in reactor water level. The DFWLC system compensated for the rising reactor water level by automatically shutting the 2A FRV, HO 2-0624A. Control room operators noted that the 2A FRV shut as expected but did not see a corresponding decrease in reactor water level. The control room operators began to shut MO 2-3206-B, the 2B FRV isolation valve, but prior to MO 2-3206-B fully shutting, manual scram criterion of +44 inches reactor water level was reached and operators manually tripped the unit.
DFWLC System Design The DFWLC system is designed to maintain water level in the reactor vessel within a specified range during plant operation by processing inputs from various plant parameters such as reactor water level, steam flow, and feedwater flow. Based on those parameters, the DFWLC system generates signals to adjust the position of two FRVs to maintain reactor water level at the desired level. Both valves are hydraulically operated and set up, or tuned, to operate within the optimal flow characteristics of the valves. Thus, when the demanded flow of the DFWLC system is 100 percent, typically the FRVs are not physically 100 percent full open. An important feature of the DFWLC system is that a lockup signal is generated by the system whenever there is at least a 10 percent difference between the actual position of an FRV and the demanded position lasting for at least 5 seconds. The time delay is meant to prevent spurious lockups from occurring at steady state conditions but short enough to prevent actual system failures from causing major reactor water level transients. When a lockup occurs, an electric signal halts all FRV motion in the position that existed at the time of the lockup. Operators can procedurally reset the lockup and restore a FRV to operation once the cause of a lockup has been investigated and corrected.
In the mid-1990s, the licensee replaced the control units for the FRVs on both units with new hydraulic actuators. During this modification, the existing 2B FRV was installed which had a maximum stroke length of 14 inches. To achieve the desired flow characteristics, the valve was tuned to the feedwater level control system to a maximum stroke length of 12 inches.
The inspectors noted that these physical characteristics of the valve were not recorded in the design modification package. Additionally, the inspectors were not able to determine if the site performed an evaluation at the time of the modification to assess the risk associated with the valve having the ability to pass significantly more flow than required should certain feedwater level control system failures occur. The inspectors noted that the licensee specifically highlighted the unique physical characteristics of the valve in QDC-3200-M-1491, Hydraulic Analysis of Condensate and Feedwater Systems for EPU Marin Recovery, revision 0, which was performed in 2006, indicating that the licensee was aware of the difference between the 2B FRV and the other FRVs at the site.
External and Internal Operating Experience On March 8, 2019, the licensee issued Nuclear Event Report (NER) NC-19-002-Y, Feedwater Regulating Valve Fleetwide Issues, revision 0, to the fleet. The NER was written to address the fleet actions necessary to improve performance of FRVs to minimize operational risk and to increase the operational reliability of the regulating valve components of the feedwater system. On October 31, 2019, NER NC-19-002-Y, revision 1, was issued which included further actions and additional industry operating experience with FRV failures.
The station completed several actions in response to NC-19-002-Y, including a review of vulnerabilities or enhancements that could be made to harden the system against subcomponent failures and a review of known single point vulnerabilities for the FRVs. The inspectors noted that the hardening review covered various generic mechanical and electrical features of the FRVs but did not specifically perform a review of the design characteristics of the valves. In the sites review of known FRV single point vulnerabilities, the site stated:
Based on the ERVR 2.0 and ERVR 3.0 reviews of the Feedwater (FW) and Feedwater Level Control (FLC) systems, there were no previously identified single point vulnerabilities (SPVs) for the Feedwater Regulating Valves (FRVs). The criterion used during the ERVR reviews for SPVs included equipment that would cause an automatic or manual reactor scram should a failure occur. The FRVs are designed to fail in place, or lockup, should a fault or failure occur.
The site went on to state that one of the original design criterions of the DFWLC system was to be designed with no SPVs. At the time of the response, the site noted that there had been issues with lockups of the FRVs during operation that required maintenance to solve, but those events had not resulted in a plant trip or unplanned shutdown. As such, the sites vulnerability evaluation of the FRVs during follow-on corporate initiatives concluded that the DFWLC system was safe from a single point vulnerability resulting from an electrical or instrumentation and control type failure. Additionally, the site concluded that a mechanical issue with the FRV would likely only result in a downpower of greater than 20 percent to isolate that feedwater line to repair. The inspectors noted that the site did not consider the presence of foreign material adversely affecting FRV operation as a failure mechanism.
Relevant fleet operating experience concerning foreign material obstruction of FRV control was included in NER NC-19-002-Y, revision 1, highlighting a manual reactor scram inserted by operators at the Braidwood station on September 23, 2019 (AR 4281429) due to foreign material found inside the control air portion of the feedwater digital valve controller. This foreign material originated from a manufacturing byproduct internal to the component.
Similarly, on March 21, 2021, operators at the Calvert Cliffs station inserted a manual scram following an FRV failure also due to foreign material originating from the vendor assembly process (AR 4410594).
On May 24, 2021, Quad Cities Unit 1 experienced an issue with the 1B FRV moving from its expected position to full open with no corresponding DFWLC system lockup. Operators attempted to take manual control of the affected valve from the control room but were unsuccessful. Eventually, the valve did lock in place in the full open position. Equipment operators were sent into the field to locally operate the valve with a hydraulic hand pump to return it to the desired position before repairs were ultimately made. Although foreign material was not determined to be the cause of the event, the inspectors noted that this event did illustrate that the DFWLC system lockup feature may not be able to mitigate all failure modes of the FRV in which the valve moves to a full open or shut position before the designed lockup happens.
On July 4, 2022, Quad Cities Unit 2 experienced an unexpected closure of the 2A FRV in which the valve went from its expected position to full shut prior to the DFWLC system lockup occurring. The resulting water level transient ultimately required control room operators to insert a manual scram on low reactor water level. The site performed a root cause investigation under AR 4509196 and determined the cause of the failure was foreign material partially or completely blocking hydraulic flow in the control servo of the FRV. The station was not able to determine the exact origin of the foreign material but believed it to be from either the seal rings used in the assembly of the servo or introduced through the addition of hydraulic oil to the system during maintenance activities. The site performed a risk assessment of the event in accordance with AD-AA-3000, Nuclear Risk Management Process, and created actions to address the specific vulnerabilities identified in the event.
The inspectors noted that no consideration was made to reevaluate the possible failure modes of the FRVs and potential consequences of those failures despite internal and external operating experience, indicating the sites original assumptions regarding the robustness of the DFWLC system may no longer be valid.
On July 12, 2022, Dresden station Unit 2 experienced a failure of the 2A FRV in which the valve went from its expected position to full open. The subsequent transient resulted in an automatic reactor scram. The failure was determined to be due to foreign material introduced during the manufacturing process which interfered with the operation of the valve (AR 4510475).
On August 12, 2022, the licensee corporate office performed a work group evaluation under AR 4511746 in response to the Quad Cities and Dresden FRV events. The evaluation concluded that although the equipment vulnerability reviews performed by Quad Cities and Dresden were consistent with the level of detail provided in operations training and the design basis provided by the vendor, the reviews did not get to the needed depth to understand the full vulnerability of the FRVs. The evaluation determined that operating experience from recent failures and scrams in the industry indicated there was an adverse trend in the failure of pneumatic controllers caused by manufacturing-produced foreign material, but the operating experience was not applied to hydraulic or other feedwater control system designs within the fleet and should have been used as a trigger to perform a deeper review of the fleet FRVs. The evaluation noted that specific discussions during site and corporate challenge boards on the effect of component failures in systems with lockup features, like the one used at Quad Cities, resulted in site responses simply stating that the system will lock up on deviation, loss of air, or hydraulics. The evaluation noted that there were no challenges or discussions with respect to abnormal directional control valve or positioner failures other than the digital system will recognize the deviation and lock the valve in position. Additionally, the evaluation stated that during the challenges it was not recognized that a failure of a single positioner results in an FRV repositioning faster than actuation of the lockup feature at stations such as Quad Cities.
On September 8, 2022, Quad Cities Unit 1 experienced a failure of the 1B FRV in the full open position before it locked up. Operators responded per station procedures and recovered reactor water level. The site performed a work group evaluation on the event under AR 4521360 and determined the cause of the failure was a failed component on the input/output module for the control system. The evaluation determined that with this specific type of failure, the system does not respond to the control system input command and the valve fails full open. The inspectors noted that, although the event was not caused by foreign material, this event illustrated again that the DFWLC lockup feature may not be able to mitigate all failure modes of the FRV in which the valve moves to a full open or shut position before the lockup happens.
2B FRV Failure On November 4, 2022, Quad Cities Unit 2 experienced a failure of the 2B FRV in the full open position and before it locked up. Operators responded in accordance with station procedures and operator training, but unlike the event occurring on Unit 1 on September 8, 2022, operators were not able to compensate for the additional feedwater flow being supplied to the reactor by the 2B FRV even with the 2A FRV fully shut. Control room operators inserted a manual scram as required by station procedures.
The site performed a root cause investigation under AR 4534873 and determined that the cause of the event was due to foreign material that was introduced during the manufacturing or assembly of the 2B FRV control servo. This foreign material interfered with the correct operation of the servo and resulted in a hard over condition which moved the 2B FRV to the fully open position. The site determined that a contributing cause of the event was that the 2B FRV had an excessive valve stroke that was longer than all the other FRVs installed at the site. During initial installation, this was not viewed as an inherent risk because the valve was tuned to only operate over a small portion of its possible valve stroke. The failure observed on November 4, 2022, caused the 2B FRV to move much further in the open direction than was previously believed possible. In the fully open position, the 2B FRV was independently capable of providing more flow than needed for 100 percent power operations. In this condition, closure of the 2A FRV could not have arrested the rising reactor water level trend.
The evaluation additionally stated that the stations operating and abnormal operating procedures lacked any discussion of the 2B FRV's unique construction because the inherent risk of this construction was never recognized during the modification process. Station procedures also lacked any guidance to specifically address a reactor water level transient resulting from a 2B FRV failure in the full open position. Similarly, the work instructions for testing the 2B FRV did not identify that the valve was physically capable of passing more than 100 percent rated flow, as the calibration procedure only calibrated the valve to operate in a normal operating band. The evaluation stated that, although there was no data to suggest that operator action could definitively have prevented the subsequent reactor scram, there were missed opportunities in the plant modification and preventive maintenance processes to evaluate current plant design and incorporate relevant mitigation strategies into operating procedures or follow-on plant modifications. Additionally, the evaluation reviewed the root cause investigation under AR 4509196 and determined that the review performed by the site at the time did not explore the ramifications of a similar failure mode in the valves open direction, resulting in another missed opportunity to mitigate this event.
Ultimately, the inspectors determined that the installation of the 2B FRV with a 14-inch stroke length introduced a latent design vulnerability into the system that went unidentified by the site until the event on November 4, 2022. Previous opportunities to identify the vulnerability through the licensees single point vulnerability assessments, as outlined in licensee procedure ER-AA-2004, "System and Component Vulnerability Identification and Mitigation,"
were unsuccessful because the licensee non-conservatively limited the scope of the vulnerability assessment to a narrow set of factors thought not to be applicable to the FRVs at the station due to the perceived robustness of the DFWLC system. When both internal and external operating experience revealed that the previous assumptions about the mitigating features of the DFWLC system were no longer valid, the site did not take meaningful action to reassess the risk of the system in accordance with AD-AA-3000 or revalidate the systems vulnerabilities in accordance with ER-AA-2004. As a result, the licensee did not have adequate station procedures or operations training to cope with the unique consequences of a failure of the 2B FRV as seen on November 4, 2022.
Corrective Actions: The licensee performed a root cause investigation of the event under AR 4534873. On November 6, 2022, the licensee established operation's department standing order 22-07 to provide specific guidance to operators regarding a repeat failure of the 2B FRV until permanent actions could be taken to eliminate the vulnerability. On January 21, 2023, the licensee installed a mechanical stop in the 2B FRV to physically limit its available stroke length and eliminate the design vulnerability noted above.
Corrective Action References: AR 4534873, "U2 Manual Scram on High Reactor Water Level"
Performance Assessment:
Performance Deficiency: The inspectors determined that the failure to have a procedure appropriate for the circumstance to address the unique plant vulnerability posed by a failure of the 2B feedwater regulation valve in the full open position was a performance deficiency.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the Procedure Quality attribute of the Initiating Events cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations.
Significance: The inspectors assessed the significance of the finding using IMC 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors screened the issue using IMC 0609, Appendix A, Exhibit 1, Initiating Events Screening Questions, and determined that this finding was of very low safety significance (Green).
Cross-Cutting Aspect: H.14 - Conservative Bias: Individuals use decision making-practices that emphasize prudent choices over those that are simply allowable. A proposed action is determined to be safe in order to proceed, rather than unsafe in order to stop. Specifically, the licensee failed to identify the latent design vulnerability of the 2B FRV through the licensees system and component vulnerability identification and mitigation program, as outlined in licensee procedure ER-AA-2004, because the scope of the vulnerability assessment was limited to a narrow set of factors thought not to be applicable to the FRVs at the station.
Enforcement:
Violation: Technical Specification 5.4.1, "Procedures," states that written procedures shall be established, implemented, and maintained covering the applicable procedures recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978. Regulatory Guide 1.33, "Quality Assurance Program Requirements," Appendix A, Section 6, lists procedures for combating emergencies and other significant events. Licensee procedure QCOA 0201-08, "Reactor High Water Level," and QCAN 901(2)-5 H-8, "1(2)B Feedwater Actuator Trouble,"
have been established by the licensee to combat a feedwater system failure under the purview of Regulatory Guide 1.33.
Contrary to the above, from approximately November 22, 1995, to November 6, 2022, neither licensee procedure QCOA 0201-08 or QCAN 901(2)-5 H-8 addressed the unique plant vulnerability posed by a failure of the 2B feedwater regulation valve in the full open position.
As a result, on November 4, 2022, following a failure of the 2B feedwater regulation valve to the full open position, operators were required to insert a manual trip of the reactor after reactor water level unexpectedly continued to rise following the automatic closure of the 2A feedwater regulation valve. The unexpected plant response was due to a latent design vulnerability which allowed the 2B feedwater regulation valve to be capable of passing greater than the required feedwater flow at full reactor power.
Enforcement Action: This violation is being treated as an NCV, consistent with Section 2.3.2 of the Enforcement Policy.
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
On April 25, 2023, the inspectors presented the integrated inspection results to Brian Wake, Site Vice President, and other members of the licensee staff.
DOCUMENTS REVIEWED
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Corrective Action
Documents
Tornado Warning Issued for QUAD CITIES STATION Area
03/31/2023
Procedures
QCOA 0010-10
Tornado Watch/Warning, Severe Thunderstorm Warning, or
Severe High Winds
M-39, Sheet 1
Diagram of Residual Heat Removal Piping
2/15/1997
Drawings
M-39, Sheet 3
Diagram of Residual Heat Removal Piping
07/29/1999
QCOP 1300-01
RCIC System Preparation for Standby Operation
Procedures
QOP 0020-03
Core Spray and RHR Room Draining
FZ 11.1.3
Unit 1 RB 554'-0" Elev. HPCI & HPCI Access Tunnel
08/2022
Fire Plans
FZ 8.2.7.A
Unit 1 TB 615' Elev. Hydrogen Seal Oil Area and MCC's
09/2022
Procedures
Fire Prevention for Hot Work
1-5799-2193 Check Valve Stuck Shut
2/25/2015
Small Water Leak Found in 901-33 Panel in Aux Elec. Room
2/25/2015
AR 306915306915Water on Aux. Elect/Cable Spreading RM and Turb Bldg
Floor
03/01/2005
1-5799-2203 Check Valve Sticking, Boiler Cond on Turb
Deck
01/08/2023
Water Found in Aux Electric
01/12/2023
Corrective Action
Documents
Aux Electric Room Historic Leak Troubleshooting
03/06/2023
Corrective Action
Documents
U2 EDG Failed to Start
09/22/2022
DG6600-01
Maintenance Rule Systems Basis Document: Diesel
Generator System
2/17/2014
INPO Report
- 540170
Emergency Diesel Generator Failed to Start
09/22/2022
Monitoring the Effectiveness of Nuclear Power Plant
Maintenance
07/2019
Miscellaneous
Industry Guideline for Monitoring the Effectiveness of
Maintenance at Nuclear Power Plants
4A
NDE Reports
QDC-13774
Failure Analysis of Time Delay Relay
11/10/2022
MSPI Failure Determination
Procedures
Maintenance Rule 18-10 - Performance Monitoring and
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Dispositioning Between (a)(1) and (a)(2)
QCOS 6600-45
Unit 2 Diesel Generator Timed Start Test
Work Orders
EM EWP Perform 1/2 DG Time Delay Relay TD1,3,4
Calibrations
09/15/2021
Shutdown Safety Management Program
OU-QC-104
Shutdown Safety Management Program Quad Cities Annex
Procedures
QCOP 0600-02
Placing Main Feedwater Regulator Online or Offline
EMD U2 125 VDC Battery Charger #2 4HR Load Test
01/10/2023
Work Orders
Unit 2 Scram on High Rx Water Lvl
11/07/2022
TIC 2-1141-14 Did Not Meet Acceptance Criteria
HPCI Pump Piping Connection Leaks
06/29/2022
U2 HPCI Pump Piping Connection Leaks During Surveillance
2/14/2022
Recommend Unit 2 SBLC Heat Trace Replacement
Unexpected 901-8 A-4 Alarm
2/13/2023
HPCI Signal Converter MGU Drive Signal (Auto)
2/25/2023
Corrective Action
Documents
Unit 1 HPCI Signal Converter Troubleshooting
03/01/2023
Schematic and Wiring Diagram Window Display for
Annunciator DG 1 and 2 Panel 2212-45
09/03/1998
Drawings
4E-2345, Sheet 1
Schematic Diagram 4160V Bus 23-1 Standby Diesel Half
Feed Breakers
03/17/1997
HPCI Turning Gear Performance on HPCI System
Operability
ECR Pipe Freeze Seal Engineering Evaluation Request
Evaluation of Leakage at Mechanical Connections for Class
and 2 System Leakage Test at Quad Cities Unit 1 and Unit
001
HPCI Signal Converter Replacement
Engineering Guidance on Performing Freeze Seal On-line
01/03/2023
Engineering
Changes
Evaluate Min Wall Thickness for CREV Freeze Seal
ASME Boiler Pressure Vessel Code
2007
Miscellaneous
QC-S-2023-0008
50.59 Screening: QCOP and QCOS 2300 Series Procedure
Changes to Leave the HPCI Turning Gear in PULL TO STOP
03/02/2023
Procedures
General Piping Analysis
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
QCAN 2212-45
E-3
Overcurrent Trip of Diesel Generator to Bus 23-1 GCB
QCAN 901 (2)-5
G-6
Standby Liquid Control Tank High/Low Temperature
TIC 3697
Temporary Instruction Change to QCOS 2300-05: HPCI
Pump Operability Test
91a
(IST) (NEIL) HPCI Pump Operability
2/08/2022
Work Orders
IM T/S FY 1-2386-B Servo Amp EC 399498
2/11/2022
Decay Heat Removal Capability of RWCU System
Engineering
Changes
Alternate Decay Heat Removal (ADHR) System Qualification
for TS 3.4.8 Action A.1
Miscellaneous
QDC-0200-M-
2339
Calculation of Reactor Drain Time in Support of DRAIN TIME
Tool
Procedures
QCAP 0260-03
Screening for Reactor Pressure Vessel Water Inventory
Control
Both Trains of SBGT Not Producing Required Flow
07/04/2022
1-1402-8B Failed to Reseat After 1400-01
2/13/2023
Time Delay Relay 10-K45B As Found Out of Spec
03/08/2023
Loose Wire Identified During Logic Test
03/09/2023
Corrective Action
Documents
PSU Q1R27 1-2301-4 LLRT Exceeded Admin Warning 15
scfh
03/27/2023
Drawings
Wiring Diagram Panel 902-47
06/10/2008
HPCI Signal Converter Replacement
Engineering
Changes
SBGT Surveillance Flowrate Note Below Expected Values
QCIS 1400-01
Core Spray Pump Discharge Pressure Calibration and
Functional Test
QCIS 1400-05
Core Spray Pump Discharge Flow Trip Unit Calibration and
Functional Test
QCOP 6600-12
Diesel Generator Air Start System Pressure Verification
QCOS 1000-28
RHR Service Water Pump Comprehensive/Performance Test
QCOS 1000-44
Unit 2 'B' Loop LPCI and Containment Cooling Modes of
HRHR Non-Outage Logic Test
Procedures
QCOS 1300-05
RCIC Pump Operability Test
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
QCOS 1400-01
Core Spray System Flow Rate Test
QCOS 1400-01
Core Spray System Flow Rate Test
QCOS 1400-10
Core Spray Operability Verification
QCOS 2300-15
HPCI Drain Pot/Steam Line Drain Level Switch Valve, and
Alarm Functional Verification
QCOS 6500-09
Functional Test of Unit 1 Second Level Undervoltage
QCOS 6600-42
Unit 2 Emergency Diesel Generator Load Test
QCOS 6600-45
Unit 2 Diesel Generator Timed Start Test
QCOS 7500-05
SBGTS Operability Test
QCTS 2000-01
Drywell Leakage Troubleshooting
QOM 1-6500-T06
TIC 3697 QCOS
2300-05
HPCI Pump Operability Test
Self-Assessments RC 4509198
Root Cause: Both Trains of SBGT Not Producing Required
Flow
QCOS 6600-41
U1 Emergency Diesel Generator Load Test
'B' Loop LPCI RHR Cooling Modes Non-Outage Logic Test
03/08/2023
'B' RHR Service Water Pump Comprehensive Test
01/17/2023
Unit 2 EDG Starting Air Header PCV Reads High Out of
Band
03/06/2023
Work Orders
Core Spray Pump a Flow Rate (IST)
03/22/2023
FRV Not Identified as SPV During ERVR4
07/19/2022
Corrective Action
Documents
Unit 2 Scram on Rx High Water Level
11/04/2022
NER NC-19-002-
Y
Feedwater Regulating Valve Fleetwide Issues
NER NC-19-002-
Y
Feedwater Regulating Valve Fleetwide Issues
QDC-0-2022-
0153
FRV Servo Valve
QDC-0-2022-
0156
FRV Hydraulic Skid Particulate
Miscellaneous
QDC-05-050
Station Design Inputs for EPU Margin Recovery Hydraulic
Analysis
11/10/2005
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Standing Order
2-07
2B FRV Failure Results in U2 SCRAM
11/06/2022
Procedures
System & Component Vulnerability Identification and
Mitigation
2