IR 05000348/2022050
| ML22272A557 | |
| Person / Time | |
|---|---|
| Site: | Farley  |
| Issue date: | 09/30/2022 |
| From: | Alan Blamey NRC/RGN-II/DRP/RPB2 |
| To: | Gayheart C Southern Nuclear Operating Co |
| References | |
| EA‑22‑095 IR 2022050 | |
| Download: ML22272A557 (28) | |
Text
September 30, 2022
SUBJECT:
JOSEPH M. FARLEY NUCLEAR PLANT-SPECIAL INSPECTION REACTIVE REPORT 05000348/2022050 AND APPARENT VIOLATION
Dear Ms. Gayheart:
On August 9, 2022, the U.S. Nuclear Regulatory Commission (NRC) completed its initial assessment of the loss of offsite power to the 4KV B train emergency power bus that resulted in a main generator/turbine trip and subsequent automatic reactor shutdown, which occurred on August 3, 2022, at Joseph M. Farley Nuclear Plant. Based on this initial assessment, the NRC sent an inspection team to your site on August 10, 2022. The basis for initiating this special inspection is further discussed in the Charter, which is included as an attachment to this report.
On August 17, 2022, the NRC completed its special inspection. On August 17, 2022, and September 19, 2022, the NRC discussed the results of this inspection with Site Vice President Delson Erb and other members of your staff. The results of this inspection are documented in the enclosed report.
Section 93812 of the enclosed report discusses a finding with an associated apparent violation for which the NRC has not yet reached a preliminary significance determination. This finding involved the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
The NRCs significance determination process (SDP) is designed to encourage an open dialogue between your staff and the NRC; however, neither the dialogue nor the written information you provide should affect the timeliness of our final determination. We ask that you promptly provide any relevant information that you would like us to consider in making our determination. We are currently evaluating the significance of this finding and will notify you in a separate correspondence once we have completed our preliminary significance review. You will be given an additional opportunity to provide additional information prior to our final significance determination unless our review concludes that the finding has very low safety significance (i.e.,
Green).
This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding.
Sincerely, Signed by Blamey, Alan on 09/30/22 Alan J. Blamey, Chief Reactor Projects Branch 2 Division of Reactor Projects Docket No. 05000348 License No. NPF2
Enclosure:
As stated SIT Charter (ML22221A092)
Inspection Report
Docket Number: 05000348 License Number: NPF-2 Report Number: 05000348/2022050 Enterprise Identifier: I2022050-0003 Licensee: Southern Nuclear Operating Co. Inc.
Facility: JOSEPH M. FARLEY NUCLEAR PLANT Location: Columbia, AL Inspection Dates: August 10, 2022 to August 17, 2022 Inspectors: M. Meeks, Senior Operations Engineer P. Meier, Senior Resident Inspector M. Read, Sr. Resident Inspector Approved By: Alan J. Blamey, Chief Reactor Projects Branch 2 Division of Reactor Projects Enclosure
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting a special inspection at JOSEPH M. FARLEY NUCLEAR PLANT, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Inadequate Design Control of the Auxiliary Feedwater System Resulting in the Inoperability of the Turbine-driven Auxiliary Feedwater Pump Cornerstone Significance/Severity Cross-Cutting Report Aspect Section Mitigating Pending None (NPP) 93812 Systems Apparent Violation AV 05000348/202205001 Open EA22095 A self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
Additional Tracking Items
Type Issue Number Title Report Section Status URI 05000348/202205002 Unit 1 Reactor Trip and 93812 Open Partial Loss of Offsite Power
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
OTHER ACTIVITIES
-TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL 93812 - Special Inspection In accordance with the attached Special Inspection Team Charter, the inspection team conducted a detailed review of the circumstances surrounding the August 3, 2022, Unit 1 event in the high voltage switchyard (HVSY) that resulted in a partial loss of offsite power, automatic reactor trip, automatic start of the B emergency diesel generator (EDG), loss of all reactor coolant pumps (RCPs), and loss of the condenser. Additionally, the team conducted a detailed review of operator decision-making and equipment challenges with the TDAFWP during post-trip recovery operations.
Description of the Event and Reactive Inspection Basis On August 3, 2022, Unit 1 was operating at 100 percent rated thermal power (RTP) and planned to remain at this power through the day. During day shift, work continued in the high voltage switchyard (HVSY) relay house to replace cables in preparation for an upcoming modification. Among other functions, the HVSY relay house enclosed energized electrical cabinets which contained sensitive relaying and metering equipment for control and protection of the HVSY. The planned work involved exposing cable trays under the floor of the relay house by manually removing floor panels or tiles located approximately two feet away from the energized electrical cabinets, moving the floor panels away from the electrical cabinets, and stacking the tiles approximately four feet away from the electrical cabinets.
The event initiated at approximately 12:58 p.m., when a transmission/distribution service organization (TDSO) technician inadvertently agitated a relay in the HVSY relay house. During movement of a floor tile, the tile was inadvertently dropped while being moved toward an electrical cabinet (instead of away from the cabinet), which in turn caused a protection relay to actuate. The initial relay actuation ultimately resulted in the automatic opening of eight circuit breakers in the HVSY and electrical isolation of the HVSY 230 kilovolt (KV) Bus 1. The isolation of 'Bus 1 resulted in an automatic main generator and turbine trip on Unit 1, subsequent automatic reactor shutdown, and a partial loss of offsite power (B train offsite power to Unit 1). Unit 2 was unaffected by this electrical transient.
The loss of HVSY Bus 1 resulted in a loss of offsite power to the 1B' startup transformer (SUT), which resulted in a loss of electrical power to 4KV buses 1B and 1C, which caused the loss of two RCPs and one circulating water pump. Due to a failure of the automatic fast bus transfer of the 1A 4KV bus from the 1B unit auxiliary transformer to the 1A SUT, Unit 1 also lost electrical power to the 1A 4KV bus, which caused a simultaneous loss of electrical power to the last RCP and last circulating water pump. The failure of the 1A 4KV bus automatic transfer feature was independent of the initiating event in the HVSY. The B EDG automatically started and restored electrical power to the 1 G 4KV (B train emergency) bus. Due to the loss of forced circulation flow in the reactor coolant system (RCS) and the loss of the condenser as an effective heat sink, the operations team stabilized the plant using natural circulation, and maintained a secondary heat sink for decay heat removal using the Auxiliary Feedwater System (AFW) and the steam generator atmospheric relief valves.
Approximately 20 minutes into the event, operators determined that the TDAFWP was not required and attempted to shut it down by closing the steam admission valves in accordance with the operating procedure. However, the conditions for an automatic TDAFWP restart remained present since there was an undervoltage signal on two out of three RCP buses. Due to that signal, the valves automatically reopened, and the pump accelerated until it tripped due to overspeed. To recover the pump, a building operator locally reset the overspeed trip mechanism and opened the throttle valve; however, the TDAFWP tripped again due to overspeed.
Management Directive 8.3, NRC Incident Investigation Program, was used to evaluate the level of NRC response for this event. Based on the deterministic criteria and risk insights related to this event, the NRC determined that the appropriate level of response was to conduct a special inspection. The NRC determined that the inspection did not need to be upgraded to an augmented inspection team response. This special inspection team was chartered to identify the circumstances surrounding this event and review the licensees actions to address the causes of the event.
Sequence of Events The inspectors interviewed licensee personnel and reviewed records to develop the sequence of events leading up to and immediately following the event described above.
All event sequence times listed below are provided in CST.
August 3, 2022 12:57 Unit 1 operating at 100 percent RTP with planned HVSY relay house work in progress.
12:58 TDSO employee inadvertently dropped a floor tile next to an electrical cabinet, which caused a chain of events including relay actuations, breakers opening, a reactor trip, loss of power to several large pumps, and an EDG start. Refer to Section Work Activities in the High Voltage Switchyard for additional details on the chain of events and consequences of the dropped floor tile.
13:15 Operators manually secured steam flow to the TDAFWP in accordance with emergency operating procedures since steam generator water levels were stabilized.
13:16 Steam admission valves automatically reopened and the TDAFWP tripped on overspeed (first overspeed trip).
13:29 Operator locally reset TDAFWP trip and throttle valve, reopened the valve, and TDAFWP tripped on overspeed (second overspeed trip).
15:15 Operators closed main steam isolation valves due to small steam leak inside turbine building.
20:41 Operators restored offsite power through the 1B startup transformer.
August 4, 2022 04:41 Operators restarted the B reactor coolant pump to reestablish forced circulation to the RCS.
15:08 TDAFWP successfully retested and returned to operable.
Work Activities in the High Voltage Switchyard
1. Review and evaluate the licensees process for control of work in the switchyard. This should
include the work control process, management of plant risk, and oversight of switchyard work.
The inspectors conducted a detailed review of the licensees procedures for control of work in the high voltage switchyard and relay house. The inspectors gathered information from work orders, risk assessments, interviews with work planners and switchyard oversight, condition reports, and personnel statements following the event.
The inspectors reviewed the licensees interface agreement with the TDSO, who owns and maintains the high voltage switchyard and relay house. The agreement stated that the licensee is responsible for monitoring bus voltages, interfacing with the control center for voltage control, operations of power system stabilizers and Unit power levels, evaluating switchyard maintenance activities against plant conditions and outages with appropriate coordination and scheduling, and following physical access control policies. The inspectors determined that the licensee provided work orders and risk assessments to control work in the switchyard, which were supplemented by TDSO documentation such as drawings, specifications, and procedures.
Licensee procedure NMP-DP001, Operational Risk Assessment, revision 23.0 contained requirements to screen work activities as Low, Medium, or High risk and identified additional requirements based on the risk level. Work activities identified as Medium and High risk required documented Risk Management Plans. The procedure also contained lists of pre-screened work activities and identified activities that could cause vibration near relays or other vibration sensitive equipment that could cause a loss of generation as a High Risk activity.
Additionally, licensee procedure NMP-GM-021, Switchyard Access and Maintenance Controls, revision 7.0 provided additional requirements for switchyard activities. Only High Risk activities required licensee direct oversight per NMP-GM021. Per procedure, Oversight personnel must manage overall risk to the plant by challenging assumptions made during the operations risk assessments by performing additional [Operational Risk Assessments] based on changes to scope, schedule, personnel, or work environment.
The inspectors determined that the licensees risk assessment and management procedures provided guidance for assessing the work activities that were scheduled in the high voltage switchyard relay house. As documented in the Results Section, an Unresolved Item (URI) was opened which covers unresolved inspection items related to the control of work in the HVSY and equipment deficiencies.
2. Verify that the bumped relay caused the breakers in the switchyard to open as they did (verify
breaker coordination) prior to the reactor trip.
The inspectors reviewed plant drawings and diagrams to determine if the high voltage switchyard circuit breakers operated as designed, given the actuation of the agitated relay, 924 BF LOR, in the high voltage switchyard relay house due to the dropped floor tile. The 924 BF LOR relay continuously monitors if breaker number 924 in the high voltage switchyard should have tripped but did not. If that does not occur, then the relay sends a signal to open eight breakers to isolate 230KV Bus 1. This actuation should isolate the 1B startup transformer and result in a loss of offsite power to the safety-related 4KV B train emergency bus. This should cause the B EDG to automatically start and provide power to the emergency bus.
The inspectors verified that the breaker coordination from the 924 BF LOR relay and expected actuations were in accordance with the design of the high voltage switchyard.
The inspectors also reviewed the main generator protection scheme designed to monitor for a fault condition in the switchyard. The KD10 relay provided main generator protection by monitoring impedance to detect line faults. The impedance required inputs of current and voltage. The relay monitored current through the main generator switchyard output breakers and voltage on the 230KV Bus 1. Upon the loss of 230KV Bus 1 voltage, impedance dropped and current increased through the generator output breakers. This drop in impedance resulted in the KD10 actuating, which generated signals to trip the main turbine and main generator.
This actuation was expected upon the loss of the 230KV Bus 1.
In the protection circuit, a KC2 relay was designed to supervise the KD10 trip circuit to prevent undesired tripping of the main generator in the event of a loss of voltage on the selected monitored bus without an electrical fault present. The KC2 relay was designed to actuate upon an electrical fault, which would have manifested as higher than the normal current passing through the main generator output breakers. In 2013, following the replacement of the Unit 1 main power transformers, relay setting calculation SE012445-002 was updated and revised the desired setpoint for the KC2 relay, but the relay was not recalibrated. The setting on KC2 on August 3, 2022, was incorrectly calibrated to the normal current passing through the main generator switchyard output breakers. With this setting, the relay was not performing its supervisory function for the KD10 trip circuit because the KC2 relay was always actuated when the Unit was at or near 100 percent rated power.
The inspectors determined that this created a vulnerability that any loss of 230KV Bus 1 voltage while operating near 100 percent RTP would result in a turbine and generator trip. This created additional vulnerabilities in the HVSY relay house in which actuation of single components would result in a reactor trip. As documented in the Results Section, a URI was opened which covers unresolved inspection items related to the control of work in the HVSY and equipment deficiencies.
3. Review the work control documents that were being used to perform switchyard
maintenance, including release of work and work scope.
The inspectors reviewed work order SNC1188321, which was used to identify and control work in the HVSY relay house. The work order included the work scope, a risk assessment, and release of work. Inspectors also performed interviews of licensee staff who provided oversight of some of the work activities and licensee work planners who provided protocols for risk screening and control of work. The inspectors reviewed post-event statements made by TDSO personnel as well as licensee personnel who were present in the switchyard relay house on August 3, 2022.
Work was released by a licensed senior reactor operator based on a review of the work package and discussions with licensee switchyard oversight. The inspectors determined that assumptions used to identify the work as Low Risk included a 2-foot exclusion area near the electrical cabinets. The inspectors determined that the level of detail in the work scope did not specifically include information related to exclusion areas, there were no procedural restrictions from TDSO personnel working inside a 2-foot exclusion area, and the 2-foot rule was an accepted practice for work activities inside the plant protected area. Furthermore, the inspectors assessed that the 2-foot rule was not consistently implemented during work activities in the switchyard relay house, which was located outside the nuclear protected area. However, the inspectors determined through interviews that the licensee provided verbal communications and precautions from oversight personnel to mitigate risk of TDSO work activities and occasionally provided direct oversight of the TDSO workers.
Inspectors determined that the licensee screened the work activities in the switchyard relay house as Low Risk despite work activities in proximity to sensitive relays. As documented in the Results Section, a URI was opened which covers unresolved inspection items related to the control of work in the HVSY and equipment deficiencies.
4. Review the work that was being performed in the switchyard, the location of the work and the
equipment in the work area to confirm the work was being performed in accordance with procedures and work instructions.
The inspectors walked down the location of the work in the HVSY relay house, reviewed work order SNC1188231, and discussed the scope of the work with licensee oversight personnel who were involved with the cable replacement project. Work instructions did not detail the removal of floor tiles or restrict work inside exclusion areas around the electrical cabinets. Although not procedurally required for Low Risk work, the licensee provided some periodic direct oversight of the TDSO workers and provided verbal communications to ensure the TDSO workers were aware of the sensitivity of the electrical equipment. There was also a barricade in the relay house before accessing the area near the sensitive panels that was used to prompt a discussion of work activities near sensitive equipment. The event started when a TDSO worker, working near sensitive electrical equipment, moved a floor tile close to the relay panel, and the tile was unexpectedly dropped. Although the movement of the floor tile was not controlled by the work instructions, inspectors determined that the expectation by licensee staff, including the person who provided some direct oversight of the work, was that tiles would be moved away from the electrical panels to mitigate risks near the electrical panels.
Inspectors determined that the general work instructions were followed by TDSO personnel but due to the Low Risk screening, there was little risk management in place to preclude the decision-making and dropped tile from the TDSO personnel. As documented in the Results Section, a URI was opened which covers unresolved inspection items related to the control of work in the HVSY and equipment deficiencies.
5. Review the licensees prompt corrective actions to correct this issue and assess the
effectiveness of the actions.
The inspectors reviewed licensee condition reports related to events in the HVSY. The inspectors interviewed licensee personnel responsible for oversight of work activities in the HVSY. The licensee immediately suspended all work activities in the HVSY until a review of work controls could be completed. If emergent and critical work was required, 100 percent direct oversight by a licensed reactor operator was required. Following a review of the events, the licensee reduced the direct oversight requirement to any licensee representative. Long term corrective actions were still being evaluated by the licensee, including evaluation of procedures, process, human performance tools, and oversight, through their corrective action program.
Inspectors determined that at the time of the inspection, the temporary corrective actions were adequate to correct the issue and were effective.
buses A, B and C from the unit auxiliary transformer (UAT) to the startup transformers (SUT)and determine
- (1) if the asbuilt design/configuration is properly modeled in the online maintenance risk analysis program,
- (2) if operators have an appropriate understanding of how they function, and
- (3) determine why the dead-bus transfer did not occur.
The inspectors reviewed design drawings, specifications and interviewed engineering personnel to understand the design of the fast dead-bus transfer from the 1B UAT to the SUTs and why the transfer did not occur on August 3, 2022, following the trip of the main generator.
The non-safety-related 1A, 1B, and 1C 4KV buses, which supply the RCPs, were being powered by the 1B UAT prior to the event. Per design, a loss of voltage (i.e., dead-bus) to the 1B UAT actuates an automatic fast transfer to align the 1A 4KV bus to the 1A SUT and the 1B and 1C to the 1B SUT. Following the trip of the main generator, the automatic transfer to the 1B SUT did not occur because the 1B SUT did not have power as a result of the bumped relay in the HVSY relay house. The automatic transfer to the 1A SUT did not occur due to a failure of the electrical relaying scheme that was independent of the initiating event in the HVSY relay house.
The inspectors determined that a fast dead-bus transfer from the 1B UAT to the 1A SUT should have occurred following a loss of the main generator since there were no problems with the 1A SUT. The inspectors did not identify any trip or overcurrent signals present on the 1A SUT or overcurrent signals present on the 1B UAT. These signals are common to the manual and automatic bus transfer. Based on inspectors review of the drawings and the fact that the licensee was able to restore offsite power through the 1A SUT without any corrective actions, inspectors determined that the fault likely occurred in the relay logic for the automatic fast dead-bus transfer and not the manual transfer logic.
The inspectors reviewed licensee drawings D177146 and D177147, which contained the electrical diagrams for the automatic transfer, and confirmed a licensee evaluation that concluded that three contacts could have failed, which would have prevented the fast dead-bus transfer. Contact EF2X should have closed upon a loss of generator excitation. Contact 21 is time-delayed and should have closed for a brief period after the opening of the UAT breaker DA01. Contact bb is an advanced contact that should have closed within four cycles after breaker DA01 tripped open. This logic should have established a limited time window where both the 2-1 and bb contacts were closed, which would have initiated the bus transfer to the 1A SUT. Since the bus transfer did not occur, the problem is likely one of the three contacts.
At the time of the NRC Special Inspection, the licensee had aligned plant loads to the 1A SUT and bypassed the 1B UAT, which is allowed per plant design. Since the three contacts had not been evaluated, inspectors were not able to determine the failure mechanism. The issue was captured in the licensees corrective action program, and the licensee intends to continue troubleshooting during the next refueling outage.
The inspectors reviewed the structure of the licensees probabilistic risk assessment (PRA) fault tree associated with a loss of the normal power supply (1B UAT) to the RCP buses and the fast transfer to the alternate power source (SUTs). The inspectors discussed the basis of the fault tree and associated basic events with the site PRA engineer. The inspectors questioned the presence of a fire flag gate in the fault tree logic that would limit risk quantification of the fault tree to only fire-initiated sequences. The licensee determined that the presence of the fire flag gate was an error and should not have been present in the fault tree logic. The licensee requantified the model without the gate and confirmed that the logic structure error did not impact the models overall core damage or large early release frequencies. The licensee entered the issue into their corrective action program as CR10900743. The inspectors concluded that the licensees PRA did model the fast transfer function of the power supply for the RCP buses from the normal to alternate power source. The inspectors also concluded that the presence of the fire flag gate did not have a significant impact on the quantification of the model or adversely impact the qualitative risk assessment performed by the licensee for the maintenance activities that took place in the switchyard on August 3, 2022.
The inspectors discussed the design of the fast dead-bus transfer and the events on August 3, 2022, with five licensed operators and determined that licensed operators are trained and knowledgeable of its design and function. The inspectors concluded that operators adequately responded to the loss of power to the balance-ofplant buses even if the specific failure of the fast dead-bus transfer was not immediately recognized during the event.
7. Identify any potential generic safety issues and make recommendations for appropriate
follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
The inspectors reviewed the events and causes to determine if any potential generic safety issues existed for the circumstances surrounding the work in the HVSY. The inspectors noted that NRC Generic Letter (GL) 2006-02, Grid Reliability and the Impact on Plant Risk and the Operability of Offsite Power dated February 1, 2006, previously detailed generic concerns related to electric grid reliability and the impact on nuclear safety. The GL covered the interfaces between nuclear power plant operators and transmission and distribution companies, including the requirements to assess and manage risk of switchyard activities in accordance with the Maintenance Rule, 10 CFR 50.65. The inspectors did not identify any potential generic safety issues that were not already discussed in NRC generic communications.
8. Collect data necessary to support completion of the significance determination process, if
applicable.
The inspectors identified a URI related to the events surrounding the assessment and management of risk inside the HVSY relay house and the incorrect setpoint on the main generator protection circuitry. Inspectors collected data to support review and closure of the URI in a later report.
Operation of the Turbine-Driven AFW pump
9. Review and assess the plant procedures controlling operation of the TDAFW pump.
The inspectors reviewed records from the event, procedures used during the event, corrective action documents, and design documents related to the control of the AFW system. The inspectors interviewed engineers and operators to identify the design of the system, the operation of the TDAFWP, and the events on August 3, 2022. The inspectors also observed a recreation of the event using the plant simulator.
The inspectors focused on emergency operating procedures and standard operating procedures (SOPs) that were used and contributed to the events on August 3, 2022, including FNP1ESP-0.1, Reactor Trip Response, revision 39.0 and FNP1SOP22.0, Auxiliary Feedwater System, revision 83.0. Inspectors also reviewed the revisions to FNP1ESP0.1 as an immediate corrective action following the event. The inspectors interviewed licensed operators and reviewed post-event personnel statements.
The inspectors noted that the procedure step 6.5 of FNP1ESP0.1 required, as a continuous operator action, to monitor steam generator levels and to stop the TDAFWP when steam generators narrow levels were greater than 28 percent and further operation of the TDAFWP was not required. The inspectors determined that the procedure did not direct operators to check for an automatic start signal present for the TDAFWP. Following the event, the licensee revised FNP1ESP0.1 to enhance guidance to operators to preclude attempting to stop the TDAFWP with an automatic start signal present. The inspectors determined that the procedure deficiency directly contributed to the TDAFWP tripping due to overspeed.
The inspectors reviewed Figure 1 of FNP1SOP22.0 which is an operator aid for resetting the TDAFWP trip and throttle valve and is posted on the wall next to the TDAFWP. The inspectors interviewed the system operators who were involved in the event. The operator aid was used during the events on August 3, 2022, following the first overspeed trip of the TDAFWP to locally reset the trip and throttle valve. Following the reset and immediately after opening the trip and throttle valve, the TDAFWP tripped a second time, again due to overspeed. The inspectors identified three deficiencies with the aid as implemented on August 3, 2022:
1. Step 1 stated, Request control room close Q1N12HV3235A/26 & Q1N12HV3235B if an auto
start signal is not present. The inspectors determined that this step did not provide adequate guidance in the event an automatic start signal was present. Because the automatic start signal was present, operations continued to the next step and proceeded to reset and open the trip and throttle valve in accordance with the procedure, which contributed to the second overspeed trip.
2. The inspectors identified that the operator aid did not include guidance to check the status of
the electronic governor controller to determine if it was ready to control the governor valve.
Following the first overspeed trip, the Speed Probes Failed signal was locked into the system, so the governor valve was fully open, and the controller would not modulate the governor valve.
Since the operator aid did not require checking the controller prior to resetting the trip and throttle valve and since operators had limited understanding of the impacts of the Speed Probes Failed signal, there was no opportunity to manually reset the controller to clear the signal.
3. The inspectors identified that the operator aid did not limit the steam admission rate to the
TDAFWP. With an automatic start signal present, the steam admission valves from the main steam lines are open, providing steam pressure up to the trip and throttle valve. A routine start of the TDAFWP is performed by opening steam admission valve Q1N12HV3226 in 2030 seconds to allow the governor time to modulate prior to reaching the overspeed trip setpoint.
Since steam admission valve Q1N12HV3226 was already open following the first overspeed trip and since the motor on the trip and throttle valve opened in approximately 8 seconds, inspectors determined that there was a high likelihood of reaching the overspeed setpoint even if the governor was able to modulate. The operator aid failed to provide adequate guidance for opening the trip and throttle valve with the steam admission valves already open.
The inspectors determined that the deficiencies with procedures FNP1SOP22.0 and FNP1ESP0.1 were caused by the licensees failure to fully understand the design basis of the AFW system. Specifically, the licensee did not understand the design vulnerabilities of operating the TDAFWP with an automatic start signal present. As a result, the licensee failed to correctly translate the design basis into procedures and instructions, including operating procedures and training. Additional details associated with this issue are documented in a finding and associated AV in the Results Section of this report 10. Review and evaluate the operators decision to terminate operation of the TDAFW during this event.
The inspectors reviewed records from the event and interviewed the licensed operators who decided to secure the TDAFWP. The inspectors also observed a recreation of the event on the plant simulator to better visualize the information available to operators during their decision-making. The inspectors determined that the operators adequately responded to the event up to step 6.5 of FNP1ESP0.1. During this step, the procedure directed operators to secure the TDAFWP and did not provide guidance to check for an automatic start signal. The inspectors determined that the operators did not recognize the automatic start signal was present due to an undervoltage condition (i.e., no power) on two out of three RCP buses. There were three bistable status lights available to operators in the control room that were associated with the TDAFWP undervoltage automatic start signal; however, these status lights were on a different panel than the switches for securing the TDAFWP. Interviewed operators identified that they followed the procedure to hold close two steam admission valve switches for at least 30 seconds. The operators did not expect the valves to reopen. However, during the event when the valves automatically reopened, operators stated that they immediately recognized that the undervoltage condition caused the automatic start signal, and that they had attempted to stop the TDAFWP with an automatic start signal locked-in. Due to the design of the facility, there was no mechanism available to the operators to reset or block the TDAFWP undervoltage automatic start signal, until voltage was restored to two of the three RCP buses.
The inspectors determined that approximately 20 minutes after the reactor trip, when directed by procedure to secure the TDAFWP, operators did not identify the undervoltage signals creating a TDAFWP automatic start signal. Although this lack of recognition contributed to the decision-making, inspectors determined that the operators followed procedure FNP1ESP0.1 aswritten.
11. Review and assess the operator training related to the operation of TDAFW pump during a transient.
Approximately 20 minutes after the reactor trip on August 3, 2022, operators attempted to secure the TDAFWP with an automatic start signal present. Upon releasing the two main control room switches, the steam admission valves reopened, and the TDAFWP tripped due to overspeed.
The inspectors interviewed operators and training staff, reviewed training materials, and observed a recreation of the event on the plant simulator. The inspectors determined that licensed operators were trained on the automatic start signals for the TDAFWP and operators demonstrated a working knowledge of the system. The inspectors determined that during the event response, operators missed the opportunity to recognize the automatic start signal, and although additional training may have increased the chance of success, it did not significantly contribute to the first overspeed trip.
Following the first overspeed trip, a system operator was dispatched to locally reset the overspeed trip and open the trip and throttle valve for the TDAFWP per licensee procedure FNP1SOP22.0 Figure 1. Upon opening the trip and throttle valve, the TDAFWP tripped on overspeed again.
The inspectors interviewed system operators and licensed operators on the recovery of the TDAFWP after the first overspeed trip and determined that there were knowledge gaps for conditions like the August 3, 2022, events. Specifically, there was limited knowledge of the electronic governor controller error codes and the impact on TDAFWP operability. This resulted in the missed opportunity to check the electronic governor controller prior to attempting to restart.
Inspectors concluded that training did not affect the decision to attempt to secure the TDAFWP, but adequate training or procedural guidance may have allowed operators to identify the Speed Probes Failed signal on the controller following the first overspeed trip of the TDAFWP.
12. Collect data necessary to support completion of the significance determination process, if applicable.
The inspectors determined that the lack of understanding of the design of the AFW system and the failure to translate the design basis into procedures and instructions, including operating procedures and training, was a performance deficiency. The inspectors gathered data necessary to support the completion of the SDP. Additional details associated with this issue are documented in a finding and associated AV in the Results Section of this report.
Turbine Auxiliary Feedwater Pump Failure 13. Review the licensees technical cause evaluation and corrective action of the TDAFWP trip after the automatic restart.
The inspectors reviewed the licensees condition reports, work orders, cause evaluations, and design documents associated with the TDAFWP. The inspectors also discussed the events with operations and engineering personnel to identify contributing causes and understand corrective actions.
Following the reactor trip, all three AFW pumps started automatically as expected and began feeding the three steam generators. The two motor-driven AFW pumps (MDAFWPs) received automatic start signals due to low steam generator levels. The single TDAFWP received automatic start signals due to both low steam generator levels and undervoltage on two-out-ofthree RCP buses. With two MD AFW pumps running and steam generator levels stabilized, operators attempted to stop the TDAFW per procedure by holding two control room switches in the STOP position for approximately 45 seconds, which closed steam admission valves Q1N12HV3235A, Q1N12HV3235B, and Q1N12HV3226. However, as the automatic start signal from the RCP bus undervoltage condition was still present, when the operator released the two control room switches to the neutral AUTO position, all three steam admission valves reopened. When the valves were closing and closed, TDAFWP speed lowered from approximately 4000 rpm (normal operating speed) to approximately 145 rpm. When the control room switches were released and the valve reopened, the TDAFWP speed increased rapidly and immediately tripped on overspeed.
The Woodward 505 digital governor controller has a design feature that when TDAFWP speed drops below a predefined setpoint (as set to 200 rpm during the event), a signal is generated for Speed Probes Failed, which fully opens the TDAFWP governor valve and prevents it from modulating closed to control pump speed. Once the TDAFWP speed reaches another predefined setpoint (as set to 100 rpm during the event), a reset signal is generated to clear the Speed Probes Failed signal and allows the digital governor to resume control of the governor valve. However, because the operators released the steam admission valve switches with TDAFWP speed within the 100200 rpm band, the Speed Probes Failed remained locked into the system. Since the governor valve was fully open and did not modulate with increasing speed, admitting steam to the turbine resulted in an overspeed trip.
Corrective actions to mitigate the design vulnerability included updating the plant simulator, revising procedures to ensure operators do not attempt to stop a running TDAFWP with an automatic start signal present, and revising standard operating procedure FNP1SOP22.0 to ensure operators have adequate guidance to reset the trip and throttle valve under all plant conditions.
Following the first overspeed trip, a system operator was dispatched to locally reset the overspeed trip and open the trip and throttle valve for the TDAFWP per licensee procedure FNP1SOP22.0 Figure 1. Upon opening the trip and throttle valve, the TDAFWP tripped on overspeed again. As the TDAFWP was not necessary for maintaining steam generator levels at that time, operations left it in the tripped condition and declared it inoperable.
The inspectors reviewed plant data, interviewed operators, and reviewed system design documents. The inspectors determined that the Speed Probes Failed signal, which remained on the controller after the first overspeed trip, would have prevented the governor from controlling the speed of the TDAFWP. Therefore, opening the trip and throttle valve with steam admission valves already fully open (due to the auto start signal being present) prevented the controller from modulating the governor valve prior to reaching the overspeed trip of the TDAFWP. Furthermore, the inspectors determined that the TDAFWP would likely have tripped due to overspeed because the trip and throttle valve stroke open time (nominally 8 seconds)was not designed to ensure pump speed control during TDAFWP startup (the TDAFWP steam admission valve Q1N12HV3226 has a design feature to slowly open in nominally 25 seconds).
The inspectors determined that procedural guidance for the system operator in FNP-1SOP22.0 Figure 1, Operator Aid for Resetting the TDAFW Pump Trip Throttle Valve was inadequate to identify and mitigate the conditions that resulted in the second overspeed trip of the TDAFWP.
The licensee revised procedural requirements for resetting the TDAFWP, established a corrective action to update the simulator to include the design vulnerability with the Speed Probes Failed signal, and provided training to operators on the system operation.
14. Review the work package and the work performed to correct the failure mode of the TDAFWP.
Inspectors reviewed licensee condition reports, evaluations, and corrective work orders and walked down components that were repaired following the events. The licensee identified a failed closed limit switch on steam admission valve Q1N12HV3235B which prevented the valve from remaining closed. Inspectors determined that this failure did not affect the events on August 3, 2022 and was adequately evaluated and repaired by the licensee.
Inspectors determined that the mechanical and electrical components associated with the TDAFWP operated as designed and only required work to be performed to enhance procedures.
15. Determine if there is a design deficiency that would cause the TDAFWP to trip when stopping with an initiation signal still present.
The inspectors reviewed design change package (DCP) 1060862601 from 2011 that installed the digital governor controller, compared electrical control drawings to event data from August 3, 2022, and interviewed engineers and operators.
The inspectors noted that the DCP detailed the function of the Woodward 505 controller and industry operations experience of the controller failing that resulted in overspeed trips. These failures were all attributed to a variety of electrical component failures which failed the controller.
The DCP stated the TDAFWP new governor software/firmware also meets all current required performance, and qualification requirements, and would have no new failure modes or effects at the level of the design function.
The inspectors reviewed the design of the protection circuitry of the Woodward 505 governor including the Speed Probes Failed signal, which indicates a loss of the controllers ability to control the governor valve. During the event on August 3, 2022, this signal would have been locked into the controller since the pump speed lowered to between 100 and 200 rpm. If the steam valves had reopened before the speed lowered to 200 rpm, the Speed Probes Failed signal would not have been generated, and the governor would have controlled the speed as the pump ramped back to 100 percent speed. A reset signal is generated if there is no start signal (the auto start signal was defeated while holding the steam admission valve switches in OFF) and the speed lowers to 100 rpm. If the steam valves had been held closed until the speed lowered to less than 100 rpm, the Speed Probes Failed signal would have been reset, and the governor would have controlled the speed as the pump ramped back to 100 percent speed.
The inspectors determined that this aspect of the Woodward governor design was not considered during the implementation of the 2011 modification to add the digital governor controller to the Unit 1 TDAFWP, and therefore was not recognized by the licensee as a potential failure mechanism of the TDAFWP. As such, standard operating procedures, including FNP1SOP22.0, were not revised to identify and preclude operation with the Speed Probes Failed signal.
16. Review the maintenance history for components that did not operate as designed.
The inspectors reviewed the maintenance history of the failed closed limit switch on steam admission valve Q1N12HV3235B, determined that the preventive maintenance and testing was adequate, and that the failure was a random failure. The inspectors also noted that the failure of the limit switch did not affect the events on August 3, 2022.
Inspectors reviewed the design and event data for the TDAFWP and concluded that all components operated as designed.
17. Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
The inspectors collected information that may constitute potentially generic safety issues from the review of the two overspeed trips of the TDAFWP on August 3, 2022. These will be evaluated by the NRCs Operational Experience program.
18. Collect data necessary to support completion of the significance determination process, if applicable.
The inspectors determined that the lack of understanding of the design of the AFW system and the failure to translate the design basis into procedures and instructions, including operating procedures and training, was a performance deficiency. The inspectors gathered data necessary to support the completion of the SDP. Additional details associated with this issue are documented in a finding and associated AV in the Results Section of this report.
INSPECTION RESULTS
Inadequate Design Control of the Auxiliary Feedwater System Resulting in the Inoperability of the Turbine-driven Auxiliary Feedwater Pump Cornerstone Significance/Severity Cross-Cutting Report Aspect Section Mitigating Pending None (NPP) 93812 Systems Apparent Violation AV 05000348/202205001 Open EA22095 A self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of 10 CFR Part 50, Appendix B, Criterion III, Design Control were identified for the licensees apparent failure to adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the turbine-driven auxiliary feedwater pump (TDAFWP) on August 3, 2022.
Description:
On August 3, 2022, following a reactor trip and automatic start of the TDAFWP, operators were directed by licensee procedure FNP1ESP0.1, Reactor Trip Response, revision 39.0, to stop the TDAFWP. Operators held the hand switches in the OFF position for approximately 45 seconds which closed the three steam admission valves and stopped steam flow to the TDAFWP. The TDAFWP speed lowered to approximately 145 rpm.
Because TDAFWP speed was within a 200-100 rpm band, the electronic governor system generated a locked-in Speed Probes Failed signal, which fully opened the TDAFWP governor valve. When the operators released the hand switches, the steam admission valves automatically reopened as designed because the automatic start signal was still present and TDAFWP speed rapidly increased. The Speed Probes Failed signal prevented the TDAFWP governor valve from controlling TDAFWP speed which caused the TDAFWP to trip due to overspeed. To recover the pump, a building operator locally reset the overspeed trip mechanism in accordance with licensee procedure FNP1SOP22.0, Auxiliary Feedwater System revision 83.0, however, both the TDAFWP undervoltage automatic start signal and the Speed Probes Failed signals were still locked-in. Based on the combination of these events, when the building operator reopened the trip and throttle valve as directed by the procedure, the TDAFWP tripped again due to overspeed. The TDAFWP was declared inoperable.
The inspectors identified two examples where the licensee did not adequately translate the design basis of the Auxiliary Feedwater System (AFW) into procedures and instructions which resulted in the inoperability of the TDAFWP on August 3, 2022.
Example 1: FNP1ESP0.1, Reactor Trip Response In 2011, the licensee modified the TDAFWP under design change package (DCP)106086201 by installing a Woodward 505 digital governor controller including a servo motor to control the position of the TDAFWP governor valve. The Woodward 505 digital governor controller has a protective design feature that will generate a Speed Probes Failed signal when it senses a loss of control, which automatically opens the governor valve to 100 percent, thereby preventing the governor valve from controlling pump speed. This signal is generated when pump speed reaches a predefined setpoint of 200 rpm and can only be reset when pump speed lowers to another predefined setpoint of 100 rpm. During a routine shutdown of the TDAFWP, the Speed Probes Failed signal would be generated at 200 rpm, and then reset at 100 rpm, as pump speed decreased.
On August 3, 2022, licensed operators attempted to close the three steam admission valves to the TDAFWP per procedure FNP1ESP0.1, Reactor Trip Response by holding two control room switches in the STOP position for at least 30 seconds. When the operator released the hand switches, the pump speed was approximately 145 rpm and the Speed Probes Failed signal was still locked into the controller. The steam admission valves went full open as designed because the undervoltage automatic start signal was still present, and the pump tripped on overspeed because the governor valve was locked-out at 100 percent open and was prevented from controlling pump speed.
Example 2: FNP1-SOP22.0. Auxiliary Feedwater System Following the first overspeed trip, a building operator attempted to reset the TDAFWP by using Figure 1 of procedure FNP1-SOP22.0. Auxiliary Feedwater System, revision 83.0, which is an operator aid for resetting the TDAFWP trip and throttle valve posted on the wall next to the pump. Following the reset and immediately after opening the trip and throttle valve, the pump tripped again due to overspeed. Step 1 of Figure 1 stated, Request control room close [steam admission valves] Q1N12HV3235A/26 & Q1N12HV3235B if an auto start signal is not present. Since the automatic start signal for the TDAFW pump was present, operations proceeded to the next steps to reset and open the trip and throttle valve in accordance with the procedure. When the operator opened the trip and throttle valve the TDAFWP experienced a second overspeed trip because the governor valve was still open due to the Speed Probe Failure signal and the governor could not modulate to control the steam entering pump.
Licensee procedure NMP-ES044, version 7.0, was used to implement DCP 1060862601 in 2011 and required an evaluation of plant site considerations including testing, training, maintenance, and operations. The DCP covered each of the items but did not specifically address changes to operating procedures.
The inspectors concluded that on August 3, 2022, the TDAFWP pump was rendered inoperable because operating procedure FNP1ESP0.1 did not provide adequate guidance to ensure the Speed Probe Failed signal was reset before operators released the hand switches for the steam admission valves. Additionally, the TDAFWP was rendered inoperable because the licensee did not update Figure 1of FNP1SOP22.0 to check the controller for a Speed Probes Failed signal and failed to provide adequate guidance to locally reset the overspeed trip mechanism if an automatic start signal was present following the modification in 2011.
Corrective Actions: The licensee entered these issues into their corrective action program and revised operating procedures to preclude securing steam flow to the TDAFWP with an automatic start signal present. Additionally, the licensee established corrective actions to review the design of the TDAFWP controller and revise additional operating procedures to identify and reset failure signals prior to starting the TDAFWP.
Corrective Action References: Condition reports 10900508, 10899966, 10899944, 10898654, and 10901729
Performance Assessment:
Performance Deficiency: A performance deficiency was identified for the licensees failure to adequately translate all aspects of the AFW system design into plant operating procedures.
As a result, operating procedures FNP1ESP0.1, Reactor Trip Response, revision 39.0 and FNP1SOP22.0 Auxiliary Feedwater System, revision 83.0 did not provide adequate guidance to secure steam flow to the TDAFWP with an automatic start signal and a Speed Probes Failed signal present.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the Design Control attribute of the Mitigating Systems cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the licensee failed to provide adequate guidance in operating procedures FNP1ESP0.1 and FNP1SOP22.0 which rendered the TDAFWP inoperable on August 3, 2022.
Significance: The inspectors assessed the significance of the finding using IMC 0609 Appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined that because the finding could not be easily screened to be of very low safety significance, a detailed risk evaluation (DRE) was required. Specifically, the procedural deficiencies resulted in the inoperability of the TDAFWP due to overspeed trips on August 3, 2022, the procedural deficiencies existed for greater than one year and could have resulted in rendering the pump inoperable under certain conditions, and the procedural deficiencies affected the operators capability to recover the pump following overspeed trip. Consequently, the inspectors determined that the finding represented a loss of the PRA function of one train (i.e., TDAFWP) of a multi-train technical specification (TS) system for greater than its TS allowed outage time and required a DRE.
Cross-Cutting Aspect: Not Present Performance. No cross-cutting aspect was assigned to this finding because the inspectors determined the finding did not reflect present licensee performance.
Enforcement:
Apparent Violation: 10 CFR 50, Appendix B, Criterion III, Design Control states, in part, that measures shall be established to assure that applicable regulatory requirements and the design basis are correctly translated into specifications, drawings, procedures, and instructions. The design basis of the AFW system is to provide an emergency source of water to three steam generators to ensure adequate steam generator water levels to cool the RCS. At any point when the two motor-driven AFW pumps were not available, the TDAFWP was required to provide adequate flow to the steam generators.
Licensee procedure NMP-ES044, version 7.0, was used to implement DCP 1060862601 in 2011 and required an evaluation of plant site considerations including testing, training, maintenance, and operations. The DCP covered each of the items but did not specifically address changes to operating procedures. Contrary to the above, two aspects of the design of the AFW system were not correctly translated into procedures and instructions during the implementation of DCP 1060862601:
1. Procedure FNP1ESP0.1, Reactor Trip Response, revision 39.0, did not preclude or
provide sufficient guidance to secure the TDAFWP with an automatic start signal present. By following the procedure and due to the control system of the Woodward 505 electronic governor, operators rendered the TDAFWP inoperable when they slowed the pump to between 100 and 200 rpm and then the steam admission valves reopened.
2. Figure 1 of procedure FNP1SOP22.0. Auxiliary Feedwater System, revision 83.0, is an
operator aid for locally resetting the TDAFWP trip and throttle valve. Following the reset and immediately after opening the trip and throttle valve, the pump tripped again due to overspeed. The procedure did not include guidance to check the status of the electronic governor controller to determine if it was ready to control, which prevented the controller from modulating the governor valve before reaching the overspeed setpoint.
Enforcement Action: This violation is being treated as an apparent violation pending a final significance (enforcement) determination.
Unresolved Item Unit 1 Reactor Trip and Partial Loss of Offsite Power 93812 (Open) URI 05000348/202205002
Description:
An Unresolved Item (URI) was identified to further review the events on August 3, 2022, which resulted in the Unit 1 reactor trip and partial loss of offsite power.
There are two distinct issues that require additional reviews to determine if a performance deficiency existed. First, the licensee evaluated high voltage switchyard (HVSY) work activities as Low Risk and a human performance event during those activities resulted in the loss of one of the two 230KV power lines in the high voltage switchyard and a loss of power to the B 4KV emergency bus. Second, due to an incorrect setpoint on a main generator protection relay, the loss of the 230KV bus resulted in a reactor trip and loss of several large non-safety-related pumps. These are being considered under one URI since both issues contributed to the event.
Regarding the first issue, due to the proximity of the work activities with respect to the trip sensitive relays, the inspectors determined that it was near enough to the electrical cabinet to be considered high risk work in accordance with the licensee procedure NMP-DP001, Operational Risk Assessment, revision 23.0. However, the term near is not defined by NMP-DP001 and more information is required to determine if the inspectors original assumption is accurate. Also, additional consideration is required to determine whether it was within the licensees ability to correct the TDSO technician from making the error. This may require additional reviews of the licensees interface agreement with the TDSO and other industry or agency guidance regarding the expectations of licensees interactions with switchyard operators.
Regarding the second issue, information was identified after the onsite portion of the Special Inspection that identified the incorrect setpoint on the KC2 relay. If the KC2 relay had worked correctly, the inadvertent switchyard house relay actuation would have only resulted in a partial loss of offsite power and an approximate loss of 10 percent RTP versus a reactor trip from 100 percent RTP. The inspectors were not able to gather enough information to determine if a preliminary performance deficiency existed and whether this may be considered the proximate cause of the event itself.
Planned Closure Actions: Regional inspectors intend to review the design and calculations of the KC-2 generator protection relay, the replacement of the main transformers when the relay setpoints should have been modified, and the impacts on the risk in the high voltage switchyard. Regional inspectors also intend to determine if the licensee had the ability to foresee and correct the TDSO technicians decision to move the floor tile next to an electrical cabinet.
Licensee Actions: As an immediate corrective action, if emergent and critical work was required, 100 percent direct oversight by a licensed reactor operator was required in the HVSY relay house. Following a review of the events, the immediate corrective action was downgraded to 100 percent oversight of all activities by a licensee representative. Long term corrective actions are still being evaluated by the licensee, including evaluation of procedures, process, human performance tools, and oversight, through their corrective action program. The NRC will follow-up on long term corrective actions in subsequent inspections.
The licensee also plans to set the KC-2 relay setpoint to the correct value during the upcoming refueling outage.
Corrective Action References: Condition reports 10898660, 10900230, 10900743, 10901689 and TE1109396
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
On August 17, 2022, and September 19, 2022, the inspectors presented the special inspection results to Site Vice President Delson Erb and other members of the licensee staff.
DOCUMENTS REVIEWED
Inspection Type Designation Description or Title Revision or
Procedure Date
93812 Corrective Action 10898654,
Documents 10898656,
10898660,
10898661,
10898662,
10898665,
10898666,
10898681,
10898692,
10898700,
10898711,
10898826,
10898855,
10898899,
10899033,
10898687
Corrective Action 10899944,
Documents 10899966,
Resulting from 10900226,
Inspection 10900230,
10900424,
10900508,
10900683,
10900743,
10901568,
10901689,
10901729,
10900354,
10901699,
TE1109097,
TE1109396
Drawings D169970 230KV Single Line Diagram 49
Inspection Type Designation Description or Title Revision or
Procedure Date
D169971 Elementary Diagram for 230KV PCB 924 29
D169971 230KV Elementary 37
D169971 Panel 1B 230KV Elementary Diagram 20
D172784 Elementary Diagram - Generator Transformer Auxiliary 05/22/1975
Relays
D172784 Elementary Diagram - Generator & Transformer Auxiliary 10/21/1975
Relays
D172784 Elementary Diagram - Generator and Transformer Auxiliary 05/22/1975
Relays
D177146 Elementary Diagram 4160V Bus 1A INC AUX TRANS & 06/09/1971
AUTO TRANSFER
D177147 Elementary Diagram 4160V Bus 1A INC AUX TRANS 1A & 06/09/1971
AUTO TRANS
FNP1STP27.1 AC Source Verification 43.0
Engineering Design Change TDAFWP Governor Replacement 11/18/2010
Changes Package
1060862601
Miscellaneous Reactor Trip 04/25/2022
Report
Procedures FNP1ESP0.1 Reactor Trip Response 39.0
FNP1SOP22.0 Auxiliary Feedwater System 83.1
FNP1UOP1.2 Startup from Unit from Hot Standby to Minimum Load 120.0
NMP-DP001 Operational Risk Awareness 23.0
NMP-GM-021 Switchyard Access and Maintenance Controls 7.0
Work Orders SNC1188231 APCO: PCB 820, 914, 924 Replacement & 1B SAT Relay 0
Upgrade
August 9, 2022
MEMORANDUM TO: Cary
- M. Read, Senior Resident Inspector (V. C. Summer)
Reactor Project Branch 4
Division of Reactor Projects
Signed by Dudes, Laura
FROM: Laura A. Dudes on 08/09/22
Regional Administrator
SUBJECT: SPECIAL INSPECTION CHARTER TO EVALUATE THE LOSS
OF OFFSITE POWER TO THE UNIT 1 4160V B TRAIN
EMERGENCY POWER BUS AND FAILURE OF THE TURBINE
DRIVEN AFW PUMP
You have been selected to lead a Special Inspection to assess the circumstances surrounding
loss of offsite power to the 4.16-kV B train emergency power bus on August 3, 2022 that
resulted in a main generator/turbine trip and subsequent automatic reactor shutdown.
Additionally, you are to assess the circumstances surrounding the operation and failure of the
turbine driven auxiliary feedwater pump (TDAFW). Your onsite inspection should begin on
August 10, 2022. Michael Meeks, Senior Operations Engineer and Peter Meier, Senior Resident
Inspector Farley will be assisting you in this inspection.
A. Basis
At approximately 12:58 CST PM on August 3, 2022, Farley Unit 1 was in mode 1 at 100%
power when the reactor automatically tripped due to a main generator lockout signal and
main turbine trip. The event started when an Alabama Power employee inadvertently
bumped a relay that caused the opening of the supply breakers for the Unit 1 B startup
transformer and a loss of offsite power to the B train emergency bus. The Unit 1 B
emergency diesel generator automatically started to restore power to the "B" train
emergency bus as expected. Unit 1 operated in natural circulation in Mode 3 after all three
reactor coolant pumps (RCP) tripped due to the loss of the A, B and C 4.16-kV
busses. During this event, the main condenser was unavailable because there was no
power for the Unit 1 circulating water pumps. Operators stabilized the plant with the auxiliary
feedwater system (AFW) and the atmospheric relief valves.
Following the reactor trip, all three AFW pumps started automatically as expected. The
turbine driven (TD) AFW pump auto started and ran approximately 20 minutes after
receiving a station blackout signal from the loss of all three RCPs (i.e., undervoltage
condition in two-of-three RCP buses). With two motor driven (MD) AFW pumps running,
operators attempted to manually stop the TDAFW, however, the auto-start signal was still
present (RCP buses remained de-energized) and the TDAFW pump restarted but
immediately tripped on overspeed. Following the overspeed trip the operators secured the
steam supply valves to allow the turbine to come to a full stop before attempting to restart it.
However, when the valve(s) were re-opened the TDAFW pump tripped again on overspeed.
The TDAFW pump was declared inoperable. Following the event, the licensee identified that
one of the two steam admission valves for the TDAFW pump was stuck open and appeared
to have caused the overspeed condition of the TDAFW pump
C. Read 2
On August 4, 2022, the licensee restored offsite power to the B emergency bus and the
Unit 1 B emergency diesel was secured. Operators started the 1B RCP for forced
circulation.
This Special Inspection is chartered to identify the circumstances surrounding the oversight
and control of the work activities in the switchyard at the time of the event, review the cause
of the loss of the Unit 1 B startup transformer, review licensed operator response to the
transient, including the decision to stop the TDAFW pump, review the technical cause
evaluation for the TDAFW overspeed trip, and review the prompt corrective actions
associated with operation of the TDAFW pump failure and the work activities in switchyard
prior to the event.
B. Scope
The inspection is expected to perform data gathering and fact-finding to address the
following:
Work Activities in the High Voltage Switchyard
Review and evaluate the licensees process for control of work in the switchyard. This
should include the work control process, management of plant risk, and oversight of
switchyard work.
Verify that the bumped relay caused the breakers in the switchyard to open as they did
(verify breaker coordination) prior to the reactor trip.
Review the work control documents that were being used to perform switchyard
maintenance, including release of work and work scope.
Review the work that was being performed in the switchyard, the location of the work
and the equipment in the work area to confirm the work was being performed in
accordance with procedures and work instructions.
Review the licensees prompt corrective actions to correct this issue and assess the
effectiveness of the actions.
Review and understand the design of the 'fast' dead-bus transfer of non-safety related
4.16-kV buses A, B and C from the unit auxiliary transformer to the startup
transformer and determine (1) if the as-built design/configuration is properly modeled in
the online maintenance risk analysis program, (2) if operators have an appropriate
understanding of how they function, and (3) determine why the dead bus transfer did not
occur.
Identify any potential generic safety issues and make recommendations for appropriate
follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
Collect data necessary to support completion of the significance determination process,
if applicable.
Operation of the Turbine Driven AFW pump
Review and assess the plant procedures controlling operation of the TDAFW pump.
Review and evaluate the operators decision to terminate operation of the TDAFW
during this event.
C. Read 3
Review and assess the operator training related to the operation of TDAFW pump during
a transient.
Collect data necessary to support completion of the significance determination process,
if applicable.
Turbine Auxiliary Feedwater Pump Failure
Review the licensees technical cause evaluation and corrective action of the TDAFW
pump trip after the automatic restart.
Review the work package and the work performed to correct the failure mode of the
TDAFW pump.
Determine if there is a design deficiency that would cause the TDAFW pump to trip when
stopping with an initiation signal still present.
Review the maintenance history for components that did not operate as designed.
Identify any potential generic safety issues and make recommendations for appropriate
follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
Collect data necessary to support completion of the significance determination process,
if applicable.
C. Guidance
Inspection Procedure 93812, "Special Inspection," provides additional guidance to be used
during the conduct of the Special Inspection. Your duties will be as described in Inspection
Procedure 93812. The inspection should emphasize fact-finding in its review of the
circumstances surrounding the event. Safety or security concerns identified that are not
directly related to the event should be reported to the Region II office for appropriate action.
You will report to the site, conduct an entrance, and begin inspection no later than
August 10, 2022. A daily status briefing of Region II management will be provided beginning
the second day on-site at approximately 4:00 p.m., Eastern Daylight Time (EDT). In
accordance with IP 93812, you should promptly recommend a change in inspection scope
or escalation if information indicates that the assumptions utilized in the MD 8.3 risk analysis
were not accurate. A report documenting the results of the inspection should be issued
within 45 days of the completion of the inspection. The report should address all applicable
areas specified in Section 03.02 of Inspection Procedure 93812. At the completion of the
inspection, you should provide recommendations for improving the reactor oversight
process baseline inspection procedures and the special inspection process based on any
lessons learned.
This charter may be modified should you develop significant new information that warrants
review. Should you have any questions concerning this charter, contact Alan Blamey at
404-997-4415.
Docket No. 50-348
License No. NPF-2
CONTACT: Alan Blamey, RII/DRP
404-997-4415
Memo ML22221A092
R-II/DRP/RPB1 R-II/Division of
OFFICE R-II/DRP/RPB2 R-II/DRP
/CRO Reactor Projects
NAME CScott CS ABlamey AB LSuggs LS MMiller MM
DATE Aug 9, 2022 Aug 9, 2022 Aug 9, 2022 Aug 9, 2022
OFFICE R-II
NAME LDudes LD
DATE Aug 9, 2022