IR 05000440/2019003
| ML19318F401 | |
| Person / Time | |
|---|---|
| Site: | Perry  |
| Issue date: | 11/14/2019 |
| From: | Billy Dickson Reactor Projects Region 3 Branch 4 |
| To: | Payne F FirstEnergy Nuclear Operating Co |
| References | |
| IR 2019003 | |
| Download: ML19318F401 (25) | |
Text
November 14, 2019
SUBJECT:
PERRY NUCLEAR POWER PLANT - INTEGRATED INSPECTION REPORT 05000440/2019003
Dear Mr. Payne:
On September 30, 2019, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at Perry Nuclear Power Plant and discussed the results of this inspection with you and other members of your staff. The results of this inspection are documented in the enclosed report.
Two findings of very low safety significance (Green) are documented in this report. One of these findings involved a violation of NRC requirements. We are treating this violation as a non-cited violation (NCV) consistent with Section 2.3.2 of the Enforcement Policy.
If you contest the violation or significance or severity of the violation documented in this inspection report, you should provide a response within 30 days of the date of this inspection report, with the basis for your denial, to the U.S. Nuclear Regulatory Commission, ATTN:
Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region III; the Director, Office of Enforcement; and the NRC Resident Inspector at Perry Nuclear Power Plant.
If you disagree with a cross-cutting aspect assignment or a finding not associated with a regulatory requirement in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 20555-0001; with copies to the Regional Administrator, Region III; and the NRC Resident Inspector at Perry Nuclear Power Plant. This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with 10 CFR 2.390, Public Inspections, Exemptions, Requests for Withholding.
Sincerely,
/RA/
Billy C. Dickson, Jr., Chief Branch 2 Division of Reactor Projects
Docket No. 05000440 License No. NPF-58
Enclosure:
As stated
Inspection Report
Docket Number:
05000440
License Number:
Report Number:
Enterprise Identifier: I-2019-003-0056
Licensee:
FirstEnergy Nuclear Operating Company
Facility:
Perry Nuclear Power Plant
Location:
Perry OH
Inspection Dates:
July 01, 2019 to September 30, 2019
Inspectors:
S. Bell, Health Physicist
J. Cassidy, Senior Health Physicist
J. Nance, Senior Resident Inspector
J. Rutkowski, Project Engineer
J. Steffes, Senior Resident Inspector
Approved By:
Billy C. Dickson, Jr., Chief
Branch 2
Division of Reactor Projects
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting an integrated inspection at Perry Nuclear Power Plant in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Failure to Properly Seat Essential Service Water "A" Pump Control Power Fuses results in Loss of Safety Function Cornerstone Significance Cross-Cutting Aspect Report Section Mitigating Systems Green NCV 05000440/2019003-01 Open
[H.9] - Training 71152 A self-revealed finding of very low safety significance (Green) and associated non-cited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the licensee's failure to accomplish work activities in accordance with the instructions, procedures, or drawings. Specifically, the licensee failed to properly install the A essential service water control power fuses in accordance with SOI-R22, Metal Clad Switchgear 5-15 kV, paragraph 2.3 such that the fuses were able to back out and cause an unplanned inoperability of several safety related systems during routine breaker operation.
LER 2019-001-00 for Perry Nuclear Power Plant, Loss of Generator Excitation Causes Turbine Trip and Reactor Protection System Actuation.
Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events Green FIN 05000440/2019003-02 Open None (NPP)71153 A self-revealed finding of very low safety significance (Green) was identified for the licensee's failure to perform adequate preventive maintenance on the main generator exciter brushes, which led to brush failures, loss of generator field excitation, generator lockout, and reactor scram. No violation of regulatory requirements was identified.
Additional Tracking Items
None.
PLANT STATUS
The plant began the inspection period at rated thermal power. On July 27, 2019, at 7:29 p.m.,
the unit scrammed in response to a turbine trip caused by closure of the turbine stop valves.
The turbine stop valve closure occurred during turbine mechanical overspeed testing due to a fault in the trip test circuitry. The licensee replaced the mechanical shutoff valve in the overspeed test circuitry and commenced a reactor startup on July 30, 2019, achieved criticality at 1:08 a.m. on July 31, 2019 and synched to the grid later that day. The unit was returned to rated thermal power at 12:40 a.m. on August 9, 2019. On September 14, 2019, the licensee commenced power reduction to 18 percent reactor power in order to address a hydraulic leak associated with number 3 turbine control valve. Following attempted repairs to the number three turbine control valve, the licensee commenced power ascension and returned the plant to rated thermal power on September 16, 2019. The reactor remained at approximately 100 percent power for the remainder of the inspection period except for rod sequence exchanges and required testing.
On April 25, 2018, FirstEnergy Solutions (FES) / FirstEnergy Nuclear Operating Company (FENOC) notified the U.S. Nuclear Regulatory Commission (NRC) that they intend to shut down all four of their operating nuclear power plants (ADAMS Accession Number ML18115A007). On March 21, 2018, FES, FirstEnergy Nuclear Generation (FENGEN), and FENOC filed for bankruptcy. On July 26, 2019, FES/FENOC submitted a letter to the NRC withdrawing the April 25, 2018, certification of permanent cessation of power operations for Davis-Besse Nuclear Power Station and Perry Nuclear Power Plant (ADAMS Accession Number ML19207A097). The NRC continues to maintain focus on public health and safety and the protection of the environment. This will include a continuous evaluation by inspectors to determine whether the licensee's financial condition is impacting safe operation of the plant.
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors performed plant status activities described in IMC 2515 Appendix D, Plant Status and conducted routine reviews using IP 71152, Problem Identification and Resolution. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
REACTOR SAFETY
71111.04Q - Equipment Alignment
Partial Walkdown Sample (IP Section 03.01) (2 Samples)
- (1) B train of motor control center and switchgear ventilation and battery exhaust systems on September 4 to 5, 2019
- (2) Division 1 diesel generator and associated subsystems while the division 2 diesel generator was inoperable for a surveillance test on September 30, 2019
71111.05Q - Fire Protection
Quarterly Inspection (IP Section 03.01) (4 Samples)
- (1) Fuel handling building 620 foot elevation on September 10, 2019
- (2) Auxiliary building 599 foot elevation fire zone 1AB-2 on September 27, 2019
- (3) Intermediate building 620 foot elevation fire zone 0IB-3 on September 27, 2019
- (4) Service building 604 foot and 620 foot elevations on September 27, 2019
71111.06 - Flood Protection Measures
Inspection Activities - Internal Flooding (IP Section 02.02a.) (1 Sample)
- (1) Liquid radwaste sumps system following EOP-3, Secondary Containment Control, procedure entry due to back leakage into the B residual heat removal pump room sump on July 9 and 10, 2019
71111.11Q - Licensed Operator Requalification Program and Licensed Operator Performance
Licensed Operator Performance in the Actual Plant/Main Control Room (IP Section 03.01) (1 Sample)
- (1) The inspectors observed and evaluated licensed operator performance in the Control Room during plant cooldown following a turbine trip and reactor scram due to a turbine overspeed testing malfunction on July 27, 2019. Additionally the inspectors observed and evaluated licensed operator performance in the Control Room for the subsequent reactor startup on July 31, 2019.
Licensed Operator Requalification Training/Examinations (IP Section 03.02) (1 Sample)
- (1) The inspectors observed and evaluated an operations crew-evaluated scenario in the plant training simulator on July 22, 2019.
71111.12 - Maintenance Effectiveness
Routine Maintenance Effectiveness Inspection (IP Section 02.01) (2 Samples)
- (1) Leak detection system following drywell identified leakage increase on August 8 to 21, 2019
- (2) Review of generator excitor brush inspections following turbine trip and reactor scram on August 8 to 19, 2019
71111.13 - Maintenance Risk Assessments and Emergent Work Control
Risk Assessment and Management Sample (IP Section 03.01) (4 Samples)
- (1) Plant risk evaluation with the plant shutdown and cooled down during troubleshooting and corrective efforts associated with the turbine trip and reactor scram that occurred during turbine mechanical overspeed testing on July 29 to 30, 2019
- (2) Risk management review for extended low power operations with the main generator offline for circuitry repairs on August 7 to 19, 2019
- (3) Risk management review for parameters indicating a small leak in the drywell on August 21 to 26, 2019
- (4) Risk assessment and management for 5A feedwater heater extraction steam supply line pin hole steam leak on inside edge of elbow with housekeeping clamp installed on August 26 to 28, 2019
71111.15 - Operability Determinations and Functionality Assessments
Operability Determination or Functionality Assessment (IP Section 02.02) (6 Samples)
- (1) Roll-up fire barrier between diesel work shop and diesel generator hallway functionality assessment on July 11 to 12, 2019
- (2) Diesel driven fire pump functionality assessment on July 15 to 16, 2019
- (3) Flex lake water pump 2 functionality assessment on August 19 to 23, 2019
- (4) Division 1 emergency diesel generator circulating lube oil pump and lube oil heater functionality assessment August 19 to 23, 2019
- (5) Nuclear closed cooling drywell inboard isolation valve failed fuse on August 30 to September 5, 2019
- (6) Turbine control valve number 3 reactor protection system lack of indications during surveillance testing and unplanned Limiting Condition for Operation entry on September 15 to 16, 2019
71111.19 - Post-Maintenance Testing
Post-Maintenance Test Sample (IP Section 03.01) (6 Samples)
- (1) Rod control and information system circuit card replacement testing on July 15, 2019
- (2) A reactor recirculation loop flow control valve controller testing following replacement of Modicon I/O card 'GND'/CR on July 23, 2019
- (3) Diesel driven fire pump test following replacement of coolant relief valve and alternator on July 24, 2019
- (4) D main steam line tunnel high temperature functional following agastat time delay relay replacement on August 16, 2019
- (5) B flex lake water pump testing following breaker linkage repair on August 29, 2019
- (6) B reactor feedwater booster pump test following seal replacement on August 30 to September 3, 2019
71111.20 - Refueling and Other Outage Activities
Refueling/Other Outage Sample (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated forced outage activities following turbine trip and subsequent reactor scram due to the turbine overspeed testing event on July 27 to 30, 2019.
71111.22 - Surveillance Testing
Surveillance Tests (other) (IP Section 03.01) (3 Samples)
- (1) SVI-E31-T5311-B, Turbine Building Main Steam Line Temperature High Channel Functional for 1E31-N361B and 1E31-N361C on July 17 to 23, 2019
- (2) SVI-R43-T1317, Diesel Generator Start and Load Division 1 on August 19 to 23, 2019
- (3) PTI-N32-P0001, Turbine Overspeed Protection Devices Trip and Turbine Lube Oil Pump Starts Weekly Test on August 26 to 29, 2019
Inservice Testing (IP Section 03.01) (1 Sample)
- (1) SVI-E22-T2001, High Pressure Core Spray Pump and Valve operability test on August 8 to 13, 2019
FLEX Testing (IP Section 03.02) (1 Sample)
- (1) Flex lake water pump A and B annual functional test on August 26 to 29, 2019
71114.06 - Drill Evaluation
Drill/Training Evolution Observation (IP Section 03.02) (1 Sample)
- (1) a simulator-based licensed operator requalification training on July 22,
RADIATION SAFETY
71124.06 - Radioactive Gaseous and Liquid Effluent Treatment
Walk Downs and Observations (IP Section 02.01) (1 Sample)
The inspectors walked down the following gaseous and liquid radioactive effluent monitoring and filtered ventilation systems to assess the material condition and verify proper alignment according to plant design:
(1)
- Unit 1 Plant Vent
- Unit 2 Plant Vent
- Off-Gas Vent
- Turbine Building/Heater Bay Vent
Calibration and Testing Program (Process & Effluent Monitors) (IP Section 02.02) (1 Sample)
The inspectors reviewed the following gaseous and liquid effluent monitor instrument calibrations and tests:
(1)
- Unit 1 Vent Noble Gas Monitor 1D17-K786
- Off-Gas Vent Noble Gas Monitor 1D17-K836
- LRW to ESW Radiation Monitor D17-K606
- Liquid Radwaste High Flow G50-N445
Sampling and Analysis (IP Section 02.03) (1 Sample)
- (1) Radioactive Effluent Sampling and Analysis Activities
- Unit 1 Plant Vent, July 25, 2019
- Unit 2 Plant Vent, July 25, 2019
- Off-Gas Vent, July 25, 2019
- Turbine Building/Heater Bay Vent, July 25, 2019
Effluent Discharges
- None were available for review during this inspection
Instrumentation and Equipment (IP Section 02.04) (1 Sample)
The inspectors reviewed the following radioactive effluent discharge system surveillance test results:
- (1) Work Order 200767889, Annulus Exhaust Gas Treatment System Train A Flow and Filter Operability Test
Dose Calculations (IP Section 02.05) (1 Sample)
- (1) Liquid and Gaseous Discharge Permits
- SVI-D175268 Effluent Sampling/Analysis & Dose Assessment, March 2018
- SVI-D175268 Effluent Sampling/Analysis & Dose Assessment, July 2018
- SVI-D175268 Effluent Sampling/Analysis & Dose Assessment, February 2019
The inspectors also reviewed the following land use census
- August 2018
Abnormal Gaseous or Liquid Tank Discharges
- None were available for review during this inspection
71124.07 - Radiological Environmental Monitoring Program
Site Inspection (IP Section 02.01) (1 Sample)
- (1) The inspectors reviewed the following:
Walkdowns, Calibrations, and Maintenance Record Review
- Environmental air sampling station; location number 1
- Environmental air sampling station; location number 3
- Environmental air sampling station; location number 6
- Environmental air sampling station; location number 7
- Environmental dosimeter location number 1
- Environmental dosimeter location number 3
- Environmental dosimeter location number 6
- Environmental dosimeter location number 10
Environmental Sample Collections and Preparation Observation
- Environmental air sampling station; location number 6
- Environmental air sampling station; location number 7
- Surface water sample; location number 59
- Municipal water sample; location number 39 Licensee Actions in Response to Missed Sample, Inoperable Sampler, Lost TLD or Anomalous Measurement
- Environmental air sampling station location number 35; loss of power; 07/17/2019
- Broadleaf vegetation sampling location number 70; missed sample; 07/2019
Sampling Program for the Potential of Licensed Material Entering Groundwater
[List 3-5 structures]
[List 3-5 structures]
Groundwater Protection Initiative (GPI) Implementation (IP Section 02.02) (1 Sample)
- (1) The inspectors evaluated the licensees voluntary groundwater protection initiative.
OTHER ACTIVITIES - BASELINE
71151 - Performance Indicator Verification MS08: Heat Removal Systems (IP Section 02.07)
- (1) Unit 1 (July 1, 2018 - June 30, 2019)
MS09: Residual Heat Removal Systems (IP Section 02.08) (1 Sample)
- (1) Unit 1 (July 1, 2018 - June 30, 2019)
MS10: Cooling Water Support Systems (IP Section 02.09) (1 Sample)
- (1) Unit 1 (July 1, 2018 - June 30, 2019)
PR01: Radiological Effluent Technical Specifications/Offsite Dose Calculation Manual Radiological Effluent Occurrences (RETS/ODCM) Radiological Effluent Occurrences Sample (IP Section 02.16) (1 Sample)
- (1) October 2018 - June 2019
71152 - Problem Identification and Resolution
Annual Follow-up of Selected Issues (IP Section 02.03) (2 Samples)
- (1) CR 2019-01671; Failure to perform adequate preventive maintenance on the main generator exciter brushes, led to brush failures, loss of generator field excitation, generator lockout, and reactor scram
- (2) CR 2019-04667; Emergency Service Water A inoperability due to control power fuses being improperly installed
71153 - Follow-up of Events and Notices of Enforcement Discretion Event Report (IP Section 03.02)
- (1) LER 05000440/2019-001-00, Loss of Generator Excitation Causes Turbine Trip and Reactor Protection System Actuation (ADAMS accession: ML19114A443). Details regarding the disposition of this LER are documented in the Results section.
Personnel Performance (IP Section 03.03) (1 Sample)
- (1) On July 27, 2019, at 7:29 p.m., a turbine trip and subsequent reactor scram occurred during the performance of turbine mechanical overspeed testing. The inspectors evaluated the unplanned event and associated licensee performance.
INSPECTION RESULTS
Failure to Properly Seat Essential Service Water A Pump Control Power Fuses results in Loss of Safety Function Cornerstone Significance Cross-Cutting Aspect Report Section Mitigating Systems
Green NCV 05000440/2019003-01 Open
[H.9] - Training 71152 A self-revealed finding of very low safety significance (Green) and associated non-cited violation of 10 CFR Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, for the licensee's failure to accomplish work activities in accordance with the instructions, procedures, or drawings. Specifically, the licensee failed to properly install the A essential service water control power fuses in accordance with SOI-R22, Metal Clad Switchgear 5-15 kV, paragraph 2.3 such that the fuses were able to back out and cause an unplanned inoperability of several safety related systems during routine breaker operation.
Description:
On May 24, 2019, after the start of the emergency service water (ESW) pump A for pump packing adjustment, two unexpected alarms annunciated in the control room indicating residual heat removal (RHR) A and Division 1 diesel generator were out of service. The operators walked down panels and noted that panel 1H13P0601 indications were illuminated for ESW Pump A Breaker Out / Power Loss / Overload. Upon investigation, the licensee discovered that the closing control power fuses (AX fuses) for the ESW A breaker were not seated properly. This resulted in the inoperability of the ESW A system and the following systems: low pressure core spray (LPCS) room cooler and LPCS, A RHR, A emergency closed cooling (ECC), and division 1 emergency diesel generator. The inspectors reviewed plant narrative logs and determined that the licensee last manipulated the AX fuses on April 25, 2019. The licensee made an eight-hour non-emergency notification to the NRC as required by 10 CFR 50.72(b)(3)(v)(D) Accident Mitigation for the inoperability of a single train system, LPCS, in Modes 1, 2, and 3.
Upon further review, the inspectors determined that the licensee had failed to do an adequate past operability review. Specifically, the licensee had determined that a non-licensed operator had not properly installed the AX fuses on April 25, 2019, but did not consider seismic implications while the fuses were improperly installed. From the time the licensee installed fuses on April 25, 2019, until the alarms annunciated in the control room on May 25, 2019, plant operators started the ESW A pump four times. It is reasonable to assume that the fuses were not completely installed and backed out because of the vibrations caused by breaker operations, both closing and opening. At the time of this inspection report issuance, the licensee continued to assess the seismic concerns raised by the inspectors.
Corrective Actions: Fuse block AX, for the ESW pump A breaker, was re-seated by the initial responder. Further investigation by the licensee determined that when properly and fully seated, the AX fuse block had no issues relative to staying in place with adequate tightness. The licensee is also pursuing a change to the Perry Switchgear and Electrical Operator Safety and Administration lesson plans to include a maintenance fundamental discussion of seating fuses and fuse blocks with physical and visual cues. The corrective actions included, but were not limited to, the importance of fully seated fuses within their receptors, ways to identify fuses and fuse block receptors which may be loose, the need to identify the issue in the work management / corrective action processes, and non-licensed operator training for seating fuses and fuse blocks, to include physical or visual cues, during a non-licensed operator continuing training cycle.
Corrective Action References: CR 2019-04667
Performance Assessment:
Performance Deficiency: The licensee's failure to appropriately install fuses such that they maintained their seismic qualification and would not back out as a result of routine breaker operation or during a seismic event was a performance deficiency.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the Equipment Performance attribute of the Mitigating Systems cornerstone and adversely affected the cornerstone objective to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences. Specifically, the failure to ensure that the ESW pump A control power fuses were properly installed negatively impacted the licensee's ability to ensure the availability, reliability, and capability of systems that respond to initiating events to prevent undesirable consequences (i.e., core damage).
Significance: The inspectors assessed the significance of the finding using Appendix A, The Significance Determination Process (SDP) for Findings At-Power. Under Exhibit 2 - Mitigating Systems Screening Questions, the inspectors answered Yes to Question #2, Does the finding represent a loss of system and/or function? The inspectors continued the significance determination with a detailed risk evaluation.
A Senior Reactor Analyst performed a detailed risk evaluation using the Standardized Plant Analysis Risk Model (SPAR) version 8.60 for Perry. The ESW pump control power fuses were last manipulated approximately 30 days prior to the alarm indications. During the 30-day period, the pump ran successfully several times. The SRA assumed that only seismic events would challenge the operation of the pump in the degraded condition. Therefore, only seismic initiating events were evaluated for a change in risk. The ESW pump A was modeled as a failure to start for an exposure period of 30 days.
The result was a change in core damage frequency less than 1E-7/yr., which is a finding of very low safety significance. The dominant core damage sequence was a seismic-induced loss of offsite power, failure of both ESW pumps, and the failure to recover power.
Cross-Cutting Aspect: H.9 - Training: The organization provides training and ensures knowledge transfer to maintain a knowledgeable, technically competent workforce and instill nuclear safety values. Specifically, the licensee failed to provide appropriate training to non-licensed operators, instead trusting in skill of the craft, to ensure fuses had been installed correctly and identify conditions where they were not on April 25, 2019.
Enforcement:
Violation: Title 10 of the Code of Federal Regulations, Part 50, Appendix B, Criterion V, Instructions, Procedures, and Drawings, requires, in part, activities affecting quality shall be prescribed by documented instructions, procedures, or drawings, of a type appropriate to the circumstances and shall be accomplished in accordance with these instructions, procedures, or drawings.
Contrary to the above on April 25, 2019, the licensee failed to accomplish activities affecting quality in accordance with instructions, procedures, and drawings. Specifically, the licensee failed to install the A ESW pump breaker AX control power fuses in accordance with instruction SOI-R22, Metal Clad Switchgear 5-15 kV, paragraph 2.3, such that the fuses backed out and caused an unplanned inoperability of several safety-related systems on May 24, 2019.
Enforcement Action: This violation is being treated as an non-cited violation, consistent with Section 2.3.2 of the Enforcement Policy.
Observation: Failure to Perform Adequate Preventive Maintenance on the Main Generator Exciter Brushes Results in Generator Lockout and Reactor Scram.
71152 The inspectors performed a detailed review of CR 2019-01671, Main Generator/Main Turbine Trip Resulting in a Reactor Scram. The licensee's casual evaluation for this issue determined that the apparent cause of the generator lockout and reactor scram was that the preventive maintenance on the generator field excitation brushes was less than adequate.
Specifically, the licensee stated that the criteria for brush inspection and replacement should have added margin to ensure brushes could not wear to the point where brush spring tension could not be maintained between inspection intervals. The inspectors determined that the vendor manual recommended that the exciter brush inspection be performed on a weekly basis and that acceptance criteria for that inspection for exciter brush length be at least one-eighth of one inch. The licensee's maintenance plan called for a monthly inspection of the the exciter brushes with at least one-eighth of one inch acceptance criteria. The inspectors determined that the longer inspection frequency in conjunction with the acceptance criteria resulted in the brushes degrading between inspection intervals such that spring tension was no longer maintained and allowed the brushes to chatter which ultimately led to the flashover event. The licensee's corrective actions altered the acceptance criteria and provided extra margin to ensure that the monthly inspection periodicity would not result in the brush spring pressure reduction such that the brushes would not chatter.
The inspectors also reviewed past operating experience and determined that there have been numerous flashover events in the industry over the past thirty-five years, with most caused by exciter brushes wearing down to the point that brush spring tension was not maintained.
Four of these events were very similar to the licensee's event in February 2019. In 1985 a nuclear operating plant experienced a flashover event and determined that the root cause was the loss of field excitation due to excessive brush wear. The brushes had been inspected seventeen days before the failure and reported as acceptable. As a result of that event exciter brush inspection was changed from once per month to every fourteen days. In 2004 another nuclear operating plant had a flashover event. The root causes of the event were the Generator/Exciter vendor recommended inspections were not implemented when the GE generator was installed in 1986 and the lessons learned from a similar Alterrex brush failure in 1993 were not applied. The fourth event occurred in November 2005 at another nuclear power plant, the cause being the exciter brushes being worn down below the point where exciter brush tension could not be maintained. These four events were opportunities for the licensee to evaluate its preventive maintenance plans for the main generator exciter brushes.
Following the event on February 25, 2019, the licensee polled the industry and found that the four plants that responded to its inquiry all inspect their main generator exciter brushes either bi-weekly or weekly. Also, only one plant used the same or similar brush replacement criteria. The other three plants reported that they had a more conservative recommendation for replacing brushes. According to the licensee, the results of the industry information shows that the past replacement criteria was less than adequate and needed to be changed.
Observation: Failure to Properly Seat Essential Service Water A Pump Control Power Fuses results in Loss of Safety Function 71152 The inspectors performed a detailed review of CR 2019-04667, Indications received for ESW Pump A Breaker Power Loss. The event caused the inoperability of a train of safety-related equipment, the loss of safety function for the Low Pressure Core Spray System, a condition prohibited by the plant's technical specifications, and caused the licensee to submit an eight-hour non-emergency Reactor Plant Event Notification Report in accordance with 10 CFR 50.72(b)(3)(v). The licensee identified that the cause of the loss of control power was due to a non-licensed operator (NLO) not recognizing that the control power fuses for emergency service water pump breaker A were not fully seated following installation.
Corrective actions implemented by the licensee included, but were not limited to, changes to switchgear and electrical operator safety training lesson plans and a discussion of the event during non-licensed operator training. The inspectors noted that the licensee's procedures used by the NLO, specifically SOI-R22, Metal Clad Switchgear 5-15 KV and SOI-P45/P49, Emergency Service Water and Screen Wash Systems, did not appear to have acceptance criteria to ensure fuses are fully seated. The acceptance criteria in the procedures was a continuity test to ensure fuse continuity. The inspectors determined there did not exist criteria to ensure proper seating of the fuse holder/clip. Through discussion with the licensee, the inspectors determined that the NLOs received initial training in their switchgear and electrical operator safety training classes on how to properly install and remove fuse clips/holders but relied upon skill of the craft thereafter. The inspectors further determined, through conversation, that NLOs are not retrained on that skill of the craft activity. The inspectors determined that the current acceptance criteria of proper fuse installation through fuse continuity checks did not reveal, and likely would not reveal, incorrect fuse installation such as that occurred on April 25, 2019, which resulted in an unplanned inoperability of several safety-related systems.
LER 2019-001-00 for Perry Nuclear Power Plant, Loss of Generator Excitation Causes Turbine Trip and Reactor Protection System Actuation.
Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events
Green FIN 05000440/2019003-02 Open
None (NPP)71153 A self-revealed finding of very low safety significance (Green) was identified for the licensee's failure to perform adequate preventive maintenance on the main generator exciter brushes, which led to brush failures, loss of generator field excitation, generator lockout, and reactor scram. No violation of regulatory requirements was identified.
Description:
On February 25, 2019, at 12:24 a.m., the Main Generator received a lockout signal from the Loss of Excitation relay following its associated time delay relay actuation. In the six seconds prior to the lockout signal, there was a decrease in grid voltage from approximately 346kV to approximately 311kV. During that same six second time period Main Generator Mega Volt Amps Reactive (MVARs) increased significantly. The equipment responded as expected for a valid loss of excitation event because once the field current stops, the generator acts as an induction motor instead of a synchronous generator and draws MVARs from the grid to support its magnetic field. The flow of MVARs in turn pulled down local grid voltage until the generator tripped, then voltage recovered. There were numerous annunciators and equipment indications following the scram due to the voltage perturbation and expected bus transfers.
Investigation into the cause of the generator trip found evidence of collector ring flashover for the inboard (positive polarity) exciter collector ring. This was confirmed as the cause of the generator loss of excitation event following visual inspection of the generator exciter. Vendor manual documentation and industry operating experience support that conclusion and mention collector ring flashover as the most prevalent cause of loss of excitation events within the industry. The generator brushes and collector rings are a vital part of the ability of the generator to work as a synchronous machine to convert mechanical, rotating energy into electrical energy. The collector rings, or slip rings, are metal rings heat shrunk onto the end of the generator rotor shaft that are maintained electrically isolated from the shaft and each other. They provide a contact surface for the generator brushes to ride upon as the shaft rotates to deliver the field current developed by the generator voltage regulator.
The slip rings are highly polished steel forgings with spiral grooves machined in the outer surface. The grooves allow the field current to even out across a few of the ring surfaces and allow a space for air flow and cooling. Each exciter brush is offset by approximately 1/4 inch.
Therefore, not all brushes ride on the same ring surface and all the ring surfaces can wear more evenly. The progression of this event is that one or more of the exciter inboard brushes reached its wear limit. The brush spring(s) was out of range to ensure positive, adequate downward pressure to the brush(es) to assure proper electrical connection was maintained.
When the brush spring(s) pressure was limited, the brush(es) began to bounce/chatter. Each time a brush fails to make adequate connection, the current it is carrying is broken. As the radial distance between the brush and collector ring surface initiates the current is strong enough to cross over the minimal air gap; this creates an arc.
An arc can impose physical damage and create surface defects in the form of pits (peaks and valleys) that act as increased friction to increase the wear rate of other brushes that may ride on the same ring surface. The longer this bouncing and subsequent arcing continues conditions for impaired electrical connection degrade in an exponential manner. Arcing also creates an environment for increased heating of the collector ring and brushes as high current discharges repeatedly occur. Once the radial distance between the brush and collector ring surface increases enough, the air gap becomes great enough that the arc does not connect the brush and collector ring, but finds the path of least resistance, some external source to ground. This is what is referred to as a flashover event.
Licensee Event Report (LER) 05000440/2019-001-00, Loss of Generator Excitation Causes Turbine Trip and Reactor Protection System Actuation, was submitted by the licensee for this event.
Corrective Actions: The licensee revised the preventive maintenance plan to modify the brush replacement criteria to add margin to account for the monthly inspection frequency.
The licensee also added detail to the brush vibration preventive maintenance task to ensure all exciter brushes are monitored during routine task performances.
Corrective Action References: CR 2019-01671
Performance Assessment:
Performance Deficiency: The inspectors determined that the inadequate preventative maintenance on the main generator exiter brushes, which resulted in a reactor scram, was a performance deficiency which was within the licnesee's ability to foresee and prevent.
Specifically, licensee procedure NOP-WM-1001, Order Planning Process, paragraph 4.3.4.6.c, requires the licensee to include appropriate acceptance criteria. However, on February 4, 2019, the monthly brush inspection failed to detect and replace the main generator's exciter brushes prior to having the brushes degrade below the minimum acceptable length, resulting in the loss of brush spring pressure, exciter arching, loss of the generator and reactor scram.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the Equipment Performance attribute of the Initiating Events cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the inadequate preventative maintenance of the generator exciter brushes impacted the Initiating Events cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations.
Significance: The inspectors assessed the significance of the finding using Appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined that this finding was a licensee performance deficiency of very low safety significance (Green) because although the finding did cause a reactor trip (scram), it did not cause the loss of mitigation equipment relied upon to transition the plant from the onset of the trip to a stable shutdown condition.
Cross-Cutting Aspect: Not Present Performance. No cross cutting aspect was assigned to this finding because the inspectors determined the finding did not reflect present licensee performance.
Enforcement:
Inspectors did not identify a violation of regulatory requirements associated with this finding.
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
- On September 30, 2019, the inspectors presented the integrated inspection results to Mr. F. Payne, Site Vice President, and other members of the licensee staff.
- On July 26, 2019, the inspectors presented the Radiation Protection Baseline Inspection inspection results to Mr. F. Payne; Site Vice President, and other members of the licensee staff.
- On October 9, 2019, the inspectors presented the Integrated Baseline Inspection inspection results to Mr. F. Payne, Site Vice President, and other members of the licensee staff.
DOCUMENTS REVIEWED
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
71111.04Q Drawings
2-0351-00000
Standby Diesel Generator Starting Air
2-0352-00000
Standby Diesel Generator Fuel Oil System
2-0353-00000
Standby Diesel Generator Lube Oil
T
2-0354-00000
Standby Diesel Generator Jacket Water
V
Procedures
VLI-M23/24
MCC, Switchgear and Miscellaneous Electrical Equipment
Area HVAC System
VLI-R44
Division 1 and 2 Diesel Generator Starting Air System
VLI-R45
Division 1 and 2 Diesel Generator Fuel Oil System (Unit 1)
VLI-R46
Division 1 and 2 Diesel Generator Jacket Water Systems
(Unit 1)
VLI-R47
Division 1 and 2 Diesel Generator Lube Oil
71111.05Q Procedures
FPI-0FH
Fuel Handling Building
FPI-0IB
Intermediate Building
FPI-1AB
Auxiliary Building Unit 1
FPI-SB
Service Building
Corrective Action
Documents
CR 2019-05701
Unplanned Entry into EOP-03 Secondary Containment
Control
07/03/2019
Drawings
2-0740-00000
Liquid Radwaste Sumps System
71111.11Q Miscellaneous
OTLC-
3058201903_PY-
SGC1
OTLC-3058201903_PY-SGC1 - Evaluated Simulator
Scenario
Procedures
Evolution Specific
Reactivity Plan
Cycle 18 - Startup 130 - Part 1
Evolution Specific
Reactivity Plan
Cycle 18 - Startup 130 - Part 2
IOI-8
Shutdown by Manual Reactor Scram
ONI-C71-1
Reactor Scram
Miscellaneous
ATL-2019-0605-
ATA-030
Perry System Health Report
2018
Work Orders
200556421
Replace Agastat E7000 TD Relay
08/15/2019
200753231
Generator Brush Monthly Inspection
Aug 8 - 19,
2019
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Corrective Action
Documents
CR 2019-06321
Turbine Trip during Weekly Turbine Overspeed Test
07/27/2019
CR 2019-06720
Drywell Identified Leakage Rose 0.1 GPM
08/10/2019
CR 2019-06983
Parameters Indicate a Small Leak in Drywell
08/20/2019
Miscellaneous
601236050
PY-1E31 Leak Detection System
Procedures
NOP-OP-1005
Shutdown Defense In Depth
NOP-OP-1007
Risk Management
PAP-1924
Risk-Informed Safety Assessment and Risk Management
PAP-1925
Shutdown Defense In Depth Assessment and Management
Corrective Action
Documents
CR 2019-05162
FLEX Equipment Remains Unavailable for Maintenance and
Allowed Out of Service Time, per FLEX Specifications 2,1,2,
2.1.4, and 2.1.8, will be Exceeded
06/12/2019
CR 2019-05928
Coolant Leak from Diesel Fire Pump when Starting Engine -
Unplanned Impairment
07/12/2019
CR 2019-05934
Diesel Driven Fire Pump Belt Loose and Coolant Leak
07/13/2019
CR 2019-06928
FLEX Lake Water Pump #2 Main Panel Breaker Failed to
Operate
08/08/2019
CR 2019-06981
Failure of Div 1 DG Lube Oil Control Power Fuse and 42
Contact Following Monthly SVI Run
08/20/2019
CR 2019-07260
Blown Mainline Fuse for Nuclear Closed Cooling Drywell
Isolation Valve
08/30/2019
CR 2019-07639
Pressure Switch 1C71N0005A Failed to Actuate During the
Performance of SVI-N31-T1151 Step 5.1.4.44 in
Accordance with Order Number 200625793
09/15/2019
Drawings
4549-43-036-9
Turbine Control Diagram
9A
B-208-151
Turbine Control (EHC) Control Valve Logic
A
Miscellaneous
eSOMS Plant
Narrative Logs
Plant Narrative Logs
08/20/2019
eSOMS Plant
Narrative Logs
Day and Evening Shift
July 10 and
2, 2019
Procedures
FPI-A-T03
Champion Fire Pump Operator Instructions
FSG-40.5
FLEX Generator Loading and Plant System Operations
NOP-LP-7300
FLEX Program for the Perry Nuclear Power Plant
NOP-OP-1009
Operability Determinations and Functionality Assessments
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
PAP-1910
PDB-R0003
Flex Specifications
PTI-P54-P0041
Semiannual Fire Door Inspection
SVI-N31-T1151
Main Turbine Valve Exercise Test
09/16/2019
Work Orders
200625793
PY-SVI-N31T1151
09/15/2019
Pressure Switch 1C71N0005A Failed to Actuate During the
Performance of SVI-N31-T1151
Procedures
SVI-E31-T0078-D
MSL Tunnel Temperature High Channel D Functional for
1E-31-N604D and 1E31-N605D
08/27/2010
Work Orders
200417053
Replacement of Scan Control Card, Vector Receiver Card,
and Several Addition Circuit Cards
07/15/2019
200556421
Replace Agastat E7000 TD Relay
05/13/2019
200790224
Replace Modicon I/O card 'GND'/CR
07/23/2019
200795274
Diesel Fire Service Pump, Replace Jacket Water Heat
Exchanger Relief Valve and Alternator
07/18/2019
200795656
Replace Mechanical Seal if Required Based on Current
Troubleshooting being Performed on Feedwater Booster
Pump B
09/12/2019
200798951
Reactor Feedwater Booster Pump B Motor Determ and Seal
Replacement
09/22/2019
200799218
Flex Lake Water Pump
08/19/2019
Corrective Action
Documents
CR 2019-06321
Turbine Trip during Weekly Turbine Overspeed Test
07/27/2019
Procedures
IOI-0004
Shutdown
IOI-0005
Maintaining Hot Shutdown
ONI-C71-1
Reactor Scram
Corrective Action
Documents
2019-06321
Prompt Functionality Assessment Form
08/14/2019
Miscellaneous
DI-236
Calculation - Perry Low Pressure Turbine Missile Probability
Analysis
05/02/2013
DI-236
Calculation Addendum - Perry Low Pressure Turbine Missile
Probability Analysis
Procedures
200798399
Turbine Overspeed Protection Devices Trip and Turbine
Lube Oil Pump Starts Test
08/06/2019
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
SOI-R43
Division 1 and 2 Diesel Generator System
SVI-R43-T1317
Diesel Generator Start and Load Division 1
Work Orders
200711841
Flex Water Pump Functional Testing
08/28/2019
200729023
SVI-E22-T2001, High Pressure Core Spray Pump and Valve
Operability Test
Aug 8 - 13,
2019
200729376
SVI-E31-T5311-B, (92D) Turbine Building Main Steam Line
Temperature High Channel Functional for 1E31-N361B and
07/17/2019
Miscellaneous
OTLC-
3058201903_PY-
SGC1
OTLC-3058201903_PY-SGC1 - Evaluated Simulator
Scenario
PNPP10481
Emergency Notification Cover Sheet
Rev
01/15/2018
PNPP7794
Initial Notification Form
Rev
01/09/2018
PNPP9100
Alert Notification Message
Rev
04/24/2019
Corrective Action
Documents
SA-BN-2019-
1476
2019 RETS/REMP Self Assessment in Preparation for the
NRC Audit
06/04/2019
Corrective Action
Documents
Resulting from
Inspection
2019-06218
2019 NRC Effluent Inspection Noble Gas Observation
07/24/2019
ATA-2019-10132
2019 Effluents NRC Followup Action for D17 Flowrate and
Total Volume
08/09/2019
Miscellaneous
Offsite Dose Calculation Manual
2018 Annual Environmental and Effluent Release Report
04/29/2019
2017 Annual Environmental and Effluent Release Report
05/01/2018
Results of Radiochemistry Cross Check Program
3rd Quarter
2017 through
1st Quarter
2019
Procedures
ICI-C-D17-20
System Calibration Instruction
Offsite Dose Calculation Manual
SOI-D17
Airborne Radiation Monitoring System (Effluents)
SOI-D17A
Process Radiation Monitoring System
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
SVI-D17-T5268
Effluent Sampling/Analysis & Dose Assessment
Self-Assessments SA-BN-2019-
1476
2019 RETS/REMP self assessment for the NRC audit
06/04/2019
Work Orders
2001703018
PY-SVI-D17T5268 Effluent Sampling/Analysis & Dose
Assessment
03/07/2018
2001753748
PY-SVI-D17T5268 Effluent Sampling/Analysis & Dose
Assessment
04/08/2019
20017753746
PY-SVI-D17T5268 Effluent Sampling/Analysis & Dose
Assessment
2/04/2019
200648169
Unit 1 Vent Noble Gas Radiation Monitor Calibration for
05/18/2017
200671740
LRW to ESW Radiation Monitor Channel Calibration for
D17-K606
2/14/2018
200693314
Liquid Radwaste High Flow Calibration for G50-N445
08/29/2018
200693360
Off-Gas Vent Noble Gas Radiation Monitor Calibration for
09/18/2018
200695479
SVI-M15-T2140-A Annulus Exhaust Gas Treatment System
Train A Flow and Filter Operability Test
03/30/2018
200703022
PY-SVI-D17T5268 Effluent Sampling/Analysis & Dose
Assessment
07/05/2018
200717059
Unit 1 Vent Noble Gas Radiation Monitor Calibration for
05/15/2019
200767889
SVI-M15-T2140-A Annulus Exhaust Gas Treatment System
Train A Flow and Filter Operability Test
09/16/2018
2007769871
Unit 1 Vent Noble Gas Radiation Monitor Calibration for
10/18/2018
Calibration
Records
Environmental Sample Pump 7004 Calibration
04/15/2019
Environmental Sample Pump 09 Calibration
04/15/2019
Environmental Sample Pump 7004 Calibration
05/02/2018
SVI-D51-T0307-A
Meteorological Monitoring System A, 60 Meter Wind Speed
Calibration
11/30/2018
SVI-D51-T0308-A
Meteorological Monitoring System A 60 Meter Wind
Direction Calibration
05/07/2019
SVI-D51-T0308-B
Meteorological Monitoring B Channel 60 Meter Wind
05/19/2019
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Direction Calibration
Corrective Action
Documents
2018-11953
New XOQ/DOX Evaluation
05/22/2018
Corrective Action
Documents
Resulting from
Inspection
2019-06219
2019 NRC Effluent Inspection ODCM Comment
07/24/2019
2019-06256
Delays Performing, Analyzing and Incorporating the Effluent
Dispersion Factors (XOQ/DOQ)
07/25/2019
Engineering
Evaluations
2018 Annual Report of the Meteorological Program
2/05/2019
Comparison of 10-Year (2006-2016) Versus Historical 7 Site
Year X/Qs and D/Qs from the Perry Nuclear Power Station
01/01/2018
Five-Year Update of Groundwater Flow Characteristics
Report
05/04/2016
Procedures
REMP-0006
Fish Sampling Instruction
REMP-0009
Surface and Drinking Water Sampling
REMP-0012
Food Product Sampling
REMP-0013
Sampling Locations
71151
Miscellaneous
NOBP-LP-4012-
Mitigating Systems Performance Index (MSPI) Unavailability
Index (UAI) & Unreliability Index (URI) for Heat Removal
System (RCIC)
07/01/2018 -
06/30/2019
NOBP-LP-4012-
Mitigating Systems Performance Index (MSPI) Unavailability
Index (UAI) & Unreliability Index (URI) for Residual Heat
Removal (RHR)
07/01/2018 -
06/30/2019
NOBP-LP-4012-
RETS/ODCM Radiological Effluent Occurrence
4th Quarter
2018 through
2nd Quarter
2019
NOBP-LP-4012-
Mitigating Systems Performance Index (MSPI) Unavailability
Index (UAI) & Unreliability Index (URI) for Emergency
07/01/2018 -
06/30/2019
Corrective Action
Documents
CR 2019-01671
Main Generator/Main Turbine Trip Resulting in a Reactor
2/25/2019
CR 2019-04667
Indications Received for ESW Pump A Breaker Power Loss
05/24/2019
Procedures
NOBP-ER-3009
Plant Health Reporting
NOP-SS-8001
FENOC Activity Tracking
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
NOP-WM-1001
Order Planning Process
NOP-WM-3001
Work Management PM Process
SOI-P45/P49
Emergency Service Water and Screen Wash Systems
SOI-R22
Metal Clad Switchgear 5-15 KV
Work Orders
200753223
Inspect Main Generator & Alterex Brush
2/12/2018
200753225
Inspect Main Generator & Alterex Brush
2/04/2019
Corrective Action
Documents
CR 2019-01671
Main Generator/Main Turbine Trip Resulting in a Reactor
2/25/2019
Miscellaneous
Scram No. 1-19-
Post Scram Restart Report Perry Nuclear Power Plant
07/30/2019
Procedures
PYBP-SITE-0019
Post Reactor Scram Evaluation
4