IR 05000528/2020002
| ML20212M129 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 07/31/2020 |
| From: | John Dixon NRC/RGN-IV/DRP/RPB-D |
| To: | Lacal M Arizona Public Service Co |
| References | |
| EA-20-075 IR 2020002 | |
| Download: ML20212M129 (32) | |
Text
July 31, 2020
SUBJECT:
PALO VERDE NUCLEAR GENERATING STATION UNITS 1, 2, AND 3 -
INTEGRATED INSPECTION REPORT 05000528/2020002, 05000529/2020002, AND 05000530/2020002
Dear Mrs. Lacal:
On June 30, 2020, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at Palo Verde Nuclear Generating Station Units 1, 2, and 3. On July 7, 2020, the NRC inspectors discussed the results of this inspection with you and other members of your staff. The results of this inspection are documented in the enclosed report.
Two findings of very low safety significance (Green) are documented in this report. None of these findings involved a violation of NRC requirements.
If you disagree with a cross-cutting aspect assignment or a finding not associated with a regulatory requirement in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 205550001; with copies to the Regional Administrator, Region IV; and the NRC Resident Inspector at Palo Verde Nuclear Generating Station Units 1, 2, and 3. This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding.
Sincerely, John L. Digitally signed by John L. Dixon Dixon 07:36:43 -05'00'
Date: 2020.07.31 John L. Dixon, Jr., Chief Reactor Projects Branch D Division of Reactor Projects Docket Nos. 05000528, 05000529, and 05000530 License Nos. NPF-41, NPF-51, and NPF-74
Enclosure:
Inspection Report 05000528/2020002, 05000529/2020002, and 05000530/2020002 w/Attachment: Detailed Risk Assessment
Inspection Report
Docket Numbers: 05000528, 05000529, and 05000530 License Numbers: NPF-41, NPF-51, and NPF-74 Report Numbers: 05000528/2020002, 05000529/2020002 and 05000530/2020002 Enterprise Identifier: I-2020-002-0001 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station Units 1, 2, and 3 Location: Tonopah, AZ Inspection Dates: April 1, 2020 to June 30, 2020 Inspectors: C. Peabody, Senior Resident Inspector R. Bywater, Resident Inspector D. You, Resident Inspector A. Athar, Resident Inspector I. Anchondo-Lopez, Reactor Inspector W. Cullum, Reactor Inspector J. Drake, Senior Reactor Inspector F. Thomas, Reactor Inspector Approved By: John L. Dixon, Jr., Chief Reactor Projects Branch D Division of Reactor Projects Enclosure
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting an integrated inspection at Palo Verde Nuclear Generating Station Units 1, 2, and 3, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Incorrect Installation of Unit 3 Reactor Coolant Pump 1B Seal Assembly Cornerstone Significance Cross-Cutting Report Aspect Section Initiating Events Green [H.6] - Design 71152 FIN 05000530/2020002-01 Margins Open/Closed A self-revealed Green finding was identified when the licensee failed to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement,
Revision 41. Specifically, when installing the Unit 3 1B reactor coolant pump seal package during the fall 2019 refueling outage, the stage 2 to stage 3 drive pins were not installed properly causing the seal to leak and ultimately requiring a forced outage to replace the seal on February 8, 2020.
Failure to Adequately Evaluate or Test the Impact of a Design Modification to the Unit 2 Main Feedwater Lube Oil Control Panel Protective Relay System Cornerstone Significance Cross-Cutting Report Aspect Section Initiating Events Green None 71153 FIN 05000529/2020002-02 Open/Closed EA-20-075 A self-revealed Green finding was identified when the licensee failed to follow Procedure 81DP-0EE10, Design Change Process, Revision 30. Specifically, the licensee failed to adequately evaluate or test the impact of a design modification to the Unit 2 main feedwater lube oil control panel protective relay system, which resulted in a spurious loss of both main feedwater pumps and subsequent reactor trip on March 3, 2020.
Additional Tracking Items
Type Issue Number Title Report Section Status LER 05000529/2020-001-00 LER 2020-001-00 for Palo 71153 Closed Verde Nuclear Generating Station (PVNGS), Unit 2,
Reactor Trip on Low Steam Generator Level Signal
PLANT STATUS
Unit 1 operated at or near full power for the duration of the inspection period.
Unit 2 entered the inspection period at full power. On April 4, 2020, Unit 2 shut down for a planned refueling outage. Unit 2 was restarted on May 4, 2020, and operated at or near full power for the remainder of the inspection period.
Unit 3 operated at or near full power for the duration of the inspection period.
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
Starting on March 20, 2020, in response to the National Emergency declared by the President of the United States on the public health risks of the coronavirus (COVID-19), resident inspectors were directed to begin telework and to remotely access licensee information using available technology. During this time the resident inspectors performed periodic site visits each week and during that time conducted plant status activities as described in IMC 2515, Appendix D; observed risk significant activities; and completed on site portions of IPs. In addition, resident and regional baseline inspections were evaluated to determine if all or portion of the objectives and requirements stated in the IP could be performed remotely. If the inspections could be performed remotely, they were conducted per the applicable IP. In some cases, portions of an IP were completed remotely and on site. The inspections documented below met the objectives and requirements for completion of the IP.
REACTOR SAFETY
71111.01 - Adverse Weather Protection
Seasonal Extreme Weather Sample (IP Section 03.01) (2 Samples)
- (1) The inspectors evaluated summer readiness of offsite and alternating current (AC)power systems on May 8, 2020.
- (2) The inspectors evaluated readiness for seasonal extreme weather conditions prior to the onset of the monsoon season on May 14, 2020.
71111.04 - Equipment Alignment
Partial Walkdown Sample (IP Section 03.01) (3 Samples)
The inspectors evaluated system configurations during partial walkdowns of the following systems/trains:
- (1) Unit 2 auxiliary feedwater system A following refueling outage testing on May 4, 2020
- (2) Unit 3 low pressure safety injection B on May 8, 2020
- (3) Unit 1 auxiliary feedwater system B on June 23, 2020
Complete Walkdown Sample (IP Section 03.02) (1 Sample)
- (1) The inspectors evaluated system configurations during a complete walkdown of the hydrogen recombiner system shared among the three units on June 8, 2020.
71111.05 - Fire Protection
Fire Area Walkdown and Inspection Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the implementation of the fire protection program by conducting a walkdown and performing a review to verify program compliance, equipment functionality, material condition, and operational readiness of the following fire areas:
- (1) Unit 2 southwest of containment building, Fire Zone 66A, on April 6, 2020
- (2) Unit 1 condensate transfer pump room and tunnel, Fire Zone 83, on April 9, 2020
- (3) Unit 3 condensate transfer pump room and tunnel, Fire Zone 83, on April 20, 2020
- (4) Unit 3 140 elevation fuel building, Fire Zone 29A, on May 28, 2020
- (5) Unit 1 west piping penetration area, Fire Zone 37C, on June 3, 2020
71111.06 - Flood Protection Measures
Inspection Activities - Internal Flooding (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated internal flooding mitigation protections in the class 1E electrical penetration rooms on May 21, 2020.
71111.07T - Heat Sink Performance Triennial Review (IP Section 03.02)
- (1) Unit 3 diesel generator A jacket water cooler, cooled by service water
- (2) Unit 2 diesel generator A air intercooler, cooled by service water
- (3) Unit 1 essential chiller B, cooled by essential cooling water
- (4) Unit 3 control room essential air cooling unit A, cooled by essential chilled water
71111.08P - Inservice Inspection Activities (PWR) PWR Inservice Inspection Activities Sample (IP Section 03.01)
- (1) The inspectors verified that the reactor coolant system boundary, steam generator tubes, reactor vessel internals, risk-significant piping system boundaries, and containment boundary are appropriately monitored for degradation and that repairs and replacements were appropriately fabricated, examined, and accepted by reviewing the following activities from April 6, 2020, to April 17, 2020. Due to the COVID 19 pandemic, only visual examinations were performed during this outage.
Therefore, one ultrasonic examination from the previous Unit 2 refueling outage was selected for review as allowed by the inspection procedure.
03.01.a - Nondestructive Examination and Welding Activities.
1. Ultrasonic examination a. Steam Generator, Weld 54-10A, reducer to pipe 2. Visual examination a. Reactor coolant system instrumentation connections 4-99 cold leg connections b. Reactor coolant system instrumentation connections, 5-36 c. Reactor coolant system instrumentation connections, 6-99 cold leg connection d. Reactor coolant pump pressure taps (4 suction taps) 6-100-1, 6-100-3, 6-100-6, 6-100-7 e. Pressurizer heaters penetrations (36), 5-37 f. Essential cooling water, 2-ewa-eoi, pipe hanger g. Essential cooling water, ewa-e01-w, pipe hanger h. Essential cooling water, 2-ew-023-h-012, pipe hanger 03.01.c - Pressurized-Water Reactor Boric Acid Corrosion Control Activities.
The inspectors reviewed 15 condition reports and associated boric acid evaluations.
03.01.d - Pressurized-Water Reactor Steam Generator Tube Examination Activities.
1. The licensee performed:
a. One hundred percent bobbin testing was performed in both steam generators.
2. No in-situ pressure testing was required.
The inspectors reviewed 18 notifications that dealt with inservice inspections issues and found that items were entered into the corrective action program at the appropriate level and addressed appropriately.
71111.11Q - Licensed Operator Requalification Program and Licensed Operator Performance
Licensed Operator Performance in the Actual Plant/Main Control Room (IP Section 03.01)
(1 Sample)
- (1) The inspectors observed and evaluated Unit 3 operators conducting a controlled power ascension following turbine valve testing on June 12, 2020.
Licensed Operator Requalification Training/Examinations (IP Section 03.02) (1 Sample)
- (1) The inspectors observed and evaluated a simulator scenario for the steam generator tube rupture emergency operating procedure on June 24, 2020.
71111.12 - Maintenance Effectiveness
Maintenance Effectiveness (IP Section 03.01) (1 Sample)
The inspectors evaluated the effectiveness of maintenance to ensure the following structures, systems, and components (SSCs) remain capable of performing their intended function:
- (1) Unit 2 safety injection system B reliability performance monitoring criteria exceeded due to a blown fuse causing the shutdown cooling isolation valve SI-654 to be nonfunctional on March 21, 2020.
Quality Control (IP Section 03.02) (1 Sample)
The inspectors evaluated the effectiveness of maintenance and quality control activities to ensure the following SSC remains capable of performing its intended function:
- (1) Review of commercial grade dedication of 15-amp fuses on April 27-28, 2020
71111.13 - Maintenance Risk Assessments and Emergent Work Control
Risk Assessment and Management Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the accuracy and completeness of risk assessments for the following planned and emergent work activities to ensure configuration changes and appropriate work controls were addressed:
- (1) Unit 2 refueling outage 22 plan on April 1, 2020
- (2) Unit 3 emergent work for auxiliary feedwater pump B on April 27, 2020
- (3) Unit 2 work controls for contingency freeze seal installation to repair valve 2PSIB-V532 on May 1, 2020
- (4) Planned outage of 525 KV switchyard east bus for capacitive coupled voltage transformer replacement on May 8, 2020
- (5) Unit 2 planned maintenance of containment spray pump B and low-pressure safety injection pump B on May 21, 2020
71111.15 - Operability Determinations and Functionality Assessments
Operability Determination or Functionality Assessment (IP Section 03.01) (11 Samples)
The inspectors evaluated the licensees justifications and actions associated with the following operability determinations and functionality assessments:
- (1) Unit 1 reactor coolant pump orbital vibration monitoring system on April 6, 2020
- (2) Unit 2 startup transformer 3 high temperature on April 21, 2020
- (3) Units 1, 2, and 3 GE Hitachi AKR-2B-50 breakers part 21 notification on April 22, 2020
- (4) Unit 2 operability of safety injection system valve SIB-UV-275 on April 29, 2020
- (5) Unit 2 evaluation of debris identified during core reload on April 29, 2020
- (6) Unit 2 operability of the containment liner on April 30, 2020
- (7) Unit 1 reactor trip circuit breaker B failed to reset multiple times on May 19, 2020
- (8) Unit 2 operability of condensate storage tank in response to CST trouble - leakage alarm received on June 1, 2020
- (9) Unit 2 operability impact of damaged conductors within AFW pump turbine temperature elements on June 2, 2020
- (10) Unit 3 containment hydrogen analyzer isolation valve HPB-UV8 on June 4, 2020
- (11) Units 1, 2, and 3 functionality of hydrogen recombiner B on June 10, 2020
71111.18 - Plant Modifications
Temporary Modifications and/or Permanent Modifications (IP Section 03.01 and/or 03.02)
(1 Sample)
The inspectors evaluated the following temporary or permanent modifications:
- (1) Design modification for the condensate low-low level trip defeat switch
71111.19 - Post-Maintenance Testing
Post-Maintenance Test Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the following post-maintenance test activities to verify system operability and functionality:
- (1) 77ST-9SB13, Unit 2 core protection calculator channel A functional testing following planned maintenance loss and restoration of instrument power on April 25, 2020
- (2) 40OP-9AF01, Unit 3 essential auxiliary feedwater system after corrective maintenance on leaking oil sight glass on April 29, 2020
- (3) 32MT-9RC03, Unit 2 reactor coolant pump 1B testing following motor replacement on May 6, 2020
- (4) Work order 5231052, startup transformer no. 2 NANX03 testing following planned maintenance and repairs on June 16, 2020
- (5) 73ST-9XI20, Unit 2 ADV-178 stroke after nitrogen check valve internals replaced on June 18, 2020
71111.20 - Refueling and Other Outage Activities
Refueling/Other Outage Sample (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated refueling outage 2R22 activities from April 4 to May 4, 2020.
71111.22 - Surveillance Testing
The inspectors evaluated the following surveillance tests:
Surveillance Tests (other) (IP Section 03.01)
- (1) Unit 2 auxiliary feedwater pump B on April 29, 2020
- (2) Unit 1 plant protection system functional testing on May 30, 2020
- (3) Unit 3 diesel generator B 24-hour loaded run, load reject, and hot restart test on
June 15, 2020 Inservice Testing (IP Section 03.01) (1 Sample)
- (1) Unit 1 essential chill water pump B inservice test on April 20, 2020
Containment Isolation Valve Testing (IP Section 03.01) (1 Sample)
- (1) Unit 2 quick opening closure device local leak rate test review on May 5,
OTHER ACTIVITIES - BASELINE
===71151 - Performance Indicator Verification The inspectors verified licensee performance indicators submittals listed below:
MS06: Emergency AC Power Systems (IP Section 02.05) ===
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS07: High Pressure Injection Systems (IP Section 02.06) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS08: Heat Removal Systems (IP Section 02.07) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS09: Residual Heat Removal Systems (IP Section 02.08) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS10: Cooling Water Support Systems (IP Section 02.09) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
71152 - Problem Identification and Resolution
Semiannual Trend Review (IP Section 02.02) (1 Sample)
- (1) The inspectors reviewed the licensees corrective action program for trends in unplanned reactor scrams and shutdowns between February 2018 and March 2020 that might be indicative of a more significant safety issue. No significant trends were observed and no findings were identified.
Annual Follow-up of Selected Issues (IP Section 02.03) (1 Sample)
The inspectors reviewed the licensees implementation of its corrective action program related to the following issues:
- (1) Incorrect installation of Unit 3 reactor coolant pump 1B seal assembly on November 22, 2019
71153 - Follow-up of Events and Notices of Enforcement Discretion Event Report (IP Section 03.02)
The inspectors evaluated the following licensee event reports (LERs):
- (1) Licensee Event Report 05000529/2020-001-00, Unit 2 Reactor Trip on Low Steam Generator Level Signal on March 3, 2020, LER 50-529-2020-001 (ADAMS Accession No. ML20121A285). The inspection conclusions associated with this LER are documented in this report under Inspection Results Section
INSPECTION RESULTS
Incorrect Installation of Unit 3 Reactor Coolant Pump 1B Seal Assembly Cornerstone Significance Cross-Cutting Report Aspect Section Initiating Events Green [H.6] - Design 71152 FIN 05000530/2020002-01 Margins Open/Closed A self-revealed Green finding was identified when the licensee failed to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Revision 41. Specifically, when installing the Unit 3 1B reactor coolant pump seal package during the fall 2019 refueling outage, the stage 2 to stage 3 drive pins were not installed properly causing the seal to leak and ultimately requiring a forced outage to replace the seal on February 8, 2020.
Description:
On November 22, 2019, Unit 3 operators identified a degrading trend for reactor coolant pump 1B second and third stage seal pressures as well as an increasing flow trend into the reactor drain tank (RDT). An evaluation completed on November 25, 2019, determined that this was likely indicative of a reactor coolant pump 1B seal leak. By February 7, 2020, reactor coolant system unidentified leak rate had increased to 0.328 gallons per minute (gpm). Unit 3 was shut down on February 8, 2020, to replace the reactor coolant pump 1B seal assembly.
A visual inspection of the disassembled seal found damage to the seal assembly as well as a misalignment of the second and third stage shaft protection sleeves. A cause evaluation found that the installation of the seal assembly on November 5, 2019 (during the Unit 3 refueling outage) was not done in accordance with procedure. Station Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Section 6.17, outlines steps that indicate the stage 2 to stage 3 drive pins for the inner pins are to be installed before the outer pins. However, the investigation found that the outer pins were installed before the inner pins.
The procedure also requires measurement taken to verify seal running clearances (Kf). It was noted that the value for Kf was larger than the previous value (2.290 vs. 2.270).
Although the procedure does not have any specific acceptance criteria for this measurement, the discrepancy was brought to the attention of the engineering department. Interviews with maintenance personnel revealed a lack of understanding of the importance of the Kf value as well as what a good Kf value should be since no acceptance criteria existed in the procedure.
Corrective Actions: Unit 3 reactor coolant pump seal assembly was replaced during the forced outage.
Corrective Action References: 20-01928
Performance Assessment:
Performance Deficiency: The licensees failure to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Section 6.17, when installing the Unit 3 reactor coolant pump 1B seal assemblies during the fall 2019 refueling outage was a performance deficiency.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the human performance attribute of the Initiating Events Cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the installation sequence errors caused seal degradation which ultimately required an unplanned plant shutdown in order to replace the seal assembly.
Significance: The inspectors assessed the significance of the finding using Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined that the finding had very low safety significance (Green) because the finding could not result in exceeding the RCS leak rate for a small loss of coolant accident (LOCA) and would not have likely affected other systems used to mitigate a LOCA, resulting in a total loss of their function (Exhibit 1, Section A).
Cross-Cutting Aspect: H.6 - Design Margins: The organization operates and maintains equipment within design margins. Margins are carefully guarded and changed only through a systematic and rigorous process. Special attention is placed on maintaining fission product barriers, defense-in-depth, and safety-related equipment. The finding has a cross-cutting aspect in the area of human performance associated with the design margins component because the licensee failed to operate and maintain their equipment within design margins. Specifically, the licensees procedure did not have an acceptance criterion for the seal running clearance measurement. As a result, the licensee was not aware that a higher than normal value could be indicative of a seal assembly mis-alignment. Ultimately this caused a degradation of the reactor coolant pump seal assembly during normal plant operations.
Enforcement:
Inspectors did not identify a violation of regulatory requirements associated with this finding.
Failure to Adequately Evaluate or Test the Impact of a Design Modification to the Unit 2 Main Feedwater Lube Oil Control Panel Protective Relay System Cornerstone Significance Cross-Cutting Report Aspect Section Initiating Events Green None 71153 FIN 05000529/2020002-02 Open/Closed EA-20-075 A self-revealed Green finding was identified when the licensee failed to follow Procedure 81DP-0EE10, Design Change Process, Revision 30. Specifically, the licensee failed to adequately evaluate or test the impact of a design modification to the Unit 2 main feedwater lube oil control panel protective relay system, which resulted in a spurious loss of both main feedwater pumps and subsequent reactor trip on March 3, 2020.
Description:
On March 3, 2020, at 3:25 p.m., an operator inadvertently opened the breaker that supplies the Unit 2 main feedwater pump lube oil control panel. At 8:49 p.m., power to the Unit 2 main feedwater pump lube oil control panel was restored by closing the supply breaker. Upon restoration of power, an erroneous low lube oil pressure signal tripped both main feedwater pumps, which resulted in an automatic reactor trip on low steam generator 1 water level.
The function of the main feedwater lube oil control panel protective relays is to monitor main feedwater bearing lube oil pressure and to take protective actions when a low-pressure condition is detected. The protective system consists of multiple digital relays and auxiliary relays configured to provide a 2 out of 3 logic on the low-pressure protection circuit. Per the vendor technical documents, the digital relays default to the tripped condition upon loss of power, whereas the auxiliary relays default to the non-tripped condition. The digital relays perform a power-on diagnostic routine, which takes approximately 10 seconds. The auxiliary relays do not have an initialization delay. Thus, during this event, when power was initially lost and then subsequently restored to the circuit, the auxiliary relays immediately actuated in response to the digital relay being tripped, providing an erroneous low lube oil pressure signal.
In 2013, the licensee modified the main feedwater pump lube oil system to address single point vulnerabilities and provide increased reliability. The original low-pressure switches were replaced with the current switching logic. However, while implementing this modification, the licensee failed to adequately evaluate the installation of the low-pressure switching logic under Design Modification Work Order ENG-DMWO 2512314. After the modification was installed, post-modification return-to-service testing should have revealed that the changes introduced a design vulnerability that could cause spurious tripping of the main feedwater pumps when power to the protective relay circuitry was cycled. However, the station failed to properly test the replacement logic system during or following this modification to determine whether the operational characteristics of the replacement system were fully compatible to support the systems functionality. The inspectors concluded that the post modification testing was inadequate and failed to identify the design vulnerability.
Procedure 81DP-0EE10, Design Change Process, Revision 30, step C.7.12.1.5, requires, in part, that design verification testing demonstrates that Failure logic will trip the component as expected (positive testing) and also that less than the expected failure will not trip the component (negative testing). Contrary to the above, on September 18, 2013, the licensees design verification testing of a main feedwater modification failed to demonstrate that less than the expected failure would not trip the component (negative testing). Specifically, design verification testing of the low-pressure switching logic installed in the main feedwater lube oil control panel protective relay system under design modification Work Order ENG-DMWO 2512314 failed to demonstrate that the component would not spuriously trip without an actual low-pressure signal. In particular, the switching logic response during a loss of power and subsequent restoration of power scenario for the associated electrical bus was not adequately tested at the time of the modification, which resulted in an unexpected trip of the main feedwater system during normal operations.
Corrective Actions: Place trip hazard labeling on the associated breakers to prevent future plant events by recognizing the reactor trip hazard of removing and restoring power to the main feedwater pump lube oil system at-power.
Corrective Action References: 20-02912
Performance Assessment:
Performance Deficiency: The inspectors determined that the failure to perform adequate design verification testing for modification Work Order ENG-DMWO 2512314 was a performance deficiency that was within the licensees ability to foresee and correct.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the design control attribute of the Initiating Events Cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the inadequate design verification resulted in a latent vulnerability upon restoration of electrical power to the system that resulted in an automatic trip on March 3, 2020.
Significance: The inspectors assessed the significance of the finding using Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. A detailed risk evaluation was required for this finding because it was a condition that caused an automatic reactor scram and the loss of main feedwater. A senior reactor analyst performed the detailed risk assessment and determined that the estimated increase in core damage frequency from the loss of main feedwater event at Palo Verde, Unit 2, on March 3, 2020, was 9.9E-7/year. The analyst applied qualitative factors to the results and to estimate that the significance was of very low safety significance (Green). The full Detailed Risk Assessment is provided as an attachment to this report.
Cross-Cutting Aspect: Not Present Performance. No cross-cutting aspect was assigned to this finding because the inspectors determined the finding did not reflect present licensee performance. The actions of the performance deficiency occurred in 2013 and are not able to be determined as a reliable indicator of current licensee performance.
Enforcement:
Inspectors did not identify a violation of regulatory requirements associated with this finding.
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
- On April 17, 2020, the inspectors presented the Unit 2 Inservice inspection results to Mr. Bruce Rash, Vice President, Nuclear Engineering and Regulatory, and other members of the licensee staff.
- On June 18, 2020, the inspectors presented the Triennial Heat Sink inspection results to Mr. Bruce Rash, Vice President, Nuclear Engineering and Regulatory, and other members of the licensee staff.
- On July 7, 2020, the inspectors presented the integrated inspection results to Mrs. Maria Lacal, Executive Vice President/Chief Nuclear Officer, and other members of the licensee staff.
DOCUMENTS REVIEWED
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.01 Corrective Action Condition Reports 20-07013
Documents
71111.01 Miscellaneous Updated Final Safety Analysis Report 20
71111.01 Miscellaneous 13-MS-A154 Tornado-Borne Missile Impact Analyses for Unprotected 1
Components
71111.01 Miscellaneous MN-850-A00001 Design Basis Tornado-Generated Missile Impact Calculation 1
for U58 Containment Penetration
71111.01 Procedures 01DP-0XX01 Control and Monitoring of Potential Tornado Borne Missiles 6
71111.01 Procedures 40OP-9ZZ19 Hot Weather Protection 7
71111.04 Corrective Action Condition Reports 20-07332, 20-07538, 20-07585, 20-07955
Documents
71111.04 Drawings 01-E-ZPL-002 Power Block Safe Shutdown Emergency Lighting 16
71111.04 Drawings 01-M-AF-001 P&I Diagram Auxiliary Feedwater System 44
71111.04 Drawings 01-M-HPP-001 Containment Hydrogen Control 22
71111.04 Drawings 01-M-SIP-001 P&I Safety Injection & Shutdown Cooling System 59
71111.04 Miscellaneous HPDBM Containment Hydrogen Control System 14
71111.04 Procedures 31MT-9HP01 Hydrogen Recombiner/Hydrogen Purge Exhaust Air 8
Filtration Unit Removal and Installation
71111.04 Procedures 32FT-9QD08 Annual Fire Protection Test for QDN-N08 14
71111.04 Procedures 36ST-9HP01 Hydrogen Recombiner Instrumentation Calibration and 17
Functional Test
71111.04 Procedures 40EP-9E003 Loss of Coolant Accident 44
71111.04 Procedures 40EP-9EO10-019 Appendix 19: Containment Hydrogen Control 0
71111.04 Procedures 40OP-9HP02 Containment Hydrogen Control and Hydrogen Purge 6
Exhaust System (HP)
71111.04 Procedures 40ST-9SI14 Train B LPSI and CS System Alignment Verification 2
71111.04 Procedures 41ST-1HP01 Hydrogen Recombiner Functional Test 19
71111.04 Work Orders 215025
71111.05 Drawings 13-A-ZYD-023 Fire Protection Auxiliary Building Floor Plan 16
71111.05 Miscellaneous 02-M-AFP-001 Auxiliary Feedwater System 34
71111.05 Miscellaneous 02-M-RDP-002 Radioactive Waste Drain System 18
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.05 Miscellaneous UFSAR Sect 9B Fire Hazards Analysis 17
71111.05 Procedures PVGS Pre-Fire Strategies Manual 26
71111.05 Procedures 14DP-0FP33 Control of Transient Combustibles 30
71111.07T Calculations 13-MC-DG-0411 Diesel Generator (DG) Heat Exchanger Minimum Flow Rate 5
vs. Inlet SP Water Temperature
71111.07T Calculations 13-MC-HJ-0003 Control Building HVAC System (HJ) Heat Load and 10
Equipment Adequacy Calculation
71111.07T Calibration Heat Exchanger Test Instrument Calibration - 3MHJAF04
Records
71111.07T Corrective Action Condition Reports 17-14358, 18-01044, 19-02995, 19-03263, 19-10943,
Documents 19-11012, 19-12422, 19-13800, 19-13802, 19-13997,
20-01497, 20-05883, 20-07557
71111.07T Corrective Action Condition Report 20-08094
Documents
Resulting from
Inspection
71111.07T Drawings 02-M-SPP-002 P & I Diagram Essential Spray Pond System 19
71111.07T Drawings 03-M-HJP-001 P & I Diagram Control Building HVAC 14
71111.07T Drawings 13-10407-M721B- Cooling Coil for HJA(B) - F04 04/01/1983
37-3
71111.07T Drawings 13-10407-M721B- Cooling Coil for HJA(B) 03/09/1987
38-5
71111.07T Drawings 13-M018-00500 Intercoolers & Piping Drawing 13
71111.07T Drawings 13-M018-00501 Intercoolers & Piping DWG 14
71111.07T Drawings 13-M018-00606 19R CPT Exchanger 10
71111.07T Drawings 13-M018-00652 Intercooler Outline Drawing PERFEX 500-L-359 2
71111.07T Drawings 13-M018-01298 Tube Layout For Diesel Generator Air Intercoolers 1
71111.07T Drawings M723-10 Refrigeration Machine Outline Drawing 3
71111.07T Engineering 4757042 Engineering Evaluation 03/24/2016
Evaluations
71111.07T Miscellaneous Closed Cooling Water Chemistry Strategic Plan 8
71111.07T Miscellaneous Essential Spray Pond Water Chemistry Strategic Plan 7
71111.07T Miscellaneous 13-MM-723 Water Chillers Specification 9
71111.07T Miscellaneous 13-VTD-P162- PERFEX Industrial Products Installation, Operation and 12/28/1992
Inspection Type Designation Description or Title Revision or
Procedure Date
0001-1 Maintenance Instructions for Heat Transfer Equipment
71111.07T Miscellaneous 13-VTD-P162- PERFEX Industrial Products Data Sheets and Drawings for 12/28/1992
0002-1 Diesel Generator Intercoolers
71111.07T Miscellaneous 73DP-9ZZ21 Heat Exchanger Visual Inspection - Diesel Generator 09/24/2019
Intercooler - Right Bank
71111.07T Miscellaneous 73DP-9ZZ21 Heat Exchanger Visual Inspection - Diesel Generator 09/23/2019
Intercooler - Left Bank
71111.07T Miscellaneous 73DP-9ZZ21 3A Jacket Water Cooler Inspection Report 02/27/2017
71111.07T Miscellaneous 73DP-9ZZ21 3A Jacket Water Cooler Inspection Report 03/04/2019
71111.07T Miscellaneous EC System Essential Chilled Water System Design Basis Manual 17
Design Basis
Manual
71111.07T Miscellaneous ERET 2991102 Essential Chiller Maintenance Template 10
71111.07T Miscellaneous EW System Essential Cooling Water System Design Basis Manual 24
Design Basis
Manual
71111.07T Miscellaneous Maintenance Rule Maintenance Rule Performance Criteria Basis Worksheet -
Performance Essential Spray Pond System (SP)
Criteria Basis
Worksheet
71111.07T Miscellaneous System Health System Health Report for Essential Spray Pond (SP) 02/01/2018 -
Report - Essential System from February 1, 2018 through July 31, 2018 07/31/2018
Spray Pond (SP)
System
71111.07T Miscellaneous System Health System Health Report for Essential Spray Pond (SP) 08/01/2018 -
Report - Essential System from August 1, 2018 through January 31, 2019 01/31/2019
Spray Pond (SP)
System
71111.07T Miscellaneous System Health System Health Report for Essential Spray Pond (SP) 02/01/2019 -
Report - Essential System from February 1, 2019 through July 31, 2019 07/31/2019
Spray Pond (SP)
System
71111.07T Miscellaneous System Health System Health Report for Essential Spray Pond (SP) 08/01/2019 -
Report - Essential System from August 1, 2019 through January 31, 2020. 01/31/2020
Inspection Type Designation Description or Title Revision or
Procedure Date
Spray Pond (SP)
System
71111.07T Miscellaneous VTD-1982-00001 ITT Standard Storage, Installation, Operation and 1
Maintenance of Heat Exchangers
71111.07T Miscellaneous VTD-C150-0014 Carrier Application Data for the Condenser and Economizer 2
71111.07T Miscellaneous VTD-C725-00004 Cryogenics Engineering Co. (Formerly CTI Nuclear) 3
Instructional Manual for the Control Room Essential Air
Handling Units
71111.07T NDE Reports 19-0783 Eddy Current Examination Report - Unit 2 DGA Intercooler 09/23/2019
B
71111.07T NDE Reports19-784 Eddy Current Examination Report - Unit 2 DGA Intercooler 09/23/19
A
71111.07T Work Orders 4660907/0 Testing of the Control Room Emergency Air Temperature 03/09/2017
Control System Train A
71111.07T Work Orders 4866373/0 Testing of the Control Room Emergency Air Temperature 11/08/2018
Control System Train A
71111.08P` Corrective Action Condition Reports 14-00265, 15-00956, 16-18718, 18-10764, 18-12437,
Documents 19-09766, 19-10498, 19-10558, 19-11147, 19-11505,
19-11971, 19-11976, 19-12237, 19-13479, 19-13515,
19-13561, 19-13648, 19-14482, 19-14972, 19-15012,
19-15458, 19-15585, 19-17818, 20-00172, 20-02472,
20-02917, 20-03007, 20-03339, 20-03343
71111.08P Corrective Action 20-05244
Documents
Resulting from
Inspection
71111.08P Engineering DMWO 2513813 Pressurizer Heater Sleeve Replacement 0
Changes
71111.08P Miscellaneous 2020_03 U2M22 U2M22 Boric Acid Walk Down Requirements 1
BAW Waiver
71111.08P Miscellaneous EPID L-2018- Safety Evaluation By The Office Of Nuclear Reactor 02/19/2019
LLR-0028 Regulation Relief Request No. 58
Regarding Certain Class 1 And 2 Welds For The Third 10-
Year Inservice Inspection Interval
Inspection Type Designation Description or Title Revision or
Procedure Date
Arizona Public Service Company Palo Verde Nuclear
Generating Station, Unit 2
Docket No. STN 50-529
71111.08P Miscellaneous MRP-058 Materials Reliability Program: Boric Acid Corrosion 2
Guidebook, Revision 2: Managing Boric Acid Corrosion
Issues at PWR Power Stations
71111.08P NDE Reports 2-054-010A - 18- Ultrasonic Examination Report for Steam Generator 10/17/2018
UTE-2014 Reducer to Pipe 54-10A
71111.08P NDE Reports 20-VE-2024 Visual Examination of Reactor Coolant Instrumentation 04/05/2020
Connection 4-99 Cold Leg Connections
71111.08P NDE Reports 20-VE-2025 Visual Examination of Reactor Coolant System 04/05/2020
Instrumentation Connections
5-36 PZR Heaters Penetrations (36), 5-37 PZR Instrument
Connections (7)
71111.08P NDE Reports 20-VE-2026 Visual Examination of Reactor Coolant System 04/05/2020
Instrumentation Connections
6-99 Cold Leg Connections, 6-100 RCP Pressure Taps (4
suction Taps) 6-100-1, 6-100-3, 6-100-6, 6-100-7 and 6-101
Hot Leg Instrument Connections (27)
71111.08P NDE Reports 20-VT-2027 Visual Examination Report for 2-EWA-E01 04/06/2020
71111.08P NDE Reports 20-VT-2028 Visual Examination for EWA-E01-W 04/06/2020
71111.08P NDE Reports 20-VT-2029 Visual Examination of 2-EW-023-H-012 04/06/2020
71111.08P Procedures 70TI-9ZC01 Boric Acid Walkdown Leak Detection 23
71111.08P Procedures 73DP-9WP04 Welding and Brazing Control 20
71111.08P Procedures 73DP-9WP05 Welding Filler Material Control 11
71111.08P Procedures 73DP-9ZC01 Boric Acid Corrosion Control Program 8
71111.08P Procedures 73TI-9ZZ05 Dry Magnetic Particle Examination 19
71111.08P Procedures 73TI-9ZZ07 Liquid Penetrant Examination 19
71111.08P Procedures 73TI-9ZZ07 Liquid Penetrant Exam 19
71111.08P Procedures 73TI-9ZZ10 Ultrasonic Weld Examinations in Ferritic Components 15
71111.08P Procedures 73TI-9ZZ17 Visual Examination of Welds, Bolting, and Components 14
71111.08P Procedures 73TI-9ZZ18 Rev Visual Examination of Components Supports 17
71111.08P Procedures 73TI-9ZZ19 Visual Examination of Pump and Valve Internal Surfaces 16
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.08P Procedures 73TI-9ZZ20 Visual Examination of Reactor Vessel Internals 12
71111.08P Procedures 73TI-9ZZ82 Visual Examination of Metal Containment Building Surfaces 8
71111.08P Procedures 73WP-0ZZ07 Welding Of Stainless And Nickel Alloys 19
71111.08P Procedures EPRI-PIPE-MPA- Procedure for Manual Phased Array of Austenitic and 2
Ferritic Pipe Welds
71111.08P Procedures EPRI-PIPE-TWS- Procedure for Manual Phased Array of Through wall sizing 1
MPA-1 in Pipe Welds
71111.08P Self-Assessments 18-14430-002 ISI Self Assessment 04/02/2019
71111.11Q Corrective Action Condition Reports 20-07885, 20-08356
Documents
71111.11Q Miscellaneous NLR20S030200 Licensed Operator Continuing Training Simulator Scenario 04/22/2020
71111.11Q Procedures 40DP-9AP09 SG Tube Rupture Technical Guideline 25
71111.11Q Procedures 40EP-9EO04 Steam Generator Tube Rupture 33
71111.11Q Procedures 40OP-9ZZ05 Power Operations 150
71111.12 Corrective Action Condition Reports 20-04986, 20-05952, 18-17794, 20-03809, 20-04152
Documents
71111.12 Miscellaneous 0547-20 Quality Receiving Checklist for APN 44400996
71111.12 Miscellaneous 0549-20 Quality Receiving Checklist for APN 44400996
71111.12 Miscellaneous 1836-17 Quality Receiving Checklist for APN 44400996
71111.12 Miscellaneous 20-04152 Maintenance Rule Expert Panel Action Record 05/28/2020
71111.12 Miscellaneous 500614161 Purchase Order Number for APN 44400996
71111.12 Procedures 12DP-0MC46 Receipt Inspection 17
71111.12 Procedures 32MT-9ZZ74 Molded Case Circuit Breaker Test 52
71111.12 Procedures 32ST-9ZZ74 Molded Case Circuit Breaker Surveillance Test 48
71111.12 Procedures 87DP-0MC39 Commercial Grade Dedication (CGD) Process 3
71111.12 Procedures 93DP-0LC18 Part 21 Reporting Process 2
71111.12 Work Orders 5233686
71111.13 Corrective Action Condition Reports 20-05805, 20-05983
Documents
71111.13 Drawings 02-P-SIF-105 Containment Building Isometric Safety Injection System 8
Shutdown Cooling Lines
71111.13 Miscellaneous Integrated Risk Assessment for CM 5180151
71111.13 Miscellaneous Switchyard System Health Report Q1-2020
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.13 Miscellaneous Schedulers Evaluation for PV Unit 2 05/21/2020
71111.13 Miscellaneous Palo Verde Unit 2 Operator Logs 05/21/2020
71111.13 Miscellaneous Unit TWO 22 Refueling Outage Final Shutdown Risk 04/02/2020
Assessment
71111.13 Miscellaneous EEOS Risk Evaluation for Unit 1, Unit 2, and Unit 3 05/08/2020
71111.13 Miscellaneous 16-15545-004 Engineering Evaluation 11/02/2016
71111.13 Miscellaneous 20-05805 Unit 3 B AFW Pump Risk Evaluation 04/26/2020
71111.13 Procedures 33MT-9ZZ02 Freeze Sealing 16
71111.13 Procedures 40DP-9OP34 Switchyard Administrative Control 22
71111.13 Work Orders 4802216, 5180151
71111.15 Corrective Action Condition Reports 19-10645, 20-04986, 20-04216, 20-05961, 20-05173,
Documents 20-05143, 20-04374, 20-04787, 20-05961, 20-06959,
20-07538, 20-07585, 20-07491, 20-07438
71111.15 Drawings 02-E-CTB-001 Elementary Diagram Condensate Transfer and Storage 2
System
71111.15 Drawings 02-J-RKS-001 Annunciator / Electronic Isolation List 43
71111.15 Drawings 02-M-AFP-001 P&I Diagram Auxiliary Feedwater System 35
71111.15 Drawings 02-M-CTP-001 P&I Diagram Condensate Transfer and Storage System 27
71111.15 Miscellaneous Decision Making Basis Summary: Repair Options for 04/07/2020
Startup Transformer NAN-X03
71111.15 Miscellaneous Zircaloy Fines and the U2 C2 Fuel Receipt 02/26/1988
71111.15 Miscellaneous 13-CN-0393 Technical Specification for the Evaluation and Repair of 04/23/2015
Containment Building Liner Plate
71111.15 Miscellaneous 20-04218-002 Engineering Evaluation 04/24/2020
71111.15 Miscellaneous 20-VT-2072 Visual Examination Report 04/21/2020
71111.15 Miscellaneous 3011469 Palo Verde Action Request
71111.15 Miscellaneous 3017378 Condition Report Disposition Request
71111.15 Miscellaneous 3017381 Condition Report Action Item
71111.15 Miscellaneous 7675R Product Specification: M5 Zirconium alloy cladding tubes 3.0
71111.15 Miscellaneous MATBZL09 Westinghouse Electric Company Nuclear Fuel Material 07/03/2013
Specification
71111.15 Miscellaneous SC 20-03 Safety Communication 20-03 from GE Hitachi 0
71111.15 Procedures 40DP-9OP26 Operations Condition Reporting Process and Operability 46
Inspection Type Designation Description or Title Revision or
Procedure Date
Determination/Functional Assessment
71111.15 Work Orders 5233686, 5235384, 5110882, 5134706
71111.18 Miscellaneous 4689510 Engineering Disposition for CD-1675 0
71111.18 Miscellaneous S-16-0094 10 CFR 50.59 Screening/ Evaluation 0
71111.18 Procedures 40EP-9EO10-044 Appendix 44: Feeding with the Condensate Pumps 0
71111.18 Procedures 40OP-9ZZ14 Feedwater and Condensate 80
71111.18 Work Orders 4689510
71111.19 Corrective Action Condition Reports 20-05888, 20-05352
Documents
71111.19 Miscellaneous 2309ER.19 Engineering Examination on Clutch - Torque Arm Assembly 1
P/N CL42030-2 S/N-8
71111.19 Procedures 32MT-9RC03 Reactor Coolant Pump Motor Testing 12
71111.19 Procedures 36MT-9SG04 Calibrate the Atmospheric Dump Valve (ADV) SG-HV-178 4
Loop (I&C Portion)
71111.19 Procedures 40OP-9AF01 Essential Auxiliary Feedwater System 68
71111.19 Procedures 40OP-9RC01 Reactor Coolant Pump Operation 50
71111.19 Procedures 73DP-0XI03 Check Valve Predictive Maintenance and Monitoring 29
Program
71111.19 Procedures 73ST-9XI20 ADVs Inservice Test 54
71111.19 Procedures 73ST-9ZZ25 Check Valve Disassembly, Inspection, and Manual Exercise 14
71111.19 Procedures 73ST-9ZZ25 Check Valve Disassembly, Inspection, and Manual Exercise 14
71111.19 Procedures 77ST-9SB13 CPCS Channel A Calibration 17
71111.19 Work Orders 5237596, 5235384, 5231052, 5088215, 5082346, 5188738,
5119759
71111.20 Corrective Action Condition Reports 20-04382, 20-04463, 20-04595, 20-04374, 20-04522,
Documents 20-04751, 20-04799, 20-04780, 20-04840
71111.20 Miscellaneous Unit TWO 22 Refueling Outage Shutdown Risk Assessment 04/02/2020
Final Report
71111.20 Miscellaneous 7675R Letter: Zircaloy Fines and the U2 C2 Fuel Receipt 02/26/1988
71111.20 Miscellaneous MATBZL09 Seamless low-tin and optimized Zirlo Tubing 6
71111.20 Procedures 40ST-9ZZ09 Containment Cleanliness Inspection 24
71111.22 Corrective Action Condition Reports 20-07979, 20-07343, 20-07377
Documents
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.22 Procedures 36ST-9SB04 PPS Functional Test - RPS/ESFAS Logic 26
71111.22 Procedures 73ST-9AF05 Auxiliary Feedwater Pump B - Comprehensive Pump Test 19
71111.22 Procedures 73ST-9CL01 73ST-9CL01 48
71111.22 Procedures 73ST-9DG08 Class 1E Diesel Generator Load Rejection, 24 Hour Rated 14
Load and Hot Start Test Train B
71111.22 Procedures 73ST-9EC01 Essential Chilled Water Pumps - Inservice Test 29
71111.22 Work Orders 5091207, 5087521, 5086899, 5104952
71152 Corrective Action Condition Reports 20-01928, 18-02605, 18-08748, 18-10686, 19-11892,
Documents 20-01928, 20-02912
71152 Procedures 31MT-9RC23 Reactor Coolant Pump Sulzer Bingham Seal Replacement 41
71152 Procedures 81DP-0CC28 Classification of Structures, Systems, and Components 16
71152 Work Orders 5042701
71153 Calculations Calculation Hydraulic Analysis of Fire Protection Water System to Unit 3 3
Number 03-MC- Power Block Vendor Sprinkler Systems
71153 Engineering ENG WO #: 20- Perform Risk Assessment of Unit 2 Trip Following 04/12/2020
Evaluations 02912-015 Restoration of Main Feed Pump Lube Oil Control Panel to
Address Safety Significance
71153 Miscellaneous Palo Verde Nuclear Generating Station, System Training 7
Manual, Volume 21, Auxiliary Feedwater (AF)
71153 Miscellaneous Palo Verde Nuclear Generating Station, System Training 7
Manual, Volume 41, Chemical and Volume Control System
(CH)
71153 Miscellaneous Palo Verde Nuclear Generating Station, System Training 6
Manual, Volume 19, Condensate System (CD)
71153 Miscellaneous Palo Verde Nuclear Generating Station, System Training 4
Manual, Volume 20, Feedwater/Feedwater Pump Turbine
System (FW/FT)
71153 Miscellaneous 13-NS-B062 At-Power PRA Study for Human Reliability Analysis 13, 15
71153 Procedures 40EP-9EO01 Standard Post Trip Actions 23
71153 Procedures 40EP-9EO02 Reactor Trip 14
71153 Procedures 40EP-9EO06 Loss of All Feedwater 22
71153 Procedures 40EP-9EO10-038 Appendix 38: Resetting AFA-P01 0
71153 Procedures 40EP-9EO10-039 Appendix 39: Local Operation of AFB-P01 0
Inspection Type Designation Description or Title Revision or
Procedure Date
71153 Procedures 40EP-9EO10-040 Appendix 40: Local Operation of AFA-P01 Using Main 0
Steam
71153 Procedures 40EP-9EO10-041 Appendix 41: Local Operation of AFN-P01 0
71153 Procedures 40EP-9EO10-042 Appendix 42: Aligning Essential Aux Feedwater Pumps 0
Suction to RMWT
71153 Procedures 40EP-9EO10-043 Appendix 43: Restarting MFPs 0
71153 Procedures 40EP-9EO10-044 Appendix 44: Feeding With the Condensate Pumps 0
71153 Procedures 40EP-9EO10-112 Appendix 112: Manual Operation of AFA-P01 During a 112
Security Event
71153 Procedures 40EP-9EO10-118 Appendix 118: Cross-Connect FP to AF 0
Detailed Risk Evaluation
Palo Verde, Unit 2, Loss of Main Feedwater
Conclusion: The estimated increase in core damage frequency from the loss of main
feedwater event at Palo Verde, Unit 2, on March 3, 2020, was 9.9E-7/year. The analyst applied
qualitative factors to the results and to estimate that the significance was of very low safety
significance (Green). This detailed risk evaluation was reviewed by a reliability and risk analyst
from the Office of Nuclear Reactor Regulation.
A. NRC Internal Events Analysis
Influential Assumptions: The following are model inputs which were discovered during conduct
of the detailed risk evaluation, which the licensee raised, or were assumptions which the final
risk estimate was sensitive to:
1. Recovery of the Main Feedwater (MFW) System
For the March 3, 2020, loss of MFW (LOMFW) event at Palo Verde, Unit 2, the analyst
determined that the failure was recoverable. The analyst, along with a license examiner,
reviewed the applicable procedures and interviewed three senior reactor operators at the
station. The analyst also reviewed the applicable human reliability analysis (HRA)
calculation performed by Palo Verde personnel and documented in Engineering Work
Order 20-02912-015, Perform risk assessment of Unit 2 trip following restoration of
Main Feed Pump Lube Oil Control Panel to address safety significance, dated April 12,
20. In this HRA calculation, Basic Event 1MFW-RECOVERHR, Operations fails to
recover Main Feedwater after an uncomplicated reactor trip, licensee PRA personnel
estimated a failure probability of 5.64E-2 (or a success rate of 94.4 percent) of
operators.
Since the SPAR model did not contain a recovery event, the analyst consulted with INL
to modify the Palo Verde SPAR model to incorporate recovery of the MFW pumps. The
analyst then performed an independent HRA calculation using the SPAR-H HRA
methodology. By procedure, recovery of MFW would be the first choice of operators for
the case where auxiliary feedwater (AFW) was unavailable due to random failures.
Operators would be tasked with performing enough troubleshooting to gain confidence
that recovering the MFW system was a viable strategy. The analyst applied an available
but poor rating to the performance shaping factors (PSF) for diagnosis and kept all other
PSFs as nominal. These factors yielded a failure probability of 5.1E-2 for MFW recovery
via the SPAR-H methodology.
2. Failure rate of Class 1E bus failures
Two of the basic events (ACP-ICC-FC-ESFA, Spurious Electrical Protection on Train A
ESF Bus Locks Out All Power Sources, and ACP-ICC-FC-ESFB, Spurious Electrical
Protection on Train B ESF Bus Locks Out All Power Sources) which were present in
some of the core damage cutsets are instrumentation and control failures associated
with the Class 1E buses. The licensees PRA model contains these failure events which
represent an occurrence at Palo Verde where the load sequencer load shed feature
locked in and would not allow any electrical equipment to be loaded and powered from a
Class 1E bus. The licensee modeled the failure probability at 6.5E-6.
The SPAR model contains a similar event estimated at 1.70E-3. The analyst noted that
the failure probability of 1.70E-3 of these events was derived from data from a 1989
Westinghouse Savannah River Site parameter report for generic failures of
instrumentation and controls equipment. The analyst discussed with INL the possibility
of a more representative value and so based on the failure occurrence and operating
history, a value of 2.61E-5 was used.
3. Condensate system dependency on the Condensate Storage Tank (CST)
The analyst noted that the SPAR model failed the condensate system and therefore
MFW recovery and condensate injection mitigative strategies when the CST failed. The
licensee noted that these strategies would remain available in this case because their
inventory source would be the main condenser which remained available for LOMFW
events. To remove this dependency, the analyst removed the pertinent cutset(s) from
the results.
Core Damage Risk Metric Estimate: The analyst incorporated these changes and estimated the
incremental conditional core damage probability (ICCDP) in accordance with Section 08.02,
Treatment of Degraded Conditions and Initiating Events, of NRC Inspection Manual Chapter 0308, Attachment 3, Significance Determination Process Technical Basis, Issue date June 16,
2016. The manual chapter prescribes estimating ICCDP using the following formula:
To estimate the CCDP, the analyst ran an initiating event analysis in accordance with
Section 8.0, Initiating Events Analyses, from the Risk Assessment of Operational Events
Handbook, Volume 1, Internal Events, Revision 2.02. Specifically, for analyst referred to
Section 8.2, Case 1 - Initiating Event Only, and set the loss of main feedwater event to 1.0
and all other initiating events to 0.0. From this run, the CCDP was estimated as:
CCDP = 1.06E-6
The baseline core damage probability was estimated using the nominal initiating event
frequency and nominal failure probabilities of all other components. The baseline CDP was
estimated as:
Baseline CDP = 7.0E-8
The analyst then obtained the ICCDP as follows:
ICCDP = 1.06E-6 - 7.0E-8
ICCDP = 9.9E-7
This ICCDP value was multiplied by one inverse year to equate the ICCDP to an increase in
core damage frequency of 9.9E-7/year.
Analysis Data: The analyst used Palo Verde SPAR model version PVNG-EQK-HWD-FLEX-
LOMFW-DEESE, which contains modifications described in the sections above to model
version 8.61, run on SAPHIRE, software version 8.2.1, for the evaluation. Truncation of 1E-11
was used in all analyses.
B. Licensees Internal Events Results
The analyst requested that the licensee perform a LOMFW analysis with and without MFW
recovery on their model(s). The results are shown in the table below:
LOMFW without MFW recovery LOMFW with MFW recovery
All hazards model 1.3E-7 3.8E-8
EOOS model (fire water 9.1E-7 7.8E-8
injection not credited)
The SPAR model can replace INL derived industry failure rates data with the failure rates that
Palo Verde uses in their model. The analyst exercised this option as a sensitivity with the same
assumptions related to recovery events and the SPAR model estimated the CCDP to be 1.5E-7.
Also, the licensee has a copy of the NRCs SPAR model which they reviewed and submitted
paper, Recommended SPAR model Adjustments, Palo Verde Unit 2 March 2020 LOMFW
Event, to the analyst on May 15, 2020. The probabilistic risk assessment group at Palo Verde
reviewed the SPAR model results and provided recommended adjustments to the SPAR model
to better reflect the LOMFW event and the systems they considered available to respond to the
event. These adjustments were:
- Crediting MFW recovery: Like the influential assumption in the NRC internal events
analysis, the licensee considered that the MFW was recoverable after the March
LOMFW event. The licensee estimated a human error probability for MFW of 5.64E-2
using their human reliability analysis tool. The analyst estimated a similar value of
5.1E-2 in the NRC internal events section above.
- Changing the failure probability of Basis Event AFW-XHE-XM-XFRRMWTTK: This basic
event models the failure of operators to transfer AFW pumps suction to the reactor
makeup water tank (RMWT) on loss of the CST. Its human error probability in the SPAR
model was 1.0E-1. The licensee proposed changing the value to the product of the
MFW recovery human error probability (5.64E-2) and the condensate injection human
error probability (3.0E-1) which was 1.69E-2. The licensee cited that these strategies
would still be available because they are dependent on the condenser hotwell, not the
CST as their inventory source.
The analyst reviewed this dependency and made the change describe in Item (3) of the
Influential Assumptions section of the NRCs internal events analysis described above
in lieu of the licensees proposed change.
- Setting Basic Event CDS-XHE-XM-2HRs to 0.0: This event assumes that the MFW
system trips and no longer feeds the steam generators 30 minutes after the reactor trip.
The licensee proposed that since for the event on March 3, 2020, MFW was lost at the
time of the reactor trip, this basic event should be set to 0.0 thereby eliminating a
condition that did not occur.
The analyst consulted with INL on Basic Event CDS-XHE-XM-2HRS and the overall
recovery of a LOMFW condition in the SPAR model. From this a model change was
incorporated and new model version PVNG-EQK-HWD-FLEX-LOMFW-DEESE was
created changing this basic event to CDS-XHE-XM-LATE to reflect that this term was
also for late recovery of AFW pumps failing to run. Thus, this change was not made for
this evaluation.
Finally, in conversations with the licensee during the estimation of the ICCDP for this event, the
analyst noted the following topics which the analyst subsequently reviewed.
- Use of fire water injection into a depressurized steam generator: The final available
method of feeding water into a steam generator is using a diesel driven fire water pump
to feed a depressurized steam generator. This steam generator feeding strategy is
modeled in the licensees model, but not in the SPAR model. Of note, the licensee did
credit this strategy in their risk evaluation for the 62-day emergency diesel generator
allowed outage time amendment in December 2016. The analyst noted two HRA events
in licensee documentation, 1AF-FP-BACKUP-HL, Operator Fails to Align FP to feed
SGs via AFN flowpath, from calculation 13-NS-B062, At-Power PRA Study for Human
Reliability Analysis, and 1AF-FLEX-SGHR-HL, Operator Fails to Align FP to AF cross-
connect to feed SG, from Letter 102-07411-MLL/TNW, Emergency License
Amendment Request to Extend Diesel Generator 3B Completion Time. When asked by
the analyst, the licensee replied these HRA events reflected the same action. The
analyst therefore reviewed the information from 1AF-FLEX-SGHR-HL since it was the
more recent of the two when analyzing. The calculation documents a 38 minute delay
time, a 20 minute execution time, and an available time window of 75 minutes. The
analyst considered these and judged that there would be barely enough time when
analyzing per the SPAR-H methodology. Also, the analyst considered that stress would
be high because this strategy would be the last available method of feeding steam
generators. The SPAR-H analysis estimated a failure rate of 5.05E-1. Next, the analyst
considered that the actions were close in time by the same crew which would yield at
least high dependency. These factors yielded a failure rate of 7.5E-1. Because this was
such a high failure rate and because dependency could have been considered complete,
the analyst did not credit this strategy. The analyst did however consider the possibility
of some success as a qualitative factor.
C. External Events Analysis
The analyst noted that this detailed risk assessment evaluates an actual event in which no
external events occurred. Additionally, the period of time that the events impacted plant
equipment was small enough that the probability of an external initiator occurring during this
time would be negligible. Therefore, the analyst assumed that the risk from external events,
given the subject performance deficiency was essentially zero. The licensee came to the same
conclusion external events analysis.
D. Large Early Release Frequency
The increase in large early release frequency from this finding was determined to be of very low
safety significance using Manual Chapter 0609, Appendix H, Containment Integrity Significance
Determination Process, because LOMFW sequences screen as having low safety significance
in pressurized water reactors with large dry containments.
Also, the analyst considered the possibility of consequential steam generator tube ruptures
using the draft guidance of Volume 5, Risk Analysis of Containment-Related Events (LERF), of
the RASP Handbook. Applying this guidance, the analyst estimated that the ICLERP was
1.4E-7. This value contains known conservatisms and is realistically lower when risk-informing
with qualitative factors. First, the same qualitative factors discussed for the estimate of the
increase in core damage frequency in section G below would also be relevant to the ICLERP
estimate. Additionally, the assumption of the release path to the environment in Volume 5 of the
RASP Handbook is conservative. Finally, the evaluation does not consider evacuation
effectiveness or differentiate between early and late releases which would qualitatively lower the
ICLERP estimate. Given the performance deficiency does not challenge or question the steam
generator tube integrity, further evaluation of the consequential steam generator tube rupture
results would support a very low safety significance.
E. Uncertainties
The analyst performed an uncertainties analysis on the results of the SPAR model in SAPHIRE
by running 5000 cases in a Monte Carlo analysis as allowed by SAPHIRE. The results
produced a distribution of outcomes with a point estimate of ICCDP of 1.0E-6. The median
case had an estimate of 5.5E-7. The analyst considered this skew towards lower values in the
determination of final significance.
F. Sensitivities
The analyst performed the sensitivity runs in the table below to evaluate potentially influential
factors of the evaluation.
Sensitivity Action Result
Removing possibility of a Set OEP-VCF-LP-CLOPT equal to 0.0 3.7E-7
consequential LOOP
Class 1E bus lockout frequency Set ACP-ICC-FC-ESFA and -ESFB equal to 1.0E-6
set to licensees value 6.5E-6
Class 1E bus lockout frequency Set ACP-ICC-FC-ESFA and -ESFB equal to 2.0E-6
set to SAPR baseline value 1.7E-3
Zero test and maintenance Ran events and condition assessment with 4.7E-7
no test and maintenance selected
Give higher success rate for Set AFW-XHE-XM-CNTRL equal to 1.0E-1 8.9E-7
TDAFW pump operation
without DC power
Use only motor driven pumps in Set AFW-PMP-CF-FR equal to 0.0 9.6E-7
common cause failure group for
AFW pumps
G. Qualitative Considerations
The analyst noted several qualitative considerations which were unable to be quantified and that
were used to inform the numerical estimate. They were:
- Some credit for fire water injection into a depressurized steam generator could be given.
While the timing for accomplishing fire water injection was barely adequate for scenarios
where all AFW was unavailable at the beginning of the postulated core damage
sequences used in this evaluation, some credit could be considered for scenarios where
the AFW pumps run for part of their 24-hour mission time. This AFW run time would
allow more time to align fire water injection and increase the probability of its success as
a mitigative strategy.
- The licensee, like many licensees, typically does not perform current maintenance
activities on similar systems. This would lower the likelihood of the licensee attempting
the recovery of the MFW panel restoration concurrently with AFW system maintenance.
Several cutsets in the results with AFW test and maintenance terms could eliminated,
lowering the increase in core damage frequency for this event.
- Long-term failures-to-run of the AFW pumps were not fully modeled. For example, if an
AFW pump failed to run at 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br />, an additional 1 to 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> would be available for
operators to employ the condensate injection strategy and therefore the failure rate of
this strategy would be lower. This effect would lower the overall increase in core
damage frequency.
A-6