IR 05000528/2020002
| ML20212M129 | |
| Person / Time | |
|---|---|
| Site: | Palo Verde  |
| Issue date: | 07/31/2020 |
| From: | John Dixon NRC/RGN-IV/DRP/RPB-D |
| To: | Lacal M Arizona Public Service Co |
| References | |
| EA-20-075 IR 2020002 | |
| Download: ML20212M129 (32) | |
Text
July 31, 2020
SUBJECT:
PALO VERDE NUCLEAR GENERATING STATION UNITS 1, 2, AND 3 -
INTEGRATED INSPECTION REPORT 05000528/2020002, 05000529/2020002, AND 05000530/2020002
Dear Mrs. Lacal:
On June 30, 2020, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection at Palo Verde Nuclear Generating Station Units 1, 2, and 3. On July 7, 2020, the NRC inspectors discussed the results of this inspection with you and other members of your staff. The results of this inspection are documented in the enclosed report.
Two findings of very low safety significance (Green) are documented in this report. None of these findings involved a violation of NRC requirements.
If you disagree with a cross-cutting aspect assignment or a finding not associated with a regulatory requirement in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the U.S. Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC 205550001; with copies to the Regional Administrator, Region IV; and the NRC Resident Inspector at Palo Verde Nuclear Generating Station Units 1, 2, and 3. This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public Inspections, Exemptions, Requests for Withholding.
Sincerely, John L. Dixon, Jr., Chief Reactor Projects Branch D Division of Reactor Projects
Docket Nos. 05000528, 05000529, and 05000530 License Nos. NPF-41, NPF-51, and NPF-74
Enclosure:
Inspection Report 05000528/2020002, 05000529/2020002, and 05000530/2020002 w/Attachment: Detailed Risk Assessment
Inspection Report
Docket Numbers:
05000528, 05000529, and 05000530
License Numbers:
Report Numbers:
05000528/2020002, 05000529/2020002 and 05000530/2020002
Enterprise Identifier: I-2020-002-0001
Licensee:
Arizona Public Service Company
Facility:
Palo Verde Nuclear Generating Station Units 1, 2, and 3
Location:
Tonopah, AZ
Inspection Dates:
April 1, 2020 to June 30, 2020
Inspectors:
C. Peabody, Senior Resident Inspector
R. Bywater, Resident Inspector
D. You, Resident Inspector
A. Athar, Resident Inspector
I. Anchondo-Lopez, Reactor Inspector
W. Cullum, Reactor Inspector
J. Drake, Senior Reactor Inspector
F. Thomas, Reactor Inspector
Approved By:
John L. Dixon, Jr., Chief
Reactor Projects Branch D
Division of Reactor Projects
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees performance by conducting an integrated inspection at Palo Verde Nuclear Generating Station Units 1, 2, and 3, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Incorrect Installation of Unit 3 Reactor Coolant Pump 1B Seal Assembly Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events Green FIN 05000530/2020002-01 Open/Closed
[H.6] - Design Margins 71152 A self-revealed Green finding was identified when the licensee failed to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement,
Revision 41. Specifically, when installing the Unit 3 1B reactor coolant pump seal package during the fall 2019 refueling outage, the stage 2 to stage 3 drive pins were not installed properly causing the seal to leak and ultimately requiring a forced outage to replace the seal on February 8, 2020.
Failure to Adequately Evaluate or Test the Impact of a Design Modification to the Unit 2 Main Feedwater Lube Oil Control Panel Protective Relay System Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events Green FIN 05000529/2020002-02 Open/Closed EA-20-075 None 71153 A self-revealed Green finding was identified when the licensee failed to follow Procedure 81DP-0EE10, Design Change Process, Revision 30. Specifically, the licensee failed to adequately evaluate or test the impact of a design modification to the Unit 2 main feedwater lube oil control panel protective relay system, which resulted in a spurious loss of both main feedwater pumps and subsequent reactor trip on March 3, 2020.
Additional Tracking Items
Type Issue Number Title Report Section Status LER 05000529/2020-001-00 LER 2020-001-00 for Palo Verde Nuclear Generating Station (PVNGS), Unit 2,
Reactor Trip on Low Steam Generator Level Signal 71153 Closed
PLANT STATUS
Unit 1 operated at or near full power for the duration of the inspection period.
Unit 2 entered the inspection period at full power. On April 4, 2020, Unit 2 shut down for a planned refueling outage. Unit 2 was restarted on May 4, 2020, and operated at or near full power for the remainder of the inspection period.
Unit 3 operated at or near full power for the duration of the inspection period.
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
Starting on March 20, 2020, in response to the National Emergency declared by the President of the United States on the public health risks of the coronavirus (COVID-19), resident inspectors were directed to begin telework and to remotely access licensee information using available technology. During this time the resident inspectors performed periodic site visits each week and during that time conducted plant status activities as described in IMC 2515, Appendix D; observed risk significant activities; and completed on site portions of IPs. In addition, resident and regional baseline inspections were evaluated to determine if all or portion of the objectives and requirements stated in the IP could be performed remotely. If the inspections could be performed remotely, they were conducted per the applicable IP. In some cases, portions of an IP were completed remotely and on site. The inspections documented below met the objectives and requirements for completion of the IP.
REACTOR SAFETY
71111.01 - Adverse Weather Protection
Seasonal Extreme Weather Sample (IP Section 03.01) (2 Samples)
- (1) The inspectors evaluated summer readiness of offsite and alternating current (AC)power systems on May 8, 2020.
- (2) The inspectors evaluated readiness for seasonal extreme weather conditions prior to the onset of the monsoon season on May 14, 2020.
71111.04 - Equipment Alignment
Partial Walkdown Sample (IP Section 03.01) (3 Samples)
The inspectors evaluated system configurations during partial walkdowns of the following systems/trains:
- (1) Unit 2 auxiliary feedwater system A following refueling outage testing on May 4, 2020
- (2) Unit 3 low pressure safety injection B on May 8, 2020
- (3) Unit 1 auxiliary feedwater system B on June 23, 2020
Complete Walkdown Sample (IP Section 03.02) (1 Sample)
- (1) The inspectors evaluated system configurations during a complete walkdown of the hydrogen recombiner system shared among the three units on June 8, 2020.
71111.05 - Fire Protection
Fire Area Walkdown and Inspection Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the implementation of the fire protection program by conducting a walkdown and performing a review to verify program compliance, equipment functionality, material condition, and operational readiness of the following fire areas:
- (1) Unit 2 southwest of containment building, Fire Zone 66A, on April 6, 2020
- (2) Unit 1 condensate transfer pump room and tunnel, Fire Zone 83, on April 9, 2020
- (3) Unit 3 condensate transfer pump room and tunnel, Fire Zone 83, on April 20, 2020
- (4) Unit 3 140 elevation fuel building, Fire Zone 29A, on May 28, 2020
- (5) Unit 1 west piping penetration area, Fire Zone 37C, on June 3, 2020
71111.06 - Flood Protection Measures
Inspection Activities - Internal Flooding (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated internal flooding mitigation protections in the class 1E electrical penetration rooms on May 21, 2020.
===71111.07T - Heat Sink Performance
Triennial Review (IP Section 03.02)===
- (1) Unit 3 diesel generator A jacket water cooler, cooled by service water
- (2) Unit 2 diesel generator A air intercooler, cooled by service water
- (3) Unit 1 essential chiller B, cooled by essential cooling water
- (4) Unit 3 control room essential air cooling unit A, cooled by essential chilled water
71111.08P - Inservice Inspection Activities (PWR) PWR Inservice Inspection Activities Sample (IP Section 03.01)
- (1) The inspectors verified that the reactor coolant system boundary, steam generator tubes, reactor vessel internals, risk-significant piping system boundaries, and containment boundary are appropriately monitored for degradation and that repairs and replacements were appropriately fabricated, examined, and accepted by reviewing the following activities from April 6, 2020, to April 17, 2020. Due to the COVID 19 pandemic, only visual examinations were performed during this outage.
Therefore, one ultrasonic examination from the previous Unit 2 refueling outage was selected for review as allowed by the inspection procedure.
03.01.a - Nondestructive Examination and Welding Activities.
1. Ultrasonic examination
a. Steam Generator, Weld 54-10A, reducer to pipe
2. Visual examination
a. Reactor coolant system instrumentation connections 4-99 cold leg
connections b. Reactor coolant system instrumentation connections, 5-36 c. Reactor coolant system instrumentation connections, 6-99 cold leg connection d. Reactor coolant pump pressure taps (4 suction taps) 6-100-1, 6-100-3,
6-100-6, 6-100-7 e. Pressurizer heaters penetrations (36), 5-37 f. Essential cooling water, 2-ewa-eoi, pipe hanger g. Essential cooling water, ewa-e01-w, pipe hanger h. Essential cooling water, 2-ew-023-h-012, pipe hanger
03.01.c - Pressurized-Water Reactor Boric Acid Corrosion Control Activities.
The inspectors reviewed 15 condition reports and associated boric acid evaluations.
03.01.d - Pressurized-Water Reactor Steam Generator Tube Examination Activities.
1. The licensee performed:
a. One hundred percent bobbin testing was performed in both steam generators.
2. No in-situ pressure testing was required.
The inspectors reviewed 18 notifications that dealt with inservice inspections issues and found that items were entered into the corrective action program at the appropriate level and addressed appropriately.
71111.11Q - Licensed Operator Requalification Program and Licensed Operator Performance
Licensed Operator Performance in the Actual Plant/Main Control Room (IP Section 03.01) (1 Sample)
- (1) The inspectors observed and evaluated Unit 3 operators conducting a controlled power ascension following turbine valve testing on June 12, 2020.
Licensed Operator Requalification Training/Examinations (IP Section 03.02) (1 Sample)
- (1) The inspectors observed and evaluated a simulator scenario for the steam generator tube rupture emergency operating procedure on June 24, 2020.
71111.12 - Maintenance Effectiveness
Maintenance Effectiveness (IP Section 03.01) (1 Sample)
The inspectors evaluated the effectiveness of maintenance to ensure the following structures, systems, and components (SSCs) remain capable of performing their intended function:
- (1) Unit 2 safety injection system B reliability performance monitoring criteria exceeded due to a blown fuse causing the shutdown cooling isolation valve SI-654 to be nonfunctional on March 21, 2020.
Quality Control (IP Section 03.02) (1 Sample)
The inspectors evaluated the effectiveness of maintenance and quality control activities to ensure the following SSC remains capable of performing its intended function:
- (1) Review of commercial grade dedication of 15-amp fuses on April 27-28, 2020
71111.13 - Maintenance Risk Assessments and Emergent Work Control
Risk Assessment and Management Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the accuracy and completeness of risk assessments for the following planned and emergent work activities to ensure configuration changes and appropriate work controls were addressed:
- (1) Unit 2 refueling outage 22 plan on April 1, 2020
- (2) Unit 3 emergent work for auxiliary feedwater pump B on April 27, 2020
- (3) Unit 2 work controls for contingency freeze seal installation to repair valve 2PSIB-V532 on May 1, 2020
- (4) Planned outage of 525 KV switchyard east bus for capacitive coupled voltage transformer replacement on May 8, 2020
- (5) Unit 2 planned maintenance of containment spray pump B and low-pressure safety injection pump B on May 21, 2020
71111.15 - Operability Determinations and Functionality Assessments
Operability Determination or Functionality Assessment (IP Section 03.01) (11 Samples)
The inspectors evaluated the licensees justifications and actions associated with the following operability determinations and functionality assessments:
- (1) Unit 1 reactor coolant pump orbital vibration monitoring system on April 6, 2020
- (2) Unit 2 startup transformer 3 high temperature on April 21, 2020
- (3) Units 1, 2, and 3 GE Hitachi AKR-2B-50 breakers part 21 notification on April 22, 2020
- (4) Unit 2 operability of safety injection system valve SIB-UV-275 on April 29, 2020
- (5) Unit 2 evaluation of debris identified during core reload on April 29, 2020
- (6) Unit 2 operability of the containment liner on April 30, 2020
- (7) Unit 1 reactor trip circuit breaker B failed to reset multiple times on May 19, 2020
- (8) Unit 2 operability of condensate storage tank in response to CST trouble - leakage alarm received on June 1, 2020
- (9) Unit 2 operability impact of damaged conductors within AFW pump turbine temperature elements on June 2, 2020
- (10) Unit 3 containment hydrogen analyzer isolation valve HPB-UV8 on June 4, 2020
- (11) Units 1, 2, and 3 functionality of hydrogen recombiner B on June 10, 2020
71111.18 - Plant Modifications
Temporary Modifications and/or Permanent Modifications (IP Section 03.01 and/or 03.02) (1 Sample)
The inspectors evaluated the following temporary or permanent modifications:
- (1) Design modification for the condensate low-low level trip defeat switch
71111.19 - Post-Maintenance Testing
Post-Maintenance Test Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the following post-maintenance test activities to verify system operability and functionality:
(1)77ST-9SB13, Unit 2 core protection calculator channel A functional testing following planned maintenance loss and restoration of instrument power on April 25, 2020 (2)40OP-9AF01, Unit 3 essential auxiliary feedwater system after corrective maintenance on leaking oil sight glass on April 29, 2020 (3)32MT-9RC03, Unit 2 reactor coolant pump 1B testing following motor replacement on May 6, 2020
- (4) Work order 5231052, startup transformer no. 2 NANX03 testing following planned maintenance and repairs on June 16, 2020 (5)73ST-9XI20, Unit 2 ADV-178 stroke after nitrogen check valve internals replaced on June 18, 2020
71111.20 - Refueling and Other Outage Activities
Refueling/Other Outage Sample (IP Section 03.01) (1 Sample)
- (1) The inspectors evaluated refueling outage 2R22 activities from April 4 to May 4, 2020.
71111.22 - Surveillance Testing
The inspectors evaluated the following surveillance tests:
Surveillance Tests (other) (IP Section 03.01)
- (1) Unit 2 auxiliary feedwater pump B on April 29, 2020
- (2) Unit 1 plant protection system functional testing on May 30, 2020
- (3) Unit 3 diesel generator B 24-hour loaded run, load reject, and hot restart test on June 15, 2020
Inservice Testing (IP Section 03.01) (1 Sample)
- (1) Unit 1 essential chill water pump B inservice test on April 20, 2020
Containment Isolation Valve Testing (IP Section 03.01) (1 Sample)
- (1) Unit 2 quick opening closure device local leak rate test review on May 5,
OTHER ACTIVITIES - BASELINE
===71151 - Performance Indicator Verification
The inspectors verified licensee performance indicators submittals listed below:
MS06: Emergency AC Power Systems (IP Section 02.05)===
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS07: High Pressure Injection Systems (IP Section 02.06) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS08: Heat Removal Systems (IP Section 02.07) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS09: Residual Heat Removal Systems (IP Section 02.08) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
MS10: Cooling Water Support Systems (IP Section 02.09) (3 Samples)
- (1) Unit 1 (April 1, 2019 through March 31, 2020)
- (2) Unit 2 (April 1, 2019 through March 31, 2020)
- (3) Unit 3 (April 1, 2019 through March 31, 2020)
71152 - Problem Identification and Resolution
Semiannual Trend Review (IP Section 02.02) (1 Sample)
- (1) The inspectors reviewed the licensees corrective action program for trends in unplanned reactor scrams and shutdowns between February 2018 and March 2020 that might be indicative of a more significant safety issue. No significant trends were observed and no findings were identified.
Annual Follow-up of Selected Issues (IP Section 02.03) (1 Sample)
The inspectors reviewed the licensees implementation of its corrective action program related to the following issues:
- (1) Incorrect installation of Unit 3 reactor coolant pump 1B seal assembly on November 22, 2019
71153 - Follow-up of Events and Notices of Enforcement Discretion Event Report (IP Section 03.02)
The inspectors evaluated the following licensee event reports (LERs):
- (1) Licensee Event Report 05000529/2020-001-00, Unit 2 Reactor Trip on Low Steam Generator Level Signal on March 3, 2020, LER 50-529-2020-001 (ADAMS Accession No. ML20121A285). The inspection conclusions associated with this LER are documented in this report under Inspection Results Section
INSPECTION RESULTS
Incorrect Installation of Unit 3 Reactor Coolant Pump 1B Seal Assembly Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events
Green FIN 05000530/2020002-01 Open/Closed
[H.6] - Design Margins 71152 A self-revealed Green finding was identified when the licensee failed to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Revision 41. Specifically, when installing the Unit 3 1B reactor coolant pump seal package during the fall 2019 refueling outage, the stage 2 to stage 3 drive pins were not installed properly causing the seal to leak and ultimately requiring a forced outage to replace the seal on February 8, 2020.
Description:
On November 22, 2019, Unit 3 operators identified a degrading trend for reactor coolant pump 1B second and third stage seal pressures as well as an increasing flow trend into the reactor drain tank (RDT). An evaluation completed on November 25, 2019, determined that this was likely indicative of a reactor coolant pump 1B seal leak. By February 7, 2020, reactor coolant system unidentified leak rate had increased to 0.328 gallons per minute (gpm). Unit 3 was shut down on February 8, 2020, to replace the reactor coolant pump 1B seal assembly.
A visual inspection of the disassembled seal found damage to the seal assembly as well as a misalignment of the second and third stage shaft protection sleeves. A cause evaluation found that the installation of the seal assembly on November 5, 2019 (during the Unit 3 refueling outage) was not done in accordance with procedure. Station Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Section 6.17, outlines steps that indicate the stage 2 to stage 3 drive pins for the inner pins are to be installed before the outer pins. However, the investigation found that the outer pins were installed before the inner pins.
The procedure also requires measurement taken to verify seal running clearances (Kf). It was noted that the value for Kf was larger than the previous value (2.290 vs. 2.270).
Although the procedure does not have any specific acceptance criteria for this measurement, the discrepancy was brought to the attention of the engineering department. Interviews with maintenance personnel revealed a lack of understanding of the importance of the Kf value as well as what a good Kf value should be since no acceptance criteria existed in the procedure.
Corrective Actions: Unit 3 reactor coolant pump seal assembly was replaced during the forced outage.
Corrective Action References: 20-01928
Performance Assessment:
Performance Deficiency: The licensees failure to follow Procedure 31MT-9RC23, Reactor Coolant Pump Sulzer Bingham Seal Replacement, Section 6.17, when installing the Unit 3 reactor coolant pump 1B seal assemblies during the fall 2019 refueling outage was a performance deficiency.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the human performance attribute of the Initiating Events Cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the installation sequence errors caused seal degradation which ultimately required an unplanned plant shutdown in order to replace the seal assembly.
Significance: The inspectors assessed the significance of the finding using Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. The inspectors determined that the finding had very low safety significance (Green) because the finding could not result in exceeding the RCS leak rate for a small loss of coolant accident (LOCA) and would not have likely affected other systems used to mitigate a LOCA, resulting in a total loss of their function (Exhibit 1, Section A).
Cross-Cutting Aspect: H.6 - Design Margins: The organization operates and maintains equipment within design margins. Margins are carefully guarded and changed only through a systematic and rigorous process. Special attention is placed on maintaining fission product barriers, defense-in-depth, and safety-related equipment. The finding has a cross-cutting aspect in the area of human performance associated with the design margins component because the licensee failed to operate and maintain their equipment within design margins. Specifically, the licensees procedure did not have an acceptance criterion for the seal running clearance measurement. As a result, the licensee was not aware that a higher than normal value could be indicative of a seal assembly mis-alignment. Ultimately this caused a degradation of the reactor coolant pump seal assembly during normal plant operations.
Enforcement:
Inspectors did not identify a violation of regulatory requirements associated with this finding.
Failure to Adequately Evaluate or Test the Impact of a Design Modification to the Unit 2 Main Feedwater Lube Oil Control Panel Protective Relay System Cornerstone Significance Cross-Cutting Aspect Report Section Initiating Events
Green FIN 05000529/2020002-02 Open/Closed EA-20-075 None 71153 A self-revealed Green finding was identified when the licensee failed to follow Procedure 81DP-0EE10, Design Change Process, Revision 30. Specifically, the licensee failed to adequately evaluate or test the impact of a design modification to the Unit 2 main feedwater lube oil control panel protective relay system, which resulted in a spurious loss of both main feedwater pumps and subsequent reactor trip on March 3, 2020.
Description:
On March 3, 2020, at 3:25 p.m., an operator inadvertently opened the breaker that supplies the Unit 2 main feedwater pump lube oil control panel. At 8:49 p.m., power to the Unit 2 main feedwater pump lube oil control panel was restored by closing the supply breaker. Upon restoration of power, an erroneous low lube oil pressure signal tripped both main feedwater pumps, which resulted in an automatic reactor trip on low steam generator 1 water level.
The function of the main feedwater lube oil control panel protective relays is to monitor main feedwater bearing lube oil pressure and to take protective actions when a low-pressure condition is detected. The protective system consists of multiple digital relays and auxiliary relays configured to provide a 2 out of 3 logic on the low-pressure protection circuit. Per the vendor technical documents, the digital relays default to the tripped condition upon loss of power, whereas the auxiliary relays default to the non-tripped condition. The digital relays perform a power-on diagnostic routine, which takes approximately 10 seconds. The auxiliary relays do not have an initialization delay. Thus, during this event, when power was initially lost and then subsequently restored to the circuit, the auxiliary relays immediately actuated in response to the digital relay being tripped, providing an erroneous low lube oil pressure signal.
In 2013, the licensee modified the main feedwater pump lube oil system to address single point vulnerabilities and provide increased reliability. The original low-pressure switches were replaced with the current switching logic. However, while implementing this modification, the licensee failed to adequately evaluate the installation of the low-pressure switching logic under Design Modification Work Order ENG-DMWO 2512314. After the modification was installed, post-modification return-to-service testing should have revealed that the changes introduced a design vulnerability that could cause spurious tripping of the main feedwater pumps when power to the protective relay circuitry was cycled. However, the station failed to properly test the replacement logic system during or following this modification to determine whether the operational characteristics of the replacement system were fully compatible to support the systems functionality. The inspectors concluded that the post modification testing was inadequate and failed to identify the design vulnerability.
Procedure 81DP-0EE10, Design Change Process, Revision 30, step C.7.12.1.5, requires, in part, that design verification testing demonstrates that Failure logic will trip the component as expected (positive testing) and also that less than the expected failure will not trip the component (negative testing). Contrary to the above, on September 18, 2013, the licensees design verification testing of a main feedwater modification failed to demonstrate that less than the expected failure would not trip the component (negative testing). Specifically, design verification testing of the low-pressure switching logic installed in the main feedwater lube oil control panel protective relay system under design modification Work Order ENG-DMWO 2512314 failed to demonstrate that the component would not spuriously trip without an actual low-pressure signal. In particular, the switching logic response during a loss of power and subsequent restoration of power scenario for the associated electrical bus was not adequately tested at the time of the modification, which resulted in an unexpected trip of the main feedwater system during normal operations.
Corrective Actions: Place trip hazard labeling on the associated breakers to prevent future plant events by recognizing the reactor trip hazard of removing and restoring power to the main feedwater pump lube oil system at-power.
Corrective Action References: 20-02912
Performance Assessment:
Performance Deficiency: The inspectors determined that the failure to perform adequate design verification testing for modification Work Order ENG-DMWO 2512314 was a performance deficiency that was within the licensees ability to foresee and correct.
Screening: The inspectors determined the performance deficiency was more than minor because it was associated with the design control attribute of the Initiating Events Cornerstone and adversely affected the cornerstone objective to limit the likelihood of events that upset plant stability and challenge critical safety functions during shutdown as well as power operations. Specifically, the inadequate design verification resulted in a latent vulnerability upon restoration of electrical power to the system that resulted in an automatic trip on March 3, 2020.
Significance: The inspectors assessed the significance of the finding using Inspection Manual Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-Power. A detailed risk evaluation was required for this finding because it was a condition that caused an automatic reactor scram and the loss of main feedwater. A senior reactor analyst performed the detailed risk assessment and determined that the estimated increase in core damage frequency from the loss of main feedwater event at Palo Verde, Unit 2, on March 3, 2020, was 9.9E-7/year. The analyst applied qualitative factors to the results and to estimate that the significance was of very low safety significance (Green). The full Detailed Risk Assessment is provided as an attachment to this report.
Cross-Cutting Aspect: Not Present Performance. No cross-cutting aspect was assigned to this finding because the inspectors determined the finding did not reflect present licensee performance. The actions of the performance deficiency occurred in 2013 and are not able to be determined as a reliable indicator of current licensee performance.
Enforcement:
Inspectors did not identify a violation of regulatory requirements associated with this finding.
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
- On April 17, 2020, the inspectors presented the Unit 2 Inservice inspection results to Mr. Bruce Rash, Vice President, Nuclear Engineering and Regulatory, and other members of the licensee staff.
- On June 18, 2020, the inspectors presented the Triennial Heat Sink inspection results to Mr. Bruce Rash, Vice President, Nuclear Engineering and Regulatory, and other members of the licensee staff.
- On July 7, 2020, the inspectors presented the integrated inspection results to Mrs. Maria Lacal, Executive Vice President/Chief Nuclear Officer, and other members of the licensee staff.
DOCUMENTS REVIEWED
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Corrective Action
Documents
Condition Reports 20-07013
Miscellaneous
Updated Final Safety Analysis Report
Miscellaneous
13-MS-A154
Tornado-Borne Missile Impact Analyses for Unprotected
Components
Miscellaneous
MN-850-A00001
Design Basis Tornado-Generated Missile Impact Calculation
for U58 Containment Penetration
Procedures
Control and Monitoring of Potential Tornado Borne Missiles
Procedures
Hot Weather Protection
Corrective Action
Documents
Condition Reports 20-07332, 20-07538, 20-07585, 20-07955
Drawings
01-E-ZPL-002
Power Block Safe Shutdown Emergency Lighting
Drawings
01-M-AF-001
P&I Diagram Auxiliary Feedwater System
Drawings
01-M-HPP-001
Containment Hydrogen Control
Drawings
01-M-SIP-001
P&I Safety Injection & Shutdown Cooling System
Miscellaneous
HPDBM
Containment Hydrogen Control System
Procedures
Hydrogen Recombiner/Hydrogen Purge Exhaust Air
Filtration Unit Removal and Installation
Procedures
Annual Fire Protection Test for QDN-N08
Procedures
Hydrogen Recombiner Instrumentation Calibration and
Functional Test
Procedures
Loss of Coolant Accident
Procedures
Appendix 19: Containment Hydrogen Control
Procedures
Containment Hydrogen Control and Hydrogen Purge
Exhaust System (HP)
Procedures
Train B LPSI and CS System Alignment Verification
Procedures
Hydrogen Recombiner Functional Test
Work Orders
215025
Drawings
13-A-ZYD-023
Fire Protection Auxiliary Building Floor Plan
Miscellaneous
2-M-AFP-001
Auxiliary Feedwater System
Miscellaneous
2-M-RDP-002
Radioactive Waste Drain System
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Miscellaneous
UFSAR Sect 9B
Fire Hazards Analysis
Procedures
PVGS Pre-Fire Strategies Manual
Procedures
Control of Transient Combustibles
Calculations
13-MC-DG-0411
Diesel Generator (DG) Heat Exchanger Minimum Flow Rate
vs. Inlet SP Water Temperature
Calculations
13-MC-HJ-0003
Control Building HVAC System (HJ) Heat Load and
Equipment Adequacy Calculation
Calibration
Records
Heat Exchanger Test Instrument Calibration - 3MHJAF04
Corrective Action
Documents
Condition Reports 17-14358, 18-01044, 19-02995, 19-03263, 19-10943,
19-11012, 19-12422, 19-13800, 19-13802, 19-13997,
20-01497, 20-05883, 20-07557
Corrective Action
Documents
Resulting from
Inspection
Condition Report
20-08094
Drawings
2-M-SPP-002
P & I Diagram Essential Spray Pond System
Drawings
03-M-HJP-001
P & I Diagram Control Building HVAC
Drawings
13-10407-M721B-
37-3
Cooling Coil for HJA(B) - F04
04/01/1983
Drawings
13-10407-M721B-
38-5
Cooling Coil for HJA(B)
03/09/1987
Drawings
13-M018-00500
Intercoolers & Piping Drawing
Drawings
13-M018-00501
Intercoolers & Piping DWG
Drawings
13-M018-00606
19R CPT Exchanger
Drawings
13-M018-00652
Intercooler Outline Drawing PERFEX 500-L-359
Drawings
13-M018-01298
Tube Layout For Diesel Generator Air Intercoolers
Drawings
M723-10
Refrigeration Machine Outline Drawing
Engineering
Evaluations
4757042
Engineering Evaluation
03/24/2016
Miscellaneous
Closed Cooling Water Chemistry Strategic Plan
Miscellaneous
Essential Spray Pond Water Chemistry Strategic Plan
Miscellaneous
13-MM-723
Water Chillers Specification
Miscellaneous
13-VTD-P162-
PERFEX Industrial Products Installation, Operation and
2/28/1992
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
0001-1
Maintenance Instructions for Heat Transfer Equipment
Miscellaneous
13-VTD-P162-
0002-1
PERFEX Industrial Products Data Sheets and Drawings for
Diesel Generator Intercoolers
2/28/1992
Miscellaneous
Heat Exchanger Visual Inspection - Diesel Generator
Intercooler - Right Bank
09/24/2019
Miscellaneous
Heat Exchanger Visual Inspection - Diesel Generator
Intercooler - Left Bank
09/23/2019
Miscellaneous
3A Jacket Water Cooler Inspection Report
2/27/2017
Miscellaneous
3A Jacket Water Cooler Inspection Report
03/04/2019
Miscellaneous
EC System
Design Basis
Manual
Essential Chilled Water System Design Basis Manual
Miscellaneous
ERET 2991102
Essential Chiller Maintenance Template
Miscellaneous
EW System
Design Basis
Manual
Essential Cooling Water System Design Basis Manual
Miscellaneous
Maintenance Rule
Performance
Criteria Basis
Worksheet
Maintenance Rule Performance Criteria Basis Worksheet -
Essential Spray Pond System (SP)
Miscellaneous
System Health
Report - Essential
Spray Pond (SP)
System
System Health Report for Essential Spray Pond (SP)
System from February 1, 2018 through July 31, 2018
2/01/2018 -
07/31/2018
Miscellaneous
System Health
Report - Essential
Spray Pond (SP)
System
System Health Report for Essential Spray Pond (SP)
System from August 1, 2018 through January 31, 2019
08/01/2018 -
01/31/2019
Miscellaneous
System Health
Report - Essential
Spray Pond (SP)
System
System Health Report for Essential Spray Pond (SP)
System from February 1, 2019 through July 31, 2019
2/01/2019 -
07/31/2019
Miscellaneous
System Health
Report - Essential
System Health Report for Essential Spray Pond (SP)
System from August 1, 2019 through January 31, 2020.
08/01/2019 -
01/31/2020
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Spray Pond (SP)
System
Miscellaneous
VTD-1982-00001
ITT Standard Storage, Installation, Operation and
Maintenance of Heat Exchangers
Miscellaneous
VTD-C150-0014
Carrier Application Data for the Condenser and Economizer
Miscellaneous
VTD-C725-00004
Cryogenics Engineering Co. (Formerly CTI Nuclear)
Instructional Manual for the Control Room Essential Air
Handling Units
NDE Reports
19-0783
Eddy Current Examination Report - Unit 2 DGA Intercooler
B
09/23/2019
Eddy Current Examination Report - Unit 2 DGA Intercooler
A
09/23/19
Work Orders
4660907/0
Testing of the Control Room Emergency Air Temperature
Control System Train A
03/09/2017
Work Orders
4866373/0
Testing of the Control Room Emergency Air Temperature
Control System Train A
11/08/2018
71111.08P` Corrective Action
Documents
Condition Reports 14-00265, 15-00956, 16-18718, 18-10764, 18-12437,
19-09766, 19-10498, 19-10558, 19-11147, 19-11505,
19-11971, 19-11976, 19-12237, 19-13479, 19-13515,
19-13561, 19-13648, 19-14482, 19-14972, 19-15012,
19-15458, 19-15585, 19-17818, 20-00172, 20-02472,
20-02917, 20-03007, 20-03339, 20-03343
Corrective Action
Documents
Resulting from
Inspection
20-05244
Engineering
Changes
DMWO 2513813
Pressurizer Heater Sleeve Replacement
Miscellaneous
20_03 U2M22
BAW Waiver
U2M22 Boric Acid Walk Down Requirements
Miscellaneous
EPID L-2018-
LLR-0028
Safety Evaluation By The Office Of Nuclear Reactor
Regulation Relief Request No. 58
Regarding Certain Class 1 And 2 Welds For The Third 10-
Year Inservice Inspection Interval
2/19/2019
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Arizona Public Service Company Palo Verde Nuclear
Generating Station, Unit 2
Docket No. STN 50-529
Miscellaneous
Materials Reliability Program: Boric Acid Corrosion
Guidebook, Revision 2: Managing Boric Acid Corrosion
Issues at PWR Power Stations
NDE Reports
2-054-010A - 18-
UTE-2014
Ultrasonic Examination Report for Steam Generator
Reducer to Pipe 54-10A
10/17/2018
NDE Reports
20-VE-2024
Visual Examination of Reactor Coolant Instrumentation
Connection 4-99 Cold Leg Connections
04/05/2020
NDE Reports
20-VE-2025
Visual Examination of Reactor Coolant System
Instrumentation Connections
5-36 PZR Heaters Penetrations (36), 5-37 PZR Instrument
Connections (7)
04/05/2020
NDE Reports
20-VE-2026
Visual Examination of Reactor Coolant System
Instrumentation Connections
6-99 Cold Leg Connections, 6-100 RCP Pressure Taps (4
suction Taps) 6-100-1, 6-100-3, 6-100-6, 6-100-7 and 6-101
Hot Leg Instrument Connections (27)
04/05/2020
NDE Reports
20-VT-2027
Visual Examination Report for 2-EWA-E01
04/06/2020
NDE Reports
20-VT-2028
Visual Examination for EWA-E01-W
04/06/2020
NDE Reports
20-VT-2029
Visual Examination of 2-EW-023-H-012
04/06/2020
Procedures
Boric Acid Walkdown Leak Detection
Procedures
Welding and Brazing Control
Procedures
Welding Filler Material Control
Procedures
Boric Acid Corrosion Control Program
Procedures
Dry Magnetic Particle Examination
Procedures
Liquid Penetrant Examination
Procedures
Liquid Penetrant Exam
Procedures
Ultrasonic Weld Examinations in Ferritic Components
Procedures
Visual Examination of Welds, Bolting, and Components
Procedures
73TI-9ZZ18 Rev
Visual Examination of Components Supports
Procedures
Visual Examination of Pump and Valve Internal Surfaces
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Procedures
Visual Examination of Reactor Vessel Internals
Procedures
Visual Examination of Metal Containment Building Surfaces
Procedures
Welding Of Stainless And Nickel Alloys
Procedures
EPRI-PIPE-MPA-
Procedure for Manual Phased Array of Austenitic and
Ferritic Pipe Welds
Procedures
EPRI-PIPE-TWS-
MPA-1
Procedure for Manual Phased Array of Through wall sizing
in Pipe Welds
Self-Assessments 18-14430-002
ISI Self Assessment
04/02/2019
Corrective Action
Documents
Condition Reports 20-07885, 20-08356
Miscellaneous
NLR20S030200
Licensed Operator Continuing Training Simulator Scenario
04/22/2020
Procedures
SG Tube Rupture Technical Guideline
Procedures
Steam Generator Tube Rupture
Procedures
Power Operations
150
Corrective Action
Documents
Condition Reports 20-04986, 20-05952, 18-17794, 20-03809, 20-04152
Miscellaneous
0547-20
Quality Receiving Checklist for APN 44400996
Miscellaneous
0549-20
Quality Receiving Checklist for APN 44400996
Miscellaneous
1836-17
Quality Receiving Checklist for APN 44400996
Miscellaneous
20-04152
Maintenance Rule Expert Panel Action Record
05/28/2020
Miscellaneous
500614161
Purchase Order Number for APN 44400996
Procedures
Receipt Inspection
Procedures
Molded Case Circuit Breaker Test
Procedures
Molded Case Circuit Breaker Surveillance Test
Procedures
Commercial Grade Dedication (CGD) Process
Procedures
Part 21 Reporting Process
Work Orders
233686
Corrective Action
Documents
Condition Reports 20-05805, 20-05983
Drawings
2-P-SIF-105
Containment Building Isometric Safety Injection System
Shutdown Cooling Lines
Miscellaneous
Integrated Risk Assessment for CM 5180151
Miscellaneous
Switchyard System Health Report Q1-2020
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Miscellaneous
Schedulers Evaluation for PV Unit 2
05/21/2020
Miscellaneous
Palo Verde Unit 2 Operator Logs
05/21/2020
Miscellaneous
Unit TWO 22 Refueling Outage Final Shutdown Risk
Assessment
04/02/2020
Miscellaneous
EEOS Risk Evaluation for Unit 1, Unit 2, and Unit 3
05/08/2020
Miscellaneous
16-15545-004
Engineering Evaluation
11/02/2016
Miscellaneous
20-05805
Unit 3 B AFW Pump Risk Evaluation
04/26/2020
Procedures
Freeze Sealing
Procedures
Switchyard Administrative Control
Work Orders
4802216, 5180151
Corrective Action
Documents
Condition Reports 19-10645, 20-04986, 20-04216, 20-05961, 20-05173,
20-05143, 20-04374, 20-04787, 20-05961, 20-06959,
20-07538, 20-07585, 20-07491, 20-07438
Drawings
2-E-CTB-001
Elementary Diagram Condensate Transfer and Storage
System
Drawings
2-J-RKS-001
Annunciator / Electronic Isolation List
Drawings
2-M-AFP-001
P&I Diagram Auxiliary Feedwater System
Drawings
2-M-CTP-001
P&I Diagram Condensate Transfer and Storage System
Miscellaneous
Decision Making Basis Summary: Repair Options for
Startup Transformer NAN-X03
04/07/2020
Miscellaneous
Zircaloy Fines and the U2 C2 Fuel Receipt
2/26/1988
Miscellaneous
13-CN-0393
Technical Specification for the Evaluation and Repair of
Containment Building Liner Plate
04/23/2015
Miscellaneous
20-04218-002
Engineering Evaluation
04/24/2020
Miscellaneous
20-VT-2072
Visual Examination Report
04/21/2020
Miscellaneous
3011469
Palo Verde Action Request
Miscellaneous
3017378
Condition Report Disposition Request
Miscellaneous
3017381
Condition Report Action Item
Miscellaneous
7675R
Product Specification: M5 Zirconium alloy cladding tubes
3.0
Miscellaneous
MATBZL09
Westinghouse Electric Company Nuclear Fuel Material
Specification
07/03/2013
Miscellaneous
Safety Communication 20-03 from GE Hitachi
Procedures
Operations Condition Reporting Process and Operability
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Determination/Functional Assessment
Work Orders
233686, 5235384, 5110882, 5134706
Miscellaneous
4689510
Engineering Disposition for CD-1675
Miscellaneous
S-16-0094
CFR 50.59 Screening/ Evaluation
Procedures
Appendix 44: Feeding with the Condensate Pumps
Procedures
Feedwater and Condensate
Work Orders
4689510
Corrective Action
Documents
Condition Reports 20-05888, 20-05352
Miscellaneous
2309ER.19
Engineering Examination on Clutch - Torque Arm Assembly
P/N CL42030-2 S/N-8
Procedures
Reactor Coolant Pump Motor Testing
Procedures
Calibrate the Atmospheric Dump Valve (ADV) SG-HV-178
Loop (I&C Portion)
Procedures
Essential Auxiliary Feedwater System
Procedures
Reactor Coolant Pump Operation
Procedures
Check Valve Predictive Maintenance and Monitoring
Program
Procedures
ADVs Inservice Test
Procedures
Check Valve Disassembly, Inspection, and Manual Exercise
Procedures
Check Valve Disassembly, Inspection, and Manual Exercise
Procedures
CPCS Channel A Calibration
Work Orders
237596, 5235384, 5231052, 5088215, 5082346, 5188738,
5119759
Corrective Action
Documents
Condition Reports 20-04382, 20-04463, 20-04595, 20-04374, 20-04522,
20-04751, 20-04799, 20-04780, 20-04840
Miscellaneous
Unit TWO 22 Refueling Outage Shutdown Risk Assessment
Final Report
04/02/2020
Miscellaneous
7675R
Letter: Zircaloy Fines and the U2 C2 Fuel Receipt
2/26/1988
Miscellaneous
MATBZL09
Seamless low-tin and optimized Zirlo Tubing
Procedures
Containment Cleanliness Inspection
Corrective Action
Documents
Condition Reports 20-07979, 20-07343, 20-07377
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Procedures
PPS Functional Test - RPS/ESFAS Logic
Procedures
Auxiliary Feedwater Pump B - Comprehensive Pump Test
Procedures
73ST-9CL01
Procedures
Class 1E Diesel Generator Load Rejection, 24 Hour Rated
Load and Hot Start Test Train B
Procedures
Essential Chilled Water Pumps - Inservice Test
Work Orders
5091207, 5087521, 5086899, 5104952
Corrective Action
Documents
Condition Reports 20-01928, 18-02605, 18-08748, 18-10686, 19-11892,
20-01928, 20-02912
Procedures
Reactor Coolant Pump Sulzer Bingham Seal Replacement
Procedures
Classification of Structures, Systems, and Components
Work Orders
5042701
Calculations
Calculation
Number 03-MC-
Hydraulic Analysis of Fire Protection Water System to Unit 3
Power Block Vendor Sprinkler Systems
Engineering
Evaluations
ENG WO #: 20-
2912-015
Perform Risk Assessment of Unit 2 Trip Following
Restoration of Main Feed Pump Lube Oil Control Panel to
Address Safety Significance
04/12/2020
Miscellaneous
Palo Verde Nuclear Generating Station, System Training
Manual, Volume 21, Auxiliary Feedwater (AF)
Miscellaneous
Palo Verde Nuclear Generating Station, System Training
Manual, Volume 41, Chemical and Volume Control System
(CH)
Miscellaneous
Palo Verde Nuclear Generating Station, System Training
Manual, Volume 19, Condensate System (CD)
Miscellaneous
Palo Verde Nuclear Generating Station, System Training
Manual, Volume 20, Feedwater/Feedwater Pump Turbine
System (FW/FT)
Miscellaneous
13-NS-B062
At-Power PRA Study for Human Reliability Analysis
13, 15
Procedures
Standard Post Trip Actions
Procedures
Procedures
Loss of All Feedwater
Procedures
Appendix 38: Resetting AFA-P01
Procedures
Appendix 39: Local Operation of AFB-P01
Inspection
Procedure
Type
Designation
Description or Title
Revision or
Date
Procedures
Appendix 40: Local Operation of AFA-P01 Using Main
Steam
Procedures
Appendix 41: Local Operation of AFN-P01
Procedures
Appendix 42: Aligning Essential Aux Feedwater Pumps
Suction to RMWT
Procedures
Appendix 43: Restarting MFPs
Procedures
Appendix 44: Feeding With the Condensate Pumps
Procedures
Appendix 112: Manual Operation of AFA-P01 During a
Security Event
2
Procedures
Appendix 118: Cross-Connect FP to AF
Detailed Risk Evaluation
Palo Verde, Unit 2, Loss of Main Feedwater
Conclusion: The estimated increase in core damage frequency from the loss of main
feedwater event at Palo Verde, Unit 2, on March 3, 2020, was 9.9E-7/year. The analyst applied
qualitative factors to the results and to estimate that the significance was of very low safety
significance (Green). This detailed risk evaluation was reviewed by a reliability and risk analyst
from the Office of Nuclear Reactor Regulation.
A. NRC Internal Events Analysis
Influential Assumptions: The following are model inputs which were discovered during conduct
of the detailed risk evaluation, which the licensee raised, or were assumptions which the final
risk estimate was sensitive to:
1. Recovery of the Main Feedwater (MFW) System
For the March 3, 2020, loss of MFW (LOMFW) event at Palo Verde, Unit 2, the analyst
determined that the failure was recoverable. The analyst, along with a license examiner,
reviewed the applicable procedures and interviewed three senior reactor operators at the
station. The analyst also reviewed the applicable human reliability analysis (HRA)
calculation performed by Palo Verde personnel and documented in Engineering Work
Order 20-02912-015, Perform risk assessment of Unit 2 trip following restoration of
Main Feed Pump Lube Oil Control Panel to address safety significance, dated April 12,
20. In this HRA calculation, Basic Event 1MFW-RECOVERHR, Operations fails to
recover Main Feedwater after an uncomplicated reactor trip, licensee PRA personnel
estimated a failure probability of 5.64E-2 (or a success rate of 94.4 percent) of
operators.
Since the SPAR model did not contain a recovery event, the analyst consulted with INL
to modify the Palo Verde SPAR model to incorporate recovery of the MFW pumps. The
analyst then performed an independent HRA calculation using the SPAR-H HRA
methodology. By procedure, recovery of MFW would be the first choice of operators for
the case where auxiliary feedwater (AFW) was unavailable due to random failures.
Operators would be tasked with performing enough troubleshooting to gain confidence
that recovering the MFW system was a viable strategy. The analyst applied an available
but poor rating to the performance shaping factors (PSF) for diagnosis and kept all other
PSFs as nominal. These factors yielded a failure probability of 5.1E-2 for MFW recovery
via the SPAR-H methodology.
2. Failure rate of Class 1E bus failures
Two of the basic events (ACP-ICC-FC-ESFA, Spurious Electrical Protection on Train A
ESF Bus Locks Out All Power Sources, and ACP-ICC-FC-ESFB, Spurious Electrical
Protection on Train B ESF Bus Locks Out All Power Sources) which were present in
some of the core damage cutsets are instrumentation and control failures associated
with the Class 1E buses. The licensees PRA model contains these failure events which
represent an occurrence at Palo Verde where the load sequencer load shed feature
locked in and would not allow any electrical equipment to be loaded and powered from a
Class 1E bus. The licensee modeled the failure probability at 6.5E-6.
The SPAR model contains a similar event estimated at 1.70E-3. The analyst noted that
the failure probability of 1.70E-3 of these events was derived from data from a 1989
Westinghouse Savannah River Site parameter report for generic failures of
instrumentation and controls equipment. The analyst discussed with INL the possibility
of a more representative value and so based on the failure occurrence and operating
history, a value of 2.61E-5 was used.
3. Condensate system dependency on the Condensate Storage Tank (CST)
The analyst noted that the SPAR model failed the condensate system and therefore
MFW recovery and condensate injection mitigative strategies when the CST failed. The
licensee noted that these strategies would remain available in this case because their
inventory source would be the main condenser which remained available for LOMFW
events. To remove this dependency, the analyst removed the pertinent cutset(s) from
the results.
Core Damage Risk Metric Estimate: The analyst incorporated these changes and estimated the
incremental conditional core damage probability (ICCDP) in accordance with Section 08.02,
Treatment of Degraded Conditions and Initiating Events, of NRC Inspection Manual Chapter 0308, Attachment 3, Significance Determination Process Technical Basis, Issue date June 16,
2016. The manual chapter prescribes estimating ICCDP using the following formula:
To estimate the CCDP, the analyst ran an initiating event analysis in accordance with
Section 8.0, Initiating Events Analyses, from the Risk Assessment of Operational Events
Handbook, Volume 1, Internal Events, Revision 2.02. Specifically, for analyst referred to
Section 8.2, Case 1 - Initiating Event Only, and set the loss of main feedwater event to 1.0
and all other initiating events to 0.0. From this run, the CCDP was estimated as:
CCDP = 1.06E-6
The baseline core damage probability was estimated using the nominal initiating event
frequency and nominal failure probabilities of all other components. The baseline CDP was
estimated as:
Baseline CDP = 7.0E-8
The analyst then obtained the ICCDP as follows:
ICCDP = 1.06E-6 - 7.0E-8
ICCDP = 9.9E-7
This ICCDP value was multiplied by one inverse year to equate the ICCDP to an increase in
core damage frequency of 9.9E-7/year.
Analysis Data: The analyst used Palo Verde SPAR model version PVNG-EQK-HWD-FLEX-
LOMFW-DEESE, which contains modifications described in the sections above to model
version 8.61, run on SAPHIRE, software version 8.2.1, for the evaluation. Truncation of 1E-11
was used in all analyses.
B. Licensees Internal Events Results
The analyst requested that the licensee perform a LOMFW analysis with and without MFW
recovery on their model(s). The results are shown in the table below:
LOMFW without MFW recovery
LOMFW with MFW recovery
All hazards model
1.3E-7
3.8E-8
EOOS model (fire water
injection not credited)
9.1E-7
7.8E-8
The SPAR model can replace INL derived industry failure rates data with the failure rates that
Palo Verde uses in their model. The analyst exercised this option as a sensitivity with the same
assumptions related to recovery events and the SPAR model estimated the CCDP to be 1.5E-7.
Also, the licensee has a copy of the NRCs SPAR model which they reviewed and submitted
paper, Recommended SPAR model Adjustments, Palo Verde Unit 2 March 2020 LOMFW
Event, to the analyst on May 15, 2020. The probabilistic risk assessment group at Palo Verde
reviewed the SPAR model results and provided recommended adjustments to the SPAR model
to better reflect the LOMFW event and the systems they considered available to respond to the
event. These adjustments were:
Crediting MFW recovery: Like the influential assumption in the NRC internal events
analysis, the licensee considered that the MFW was recoverable after the March
LOMFW event. The licensee estimated a human error probability for MFW of 5.64E-2
using their human reliability analysis tool. The analyst estimated a similar value of
5.1E-2 in the NRC internal events section above.
Changing the failure probability of Basis Event AFW-XHE-XM-XFRRMWTTK: This basic
event models the failure of operators to transfer AFW pumps suction to the reactor
makeup water tank (RMWT) on loss of the CST. Its human error probability in the SPAR
model was 1.0E-1. The licensee proposed changing the value to the product of the
MFW recovery human error probability (5.64E-2) and the condensate injection human
error probability (3.0E-1) which was 1.69E-2. The licensee cited that these strategies
would still be available because they are dependent on the condenser hotwell, not the
CST as their inventory source.
The analyst reviewed this dependency and made the change describe in Item (3) of the
Influential Assumptions section of the NRCs internal events analysis described above
in lieu of the licensees proposed change.
Setting Basic Event CDS-XHE-XM-2HRs to 0.0: This event assumes that the MFW
system trips and no longer feeds the steam generators 30 minutes after the reactor trip.
The licensee proposed that since for the event on March 3, 2020, MFW was lost at the
time of the reactor trip, this basic event should be set to 0.0 thereby eliminating a
condition that did not occur.
The analyst consulted with INL on Basic Event CDS-XHE-XM-2HRS and the overall
recovery of a LOMFW condition in the SPAR model. From this a model change was
incorporated and new model version PVNG-EQK-HWD-FLEX-LOMFW-DEESE was
created changing this basic event to CDS-XHE-XM-LATE to reflect that this term was
also for late recovery of AFW pumps failing to run. Thus, this change was not made for
this evaluation.
Finally, in conversations with the licensee during the estimation of the ICCDP for this event, the
analyst noted the following topics which the analyst subsequently reviewed.
Use of fire water injection into a depressurized steam generator: The final available
method of feeding water into a steam generator is using a diesel driven fire water pump
to feed a depressurized steam generator. This steam generator feeding strategy is
modeled in the licensees model, but not in the SPAR model. Of note, the licensee did
credit this strategy in their risk evaluation for the 62-day emergency diesel generator
allowed outage time amendment in December 2016. The analyst noted two HRA events
in licensee documentation, 1AF-FP-BACKUP-HL, Operator Fails to Align FP to feed
SGs via AFN flowpath, from calculation 13-NS-B062, At-Power PRA Study for Human
Reliability Analysis, and 1AF-FLEX-SGHR-HL, Operator Fails to Align FP to AF cross-
connect to feed SG, from Letter 102-07411-MLL/TNW, Emergency License
Amendment Request to Extend Diesel Generator 3B Completion Time. When asked by
the analyst, the licensee replied these HRA events reflected the same action. The
analyst therefore reviewed the information from 1AF-FLEX-SGHR-HL since it was the
more recent of the two when analyzing. The calculation documents a 38 minute delay
time, a 20 minute execution time, and an available time window of 75 minutes. The
analyst considered these and judged that there would be barely enough time when
analyzing per the SPAR-H methodology. Also, the analyst considered that stress would
be high because this strategy would be the last available method of feeding steam
generators. The SPAR-H analysis estimated a failure rate of 5.05E-1. Next, the analyst
considered that the actions were close in time by the same crew which would yield at
least high dependency. These factors yielded a failure rate of 7.5E-1. Because this was
such a high failure rate and because dependency could have been considered complete,
the analyst did not credit this strategy. The analyst did however consider the possibility
of some success as a qualitative factor.
C. External Events Analysis
The analyst noted that this detailed risk assessment evaluates an actual event in which no
external events occurred. Additionally, the period of time that the events impacted plant
equipment was small enough that the probability of an external initiator occurring during this
time would be negligible. Therefore, the analyst assumed that the risk from external events,
given the subject performance deficiency was essentially zero. The licensee came to the same
conclusion external events analysis.
D. Large Early Release Frequency
The increase in large early release frequency from this finding was determined to be of very low
safety significance using Manual Chapter 0609, Appendix H, Containment Integrity Significance
Determination Process, because LOMFW sequences screen as having low safety significance
in pressurized water reactors with large dry containments.
Also, the analyst considered the possibility of consequential steam generator tube ruptures
using the draft guidance of Volume 5, Risk Analysis of Containment-Related Events (LERF), of
the RASP Handbook. Applying this guidance, the analyst estimated that the ICLERP was
1.4E-7. This value contains known conservatisms and is realistically lower when risk-informing
with qualitative factors. First, the same qualitative factors discussed for the estimate of the
increase in core damage frequency in section G below would also be relevant to the ICLERP
estimate. Additionally, the assumption of the release path to the environment in Volume 5 of the
RASP Handbook is conservative. Finally, the evaluation does not consider evacuation
effectiveness or differentiate between early and late releases which would qualitatively lower the
ICLERP estimate. Given the performance deficiency does not challenge or question the steam
generator tube integrity, further evaluation of the consequential steam generator tube rupture
results would support a very low safety significance.
E. Uncertainties
The analyst performed an uncertainties analysis on the results of the SPAR model in SAPHIRE
by running 5000 cases in a Monte Carlo analysis as allowed by SAPHIRE. The results
produced a distribution of outcomes with a point estimate of ICCDP of 1.0E-6. The median
case had an estimate of 5.5E-7. The analyst considered this skew towards lower values in the
determination of final significance.
F. Sensitivities
The analyst performed the sensitivity runs in the table below to evaluate potentially influential
factors of the evaluation.
Sensitivity
Action
Result
Removing possibility of a
consequential LOOP
Set OEP-VCF-LP-CLOPT equal to 0.0
3.7E-7
Class 1E bus lockout frequency
set to licensees value
Set ACP-ICC-FC-ESFA and -ESFB equal to
6.5E-6
1.0E-6
Class 1E bus lockout frequency
set to SAPR baseline value
Set ACP-ICC-FC-ESFA and -ESFB equal to
1.7E-3
2.0E-6
Zero test and maintenance
Ran events and condition assessment with
no test and maintenance selected
4.7E-7
Give higher success rate for
TDAFW pump operation
without DC power
Set AFW-XHE-XM-CNTRL equal to 1.0E-1
8.9E-7
Use only motor driven pumps in
common cause failure group for
AFW pumps
Set AFW-PMP-CF-FR equal to 0.0
9.6E-7
G. Qualitative Considerations
The analyst noted several qualitative considerations which were unable to be quantified and that
were used to inform the numerical estimate. They were:
Some credit for fire water injection into a depressurized steam generator could be given.
While the timing for accomplishing fire water injection was barely adequate for scenarios
where all AFW was unavailable at the beginning of the postulated core damage
sequences used in this evaluation, some credit could be considered for scenarios where
the AFW pumps run for part of their 24-hour mission time. This AFW run time would
allow more time to align fire water injection and increase the probability of its success as
a mitigative strategy.
The licensee, like many licensees, typically does not perform current maintenance
activities on similar systems. This would lower the likelihood of the licensee attempting
the recovery of the MFW panel restoration concurrently with AFW system maintenance.
Several cutsets in the results with AFW test and maintenance terms could eliminated,
lowering the increase in core damage frequency for this event.
Long-term failures-to-run of the AFW pumps were not fully modeled. For example, if an
AFW pump failed to run at 10 hours1.157407e-4 days <br />0.00278 hours <br />1.653439e-5 weeks <br />3.805e-6 months <br />, an additional 1 to 2 hours2.314815e-5 days <br />5.555556e-4 hours <br />3.306878e-6 weeks <br />7.61e-7 months <br /> would be available for
operators to employ the condensate injection strategy and therefore the failure rate of
this strategy would be lower. This effect would lower the overall increase in core
damage frequency.