ML22143A881
| ML22143A881 | |
| Person / Time | |
|---|---|
| Site: | Diablo Canyon  |
| Issue date: | 05/26/2022 |
| From: | Jason Kozal NRC/RGN-IV/DORS/PBA |
| To: | Gerfen P Pacific Gas & Electric Co |
| Shared Package | |
| ML22143A880 | List: |
| References | |
| IR 2022001 | |
| Download: ML22143A881 (22) | |
See also: IR 05000275/2022001
Text
May 26, 2022
Ms. Paula A. Gerfen, Senior Vice President,
and Chief Nuclear Officer
Pacific Gas and Electric Company
P.O. Box 56
Mail Code 104/6
Avila Beach, CA 93424
SUBJECT: DIABLO CANYON POWER PLANT, UNITS 1 AND 2 - REVISED INTEGRATED
INSPECTION REPORT 05000275/2022001 AND 05000323/2022001
Dear Ms. Gerfen:
On March 31, 2022, the U.S. Nuclear Regulatory Commission (NRC) completed an inspection
at Diablo Canyon Power Plant, Units 1 and 2. The results of this inspection were originally
issued in a report, dated May 05, 2022 (Agencywide Document Access and Management
System (ADAMS) Accession No. ML22118A936).
The NRC staff subsequently determined that due to an administrative oversight, the SUMMARY
section of the report incorrectly listed Additional Tracking Items as None, when it should have
reflected closure of the two licensee event reports (LERs) discussed in Section 71153, Follow
Up of Events and Notices of Enforcement Discretion. The revised inspection report correcting
this error is enclosed. This change had no impact on the findings documented in this report, but
consistent with NRC processes, this report is being reissued in whole.
This letter, its enclosure, and your response (if any) will be made available for public inspection
and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document
Room in accordance with Title 10 of the Code of Federal Regulations 2.390, Public
Inspections, Exemptions, Requests for Withholding.
Sincerely,
Jason W. Digitally signed by
Jason W. Kozal
Kozal Date: 2022.05.26
12:41:37 -05'00'
Jason W. Kozal, Chief
Projects Branch A
Division of Operating Reactor Safety
Docket Nos. 05000275 and 05000323
License Nos. DPR-80 and DPR-82
Enclosure:
As stated
cc w/ encl: Distribution via LISTSERV
ML22143A881 KEYWORD- NRC-002
SUNSI Review X Non-Sensitive X Publicly Available
X PVossmar Sensitive Non-Publicly Available
OFFICE SPE:DORS/A BC:DORS/A
NAME PVossmar /RA/ JKozal /RA/
DATE 5/23/2022 5/26/2022
U.S. NUCLEAR REGULATORY COMMISSION
Inspection Report
Docket Numbers: 05000275 and 05000323
License Numbers: DPR-80 and DPR-82
Report Numbers: 05000275/2022001 and 05000323/2022001
Enterprise Identifier: I-2022-001-0013
Licensee: Pacific Gas and Electric Company
Facility: Diablo Canyon Power Plant, Units 1 and 2
Location: Avila Beach, CA
Inspection Dates: January 1, 2022, to March 31, 2022
Inspectors: A. Athar, Resident Inspector
N. Day, Acting Senior Resident Inspector
M. Doyle, Operations Engineer
J. Drake, Acting Senior Resident Inspector
N. Hernandez, Acting Senior Resident Inspector
Approved By: Jason W. Kozal, Chief
Projects Branch A
Division of Operating Reactor Safety
Enclosure
SUMMARY
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring the licensees
performance by conducting an integrated inspection at Diablo Canyon Power Plant, Units 1
and 2, in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the
NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer
to https://www.nrc.gov/reactors/operating/oversight.html for more information.
List of Findings and Violations
Failure to Promptly Correct a Condition Adverse to Quality which Resulted in an Inoperable
Auxiliary Saltwater Pump
Cornerstone Significance Cross-Cutting Report
Aspect Section
Mitigating Green [H.6] - Design 71152A
Systems NCV 05000275,05000323/2022001-02 Margins
Open/Closed
A self-revealed Green, non-cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, was
identified when the licensee failed to promptly identify and correct a condition adverse to
quality. Specifically, excessive moisture and debris on the Unit 1 auxiliary saltwater pump 1-1
caused a ground on the motor that challenged the presumption of operability, and the system
was required to be declared inoperable and removed from service for troubleshooting and
corrective maintenance.
Failure to Follow Procedures Results in Inoperable Emergency Diesel Generator
Cornerstone Significance Cross-Cutting Report
Aspect Section
Mitigating Green [H.5] - Work 71153
Systems NCV 05000323/2022001-01 Management
Open/Closed
A self-revealed Green, non-cited violation of Technical Specification 5.4.1.a, was identified for
the licensees failure to implement adequate work instructions for post-maintenance testing of
the emergency diesel generator following tuning of the governor. Specifically, following
corrective maintenance for governor troubleshooting, the licensee utilized an informal work
instruction to perform tuning of the electric governor and failed to ensure that all testing
described in Procedure MP E-21.6, Diesel Generator Electrical Governor and Voltage
Regulator Adjustments, Revision 22, was identified and performed, to ensure that the
emergency diesel generator 2-3 would be able to perform its specified safety function.
Inadequate Alarm Response Procedure
Cornerstone Significance Cross-Cutting Report
Aspect Section
Initiating Events Green None (NPP) 71153
NCV 05000275,05000323/2022001-03
Open/Closed
The inspectors identified a Green, non-cited violation of Technical Specification 5.4.1.a, for
the licensee's failure to establish an adequate procedure for responding to the "FDWTR
HTRS LEVEL HI" alarm, as recommended by Section 5 of Regulatory Guide 1.33, Revision 2,
Appendix A, dated February 1978.
2
Specifically, step 2.3.6 of Annunciator Response Procedure AR PK10-16, FDWTR HTRS
LEVEL HI, Revision 8A, directed the control room crew to obtain management advice before
taking certain actions to mitigate a feedwater heater tube leak, which could result in undue
delay.
Additional Tracking Items
Type Issue Number Title Report Section Status
LER 05000323/2021-002-00 Unit 2 Manual Reactor Trip 71153 Closed
Due to Increased Water
Level in a Feedwater Heater
LER 05000323/2021-001-00 Emergency Diesel Generator 71153 Closed
Declared Inoperable due to
Low Frequency Condition
Discovery during Routine
Surveillance
3
PLANT STATUS
Unit 1 began the inspection period at rated thermal power. On February 23, 2022, Unit 1
reduced power to approximately 50 percent for planned condenser cleaning. Unit 1 returned to
rated thermal power on February 25, 2022. On March 6, 2022, Unit 1 entered end-of-fuel-cycle
coast-down in preparation for a planned refueling outage. On March 26, 2022, the unit was shut
down to enter planned Refueling Outage 23 and remained shut down for the remainder of the
inspection period.
Unit 2 operated at or near rated thermal power for the entire inspection period.
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedures (IPs) in
effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with
their attached revision histories are located on the public website at http://www.nrc.gov/reading-
rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared
complete when the IP requirements most appropriate to the inspection activity were met
consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection
Program - Operations Phase. The inspectors performed activities described in IMC 2515,
Appendix D, Plant Status, observed risk significant activities, and completed on-site portions of
IPs. The inspectors reviewed selected procedures and records, observed activities, and
interviewed personnel to assess licensee performance and compliance with Commission rules
and regulations, license conditions, site procedures, and standards.
Starting on March 20, 2020, in response to the National Emergency declared by the President
of the United States on the public health risks of the coronavirus (COVID-19), resident and
regional inspectors were directed to begin telework and to remotely access licensee information
using available technology. During this time, the resident inspectors performed periodic site
visits each week, increasing the amount of time on site as local COVID-19 conditions permitted.
As part of their onsite activities, resident inspectors conducted plant status activities as
described in IMC 2515, Appendix D; observed risk-significant activities; and completed on site
portions of IPs. In addition, resident and regional baseline inspections were evaluated to
determine if all or a portion of the objectives and requirements stated in the IP could be
performed remotely. If the inspections could be performed remotely, they were conducted per
the applicable IP. In some cases, portions of an IP were completed remotely and on site. The
inspections documented below met the objectives and requirements for completion of the IP.
REACTOR SAFETY
71111.01 - Adverse Weather Protection
External Flooding Sample (IP Section 03.03) (1 Sample)
(1) The inspectors evaluated that flood protection barriers, mitigation plans, procedures,
and equipment are consistent with the licensees design requirements and risk
analysis assumptions for coping with external flooding on March 22, 2022.
4
71111.04 - Equipment Alignment
Partial Walkdown Sample (IP Section 03.01) (4 Samples)
The inspectors evaluated system configurations during partial walkdowns of the following
systems/trains:
(1) Unit 1 residual heat removal pump 1-2 on January 24, 2022
(2) Unit 1 emergency diesel generator 1-1 starting air system on February 25, 2022
(3) Unit 2 residual heat removal pump 2-1 on March 14, 2022
(4) Unit 1 reactor vessel refueling level indication system on March 28, 2022
71111.05 - Fire Protection
Fire Area Walkdown and Inspection Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the implementation of the fire protection program by conducting a
walkdown and performing a review to verify program compliance, equipment functionality,
material condition, and operational readiness of the following fire areas:
(1) Fire on owner controlled area near 500 kV offsite line on February 2, 2022
(2) Unit 1 turbine building component cooling water heat exchanger room, 85-foot
elevation, on February 16, 2022
(3) Unit 2 turbine building component cooling water heat exchanger room, 85-foot
elevation, on March 15, 2022
(4) Unit 1 emergency diesel generator 1-1, 1-2, and 1-3 rooms, 85-foot elevation, on
March 16, 2022
(5) Unit 1 turbine building on the 119-foot elevation 4 kV rooms on March 27, 2022
Fire Brigade Drill Performance Sample (IP Section 03.02) (1 Sample)
(1) The inspectors evaluated the onsite fire brigade training and performance during an
unannounced fire drill on January 30, 2022.
71111.06 - Flood Protection Measures
Inspection Activities - Internal Flooding (IP Section 03.01) (1 Partial)
(1) (Partial)
The inspectors evaluated internal flooding mitigation protections in the Unit 2
emergency diesel generator 2-3 room on March 2, 2022. The inspectors completed
inspection procedure Sections 03.01.1 and 03.01.2.
71111.07A - Heat Exchanger/Sink Performance
Annual Review (IP Section 03.01) (1 Sample)
(1) The inspectors evaluated readiness and performance of the Unit 1 component
cooling water heat exchangers 1-1 and 1-2 thermal performance testing on
March 1, 2022.
5
71111.11A - Licensed Operator Requalification Program and Licensed Operator Performance
Requalification Examination Results (IP Section 03.03) (1 Sample)
(1) The inspectors reviewed and evaluated the licensed operator examination failure
rates for the requalification annual operating exam administered on March 29, 2022.
71111.11Q - Licensed Operator Requalification Program and Licensed Operator Performance
Licensed Operator Performance in the Actual Plant/Main Control Room (IP Section 03.01)
(1 Sample)
(1) The inspectors observed and evaluated licensed operator performance in the control
room during the Unit 1 shutdown for Refueling Outage 1R23 on March 27, 2022.
Licensed Operator Requalification Training/Examinations (IP Section 03.02) (1 Sample)
(1) The inspectors observed and evaluated licensed operator annual exams on
February 9, 2022.
71111.12 - Maintenance Effectiveness
Maintenance Effectiveness (IP Section 03.01) (1 Partial)
(1) (Partial)
The inspectors evaluated the effectiveness of maintenance to ensure the Unit 2
condensate system remained capable of performing its intended function. The
inspectors completed inspection procedure Sections 03.01.1, 03.01.2, and 03.01.4.
Quality Control (IP Section 03.02) (1 Sample)
The inspectors evaluated the effectiveness of maintenance and quality control activities to
ensure the following structure, system, or component remains capable of performing its
intended function:
(1) Emergency diesel generator lube oil storage on January 7, 2022
71111.13 - Maintenance Risk Assessments and Emergent Work Control
Risk Assessment and Management Sample (IP Section 03.01) (4 Samples)
The inspectors evaluated the accuracy and completeness of risk assessments for the
following planned and emergent work activities to ensure configuration changes and
appropriate work controls were addressed:
(1) Unit 1 emergency diesel generator 1-1 major maintenance outage window on
January 12, 2022
(2) Unit 2 concurrently exercising boration pathway valves on February 15, 2022
(3) Unit 1 condenser differential pressure emergent trend on February 15, 2022
(4) Unit 1 emergency diesel generator 1-2 major maintenance outage window on
February 24, 2022
6
71111.15 - Operability Determinations and Functionality Assessments
Operability Determination or Functionality Assessment (IP Section 03.01) (4 Samples)
The inspectors evaluated the licensee's justifications and actions associated with the
following operability determinations and functionality assessments:
(1) Unit 1 emergency diesel generator 1-2 lube oil heat exchanger leak on
January 25, 2022
(2) Emergency plan equipment discussed in Notification 51143190 and 51143191 on
February 9, 2022
(3) Operability of concurrent outage of containment fan cooler unit 1-4 and emergency
diesel generator 1-2 on February 18, 2022
(4) Unit 1 residual heat removal valve RHR-1-8072 following discovery of through wall
leak on leak-off line RHR-Line-2524 on March 31, 2022
71111.18 - Plant Modifications
Temporary Modifications and/or Permanent Modifications (IP Section 03.01 and/or 03.02)
(1 Sample)
The inspectors evaluated the following temporary or permanent modifications:
(1) Installation of a sixth emergency diesel generator on February 17, 2022
71111.19 - Post-Maintenance Testing
Post-Maintenance Test Sample (IP Section 03.01) (5 Samples)
The inspectors evaluated the following post-maintenance testing activities to verify system
operability and/or functionality:
(1) Unit 1 emergency diesel generator 1-1 major maintenance outage window on
January 14, 2022
(2) Unit 2 component cooling water pump 2-3 maintenance outage window on
January 18, 2022
(3) Unit 2 component cooling water heat exchanger 2-1 and auxiliary saltwater pump 2-1
maintenance outage window on January 31, 2022
(4) Unit 1 control room ventilation maintenance outage window on February 9, 2022
(5) Unit 1 emergency diesel generator 1-2 major maintenance outage window on
February 26, 2022
71111.20 - Refueling and Other Outage Activities
Refueling/Other Outage Sample (IP Section 03.01) (1 Partial)
(1) (Partial)
The inspectors evaluated Refueling Outage 1R23 activities from March 26, 2022, to
March 31, 2022. The inspectors completed inspection procedure Sections 3.01.a
and 3.01.b..
7
71111.22 - Surveillance Testing
The inspectors evaluated the following surveillance testing activities to verify system operability
and/or functionality:
Surveillance Tests (other) (IP Section 03.01) (5 Samples)
(1) Unit 2 emergency diesel generator 2-2 routine surveillance test on January 20, 2022
(2) Observation of operator rounds on February 3, 2022
(3) Unit 1 safety bus G 4 kV under voltage testing per Work Order 64219827 on
February 22, 2022
(4) Unit 1, Work Order 64253229, I-7-P456. A pressurizer power-operated relief valve
calibration on February 28, 2022
(5) Unit 1, 4 kV, bus G non-safety injection auto-transfer test on March 27, 2022
Inservice Testing (IP Section 03.01) (1 Sample)
(1) Unit 1 containment spray pump 1-1 routine surveillance test on January 28, 2022
71114.06 - Drill Evaluation
Select Emergency Preparedness Drills and/or Training for Observation (IP Section 03.01)
(1 Sample)
(1) The inspectors evaluated a tabletop emergency preparedness drill in the Emergency
Operation Facility on January 25, 2022.
OTHER ACTIVITIES - BASELINE
71151 - Performance Indicator Verification
The inspectors verified licensee performance indicator submittals listed below:
IE01: Unplanned Scrams per 7000 Critical Hours Sample (IP Section 02.01) (2 Samples)
(1) Unit 1 (January 1, 2021, through December 31, 2021)
(2) Unit 2 (January 1, 2021, through December 31, 2021)
IE03: Unplanned Power Changes per 7000 Critical Hours Sample (IP Section 02.02)
(2 Samples)
(1) Unit 1 (January 1, 2021, through December 31, 2021)
(2) Unit 2 (January 1, 2021, through December 31, 2021)
IE04: Unplanned Scrams with Complications (USwC) Sample (IP Section 02.03) (2 Samples)
(1) Unit 1 (January 1, 2021, through December 31, 2021)
(2) Unit 2 (January 1, 2021, through December 31, 2021)
8
71152A - Annual Follow-up Problem Identification and Resolution
Annual Follow-up of Selected Issues (Section 03.03) (1 Sample)
The inspectors reviewed the licensees implementation of its corrective action program
related to the following issues:
(1) Unit 1 auxiliary saltwater pump 1-1 multiple ground alarms on March 10, 2022. The
inspector completed Section 3.0.3. This completes the sample that was partially
completed and documented in Integrated Inspection Report 05000275/2021004 and
05000323/2021004 (ADAMS Accession No. ML22024A495).
71153 - Follow Up of Events and Notices of Enforcement Discretion
Event Report (IP Section 03.02) (2 Samples)
The inspectors evaluated the following licensee event reports (LERs):
(1) LER 05000323/2021-001-00, Emergency Diesel Generator Declared Inoperable due
to Low Frequency Condition Discovery during Routine Surveillance (ADAMS
Accession No. ML21263A062). The inspection conclusions associated with this LER
are documented in this report under Inspection Results Section 71153.
(2) LER 05000323/2021-002-00, Unit 2 Manual Reactor Trip Due to Increased Water
Level in a Feedwater Heater (ADAMS Accession No. ML21348A112). The inspection
conclusions associated with this LER are documented under Inspection Results
Section 71153.
INSPECTION RESULTS
Failure to Promptly Correct a Condition Adverse to Quality which Resulted in an Inoperable
Auxiliary Saltwater Pump
Cornerstone Significance Cross-Cutting Report
Aspect Section
Mitigating Green [H.6] - Design 71152A
Systems NCV 05000275,05000323/2022001-02 Margins
Open/Closed
A self-revealed Green, non-cited violation of 10 CFR Part 50, Appendix B, Criterion XVI, was
identified when the licensee failed to promptly identify and correct a condition adverse to
quality. Specifically, excessive moisture and debris on the Unit 1 auxiliary saltwater pump 1-1
caused a ground on the motor that challenged the presumption of operability, and the system
was required to be declared inoperable and removed from service for troubleshooting and
corrective maintenance.
Description: The auxiliary salt water (ASW) system design includes two safety-related motor
driven pumps that draw water from the Pacific Ocean to provide safety-related cooling water
for heat removal. The pump motors are housed in an enclosed vault adjacent to the Pacific
Ocean that includes forced ventilation through a raised snorkel/intake riser. Through the
inherent design of the vault, the ventilation introduces salty air into the environment where the
motor is housed on top of the auxiliary saltwater pump.
During normal operations, Diablo Canyon Power Plant does not need to operate both
auxiliary saltwater pumps. In order to perform scheduled work and minimize runtime wear,
9
the auxiliary saltwater pumps are swapped between a standby and running pump as
appropriate (typically each week). While in the standby (non-running) state, forced air is still
provided into the room for appropriate environmental temperatures. To support readiness and
long-term reliability, large motors typically have nonsafety-related components. A typical
component that supports the readiness in moisture environments is a motor heater. The
inspectors note that, although the heater itself is not safety-related, the motor is. In the case
of standby operations (nonrunning no heat/drying for moisture removal), excessive moisture
builds up that challenges the motor, which is a safety-related component and can affect the
reliability and operability of the ASW system.
In September 2019, during routine maintenance, Diablo Canyon identified that the motor
heater for ASW pump 1-1 was degraded. In January 2020, the station established a monthly
monitoring plan to observe the temperature difference between the stator and the lower
bearing; ultimately this monitoring strategy was ineffective due to the relatively long time
between evaluations of these parameters. The heater continued to degrade and completely
failed July 1, 2021. The degradation of the motor heater allowed elevated levels of
contamination and moisture buildup on the Auxiliary Saltwater Pump 1-1 motor windings,
which went undetected by the monthly monitoring plan. Over time, the contamination and
moisture accumulated enough to cause a ground fault in the Auxiliary Saltwater Pump 1-1
motor.
On July 5, 2021, the scheduled train swap occurred, with a routine start of the ASW
pump 1-1. The pump started and ran; however, almost immediately, unexpected main control
room ground detection system annunciators alarmed, and the operators were required to
secure the running pump due to a detected ground. Operators declared the pump inoperable,
and troubleshooting commenced. Insulation performance testing revealed excessive moisture
and debris on the pump motor, which spectroscopy confirmed to be sea salt debris. The
pump motor was replaced with a spare motor and the licensee performed post-service
analysis on the motor that was removed from service. The licensees analysis determined
that if the motor was needed in an accident scenario, the motor ground would have cleared
due to self-heating and moisture removal; nonetheless, this issue rendered the system
inoperable and unavailable for 75 hours8.680556e-4 days <br />0.0208 hours <br />1.240079e-4 weeks <br />2.85375e-5 months <br />.
Corrective Actions: Diablo Canyon Power Plant replaced the motor with a spare auxiliary
saltwater pump motor and declared ASW pump 1-1 operable on July 8, 2021, performed a
root cause analysis, revised equipment reliability process procedures to drive management
review of risk, created plant computer monitoring points for auxiliary saltwater pump motor
heaters, and revised the conduct of engineering and technical evaluation procedures to add
specific guidance to improve adverse condition monitoring.
Corrective Action References: Notification 5114756
Performance Assessment:
Performance Deficiency: Diablo Canyon Power Plant Unit 1 failed to promptly identify and
correct moisture and debris build up on the Unit 1 ASW pump 1-1 motor internal windings
which is a condition adverse to quality. Failing to identify and correct a condition adverse to
quality is a performance deficiency that was within the licensees ability to foresee and
correct.
Screening: The inspectors determined the performance deficiency was more than minor
because it was associated with the Equipment Performance attribute of the Mitigating
10
Systems cornerstone and adversely affected the cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events to prevent
undesirable consequences. Specifically, the licensees failure to promptly identify and correct
moisture and debris build up on the Unit 1 ASW pump 1-1 motor internal windings caused
75 hours8.680556e-4 days <br />0.0208 hours <br />1.240079e-4 weeks <br />2.85375e-5 months <br /> of unavailability of the ASW pump 1-1.
Significance: The inspectors assessed the significance of the finding using Inspection Manual
Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-
Power. Using Exhibit 2, Mitigating System Screening Questions, the finding was screened
as having very low safety significance (Green) because it was not a design deficiency; did not
represent a loss of system function; did not represent an actual loss of function of at least a
single train for longer than its technical specification allowed outage time; and did not result in
the loss of a high safety-significant nontechnical specification train.
Cross-Cutting Aspect: H.6 - Design Margins: The organization operates and maintains
equipment within design margins. Margins are carefully guarded and changed only through a
systematic and rigorous process. Special attention is placed on maintaining fission product
barriers, defense-in-depth, and safety related equipment. The inspectors noted that Diablo
Canyon found degraded current on the auxiliary saltwater pump motor heater in
September 2019. However, the licensee failed to replace the motor heater before it failed in
July 2021, even when presented with opportunities to replace it during a refuel outage.
During this time, the station justified not replacing the heater and failed to maintain plant
equipment with appropriate design margin.
Enforcement:
Violation: Title 10 CFR Part 50, Appendix B, Criterion XVI, requires, in part, conditions
adverse to quality are promptly identified and corrected.
Contrary to the above, on July 5, 2021, Diablo Canyon Power Plant failed to promptly identify
and correct a condition adverse to quality. Specifically, the excessive moisture and debris on
the Unit 1 auxiliary saltwater pump 1-1, a condition adverse to quality, caused a ground on
the motor that challenged the presumption of operability and required the system to be
declared inoperable and removed from service for troubleshooting and corrective
maintenance.
Enforcement Action: This violation is being treated as a non-cited violation, consistent with
Section 2.3.2 of the Enforcement Policy.
11
Failure to Follow Procedures Results in Inoperable Emergency Diesel Generator
Cornerstone Significance Cross-Cutting Report
Aspect Section
Mitigating Green [H.5] - Work 71153
Systems NCV 05000323/2022001-01 Management
Open/Closed
A self-revealed Green, non-cited violation of Technical Specification 5.4.1.a, Procedures,
was identified for the licensees failure to establish and implement adequate work instructions
for post-maintenance testing of the emergency diesel generator (EDG) following tuning of the
governor. Specifically, following corrective maintenance for governor troubleshooting, the
licensee utilized an informal work instruction to perform tuning of the electric governor and
failed to ensure that all testing described in Procedure MP E-21.6, Diesel Generator
Electrical Governor and Voltage Regulator Adjustments, Revision 22, was identified and
performed, to ensure that the EDG 2-3 would be able to perform its specified safety function.
Description: On June 28, 2021, EDG 2-3 was removed from service for governor tuning.
EDG 2-3 had been experiencing load and frequency oscillations during its last two
surveillances, and the licensee had determined that the governor control system had been
impacted by fuel rack resistance changes from previous fuel system maintenance. The
personnel scheduled to perform the governor tuning had not previously performed this work.
However, only an informal work instruction was prepared to provide guidance for the tuning
and post-maintenance testing. Due to the complex nature of the governor tuning work scope,
the work order instructions were left vague with detailed actions contained in an informal
guidance document. The maintenance work order instructions were to adjust the governor
controls "per Engineering direction."
Since the document providing direction was an informal work plan, it was not required to be
place kept, and place keeping was not performed. Governor tuning maintenance was
completed on June 29, 2021; however, a fuel oil leak developed and testing was delayed until
the next shift. The maintenance verification testing was scoped, but due to insufficient detail
and lack of place keeping, the as-left isochronous (ISOC) frequency set-point on the governor
was not verified. EDG 2-3 was declared operable and placed in standby.
On July 21, 2021, EDG 2-3 was started on a simulated undervoltage signal for routine
monthly surveillance testing per Procedure STP M-9A3. The EDG was started with the mode
select switch in the "AUTO" position which indicated the diesel was aligned for dedicated
supply to its associated 4 kV vital electrical bus. Upon starting, the as-read steady-state
frequency was 58.9 Hz which was outside of the required technical specification acceptance
criteria of 59.2 Hz to 60.8 Hz. This resulted in a failed surveillance and the EDG being
declared inoperable since June 28, 2021. Engineering determined that the governor as-left
ISOC frequency set-point was the problem. The EDG 2-3 ISOC frequency set-point was
restored within limits per Procedure MP E-21.6. The EDG 2-3 passed all post-maintenance
testing requirements and was declared operable on July 22, 2021, at 10:55 p.m.
Corrective Actions: The licensee restored the correct governor settings, issued licensee event
report (LER) 05000323/2021-001-00, "Emergency Diesel Generator Declared Inoperable due
to Low Frequency Condition Discovery during Routine Surveillance," (ADAMS Accession
No. ML21263A062), updated Maintenance Procedure MP E-21.6 to include a note about
partial procedure use and impacts on ISOC frequency to ensure sufficient maintenance
verification testing, and initiated action to provide EDG governor control system training for
select system material experts.
12
Corrective Action References: Notification 51113496
Performance Assessment:
Performance Deficiency: The inspectors determined that the licensees failure to provide
adequate work instructions and follow procedures related to tuning and testing the EDG 2-3
governor was a performance deficiency that was reasonably within the licensees ability to
foresee and should have been prevented. Specifically, use of an informal work instruction to
perform complex corrective maintenance on the governor resulted in EDG 2-3 being
inoperable for greater than the technical specification allowed outage time.
Screening: The inspectors determined the performance deficiency was more than minor
because it was associated with the Equipment Performance attribute of the Mitigating
Systems cornerstone and adversely affected the cornerstone objective to ensure the
availability, reliability, and capability of systems that respond to initiating events to prevent
undesirable consequences. Specifically, the failure to provide adequate guidance for work on
the tuning and post maintenance testing of the governor resulted in EDG 2-3 being
inoperable for 25 days.
Significance: The inspectors assessed the significance of the finding using Inspection Manual
Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-
Power. Using Exhibit 2, Mitigating System Screening Questions, the finding was screened
as having very low safety significance (Green) because it was not a design deficiency; did not
represent a loss of system and/or train function; did not represent an actual loss of function of
at least a single train for longer than its technical specification allowed outage time; and did
not result in the loss of a high safety-significant nontechnical specification train. EDG 2-3
maintained its ability to perform its safety function.
Cross-Cutting Aspect: H.5 - Work Management: The organization implements a process of
planning, controlling, and executing work activities such that nuclear safety is the overriding
priority. The work process includes the identification and management of risk commensurate
to the work and the need for coordination with different groups or job activities. Specifically,
the licensee used an informal work instruction rather than developing a formalized
procedure/guidance for the operators to complete the complex task of tuning and testing the
EDG 2-3 governor.
Enforcement:
Violation: Technical Specification Section 5.4.1.a states, in part, that written procedures shall
be established, implemented, and maintained covering the applicable procedures
recommended in Regulatory Guide 1.33, Revision 2, Appendix A, February 1978. NRC
Regulatory Guide 1.33, Appendix A, Section 9.a, recommends that maintenance that can
affect the performance of safety-related equipment should be properly preplanned and
performed in accordance with written procedures, documented instructions, or drawings
appropriate to the circumstances.
Contrary to the above, on June 28, 2021, the licensee failed to ensure that maintenance that
could affect the performance of safety-related equipment was properly preplanned and
performed in accordance with written procedures, documented instructions, or drawings
appropriate to the circumstances. Specifically, the licensee utilized an informal work
instruction to perform the complex task of tuning and testing the electric governor for EDG 2-3
and failed to complete all required post-maintenance testing required to demonstrate that
13
structures, systems, and components will perform satisfactorily in service. As a result, EDG 2-
3 was inoperable for approximately 25 days.
Enforcement Action: This violation is being treated as a non-cited violation, consistent with
Section 2.3.2 of the Enforcement Policy.
Inadequate Alarm Response Procedure
Cornerstone Significance Cross-Cutting Report
Aspect Section
Initiating Events Green None (NPP) 71153
NCV 05000275,05000323/2022001-03
Open/Closed
The inspectors identified a Green, non-cited violation of Technical Specification 5.4.1.a, for
the licensee's failure to establish an adequate procedure for responding to the "FDWTR
HTRS LEVEL HI" alarm, as recommended by Section 5 of Regulatory Guide 1.33,
Revision 2, Appendix A, dated February 1978. Specifically, step 2.3.6 of Annunciator
Response Procedure AR PK10-16, FDWTR HTRS LEVEL HI, Revision 8A, directed the
control room crew to obtain management advice before taking certain actions to mitigate a
feedwater heater tube leak, which could result in undue delay.
Description: On October 15, 2021, at 8:49 a.m., the Unit 2 feedwater heaters high level
(FDWTR HTRS LEVEL HI) annunciator alarmed in the control room. Based on multiple
diverse indications, station personnel verified excessive condensate water on the shell side of
feedwater heater 2-5B.
The station entered Annunciator Response Procedure AR PK10-16, FDWTR HTRS LEVEL
HI, Revision 8A, upon receiving the alarm in the control room. Step 2.2.3 of Procedure AR
PK 10-16 outlines a methodology to determine whether the high level condition is the result of
a level control system malfunction or a feedwater heater tube leak. By 12:26 p.m. that day,
operations personnel had completed this step and had determined that there was no
malfunction of the level control system, indicating the presence of a tube leak in the
feedwater heater 2-5B. At 12:33 p.m., the instrumentation and controls maintenance
supervisor had concurred that there was no malfunction of the level control system. Thus, the
station had strong evidence at 12:33 p.m. that there was a tube leak in feedwater heater 2-
5B.
Step 2.3.6 of Procedure AR PK 10-16 provides instructions in the event that a tube leak has
been diagnosed. This step directs the station to perform either or both of the following, as
deemed advisable by plant management after an assessment of event severity:
- Conduct an orderly plant shutdown
- Dispatch an operator to isolate the affected feedwater heater
This step was entered at 12:33 p.m., and a plant management meeting occurred at 3:00 p.m.
to discuss the options listed. The station ultimately decided to dispatch an operator to isolate
feedwater heater 2-5B, which first required a downpower to 90 percent due to thermodynamic
considerations of taking a feedwater heater out of service. The inspectors noted that the
downpower began at 4:09 p.m.almost 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> after the tube leak was initially diagnosed.
The downpower was completed at 5:12 p.m., and an operator was then dispatched to isolate
feedwater heater 2-5B.
14
While the operator was attempting the close the condensate outlet isolation valve, the level in
the sight glass for feedwater heater 2-5B continued to rise, until it was off-scale (out of sight)
high. As required by procedure, the control room crew manually tripped the Unit 2 reactor due
to the feedwater heater 2-5B being off-scale high. Immediately following the trip, the crew
closed the main steam isolation valves (MSIVs) per Procedure AR PK 10-16. This action also
isolated the main feedwater pumps, which are steam-driven.
The inspectors noted that step 2.3.6 of Procedure AR PK 10-16 directs the station to obtain
management advice before taking action to mitigate the feedwater heater tube leak. On
October 15, the station gathered a senior management meeting to complete this step, before
taking action to mitigate the feedwater heater tube leak. After the downpower was completed
and an operator was dispatched to isolate the affected feedwater heater, the operator was
unable to complete the isolation of feedwater heater 2-5B before the feedwater heater level
went off-scale (out of sight) high and the control room crew had to manually trip the reactor.
During the inspection, the licensee provided additional information including timelines
describing activities occurring in parallel with plant management meeting preparations,
evidence that a plant trip was unavoidable due to the extensive FWH damage in this case,
and operating experience suggesting this event progressed significantly more rapidly than
past events. Upon review of this information, the inspectors concluded that there was not
sufficient basis to conclude the performance deficiency caused the October 15 plant trip and
loss of the condenser and feedwater pumps. However, the inspectors concluded that
Procedure AR PK 10-16 was inadequate because step 2.3.6 directed control room operators
to gain management input before taking action to mitigate the feedwater heater tube leak,
which could take the decision-making out of the control room and incur an undue delay.
Corrective Actions: The licensee removed the statement "as deemed advisable by plant
management," from step 2.3.6 of Procedure AR PK 10-16, and instead directed the control
room operators to take action to mitigate a feedwater heater tube leak as directed by the shift
manager.
Corrective Action References: Notifications 51148759 and 51145638
Performance Assessment:
Performance Deficiency: The failure to establish an adequate procedure for responding to the
"FDWTR HTRS LEVEL HI" alarm in accordance with the requirements of plant technical
specifications was a performance deficiency that was within the licensee's ability to foresee
and correct.
Screening: The inspectors determined the performance deficiency was more than minor
because if left uncorrected, it would have the potential to lead to a more significant safety
concern. Specifically, the undue delay caused by convening a management meeting may
lead to a more significant safety concern while implementing this procedure.
Significance: The inspectors assessed the significance of the finding using Inspection Manual
Chapter 0609, Appendix A, The Significance Determination Process (SDP) for Findings At-
Power. Using Exhibit 1, Initiating Events Screening Questions, the finding was screened as
having very low safety significance (Green) because it did not cause a reactor trip.
Cross-Cutting Aspect: Not Present Performance. No cross-cutting aspect was assigned to
this finding because the inspectors determined the finding did not reflect present licensee
15
performance. Specifically, the latest revision of Annunciator Response Procedure AR PK10-
16, FDWTR HTRS LEVEL HI, Revision 8A, was issued on October 12, 2016, which was
outside the 3-year window normally considered to be present performance.
Enforcement:
Violation: Technical Specification 5.4.1.a requires, in part, that written procedures shall be
established, implemented, and maintained covering the applicable procedures recommended
in Regulatory Guide 1.33, Revision 2, Appendix A, dated February 1978. Regulatory
Guide 1.33, Revision 2, Appendix A, Section 5, identifies procedures pertaining to alarm
conditions as recommended procedures.
Contrary to the above, prior to October 15, 2021, the licensee failed to establish adequate
written procedures to respond to the FDWTR HTRS LEVEL HI alarm. Specifically,
Annunciator Response Procedure AR PK10-16, FDWTR HTRS LEVEL HI, Revision 8A,
Step 2.3.6, directed control room operators to gain management input before taking action to
mitigate a feedwater heater tube leak, which could take the decision-making out of the control
room and incur an undue delay.
Enforcement Action: This violation is being treated as a non-cited violation, consistent with
Section 2.3.2 of the Enforcement Policy.
EXIT MEETINGS AND DEBRIEFS
The inspectors verified no proprietary information was retained or documented in this report.
- On April 7, 2022, the inspectors presented the integrated inspection results to
Ms. P. Gerfen, Senior Vice President, Generation and Chief Nuclear Officer, and other
members of the licensee staff.
16
DOCUMENTS REVIEWED Chief Nuclear Officer
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.01 Procedures CP M-16 Severe Weather 20
71111.04 Corrective Action Notifications 51141818, 51141819, 51141871, 51141890
Documents
71111.04 Procedures OP A-2:X RVRLIS Alignments for Refueling Outages 10
71111.04 Procedures OP B-2:I RHR System Alignment Verification for Plant Startup 26
71111.05 Corrective Action Notifications 51142578
Documents
71111.05 Corrective Action Notifications 51071040, 51094718, 51132042
Documents
71111.05 Drawings 111805-49 Fire Preplan Turbine Building Elevation 85' 3
71111.05 Drawings 11805-54 Fire Preplan Turbine Building Elevation 119' 3
71111.05 Miscellaneous Generic Safety
Plan GPAAHHR
and GPAASITE
71111.05 Procedures Generic Hot Work Fire Lesson Plan 4
71111.07A Corrective Action Notifications 51144881, 51144943, 51144976
Documents
71111.07A Procedures PEP M-234 CCW Heat Exchanger Performance Test 21
71111.07A Work Orders WO 64187929, 64187939
71111.11A Miscellaneous ARQ results ARQ Results 03/28/2022
71111.11Q Procedures E-0 Reactor Trip or Safety Injection 45D
71111.11Q Procedures E-0.1 Reactor Trip Response 43
71111.11Q Procedures OP L-4 Normal Operation at Power 103
71111.12 Miscellaneous PG&E Letter No. Diablo Canyon Units 1 and 2 License Amendment Request 02/14/1992
DCL 92-036, 92-03, Revision of Technical Specifications 3/4.8.1 and
Docket No. 50- 3/4.8.2 Increase Emergency Diesel Generator Fuel Oil
275, OL-DPR-80, Storage Requirements
Docket No. 50-
323, OL-DPR-82
71111.12 Miscellaneous PG&E Letter No. Diablo Canyon Units 1 and 2 Response to NRC Questions 06/05/1992
DCL 92-131 on License Amendment Request 92-03, Revision of
Docket No. 50- Technical Specifications 3/4.8.1 and 3/4.8.2 Increase
17
Inspection Type Designation Description or Title Revision or
Procedure Date
275, OL-DPR-80, Emergency Diesel Generator Fuel Oil Storage Requirements
Docket No. 50-
323, OL-DPR-82
71111.12 Miscellaneous SISI Manual Seismically Induced Systems Interaction Manual 13
71111.12 Miscellaneous STP M-9G Diesel Generator 24 Hour Load Test and Hot Restart Test 04/22/2017
Completed
Surveillance
Results
71111.12 Miscellaneous STP M-9G Diesel Generator 24 Hour Load Test and Hot Restart Test 04/19/2019
Completed
Surveillance
Results
71111.12 Miscellaneous STP M-9G Diesel Generator 24 Hour Load Test and Hot Restart Test 05/14/2021
Completed for 4-22-2017, 4-19-2019, 5-14-21
Surveillance
Results
71111.12 Procedures Behavior Of Loose Equipment In A Seismic Event 06/1985
71111.12 Procedures STP M-9G Diesel Generator 24 Hour Load Test and Hot Restart Test 59
71111.13 Corrective Action Notifications 51139208, 51140737, 51140750, 51140937, 51140954,
Documents 51141885
71111.13 Miscellaneous Unit 1 Condenser Station Prediction of Condenser DP 02/15/2022
DP Trends
71111.13 Procedures AD7.ID14 Assessment of Integrated Risk 29
71111.13 Procedures OM7.ID7 Emerging Issue and Event Investigations
71111.13 Procedures OP O-36 Protected Equipment Postings 27
71111.13 Procedures OP1.DC10 Conduct of Operations 65
71111.13 Procedures OP1.ID3 Reactivity Management Program 18
71111.13 Procedures STP V-3E1 Exercising Valve FCV-110A Boric Acid to Blender 22
71111.13 Procedures STP V-3E2 Exercising Valve FCV-110B Blender to Charging Header 10
71111.15 Corrective Action Notifications 50985660, 51123152, 51131514, 51135585, 51147276
Documents
71111.15 Drawings 102010-3 Residual Heat Removal System 52
71111.15 Drawings 102028-20 ASME Code Boundary 53
71111.15 Drawings 224757-1 Residual Heat Removal System Suction Loop 4 V-8702 3
18
Inspection Type Designation Description or Title Revision or
Procedure Date
Leak-off & Thermal Expansion
71111.15 Miscellaneous 9000039632 Emergency Diesel Generator Fuel Oil Storage 0
71111.15 Procedures EP MT-21 Emergency Kits 19
71111.15 Procedures MP M-21-HX.2 Diesel Generator Lube Oil Heat Exchanger Maintenance 8
71111.18 Calculations Calc M-786 Determine Required Diesel Fuel Oil Storage to Meet DCPP 6
Licensing Basis of Operating "Minimum ESF Loads"
71111.18 Calculations Calc M-786 Determine Required Diesel Fuel Oil Storage to Meet DCPP 8
Licensing Basis of Operating "Minimum ESF Loads"
71111.18 Engineering DCN DC2-EM- Install sixth Diesel Engine Generator 1
Changes 44405
71111.18 Engineering DCN DC2-EM- Diesel Engine Generator 2-3 0
Changes 46476
71111.18 Engineering DCP M-44405 Install sixth Diesel Engine Generator 3
Changes
71111.19 Corrective Action Notifications 51140355, 51140689, 51140691, 51140700, 51140702,
Documents 51140708, 51140711, 51140724, 51140732, 51140733,
51140734, 51140735, 51140736, 51140738, 51140739,
51140780, 51140784, 51140793, 51140818, 51140856,
51140857, 51140870, 51140882, 51140883, 51141044,
51141058, 51141062, 51141078, 51141379, 51142497,
51142521, 51142533
71111.19 Work Orders WO 60130870, 64131387, 64131388, 64132042, 64132052,
64174747, 64213744, 64222825, 64238407
71111.20 Procedures OP B-1A CVCS Borate the RCS to Refuel 3
71111.20 Procedures OP B-9:I Primary Sampling - Make System Available 13
71111.20 Procedures OP L-5 Plant Cooldown from Minimum Load to Cold Shutdown 110
71111.20 Procedures OP L-6 Cold Shutdown Refueling 88
71111.22 Corrective Action Notifications 51141538, 51142224, 51147252, 51147253, 51147446
Documents
71111.22 Miscellaneous U1 Auxiliary Building Round for February 03, 2022, and U2
Turbine Building Rounds for February 04, 2022
71111.22 Procedures STP M-13G 4 kV Bus G Non-SI Auto-Transfer Test 61
71111.22 Procedures STP M-16A Slave Relay Test of Trains A and B K603 (Safety Injection) 30
71111.22 Procedures STP M-9A2 Diesel Engine Generator 2-2 Routine Surveillance Test 13
19
Inspection Type Designation Description or Title Revision or
Procedure Date
71111.22 Procedures STP P-CSP-11 Routine Surveillance Test of Containment Spray Pump 1-1 21A
71111.22 Procedures STP V-3I3A Full Stroke Exercise of Containment Spray Valve CS-9001A 8
71111.22 Work Orders WO 64172567, 64218102, 64252011, 64252019, 64253229,
64261233
71114.06 Corrective Action Notifications 51141937
Documents
71151 Corrective Action Notifications 51142375
Documents
71152A Corrective Action Notification and 5114756
Documents Causal
Evaluation
71153 Corrective Action Notifications 51088141, 51115230, 51117574, 51126031, 51126111,
Documents 51133450, 51133724
71153 Procedures AD13.ID4 Post Maintenance Testing 37
71153 Procedures AR PK 10-16 FDWTR HTRS LEVEL HI Annunciator Response Procedure 8A
71153 Procedures MP M-21.8 Diesel Engine Governor Actuator Maintenance 26
71153 Procedures MP_E-21.6 Diesel Generator Electrical Governor and Voltage Regulator 22
Adjustments
71153 Procedures MP_E-21.GOV Emergency Diesel Generator Electrical Governor 4
Maintenance
71153 Procedures OP C-7:IV Condensate System - Feedwater Heaters Remove from 11
Service
71153 Work Orders WO 60131137, 60136440, 60136888, 60138461, 60138932
20