ML22028A392

Licensee Regulatory Conference Presentation

ML22028A392
Person / Time
Site:	Davis Besse
Issue date:	02/01/2022
From:	Energy Harbor Nuclear Corp
To:	NRC/RGN-III
Shared Package
ML22028A384	List: ML22028A392 ML22028A397
References
Download: ML22028A392 (37)

Text

Regulatory Conference Related to Davis-Besse EDG FFSS Finding February 1, 2022

Purpose of Meeting

• Provide information and perspective on the uncertainties with significance determination related to the 5 areas outlined in the Inspection Report Preliminary Greater Than Green Finding Dated Dec 16, 2021

• Provide information and perspective on the failure analysis performed on the Field Flash Selector Switch (FFSS) after the inspection team was on site.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 2

Agenda

• Apparent Violation and Performance Deficiency

• Licensee Perspective

• Summary of FFSS Failure

• PRA Considerations

• Independent Failure Analysis

• Conclusions

• Questions and Wrap-up Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 3

Apparent Violation & Performance Deficiency

• On May 27, 2021, a self-revealed finding with its safety significance as yet to be determined (TBD) and an associated Apparent Violation (AV) of TS 5.4.1.a were identified for the licensees apparent failure to develop a preventive maintenance schedule for the inspection of emergency diesel generator (EDG) field flash selector switches (FFSSs)

• The Team concluded that the failure to develop a preventive maintenance schedule for the inspection of the FFSS, was contrary to Technical Specification 5.4.1 and Section 9.b of Regulatory Guide 1.33, Revision 2, February 1978, and constituted a performance deficiency

• Preliminary Greater Than Green Finding - The failure to inspect the switch contributed to the long-term degradation of the switch electrical contacts and ultimately contributed to an EDG failure during fast start testing Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 4

Licensee Perspective

• Energy Harbor staff determined the significance of the May 27th event could be characterized as Low to Moderate (White) based on PRA considerations

• Independent failure analysis of the FFSS performed after the on-site portion of the NRC inspection ruled out silver sulfidation and determined foreign material (FM) as the most likely cause

• The results of the failure analysis led Davis-Besse to change our perspective from the initial investigation:

• Davis-Besse staff has determined that a PM to inspect the contacts of the FFSS would likely have not prevented the May 27th event

• Davis-Besse staff perspective is that the significance of the performance deficiency associated with the FFSS should not be based on the May 27th event Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 5

Summary of EDG#1 FFSS Failure 5/27/21 Fast-Start Surveillance Test - EDG 1 184 Day Test

• Performed with the FFSS in the 400 RPM (Emergency Start) position

• The normal position of the switch is on the 400 RPM position

• The monthly test for the EDG moves the switch to the 800 RPM (Idle Start) position for the test and then moves it back to the 400 RPM position

• Failed to reach a rated voltage and frequency

• Engine started as expected and reached 900 RPM

• No Indications that the field flashed

• Decision made to stop the test and shut down the engine

• When the FFSS was placed into the 800 RPM position - the field flashed, proper voltage & frequency observed

• The idle shutdown continued, and the engine was stopped to allow troubleshooting Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 6

Summary of EDG#1 FFSS Failure Problem Solving Team

• Discovered intermittent open contact on the FFSS 400 RPM contacts

• Initial in-situ check showed closed contact

• The switch was cycled 5 times in-situ, 2 of 5 times open contact observed

• Replaced FFSS and tested satisfactorily on 5/28/21

• Surface contamination is visually observed on the FFSS contacts

• The vendor manual recommended inspection and cleaning to remove sulfide buildup and contact pitting - no periodicity specified

• No PM existed to inspect/clean the FFSS Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 7

Summary of EDG#1 FFSS Failure Actions Taken/Planned

• Replaced old FFSS with a new FFSS - complete

• Test procedure enhancement - perform continuity checks following switch manipulation - complete (interim action)

• Procedure enhancement - Field Flash Pushbutton use added to the Emergency Use section of the operating procedure - complete

• Initiated enhancement PMs to Inspect/Replace FFSS - complete

• EDG Reliability Assessment - complete

• Modifications being considered:

• Install an indicating light for the switch contacts when returned to the 400 RPM position to provide positive indication of circuity integrity.

• Redesign of the field flash circuit to remove the dependence on the 400 RPM contact functioning and eliminate the vulnerability.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 8

Summary of EDG#1 FFSS Failure Timeline of Events Successful 184 Successful Successful Successful Successful Successful Field Flash day EDG test: Monthly EDG Monthly EDG Monthly EDG Monthly EDG Monthly EDG Selector Switch FFSS set to 400 Test: FFSS Test: FFSS Test: FFSS Test: FFSS Test: FFSS Replaced RPM. FFSS not moved to 800 moved to 800 moved to 800 moved to 800 moved to 800 manipulated RPM, tested, & RPM, tested, & RPM, tested, & RPM, tested, & RPM, tested, & May 28th following test returned to 400 returned to 400 returned to 400 returned to 400 returned to 400 Field Flash RPM after test: RPM after test: RPM after test: RPM after test: RPM after test:

Selector Switch Nov. 12th Found Failed:

Dec 10th Jan 7th Feb 4th Mar 4th April 1st & 29th May 27th Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 9

PRA Considerations

• Conservatisms in the PRA model were evaluated for the risk associated with the May 27th event

• Key areas evaluated are:

• Exposure time

• Operator Recovery

• Model Conservatisms

• The significance of the event is estimated to be White, based on consideration of these key areas Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 10

PRA Considerations Exposure Time - EDG#1 FFSS Failure Successful Monthly EDG Last Monthly Field Flash EDG test with Test: FFSS moved EDG Test: Selector Switch FFSS set to 400 to 400 RPM and FFSS moved Replaced RPM. FFSS not the switch to 400 RPM manipulated contacts are and the switch May 28th 0935 following test potentially not contacts are Field Flash made: not made:

Selector Switch Nov. 12th 0446 Found Failed:

Dec. 10th 1129 April 29 27th 1122 May 27th Standard T/2:

T/2=98.6 T/2 Analysis would be Nov. 12 2020 0446 to May 28th 0935 2021: 197.2 days, T/2=98.6 NRC Used:

83+31=114 TIME UNKNOWN IF FFSS WAS FAILED TIME KNOWN FFSS WAS FAILED NRC USING T/2 Nov. 12th to April 27th Full T for April 27th 2021 to May 28 2021 17:30.

NRC using 166 days or T/2= 83. NRC is using 31 days .

Challenge 70+29= 99 TIME KNOWN FFSS WAS SUCCESS TIME UNKNOWN IF FFSS WAS FAILED TIME KNOWN FFSS WAS FAILED November 12th to December 10th November 12th should be December 10th April 27th should be April 29th. T=29 FFSS was NOT MOVED after success April 27th Date should be April 29th EDG was available at 0935 not 17:30 T=140, T/2=70 Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 11

PRA Considerations Operator Response/Recovery

• Following fast start failure of EDG 1 during a Loss of Offsite Power with SBODG and EDG 2 unavailable

• Crew would take actions for the Reactor Trip resulting from Loss of Offsite Power

• EDG 1 would be emergency shutdown during the initial start

• Extended Loss of AC and entering DB-OP-02700, Station Blackout. DB-OP-02700 has actions to mitigate a station blackout concurrent with a maximum of 111 gpm RCS leakage

• The Shift Manager would pursue the recovery of EDG 1

• Equipment Operators would be given direction to reset the lockout IAW DB-OP-06316, Emergency Diesel Generator Operating Procedure

• Post start checks would identify the lack of voltage indication for the EDG with the red Field Flashed light still illuminated

• Using guidance in the normal operating procedure and Conduct of Operations, the operating crew would depress the Field Flash Pushbutton to flash the EDG Field

• E Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 12

PRA Considerations Operator Response/Recovery Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 13

PRA Considerations Operator Response/Recovery Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 14

PRA Considerations Dominant Fire PRA Risk Scenarios

1) Fire in high voltage switchgear room 2 (Q-01).

2) Fire in control room (FF-01) or cable spread room (DD-01) requires control room abandonment due to loss of control.

3) Fire in low voltage switchgear room #2 (X-01) causes an overcurrent trip of Bus A Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 15

PRA Considerations Conservatisms or non-conservatisms in the Davis-Besse fire PRA Model Specific Non-Conservatisms:

• None Model Specific Conservatisms:

• No credit given for RCPs tripping due to loss of power from fire impacts.

• Impact of removing the conservatism:

• All-hazards delta CDP reduction of 18%

• DC power is failed from low voltage switchgear ventilation failures even though the heat loading in the low ventilation SWGR is minimal when 480V bus power is lost.

• Impact of removing the ventilation failures impact on the DC power system only for battery duration.

• All-hazards delta CDP reduction of 12%

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 16

PRA Considerations Conservatisms or non-conservatisms in the Davis-Besse fire PRA EDG Recovery:

• No Credit given for recovery of the emergency diesel generator.

• Impact crediting recovery (including recovery HEP of 0.723)

• All-hazards delta CDP reduction of 27%

Combined effect of conservatism:

• Impact of removing conservatisms and considering EDG recovery concurrently:

• All-hazards delta CDP reduction of 49%

• Results in an all-hazards delta CDP of 8.79E-6, which is within the range of White Significance Analysis of the effects of Conservatisms is documented in PRA-DB1-22-001-R00 Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 17

PRA Considerations Conservatisms or non-conservatisms in the Davis-Besse fire PRA Non-quantifiable Sources of conservatism:

• NUREG-2230: Electrical Cabinet Fires

• NUREG-2178 vol. 2: Electrical Enclosure Fires, Motors and Dry transformers, Main Control board fire progression, Plant Trip probability <1.

• Both of these require extensive Fire Detail Modeling to determine impact Other possible conservatisms investigated:

• NUREG-1921 supplement 2: MCR Abandonment

• Currently in alignment

• EPRI 3002016004: Alternate MCR Abandonment HRA

• Negligible impact to analysis

• NUREG-2232, NUREG-2233: Transient Fires

• Negligible impact to analysis Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 18

Independent Failure Analysis - MPR & Exelon Power Labs

• MPR and Associates

• Non-intrusive inspection

• Contact resistance checks

• Functional testing

• Failure modes analysis

• Exelon Power Labs

• Contact resistance checks

• Intrusive inspection

• MPR And Associates prepared final failure analysis report Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 19

Independent Failure Analysis - MPR & Exelon Power Labs Contact Resistance

• Resistance values were consistently low and did not show an improving trend associated with FFSS operation

• Indicates that operation of the switch during troubleshooting onsite and at the vendor simply maintained the as-found contact surface contamination and did not improve/condition the contacts

• Contacts 1 (spare) and 3 (400 RPM)

• Closed except during some monthly surveillances

• All 100 measurements of contacts 1 and 3 resistance were well below the 72 Ohms determined in the lab to be needed to prevent the field flash contactor pickup (90% below 1 ohm; 2.6 ohm maximum)

• Contacts 2 (800 RPM) and 4 (spare)

• Open when 1 and 3 are closed

• Higher resistance values than contacts 1 and 3, with an average reading of 3.7 Ohms, and only 1 of 100 just above 72 Ohms (73 Ohms)

• No contact resistance measured in the lab was in the range of thousands of Ohms as was observed 2 of 5 times while the switch was still installed following the failure. All lab-measured resistance values were orders of magnitude smaller and in a relatively narrow band.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 20

Independent Failure Analysis - MPR & Exelon Power Labs Functional Testing and Conclusions

• Vendor recommended PM would not have prevented this issue

• During the simulated functional testing with a field flash contactor, the 3-3C (400 RPM) and 2-2C (800 RPM) contacts functioned properly 100% of the time without failure.

• It was identified the more heavily fouled contacts were for the 800 RPM circuit (slow start), which had not failed in the plant and did not fail in testing.

• The visual condition of the 400 RPM contacts would not have led to cleaning/replacement of the FFSS

• The resistance values for both the 400 & 800 RPM indicated acceptable contact performance

• These visual and resistance measurement findings of the 400 RPM contacts, if observed during a PM, would have resulted in continued use of the FFSS Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 21

Independent Failure Analysis - MPR & Exelon Power Labs Surface Contamination

• Contact Surfaces:

400 RPM 800 RPM Emergency Start Idle Start Normally Closed Normally Open Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 22

Independent Failure Analysis - MPR & Exelon Power Labs Surface Contamination

• Based on the evaluation of the contact contamination and resistance measurements, MPR concluded the surface contamination was not the cause of the FFSS Failure.

• The black substance on the switch contacts is silver sulfide. This forms when silver reacts to sulfur found in the environment, such as Sulfur Oxides found in diesel exhaust.

• Silver sulfide buildup can cause electrical failures, especially on switches that are infrequently cycled, due to buildup.

It is crumbly and does not flake off in large pieces.

• Contacts that are open have a greater buildup than the normally closed contacts, as observed with the FFSS.

• The GE SBM switch does have a self-cleaning wipe feature and the monthly cycling would help maintain acceptable surface conditions.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 23

Independent Failure Analysis - MPR & Exelon Power Labs Switch Alignment/Compression

• Observed proper switch alignment and spring compression

• All 4 moving contact springs were inspected prior to disassembly

• These conical springs were installed correctly with the larger end nearest the contact and centered over the contact carrier nub

• The contacts aligned consistently with each switch operation and appeared to be centered on each other. No issues were observed with contact alignment.

• Following disassembly, spring compression testing was performed with the moving contacts still connected to the cam followers

• The values of all 4 contacts were consistent with each other and very repeatable from trial to trial Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 24

Independent Failure Analysis - MPR & Exelon Power Labs Disassembly and Inspection

• The switch operator mechanism was in very good condition

• The latch arm rested firmly in the cam indents and the latch spring was positioned correctly

• The stop tabs were set to provide the desired switch handle and shaft range of motion

• No foreign material was found in either stage body or in the switch operator mechanism housing

• All components within both contact stages were found in very good condition

• The cams & cam followers showed little to no wear

• The stationary contacts were in position and seated firmly in the stage body

• The moving contact assemblies pivoted smoothly on the respective cam follower Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 25

Independent Failure Analysis - MPR & Exelon Power Labs Scanning Electron Microscope/Energy Dispersive Spectroscopy (SEM/EDS)

• Examination by SEM/EDS identified foreign material on a contact surface

• Contact 2-2C (800 rpm) was visually the worst of the set with the black substance covering nearly all four contacts

• Identified black substance as sulfur. The silver-plated contacts are succumbing to silver sulfidation.

• Contact 3-3C (400 rpm) did not have as much sulfur cover as the even numbered contacts

• There was no wear or unusual scarring on the mating surfaces that would identify it as having a potential for poor electrical connection

• Stationary Contact 3C (mating area) - Detected nickel (Ni) [Discovery

- Cause - Foreign Material]

• Material Identification of Switch Parts

• A transverse cross section of the stationary Contact 4C was completed.

The silver on the contact surface was attached to a copper-zinc-tin (Cu-Zn-Sn) alloy. No nickel (Ni) was detected.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 26

Independent Failure Analysis - MPR & Exelon Power Labs Scanning Electron Microscope/Energy Dispersive Spectroscopy (SEM/EDS)

• The following switch parts were tested:

• Terminal screws (on outside of switch body) - nickel plated brass

• Shaft - material was consistent with carbon steel with zinc plating

• Stationary contact supports - material is consistent with a copper-zinc-tin alloy base material with tin plating

• Movable contact support - material is consistent with brass (copper and zinc)

• Latch arm - the material is consistent with a carbon steel base material with a zinc-chromium plating

• Moving contact assembly spring - stainless steel

• Nickel should not be on Stationary Contact 3C (400 rpm)

• Source of nickel on 3C unknown; possibly terminal screw Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 27

Independent Failure Analysis - MPR & Exelon Power Labs Failure Analysis Results and Conclusions

• Resistance and Functional Tests

• Intermittent, very high contact resistances (> 1000 Ohms) measured in place at the station could not be repeated in laboratory

• 400 rpm contacts - All laboratory-measured resistances were significantly lower than those measured at the station, and all were well below the measured threshold resistance needed to prevent field flash contactor pickup.

All functional tests picked up contactor

• 800 rpm contacts - Were found to be in worse condition than 400 rpm contacts by visual inspection, resistance measurement, and material analysis; all but one measured resistance was below the measured threshold resistance. All functional tests picked up contactor

• Internal Inspection and Material Analysis

• All contact surfaces were found to be in good shape, but with some surface sulfidation present (worse for 800 rpm contacts). 400 rpm contacts had little to no sulfidation at mating areas.

• Switch operating mechanism and contact mechanisms were in good operating condition

• One moving contact, 3C, had nickel deposits on surface, mostly in the mating area. Contacts are silver plated, and nickel should not be present (i.e., it is foreign material)

• Source of nickel on 3C contact surface unknown; could be terminal screw plating or another unidentified source

• Conclusion

• Evidence supports the postulated failure cause being high contact resistance due to presence of foreign material

• The observed switch visual and functional condition would not have prompted a PM to clean contacts or replace the switch Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 28

Independent Failure Analysis - MPR & Exelon Power Labs Manipulation of the FFSS during initial troubleshooting

• Initial FFSS contact 3-3C (400 RPM) checks following EDG Shutdown showed continuity across contact.

• With the switch still installed additional troubleshooting operated the FFSS 5 times.

• Following removal, the switch was delivered to the PSDM team and then to the system engineer.

• The switch was manipulated for visual inspection between removal and delivery to the system engineer.

• Pictures of 3-3C contact surfaces were taken on 6/3/21.

• After these pictures were taken the following was performed on the switch between 6/3/21 and 9/9/21

• FFSS cycled for visual inspections of contacts.

• Plate on front end of switch was removed for visual inspection Switch operator Mechanism and re-installed.

• Main bolts loosened slightly (~ 1/4) to attempt better visual inspection of contact surfaces and bolts were re-snugged.

• On 9/9/21 switch is prepared for shipment.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 29

Independent Failure Analysis - MPR & Exelon Power Labs Other Postulated Failure Modes Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 30

Additional Considerations How a FFSS PM will not prevent failure by FM

• Presence of FM is a unique and unpredictable incident and not the result of a performance deficiency

• A FFSS Preventive Maintenance task to inspect is not likely to prevent failure by FM

• Introduction of FM is not ruled by a timing function that is predictable

• Example:

• A FFSS PM to inspect is performed on Day X

• FM is introduced in FFSS on Day X+1

• FFSS fails performance on Day X+2

• FFSS failure by FM NOT preventable by PM Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 31

Additional Considerations Foreign Material Source Source of Foreign material is indeterminate The foreign material most likely came from within the switch (terminal screw)

- Additional testing identified nickel on the FFSS terminal screws

- Small openings exist that could allow material from the terminal screws to fall onto the switch contacts.

- Switch is open to environment, but switch is configured so opening faces down

- There is no forced air flow within the cabinet

- A visual inspection of the cabinet was performed and the cabinet was found to be clean Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 32

Additional Considerations Operating Experience Three SBM switch failure OEs related to contact contamination and identified by Davis-Besse were reviewed and were determined to not be comparable to the FFSS failure at Davis-Besse based on the failure analysis performed.

Plant 1:

- Identified dirt, fuzz-like foreign material and contact surface sulfidation as the cause.

- Davis-Besse FFSS was found to be in good material condition and no dirt or fuzz-like material was found, thus this failure is not comparable.

Plant 2:

- Identified contact corrosion, but no detailed failure analysis performed to support.

- Since there was no detailed analysis performed of the failed switch this OE could not be compared to the failure analysis results of the Davis-Besse FFSS.

Plant 3:

- Identified contact contamination due to silver sulfide as the cause and lack of regular operation was identified as contributing (only during deep downpowers or forced outage maintenance).

- Davis-Besse FFSS 400 RPM contacts were found to be significantly less fouled than the Plant 3 contact surfaces.

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 33

Responses to 5 Areas to support Significance Determination We have presented and provided written responses for:

• The assumed exposure period used in the NRC preliminary evaluation

• Actions that can be taken by operators to recover from a fast start failure of the EDG, including operator training on the FFSS

• The feasibility and reliability of the operator actions to recover from the fast start failure of the EDG, particularly during the dominant fire risk scenarios

• Conservatisms or non-conservatisms in your fire PRA that could affect the outcome of this evaluation

• Why you believe the contact contamination is not a credible cause of the FFSS that occurred during testing, including how inspection and preventive maintenance would not prevent the failure Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 34

Conclusions Summary of Energy Harbor Position

• Based on our PRA insights and review of conservatisms, the significance of the event would be White

• An independent failure analysis eliminated silver sulfidation as the cause of the FFSS failure and determined the failure was most likely caused by foreign material

• It is Davis-Besses perspective that an inspection of the FFSS would not have prevented the failure of the EDG on May 27th

• The significance of not having a PM schedule for the inspection of the FFSS should be Green (Very Low)

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 35

Questions?

Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 36

Additional Information Slides Field Flash Circuit FFSS Confidential, Subject to Confidentiality Agreements For Discussion Purposes Only, Non-Disclosable Information 37