IR 05000280/2010006: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
(2 intermediate revisions by the same user not shown)
Line 3: Line 3:
| issue date = 09/10/2010
| issue date = 09/10/2010
| title = IR 05000280-10-006; Virginia Electric and Power Company; 06/10/2010 - 08/06/2010; Surry Power Station - Special Inspection Report
| title = IR 05000280-10-006; Virginia Electric and Power Company; 06/10/2010 - 08/06/2010; Surry Power Station - Special Inspection Report
| author name = Wert L D
| author name = Wert L
| author affiliation = NRC/RGN-II/DRP
| author affiliation = NRC/RGN-II/DRP
| addressee name = Heacock D A
| addressee name = Heacock D
| addressee affiliation = Virginia Electric & Power Co (VEPCO)
| addressee affiliation = Virginia Electric & Power Co (VEPCO)
| docket = 05000280
| docket = 05000280
Line 18: Line 18:


=Text=
=Text=
{{#Wiki_filter:
{{#Wiki_filter:UNITED STATES ptember 10, 2010
[[Issue date::September 10, 2010]]


Mr. David President and Chief Nuclear Officer Virginia Electric and Power Company Innsbrook Technical Center 5000 Dominion Boulevard Glen Allen, VA 23060-6711
==SUBJECT:==
 
SURRY POWER STATION - SPECIAL INSPECTION REPORT 05000280/2010006
SUBJECT: SURRY POWER STATION - SPECIAL INSPECTION REPORT 05000280/2010006


==Dear Mr. Heacock:==
==Dear Mr. Heacock:==
On August 6, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Surry Nuclear Station Unit 1. The purpose of this inspection was to inspect and assess the loss of vital instrument bus 1-III and the subsequent Unit 1 reactor trip, safety injection and nuclear instrument cabinet fire. A special inspection was warranted based on the risk and deterministic criteria specified in Management Directive 8.3, "NRC Incident Investigation Program." The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on [[Exit meeting date::August 6, 2010]], with Mr. Bischof and other members of your staff. The determination that the special inspection would be conducted was made by the NRC on June 9, 2010 and the Special Inspection Team (SIT) was dispatched to the site on June 10, 2010. The inspection focus areas are detailed in the Special Inspection Team Charter (Attachment 2).
On August 6, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Surry Nuclear Station Unit 1. The purpose of this inspection was to inspect and assess the loss of vital instrument bus 1-III and the subsequent Unit 1 reactor trip, safety injection and nuclear instrument cabinet fire. A special inspection was warranted based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on [[Exit meeting date::August 6, 2010]], with Mr. Bischof and other members of your staff. The determination that the special inspection would be conducted was made by the NRC on June 9, 2010 and the Special Inspection Team (SIT) was dispatched to the site on June 10, 2010. The inspection focus areas are detailed in the Special Inspection Team Charter (Attachment 2).


The inspection was performed in accordance with Inspection Procedure 93812, A Special Inspection,@ and focused on the areas discussed in the inspection charter. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission
The inspection was performed in accordance with Inspection Procedure 93812, ASpecial Inspection,@ and focused on the areas discussed in the inspection charter. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission=s rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.
=s rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.


The report documents one NRC identified finding of very low safety significance (Green), which was determined to be a violation of NRC requirements. However, because of the very low safety significance of this issue and because it was entered into your corrective action program, the NRC is treating this as a non-cited violation (NCV) consistant with Section VI.A.1 of the NRC Enforcement Policy. If you wish to contest the NCV, you should provide a written response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC, 20555-0001; with copies to the Regional Administrator Region II; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC, 20555-0001; and the NRC Senior Resident Inspector at the Surry Power Station.
The report documents one NRC identified finding of very low safety significance (Green), which was determined to be a violation of NRC requirements. However, because of the very low safety significance of this issue and because it was entered into your corrective action program, the NRC is treating this as a non-cited violation (NCV) consistant with Section VI.A.1 of the NRC Enforcement Policy. If you wish to contest the NCV, you should provide a written response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC, 20555-0001; with copies to the Regional Administrator Region II; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC, 20555-0001; and the NRC Senior Resident Inspector at the Surry Power Station.


VEPCO 2 In addition, if you disagree with the cross-cutting aspect assigned to the finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, RII, and the NRC Senior Resident Inspector at the Surry Power Station."
VEPCO   2 In addition, if you disagree with the cross-cutting aspect assigned to the finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, RII, and the NRC Senior Resident Inspector at the Surry Power Station.


In accordance with 10 CFR 2.390 of the NRC's "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRC's document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).
In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).


Sincerely,/RA/  
Sincerely,
 
/RA/
Leonard D. Wert, Jr., Director Division of Reactor Projects Docket No.: 50-280 License No.: DPR-32  
Leonard D. Wert, Jr., Director Division of Reactor Projects Docket No.: 50-280 License No.: DPR-32


===Enclosure:===
===Enclosure:===
Inspection Report 05000280/2010006  
Inspection Report 05000280/2010006 w/Attachments: 1. Supplemental Information 2. Surry SIT Charter


===w/Attachments:===
REGION II==
1. Supplemental Information 2. Surry SIT Charter
Docket No.: 50-280 License No.: DPR-32 Report No: 05000280/2010006 Licensee: Virginia Electric and Power Company (VEPCO)
Facility: Surry Power Station, Unit 1 Location: 5850 Hog Island Road Surry, VA 23883 Dates: June 10, 2010 through August 6, 2010 Inspectors: A. Hutto, Senior Resident Inspector (Lead Inspector)
R. Clagg, Resident Inspector S. Walker, Senior Reactor Inspector Approved by: Leonard D. Wert, Jr., Director Division of Reactor Projects Enclosure


cc w/encl. (See page 3)
=SUMMARY OF FINDINGS=
IR 05000280/2010-006; 06/10/2010 - 08/06/2010; Surry Power Station, Unit 1, Special


__ML102560333_____________ X SUNSI REVIEW COMPLETE OFFICE RII:DRS RII:DRS RII:DRP RII:DRP RII:DRP RII:DRP RII:DRP SIGNATURE GAH2 SAW4 RLC4 DCA JSD GJM1 LXW1 NAME AHutto SWalker RClagg DArnett JDodson GMcCoy LWert DATE 09/02/2010 09/07/2010 09/02/2010 09/13/2010 09/02/2010 09/02/2010 09/10/2010 E-MAIL COPY? YES NO YES NO YES NO YES NO YES NO YES NO YES NO VEPCO 3 cc w/encl. Gerald T. Bischof Site Vice President Surry Power Station Virginia Electric and Power Company Electronic Mail Distribution B. L. (Sonny) Stanley Director, Nuclear Safety and Licensing Virginia Electric and Power Company Electronic Mail Distribution Lillian M. Cuoco, Esq. Senior Counsel Dominion Resources Services, Inc. Electronic Mail Distribution
Inspection.


Chris L. Funderburk Director, Nuclear Licensing & Operations Support Virginia Electric and Power Company Electronic Mail Distribution
This inspection was conducted by a team consisting of a senior reactor inspector from the NRC=s Region II office and resident inspectors from the Catawba and North Anna Nuclear Stations. One finding was identified and was determined to be a non-cited violation (NCV).


Ginger L. Alligood Virginia Electric and Power Company Electronic Mail Distribution
The significance of most findings is indicated by their color (Green, White, Yellow, Red)using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP).


Virginia State Corporation Commission Division of Energy Regulation P.O. Box 1197 Richmond, VA 23209
The cross-cutting aspect was determined using IMC 0310, Components Within the Cross-Cutting Areas. Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process Revision 4, dated December 2006.


Attorney General Supreme Court Building 900 East Main Street Richmond, VA 23219
===NRC - Identified and Self-Revealing Findings===
 
Senior Resident Inspector Surry Power Station U.S. Nuclear Regulatory Commission 5850 Hog Island Rd Surry, VA 23883 Michael M. Cline Director Virginia Department of Emergency Services Management Electronic Mail Distribution
 
VEPCO 4 Letter to David from Leonard D. Wert, Jr., dated September 10, 2010.
 
SUBJECT: SURRY POWER STATION - SPECIAL INSPECTION REPORT 05000280/2010006 Distribution w/encl:
C. Evans, RII OE Mail RIDSNRRDIRS PUBLIC RidsNrrPMSurry Resource Enclosure U.S. NUCLEAR REGULATORY COMMISSION REGION II
 
Docket No.: 50-280 License No.: DPR-32 Report No: 05000280/2010006 Licensee: Virginia Electric and Power Company (VEPCO)
 
Facility: Surry Power Station, Unit 1 Location: 5850 Hog Island Road Surry, VA 23883 Dates: June 10, 2010 through August 6, 2010
 
Inspectors: A. Hutto, Senior Resident Inspector (Lead Inspector) R. Clagg, Resident Inspector S. Walker, Senior Reactor Inspector
 
Approved by: Leonard D. Wert, Jr., Director Division of Reactor Projects
 
Enclosure
 
=SUMMARY OF FINDINGS=
IR 05000280/2010-006; 06/10/2010 - 08/06/2010; Surry Power Station, Unit 1, Special Inspection.
 
This inspection was conducted by a team consisting of a senior reactor inspector from the NRC=s Region II office and resident inspectors from the Catawba and North Anna Nuclear Stations. One finding was identified and was determined to be a non-cited violation (NCV). The significance of most findings is indicated by their color (Green, White, Yellow, Red) using Inspection Manual Chapter (IMC) 0609, "Significance Determination Process" (SDP). The cross-cutting aspect was determined using IMC 0310, "Components Within the Cross-Cutting Areas."  Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRC's program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process" Revision 4, dated December 2006.
 
===A. NRC - Identified and Self-Revealing Findings===


===Cornerstone: Initiating Events===
===Cornerstone: Initiating Events===
: '''Green.'''
: '''Green.'''
An NRC identified non-cited violation of 10 CFR 50, Appendix B, Criteria XVI, Corrective Action, was identified for the licensee's failure to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrument (NI) cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009. The issue was entered into the licensee's corrective action program as condition report CR383881. All the RC filters in the Surry Unit 1 and 2 NI cabinets have been replaced with new RC filters.
An NRC identified non-cited violation of 10 CFR 50, Appendix B, Criteria XVI,
Corrective Action, was identified for the licensees failure to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrument (NI) cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009. The issue was entered into the licensees corrective action program as condition report CR383881. All the RC filters in the Surry Unit 1 and 2 NI cabinets have been replaced with new RC filters.
 
The finding was determined to be of more than minor significance because it is associated with the equipment performance attribute of the Initiating Events cornerstone.


The finding was determined to be of more than minor significance because it is associated with the equipment performance attribute of the Initiating Events cornerstone. It adversely affected the cornerstone objective of protection against external events, i.e., fire. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore a phase 2 analysis utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding involved MCR fire scenarios, a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage.
It adversely affected the cornerstone objective of protection against external events, i.e.,
fire. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore a phase 2 analysis utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding involved MCR fire scenarios, a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage.


Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of <1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010. (Section 4OA5.4)
Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of <1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010. (Section 4OA5.4)


===B. Licensee-Identified Violations===
===Licensee-Identified Violations===


None.
None.
Line 97: Line 77:
=REPORT DETAILS=
=REPORT DETAILS=


Summary of Plant Events On June 8, 2010, power to Unit 1 vital AC busses 1-III and 1-IIIA was lost when the uninterruptible power supply inverter swapped to the alternate AC source, which was out of service for scheduled maintenance. The loss of the vital bus caused the 'A' main feed pump recirculation valve to fail to open and also caused 2 of the 3 main feedwater regulating valves to fail and enter an automatic hold mode of operation. This resulted in a reduction in main feedwater flow causing an automatic reactor trip due to a feed/steam flow mismatch in conjunction with low steam generator level. The loss of vital bus 1-III also resulted in a safety injection due to a loss of some vital instrumentation (powered by bus 1-III) along with an RCS cooldown below 543 F due to the inability of the steam dumps to modulate for temperature control. The loss of vital AC bus power also resulted in a loss of numerous field inputs to the plant computer system (PCS) and loss of the safety parameter display system (SPDS). The main control room (MCR) annunciators and sufficient MCR instrumentation remained operable to monitor critical safety functions. All three auxiliary feedwater pumps automatically initiated as designed on low-low steam generator level following the trip. Finally the reactor coolant pump (RCP) 1A was operated for approximately 10 minutes without component cooling water (CCW) cooling to the thermal barrier and motor oil coolers. Thermal barrier cooling was not a significant issue because seal injection flow was not lost.
Summary of Plant Events On June 8, 2010, power to Unit 1 vital AC busses 1-III and 1-IIIA was lost when the uninterruptible power supply inverter swapped to the alternate AC source, which was out of service for scheduled maintenance. The loss of the vital bus caused the 'A' main feed pump recirculation valve to fail to open and also caused 2 of the 3 main feedwater regulating valves to fail and enter an automatic hold mode of operation. This resulted in a reduction in main feedwater flow causing an automatic reactor trip due to a feed/steam flow mismatch in conjunction with low steam generator level. The loss of vital bus 1-III also resulted in a safety injection due to a loss of some vital instrumentation (powered by bus 1-III) along with an RCS cooldown below 543 F due to the inability of the steam dumps to modulate for temperature control. The loss of vital AC bus power also resulted in a loss of numerous field inputs to the plant computer system (PCS) and loss of the safety parameter display system (SPDS). The main control room (MCR) annunciators and sufficient MCR instrumentation remained operable to monitor critical safety functions. All three auxiliary feedwater pumps automatically initiated as designed on low-low steam generator level following the trip. Finally the reactor coolant pump (RCP) 1A was operated for approximately 10 minutes without component cooling water (CCW)cooling to the thermal barrier and motor oil coolers. Thermal barrier cooling was not a significant issue because seal injection flow was not lost.


During the post-trip transient, pressurizer power operated relief valve (PORV), 1-RC-PCV-1455C, cycled 14 times to maintain RCS pressure due to the safety injection and the loss of normal letdown. Also, following the event, failures of the resistor/capacitor (RC) filters in the nuclear instrument (NI) 42 cabinet resulted in a small control room fire. Approximately 3 hours later, a second RC filter failed in NI-44.
During the post-trip transient, pressurizer power operated relief valve (PORV), 1-RC-PCV-1455C, cycled 14 times to maintain RCS pressure due to the safety injection and the loss of normal letdown. Also, following the event, failures of the resistor/capacitor (RC) filters in the nuclear instrument (NI) 42 cabinet resulted in a small control room fire. Approximately 3 hours later, a second RC filter failed in NI-44.


Inspection Scope Based on the probabilistic risk and deterministic criteria specified in Management Directive 8.3, ANRC Incident Investigation Program,@ Inspection Procedure 71153, AEvent Follow-up,@ and the significance of the operational events which occurred, a Special Inspection was initiated in accordance with Inspection Procedure 93812, ASpecial Inspection Team.
Inspection Scope Based on the probabilistic risk and deterministic criteria specified in Management Directive 8.3, ANRC Incident Investigation Program,@ Inspection Procedure 71153, AEvent Follow-up,@ and the significance of the operational events which occurred, a Special Inspection was initiated in accordance with Inspection Procedure 93812, ASpecial Inspection Team.@ The inspection focus areas included the following charter items:
    $    Develop a sequence of events, including operator actions in response to the loss of vital AC bus 1-III.


@  The inspection focus areas included the following charter items:
    $   Review and assess use of alarm response/abnormal operating/emergency operating procedures during the event.
$ Develop a sequence of events, including operator actions in response to the loss of vital AC bus 1-III.


$ Review and assess use of alarm response/abnormal operating/emergency operating procedures during the event.
    $   Assess the available information on the loss of the 125 vital AC bus 1-III and the maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices.


$ Assess the available information on the loss of the 125 vital AC bus 1-III and the maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices.
$   Review licensee documents and other information to assess if the licensee knew or should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage.


$ Review licensee documents and other information to assess if the licensee knew or should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage.
  $   Ascertain the type of fires that actually occurred and the zone of influence of the fires.


$ Ascertain the type of fires that actually occurred and the zone of influence of the fires.
  $   Assess and review the licensees evaluation regarding 1A RCP loss of CCW for 10 minutes.


$ Assess and review the licensee's evaluation regarding 1A RCP loss of CCW for 10 minutes.  $ Review the licensee's corrective actions (CAs), causal analysis and extent of condition associated with the loss of the vital bus and with the RC filter failures.
  $   Review the licensees corrective actions (CAs), causal analysis and extent of condition associated with the loss of the vital bus and with the RC filter failures.


$ Collect data necessary to support completion of the significance determination process, if applicable.
  $   Collect data necessary to support completion of the significance determination process, if applicable.


$ Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
  $   Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).


Document the inspection findings and conclusions in an inspection report within 30 days of the inspection.
Document the inspection findings and conclusions in an inspection report within 30 days of the inspection.
Line 127: Line 107:
{{IP sample|IP=IP 93812}}
{{IP sample|IP=IP 93812}}
===.1 Develop a sequence of events, including operator actions in response to the loss of vital===
===.1 Develop a sequence of events, including operator actions in response to the loss of vital===
AC bus 1-III, (Charter Item 1)
AC bus 1-III, (Charter Item 1)


Line 133: Line 114:


====b. Findings and Observations====
====b. Findings and Observations====
Tuesday June 8, 2010 02:27:00 Uninterruptable power supply (UPS) 1-EP-1A-2 regulating line conditioner (RLC) was tagged out for planned preventive maintenance (PM) and to replace the low and high voltage cards. 09:48:00 While landing two leads (energized with 125Vdc) to a terminal board in UPS, one lead slipped from electrician's hand and contacted an adjacent terminal strip causing an electrical arc and popping sound. An electrical short caused a DC fuse to blow which deenergized a timer relay to the static switch. The inverter, per design, automatically swapped to the RLC which at the time was de-energized (tagged out). This resulted in a loss of 120V vital AC power to vital bus (VB) VB-1-III and VB-1-IIIA.
Tuesday June 8, 2010 02:27:00       Uninterruptable power supply (UPS) 1-EP-1A-2 regulating line conditioner (RLC) was tagged out for planned preventive maintenance (PM) and to replace the low and high voltage cards.
 
09:48:00       While landing two leads (energized with 125Vdc) to a terminal board in UPS, one lead slipped from electricians hand and contacted an adjacent terminal strip causing an electrical arc and popping sound. An electrical short caused a DC fuse to blow which deenergized a timer relay to the static switch. The inverter, per design, automatically swapped to the RLC which at the time was de-energized (tagged out). This resulted in a loss of 120V vital AC power to vital bus (VB) VB-1-III and VB-1-IIIA.


Loss of power to VB-1-III & IIIA caused the 1A main feedwater pump recirculation valve to fail open diverting feed flow to main condenser, the C steam generator (SG) main feed regulator valve (MFRV) failed to the auto lock position preventing adjustment of feed flow to the C SG. The A MFRV unexpectedly went to the closed position stopping feed flow to the A SG. 09:49:23  Automatic reactor trip due to feed flow/steam flow mismatch with low SG level 09:49:23 Turbine trip on reactor rip initiated 09:49:27  Manual reactor trip. Trip initiated per 1-E-0, Reactor Trip or Safety Injection Actuation 09:49:29 Motor driven auxiliary feedwater pumps (AFW) started on low SG level 09:49:31 Turbine driven AFW pump started on low SG level  09:49:39 Feedwater isolation on safety injection signal 09:50:01 Automatic safety injection (SI) on hi steam flow with low Tave < 543F 09:50:00 Main feedwater pumps A and B tripped on feedwater isolation signal 09:50:03 Main steam isolation on high steam flow with low Tave SI signal 09:57 1A reactor coolant pump secured, component cooling water to pump was isolated due to loss VB-1-III 10:00  SI manually reset, first charging pump secured 10:02  Pressurizer (PZR) level 100%, off scale high 10:05  High head SI isolated to cold legs 10:07 Transition to procedure 1-ES-1.1, SI Termination, from 1-E-0 as criteria were met to terminate SI 10:11  B low head SI pump secured 10:16 PZR PORV PCV-1445C cycled, first of 14 cycles 10:16  Letdown established on 45 gallon per minute orifice 10:37  Last PORV cycle 11:21  Fire in N-42 cabinet (PR, IR, SR remain energized, no blown fuses.) 11:24 Fire out in N-42 cabinet. Manual CO 2 and dry chemical fire extinguishers discharged. 14:45 N-44 control power fuse blows, N-44 and N-43 de-energized. P-10 reset disabling high voltage power supply to source range (SR) NIs 14:50 SR NIs restored 15:24 Power restored to vital AC buses 1-III and 1-IIIA, all plant parameters are stable and in Mode 3 19:45 Last actions of 1-ES-1.1, SI Termination, are completed, procedure exited.
Loss of power to VB-1-III & IIIA caused the 1A main feedwater pump recirculation valve to fail open diverting feed flow to main condenser, the C steam generator (SG) main feed regulator valve (MFRV) failed to the auto lock position preventing adjustment of feed flow to the C SG. The A MFRV unexpectedly went to the closed position stopping feed flow to the A SG.


===.2 Review and assess use of alarm response/abnormal operating/emergency operating procedures during the event.===
09:49:23 Automatic reactor trip due to feed flow/steam flow mismatch with low SG level 09:49:23 Turbine trip on reactor rip initiated 09:49:27 Manual reactor trip. Trip initiated per 1-E-0, Reactor Trip or Safety Injection Actuation 09:49:29 Motor driven auxiliary feedwater pumps (AFW) started on low SG level 09:49:31 Turbine driven AFW pump started on low SG level 09:49:39 Feedwater isolation on safety injection signal 09:50:01 Automatic safety injection (SI) on hi steam flow with low Tave < 543F 09:50:00 Main feedwater pumps A and B tripped on feedwater isolation signal 09:50:03 Main steam isolation on high steam flow with low Tave SI signal 09:57    1A reactor coolant pump secured, component cooling water to pump was isolated due to loss VB-1-III 10:00    SI manually reset, first charging pump secured 10:02    Pressurizer (PZR) level 100%, off scale high 10:05    High head SI isolated to cold legs 10:07    Transition to procedure 1-ES-1.1, SI Termination, from 1-E-0 as criteria were met to terminate SI 10:11    B low head SI pump secured 10:16    PZR PORV PCV-1445C cycled, first of 14 cycles 10:16    Letdown established on 45 gallon per minute orifice 10:37    Last PORV cycle 11:21    Fire in N-42 cabinet (PR, IR, SR remain energized, no blown fuses.)
The following specific aspects of the event should be included: rapid cooldown of primary, reactor coolant pump operation, safety injection operation including pressurizer level response, fire incident response, (Charter Item 2)
 
11:24    Fire out in N-42 cabinet. Manual CO2 and dry chemical fire extinguishers discharged.
 
14:45    N-44 control power fuse blows, N-44 and N-43 de-energized. P-10 reset disabling high voltage power supply to source range (SR) NIs 14:50    SR NIs restored 15:24    Power restored to vital AC buses 1-III and 1-IIIA, all plant parameters are stable and in Mode 3 19:45    Last actions of 1-ES-1.1, SI Termination, are completed, procedure exited.
 
===.2 Review and assess use of alarm response/abnormal operating/emergency operating===
 
procedures during the event. The following specific aspects of the event should be included: rapid cooldown of primary, reactor coolant pump operation, safety injection operation including pressurizer level response, fire incident response, (Charter Item 2)


====a. Inspection Scope====
====a. Inspection Scope====
The inspectors reviewed and assessed the licensee's use of alarm (annunciator) response procedures, abnormal operating procedures, and emergency operating (emergency) procedures during the event. The inspectors reviewed Historian (HSR) alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. The inspectors also reviewed the licensee's licensed operator requalification program to determine the type of training that control room personnel received regarding the loss of a vital bus. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and commensurate with the licensee's training program. The inspectors also observed a re-creation of the event in the plant simulator to assess the licensee's response. Documents reviewed are listed in Attachment 1.
The inspectors reviewed and assessed the licensees use of alarm (annunciator)response procedures, abnormal operating procedures, and emergency operating (emergency) procedures during the event. The inspectors reviewed Historian (HSR)alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. The inspectors also reviewed the licensees licensed operator requalification program to determine the type of training that control room personnel received regarding the loss of a vital bus. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and commensurate with the licensees training program. The inspectors also observed a re-creation of the event in the plant simulator to assess the licensees response.


The inspectors reviewed and assessed the licensee's fire incident response. The inspectors reviewed operator logs, licensee fire protection procedures and fire protection strategies, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and appropriate for the given conditions. Documents reviewed are listed in Attachment 1.
Documents reviewed are listed in Attachment 1.
 
The inspectors reviewed and assessed the licensees fire incident response. The inspectors reviewed operator logs, licensee fire protection procedures and fire protection strategies, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and appropriate for the given conditions. Documents reviewed are listed in Attachment 1.


====b. Observations and Findings====
====b. Observations and Findings====
No findings were identified. The inspectors determined that the licensee utilized the appropriate procedures based on the given plant conditions. The plant conditions encountered and systems response were as expected and modeled in the simulator, based on the equipment and instrumentation lost due to the de-energization of the 1-III and 1-IIIA vital AC buses, with the exception of the 1A main feedwater regulating valve going closed. The licensee performed comprehensive troubleshooting on the valve in an effort to determine the cause of the closure but could not find any discrepancies. The 1A main feedwater valve controller was replaced as a contingency and retested satisfactorily prior to startup. The inspectors concluded that the licensee's operator requalification program was adequate in the depth and breadth of training devoted to loss of a vital bus and was demonstrated by the operators' correct response to the event. The inspectors evaluated the licensee's actions as compliant with plant procedures and the licensee's training program. The inspectors determined that the licensee actions for the fire incident response were proper and in accordance with plant procedures.
No findings were identified. The inspectors determined that the licensee utilized the appropriate procedures based on the given plant conditions. The plant conditions encountered and systems response were as expected and modeled in the simulator, based on the equipment and instrumentation lost due to the de-energization of the 1-III and 1-IIIA vital AC buses, with the exception of the 1A main feedwater regulating valve going closed. The licensee performed comprehensive troubleshooting on the valve in an effort to determine the cause of the closure but could not find any discrepancies. The 1A main feedwater valve controller was replaced as a contingency and retested satisfactorily prior to startup. The inspectors concluded that the licensees operator requalification program was adequate in the depth and breadth of training devoted to loss of a vital bus and was demonstrated by the operators correct response to the event. The inspectors evaluated the licensees actions as compliant with plant procedures and the licensees training program. The inspectors determined that the licensee actions for the fire incident response were proper and in accordance with plant procedures.
 
===.3 Assess the available information on the loss of the 125 VAC bus 1-III and the===


===.3 Assess the available information on the loss of the 125 VAC bus 1-III and the maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices, (Charter Item 3)===
maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices, (Charter Item 3)


====a. Inspection Scope====
====a. Inspection Scope====
The inspection team reviewed corrective action documents, maintenance procedures, work orders, system drawings, vendor technical manual information, maintenance training requirements and qualification records, and interviewed engineering and maintenance personnel, to assess the maintenance practices associated with the event. The inspectors also reviewed the licensee's maintenance risk assessments associated with the regulating line conditioner circuit board replacement, scheduled work orders associated with the uninterruptible power supplies, and interviewed engineering personnel to identify and assess other potential vulnerabilities or maintenance practices.
The inspection team reviewed corrective action documents, maintenance procedures, work orders, system drawings, vendor technical manual information, maintenance training requirements and qualification records, and interviewed engineering and maintenance personnel, to assess the maintenance practices associated with the event.
 
The inspectors also reviewed the licensees maintenance risk assessments associated with the regulating line conditioner circuit board replacement, scheduled work orders associated with the uninterruptible power supplies, and interviewed engineering personnel to identify and assess other potential vulnerabilities or maintenance practices.


b Findings No findings were identified. The inspectors found that the RLC circuit board replacement PM had been performed at least eight previous times on Units 1 and 2 without incident. During the development of maintenance procedure 0-ECM-0103-02, Station Battery UPS System Maintenance, it was not recognized during technical reviews by the engineering subject matter expert (SME), that an electrical fault during RLC circuit board replacement could cause a transfer of power from the normal supply to the tagged out alternate supply. The SME believed that the DC circuit that remained energized during the maintenance only affected alarms and indications. This was based on the fact that there were no vendor technical manual information or precautions highlighting this vulnerability, and that system drawings did not clearly link the DC circuit to operation of the static switch that transferred the power source. As a result, there were no precautions in the procedure related to lifting and landing the energized leads that caused the transient. The licensee's lack of understanding of the potential consequences of the maintenance was also reflected in the licensee's risk management tool for online maintenance. The work was coded to accurately reflect the perceived risk and therefore did not contribute to a significant increase in the online risk projected for the day. The previous multiple successful completions of the PM served to reinforce the licensee's perception that there was no risk of losing power to the vital bus as a result of the maintenance. The inspectors determined that the licensee's assessment was reasonable given the information available at the time.
b Findings No findings were identified. The inspectors found that the RLC circuit board replacement PM had been performed at least eight previous times on Units 1 and 2 without incident. During the development of maintenance procedure 0-ECM-0103-02, Station Battery UPS System Maintenance, it was not recognized during technical reviews by the engineering subject matter expert (SME), that an electrical fault during RLC circuit board replacement could cause a transfer of power from the normal supply to the tagged out alternate supply. The SME believed that the DC circuit that remained energized during the maintenance only affected alarms and indications. This was based on the fact that there were no vendor technical manual information or precautions highlighting this vulnerability, and that system drawings did not clearly link the DC circuit to operation of the static switch that transferred the power source. As a result, there were no precautions in the procedure related to lifting and landing the energized leads that caused the transient. The licensees lack of understanding of the potential consequences of the maintenance was also reflected in the licensees risk management tool for online maintenance. The work was coded to accurately reflect the perceived risk and therefore did not contribute to a significant increase in the online risk projected for the day. The previous multiple successful completions of the PM served to reinforce the licensees perception that there was no risk of losing power to the vital bus as a result of the maintenance. The inspectors determined that the licensees assessment was reasonable given the information available at the time.


The inspectors found that the technician that performed the maintenance was knowledgeable, experienced and fully qualified to perform the maintenance. The maintenance was performed in accordance with the procedure and the applicable electrical safety work practices. The task that led to the transient required the landing of three leads, two of which were energized, on the same terminal. The increased difficulty of the task contributed to the technician losing control of the energized lead. The technician had previously performed the circuit board replacement PM a number of times without incident.
The inspectors found that the technician that performed the maintenance was knowledgeable, experienced and fully qualified to perform the maintenance. The maintenance was performed in accordance with the procedure and the applicable electrical safety work practices. The task that led to the transient required the landing of three leads, two of which were energized, on the same terminal. The increased difficulty of the task contributed to the technician losing control of the energized lead. The technician had previously performed the circuit board replacement PM a number of times without incident.


The inspectors found that twenty five work orders were scheduled during the next 12 months associated with Unit 1 and Unit 2 UPS, a number of which dealt with the RLC. The next scheduled RLC circuit board replacement was in September 2010 on 02-EP-UPS-2A-2. As a compensatory action or short term corrective action, the licensee has re-coded all vital bus UPS RLC PMs so that they can only be performed during Mode 5.
The inspectors found that twenty five work orders were scheduled during the next 12 months associated with Unit 1 and Unit 2 UPS, a number of which dealt with the RLC.


===.4 Review licensee documents and other information to assess if the licensee knew or should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage, (Charter Item 4)===
The next scheduled RLC circuit board replacement was in September 2010 on 02-EP-UPS-2A-2. As a compensatory action or short term corrective action, the licensee has re-coded all vital bus UPS RLC PMs so that they can only be performed during Mode 5.
 
===.4 Review licensee documents and other information to assess if the licensee knew or===
 
should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage, (Charter Item 4)


====a. Inspection Scope====
====a. Inspection Scope====
The inspectors assessed whether the licensee had previous opportunities to identify degradation or failure modes of the RC filters similar to what occurred on June 8, 2010. The inspectors reviewed vendor information for the NI cabinets to verify the licensee had incorporated appropriate vendor recommended maintenance practices in their surveillance, maintenance, and testing procedures, specifically for inspection and cleaning. Electrical drawings and schematics were reviewed to verify design requirements were accurately reflected in the drawings as well as in the field. Internal and external operating experience documents were reviewed to verify the licensee had performed adequate assessments of industry knowledge and related events with regards to RC filters. The inspectors also reviewed the licensee's corrective action database to determine whether there were previous opportunities to identify and correct similar issues, and whether those instances were adequately evaluated and properly dispositioned.
The inspectors assessed whether the licensee had previous opportunities to identify degradation or failure modes of the RC filters similar to what occurred on June 8, 2010.
 
The inspectors reviewed vendor information for the NI cabinets to verify the licensee had incorporated appropriate vendor recommended maintenance practices in their surveillance, maintenance, and testing procedures, specifically for inspection and cleaning. Electrical drawings and schematics were reviewed to verify design requirements were accurately reflected in the drawings as well as in the field. Internal and external operating experience documents were reviewed to verify the licensee had performed adequate assessments of industry knowledge and related events with regards to RC filters. The inspectors also reviewed the licensees corrective action database to determine whether there were previous opportunities to identify and correct similar issues, and whether those instances were adequately evaluated and properly dispositioned.


====b. Findings====
====b. Findings====
Review of the vendor manuals for the NI cabinets identified that the recommended preventive maintenance was primarily limited to the upper half of the cabinets known as the drawer assembly. Recommended practices included inspection of the drawer assembly components for signs of overheating or deterioration. The lower half of the NI cabinets are known as the termination cabinets. The termination cabinets are where the RC filters are located. The inspectors noted that none of the documents reviewed explicitly addressed inspection of the termination cabinets for mechanical degradation or component deterioration. As part of the licensee's corrective actions, all other locations for RC filters were inspected for potential component degradation. During walk-downs of the NI cabinets, the inspectors did not note any design discrepancies between the NI cabinet schematics and the actual equipment in the field.
Review of the vendor manuals for the NI cabinets identified that the recommended preventive maintenance was primarily limited to the upper half of the cabinets known as the drawer assembly. Recommended practices included inspection of the drawer assembly components for signs of overheating or deterioration. The lower half of the NI cabinets are known as the termination cabinets. The termination cabinets are where the RC filters are located. The inspectors noted that none of the documents reviewed explicitly addressed inspection of the termination cabinets for mechanical degradation or component deterioration. As part of the licensees corrective actions, all other locations for RC filters were inspected for potential component degradation. During walk-downs of the NI cabinets, the inspectors did not note any design discrepancies between the NI cabinet schematics and the actual equipment in the field.


An initial assessment and review of the available operating experience databases with respect to RC filters and NI cabinets did not immediately identify any similar instances to the June 8, 2010 event. However, during the event investigation, the licensee identified a similar event at another plant where the plant experienced a loss of control power for specific nuclear instrumentation when the transient voltage suppressors (the equivalent of a RC filter) began to overheat and burn inside the NI cabinet.
An initial assessment and review of the available operating experience databases with respect to RC filters and NI cabinets did not immediately identify any similar instances to the June 8, 2010 event. However, during the event investigation, the licensee identified a similar event at another plant where the plant experienced a loss of control power for specific nuclear instrumentation when the transient voltage suppressors (the equivalent of a RC filter) began to overheat and burn inside the NI cabinet.


The inspectors reviewed condition report (CR) 358071 which documented a similar occurrence in November 2009 when an RC filter overheated and burned in the N-43 NI cabinet. Specifically, while performing maintenance on the NI cabinet, licensee staff observed some arcing and sparking and subsequently discovered a RC filter that caught fire. During the inspectors' review, they noted the CR stated that following the discovery, the Instrumentation and Control (I&C) group should investigate the event to determine what could have caused the overheating. Upon follow-up, the inspectors found this investigation was not conducted, and the CR was subsequently closed with no apparent cause or corrective actions in place. The inspectors considered this a missed opportunity to discover the deteriorating and degraded condition of the RC filters and the potential for them to arc and cause a fire in the control room. In addition to the failure of the licensee to evaluate the fire in the NI cabinet, the inspectors also identified in the CR change history, that the description had been changed from "small electrical fire" to "electrical resistor smoldering."  After conducting interviews with licensee staff involved in the November 2009 event, it was determined that it was actually a fire/flame that was observed and not smoldering.
The inspectors reviewed condition report (CR) 358071 which documented a similar occurrence in November 2009 when an RC filter overheated and burned in the N-43 NI cabinet. Specifically, while performing maintenance on the NI cabinet, licensee staff observed some arcing and sparking and subsequently discovered a RC filter that caught fire. During the inspectors review, they noted the CR stated that following the discovery, the Instrumentation and Control (I&C) group should investigate the event to determine what could have caused the overheating. Upon follow-up, the inspectors found this investigation was not conducted, and the CR was subsequently closed with no apparent cause or corrective actions in place. The inspectors considered this a missed opportunity to discover the deteriorating and degraded condition of the RC filters and the potential for them to arc and cause a fire in the control room. In addition to the failure of the licensee to evaluate the fire in the NI cabinet, the inspectors also identified in the CR change history, that the description had been changed from small electrical fire to electrical resistor smoldering. After conducting interviews with licensee staff involved in the November 2009 event, it was determined that it was actually a fire/flame that was observed and not smoldering.


The licensee's final root cause report for the June 8, 2010, event determined the event described in CR 358071 as a missed opportunity to perform causal analysis on the degraded RC filters. Also, the root cause report determined that the change in the description from "fire" to "smoldering" impacted the significance of the CR and the level of review the CR received.
The licensees final root cause report for the June 8, 2010, event determined the event described in CR 358071 as a missed opportunity to perform causal analysis on the degraded RC filters. Also, the root cause report determined that the change in the description from fire to smoldering impacted the significance of the CR and the level of review the CR received.


As part of their immediate corrective actions for this issue, the licensee tested and replaced all of the RC filters in the NI cabinets. During subsequent investigation of the removed filters, the licensee determined that many were in a degraded condition.
As part of their immediate corrective actions for this issue, the licensee tested and replaced all of the RC filters in the NI cabinets. During subsequent investigation of the removed filters, the licensee determined that many were in a degraded condition.


Various indicators were used in determining degradation such as: visual cracking, measured capacitance, measured impedance, and capacitance leakage. Capacitance leakage testing can indicate a form of component degradation which may produce a short circuit or high resistance leakage path. Some RC filters exhibited sparking during this testing.
Various indicators were used in determining degradation such as: visual cracking, measured capacitance, measured impedance, and capacitance leakage. Capacitance leakage testing can indicate a form of component degradation which may produce a short circuit or high resistance leakage path. Some RC filters exhibited sparking during this testing.


Failure to Identify and Correct Degraded RC Suppressors  
Failure to Identify and Correct Degraded RC Suppressors


=====Introduction:=====
=====Introduction:=====
An NRC identified non-cited violation of 10 CFR 50 Appendix B, Criteria XVI, Corrective Action was identified for the licensee's failure to identify and correct degraded RC filters associated with Unit 1 nuclear instrument cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009.
An NRC identified non-cited violation of 10 CFR 50 Appendix B, Criteria XVI, Corrective Action was identified for the licensees failure to identify and correct degraded RC filters associated with Unit 1 nuclear instrument cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009.


=====Description:=====
=====Description:=====
In November 2009 while performing maintenance on Unit 2 nuclear instrumentation cabinet N-43, some arcing and sparking was observed by the licensee from the rear of the NI cabinet. Upon further investigation, it was observed that a RC filter on a spare termination had caused the observed arcing and subsequently caught fire. The fire was immediately extinguished, the work was stopped, and the licensee initiated CR 358071 to evaluate the event.
In November 2009 while performing maintenance on Unit 2 nuclear instrumentation cabinet N-43, some arcing and sparking was observed by the licensee from the rear of the NI cabinet. Upon further investigation, it was observed that a RC filter on a spare termination had caused the observed arcing and subsequently caught fire. The fire was immediately extinguished, the work was stopped, and the licensee initiated CR 358071 to evaluate the event.


In June 2010 following a similar event where an RC filter in the Unit 1 NI cabinet N-42 overheated and caught fire, and an RC filter in cabinet N-44 overheated and smoldered, the inspectors reviewed the licensee's response to the November 2009 event documented in CR 358071. The inspectors noted the licensee did not perform an investigation, as stated in the CR, of the event to determine possible reasons for the arcing and resulting fire of the RC filter in the Unit 2 cabinet N-43. No corrective actions were taken to determine the cause or the extent of condition.
In June 2010 following a similar event where an RC filter in the Unit 1 NI cabinet N-42 overheated and caught fire, and an RC filter in cabinet N-44 overheated and smoldered, the inspectors reviewed the licensees response to the November 2009 event documented in CR 358071. The inspectors noted the licensee did not perform an investigation, as stated in the CR, of the event to determine possible reasons for the arcing and resulting fire of the RC filter in the Unit 2 cabinet N-43. No corrective actions were taken to determine the cause or the extent of condition.


The inspectors also identified that the description of CR 358071 had been changed from "fire" to "smoldering" which subsequently had an impact on how the licensee dispositioned the CR. The licensee has since performed a root cause evaluation for the June 8, 2010, event and determined that the organization missed an opportunity to perform a cause analysis on the November 2009 event and that the inaccurate description documented in CR 358071 contributed to a decreased significance and organization response. As part of the corrective actions, the licensee has replaced all the RC filters in the NI cabinets on both units and has corrective actions to update the aging management program for the RC filters since recognizing the age degradation issue and potential consequences. The inspectors determined that it was within the licensee's ability to identify and correct the degraded Unit 1 RC filters prior to failure, based on internal OE information related to the November 2009 Unit 2 RC filter failure.
The inspectors also identified that the description of CR 358071 had been changed from fire to smoldering which subsequently had an impact on how the licensee dispositioned the CR. The licensee has since performed a root cause evaluation for the June 8, 2010, event and determined that the organization missed an opportunity to perform a cause analysis on the November 2009 event and that the inaccurate description documented in CR 358071 contributed to a decreased significance and organization response. As part of the corrective actions, the licensee has replaced all the RC filters in the NI cabinets on both units and has corrective actions to update the aging management program for the RC filters since recognizing the age degradation issue and potential consequences. The inspectors determined that it was within the licensees ability to identify and correct the degraded Unit 1 RC filters prior to failure, based on internal OE information related to the November 2009 Unit 2 RC filter failure.


=====Analysis:=====
=====Analysis:=====
Line 194: Line 196:
Specifically, the degraded condition of the RC filters in the NI cabinets affected the Reactor Protection System and resulted in loss of source range NIs for a period of time, affecting the transient initiator contributor. The fire that resulted from the overheating and arcing of the RC filters increased the external event initiator.
Specifically, the degraded condition of the RC filters in the NI cabinets affected the Reactor Protection System and resulted in loss of source range NIs for a period of time, affecting the transient initiator contributor. The fire that resulted from the overheating and arcing of the RC filters increased the external event initiator.


The licensee failed to identify and correct degraded R/C filters associated with the Unit 1 Nuclear Instrumentation (NI) Cabinets NI-42 and NI-44 based on internal operating experience from a similar degraded condition on Unit 2 in 2009. The degraded RC filters resulted in fires in the Unit 2 main control room (MCR) NI cabinets NI-42 and NI-44. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore phase 2 analyses utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding consisted of MCR fire scenarios a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage. Human error probability for MCR actions was increased to account for use of self contained breathing apparatus during the fire scenarios. Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of <1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010.
The licensee failed to identify and correct degraded R/C filters associated with the Unit 1 Nuclear Instrumentation (NI) Cabinets NI-42 and NI-44 based on internal operating experience from a similar degraded condition on Unit 2 in 2009. The degraded RC filters resulted in fires in the Unit 2 main control room (MCR) NI cabinets NI-42 and NI-44. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore phase 2 analyses utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding consisted of MCR fire scenarios a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage. Human error probability for MCR actions was increased to account for use of self contained breathing apparatus during the fire scenarios. Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of
      <1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010.


=====Enforcement:=====
=====Enforcement:=====
10 CFR Part 50, Appendix B, Criteria XVI, states, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. Contrary to the above, the licensee failed to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrumentation cabinets for N-42 and N-44, a condition adverse to quality, which should have been identified after a similar degraded condition was identified on Unit 2 in November 2009. This issue has been entered into the licensee's corrective action program as CR 383881. Because the finding is of very low safety significance (Green) and it was entered into the licensee's CAP as CR 383881, this violation is being treated as an NCV, consistant with section VI.A.1 of the NRC Enforcement Policy: NCV 05000280/2010006-01, Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters.
10 CFR Part 50, Appendix B, Criteria XVI, states, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. Contrary to the above, the licensee failed to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrumentation cabinets for N-42 and N-44, a condition adverse to quality, which should have been identified after a similar degraded condition was identified on Unit 2 in November 2009. This issue has been entered into the licensees corrective action program as CR 383881. Because the finding is of very low safety significance (Green)and it was entered into the licensees CAP as CR 383881, this violation is being treated as an NCV, consistant with section VI.A.1 of the NRC Enforcement Policy: NCV 05000280/2010006-01, Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters.
 
===.5 Ascertain the type of fires that actually occurred and the zone of influence of the fires,===


===.5 Ascertain the type of fires that actually occurred and the zone of influence of the fires, (Charter Item 5)===
      (Charter Item 5)


====a. Inspection Scope====
====a. Inspection Scope====
Line 205: Line 210:


====b. Findings====
====b. Findings====
No findings were identified. For the fire that occurred in the Unit 1 N-42 cabinet, the inspectors evaluated the evidence gathered by the licensee to determine an accurate zone of influence (ZOI). The zone of influence is considered the surrounding area affected by an ignition source and its potential flame plume and radiant heating effects. In this case, the ignition source is the RC filter at terminations 1 and 2. The inspector reviewed photographs taken by the licensee that showed the affects of RC filter in N-42 at terminations 1 and 2 arcing and catching fire.
No findings were identified. For the fire that occurred in the Unit 1 N-42 cabinet, the inspectors evaluated the evidence gathered by the licensee to determine an accurate zone of influence (ZOI). The zone of influence is considered the surrounding area affected by an ignition source and its potential flame plume and radiant heating effects.
 
In this case, the ignition source is the RC filter at terminations 1 and 2. The inspector reviewed photographs taken by the licensee that showed the affects of RC filter in N-42 at terminations 1 and 2 arcing and catching fire.
 
Each RC filter is approximately 1 inch long. The terminations for the RC filters are aligned vertically in the rear of the NI cabinet. From the evidence reviewed, it was determined that once the RC filter in N-42 caught fire, it burned with sufficient intensity to significantly burn the RC filter immediately below it at terminations 3 and 4. This RC filter was found severely damaged. The flame travelled up along the termination raceway and affected the RC filter at terminations 11 and 12 (approximately 1 inch above RC filter at terminations 1 and 2). This RC filter caught fire and propagated the flame further up the termination board. The flame then charred and burned two more RC filters at terminations 7 and 8, and 5 and 6, respectively (approximately 1 inch above RC filter at terminations 11 and 12). Based on the photographs, NI cabinet schematics, and data charts provided by the licensee, the zone of influence of the flame plume was determined to be a total of 6 along the termination board - from 1 inch below the ignition source (RC filter at termination 1 and 2) to 5 inches above the ignition source.


Each RC filter is approximately 1 inch long. The terminations for the RC filters are aligned vertically in the rear of the NI cabinet. From the evidence reviewed, it was determined that once the RC filter in N-42 caught fire, it burned with sufficient intensity to significantly burn the RC filter immediately below it at terminations 3 and 4. This RC filter was found severely damaged. The flame travelled up along the termination raceway and affected the RC filter at terminations 11 and 12 (approximately 1 inch above RC filter at terminations 1 and 2). This RC filter caught fire and propagated the flame further up the termination board. The flame then charred and burned two more RC filters at terminations 7 and 8, and 5 and 6, respectively (approximately 1 inch above RC filter at terminations 11 and 12). Based on the photographs, NI cabinet schematics, and data charts provided by the licensee, the zone of influence of the flame plume was determined to be a total of 6" along the termination board - from 1 inch below the ignition source (RC filter at termination 1 and 2) to 5 inches above the ignition source. The flames did not appear to spread outside the width of the RC filters. Based on the dimensions, the fire was within the expected ZOI. Smoke residue was observed to have been found in the same general ZOI. Termination wire damage was observed as well, but within the same general ZOI.
The flames did not appear to spread outside the width of the RC filters. Based on the dimensions, the fire was within the expected ZOI. Smoke residue was observed to have been found in the same general ZOI. Termination wire damage was observed as well, but within the same general ZOI.


The RC filter that overheated in NI-44, terminations 1 and 2, approximately 3 hours after the RC filter had ignited in N-42, only charred and did not cause any damage to the surrounding components.
The RC filter that overheated in NI-44, terminations 1 and 2, approximately 3 hours after the RC filter had ignited in N-42, only charred and did not cause any damage to the surrounding components.


===.6 Assess and review the licensee's evaluation regarding 1A RCP loss of CCW for 10 minutes, (Charter Item 6)===
===.6 Assess and review the licensees evaluation regarding 1A RCP loss of CCW for 10===
 
minutes, (Charter Item 6)


====a. Inspection Scope====
====a. Inspection Scope====
The inspectors reviewed and assessed the licensee's evaluation of the loss of component cooling water (CCW) flow to the 1A reactor coolant pump (RCP) for 10 minutes during the event. The inspectors reviewed HSR alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures. The inspectors also reviewed PCS data and equipment design documentation to validate that temperature thresholds for the 1A RCP were not exceeded.
The inspectors reviewed and assessed the licensees evaluation of the loss of component cooling water (CCW) flow to the 1A reactor coolant pump (RCP) for 10 minutes during the event. The inspectors reviewed HSR alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures. The inspectors also reviewed PCS data and equipment design documentation to validate that temperature thresholds for the 1A RCP were not exceeded.


====b. Observations and Findings====
====b. Observations and Findings====
No findings were identified. The inspectors concluded that the licensee utilized the appropriate procedures based on the given plant conditions. The inspectors also evaluated the licensee's actions as compliant with plant procedures. In addition, the inspectors verified that the vendor recommended maximum temperatures for the 1A RCP were not exceeded.
No findings were identified. The inspectors concluded that the licensee utilized the appropriate procedures based on the given plant conditions. The inspectors also evaluated the licensees actions as compliant with plant procedures. In addition, the inspectors verified that the vendor recommended maximum temperatures for the 1A RCP were not exceeded.


===.7 Review the licensee's corrective actions (CAs), causal analysis and extent of condition associated with the loss of the vital bus and with the RC filter failures, (Charter Items 7===
===.7 Review the licensees corrective actions (CAs), causal analysis and extent of condition===


and 8)
associated with the loss of the vital bus and with the RC filter failures, (Charter Items 7      and 8)


====a. Inspection Scope====
====a. Inspection Scope====
The inspectors reviewed the licensee's root cause evaluation reports RCE001013, Failure of RC Suppressors in the NIS Cabinets, and RCE001014, Surry Unit 1 Reactor Trip due to a Loss of Vital Bus 1-III, to assess the adequacy of the licensee's causal analysis, corrective actions and extent of condition associated with these issues.
The inspectors reviewed the licensees root cause evaluation reports RCE001013, Failure of RC Suppressors in the NIS Cabinets, and RCE001014, Surry Unit 1 Reactor Trip due to a Loss of Vital Bus 1-III, to assess the adequacy of the licensees causal analysis, corrective actions and extent of condition associated with these issues.


====b. Observations and Findings====
====b. Observations and Findings====
No findings were identified. The inspectors found the root cause evaluations performed by the licensee were thorough for both issues and that reasonable root causes were identified. Analysis methods used by the licensee included barrier analysis, task analysis, and human error evaluation flow charting for the loss of vital bus event, and failure modes affects analysis and fault tree analysis for the RC filter failures.
No findings were identified. The inspectors found the root cause evaluations performed by the licensee were thorough for both issues and that reasonable root causes were identified. Analysis methods used by the licensee included barrier analysis, task analysis, and human error evaluation flow charting for the loss of vital bus event, and failure modes affects analysis and fault tree analysis for the RC filter failures.


The licensee determined that the direct root cause of the loss of vital bus was human error by the technician when he allowed an energized lead to slip from his hand during the RLC maintenance. An additional root cause identified by the licensee was a knowledge deficiency associated with the operation and inputs to the inverter sync switch. This knowledge deficiency resulted in a lack of understanding of the potential consequences of the maintenance. As a result, barriers that could have prevented the event were not identified and implemented. The licensee determined that the root cause for the RC filter failures was a short circuit of the capacitor which resulted in high current flow and heat, which ignited the epoxy casing of the filters. The most probable cause of the short was due to hardening and cracking of the RC filters' epoxy insulation due to aging.
The licensee determined that the direct root cause of the loss of vital bus was human error by the technician when he allowed an energized lead to slip from his hand during the RLC maintenance. An additional root cause identified by the licensee was a knowledge deficiency associated with the operation and inputs to the inverter sync switch. This knowledge deficiency resulted in a lack of understanding of the potential consequences of the maintenance. As a result, barriers that could have prevented the event were not identified and implemented. The licensee determined that the root cause for the RC filter failures was a short circuit of the capacitor which resulted in high current flow and heat, which ignited the epoxy casing of the filters. The most probable cause of the short was due to hardening and cracking of the RC filters epoxy insulation due to aging.


The inspectors found that corrective actions implemented by the licensee for both issues addressed the root causes and should be effective in preventing reoccurrence. In addition to recoding all UPS RLC PMs to be worked during Mode 5, corrective actions identified by the licensee for the loss of vital bus included additional training related to the event for engineering and maintenance personnel, improvements in procedures for operational risk assessment, and improvements in technical review requirements for maintenance technical procedure development. Corrective actions completed for the RC filter failures included the replacement of all RC filters in the Unit 1 and 2 NI cabinets with new filters. Additional corrective actions planned are to replace RC filters in the Unit 1 and 2 protection channel relay racks and to develop a preventive maintenance strategy for RC filters in the NI, reactor protection, ultrasonic flow measuring, radiation monitoring and vital bus UPS systems.
The inspectors found that corrective actions implemented by the licensee for both issues addressed the root causes and should be effective in preventing reoccurrence. In addition to recoding all UPS RLC PMs to be worked during Mode 5, corrective actions identified by the licensee for the loss of vital bus included additional training related to the event for engineering and maintenance personnel, improvements in procedures for operational risk assessment, and improvements in technical review requirements for maintenance technical procedure development. Corrective actions completed for the RC filter failures included the replacement of all RC filters in the Unit 1 and 2 NI cabinets with new filters. Additional corrective actions planned are to replace RC filters in the Unit 1 and 2 protection channel relay racks and to develop a preventive maintenance strategy for RC filters in the NI, reactor protection, ultrasonic flow measuring, radiation monitoring and vital bus UPS systems.


The inspectors found that the licensee's extent of condition reviews for the loss of vital bus and the RC filter failures were sufficiently broad to address other potential vulnerabilities related to these events. Appropriate corrective actions were identified to address the additional scope of these issues.
The inspectors found that the licensees extent of condition reviews for the loss of vital bus and the RC filter failures were sufficiently broad to address other potential vulnerabilities related to these events. Appropriate corrective actions were identified to address the additional scope of these issues.


===.8 Collect data necessary to support completion of the significance determination process, (Charter Item 9)===
===.8 Collect data necessary to support completion of the significance determination process,===


    (Charter Item 9)
The inspectors interviewed licensee personnel and reviewed operations logs, licensee procedures, corrective action program documents, work orders, root cause evaluations, operability assessments, and engineering evaluations to gather data necessary to develop and assess the safety significance of any findings. This information was provided to the regional Senior Reactor Analyst to assess the significance of the event and to support a Phase 3 analysis for the NI RC filter fire in the main control room.
The inspectors interviewed licensee personnel and reviewed operations logs, licensee procedures, corrective action program documents, work orders, root cause evaluations, operability assessments, and engineering evaluations to gather data necessary to develop and assess the safety significance of any findings. This information was provided to the regional Senior Reactor Analyst to assess the significance of the event and to support a Phase 3 analysis for the NI RC filter fire in the main control room.


===.9 Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, Bulletins), (Charter Item 10)===
===.9 Identify any potential generic safety issues and make recommendations for appropriate===
 
follow-up action (e.g., Information Notices, Generic Letters, Bulletins), (Charter Item 10)


====a. Inspection Scope====
====a. Inspection Scope====
The inspectors reviewed the licensee's root cause evaluation including relevant operating experience, corrective action program documents, work orders, and the NRC Operating Experience (OpE) database to determine the potential for generic safety issues related to the degraded conditions and causes pertaining to this event.
The inspectors reviewed the licensees root cause evaluation including relevant operating experience, corrective action program documents, work orders, and the NRC Operating Experience (OpE) database to determine the potential for generic safety issues related to the degraded conditions and causes pertaining to this event.


====b. Findings and observations====
====b. Findings and observations====
Line 252: Line 266:
On August 6, 2010, the inspection results were presented to Mr. Bischoff and other members of his staff, who acknowledged the findings. The inspectors asked the licensee whether any of the material examined during the inspection should be considered proprietary. No proprietary information was identified.
On August 6, 2010, the inspection results were presented to Mr. Bischoff and other members of his staff, who acknowledged the findings. The inspectors asked the licensee whether any of the material examined during the inspection should be considered proprietary. No proprietary information was identified.


ATTACHMENT:
ATTACHMENT:  


=SUPPLEMENTAL INFORMATION=
=SUPPLEMENTAL INFORMATION=
Line 259: Line 273:


===Licensee Personnel===
===Licensee Personnel===
: [[contact::M. Adams]], Director, Station Engineering
: [[contact::M. Adams]], Director, Station Engineering
: [[contact::G. Bischof]], Site Vice President  
: [[contact::G. Bischof]], Site Vice President
: [[contact::B. Garber]], Supervisor, Licensing
: [[contact::B. Garber]], Supervisor, Licensing
: [[contact::K. Grover]], Manager, Operations
: [[contact::K. Grover]], Manager, Operations
: [[contact::A. Harrow]], Supervisor, Electrical Systems
: [[contact::A. Harrow]], Supervisor, Electrical Systems
: [[contact::R. Johnson]], Manager, Outage and Planning  
: [[contact::R. Johnson]], Manager, Outage and Planning
: [[contact::R. Manrique]], Supervisor, Primary Systems  
: [[contact::R. Manrique]], Supervisor, Primary Systems
: [[contact::C. Olsen]], Manager, Site Engineering  
: [[contact::C. Olsen]], Manager, Site Engineering
: [[contact::L. Ragland]], Supervisor, Health Physics Operations
: [[contact::L. Ragland]], Supervisor, Health Physics Operations
: [[contact::K. Sloane]], Plant Manager (Nuclear)  
: [[contact::K. Sloane]], Plant Manager (Nuclear)
: [[contact::B. Stanley]], Director, Station Safety and Licensing  
: [[contact::B. Stanley]], Director, Station Safety and Licensing
: [[contact::M. Wilda]], Supervisor, Auxiliary Systems  
: [[contact::M. Wilda]], Supervisor, Auxiliary Systems
: [[contact::J. Eggart]], Manager, Radiation Protection & Chemistry  
: [[contact::J. Eggart]], Manager, Radiation Protection & Chemistry
: [[contact::B. Hilt]], Supervisor, HP Technical Services  
: [[contact::B. Hilt]], Supervisor, HP Technical Services
: [[contact::D. White]], Supervisor, ALARA  
: [[contact::D. White]], Supervisor, ALARA
: [[contact::D. Godwin]], Supervisor, Nuclear Engineering  
: [[contact::D. Godwin]], Supervisor, Nuclear Engineering


==LIST OF ITEMS==
==LIST OF ITEMS==
Line 281: Line 295:


===Opened===
===Opened===
: 05000280/2010006-01
: 05000280/2010006-01                   NCV         Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters (Section 4OA5.4)
NCV Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters (Section 4OA5.4)  


==LIST OF DOCUMENTS REVIEWED==
==LIST OF DOCUMENTS REVIEWED==
===Procedures===
: 0-ECM-0103-02, Station Battery UPS System Maintenance, Rev. 25 0-AP-50.00, Opposite Unit Emergency, Rev. 32
: 0-FS-FP-116, Control Room Elevation 27 Feet - 6 inches, Rev. 4 1-AP-10.03, Loss of Vital Bus III, Rev. 16 1-AP-9.00, RCP Abnormal Conditions, Rev. 29 1-E-0, Reactor Trip or Safety Injection, Rev. 63 1-ES-1.1, SI Termination, Rev. 42
: 1-GOP-2.3, Unit Shutdown Stabilizing at HSD, Rev. 32 1C-A1, RCP 1A CC Return Low Flow, Rev. 2 1C-A8, PRZR PRESS CNTRL HI Output, Rev. 2 1C-F7, PRZR Relief TK HI PRESS, Rev. 6 1D-F3, REGEN HX Letdown Line HI Temp, Rev. 0
: RQ-10.4-ST-1, LORP Loss of Vital Bus or Loss of DC Bus, Rev. 0
: MA-AA-103, Conduct of Troubleshooting, Rev. 4
: IMP-C-NI-19, Nuclear Instrumentation Maintenance, Rev. 15
: PI-AA-200, Condition Reporting, Rev. 12 1-IPT-CC-NI-N-42, Nuclear Instrumentation Power Range N-42 Channel Calibration, Rev. 1 
===Drawings===
: 11448-ESK-20A, Schematic 1-EP-UPS-1A-1 15 KVA Inverter, Unit 1, Rev. 1 5965D01, Interconnecting Wiring Diagram Notes and Legends, Unit 1, Rev. 10 11448-FE-3CC, Wiring Diagram Nuclear Instrumentation Panels 1&2, Unit 1, Rev. 14 11448-FE-3CD, Wiring Diagram Nuclear Instrumentation Panels 3&4, Unit 1, Rev. 14 11448-FE-64HA, Conduit Plan Fire Detection System- Service Building and Control Room, Unit
: 1, Rev. 10 11448-RE-25G, Vertical Control Boards, Section 1-5, Unit 1, Rev. 5
: Corrective Action Documents
: CR383847, Unit 1 Reactor trip with safety injection
: CR383880, N-44 PRNI fuse blown - SRNIs lost due to P-10 permissive
: CR383881, Fire in MCR (N-32/N-36/N-42 cabinet - terminal board 224 terminals 1 and 2)
: CR383883,
: EPIP-1.01 entered due to unit pressurizer PORV lifting
: CR383889, Inadvertent short in 01-EP-UPS-1A-2
: CR384024, Cracked filters in power range channel N-43
: CR384061, NI RC filters failing capacitor leakage test
: CR384098, 1FW-FCV-1478 failed close during Unit 1 loss of vital bus
: CR384200, Cracks identified in voltage suppression RC filters
: CR384320, Anomaly in DC leakage current for NI RC filters
: CR384279, Loss of power to Unit 2 remote monitoring panel from loss of VB 1-III
: CR358071, Electrical resistor smoldering in the back of the N43 cabinet
: CR358082, Possible damage to electrical resistor in the back of the N43 cabinet Attachment 1 Training Documents
: DNAP 1909, Electrical Safety PSECTP(N)2009-3, Power Station Electrician Continuing Training PSEDP(N) JPM 9.09, Perform Maintenance on Uninterruptable Power Supply SystemInstrument and Control Start Up Checklists,
: CL-1 through
: CL-5, Unit 1, Rev. 1
: Root Cause Evaluation Reports Root Cause Evaluation RCE001013, Failure of RC Suppressors in the NIS Cabinets Root Cause Evaluation RCE001014, Surry Unit 1 Reactor Trip due to a Loss of Vital Bus 1-III 
===Work Orders===
: WO 38102682694, Troubleshoot/Repair NI Channel, Remove/Replace Suppression Filter,  dated 11/15/09
: WO 38102470562, 2-IPT-CC-NI-N-43, N-43 Channel Calibration, dated 11/14/09
: WO 38102107035, PM: RLC Circuit Board Replacement, 01-EP-UPS-1A-2-Panel, dated
: 6/08/10
: WO 38102805466, Troubleshoot 1-FW-LOOP-L-1478, dated 6/11/10
: UFSAR Section 7.2, Reactor Protection System 
===Miscellaneous===
: Emergency Action Level Technical Basis Document,
: HU3.1 Vendor Technical Manual, 38-S984-00001, Solid State Controls Inc., Instruction and Operating
: Manual W/Drawings, Equipment: 1 Phase 15 KVA UPS System Tag-Out: 1-10-EP-0012, 1-EP-UPS-1A-2, Uninterruptable Power Supply Panel Vendor Technical Manual, 38-W893-00061, Technical Manual Nuclear Instrumentation System
: Units 1 and 2, Rev. 10
: PTE 10000013989, RC Filter NI Replacement In-stock Upgrade and Future Purchase, dated
: 6/10/10
: ET-S-10-0060, Assembly, Testing, and Installation for 0.22uf and 150 Ohm RC Filter, Surry,
: Unit 1, Ver. 0, dated 6/10/10 Inspection Results for Inspection Lot 070000034600
==LIST OF ACRONYMS==
: [[ADAMS]] [[Agencywide Document Access and Management System]]
: [[AFW]] [[Auxiliary Feedwater System]]
: [[CA]] [[Corrective Action]]
: [[CAP]] [[Corrective Action Program]]
: [[CCW]] [[Component Cooling Water]]
: [[CFR]] [[Code of Federal Regulations]]
: [[CR]] [[Condition Report]]
: [[EDG]] [[Emergency Diesel Generator]]
: [[IMC]] [[Inspection Manual Chapter]]
: [[JPM]] [[Job Performance Measures]]
: [[LHSI]] [[Low Head Safety Injection]]
: [[MCR]] [[Main Control Room]]
: [[NCV]] [[Non-cited Violation]]
: [[NEI]] [[Nuclear Energy Institute]]
: [[NI]] [[Nuclear Instrument]]
: [[NRC]] [[Nuclear Regulatory Commission]]
: [[OD]] [[Operability Determination]]
: [[OE]] [[Operating Experience]]
: [[PARS]] [[Publicly Available Records]]
: [[PCP]] [[Process Control Program]]
: [[PCS]] [[Plant Computer System]]
: [[PM]] [[Preventive Maintenance]]
: [[PORV]] [[Power Operated Relief Valve]]
: [[RCE]] [[Root Cause Evaluation]]
: [[RCP]] [[Reactor Coolant Pump]]
: [[RCS]] [[Reactor Coolant System]]
: [[RLC]] [[Regulating Line Conditioner]]
: [[RTP]] [[Rated Thermal Power]]
: [[SDP]] [[Significance Determination Process]]
: [[SG]] [[Steam Generator]]
: [[SI]] [[Safety Injection]]
: [[SIT]] [[Special Inspection Team]]
: [[SPDS]] [[Safety Parameter Display System]]
: [[SR]] [[Surveillance Requirements]]
: [[TDAFWP]] [[Turbine Driven Auxiliary Feedwater Pump]]
: [[TS]] [[Technical Specifications]]
: [[UFSAR]] [[Updated Final Safety Analysis Report]]
: [[UPS]] [[Uninterruptable Power Supply]]
: [[VEPCO]] [[Virginia Electric and Power Company]]
: [[VPAP]] [[Virginia Power Administrative Procedure]]
: [[WO]] [[Work Order]]
: [[SURRY]] [[SPECIAL INSPECTION]]
: [[TEAM]] [[(]]
: [[SIT]] [[) CHARTER]]
: [[LOSS]] [[]]
: [[OF]] [[]]
: [[VITAL]] [[]]
: [[BUS]] [[, REACTOR TRIP,]]
: [[AND]] [[]]
: [[SAFETY]] [[INJECTION]]
: [[A.]] [[Basis  On June 8, 2010, a Unit 1 vital]]
AC bus was lost when the uninterruptible power
supply inverter swapped to the alternate AC source was which out of service for scheduled maintenance. The loss of the vital bus caused the 'A' main feed pump recirculation valve to fail open and also caused 2 of the 3 main feedwater regulating valves to fail and enter an automatic hold mode of operation. This resulted in a reduction in main feedwater flow causing an automatic reactor trip due to a
feed/steam flow mismatch in conjunction with low steam generator level. The loss of vital bus 1-III also resulted in a safety injection due to a loss of some vital instrumentation (fed by bus 1-III) along with the momentary
: [[RCS]] [[cooldown below 543 F due to the inability of the steam dumps to modulate for temperature control. This event also resulted in a loss of numerous field inputs to the Plant Computer System []]
PCS] and loss of the Safety Parameter Display System (SPDS). The Main Control Room (MCR) annunciators and sufficient MCR instrumentation remained operable to
monitor critical safety functions. All three auxiliary feedwater pumps automatically initiated as designed on low-low steam generator level following the trip. Finally the reactor coolant pump (RCP) 1A was run for approximately 10 minutes without component cooling water (CCW) cooling to the thermal barrier and motor oil coolers. Thermal barrier cooling was not an issue because seal injection flow was not lost;
however the licensee is evaluating the 1A
: [[RCP]] [[due to loss of cooling for 10 minutes. For additional information, please refer to]]
EN 45986.
During the post-trip transient, pressurizer
: [[PORV]] [[[Power Operated Relief Valve],]]
PCV-1455C, cycled 14 times as required to maintain RCS pressure due to the safety
injection and the loss of normal letdown.
Also, following the event, failures of the resistor/capacitor (RC) filters in the nuclear instrument (NI) 42 cabinet resulted in a small control room fire. Approximately 3 hours later, a second
: [[RC]] [[filter failed in]]
: [[NI]] [[-44. In addition to the failures of the]]
: [[RC]] [[filters during this event, a similar failure of a]]
: [[RC]] [[filter in the U2]]
: [[NI]] [[43 cabinet occurred during the 2009 refueling outage. The extent of condition and the potential generic safety implications associated with the]]
RC filter failures are a concern.
In accordance with Management Directive 8.3, NRC Incident Investigation Program,
deterministic and conditional risk criteria were used to evaluate the level of
: [[NRC]] [[response for this operational event. Several deterministic criteria were met. This issue involved significant unexpected system interaction and potential generic implications. The loss of vital]]
: [[AC]] [[to Unit 1 resulted in the loss of numerous inputs to the plant computer system and the safety parameter display system. The trip was complex and included the pressurizer (PZR) going solid and multiple cycles of the]]
: [[PZR]] [[]]
PORV. The conditional core damage probability (CCDP) for the event met the
criterion for a Special Inspection. Region
: [[II]] [[determined that the appropriate level of]]
NRC response was to conduct a Special Inspection. 
Attachment 2 This Special Inspection is chartered to review the circumstances surrounding the loss of the Unit 1
: [[120VAC]] [[vital bus 1-]]
III and the response of plant equipment. In addition, the team will review the operator actions taken in response to the loss of vital AC, the plant trip and safety injection and the fire in the NI cabinets.
B. Scope    The inspection is expected to perform data gathering and fact-finding in order to address the following:
1. Develop a sequence of events, including operator actions in response to the loss of vital
: [[AC]] [[bus 1-]]
III.
2. Review and assess use of alarm response/abnormal operating/emergency operating procedures during the event. The following specific aspects of the event should be included: 
* Rapid cooldown of  primary 
* RCP operation 
* Safety Injection operation  including PZR level response 
* Fire Incident response
3. Assess the available information on the loss of the
: [[125 VAC]] [[bus 1-]]
III and the maintenance practices associated with the event. Review vital AC system work
orders and related information to identify other potential vulnerabilities or maintenance practices.
4. Review licensee documents and other information to assess if the licensee knew or should have known that the
: [[RC]] [[filters in the]]
NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage. Considerations should include:
* Operational decision making 
* Operational experience (internal and external) 
* Vendor information on expected service life (age of capacitors),  recommended preventative  maintenance, and if any generic communications were issued on similar capacitors
5. Ascertain the type of fires that actually occurred and the zone of influence of the fires. 6. Assess and review the licensee's evaluation regarding 1A
: [[RCP]] [[loss of]]
CCW for 10 minutes.
7. Review the licensee's corrective actions (CAs), causal analysis and extent of condition associated with the loss of the vital bus.
8. Review the licensee's corrective actions (CAs), causal analysis and extent of condition associated with the RC filter failures. 


Attachment 2 9. Collect data necessary to support completion of the significance determination process, if applicable. Considerations should include: 
* Main feedwater recovery actions 
* Condensate storage tank refill actions 
* Unit 2 auxiliary feedwater crosstie actions 
* PORV block valve actions
10. Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).
}}
}}

Latest revision as of 16:49, 21 December 2019

IR 05000280-10-006; Virginia Electric and Power Company; 06/10/2010 - 08/06/2010; Surry Power Station - Special Inspection Report
ML102560333
Person / Time
Site: Surry Dominion icon.png
Issue date: 09/10/2010
From: Wert L
Division Reactor Projects II
To: Heacock D
Virginia Electric & Power Co (VEPCO)
References
IR-10-006
Download: ML102560333 (27)
v • d • e


Text

UNITED STATES ptember 10, 2010

SUBJECT:

SURRY POWER STATION - SPECIAL INSPECTION REPORT 05000280/2010006

Dear Mr. Heacock:

On August 6, 2010, the U.S. Nuclear Regulatory Commission (NRC) completed a special inspection at your Surry Nuclear Station Unit 1. The purpose of this inspection was to inspect and assess the loss of vital instrument bus 1-III and the subsequent Unit 1 reactor trip, safety injection and nuclear instrument cabinet fire. A special inspection was warranted based on the risk and deterministic criteria specified in Management Directive 8.3, NRC Incident Investigation Program. The enclosed inspection report documents the inspection results, which were discussed at the exit meeting on August 6, 2010, with Mr. Bischof and other members of your staff. The determination that the special inspection would be conducted was made by the NRC on June 9, 2010 and the Special Inspection Team (SIT) was dispatched to the site on June 10, 2010. The inspection focus areas are detailed in the Special Inspection Team Charter (Attachment 2).

The inspection was performed in accordance with Inspection Procedure 93812, ASpecial Inspection,@ and focused on the areas discussed in the inspection charter. The inspection examined activities conducted under your license as they relate to safety and compliance with the Commission=s rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel.

The report documents one NRC identified finding of very low safety significance (Green), which was determined to be a violation of NRC requirements. However, because of the very low safety significance of this issue and because it was entered into your corrective action program, the NRC is treating this as a non-cited violation (NCV) consistant with Section VI.A.1 of the NRC Enforcement Policy. If you wish to contest the NCV, you should provide a written response within 30 days of the date of this inspection report, with the basis for your denial, to the Nuclear Regulatory Commission, ATTN: Document Control Desk, Washington, DC, 20555-0001; with copies to the Regional Administrator Region II; the Director, Office of Enforcement, United States Nuclear Regulatory Commission, Washington, DC, 20555-0001; and the NRC Senior Resident Inspector at the Surry Power Station.

VEPCO 2 In addition, if you disagree with the cross-cutting aspect assigned to the finding in this report, you should provide a response within 30 days of the date of this inspection report, with the basis for your disagreement, to the Regional Administrator, RII, and the NRC Senior Resident Inspector at the Surry Power Station.

In accordance with 10 CFR 2.390 of the NRCs Rules of Practice, a copy of this letter, its enclosure, and your response (if any) will be available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of the NRCs document system (ADAMS). ADAMS is accessible from the NRC Website at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room).

Sincerely,

/RA/

Leonard D. Wert, Jr., Director Division of Reactor Projects Docket No.: 50-280 License No.: DPR-32

Enclosure:

Inspection Report 05000280/2010006 w/Attachments: 1. Supplemental Information 2. Surry SIT Charter

REGION II==

Docket No.: 50-280 License No.: DPR-32 Report No: 05000280/2010006 Licensee: Virginia Electric and Power Company (VEPCO)

Facility: Surry Power Station, Unit 1 Location: 5850 Hog Island Road Surry, VA 23883 Dates: June 10, 2010 through August 6, 2010 Inspectors: A. Hutto, Senior Resident Inspector (Lead Inspector)

R. Clagg, Resident Inspector S. Walker, Senior Reactor Inspector Approved by: Leonard D. Wert, Jr., Director Division of Reactor Projects Enclosure

SUMMARY OF FINDINGS

IR 05000280/2010-006; 06/10/2010 - 08/06/2010; Surry Power Station, Unit 1, Special

Inspection.

This inspection was conducted by a team consisting of a senior reactor inspector from the NRC=s Region II office and resident inspectors from the Catawba and North Anna Nuclear Stations. One finding was identified and was determined to be a non-cited violation (NCV).

The significance of most findings is indicated by their color (Green, White, Yellow, Red)using Inspection Manual Chapter (IMC) 0609, Significance Determination Process (SDP).

The cross-cutting aspect was determined using IMC 0310, Components Within the Cross-Cutting Areas. Findings for which the SDP does not apply may be Green or be assigned a severity level after NRC management review. The NRCs program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, Reactor Oversight Process Revision 4, dated December 2006.

NRC - Identified and Self-Revealing Findings

Cornerstone: Initiating Events

Green.

An NRC identified non-cited violation of 10 CFR 50, Appendix B, Criteria XVI,

Corrective Action, was identified for the licensees failure to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrument (NI) cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009. The issue was entered into the licensees corrective action program as condition report CR383881. All the RC filters in the Surry Unit 1 and 2 NI cabinets have been replaced with new RC filters.

The finding was determined to be of more than minor significance because it is associated with the equipment performance attribute of the Initiating Events cornerstone.

It adversely affected the cornerstone objective of protection against external events, i.e.,

fire. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore a phase 2 analysis utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding involved MCR fire scenarios, a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage.

Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of <1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010. (Section 4OA5.4)

Licensee-Identified Violations

None.

REPORT DETAILS

Summary of Plant Events On June 8, 2010, power to Unit 1 vital AC busses 1-III and 1-IIIA was lost when the uninterruptible power supply inverter swapped to the alternate AC source, which was out of service for scheduled maintenance. The loss of the vital bus caused the 'A' main feed pump recirculation valve to fail to open and also caused 2 of the 3 main feedwater regulating valves to fail and enter an automatic hold mode of operation. This resulted in a reduction in main feedwater flow causing an automatic reactor trip due to a feed/steam flow mismatch in conjunction with low steam generator level. The loss of vital bus 1-III also resulted in a safety injection due to a loss of some vital instrumentation (powered by bus 1-III) along with an RCS cooldown below 543 F due to the inability of the steam dumps to modulate for temperature control. The loss of vital AC bus power also resulted in a loss of numerous field inputs to the plant computer system (PCS) and loss of the safety parameter display system (SPDS). The main control room (MCR) annunciators and sufficient MCR instrumentation remained operable to monitor critical safety functions. All three auxiliary feedwater pumps automatically initiated as designed on low-low steam generator level following the trip. Finally the reactor coolant pump (RCP) 1A was operated for approximately 10 minutes without component cooling water (CCW)cooling to the thermal barrier and motor oil coolers. Thermal barrier cooling was not a significant issue because seal injection flow was not lost.

During the post-trip transient, pressurizer power operated relief valve (PORV), 1-RC-PCV-1455C, cycled 14 times to maintain RCS pressure due to the safety injection and the loss of normal letdown. Also, following the event, failures of the resistor/capacitor (RC) filters in the nuclear instrument (NI) 42 cabinet resulted in a small control room fire. Approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> later, a second RC filter failed in NI-44.

Inspection Scope Based on the probabilistic risk and deterministic criteria specified in Management Directive 8.3, ANRC Incident Investigation Program,@ Inspection Procedure 71153, AEvent Follow-up,@ and the significance of the operational events which occurred, a Special Inspection was initiated in accordance with Inspection Procedure 93812, ASpecial Inspection Team.@ The inspection focus areas included the following charter items:

$ Develop a sequence of events, including operator actions in response to the loss of vital AC bus 1-III.

$ Review and assess use of alarm response/abnormal operating/emergency operating procedures during the event.

$ Assess the available information on the loss of the 125 vital AC bus 1-III and the maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices.

$ Review licensee documents and other information to assess if the licensee knew or should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage.

$ Ascertain the type of fires that actually occurred and the zone of influence of the fires.

$ Assess and review the licensees evaluation regarding 1A RCP loss of CCW for 10 minutes.

$ Review the licensees corrective actions (CAs), causal analysis and extent of condition associated with the loss of the vital bus and with the RC filter failures.

$ Collect data necessary to support completion of the significance determination process, if applicable.

$ Identify any potential generic safety issues and make recommendations for appropriate follow-up action (e.g., Information Notices, Generic Letters, and Bulletins).

Document the inspection findings and conclusions in an inspection report within 30 days of the inspection.

OTHER ACTIVITIES

4OA5 Other Activities - Special Inspection

Main article: IP 93812

.1 Develop a sequence of events, including operator actions in response to the loss of vital

AC bus 1-III, (Charter Item 1)

a. Inspection Scope

The team identified the events that occurred at Surry and documented the specific events in chronological order. In order to develop this sequence of events, the inspection team reviewed corrective action documents and control room logs. The inspection team also interviewed several licensee staff members in the operations, engineering and maintenance departments in order to validate and further establish the sequence of events documented in this report.

b. Findings and Observations

Tuesday June 8, 2010 02:27:00 Uninterruptable power supply (UPS) 1-EP-1A-2 regulating line conditioner (RLC) was tagged out for planned preventive maintenance (PM) and to replace the low and high voltage cards.

09:48:00 While landing two leads (energized with 125Vdc) to a terminal board in UPS, one lead slipped from electricians hand and contacted an adjacent terminal strip causing an electrical arc and popping sound. An electrical short caused a DC fuse to blow which deenergized a timer relay to the static switch. The inverter, per design, automatically swapped to the RLC which at the time was de-energized (tagged out). This resulted in a loss of 120V vital AC power to vital bus (VB) VB-1-III and VB-1-IIIA.

Loss of power to VB-1-III & IIIA caused the 1A main feedwater pump recirculation valve to fail open diverting feed flow to main condenser, the C steam generator (SG) main feed regulator valve (MFRV) failed to the auto lock position preventing adjustment of feed flow to the C SG. The A MFRV unexpectedly went to the closed position stopping feed flow to the A SG.

09:49:23 Automatic reactor trip due to feed flow/steam flow mismatch with low SG level 09:49:23 Turbine trip on reactor rip initiated 09:49:27 Manual reactor trip. Trip initiated per 1-E-0, Reactor Trip or Safety Injection Actuation 09:49:29 Motor driven auxiliary feedwater pumps (AFW) started on low SG level 09:49:31 Turbine driven AFW pump started on low SG level 09:49:39 Feedwater isolation on safety injection signal 09:50:01 Automatic safety injection (SI) on hi steam flow with low Tave < 543F 09:50:00 Main feedwater pumps A and B tripped on feedwater isolation signal 09:50:03 Main steam isolation on high steam flow with low Tave SI signal 09:57 1A reactor coolant pump secured, component cooling water to pump was isolated due to loss VB-1-III 10:00 SI manually reset, first charging pump secured 10:02 Pressurizer (PZR) level 100%, off scale high 10:05 High head SI isolated to cold legs 10:07 Transition to procedure 1-ES-1.1, SI Termination, from 1-E-0 as criteria were met to terminate SI 10:11 B low head SI pump secured 10:16 PZR PORV PCV-1445C cycled, first of 14 cycles 10:16 Letdown established on 45 gallon per minute orifice 10:37 Last PORV cycle 11:21 Fire in N-42 cabinet (PR, IR, SR remain energized, no blown fuses.)

11:24 Fire out in N-42 cabinet. Manual CO2 and dry chemical fire extinguishers discharged.

14:45 N-44 control power fuse blows, N-44 and N-43 de-energized. P-10 reset disabling high voltage power supply to source range (SR) NIs 14:50 SR NIs restored 15:24 Power restored to vital AC buses 1-III and 1-IIIA, all plant parameters are stable and in Mode 3 19:45 Last actions of 1-ES-1.1, SI Termination, are completed, procedure exited.

.2 Review and assess use of alarm response/abnormal operating/emergency operating

procedures during the event. The following specific aspects of the event should be included: rapid cooldown of primary, reactor coolant pump operation, safety injection operation including pressurizer level response, fire incident response, (Charter Item 2)

a. Inspection Scope

The inspectors reviewed and assessed the licensees use of alarm (annunciator)response procedures, abnormal operating procedures, and emergency operating (emergency) procedures during the event. The inspectors reviewed Historian (HSR)alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. The inspectors also reviewed the licensees licensed operator requalification program to determine the type of training that control room personnel received regarding the loss of a vital bus. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and commensurate with the licensees training program. The inspectors also observed a re-creation of the event in the plant simulator to assess the licensees response.

Documents reviewed are listed in Attachment 1.

The inspectors reviewed and assessed the licensees fire incident response. The inspectors reviewed operator logs, licensee fire protection procedures and fire protection strategies, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures and appropriate for the given conditions. Documents reviewed are listed in Attachment 1.

b. Observations and Findings

No findings were identified. The inspectors determined that the licensee utilized the appropriate procedures based on the given plant conditions. The plant conditions encountered and systems response were as expected and modeled in the simulator, based on the equipment and instrumentation lost due to the de-energization of the 1-III and 1-IIIA vital AC buses, with the exception of the 1A main feedwater regulating valve going closed. The licensee performed comprehensive troubleshooting on the valve in an effort to determine the cause of the closure but could not find any discrepancies. The 1A main feedwater valve controller was replaced as a contingency and retested satisfactorily prior to startup. The inspectors concluded that the licensees operator requalification program was adequate in the depth and breadth of training devoted to loss of a vital bus and was demonstrated by the operators correct response to the event. The inspectors evaluated the licensees actions as compliant with plant procedures and the licensees training program. The inspectors determined that the licensee actions for the fire incident response were proper and in accordance with plant procedures.

.3 Assess the available information on the loss of the 125 VAC bus 1-III and the

maintenance practices associated with the event. Review vital AC system work orders and related information to identify other potential vulnerabilities or maintenance practices, (Charter Item 3)

a. Inspection Scope

The inspection team reviewed corrective action documents, maintenance procedures, work orders, system drawings, vendor technical manual information, maintenance training requirements and qualification records, and interviewed engineering and maintenance personnel, to assess the maintenance practices associated with the event.

The inspectors also reviewed the licensees maintenance risk assessments associated with the regulating line conditioner circuit board replacement, scheduled work orders associated with the uninterruptible power supplies, and interviewed engineering personnel to identify and assess other potential vulnerabilities or maintenance practices.

b Findings No findings were identified. The inspectors found that the RLC circuit board replacement PM had been performed at least eight previous times on Units 1 and 2 without incident. During the development of maintenance procedure 0-ECM-0103-02, Station Battery UPS System Maintenance, it was not recognized during technical reviews by the engineering subject matter expert (SME), that an electrical fault during RLC circuit board replacement could cause a transfer of power from the normal supply to the tagged out alternate supply. The SME believed that the DC circuit that remained energized during the maintenance only affected alarms and indications. This was based on the fact that there were no vendor technical manual information or precautions highlighting this vulnerability, and that system drawings did not clearly link the DC circuit to operation of the static switch that transferred the power source. As a result, there were no precautions in the procedure related to lifting and landing the energized leads that caused the transient. The licensees lack of understanding of the potential consequences of the maintenance was also reflected in the licensees risk management tool for online maintenance. The work was coded to accurately reflect the perceived risk and therefore did not contribute to a significant increase in the online risk projected for the day. The previous multiple successful completions of the PM served to reinforce the licensees perception that there was no risk of losing power to the vital bus as a result of the maintenance. The inspectors determined that the licensees assessment was reasonable given the information available at the time.

The inspectors found that the technician that performed the maintenance was knowledgeable, experienced and fully qualified to perform the maintenance. The maintenance was performed in accordance with the procedure and the applicable electrical safety work practices. The task that led to the transient required the landing of three leads, two of which were energized, on the same terminal. The increased difficulty of the task contributed to the technician losing control of the energized lead. The technician had previously performed the circuit board replacement PM a number of times without incident.

The inspectors found that twenty five work orders were scheduled during the next 12 months associated with Unit 1 and Unit 2 UPS, a number of which dealt with the RLC.

The next scheduled RLC circuit board replacement was in September 2010 on 02-EP-UPS-2A-2. As a compensatory action or short term corrective action, the licensee has re-coded all vital bus UPS RLC PMs so that they can only be performed during Mode 5.

.4 Review licensee documents and other information to assess if the licensee knew or

should have known that the RC filters in the NI-42 and 44 cabinets were susceptible to arcing and failure based on similar occurrences in the Unit 2 NI 43 cabinet during the last refueling outage, (Charter Item 4)

a. Inspection Scope

The inspectors assessed whether the licensee had previous opportunities to identify degradation or failure modes of the RC filters similar to what occurred on June 8, 2010.

The inspectors reviewed vendor information for the NI cabinets to verify the licensee had incorporated appropriate vendor recommended maintenance practices in their surveillance, maintenance, and testing procedures, specifically for inspection and cleaning. Electrical drawings and schematics were reviewed to verify design requirements were accurately reflected in the drawings as well as in the field. Internal and external operating experience documents were reviewed to verify the licensee had performed adequate assessments of industry knowledge and related events with regards to RC filters. The inspectors also reviewed the licensees corrective action database to determine whether there were previous opportunities to identify and correct similar issues, and whether those instances were adequately evaluated and properly dispositioned.

b. Findings

Review of the vendor manuals for the NI cabinets identified that the recommended preventive maintenance was primarily limited to the upper half of the cabinets known as the drawer assembly. Recommended practices included inspection of the drawer assembly components for signs of overheating or deterioration. The lower half of the NI cabinets are known as the termination cabinets. The termination cabinets are where the RC filters are located. The inspectors noted that none of the documents reviewed explicitly addressed inspection of the termination cabinets for mechanical degradation or component deterioration. As part of the licensees corrective actions, all other locations for RC filters were inspected for potential component degradation. During walk-downs of the NI cabinets, the inspectors did not note any design discrepancies between the NI cabinet schematics and the actual equipment in the field.

An initial assessment and review of the available operating experience databases with respect to RC filters and NI cabinets did not immediately identify any similar instances to the June 8, 2010 event. However, during the event investigation, the licensee identified a similar event at another plant where the plant experienced a loss of control power for specific nuclear instrumentation when the transient voltage suppressors (the equivalent of a RC filter) began to overheat and burn inside the NI cabinet.

The inspectors reviewed condition report (CR) 358071 which documented a similar occurrence in November 2009 when an RC filter overheated and burned in the N-43 NI cabinet. Specifically, while performing maintenance on the NI cabinet, licensee staff observed some arcing and sparking and subsequently discovered a RC filter that caught fire. During the inspectors review, they noted the CR stated that following the discovery, the Instrumentation and Control (I&C) group should investigate the event to determine what could have caused the overheating. Upon follow-up, the inspectors found this investigation was not conducted, and the CR was subsequently closed with no apparent cause or corrective actions in place. The inspectors considered this a missed opportunity to discover the deteriorating and degraded condition of the RC filters and the potential for them to arc and cause a fire in the control room. In addition to the failure of the licensee to evaluate the fire in the NI cabinet, the inspectors also identified in the CR change history, that the description had been changed from small electrical fire to electrical resistor smoldering. After conducting interviews with licensee staff involved in the November 2009 event, it was determined that it was actually a fire/flame that was observed and not smoldering.

The licensees final root cause report for the June 8, 2010, event determined the event described in CR 358071 as a missed opportunity to perform causal analysis on the degraded RC filters. Also, the root cause report determined that the change in the description from fire to smoldering impacted the significance of the CR and the level of review the CR received.

As part of their immediate corrective actions for this issue, the licensee tested and replaced all of the RC filters in the NI cabinets. During subsequent investigation of the removed filters, the licensee determined that many were in a degraded condition.

Various indicators were used in determining degradation such as: visual cracking, measured capacitance, measured impedance, and capacitance leakage. Capacitance leakage testing can indicate a form of component degradation which may produce a short circuit or high resistance leakage path. Some RC filters exhibited sparking during this testing.

Failure to Identify and Correct Degraded RC Suppressors

Introduction:

An NRC identified non-cited violation of 10 CFR 50 Appendix B, Criteria XVI, Corrective Action was identified for the licensees failure to identify and correct degraded RC filters associated with Unit 1 nuclear instrument cabinets for N-42 and N-44 based on a similar degraded condition identified on Unit 2 NI cabinet N-43 in November 2009.

Description:

In November 2009 while performing maintenance on Unit 2 nuclear instrumentation cabinet N-43, some arcing and sparking was observed by the licensee from the rear of the NI cabinet. Upon further investigation, it was observed that a RC filter on a spare termination had caused the observed arcing and subsequently caught fire. The fire was immediately extinguished, the work was stopped, and the licensee initiated CR 358071 to evaluate the event.

In June 2010 following a similar event where an RC filter in the Unit 1 NI cabinet N-42 overheated and caught fire, and an RC filter in cabinet N-44 overheated and smoldered, the inspectors reviewed the licensees response to the November 2009 event documented in CR 358071. The inspectors noted the licensee did not perform an investigation, as stated in the CR, of the event to determine possible reasons for the arcing and resulting fire of the RC filter in the Unit 2 cabinet N-43. No corrective actions were taken to determine the cause or the extent of condition.

The inspectors also identified that the description of CR 358071 had been changed from fire to smoldering which subsequently had an impact on how the licensee dispositioned the CR. The licensee has since performed a root cause evaluation for the June 8, 2010, event and determined that the organization missed an opportunity to perform a cause analysis on the November 2009 event and that the inaccurate description documented in CR 358071 contributed to a decreased significance and organization response. As part of the corrective actions, the licensee has replaced all the RC filters in the NI cabinets on both units and has corrective actions to update the aging management program for the RC filters since recognizing the age degradation issue and potential consequences. The inspectors determined that it was within the licensees ability to identify and correct the degraded Unit 1 RC filters prior to failure, based on internal OE information related to the November 2009 Unit 2 RC filter failure.

Analysis:

The failure to identify and correct the degraded Unit 1 RC filters following a similar Unit 2 RC filter failure in November, 2009 is a performance deficiency. The finding was determined to be of more than minor significance because it is associated with equipment performance attribute of the Initiating Events Cornerstone. It adversely affected the cornerstone objective of protection against external events, i.e., fire.

Specifically, the degraded condition of the RC filters in the NI cabinets affected the Reactor Protection System and resulted in loss of source range NIs for a period of time, affecting the transient initiator contributor. The fire that resulted from the overheating and arcing of the RC filters increased the external event initiator.

The licensee failed to identify and correct degraded R/C filters associated with the Unit 1 Nuclear Instrumentation (NI) Cabinets NI-42 and NI-44 based on internal operating experience from a similar degraded condition on Unit 2 in 2009. The degraded RC filters resulted in fires in the Unit 2 main control room (MCR) NI cabinets NI-42 and NI-44. The performance deficiency was screened using phase 1 of the Significance Determination Process (SDP) and was determined to be a fire initiator contributor and to have impact on post fire safe shutdown, therefore phase 2 analyses utilizing Inspection Manual chapter 0609 Appendix F was required. Since the finding consisted of MCR fire scenarios a phase 3 analysis was required. A phase 3 risk analysis was performed by a regional SRA in accordance with IMC 0609 Appendix F, NUREG/CR6850, and utilizing the latest Surry SPAR probabilistic risk analysis model. The fire scenarios were determined to impact MCR operator actions but would not credibly require MCR evacuation for either habitability or safe shutdown functional requirements. The dominant sequence was a fire induced reactor trip transient initiator, with failures of auxiliary feedwater, main feedwater and failure to implement feed and bleed leading to core damage. Human error probability for MCR actions was increased to account for use of self contained breathing apparatus during the fire scenarios. Factors which mitigated the risk of the fire were the minimal fire growth potential and the potential for NI cabinet fires to damage SSD equipment. The risk evaluation result was an increase of

<1E-6 for core damage frequency, a finding of very low risk significance (Green). This finding involved the cross cutting area of problem identification and resolution, the component of operating experience (OE), and the aspect of evaluating internal OE (P.2.a), because the licensee did not effectively evaluate the internal operating experience gained from the November 2009 RC filter failure prior to the failure of the RC filters on June 8, 2010.

Enforcement:

10 CFR Part 50, Appendix B, Criteria XVI, states, in part, that measures shall be established to assure that conditions adverse to quality, such as failures, malfunctions, deficiencies, deviations, defective material and equipment, and nonconformances are promptly identified and corrected. Contrary to the above, the licensee failed to identify and correct degraded RC filters associated with Unit 1 Nuclear Instrumentation cabinets for N-42 and N-44, a condition adverse to quality, which should have been identified after a similar degraded condition was identified on Unit 2 in November 2009. This issue has been entered into the licensees corrective action program as CR 383881. Because the finding is of very low safety significance (Green)and it was entered into the licensees CAP as CR 383881, this violation is being treated as an NCV, consistant with section VI.A.1 of the NRC Enforcement Policy: NCV 05000280/2010006-01, Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters.

.5 Ascertain the type of fires that actually occurred and the zone of influence of the fires,

(Charter Item 5)

a. Inspection Scope

The inspectors assessed the evidence gathered by the licensee following the fire in the NI cabinets to evaluate the extent of the damage caused by the degraded RC filters in NI cabinet N-42, terminal block 224, terminations 1 and 2 and NI cabinet N-44, terminal block 424, terminations 1 and 2; including all the affected RC filters that were burned, charred, or damaged by smoke. The team interviewed licensee staff that observed the fire and those that extinguished the fire. A walk-down of the affected equipment was conducted in order to assess the appropriate zone of influence. Schematics of the affected equipment were used to identify and verify proper field design configuration.

b. Findings

No findings were identified. For the fire that occurred in the Unit 1 N-42 cabinet, the inspectors evaluated the evidence gathered by the licensee to determine an accurate zone of influence (ZOI). The zone of influence is considered the surrounding area affected by an ignition source and its potential flame plume and radiant heating effects.

In this case, the ignition source is the RC filter at terminations 1 and 2. The inspector reviewed photographs taken by the licensee that showed the affects of RC filter in N-42 at terminations 1 and 2 arcing and catching fire.

Each RC filter is approximately 1 inch long. The terminations for the RC filters are aligned vertically in the rear of the NI cabinet. From the evidence reviewed, it was determined that once the RC filter in N-42 caught fire, it burned with sufficient intensity to significantly burn the RC filter immediately below it at terminations 3 and 4. This RC filter was found severely damaged. The flame travelled up along the termination raceway and affected the RC filter at terminations 11 and 12 (approximately 1 inch above RC filter at terminations 1 and 2). This RC filter caught fire and propagated the flame further up the termination board. The flame then charred and burned two more RC filters at terminations 7 and 8, and 5 and 6, respectively (approximately 1 inch above RC filter at terminations 11 and 12). Based on the photographs, NI cabinet schematics, and data charts provided by the licensee, the zone of influence of the flame plume was determined to be a total of 6 along the termination board - from 1 inch below the ignition source (RC filter at termination 1 and 2) to 5 inches above the ignition source.

The flames did not appear to spread outside the width of the RC filters. Based on the dimensions, the fire was within the expected ZOI. Smoke residue was observed to have been found in the same general ZOI. Termination wire damage was observed as well, but within the same general ZOI.

The RC filter that overheated in NI-44, terminations 1 and 2, approximately 3 hours3.472222e-5 days <br />8.333333e-4 hours <br />4.960317e-6 weeks <br />1.1415e-6 months <br /> after the RC filter had ignited in N-42, only charred and did not cause any damage to the surrounding components.

.6 Assess and review the licensees evaluation regarding 1A RCP loss of CCW for 10

minutes, (Charter Item 6)

a. Inspection Scope

The inspectors reviewed and assessed the licensees evaluation of the loss of component cooling water (CCW) flow to the 1A reactor coolant pump (RCP) for 10 minutes during the event. The inspectors reviewed HSR alarm messages, operator logs, and PCS data to evaluate that appropriate procedures were utilized based on given plant conditions. In addition, the inspectors reviewed completed procedures, operator logs, and interviewed licensee personnel to verify that actions taken were in accordance with plant procedures. The inspectors also reviewed PCS data and equipment design documentation to validate that temperature thresholds for the 1A RCP were not exceeded.

b. Observations and Findings

No findings were identified. The inspectors concluded that the licensee utilized the appropriate procedures based on the given plant conditions. The inspectors also evaluated the licensees actions as compliant with plant procedures. In addition, the inspectors verified that the vendor recommended maximum temperatures for the 1A RCP were not exceeded.

.7 Review the licensees corrective actions (CAs), causal analysis and extent of condition

associated with the loss of the vital bus and with the RC filter failures, (Charter Items 7 and 8)

a. Inspection Scope

The inspectors reviewed the licensees root cause evaluation reports RCE001013, Failure of RC Suppressors in the NIS Cabinets, and RCE001014, Surry Unit 1 Reactor Trip due to a Loss of Vital Bus 1-III, to assess the adequacy of the licensees causal analysis, corrective actions and extent of condition associated with these issues.

b. Observations and Findings

No findings were identified. The inspectors found the root cause evaluations performed by the licensee were thorough for both issues and that reasonable root causes were identified. Analysis methods used by the licensee included barrier analysis, task analysis, and human error evaluation flow charting for the loss of vital bus event, and failure modes affects analysis and fault tree analysis for the RC filter failures.

The licensee determined that the direct root cause of the loss of vital bus was human error by the technician when he allowed an energized lead to slip from his hand during the RLC maintenance. An additional root cause identified by the licensee was a knowledge deficiency associated with the operation and inputs to the inverter sync switch. This knowledge deficiency resulted in a lack of understanding of the potential consequences of the maintenance. As a result, barriers that could have prevented the event were not identified and implemented. The licensee determined that the root cause for the RC filter failures was a short circuit of the capacitor which resulted in high current flow and heat, which ignited the epoxy casing of the filters. The most probable cause of the short was due to hardening and cracking of the RC filters epoxy insulation due to aging.

The inspectors found that corrective actions implemented by the licensee for both issues addressed the root causes and should be effective in preventing reoccurrence. In addition to recoding all UPS RLC PMs to be worked during Mode 5, corrective actions identified by the licensee for the loss of vital bus included additional training related to the event for engineering and maintenance personnel, improvements in procedures for operational risk assessment, and improvements in technical review requirements for maintenance technical procedure development. Corrective actions completed for the RC filter failures included the replacement of all RC filters in the Unit 1 and 2 NI cabinets with new filters. Additional corrective actions planned are to replace RC filters in the Unit 1 and 2 protection channel relay racks and to develop a preventive maintenance strategy for RC filters in the NI, reactor protection, ultrasonic flow measuring, radiation monitoring and vital bus UPS systems.

The inspectors found that the licensees extent of condition reviews for the loss of vital bus and the RC filter failures were sufficiently broad to address other potential vulnerabilities related to these events. Appropriate corrective actions were identified to address the additional scope of these issues.

.8 Collect data necessary to support completion of the significance determination process,

(Charter Item 9)

The inspectors interviewed licensee personnel and reviewed operations logs, licensee procedures, corrective action program documents, work orders, root cause evaluations, operability assessments, and engineering evaluations to gather data necessary to develop and assess the safety significance of any findings. This information was provided to the regional Senior Reactor Analyst to assess the significance of the event and to support a Phase 3 analysis for the NI RC filter fire in the main control room.

.9 Identify any potential generic safety issues and make recommendations for appropriate

follow-up action (e.g., Information Notices, Generic Letters, Bulletins), (Charter Item 10)

a. Inspection Scope

The inspectors reviewed the licensees root cause evaluation including relevant operating experience, corrective action program documents, work orders, and the NRC Operating Experience (OpE) database to determine the potential for generic safety issues related to the degraded conditions and causes pertaining to this event.

b. Findings and observations

No findings were identified. The root cause of the RC filter fires was related to aging degradation/cracking of the epoxy covering of filters. Fires and/or overheating occurred during two separate events at Surry in 2009 and 2010. Additionally, a similar RC surge suppressor overheating event at Turkey Point related to aging was identified by the licensee as a result of an industry OE search. As a result of the RC filter fires being attributed to a generic aging issue and the likelihood that similar RC filters of comparable age exist at other nuclear plants, the inspection team recommended that an information notice be issued by the NRC to communicate this issue to the industry.

4OA6 Meetings, Including Exit

On August 6, 2010, the inspection results were presented to Mr. Bischoff and other members of his staff, who acknowledged the findings. The inspectors asked the licensee whether any of the material examined during the inspection should be considered proprietary. No proprietary information was identified.

ATTACHMENT:

SUPPLEMENTAL INFORMATION

KEY POINTS OF CONTACT

Licensee Personnel

M. Adams, Director, Station Engineering
G. Bischof, Site Vice President
B. Garber, Supervisor, Licensing
K. Grover, Manager, Operations
A. Harrow, Supervisor, Electrical Systems
R. Johnson, Manager, Outage and Planning
R. Manrique, Supervisor, Primary Systems
C. Olsen, Manager, Site Engineering
L. Ragland, Supervisor, Health Physics Operations
K. Sloane, Plant Manager (Nuclear)
B. Stanley, Director, Station Safety and Licensing
M. Wilda, Supervisor, Auxiliary Systems
J. Eggart, Manager, Radiation Protection & Chemistry
B. Hilt, Supervisor, HP Technical Services
D. White, Supervisor, ALARA
D. Godwin, Supervisor, Nuclear Engineering

LIST OF ITEMS

OPENED, CLOSED AND DISCUSSED

Opened

05000280/2010006-01 NCV Failure to Identify and Correct Degraded Unit 1 Nuclear Instrument RC Filters (Section 4OA5.4)

LIST OF DOCUMENTS REVIEWED

Retrieved from "https://kanterella.com/w/index.php?title=IR_05000280/2010006&oldid=2284771"