IR 05000440/2018040: Difference between revisions

From kanterella
Jump to navigation Jump to search
(Created page by program invented by StriderTol)
(Created page by program invented by StriderTol)
 
(5 intermediate revisions by the same user not shown)
Line 19: Line 19:


=Text=
=Text=
{{#Wiki_filter:UNITED STATES NUCLEAR REGULATORY COMMISSION REGION III 2443 WARRENVILLE RD. SUITE 210 LISLE, ILLINOIS 60532-4352 March 27, 2018 EA-17-043
{{#Wiki_filter:UNITED STATES NUCLEAR REGULATORY COMMISSION rch 27, 2018


Mr. David Hamilton Site Vice President FirstEnergy Nuclear Operating Company Perry Nuclear Power Plant Mail Stop A-PY-A290 P.O. Box 97, 10 Center Road Perry, OH 44081-0097
==SUBJECT:==
PERRY NUCLEAR POWER PLANTNRC SUPPLEMENTAL INSPECTION REPORT 05000440/2018040 AND ASSESSMENT FOLLOW-UP LETTER


SUBJECT: PERRY NUCLEAR POWER PLANT-NRC SUPPLEMENTAL INSPECTION REPORT 05000440/2018040 AND ASSESSMENT FOLLOW-UP LETTER
==Dear Mr. Hamilton:==
The U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection Response To Action Matrix Column 2 Inputs, at your Perry Nuclear Power Plant. On February 15, 2018, the NRC discussed the results of this inspection and the implementation of your corrective actions with Mr. Frank Payne and other members of your staff in an exit Meeting. On February 16, 2018, the NRC discussed the causes and your corrective actions identified as a result of the WHITE finding, and the criteria required for returning to the Licensee Response Column of the Action Matrix with Mr. Frank Payne in a Regulatory Performance Meeting. The results of this inspection are documented in the enclosed report.
 
By letter dated August 24, 2017, (ADAMS ML17236A187), the U.S. Nuclear Regulatory Commission (NRC) informed you that as a result of a White (low-to-moderate safety significance) finding in the Mitigating Systems Cornerstone that the NRC assessed Perry Nuclear Power Plant performance to be in the Regulatory Response Column of the Reactor Oversight Process (ROP) Action Matrix, effective the second quarter of 2017. In the same letter, the NRC informed you of our intent to perform a supplemental inspection using Inspection Procedure 95001 upon notification of your readiness for the inspection. By letter dated October 31, 2017, we were notified by your staff that you were ready for the NRC to conduct our supplemental inspection to review the actions taken to address the White finding.


==Dear Mr. Hamilton:==
The NRC performed this inspection to review your stations actions in response to a White finding in the Mitigating System cornerstone which was documented in NRC Inspection Report 05000440/2017009 (ML17156A750). The finding involved the failure to evaluate the effects of voltage suppression diode failure on the Emergency Diesel Generator (EDG) control circuit, which was a component subject to the requirements of Title 10 of the Code of Federal Regulations (CFR) Part 50, Appendix B. Specifically, FirstEnergy failed to consider the effect of a shorted diode on the control circuitry of the EDG, and, as a result, failed to recognize that installation of voltage suppression diodes across control relays, with no mitigation for diode failure, was not suitable for the EDG control circuit. This introduction of new components (diodes) into the control circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering the EDG inoperable and unable to start.
The U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection using Inspection Procedure 95001, "Supplemental Inspec tion Response To Action Matrix Column 2 Inputs," at your Perry Nuclear Power Plant. On February 15, 2018, the NRC discussed the results of this inspection and the implementation of your corrective actions with Mr. Frank Payne and other members of your staff in an exit Meeting. On February 16, 2018, the NRC discussed the causes and your corrective actions identified as a result of the WHITE finding, and the criteria required for returning to the Licensee Response Column of the Action Matrix with Mr. Frank Payne in a Regulatory Performance Meeting. The results of this inspection are documented in the enclosed report. By letter dated August 24, 2017, (ADAMS ML17236A187), the U.S. Nuclear Regulatory Commission (NRC) informed you that as a result of a White (low-to-moderate safety significance) finding in the Mitigating Systems Cornerstone that the NRC assessed Perry Nuclear Power Plant performance to be in the Regulatory Response Column of the Reactor Oversight Process (ROP) Action Matrix, effective the second quarter of 2017. In the same letter, the NRC informed you of our intent to perform a supplemental inspec tion using Inspection Procedure 95001 upon notification of your readiness for the inspection. By letter dated October 31, 2017, we were notified by your staff that you were ready for the NRC to conduct our supplemental inspection to review the actions taken to address the White finding. The NRC performed this inspection to review your station's actions in response to a White finding in the Mitigating System cornerstone which was documented in NRC Inspection Report 05000440/2017009 (ML17156A750). The finding involved the failure to evaluate the effects of voltage suppression diode failure on the Emergency Diesel Generator (EDG) control circuit, which was a component subject to the requirements of Title 10 of the Code of Federal Regulations (CFR) Part 50, Appendix B. Specifically, FirstEnergy failed to consider the effect of a shorted diode on the control circuitry of the EDG, and, as a result, failed to recognize that installation of voltage suppression diodes across control relays, with no mitigation for diode failure, was not suitable for the EDG control circuit. This introduction of new components (diodes) into the control circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering the EDG inoperable and unable to start. The NRC determined that the completed or planned corrective actions were sufficient to address the performance issue that led to the White finding previously described and were prioritized commensurate with the safety significance of the issue. In addition, the NRC determined that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event. No finding or violations were documented. After reviewing Perry Nuclear Power Plant's performance in addressing the White finding subject of this Inspection Procedure 95001, the NRC concluded your actions met the objectives of the inspection. Therefore, in accordance with the guidance in Inspection Manual


Chapter (IMC) 0305, "Operating Reactor Assessment Program," the White finding will only be considered in assessing plant performance for a total of four quarters. As a result, the NRC determined the performance at Perry Nuclear Power Plant to be in the Licensee Response Column of the ROP Action Matrix as of April 1, 2018.
The NRC determined that the completed or planned corrective actions were sufficient to address the performance issue that led to the White finding previously described and were prioritized commensurate with the safety significance of the issue. In addition, the NRC determined that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event.


This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with 10 CFR 2.390, "Public Inspections, Exemptions, Requests for Withholding."
No finding or violations were documented.


Sincerely,/RA/
After reviewing Perry Nuclear Power Plants performance in addressing the White finding subject of this Inspection Procedure 95001, the NRC concluded your actions met the objectives of the inspection. Therefore, in accordance with the guidance in Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program, the White finding will only be considered in assessing plant performance for a total of four quarters. As a result, the NRC determined the performance at Perry Nuclear Power Plant to be in the Licensee Response Column of the ROP Action Matrix as of April 1, 2018.
Billy Dickson, Chief


Branch 5 Division of Reactor Projects Docket No. 50-440 License No. NPF-58
This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with 10 CFR 2.390, Public Inspections, Exemptions, Requests for Withholding.


===Enclosure:===
Sincerely,
/RA/
Billy Dickson, Chief Branch 5 Division of Reactor Projects Docket No. 50-440 License No. NPF-58 Enclosure:
Inspection Report 05000440/2018040 cc: Distribution via LISTSERV
Inspection Report 05000440/2018040 cc: Distribution via LISTSERV


=SUMMARY=
=SUMMARY=
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring licensee's performance by conducting a supplemental inspection at Perry Nuclear Power Plant in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRC's program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information. NRC and self-revealed findings, violations, and additional items are summarized in the table below.
The U.S. Nuclear Regulatory Commission (NRC) continued monitoring licensees performance by conducting a supplemental inspection at Perry Nuclear Power Plant in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information. NRC and self-revealed findings, violations, and additional items are summarized in the table below.


List of Findings and Violations No findings or violations were identified.
List of Findings and Violations No findings or violations were identified.


Additional Tracking Items Type Issue Number Title Report Section Status NOV 05000440/2017009-01 Division 2 Diesel Generator Failure to Start due to a Failed Diode in the 125 VDC Control Power Circuit 95001 Closed 3 TABLE OF CONTENTS
Additional Tracking Items Type     Issue Number                 Title                       Report           Status Section NOV     05000440/2017009-01         Division 2 Diesel Generator 95001            Closed Failure to Start due to a Failed Diode in the 125 VDC Control Power Circuit
 
TABLE OF CONTENTS


=INSPECTION SCOPES=
=INSPECTION SCOPES=
................................................................................................................ 4


==OTHER ACTIVITIES==
==OTHER ACTIVITIES==
-TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL
TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL


==INSPECTION RESULTS==
==INSPECTION RESULTS==
Line 60: Line 65:


=DOCUMENTS REVIEWED=
=DOCUMENTS REVIEWED=
......................................................................................................... 13
......................................................................................................... 13
INSPECTION SCOPES
INSPECTION SCOPES
Inspections were conducted using the appropriate portions of the inspection procedure (IPs) in
Inspections were conducted using the appropriate portions of the inspection procedure (IPs) in
effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with their attached revision histories are located on the public website at http://www.nrc.gov/reading-rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared complete when the IP requirements most appropriate to the inspection activity were met consistent with Inspection Manual Chapter (IMC) 2515, "Light-Water Reactor Inspection
effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with
Program - Operations Phase.The inspectors reviewed selected procedures and records, observed activities, and interviewed personnel to assess licensee performance and compliance with Commission rules and regulations, license conditions, site procedures, and standards.
their attached revision histories are located on the public website at http://www.nrc.gov/reading-
OTHER ACTIVITIES-TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL
rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared
complete when the IP requirements most appropriate to the inspection activity were met
consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection
Program - Operations Phase. The inspectors reviewed selected procedures and records,
observed activities, and interviewed personnel to assess licensee performance and compliance
with Commission rules and regulations, license conditions, site procedures, and standards.
OTHER ACTIVITIESTEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
Perry Nuclear Power Plant entered the Regulatory Response column of NRC's Action Matrix
Perry Nuclear Power Plant entered the Regulatory Response column of NRCs Action Matrix
in the 2nd quarter of 2017 due to a White (low-to-moderate safety significance) inspection
in the 2nd quarter of 2017 due to a White (low-to-moderate safety significance) inspection
finding, 05000440/2017009-01, in the Mitigating Systems Cornerstone. The finding was
finding, 05000440/2017009-01, in the Mitigating Systems Cornerstone. The finding was
associated with the station's failure
associated with the stations failure to evaluate the effects of voltage suppression diode failure
to evaluate the effects of voltage suppression diode failure
on the Emergency Diesel Generator (EDG) control circuit. The finding was characterized as
on the Emergency Diesel Generator (EDG) control circuit. The finding was characterized as having White safety significance based on the results of a risk assessment performed by a
having White safety significance based on the results of a risk assessment performed by a
region-based Senior Reactor Analyst. This issue was documented in NRC Inspection
region-based Senior Reactor Analyst. This issue was documented in NRC Inspection
Reports (IRs) 05000440/2017009 and 05000440/2017010. By letter dated November 3, 2017, the licensee notified the NRC that it had completed its evaluation of the circumstances surrounding the degraded performance and were ready for the
Reports (IRs) 05000440/2017009 and 05000440/2017010.
NRC to assess the evaluation and subsequent corrective actions. In preparation for the inspection, the licensee performed Root Cause Evaluation (RCE), "PRA Evaluation Associated with the Div. 2 EDG MSPI Failure (CR 2016-14456) Indicates a Preliminary Level of
By letter dated November 3, 2017, the licensee notified the NRC that it had completed its
Significance that Warrants a Root Cause Evaluation," Revision 2, dated October 9, 2017. It
evaluation of the circumstances surrounding the degraded performance and were ready for the
identified weaknesses that existed in various organizations which allowed for a risk-significant finding, and to determine the organizational attributes that resulted in the White finding.
NRC to assess the evaluation and subsequent corrective actions. In preparation for the
inspection, the licensee performed Root Cause Evaluation (RCE), PRA Evaluation Associated
with the Div. 2 EDG MSPI Failure (CR 2016-14456) Indicates a Preliminary Level of
Significance that Warrants a Root Cause Evaluation, Revision 2, dated October 9, 2017. It
identified weaknesses that existed in various organizations which allowed for a risk-significant
finding, and to determine the organizational attributes that resulted in the White finding.
The affected diesel generators are identified as the Standby Diesel Generator and Emergency
The affected diesel generators are identified as the Standby Diesel Generator and Emergency
Diesel Generator in this report's reference documents and are used interchangeably. This
Diesel Generator in this reports reference documents and are used interchangeably. This
report uses Emergency Diesel Generator for consistency.
report uses Emergency Diesel Generator for consistency.
The inspectors reviewed the licensee's root causes, contributing causes, extent of condition, and extent of cause determinations. The inspectors assessed whether the licensee's corrective
The inspectors reviewed the licensees root causes, contributing causes, extent of condition,
and extent of cause determinations. The inspectors assessed whether the licensees corrective
actions to address the root and contributing causes were sufficient to prevent recurrence. The
actions to address the root and contributing causes were sufficient to prevent recurrence. The
highlights of the performance review
highlights of the performance review and NRCs assessment are documented below.
and NRC's assessment are documented below.  
Evaluation of Inspection Requirements
 
  (1) Problem Identification
Evaluation of Inspection Requirements
(1) Problem Identification
a) [Describe who identified the issue and under what conditions the issue was
a) [Describe who identified the issue and under what conditions the issue was
identified.
identified.]
] Root cause CR 2016-14456 evaluation concluded that the diode failure (shorted) for relay R10BB, 1R43Q0003B, which resulted in loss of the 125 Volts, Direct
Root cause CR 2016-14456 evaluation concluded that the diode failure (shorted) for
Current, (VDC) control power to the Division 2 EDG, was self-revealed on November 6, 2016, while performing surv
relay R10BB, 1R43Q0003B, which resulted in loss of the 125 Volts, Direct
eillance test PTI-R43-P0006B, "Division 2
Current, (VDC) control power to the Division 2 EDG, was self-revealed on
Diesel Generator Pneumatic Logic Board Functional Check.Operators received various unexpected alarms when a jumper was installed to simulate an
November 6, 2016, while performing surveillance test PTI-R43-P0006B, Division 2
emergency (auto) start as part of the surveillance test. b) [Describe how long the issue existed and prior opportunities for identification.
Diesel Generator Pneumatic Logic Board Functional Check. Operators received
] Root cause CR 2016-14456 evaluation concluded this issue existed when the suppression diodes were installed in the control circuitry of the Division 2 EDG on April 21, 2007. The licensee identified that an opportunity was missed after the diode on relay R11 failed on May 6, 2016. On April 21, 2007, the licensee initiated and discussed in CR-2016-06450 that the new Allen Brandley relays R11A, R10BB, RSS, K4, and associated diodes, were installed on the Division 2 EDG by ECP 04-0049. This
various unexpected alarms when a jumper was installed to simulate an
emergency (auto) start as part of the surveillance test.
b) [Describe how long the issue existed and prior opportunities for identification.]
Root cause CR 2016-14456 evaluation concluded this issue existed when the
suppression diodes were installed in the control circuitry of the Division 2 EDG on
April 21, 2007.
The licensee identified that an opportunity was missed after the diode on relay R11
failed on May 6, 2016. On April 21, 2007, the licensee initiated and discussed in
CR-2016-06450 that the new Allen Brandley relays R11A, R10BB, RSS, K4, and
associated diodes, were installed on the Division 2 EDG by ECP 04-0049. This
relay was energized on May 06, 2016 during a monthly Surveillance test. The diode
relay was energized on May 06, 2016 during a monthly Surveillance test. The diode
in parallel with the relay R11A failed causing alarms, abnormal indications, and
in parallel with the relay R11A failed causing alarms, abnormal indications, and
required operator action to shutdown the DIV2 ED
required operator action to shutdown the DIV2 EDG. The diode was replaced based
: [[contact::G. The diode was replaced based on CR-2016-06450 on May 06]], 2016. The licensee did not then consider the effects of a shorted diode, for other diodes in the system and other EDGs. c) [Describe the plant-specific risk consequences, as applicable, and compliance concerns associated with the issue.
on CR-2016-06450 on May 06, 2016. The licensee did not then consider the
] The licensee did not document a Probabilistic Risk Assessment (PRA) evaluation for
effects of a shorted diode, for other diodes in the system and other EDGs.
this condition. The Licensee discussed
c) [Describe the plant-specific risk consequences, as applicable, and compliance
the NRC risk assessment that concluded the safety significance for this issue as White (low-to-moderate) based on the
concerns associated with the issue.]
The licensee did not document a Probabilistic Risk Assessment (PRA) evaluation for
this condition. The Licensee discussed the NRC risk assessment that concluded the
safety significance for this issue as White (low-to-moderate) based on the
Division 2 EDG not operable and unable to perform its emergency start function from
Division 2 EDG not operable and unable to perform its emergency start function from
April 2, 2015, when it passes a surveillance test, until November 8, 2016 when the emergency start diode was replaced after failure and the Division 2 EDG was returned to service. Because the licensee was not aware of the Div 2 EDG's
April 2, 2015, when it passes a surveillance test, until November 8, 2016 when the
inoperability during the unit's operation cycle, the required actions of TS 3.8.1.F.1
emergency start diode was replaced after failure and the Division 2 EDG was
returned to service. Because the licensee was not aware of the Div 2 EDGs
inoperability during the units operation cycle, the required actions of TS 3.8.1.F.1
and 3.8.1.F.2 were not followed. The evaluation also recognized this determination
and 3.8.1.F.2 were not followed. The evaluation also recognized this determination
resulted in being placed in the Regulatory Response Column of the NRC's Reactor Oversight Process (ROP) Action Matrix.    
resulted in being placed in the Regulatory Response Column of the NRCs Reactor
(2) Root Cause, Extent-of-Condition, and Extent-of-Cause Evaluation
Oversight Process (ROP) Action Matrix.
(2) Root Cause, Extent-of-Condition, and Extent-of-Cause Evaluation
a) [Describe systematic methodology used to identify the root and contributing causes.]
a) [Describe systematic methodology used to identify the root and contributing causes.]
The licensee assigned a multi-disciplined team
The licensee assigned a multi-disciplined team to perform a Root Cause Evaluation
to perform a Root Cause Evaluation to identify the root and contributing causes of the EDG inoperability and the inability
to identify the root and contributing causes of the EDG inoperability and the inability
to start. In its root cause analysis, the licensee used the following processes to identify the root and contributing causes:  
to start. In its root cause analysis, the licensee used the following processes to
* Data gathering through interviews and document review;  
identify the root and contributing causes:
* Event and Causal Factor Charting;  
      *   Data gathering through interviews and document review;
* Internal and External Laboratory Analysis;  
      *   Event and Causal Factor Charting;
* Equipment Apparent Cause Evaluation (EACE);  
      *   Internal and External Laboratory Analysis;
* Latent Organization Weakness Analysis;  
      *   Equipment Apparent Cause Evaluation (EACE);
* Performance Improvement International Failure Modes Analysis; and  
      *   Latent Organization Weakness Analysis;
* MORT Analysis.
      *   Performance Improvement International Failure Modes Analysis; and
      *   MORT Analysis.
The licensee used the above processes to evaluate equipment and human
The licensee used the above processes to evaluate equipment and human
performance issues associated with the event. The processes used were systematic processes governed by and implemented in accordance with the licensee's procedures. The processes were used to ensure the information and circumstances
performance issues associated with the event. The processes used were systematic
processes governed by and implemented in accordance with the licensees
procedures. The processes were used to ensure the information and circumstances
surrounding the problem were understood. The results of these processes revealed
surrounding the problem were understood. The results of these processes revealed
two root causes and one contributing cause. The initial root cause evaluation
two root causes and one contributing cause. The initial root cause evaluation
attributed the failure to a manufacturing defect in the diode with a contributing cause
attributed the failure to a manufacturing defect in the diode with a contributing cause
related to engineering design practices. The Notice of Violation (NOV) was issued in August of 2017 for failing to recognize that installation of voltage suppression diodes across control relays, with no mitigation for diode failure, was not suitable for the
related to engineering design practices. The Notice of Violation (NOV) was issued in
August of 2017 for failing to recognize that installation of voltage suppression diodes
across control relays, with no mitigation for diode failure, was not suitable for the
EDG control circuit. This introduction of new components (diodes) into the control
EDG control circuit. This introduction of new components (diodes) into the control
circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering
circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering
the EDG inoperable and unable to start. After the NOV was issued, Revision 2 of the
the EDG inoperable and unable to start. After the NOV was issued, Revision 2 of the
Root Cause Evaluation identified using diodes as a common design practice as a second root cause. This aligned closer with the NOV. The licensee also determined
Root Cause Evaluation identified using diodes as a common design practice as a
that discrepancies with vendor acceptance criteria used during initial dedication was a contributing cause. b) [Describe level of detail of detail used in the root cause evaluation and indicate whether it was commensurate with the significance of the problem.
second root cause. This aligned closer with the NOV. The licensee also determined
]
that discrepancies with vendor acceptance criteria used during initial dedication was
In its root cause analysis, the licensee classified the event as a "significant condition adverse to quality" and applied thorough processes and methodologies to identify the root and contributing causes associated with the licensee's failure
a contributing cause.
to evaluate the effects of voltage suppression diode failure on the EDG control circuit. The analysis
b) [Describe level of detail of detail used in the root cause evaluation and indicate
whether it was commensurate with the significance of the problem.]
In its root cause analysis, the licensee classified the event as a significant condition
adverse to quality and applied thorough processes and methodologies to identify the
root and contributing causes associated with the licensees failure to evaluate the
effects of voltage suppression diode failure on the EDG control circuit. The analysis
performed in the root cause were of sufficient detail commensurate with the
performed in the root cause were of sufficient detail commensurate with the
significance of the problem. The licensee's root cause evaluation included a timeline of events. The licensee conducted interviews, and performed work history and corrective action document reviews to capture the issues leading up to the event.
significance of the problem. The licensees root cause evaluation included a timeline
The root cause evaluation resulted in corrective actions that promptly addressed the design issues with the use of the diodes in the EDG control circuitry that would
of events. The licensee conducted interviews, and performed work history and
prevent recurrence. The engineering deficiency that was the basis of the NOV occurred in 2005 and the root cause evaluation addressed engineering issues and
corrective action document reviews to capture the issues leading up to the event.
The root cause evaluation resulted in corrective actions that promptly addressed the
design issues with the use of the diodes in the EDG control circuitry that would
prevent recurrence. The engineering deficiency that was the basis of the NOV
occurred in 2005 and the root cause evaluation addressed engineering issues and
performance since then to characterize current behavior and develop corrective
performance since then to characterize current behavior and develop corrective
actions. c) [Describe prior occurrences and operating experience considered during the root cause evaluation.
actions.
] The licensee included an evaluation of internal and external Operating Experience (OE) for issues with issues related to the loss of control power resulting from failed, shorted diodes. The licensee found that failures of diodes used to
c) [Describe prior occurrences and operating experience considered during the root
cause evaluation.]
The licensee included an evaluation of internal and external Operating
Experience (OE) for issues with issues related to the loss of control power resulting
from failed, shorted diodes. The licensee found that failures of diodes used to
suppress the inductive kickback of DC coils are not common at Perry nor within
suppress the inductive kickback of DC coils are not common at Perry nor within
the FENOC fleet.
the FENOC fleet.
Review of the external operating experience from INPO and NRC Websites was also conducted that revealed seven issues related to diode failures. Three of these
Review of the external operating experience from INPO and NRC Websites was also
conducted that revealed seven issues related to diode failures. Three of these
occurred prior to completion of the engineering change package in 2005 that
occurred prior to completion of the engineering change package in 2005 that
subsequently installed the suppression diodes in the EDG control circuit. While
subsequently installed the suppression diodes in the EDG control circuit. While
several of these events included failed, shorted diodes, none of the applications were
several of these events included failed, shorted diodes, none of the applications were
for suppression diodes. One event at Seabrook in 2003, OE 16764, concluded the failed diodes were from manufacturing defects. At the time the licensee's initial screening of this OE did not identify any specific diode issues that would likely have resulted in the corrective actions necessary to address the issue identified in Root Cause (CR) 2016-14456.
for suppression diodes. One event at Seabrook in 2003, OE 16764, concluded the
The inspectors did identify one issue at Cooper, OE 214246, concerning the failure of a blocking diodes in the transient suppression circuit of the EDG was not included
failed diodes were from manufacturing defects. At the time the licensees initial
screening of this OE did not identify any specific diode issues that would likely have
resulted in the corrective actions necessary to address the issue identified in Root
Cause (CR) 2016-14456.
The inspectors did identify one issue at Cooper, OE 214246, concerning the failure
of a blocking diodes in the transient suppression circuit of the EDG was not included
in root cause CR 2016-14456. The licensee reviewed this OE and concluded it was
in root cause CR 2016-14456. The licensee reviewed this OE and concluded it was
missed during the external OE review. The licensee wrote CR 05-02410 that
missed during the external OE review. The licensee wrote CR 05-02410 that
determined the inclusion of this OE would not have changed the conclusion in the root cause report.
determined the inclusion of this OE would not have changed the conclusion in the
d) [Describe the extent of condition and the extent of cause evaluations performed.
root cause report.
] Extent of Condition is defined in NRC Inspection Manual 95001 as the extent to which the actual condition exists with other plant processes, equipment, or human
d) [Describe the extent of condition and the extent of cause evaluations performed.]
performance. The licensee determined there were two conditions of concern. The first condition was that suppression (flyback) diodes were installed in class 1E 125VDC circuitry whose failure could go undetected for an extended period
Extent of Condition is defined in NRC Inspection Manual 95001 as the extent to
which the actual condition exists with other plant processes, equipment, or human
performance. The licensee determined there were two conditions of concern.
The first condition was that suppression (flyback) diodes were installed in
class 1E 125VDC circuitry whose failure could go undetected for an extended period
of time. To evaluate this equipment condition, the licensee identified the extent to
of time. To evaluate this equipment condition, the licensee identified the extent to
where the same component is used in the same or similar applications. In addition,
where the same component is used in the same or similar applications. In addition,
the licensee identified whether similar components existed in same or similar applications. A review of Perry stock codes was performed to see where Allen Bradley 199-FSMZ-1 flyback diodes were installed in the plant. The review identified that these diodes were only utilized on each of the Division 1 and 2 EDGs
the licensee identified whether similar components existed in same or similar
across relays R11A, R10BB, RSS, and K4. These diodes were all installed as part of the ECP 04-0049 which upgraded the speed control governing system from a
applications. A review of Perry stock codes was performed to see where Allen
Woodward EGA/EGB-35C to a Woodward2301A/EGB-35P for the Division 1 and 2 EDGs. All eight of these diodes have since been removed by the implementation of
Bradley 199-FSMZ-1 flyback diodes were installed in the plant. The review
identified that these diodes were only utilized on each of the Division 1 and 2 EDGs
across relays R11A, R10BB, RSS, and K4. These diodes were all installed as part
of the ECP 04-0049 which upgraded the speed control governing system from a
Woodward EGA/EGB-35C to a Woodward2301A/EGB-35P for the Division 1 and 2
EDGs. All eight of these diodes have since been removed by the implementation of
ECP 16-0348 in December 2016. There was no other location identified in the plant
ECP 16-0348 in December 2016. There was no other location identified in the plant
where this particular diode is used. Lab testing of the Division 1 R10BB Relay (the same relay that failed in the Division 2) after removal provided assurance that the relay surge suppressor would have performed had an EDG Division 1 Auto-Start occurred during the time it had
where this particular diode is used.
been installed. The licensee also identified systems and their respective drawings that were likely to include 1E components. It was determined that there were 13 drawings warranting
Lab testing of the Division 1 R10BB Relay (the same relay that failed in the
further review by the Responsible System Engineers to identify diodes/relays that are not tested at least quarterly. This review found there were four systems as follows:
Division 2) after removal provided assurance that the relay surge suppressor would
Reactor Recirculation System (2 total) Residual Heat Removal System (52 total) Division 3 HPCS Diesel Generator (4 total) Reactor Core Isolation Cooling System (18 total)
have performed had an EDG Division 1 Auto-Start occurred during the time it had
been installed.
The licensee also identified systems and their respective drawings that were likely to
include 1E components. It was determined that there were 13 drawings warranting
further review by the Responsible System Engineers to identify diodes/relays that are
not tested at least quarterly. This review found there were four systems as follows:
Reactor Recirculation System (2 total)
Residual Heat Removal System (52 total)
Division 3 HPCS Diesel Generator (4 total)
Reactor Core Isolation Cooling System (18 total)
These components were tested and found satisfactory through continuity tests or
These components were tested and found satisfactory through continuity tests or
performance of surveillance tests. The diodes in these systems were installed in accordance with industry and vendor design practice and not Perry engineering modifications. Their reliability has been established over the many numbers of years
performance of surveillance tests. The diodes in these systems were installed in
accordance with industry and vendor design practice and not Perry engineering
modifications. Their reliability has been established over the many numbers of years
of operating experience. The consequences of failures of any of the diodes were
of operating experience. The consequences of failures of any of the diodes were
evaluated by engineering and operations and found to be no different than a failure
evaluated by engineering and operations and found to be no different than a failure
of any part in the system and designed to be identified during the normal scheduled surveillances.
of any part in the system and designed to be identified during the normal scheduled
surveillances.
The second condition defined by the licensee was associated with Plant Processes
The second condition defined by the licensee was associated with Plant Processes
and Human Performance. Specifically, related to the NRC performance deficiency
and Human Performance. Specifically, related to the NRC performance deficiency
cited in the NOV as the failure to evaluate the effects of voltage suppression diode
cited in the NOV as the failure to evaluate the effects of voltage suppression diode
failure on the EDG control circuit. The licensee reviewed the corrective action program to identify issues that may have existed because the failure effects of components added/changed by the modification process were not properly evaluated resulting in a system/component failure. The review identified evidence that the
failure on the EDG control circuit. The licensee reviewed the corrective action
program to identify issues that may have existed because the failure effects of
components added/changed by the modification process were not properly evaluated
resulting in a system/component failure. The review identified evidence that the
failure effects of components added/changed by the modification process at Perry
failure effects of components added/changed by the modification process at Perry
were not properly evaluated and resulted in a system/component failure. Seven issues were identified in the last ten years, however none within the last three years. The licensee created CA 2016-14456-015 to perform a self-assessment of the safety related design modifications issued at Perry since 2005. This assessment will verify that proper failure modes and effects have been considered for the new
were not properly evaluated and resulted in a system/component failure. Seven
components added to MSPI systems. This corrective action will address the potential existence of latent design deficiencies that have not yet resulted in an equipment failure. This corrective action completion date is scheduled for
issues were identified in the last ten years, however none within the last three years.
The licensee created CA 2016-14456-015 to perform a self-assessment of the
safety related design modifications issued at Perry since 2005. This assessment will
verify that proper failure modes and effects have been considered for the new
components added to MSPI systems. This corrective action will address the
potential existence of latent design deficiencies that have not yet resulted in an
equipment failure. This corrective action completion date is scheduled for
March 30, 2018.
March 30, 2018.
The extent of cause (EOC) evaluation determines the extent to which the causes of
The extent of cause (EOC) evaluation determines the extent to which the causes of
an identified problem have impacted other plant processes, equipment, or human
an identified problem have impacted other plant processes, equipment, or human
performance.  
performance.
 
The Root Cause Evaluation CR 2016-14456, identified two root causes and one
The Root Cause Evaluation CR 2016-14456, identified two root causes and one contributing cause including:
contributing cause including:
Root Cause: Using diodes as a common design practice Root Cause: Manufacturing Defect
Root Cause: Using diodes as a common design practice
Contributing Cause: Discrepancies with vendor (ESI) acceptance criteria used during initial dedication testing. The inspectors only reviewed the extent of the root cause related to using diodes as a common design practice. This evaluation is similar to the extent of condition evaluation associated with Plant Processes and Human Performance discussed above. While the other root cause associated with a manufacturing defect was useful to the licensee, the inspectors determined that absent the performance
Root Cause: Manufacturing Defect
Contributing Cause: Discrepancies with vendor (ESI) acceptance criteria used
during initial dedication testing.
The inspectors only reviewed the extent of the root cause related to using diodes as
a common design practice. This evaluation is similar to the extent of condition
evaluation associated with Plant Processes and Human Performance discussed
above. While the other root cause associated with a manufacturing defect was
useful to the licensee, the inspectors determined that absent the performance
deficiency cited in the NOV, the failure of the diode in the R10BB relay due to the
deficiency cited in the NOV, the failure of the diode in the R10BB relay due to the
manufacturing defect would not have had the same consequences. To evaluate the extent of using diodes as a common design practice, the licensee reviewed safety-related design changes implemented since 2005 identify other locations where surge suppression devices were installed in the plant that may not
manufacturing defect would not have had the same consequences.
be needed and hence may have created an unnecessary failure mechanism. To identify if other types of incorrect mindset or misapplication of common design standards existed within engineering that have affected the quality of engineering
To evaluate the extent of using diodes as a common design practice, the licensee
reviewed safety-related design changes implemented since 2005 identify other
locations where surge suppression devices were installed in the plant that may not
be needed and hence may have created an unnecessary failure mechanism. To
identify if other types of incorrect mindset or misapplication of common design
standards existed within engineering that have affected the quality of engineering
changes, the Perry site performance indicators associated with engineering changes
changes, the Perry site performance indicators associated with engineering changes
were reviewed. As a result of the evaluations, the licensee did not identify any other applications where diodes were not needed. Also, an adverse trend in engineering product quality identified in the Engineering indicators was closed in the first half of 2016 based on a sustained improvement in engineering product/ECP quality. e) [Indicate whether safety culture components were appropriate considered during the root cause, extent of condition, and extent of cause evaluations performed.
were reviewed.
] Safety Culture components were adequately addressed in the root cause by evaluation of the causal factors against the NRC safety culture aspects and the categories for INPO traits of a healthy safety culture. The licensee identified no
As a result of the evaluations, the licensee did not identify any other applications
where diodes were not needed. Also, an adverse trend in engineering product
quality identified in the Engineering indicators was closed in the first half of 2016
based on a sustained improvement in engineering product/ECP quality.
e) [Indicate whether safety culture components were appropriate considered during the
root cause, extent of condition, and extent of cause evaluations performed.]
Safety Culture components were adequately addressed in the root cause by
evaluation of the causal factors against the NRC safety culture aspects and the
categories for INPO traits of a healthy safety culture. The licensee identified no
safety culture aspects as significant contributors to the event. No cross-cutting issue
safety culture aspects as significant contributors to the event. No cross-cutting issue
was assigned by the NRC in the NOV since the performance deficiency associated
was assigned by the NRC in the NOV since the performance deficiency associated
with this finding occurred more than three years ago; therefore, it did not reflect current licensee performance.
with this finding occurred more than three years ago; therefore, it did not reflect
current licensee performance.
The root cause did identify related cross cutting aspects that could be related to the
The root cause did identify related cross cutting aspects that could be related to the
design aspects of the NOV's performance deficiency. Specific aspects identified
design aspects of the NOVs performance deficiency. Specific aspects identified
were, (1) Avoid Complacency, (2) Consistent Process and, (3) Conservative Bias. Corrective actions CA-2016-14456-02 and CA-2016-14456-013 were initiated to address these safety culture aspects. The first action provided training and lessons
were, (1) Avoid Complacency, (2) Consistent Process and, (3) Conservative Bias.
Corrective actions CA-2016-14456-02 and CA-2016-14456-013 were initiated to
address these safety culture aspects. The first action provided training and lessons
learned to all engineering staff that emphasized minimizing installation of
learned to all engineering staff that emphasized minimizing installation of
components that are not needed. The second action was to discuss the lessons
components that are not needed. The second action was to discuss the lessons
learned from this issue to all supervisory staff. Both of these actions were
learned from this issue to all supervisory staff. Both of these actions were
completed.  
completed.
(3) Corrective Actions a) [Describe corrective actions taken, planned, or why unnecessary for each root and contributing cause. Specifically identify those corrective action taken or planned to
(3) Corrective Actions
prevent recurrence. Planned corrective actions to prevent recurrence will be inspected using IP 71152.
a) [Describe corrective actions taken, planned, or why unnecessary for each root and
] The Root Cause Evaluation in CR 2016-14456, identified two root causes and one contributing cause including: Root Cause: Using diodes as a common design practice Root Cause: Manufacturing Defect
contributing cause. Specifically identify those corrective action taken or planned to
Contributing Cause: Discrepancies with vendor acceptance criteria used during
prevent recurrence. Planned corrective actions to prevent recurrence will be
inspected using IP 71152.]
The Root Cause Evaluation in CR 2016-14456, identified two root causes and one
contributing cause including:
Root Cause: Using diodes as a common design practice
Root Cause: Manufacturing Defect
Contributing Cause: Discrepancies with vendor acceptance criteria used during
initial dedication testing.
initial dedication testing.
The NOV cited the failure to consider the effect of a shorted diode on the control circuitry of the Division 2 EDG, and, as a result, the failure to recognize that
The NOV cited the failure to consider the effect of a shorted diode on the control
circuitry of the Division 2 EDG, and, as a result, the failure to recognize that
installation of voltage suppression diodes across control relays, with no mitigation for
installation of voltage suppression diodes across control relays, with no mitigation for
diode failure, was not suitable for the EDG control circuit and was a violation
diode failure, was not suitable for the EDG control circuit and was a violation
of 10 CFR, Part 50, Appendix B, Criterion III, "Design Control."
of 10 CFR, Part 50, Appendix B, Criterion III, Design Control.
Corrective actions that addressed the NOV were conducted under CR 2016-13183. These actions included an evaluation by engineering that all the suppression diodes
Corrective actions that addressed the NOV were conducted under CR 2016-13183.
These actions included an evaluation by engineering that all the suppression diodes
installed in the control circuitry were not needed and therefore could be removed
installed in the control circuitry were not needed and therefore could be removed
permanently in ECP 16-0348-004. Removal of the all the suppression diodes
permanently in ECP 16-0348-004. Removal of the all the suppression diodes
was completed for the Division 1 EDG by Work Order (WO) 200701159 on December 17, 2016 and for the Division 2 EDG by WO 200701164 on December 22, 2016. These actions were completed prior to issuance of the NOV.  
was completed for the Division 1 EDG by Work Order (WO) 200701159 on
 
December 17, 2016 and for the Division 2 EDG by WO 200701164 on
December 22, 2016. These actions were completed prior to issuance of the NOV.
The actions above addressed the licensee root cause of using diodes as a common
The actions above addressed the licensee root cause of using diodes as a common
design practice. To further address this root cause the licensee initiated corrective actions that performed or will perform reviews of engineering packages since 2006 to determine current engineering practices and standards. Additional corrective actions
design practice. To further address this root cause the licensee initiated corrective
actions that performed or will perform reviews of engineering packages since 2006 to
determine current engineering practices and standards. Additional corrective actions
included training for all design engineers on this issue, ensuring this issue is part of
included training for all design engineers on this issue, ensuring this issue is part of
the mentoring program discussions for all new engineers, discussions of this issue with all plant supervisors, and finally revisions to the EDG systems training plan.
the mentoring program discussions for all new engineers, discussions of this issue
To address the other licensee root cause of a manufacturing defect, corrective actions included the removal of the diodes as discussed above. In addition, another
with all plant supervisors, and finally revisions to the EDG systems training plan.
corrective action was to complete or verify complete the 10 CFR 21 Requirements  
To address the other licensee root cause of a manufacturing defect, corrective
(internally or externally) of the defective diodes.
actions included the removal of the diodes as discussed above. In addition, another
The licensee implemented an action for the contributing cause for the discrepancies with vendor acceptance criteria used during initial dedication testing. The diode
corrective action was to complete or verify complete the 10 CFR 21 Requirements
      (internally or externally) of the defective diodes.
The licensee implemented an action for the contributing cause for the discrepancies
with vendor acceptance criteria used during initial dedication testing. The diode
supplier revised the dedication test plan to change the acceptance criteria for the
supplier revised the dedication test plan to change the acceptance criteria for the
reverse bias testing of diodes from 500 micro amps leakage current to 5 micro amps.
reverse bias testing of diodes from 500 micro amps leakage current to 5 micro amps.
b) [Describe the prioritization of each corrective action.
b) [Describe the prioritization of each corrective action.]
] Corrective actions were assigned a priority in accordance with procedure NOBP-LP-2011, Root Cause Analysis, based on safety precedence. Priorities
Corrective actions were assigned a priority in accordance with procedure
included: 1. Design Hazard; 2. Safety Devices;
NOBP-LP-2011, Root Cause Analysis, based on safety precedence. Priorities
included:
1. Design Hazard;
2. Safety Devices;
3. Safety Warnings;
3. Safety Warnings;
4. Procedures;
4. Procedures;
5. Training/Awareness; and
5. Training/Awareness; and
6. Notify Management of Risk and Accept without Action. c) [Describe the schedule established for implementing and completing each corrective
6. Notify Management of Risk and Accept without Action.
action.] The Root Cause Evaluation contains the schedule for implementing the Corrective Actions. Corrective actions to address the performance deficiency in the NOV were
c) [Describe the schedule established for implementing and completing each corrective
action.]
The Root Cause Evaluation contains the schedule for implementing the Corrective
Actions. Corrective actions to address the performance deficiency in the NOV were
completed in December 2016 when the suppression diodes were removed from the
completed in December 2016 when the suppression diodes were removed from the
control circuit relays in both the Division 1 and Division 2 EDGs. Most of the other
control circuit relays in both the Division 1 and Division 2 EDGs. Most of the other
corrective actions were completed in 2017. The only corrective action not completed
corrective actions were completed in 2017. The only corrective action not completed
by the time the 95001 inspection occurred was to perform a self-assessment of the attached list of safety related and/or augmented quality design change packages issued at the Perry Plant since 2005 discussed in 1 d) above. d) [Describe the quantitative or qualitative measures of success that have been developed for determining the effectiveness of the corrective actions to prevent
by the time the 95001 inspection occurred was to perform a self-assessment of the
recurrence.
attached list of safety related and/or augmented quality design change packages
] The root cause evaluation contains several measures for determining the effectiveness of the planned and completed corrective actions. Specifically:  
issued at the Perry Plant since 2005 discussed in 1 d) above.
 
d) [Describe the quantitative or qualitative measures of success that have been
developed for determining the effectiveness of the corrective actions to prevent
recurrence.]
The root cause evaluation contains several measures for determining the
effectiveness of the planned and completed corrective actions. Specifically:
1- Effectiveness Review (ER) 2016-14456-1, to be completed by
1- Effectiveness Review (ER) 2016-14456-1, to be completed by
January 31, 2019, is to verify that between July 1, 2017 and January 1, 2019, no new component failures have occurred in the 125 VDC control circuitry of the
January 31, 2019, is to verify that between July 1, 2017 and January 1, 2019, no
Division 1 and 2 EDGs that are attributed to the removal of the diodes. 2- Effectiveness Review (ER) 2016-14456-2, to be completed by September 1, 2019, is to perform a review of all safety related/augmented quality design change ECPs issued between July 1, 2017 and July 1, 2019 that involve electrical and/or instrumentation and control changes. For each ECP, the plant
new component failures have occurred in the 125 VDC control circuitry of the
Division 1 and 2 EDGs that are attributed to the removal of the diodes.
2- Effectiveness Review (ER) 2016-14456-2, to be completed by
September 1, 2019, is to perform a review of all safety related/augmented quality
design change ECPs issued between July 1, 2017 and July 1, 2019 that involve
electrical and/or instrumentation and control changes. For each ECP, the plant
will verify that suppression diodes were properly sized, a failure modes and
will verify that suppression diodes were properly sized, a failure modes and
effects analysis or equivalent was completed, and it did not add unnecessary
effects analysis or equivalent was completed, and it did not add unnecessary
components. New condition reports will be generated if these criteria were not
components. New condition reports will be generated if these criteria were not
met. e) [Indicate which corrective actions taken or planned adequately address the Notice of
met.
Violation (NOV) as applicable.
e) [Indicate which corrective actions taken or planned adequately address the Notice of
] The NOV cited the failure to consider the effect of a shorted diode on the control circuitry of the Division 2 EDG, and, as a result, the failure to recognize that installation of voltage suppression diodes across control relays, with no mitigation for
Violation (NOV) as applicable.]
The NOV cited the failure to consider the effect of a shorted diode on the control
circuitry of the Division 2 EDG, and, as a result, the failure to recognize that
installation of voltage suppression diodes across control relays, with no mitigation for
diode failure, was not suitable for the EDG control circuit and was a violation
diode failure, was not suitable for the EDG control circuit and was a violation
of 10 CFR, Part 50, Appendix B, Criterion III, "Design Control."
of 10 CFR, Part 50, Appendix B, Criterion III, Design Control.
Corrective actions that addressed the NOV were conducted under CR-2016-13183. These actions included an evaluation by engineering that all the suppression diodes
Corrective actions that addressed the NOV were conducted under CR-2016-13183.
These actions included an evaluation by engineering that all the suppression diodes
installed in the control circuitry were not needed and could be removed permanently
installed in the control circuitry were not needed and could be removed permanently
in ECP 16-0348-004. Removal of the all the suppression diodes was completed
in ECP 16-0348-004. Removal of the all the suppression diodes was completed
for the Division 1 EDG by Work Order (WO) 200701159 completed on
for the Division 1 EDG by Work Order (WO) 200701159 completed on
December 17, 2016 and for the Division 2 EDG by WO 200701164 completed on December 22, 2016. These actions were completed prior to issuance of the NO
December 17, 2016 and for the Division 2 EDG by WO 200701164 completed on
: [[contact::V. (4) Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues. The licensee did not request credit for self-identification of an old design issue; therefore]], the risk-
December 22, 2016. These actions were completed prior to issuance of the NOV.
    (4) Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues. The licensee did
not request credit for self-identification of an old design issue; therefore, the risk-
significant issue was not evaluated against the IMC 0305 criteria for treatment of an old
significant issue was not evaluated against the IMC 0305 criteria for treatment of an old
design issue.
design issue.
INSPECTION RESULTS
INSPECTION RESULTS
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
Observation IP 95001 Assessment of licensee's evaluation and corrective actions.
Observation                                         IP 95001
Assessment of licensees evaluation and corrective actions.
The NRC determined that completed or planned corrective actions were sufficient to address
The NRC determined that completed or planned corrective actions were sufficient to address
the performance issue that led to the White finding previously described and were prioritized commensurate with the safety significance of the issue. In addition, the NRC determined that the root cause evaluation was conducted to a level of detail commensurate with the
the performance issue that led to the White finding previously described and were prioritized
commensurate with the safety significance of the issue. In addition, the NRC determined that
the root cause evaluation was conducted to a level of detail commensurate with the
significance of the problem and reached reasonable conclusions as to the root and
significance of the problem and reached reasonable conclusions as to the root and
contributing causes of the event.
contributing causes of the event.
The inspectors did observe that some of the corrective actions completed prior to the issuance of the NOV were not revised or changed to ensure they addressed the performance
The inspectors did observe that some of the corrective actions completed prior to the
issuance of the NOV were not revised or changed to ensure they addressed the performance
deficiency cited in the NOV. An example included the action to provide lessons learned to the
deficiency cited in the NOV. An example included the action to provide lessons learned to the
engineering department on the results of this root cause investigation. The inspectors found
engineering department on the results of this root cause investigation. The inspectors found
the focus to be on the causal factors associated with the development of ECP 04-0049.
the focus to be on the causal factors associated with the development of ECP 04-0049.
Specifically: 1. Need to perform testing during design development to validate proper sizing of diodes 2. Minimizing the installation of components that are not required for the modification to minimize probability of equipment failures. The inspectors noted this training had minimal
Specifically: 1. Need to perform testing during design development to validate proper sizing of
diodes 2. Minimizing the installation of components that are not required for the modification
to minimize probability of equipment failures. The inspectors noted this training had minimal
discussion on the failure to consider the effects of a shorted diode and was not a focus of the
discussion on the failure to consider the effects of a shorted diode and was not a focus of the
training. The inspectors considered this action would have better addressed the performance deficiency of the NOV if it would have been revised with this emphasis.
training. The inspectors considered this action would have better addressed the performance
After reviewing Perry Nuclear Power Plant's performance in addressing the White finding
deficiency of the NOV if it would have been revised with this emphasis.
subject of this Inspection Procedure 95001, "Supplemental Inspection Response To Action
After reviewing Perry Nuclear Power Plants performance in addressing the White finding
Matrix Column 2 Inputs," the NRC concluded actions taken by Perry Nuclear Power Plant met
subject of this Inspection Procedure 95001, Supplemental Inspection Response To Action
Matrix Column 2 Inputs, the NRC concluded actions taken by Perry Nuclear Power Plant met
the objectives of the inspection. Therefore, violation 05000440/2017009-01 is closed.
the objectives of the inspection. Therefore, violation 05000440/2017009-01 is closed.
 
No findings or violations were identified.
No findings or violations were identified.  
 
EXIT MEETINGS AND DEBRIEFS
EXIT MEETINGS AND DEBRIEFS
The inspectors confirmed that proprietary information was controlled to protect from public disclosure. No proprietary information was documented in this report.  
The inspectors confirmed that proprietary information was controlled to protect from public
* On February 15, 2018, the inspectors presented the Supplemental Procedure 95001 inspection results to Mr. Frank Payne, and other members of the licensee staff.  
disclosure. No proprietary information was documented in this report.
* On February 15, 2018, the NRC discussed the performance of the Perry Nuclear Power Plant in accordance with IMC 0305, Section 10.01. during a regulatory performance meeting. The meeting was attended by the Mr. Billy Dickson, Chief, Branch 5 and Mr. Frank
*   On February 15, 2018, the inspectors presented the Supplemental Procedure 95001
Payne, Plant General Manager, and other senior licensee staff. The NRC and licensee discussed the issues related to the WHITE finding resulted in Perry Nuclear Power Plant remaining in the Regulatory Response Column of the Action Matrix.
inspection results to Mr. Frank Payne, and other members of the licensee staff.
*   On February 15, 2018, the NRC discussed the performance of the Perry Nuclear Power
Plant in accordance with IMC 0305, Section 10.01. during a regulatory performance
meeting. The meeting was attended by the Mr. Billy Dickson, Chief, Branch 5 and Mr. Frank
Payne, Plant General Manager, and other senior licensee staff. The NRC and licensee
discussed the issues related to the WHITE finding resulted in Perry Nuclear Power Plant
remaining in the Regulatory Response Column of the Action Matrix.
DOCUMENTS REVIEWED
DOCUMENTS REVIEWED
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs
Condition Reports - CR-2016-13183; During performance of PTI-R43-P0006B CB-1, CB-2, CB-3, CB-4 DC Breakers Trip when loaded (superseded by root cause evaluation CR-2016-14456) - CR-2016-14251; Additional vulnerabilities identified in surge suppression diode on the DIV 1 and DIV 2 Diesel Generators - CR-2016-14456; PRA Evaluation Associated with Div 2 EDG MSPI Failure (CR-2016-13183) Indicates a Preliminary Level of Significance that Warrants a Root Cause Evaluation - CR-2017-05709; 2017 Pre-NRC PI&R Assessment does not meet Smart Criteria  
Condition Reports
- CR-2017-06396; NRC Apparent Violation- Unsuitable Application of Surge Suppression Diodes in Standby Diesel Generator Control Power Circuitry - CR-2018-00268; Evaluation of violation 05000400/2017009-01 "Unsuitable Application of Surge Suppression Diodes in Standby Diesel Generator Control Power Circuitry"
- CR-2016-13183; During performance of PTI-R43-P0006B CB-1, CB-2, CB-3, CB-4 DC
Condition Reports Generated During Inspection - CR-2018-01316; NRC 95001 Inspection; OE Review for CR-2016-14456 - CR-2018-01335; NRC 95001 Inspection; Training Needs Analysis for Root
Breakers Trip when loaded (superseded by root cause evaluation CR-2016-14456)
Cause 2016-14456  
- CR-2016-14251; Additional vulnerabilities identified in surge suppression diode on the DIV 1
 
and DIV 2 Diesel Generators
Procedures - NOBP-LP-2011; Root Cause Analysis; Revision 22  
- CR-2016-14456; PRA Evaluation Associated with Div 2 EDG MSPI Failure (CR-2016-13183)
Indicates a Preliminary Level of Significance that Warrants a Root Cause Evaluation
- CR-2017-05709; 2017 Pre-NRC PI&R Assessment does not meet Smart Criteria
- CR-2017-06396; NRC Apparent Violation- Unsuitable Application of Surge Suppression
Diodes in Standby Diesel Generator Control Power Circuitry
- CR-2018-00268; Evaluation of violation 05000400/2017009-01 "Unsuitable Application of
Surge Suppression Diodes in Standby Diesel Generator Control Power Circuitry"
Condition Reports Generated During Inspection
- CR-2018-01316; NRC 95001 Inspection; OE Review for CR-2016-14456
- CR-2018-01335; NRC 95001 Inspection; Training Needs Analysis for Root
Cause 2016-14456
Procedures
- NOBP-LP-2011; Root Cause Analysis; Revision 22
- NOBP-ER-3017; Selection and Devlopment of Engineering Mentors; Revision 00
- NOBP-ER-3017; Selection and Devlopment of Engineering Mentors; Revision 00
Others - ECP 16-0348-004; Modify Div. 2 Diesel Generator control circuitry for relay flyback diode failure; dated December 14, 2016 - Equipment Apparent Cause Evaluation Template (EACE) Diode Failure on Relay R10BB; December 2, 2016 - Failure Analysis Report CF16-060; BETA Laboratory Services Section; Diode, Surge Suppressor; November 11, 2016  
Others
- Design Basis Assessment Report- Second Half 2017 - E-Mail from  
- ECP 16-0348-004; Modify Div. 2 Diesel Generator control circuitry for relay flyback diode
: [[contact::M. Koberling to JPR Mentors and Supervisors; November 30]], 2017  
failure; dated December 14, 2016
- Job Performance Requirement, JPR 1.1; Engineering Change, Preparation, Approval, Turnover and Closeout, July 17, 2017
- Equipment Apparent Cause Evaluation Template (EACE) Diode Failure on Relay R10BB;
December 2, 2016
- Failure Analysis Report CF16-060; BETA Laboratory Services Section; Diode, Surge
Suppressor; November 11, 2016
- Design Basis Assessment Report- Second Half 2017
- E-Mail from  
: [[contact::M. Koberling to JPR Mentors and Supervisors; November 30]], 2017
- Job Performance Requirement, JPR 1.1; Engineering Change, Preparation, Approval,
Turnover and Closeout, July 17, 2017
14
}}
}}

Latest revision as of 18:03, 2 November 2019

NRC Supplemental Inspection Report 05000440/2018040 and Assessment Follow-Up Letter
ML18086B212
Person / Time
Site: Perry FirstEnergy icon.png
Issue date: 03/27/2018
From: Billy Dickson
NRC/RGN-III/DRP/B5
To: Hamilton D
FirstEnergy Nuclear Operating Co
References
EA-17-043 IR 2018040
Download: ML18086B212 (17)


Text

UNITED STATES NUCLEAR REGULATORY COMMISSION rch 27, 2018

SUBJECT:

PERRY NUCLEAR POWER PLANTNRC SUPPLEMENTAL INSPECTION REPORT 05000440/2018040 AND ASSESSMENT FOLLOW-UP LETTER

Dear Mr. Hamilton:

The U.S. Nuclear Regulatory Commission (NRC) completed a supplemental inspection using Inspection Procedure 95001, Supplemental Inspection Response To Action Matrix Column 2 Inputs, at your Perry Nuclear Power Plant. On February 15, 2018, the NRC discussed the results of this inspection and the implementation of your corrective actions with Mr. Frank Payne and other members of your staff in an exit Meeting. On February 16, 2018, the NRC discussed the causes and your corrective actions identified as a result of the WHITE finding, and the criteria required for returning to the Licensee Response Column of the Action Matrix with Mr. Frank Payne in a Regulatory Performance Meeting. The results of this inspection are documented in the enclosed report.

By letter dated August 24, 2017, (ADAMS ML17236A187), the U.S. Nuclear Regulatory Commission (NRC) informed you that as a result of a White (low-to-moderate safety significance) finding in the Mitigating Systems Cornerstone that the NRC assessed Perry Nuclear Power Plant performance to be in the Regulatory Response Column of the Reactor Oversight Process (ROP) Action Matrix, effective the second quarter of 2017. In the same letter, the NRC informed you of our intent to perform a supplemental inspection using Inspection Procedure 95001 upon notification of your readiness for the inspection. By letter dated October 31, 2017, we were notified by your staff that you were ready for the NRC to conduct our supplemental inspection to review the actions taken to address the White finding.

The NRC performed this inspection to review your stations actions in response to a White finding in the Mitigating System cornerstone which was documented in NRC Inspection Report 05000440/2017009 (ML17156A750). The finding involved the failure to evaluate the effects of voltage suppression diode failure on the Emergency Diesel Generator (EDG) control circuit, which was a component subject to the requirements of Title 10 of the Code of Federal Regulations (CFR) Part 50, Appendix B. Specifically, FirstEnergy failed to consider the effect of a shorted diode on the control circuitry of the EDG, and, as a result, failed to recognize that installation of voltage suppression diodes across control relays, with no mitigation for diode failure, was not suitable for the EDG control circuit. This introduction of new components (diodes) into the control circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering the EDG inoperable and unable to start.

The NRC determined that the completed or planned corrective actions were sufficient to address the performance issue that led to the White finding previously described and were prioritized commensurate with the safety significance of the issue. In addition, the NRC determined that the root cause evaluation was conducted to a level of detail commensurate with the significance of the problem and reached reasonable conclusions as to the root and contributing causes of the event.

No finding or violations were documented.

After reviewing Perry Nuclear Power Plants performance in addressing the White finding subject of this Inspection Procedure 95001, the NRC concluded your actions met the objectives of the inspection. Therefore, in accordance with the guidance in Inspection Manual Chapter (IMC) 0305, Operating Reactor Assessment Program, the White finding will only be considered in assessing plant performance for a total of four quarters. As a result, the NRC determined the performance at Perry Nuclear Power Plant to be in the Licensee Response Column of the ROP Action Matrix as of April 1, 2018.

This letter, its enclosure, and your response (if any) will be made available for public inspection and copying at http://www.nrc.gov/reading-rm/adams.html and at the NRC Public Document Room in accordance with 10 CFR 2.390, Public Inspections, Exemptions, Requests for Withholding.

Sincerely,

/RA/

Billy Dickson, Chief Branch 5 Division of Reactor Projects Docket No. 50-440 License No. NPF-58 Enclosure:

Inspection Report 05000440/2018040 cc: Distribution via LISTSERV

SUMMARY

The U.S. Nuclear Regulatory Commission (NRC) continued monitoring licensees performance by conducting a supplemental inspection at Perry Nuclear Power Plant in accordance with the Reactor Oversight Process. The Reactor Oversight Process is the NRCs program for overseeing the safe operation of commercial nuclear power reactors. Refer to https://www.nrc.gov/reactors/operating/oversight.html for more information. NRC and self-revealed findings, violations, and additional items are summarized in the table below.

List of Findings and Violations No findings or violations were identified.

Additional Tracking Items Type Issue Number Title Report Status Section NOV 05000440/2017009-01 Division 2 Diesel Generator 95001 Closed Failure to Start due to a Failed Diode in the 125 VDC Control Power Circuit

TABLE OF CONTENTS

INSPECTION SCOPES

OTHER ACTIVITIES

TEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL

INSPECTION RESULTS

............................................................................................................

EXIT MEETINGS AND DEBRIEFS

............................................................................................ 13

DOCUMENTS REVIEWED

......................................................................................................... 13

INSPECTION SCOPES

Inspections were conducted using the appropriate portions of the inspection procedure (IPs) in

effect at the beginning of the inspection unless otherwise noted. Currently approved IPs with

their attached revision histories are located on the public website at http://www.nrc.gov/reading-

rm/doc-collections/insp-manual/inspection-procedure/index.html. Samples were declared

complete when the IP requirements most appropriate to the inspection activity were met

consistent with Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection

Program - Operations Phase. The inspectors reviewed selected procedures and records,

observed activities, and interviewed personnel to assess licensee performance and compliance

with Commission rules and regulations, license conditions, site procedures, and standards.

OTHER ACTIVITIESTEMPORARY INSTRUCTIONS, INFREQUENT AND ABNORMAL

95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs

Perry Nuclear Power Plant entered the Regulatory Response column of NRCs Action Matrix

in the 2nd quarter of 2017 due to a White (low-to-moderate safety significance) inspection

finding,05000440/2017009-01, in the Mitigating Systems Cornerstone. The finding was

associated with the stations failure to evaluate the effects of voltage suppression diode failure

on the Emergency Diesel Generator (EDG) control circuit. The finding was characterized as

having White safety significance based on the results of a risk assessment performed by a

region-based Senior Reactor Analyst. This issue was documented in NRC Inspection

Reports (IRs) 05000440/2017009 and 05000440/2017010.

By letter dated November 3, 2017, the licensee notified the NRC that it had completed its

evaluation of the circumstances surrounding the degraded performance and were ready for the

NRC to assess the evaluation and subsequent corrective actions. In preparation for the

inspection, the licensee performed Root Cause Evaluation (RCE), PRA Evaluation Associated

with the Div. 2 EDG MSPI Failure (CR 2016-14456) Indicates a Preliminary Level of

Significance that Warrants a Root Cause Evaluation, Revision 2, dated October 9, 2017. It

identified weaknesses that existed in various organizations which allowed for a risk-significant

finding, and to determine the organizational attributes that resulted in the White finding.

The affected diesel generators are identified as the Standby Diesel Generator and Emergency

Diesel Generator in this reports reference documents and are used interchangeably. This

report uses Emergency Diesel Generator for consistency.

The inspectors reviewed the licensees root causes, contributing causes, extent of condition,

and extent of cause determinations. The inspectors assessed whether the licensees corrective

actions to address the root and contributing causes were sufficient to prevent recurrence. The

highlights of the performance review and NRCs assessment are documented below.

Evaluation of Inspection Requirements

(1) Problem Identification

a) [Describe who identified the issue and under what conditions the issue was

identified.]

Root cause CR 2016-14456 evaluation concluded that the diode failure (shorted) for

relay R10BB, 1R43Q0003B, which resulted in loss of the 125 Volts, Direct

Current, (VDC) control power to the Division 2 EDG, was self-revealed on

November 6, 2016, while performing surveillance test PTI-R43-P0006B, Division 2

Diesel Generator Pneumatic Logic Board Functional Check. Operators received

various unexpected alarms when a jumper was installed to simulate an

emergency (auto) start as part of the surveillance test.

b) [Describe how long the issue existed and prior opportunities for identification.]

Root cause CR 2016-14456 evaluation concluded this issue existed when the

suppression diodes were installed in the control circuitry of the Division 2 EDG on

April 21, 2007.

The licensee identified that an opportunity was missed after the diode on relay R11

failed on May 6, 2016. On April 21, 2007, the licensee initiated and discussed in

CR-2016-06450 that the new Allen Brandley relays R11A, R10BB, RSS, K4, and

associated diodes, were installed on the Division 2 EDG by ECP 04-0049. This

relay was energized on May 06, 2016 during a monthly Surveillance test. The diode

in parallel with the relay R11A failed causing alarms, abnormal indications, and

required operator action to shutdown the DIV2 EDG. The diode was replaced based

on CR-2016-06450 on May 06, 2016. The licensee did not then consider the

effects of a shorted diode, for other diodes in the system and other EDGs.

c) [Describe the plant-specific risk consequences, as applicable, and compliance

concerns associated with the issue.]

The licensee did not document a Probabilistic Risk Assessment (PRA) evaluation for

this condition. The Licensee discussed the NRC risk assessment that concluded the

safety significance for this issue as White (low-to-moderate) based on the

Division 2 EDG not operable and unable to perform its emergency start function from

April 2, 2015, when it passes a surveillance test, until November 8, 2016 when the

emergency start diode was replaced after failure and the Division 2 EDG was

returned to service. Because the licensee was not aware of the Div 2 EDGs

inoperability during the units operation cycle, the required actions of TS 3.8.1.F.1

and 3.8.1.F.2 were not followed. The evaluation also recognized this determination

resulted in being placed in the Regulatory Response Column of the NRCs Reactor

Oversight Process (ROP) Action Matrix.

(2) Root Cause, Extent-of-Condition, and Extent-of-Cause Evaluation

a) [Describe systematic methodology used to identify the root and contributing causes.]

The licensee assigned a multi-disciplined team to perform a Root Cause Evaluation

to identify the root and contributing causes of the EDG inoperability and the inability

to start. In its root cause analysis, the licensee used the following processes to

identify the root and contributing causes:

  • Data gathering through interviews and document review;
  • Event and Causal Factor Charting;
  • Internal and External Laboratory Analysis;
  • Equipment Apparent Cause Evaluation (EACE);
  • Latent Organization Weakness Analysis;
  • Performance Improvement International Failure Modes Analysis; and
  • MORT Analysis.

The licensee used the above processes to evaluate equipment and human

performance issues associated with the event. The processes used were systematic

processes governed by and implemented in accordance with the licensees

procedures. The processes were used to ensure the information and circumstances

surrounding the problem were understood. The results of these processes revealed

two root causes and one contributing cause. The initial root cause evaluation

attributed the failure to a manufacturing defect in the diode with a contributing cause

related to engineering design practices. The Notice of Violation (NOV) was issued in

August of 2017 for failing to recognize that installation of voltage suppression diodes

across control relays, with no mitigation for diode failure, was not suitable for the

EDG control circuit. This introduction of new components (diodes) into the control

circuitry resulted in the eventual failure of the EDG control circuit, thereby rendering

the EDG inoperable and unable to start. After the NOV was issued, Revision 2 of the

Root Cause Evaluation identified using diodes as a common design practice as a

second root cause. This aligned closer with the NOV. The licensee also determined

that discrepancies with vendor acceptance criteria used during initial dedication was

a contributing cause.

b) [Describe level of detail of detail used in the root cause evaluation and indicate

whether it was commensurate with the significance of the problem.]

In its root cause analysis, the licensee classified the event as a significant condition

adverse to quality and applied thorough processes and methodologies to identify the

root and contributing causes associated with the licensees failure to evaluate the

effects of voltage suppression diode failure on the EDG control circuit. The analysis

performed in the root cause were of sufficient detail commensurate with the

significance of the problem. The licensees root cause evaluation included a timeline

of events. The licensee conducted interviews, and performed work history and

corrective action document reviews to capture the issues leading up to the event.

The root cause evaluation resulted in corrective actions that promptly addressed the

design issues with the use of the diodes in the EDG control circuitry that would

prevent recurrence. The engineering deficiency that was the basis of the NOV

occurred in 2005 and the root cause evaluation addressed engineering issues and

performance since then to characterize current behavior and develop corrective

actions.

c) [Describe prior occurrences and operating experience considered during the root

cause evaluation.]

The licensee included an evaluation of internal and external Operating

Experience (OE) for issues with issues related to the loss of control power resulting

from failed, shorted diodes. The licensee found that failures of diodes used to

suppress the inductive kickback of DC coils are not common at Perry nor within

the FENOC fleet.

Review of the external operating experience from INPO and NRC Websites was also

conducted that revealed seven issues related to diode failures. Three of these

occurred prior to completion of the engineering change package in 2005 that

subsequently installed the suppression diodes in the EDG control circuit. While

several of these events included failed, shorted diodes, none of the applications were

for suppression diodes. One event at Seabrook in 2003, OE 16764, concluded the

failed diodes were from manufacturing defects. At the time the licensees initial

screening of this OE did not identify any specific diode issues that would likely have

resulted in the corrective actions necessary to address the issue identified in Root

Cause (CR) 2016-14456.

The inspectors did identify one issue at Cooper, OE 214246, concerning the failure

of a blocking diodes in the transient suppression circuit of the EDG was not included

in root cause CR 2016-14456. The licensee reviewed this OE and concluded it was

missed during the external OE review. The licensee wrote CR 05-02410 that

determined the inclusion of this OE would not have changed the conclusion in the

root cause report.

d) [Describe the extent of condition and the extent of cause evaluations performed.]

Extent of Condition is defined in NRC Inspection Manual 95001 as the extent to

which the actual condition exists with other plant processes, equipment, or human

performance. The licensee determined there were two conditions of concern.

The first condition was that suppression (flyback) diodes were installed in

class 1E 125VDC circuitry whose failure could go undetected for an extended period

of time. To evaluate this equipment condition, the licensee identified the extent to

where the same component is used in the same or similar applications. In addition,

the licensee identified whether similar components existed in same or similar

applications. A review of Perry stock codes was performed to see where Allen

Bradley 199-FSMZ-1 flyback diodes were installed in the plant. The review

identified that these diodes were only utilized on each of the Division 1 and 2 EDGs

across relays R11A, R10BB, RSS, and K4. These diodes were all installed as part

of the ECP 04-0049 which upgraded the speed control governing system from a

Woodward EGA/EGB-35C to a Woodward2301A/EGB-35P for the Division 1 and 2

EDGs. All eight of these diodes have since been removed by the implementation of

ECP 16-0348 in December 2016. There was no other location identified in the plant

where this particular diode is used.

Lab testing of the Division 1 R10BB Relay (the same relay that failed in the

Division 2) after removal provided assurance that the relay surge suppressor would

have performed had an EDG Division 1 Auto-Start occurred during the time it had

been installed.

The licensee also identified systems and their respective drawings that were likely to

include 1E components. It was determined that there were 13 drawings warranting

further review by the Responsible System Engineers to identify diodes/relays that are

not tested at least quarterly. This review found there were four systems as follows:

Reactor Recirculation System (2 total)

Residual Heat Removal System (52 total)

Division 3 HPCS Diesel Generator (4 total)

Reactor Core Isolation Cooling System (18 total)

These components were tested and found satisfactory through continuity tests or

performance of surveillance tests. The diodes in these systems were installed in

accordance with industry and vendor design practice and not Perry engineering

modifications. Their reliability has been established over the many numbers of years

of operating experience. The consequences of failures of any of the diodes were

evaluated by engineering and operations and found to be no different than a failure

of any part in the system and designed to be identified during the normal scheduled

surveillances.

The second condition defined by the licensee was associated with Plant Processes

and Human Performance. Specifically, related to the NRC performance deficiency

cited in the NOV as the failure to evaluate the effects of voltage suppression diode

failure on the EDG control circuit. The licensee reviewed the corrective action

program to identify issues that may have existed because the failure effects of

components added/changed by the modification process were not properly evaluated

resulting in a system/component failure. The review identified evidence that the

failure effects of components added/changed by the modification process at Perry

were not properly evaluated and resulted in a system/component failure. Seven

issues were identified in the last ten years, however none within the last three years.

The licensee created CA 2016-14456-015 to perform a self-assessment of the

safety related design modifications issued at Perry since 2005. This assessment will

verify that proper failure modes and effects have been considered for the new

components added to MSPI systems. This corrective action will address the

potential existence of latent design deficiencies that have not yet resulted in an

equipment failure. This corrective action completion date is scheduled for

March 30, 2018.

The extent of cause (EOC) evaluation determines the extent to which the causes of

an identified problem have impacted other plant processes, equipment, or human

performance.

The Root Cause Evaluation CR 2016-14456, identified two root causes and one

contributing cause including:

Root Cause: Using diodes as a common design practice

Root Cause: Manufacturing Defect

Contributing Cause: Discrepancies with vendor (ESI) acceptance criteria used

during initial dedication testing.

The inspectors only reviewed the extent of the root cause related to using diodes as

a common design practice. This evaluation is similar to the extent of condition

evaluation associated with Plant Processes and Human Performance discussed

above. While the other root cause associated with a manufacturing defect was

useful to the licensee, the inspectors determined that absent the performance

deficiency cited in the NOV, the failure of the diode in the R10BB relay due to the

manufacturing defect would not have had the same consequences.

To evaluate the extent of using diodes as a common design practice, the licensee

reviewed safety-related design changes implemented since 2005 identify other

locations where surge suppression devices were installed in the plant that may not

be needed and hence may have created an unnecessary failure mechanism. To

identify if other types of incorrect mindset or misapplication of common design

standards existed within engineering that have affected the quality of engineering

changes, the Perry site performance indicators associated with engineering changes

were reviewed.

As a result of the evaluations, the licensee did not identify any other applications

where diodes were not needed. Also, an adverse trend in engineering product

quality identified in the Engineering indicators was closed in the first half of 2016

based on a sustained improvement in engineering product/ECP quality.

e) [Indicate whether safety culture components were appropriate considered during the

root cause, extent of condition, and extent of cause evaluations performed.]

Safety Culture components were adequately addressed in the root cause by

evaluation of the causal factors against the NRC safety culture aspects and the

categories for INPO traits of a healthy safety culture. The licensee identified no

safety culture aspects as significant contributors to the event. No cross-cutting issue

was assigned by the NRC in the NOV since the performance deficiency associated

with this finding occurred more than three years ago; therefore, it did not reflect

current licensee performance.

The root cause did identify related cross cutting aspects that could be related to the

design aspects of the NOVs performance deficiency. Specific aspects identified

were, (1) Avoid Complacency, (2) Consistent Process and, (3) Conservative Bias.

Corrective actions CA-2016-14456-02 and CA-2016-14456-013 were initiated to

address these safety culture aspects. The first action provided training and lessons

learned to all engineering staff that emphasized minimizing installation of

components that are not needed. The second action was to discuss the lessons

learned from this issue to all supervisory staff. Both of these actions were

completed.

(3) Corrective Actions

a) [Describe corrective actions taken, planned, or why unnecessary for each root and

contributing cause. Specifically identify those corrective action taken or planned to

prevent recurrence. Planned corrective actions to prevent recurrence will be

inspected using IP 71152.]

The Root Cause Evaluation in CR 2016-14456, identified two root causes and one

contributing cause including:

Root Cause: Using diodes as a common design practice

Root Cause: Manufacturing Defect

Contributing Cause: Discrepancies with vendor acceptance criteria used during

initial dedication testing.

The NOV cited the failure to consider the effect of a shorted diode on the control

circuitry of the Division 2 EDG, and, as a result, the failure to recognize that

installation of voltage suppression diodes across control relays, with no mitigation for

diode failure, was not suitable for the EDG control circuit and was a violation

of 10 CFR, Part 50, Appendix B, Criterion III, Design Control.

Corrective actions that addressed the NOV were conducted under CR 2016-13183.

These actions included an evaluation by engineering that all the suppression diodes

installed in the control circuitry were not needed and therefore could be removed

permanently in ECP 16-0348-004. Removal of the all the suppression diodes

was completed for the Division 1 EDG by Work Order (WO) 200701159 on

December 17, 2016 and for the Division 2 EDG by WO 200701164 on

December 22, 2016. These actions were completed prior to issuance of the NOV.

The actions above addressed the licensee root cause of using diodes as a common

design practice. To further address this root cause the licensee initiated corrective

actions that performed or will perform reviews of engineering packages since 2006 to

determine current engineering practices and standards. Additional corrective actions

included training for all design engineers on this issue, ensuring this issue is part of

the mentoring program discussions for all new engineers, discussions of this issue

with all plant supervisors, and finally revisions to the EDG systems training plan.

To address the other licensee root cause of a manufacturing defect, corrective

actions included the removal of the diodes as discussed above. In addition, another

corrective action was to complete or verify complete the 10 CFR 21 Requirements

(internally or externally) of the defective diodes.

The licensee implemented an action for the contributing cause for the discrepancies

with vendor acceptance criteria used during initial dedication testing. The diode

supplier revised the dedication test plan to change the acceptance criteria for the

reverse bias testing of diodes from 500 micro amps leakage current to 5 micro amps.

b) [Describe the prioritization of each corrective action.]

Corrective actions were assigned a priority in accordance with procedure

NOBP-LP-2011, Root Cause Analysis, based on safety precedence. Priorities

included:

1. Design Hazard;

2. Safety Devices;

3. Safety Warnings;

4. Procedures;

5. Training/Awareness; and

6. Notify Management of Risk and Accept without Action.

c) [Describe the schedule established for implementing and completing each corrective

action.]

The Root Cause Evaluation contains the schedule for implementing the Corrective

Actions. Corrective actions to address the performance deficiency in the NOV were

completed in December 2016 when the suppression diodes were removed from the

control circuit relays in both the Division 1 and Division 2 EDGs. Most of the other

corrective actions were completed in 2017. The only corrective action not completed

by the time the 95001 inspection occurred was to perform a self-assessment of the

attached list of safety related and/or augmented quality design change packages

issued at the Perry Plant since 2005 discussed in 1 d) above.

d) [Describe the quantitative or qualitative measures of success that have been

developed for determining the effectiveness of the corrective actions to prevent

recurrence.]

The root cause evaluation contains several measures for determining the

effectiveness of the planned and completed corrective actions. Specifically:

1- Effectiveness Review (ER) 2016-14456-1, to be completed by

January 31, 2019, is to verify that between July 1, 2017 and January 1, 2019, no

new component failures have occurred in the 125 VDC control circuitry of the

Division 1 and 2 EDGs that are attributed to the removal of the diodes.

2- Effectiveness Review (ER) 2016-14456-2, to be completed by

September 1, 2019, is to perform a review of all safety related/augmented quality

design change ECPs issued between July 1, 2017 and July 1, 2019 that involve

electrical and/or instrumentation and control changes. For each ECP, the plant

will verify that suppression diodes were properly sized, a failure modes and

effects analysis or equivalent was completed, and it did not add unnecessary

components. New condition reports will be generated if these criteria were not

met.

e) [Indicate which corrective actions taken or planned adequately address the Notice of

Violation (NOV) as applicable.]

The NOV cited the failure to consider the effect of a shorted diode on the control

circuitry of the Division 2 EDG, and, as a result, the failure to recognize that

installation of voltage suppression diodes across control relays, with no mitigation for

diode failure, was not suitable for the EDG control circuit and was a violation

of 10 CFR, Part 50, Appendix B, Criterion III, Design Control.

Corrective actions that addressed the NOV were conducted under CR-2016-13183.

These actions included an evaluation by engineering that all the suppression diodes

installed in the control circuitry were not needed and could be removed permanently

in ECP 16-0348-004. Removal of the all the suppression diodes was completed

for the Division 1 EDG by Work Order (WO) 200701159 completed on

December 17, 2016 and for the Division 2 EDG by WO 200701164 completed on

December 22, 2016. These actions were completed prior to issuance of the NOV.

(4) Evaluation of IMC 0305 Criteria for Treatment of Old Design Issues. The licensee did

not request credit for self-identification of an old design issue; therefore, the risk-

significant issue was not evaluated against the IMC 0305 criteria for treatment of an old

design issue.

INSPECTION RESULTS

95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs

Observation IP 95001

Assessment of licensees evaluation and corrective actions.

The NRC determined that completed or planned corrective actions were sufficient to address

the performance issue that led to the White finding previously described and were prioritized

commensurate with the safety significance of the issue. In addition, the NRC determined that

the root cause evaluation was conducted to a level of detail commensurate with the

significance of the problem and reached reasonable conclusions as to the root and

contributing causes of the event.

The inspectors did observe that some of the corrective actions completed prior to the

issuance of the NOV were not revised or changed to ensure they addressed the performance

deficiency cited in the NOV. An example included the action to provide lessons learned to the

engineering department on the results of this root cause investigation. The inspectors found

the focus to be on the causal factors associated with the development of ECP 04-0049.

Specifically: 1. Need to perform testing during design development to validate proper sizing of

diodes 2. Minimizing the installation of components that are not required for the modification

to minimize probability of equipment failures. The inspectors noted this training had minimal

discussion on the failure to consider the effects of a shorted diode and was not a focus of the

training. The inspectors considered this action would have better addressed the performance

deficiency of the NOV if it would have been revised with this emphasis.

After reviewing Perry Nuclear Power Plants performance in addressing the White finding

subject of this Inspection Procedure 95001, Supplemental Inspection Response To Action

Matrix Column 2 Inputs, the NRC concluded actions taken by Perry Nuclear Power Plant met

the objectives of the inspection. Therefore, violation 05000440/2017009-01 is closed.

No findings or violations were identified.

EXIT MEETINGS AND DEBRIEFS

The inspectors confirmed that proprietary information was controlled to protect from public

disclosure. No proprietary information was documented in this report.

  • On February 15, 2018, the inspectors presented the Supplemental Procedure 95001

inspection results to Mr. Frank Payne, and other members of the licensee staff.

  • On February 15, 2018, the NRC discussed the performance of the Perry Nuclear Power

Plant in accordance with IMC 0305, Section 10.01. during a regulatory performance

meeting. The meeting was attended by the Mr. Billy Dickson, Chief, Branch 5 and Mr. Frank

Payne, Plant General Manager, and other senior licensee staff. The NRC and licensee

discussed the issues related to the WHITE finding resulted in Perry Nuclear Power Plant

remaining in the Regulatory Response Column of the Action Matrix.

DOCUMENTS REVIEWED

95001 - Supplemental Inspection Response to Action Matrix Column 2 Inputs

Condition Reports

- CR-2016-13183; During performance of PTI-R43-P0006B CB-1, CB-2, CB-3, CB-4 DC

Breakers Trip when loaded (superseded by root cause evaluation CR-2016-14456)

- CR-2016-14251; Additional vulnerabilities identified in surge suppression diode on the DIV 1

and DIV 2 Diesel Generators

- CR-2016-14456; PRA Evaluation Associated with Div 2 EDG MSPI Failure (CR-2016-13183)

Indicates a Preliminary Level of Significance that Warrants a Root Cause Evaluation

- CR-2017-05709; 2017 Pre-NRC PI&R Assessment does not meet Smart Criteria

- CR-2017-06396; NRC Apparent Violation- Unsuitable Application of Surge Suppression

Diodes in Standby Diesel Generator Control Power Circuitry

- CR-2018-00268; Evaluation of violation 05000400/2017009-01 "Unsuitable Application of

Surge Suppression Diodes in Standby Diesel Generator Control Power Circuitry"

Condition Reports Generated During Inspection

- CR-2018-01316; NRC 95001 Inspection; OE Review for CR-2016-14456

- CR-2018-01335; NRC 95001 Inspection; Training Needs Analysis for Root

Cause 2016-14456

Procedures

- NOBP-LP-2011; Root Cause Analysis; Revision 22

- NOBP-ER-3017; Selection and Devlopment of Engineering Mentors; Revision 00

Others

- ECP 16-0348-004; Modify Div. 2 Diesel Generator control circuitry for relay flyback diode

failure; dated December 14, 2016

- Equipment Apparent Cause Evaluation Template (EACE) Diode Failure on Relay R10BB;

December 2, 2016

- Failure Analysis Report CF16-060; BETA Laboratory Services Section; Diode, Surge

Suppressor; November 11, 2016

- Design Basis Assessment Report- Second Half 2017

- E-Mail from

M. Koberling to JPR Mentors and Supervisors; November 30, 2017

- Job Performance Requirement, JPR 1.1; Engineering Change, Preparation, Approval,

Turnover and Closeout, July 17, 2017

14