ML20239A835

Results of the ROP Self-Assessment Effectiveness Review of the Cross-Cutting Issues Program

ML20239A835
Person / Time
Issue date:	09/21/2020
From:	Mark Franke, Chris Miller NRC/NRR/DRO, Division of Reactor Safety II
To:	Ho Nieh Office of Nuclear Reactor Regulation
Alex Garmoe, NRR/DRO/IRAB, 415-3814
Shared Package
ML20239A806	List: ML20239A834 ML20239A835 ML20239A836 ML20239A837
References
Download: ML20239A835 (32)

Text

UNITED STATES NUCLEAR REGULATORY COMMISSION WASHINGTON, D.C. 20555-0001 September 21, 2020 MEMORANDUM TO: Ho K. Nieh, Director Office of Nuclear Reactor Regulation FROM: Christopher G. Miller, Director /RA/

Division of Reactor Oversight Office of Nuclear Reactor Regulation Mark E. Franke, Director /RA/

Division of Reactor Safety Region II Office

SUBJECT:

RESULTS OF THE REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT EFFECTIVENESS REVIEW OF THE CROSS-CUTTING ISSUES PROGRAM The report attached to this memorandum presents results from the Reactor Oversight Process (ROP) self-assessment effectiveness review of the Cross-Cutting Issues (CCI) program. A team comprised of staff from the Office of Nuclear Reactor Regulation (NRR), the Office of Research, and three of the regional offices evaluated implementation of the CCI program and the effect of program changes implemented in calendar year 2015. Specific recommendations for your consideration and direction are included in the report. The report contains two Appendices that discuss the teams data review in detail and summarize stakeholder feedback.

Appendix A contains essential supporting information for understanding the teams conclusions and recommendations.

The Division of Reactor Oversight (DRO) has reviewed the work of the team, and is very appreciative of the strong, data driven work the team has accomplished in their effectiveness review. In addition to making clear ties to the effectiveness of the CCI program using finding trends, CCI program information, and plant performance data, the team has also made some insightful recommendations that could help improve the program. Their recommendations take into account not only the potential future benefit of the changes, but also the level of effort required to accomplish them. Ways to improve program clarity, and simplicity are particularly attractive, given the complicated method of classifying the data now, under 23 aspects and waiting varying lengths of time to act on that data. DRO looks forward to engaging with NRR, NSIR, and the Regions to determine which of these recommendations should be pursued in the near term, with eventual engagement in a public forum.

The effectiveness review charter is available in the Agencywide Documents Access and Management System (ADAMS) at Accession No. ML19179A105. In addition to this memorandum and report, the team documented additional information about the options considered, stakeholder feedback, and CCI program history and evolution. All of this information is available in ADAMS Package ML20239A806.

H. Nieh 2

Enclosures:

1. Report of the ROP Self-Assessment Effectiveness Review of the CCI Program

2. Appendix A, Data Analysis,

3. Appendix B, Stakeholder Views of the CCI Program, CONTACT: Alex Garmoe, NRR/DRO 301-415-3814

H. Nieh 2

SUBJECT:

RESULTS OF THE REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT EFFECTIVENESS REVIEW OF THE CROSS-CUTTING ISSUES PROGRAM DATED SEPTEMBER 21, 2020 DISTRIBUTION:

PUBLIC RidsNrrDro RidsNrrDroIrab RidsNrrOd RidsRgn1MailCenter RidsRgn2MailCenter RidsRgn3MailCenter RidsRgn4MailCenter HNieh, NRR MGavrilas, NRR CMiller, NRR MFerdas, NRR THipschman, NRR MFranke, RII AMasters, NRR AGarmoe, NRR HFreeman, RIV WSchaup, RIII RSigmon, NRR SMorrow, NRR DLew, RI RLorson, RI LDude, RII JMunday, RII JGeissner, RIII KOBrien, RIII SMorris, RIV TVegel, RIV RidsRgn1MailCenter RidsRgn2MailCenter RidsRG3MailCenter RidsRgn4MailCenter ADAMS ACCESSION Nos.:

ML20239A806 (Package)

ML20239A835 (Memo) *concurred via email NRR-106 OFFICE NRR/DRO/IOEB RES/DRA/HFRB RIV/DRP/RPBA RIII/DRP/RPB1/LRO NAME RSigmon* SMorrow* HFreeman* WSchaup*

DATE 09/15/2020 09/15/2020 09/21/2020 09/14/2020 OFFICE NRR/DRO/IRAB NRR/DRO/IRAB RII/DRS NRR/DRO NAME AGarmoe* AMasters* MFranke* CMiller*

DATE 09/15/2020 09/15/2020 09/15/2020 9/21/2020 OFFICIAL RECORD COPY

Report of the Reactor Oversight Process Self-Assessment Effectiveness Review of the Cross-Cutting Issues Program

Background

In summer 2019, a multi-disciplinary team of staff from multiple agency offices was assigned to conduct an effectiveness review of the Cross-Cutting Issue (CCI) program. The team charter is available in Agencywide Documents Access and Management System (ADAMS) at Accession No. ML19179A105. The team was comprised of the following individuals:

Mark Franke (Executive Sponsor) Director, Division of Reactor Safety, Region II Anthony Masters (Advisor) Chief, ROP Assessment Branch, NRR/DRO Alex Garmoe (Team Lead) Senior Reactor Operations Engineer, NRR/DRO William Schaup Senior Resident Inspector, Region III/DRP Harry Freeman Senior Project Engineer, Region IV/DRS Stephanie Morrow Human Factors Engineer, RES/DRA Rebecca Sigmon Reactor Systems Engineer, NRR/DRO Nick Hobbs Resident Inspector, Region II/DRP Effectiveness reviews were added to the Reactor Oversight Process (ROP) self-assessment program as part of the November 23, 2015, revision to Inspection Manual Chapter (IMC) 0307, Reactor Oversight Process Self-Assessment Program. These effectiveness reviews are intended to assess recently implemented ROP changes to evaluate their effectiveness to ensure that the intended results have been realized and to evaluate any unintended consequences. Additionally, in late 2018 staff began the ROP enhancement project, which solicited and reviewed recommendations from internal and external stakeholders on ways to improve the ROP.

The history of the CCI program and the bases behind program changes over time is discussed in a separate document developed to accompany this report, available in ADAMS as Accession No. ML20239A834. The CCI program initially operated under the premise that CCIs would be revealed through greater than Green inspection findings or degraded performance indicators (PIs), as noted in the Staff Requirements Memorandum (SRM) for SECY-00-0049, which stated, The staff should show that cross-cutting issues they identify have a clear and strong link to significant inspection findings or degraded PIs before the staff attempts to take action on programmatic concerns. The most significant changes in the history of the CCI program, which implemented granularity to the program that remains to this day, came in response to the Davis-Besse vessel head degradation event, which revealed that cultural contributors can lead to risk significant issues that other aspects of the ROP may not detect in advance. As stated in SECY-04-111, A weak Safety Culture was identified as a root cause of the reactor head degradation at the Davis-Besse nuclear power plant. Because the NRC staff was limited in its ability to find this problem prior to the incident, the staff believes implementation of assessment processes which facilitate identification of negative trends in the area of Safety Culture should be explored.

Davis-Besse was in Column 1 of the Action Matrix for the first two years of the ROP, and then transitioned directly to IMC 0350 oversight and was shut down for several years. Based on the SRM for SECY-02-0166, Davis-Besse Lessons Learned Task Force Recommendation 3.3.4(5),

and Government Accountability Office Report GAO-04-415, Nuclear Regulation - NRC Needs to More Aggressively and Comprehensively Resolve Issues Related to the Davis-Besse Nuclear Power Plants Shutdown, the staff proposed to more fully address safety culture via the CCI Enclosure 1

program, which the Commission approved in the SRM for SECY-04-111. The Commission direction and staff action in response to Davis-Besse show that an effective CCI program must be able to identify and raise awareness to underlying cultural issues even if they have not yet revealed themselves through significant events.

The last effectiveness review of the CCI program was performed in 2014 (ADAMS Accession No. ML14099A171). Changes to the CCI program were implemented in the April 9, 2015, revision to IMC 0305. The team devoted specific attention to reviewing the changes made in 2015 to assess whether changes addressed the problem statement as intended and whether there were any unintended implications on the overall effectiveness of the CCI program.

Table 1: Cross-Cutting Areas and Aspects in the Current CCI Program Human Performance SCWE H.1 Resources H.8 Procedure Adherence S.1 SCWE Policy H.2 Field Presence H.9 Training S.2 Alternate Process H.3 Change Management H.10 Bases for Decisions S.3 Free Flow of Information H.4 Teamwork H.11 Challenge the Unknown H.5 Work Management H.12 Avoid Complacency H.6 Design Margins H.13 Consistent Process H.7 Documentation H.14 Conservative Bias PI&R Other P.1 Identification X.1 Incentives X.7 Job Ownership P.2 Evaluation X.2 Commitment to Safety X.8 Benchmarking P.3 Resolution X.3 Roles and Authorities X.9 Work Process P.4 Trending X.4 Constant Examination Communication P.5 Operating Experience X.5 Leader Behaviors X.10 Expectations P.6 Self-Assessment X.6 Standards X.11 Challenge Assumptions X.12 Accountability for Decisions Table 2: Cross-cutting theme and CCI thresholds for pre-2015 and 2015+ program iterations 2006-2014 2015-present Theme: 4+ findings in any single HU or PI&R Theme: 6+ findings in any single HU or PI&R aspect aspect 12+ findings in the PI&R area 20+ findings in the HU area SCCI: Theme present + agency concern with CCI: Theme present for three consecutive licensee actions assessment periods SCWE Theme: 1+ finding in any single SCWE aspect; Theme: 1+ finding in any single SCWE aspect; Chilling effect letter; Chilling effect letter; Escalated enforcement involving Escalated enforcement involving discrimination discrimination affecting more than one individual SCCI: Impact on SCWE is not isolated + CCI: Theme present for four consecutive agency concern with licensee actions assessment periods The current cross-cutting areas and aspects are shown in Table 1. There are three cross-cutting areas (Human Performance (HU), Problem Identification and Resolution (PI&R),

and Safety Conscious Work Environment (SCWE)) that are always available for use. The Other cross-cutting aspects are within the scope of nuclear safety culture but do not fit within 2

the three cross-cutting areas. Under existing program guidance they are only available for use in supplemental inspections.

The CCI program underwent significant changes in 2015 and has since accumulated several years of run-time. Several recommendations submitted as part of the ROP enhancement initiative involved reviewing or modifying the CCI program. These together formed the basis for conducting an effectiveness review of the CCI program at this time.

Effectiveness Criteria In order to be considered effective, a program should be able to demonstrate that it is meeting its stated objective. The objective of the CCI program is stated in section 14 of IMC 0305 as:

The NRC identifies a cross-cutting issue (CCI) to inform the licensee that the NRC has a concern with the licensees performance in the cross-cutting area and to encourage the licensee to take appropriate actions before more significant performance issues emerge This objective is comprised of several key components that the team assessed to determine whether the objective of the program is being met through program implementation:

1. identify cross-cutting issues

2. inform licensees of NRC concerns

3. encourage licensee actions before more significant performance issues emerge In addition, SECY-99-007 discusses the importance of applying the NRCs Principles of Good Regulation to the ROP to instill confidence that nuclear power facilities are regulated in a manner that meets the NRCs mission. These principles can also be applied to assess the effectiveness of the CCI program.

Independence Openness Efficiency Clarity Reliability In assessing the effectiveness of the CCI program the team considered the stated objective of the program, history of the program, and the Principles of Good Regulation as information was reviewed.

Data and Information Reviewed The teams review of information and data is fully explained in Appendix A, Data Analysis, which is essential information to understand the conclusions and recommendations in this report.

The team reviewed the information listed in the teams charter, which included inspection findings and plant performance data from the operating reactor assessment program, documentation of cross-cutting themes and issues in assessment letters, current and historical inspection manual chapter and inspection procedure guidance, staff generated Commission papers and Commission responses, and feedback from internal and external stakeholders. The team also reached out to Consejo de Seguridad Nuclear (the nuclear regulatory agency in 3

Spain) to review their safety culture oversight process since they implement a very similar oversight process as the ROP that includes a similar substantive cross-cutting issues process.

The teams review of data from the operating reactor assessment program primarily consisted of inspection findings with cross-cutting aspects in the HU, PI&R, and SCWE cross-cutting areas from 2007 to 2019. This time period was the focus of data review since specific cross-cutting aspects were not assigned to inspection findings prior to implementation of the 2006 CCI program. The majority of the data pertained to inspection findings with cross-cutting aspects in HU or PI&R since the SCWE cross-cutting aspects have rarely been assigned to inspection findings. The threshold or criteria for a theme in the SCWE cross-cutting area (a chilling effect letter, certain escalated enforcement actions, or a single finding with a SCWE cross-cutting aspect) are substantially different from the threshold for an HU or PI&R cross-cutting theme and, unlike the criteria for an HU or PI&R CCI, a SCWE theme or CCI is not based on cross-cutting aspect trends.

Much of the teams data analysis used greater-than-Green assessment inputs, the majority of which have been White, as a measure of declining performance. This was not done to imply when the CCI program should intervene but was used since it represents a readily retrievable, sizable population of data that could be assessed. The CCI program objective is to identify cross-cutting concerns and encourage licensee actions before more significant performance issues, which is otherwise undefined, emerge. While the label for a White finding is low to moderate safety significance, at least as far back as 2003 IMC 0305 has included in the list of definitions safety significant finding/performance indicator, which is defined as [a]n inspection finding with a safety significance greater than green or a performance indicator that is greater than green. Further, since roughly 98 percent of findings and PIs have been Green, White findings or PIs are more significant than the vast majority of issues. Conversely, Figure 1 of IMC 0305 does not use the term significant degradation in safety performance until Column 4 of the Action Matrix. Thus, more significant performance issues could potentially be viewed as referring to movement to Column 2, especially since a plant may now remain in Column 2 with multiple White inputs, or movement to Column 3 or higher in the Action Matrix.

The teams review of information and data included analysis of all cross-cutting aspects issued since 2006, assessment of plants that reached CCI program thresholds, and assessment of plants in which performance declined. In reviewing plants in which performance declined, the team focused on plants that met Column 4 criteria and, for more recent years, plants that incurred the highest number of greater-than-Green assessment inputs and spent the most time outside of Column 1. The team made the following observations based on the information reviewed:

• There is a statistically significant predictive association between accumulating cross-cutting aspects and future performance decline, supporting the premise of the CCI program as a forward-looking component of a largely reactive oversight process.

• Whether looking at the pre-2015 or 2015 through present (2015+) CCI programs, plants that reached cross-cutting theme thresholds were at least twice as likely as plants without themes to exhibit declining performance as measured by Action Matrix movement out of Column 1 or nearly twice as likely to experience a precursor event as determined by the Accident Sequence Precursor program within the next year.

• Based on historical data review, the 2015+ CCI program is less susceptible to false positives or data scatter than the pre-2015 program (for this review a false positive was 4

defined as a plant that reached CCI program thresholds but did not exhibit performance decline within the following year).

• A detailed review of the five plants that met criteria for entering Column 4 since 2006 revealed that the 2015+ program would not have identified a CCI at any of the plants until at least some performance decline had already occurred and in some cases not at all.

Assessment of Effectiveness Identify cross-cutting issues The percentage of findings that were assigned a cross-cutting aspect remained steady over the years, both in terms of total aspects and by cross-cutting area (see Figure 1). Roughly 80 to 85 percent of all inspection findings have been assigned a cross-cutting aspect throughout program history. The bulk of cross-cutting aspects have been in the HU cross-cutting area. There has been a minor trend toward more HU cross-cutting aspects over roughly the past ten years. The team did not find this minor trend to be notable enough to explore further.

Figure 1: Percentage of Inspection Findings Assigned a Cross-Cutting Aspect The team concluded that the program was being used by inspectors and the existing cross-cutting areas and aspects covered the majority of issues inspectors encounter. Because the assignment of cross-cutting aspects is limited to issues reflecting current performance, defined as within the past three years, it is expected that some findings will not be assigned a cross-cutting aspect in one of the three cross-cutting areas. A review of the definitions of the cross-cutting aspects revealed overlap between many of the aspects, and experience from team members and feedback from stakeholders indicated that in some cases more than one cross-cutting aspect could apply to a single inspection finding. Often in these situations increased effort is expended evaluating and determining the proper cross-cutting aspect, in most cases for a finding of very low safety significance in which a risk-informed application of resources would 5

support less effort, not more. While this appears to be a fairly widely held observation, the team could not further estimate the actual burden since the agencys time reporting system does not specifically track the time spent selecting a cross-cutting aspect. Additionally, a cross-cutting aspect is defined as the proximate cause of a performance deficiency. This has often been more simply stated the cross-cutting aspect cannot be the same as the performance deficiency, and has resulted in inspectors sometimes veering away from assigning the most relevant cross-cutting aspect if it is viewed as too close to the performance deficiency, especially in the case of findings involving PI&R deficiencies. Beyond just the added resource burden, multiple overlapping cross-cutting aspects that may reasonably apply to a finding and the application of the cross-cutting aspect definition could lead to questions about the credibility and reliability of program conclusions identified through trending the cross-cutting aspects.

The team also looked at whether the existing structure of the program supported the proactive identification of concerns in cross-cutting areas before significant performance issues emerged.

First, the team identified plants that had cross-cutting themes or CCIs in the 2015+ time period to determine whether the program was identifying plants with actual cross-cutting concerns.

Then the team reviewed plants that exhibited performance decline and looked back at how the CCI program assessed them. Two performance outliers in the 2015+ timeframe were identified, with the assessment of both sites including multiple greater-than-Green (GTG) inputs and extended time in Column 2. No plant moved to Column 3 or beyond in this time period though one of the 2015+ plants identified would have moved to Column 3, had the criteria for entering Column 3 not been changed from two White inputs to three White inputs in 2015. Similarly, no plant has met the criteria for a CCI in HU or PI&R since program changes implemented in 2015.

The team sought to review additional data points and to assess plants with more significant performance declines.

The team looked to the pre-2015 timeframe and reviewed the five plants that were in Column 1 and then at some point met criteria for movement to Column 4 to determine how the 2015+ CCI program would have assessed those plants. Five plants were identified that met the criteria for movement to Column 4 of the Action Matrix, one of which ultimately proceeded to the IMC 0350 process and one remained in Column 3 based on an Action Matrix deviation. In total seven plants were reviewed in detail, two in the 2015+ timeframe and five in the pre-2015 timeframe.

The 2015+ CCI program would not have opened a CCI for any of these seven cases before performance declined, and in some cases not even after performance declined, whereas in every case the pre-2015 CCI criteria would have been met before the performance decline.

Discussion with individuals involved in performance assessment of the 2015+ outlier plants revealed that staff was concerned about the human performance cross-cutting area as performance declined and review of inspection reports and lessons learned documents for the Column 4 plants revealed cultural issues in cross-cutting areas as contributors to the performance decline. Appendix A provides more detail about each specific example.

Based on the review described in this section, the team determined that the CCI program is being implemented by inspectors through the assignment of cross-cutting aspects to findings at a reasonable rate. Cross-cutting aspects are also being reviewed for cross-cutting themes at annual assessment and 2nd quarter review time periods. However, stakeholder feedback and team experience indicated that, in situations where multiple cross-cutting aspects could apply, increased resources are often spent determining the proper cross-cutting aspect, which is counter to the goal of risk-informed resource allocation to issues of very low safety significance.

In addition, in the teams view the credibility and reliability of program conclusions may be questioned. This is because program conclusions are based on trending of cross-cutting aspects, in which more than one could have applied to some of the inspection findings, and 6

through implementation of a cross-cutting aspect definition that may at times steer inspectors away from the most applicable aspect.

Of the seven performance outlier or Column 4 plants reviewed in detail, the current CCI criteria would not have been met for any plant before performance declined, and in some cases not at all.

Inform licensees of NRC concerns The objective of the CCI program stated in IMC 0305 includes, in part, informing licensees of concerns in cross-cutting areas. A substantive cross-cutting issue (SCCI) was initially defined in IMC 0305 as a common performance theme (i.e., related causal factor) as evidenced by a significant number of current inspection findings in the cross-cutting areas. The SCCI was only to be described in an assessment letter if multiple findings in a cross-cutting area represented a theme, and the assessment letter was to describe the examples, place them into the proper safety context, and describe the agencys plan to monitor the issue.

In 2006 the more structured SCCI program was implemented. At this point the concept of a cross-cutting theme was introduced and was defined as four or more findings with a common cross-cutting aspect in the HU or PI&R cross-cutting areas, with separate criteria for a SCWE theme. Guidance in IMC 0305 provided for detailed documentation of SCCIs including placing the issues into the proper safety context and discussing planned follow-up actions.

The program revisions implemented in 2015 defined a cross-cutting theme as more than 6 findings with a common cross-cutting aspect in any HU or PI&R aspect, more than 20 total HU findings, or more than 12 PI&R findings. Separate criteria continued to exist for a SCWE theme.

A CCI was defined as an open theme for at least three consecutive assessment letters, with the three assessment letter waiting period (four in the case of SCWE) intended to provide the licensee an opportunity to address the cross-cutting theme before a CCI is opened. Each cross-cutting theme was to be documented in an assessment letter and agency follow-up actions began with the first occurrence of a cross-cutting theme. A cross-cutting theme would no longer exist if the threshold was no longer reached, though once a CCI was opened it would stay open until the closure criteria established in the assessment letter were met.

Stakeholder feedback and the teams review found that the 2015 changes introduced confusion or lack of clarity into the program. Both themes and CCIs are documented in assessment letters and the agency begins reviewing licensee actions once a theme is present. However, since themes merely exist or dont exist based on number of cross-cutting aspects, a theme cannot be held open like a CCI even if the licensee has taken no meaningful or effective action or if the agencys initial review reveals concerns. The program precludes the agency from taking formal regulatory action for a cross-cutting area until a theme has been present for three consecutive assessment periods, even though data show that reaching the theme threshold even once requires far more than the typical number of cross-cutting aspects issued to a plant (the theme thresholds in HU and PI&R would have required five to six times the industry average number of findings in each area in 2019, for example). Stakeholder feedback indicated that these elevated theme thresholds introduced in 2015 allowed licensees to see trends developing and take action before thresholds were reached, which the team noted appears to obviate the need for a three consecutive assessment letter waiting period.

7

No plant has reached CCI criteria since implementation of the higher thresholds and three assessment period waiting time established in the 2015 iteration of the CCI program, except for a single SCWE CCI due to a chilling effect letter. Thus, the team had limited data for assessing the informing of licensees of cross-cutting concerns via identification of a CCI. However, there are plants that have reached cross-cutting theme thresholds since the 2015+ CCI program was implemented. Since cross-cutting themes are also documented in assessment letters the team reviewed the documentation of the themes. The CCI program objective implies that the agency cannot formally express concerns in a cross-cutting area until a CCI is opened once a theme has been present for three consecutive assessment letters. The assessment letters reviewed stated that a cross-cutting theme existed because a threshold was exceeded and did not provide further insight into the behaviors or issues that led to the theme.

Figure 2: Assessment Discussion for HU Cross-Cutting Theme at a Particular Plant In one example, shown in Figure 2, a cross-cutting theme was repetitively opened and closed at a site with no descriptive language included in the assessment letters and no acknowledgement of the repetitive nature of the theme. Despite the presence of at least informal staff concerns in the cross-cutting area, the documentation of this and other cross-cutting themes was minimal and lacked context. Interested members of the public, the affected licensee, or other 8

stakeholders not directly involved would not be aware of the agencys full assessment of the cross-cutting theme and would have to individually review numerous inspection reports to determine which findings resulted in the cross-cutting theme and what the common characteristics of the findings were.

Based on the review described in this section, the team determined that licensees would be informed of cross-cutting concerns when CCI criteria are met, and in the one case since 2015 in which a CCI was identified the licensee was notified of the cross-cutting concern. Discussion with stakeholders and the teams review of information found that the change in 2015 to require a cross-cutting theme for three consecutive assessment periods before a CCI is opened, while intended to provide licensees additional opportunity to address the theme before the agency expresses concern, has created confusion and restricts the agencys ability to formally express concerns in a cross-cutting area. Cross-cutting themes are documented in assessment letters and staff is provided guidance to begin reviewing what the licensee does to address the theme, even though program guidance indicates the agency is not formally concerned until a CCI is opened. A theme automatically ceases to exist if the number of findings drops below the theme threshold, even if the agency developed concerns based on its review of licensee actions in response to the theme. Opportunities exist to enhance the clarity of CCI program insights communicated through themes and CCIs and improve documentation in assessment letters.

Encourage licensee actions before more significant performance issues emerge The team reached out to the inspector community and verified the industry assertion that at least some licensees have procedural guidance in place to monitor for developing cross-cutting trends and enter developing trends into the corrective action program (CAP) before the agencys program thresholds are exceeded. What actions are taken as a result of entry into the CAP would depend on each specific instance. Nevertheless, the team is comfortable that the mere existence of program thresholds provides at least some encouragement to licensees to be aware of CCI program data.

On the other hand, the team found that once CCI program thresholds are reached the program may provide only limited encouragement for licensees to take action. The instance outlined in Figure 2, and also shown in Figure 6, shows that a cross-cutting theme will cease to exist based on the number of findings dropping back below program thresholds, regardless of whether any assessment or actions have been taken by the licensee and whether the agency is comfortable with licensee performance in the cross-cutting area. In addition, the plant-specific information discussed in Appendix A and shown in Figures 6 through 12 shows how infrequent it is, even for plants that reached Column 4 of the Action Matrix, to reach the 2015+ theme thresholds at all, much less for more than one consecutive assessment period.

Based on the review described in this section, the team determined that the presence of the CCI program provides at least some encouragement to monitor cross-cutting aspects and take action to avoid reaching program thresholds. However, once a theme threshold is reached the program may provide only limited encouragement to take proactive action since it is unlikely that a theme would repeat and meet the criteria to open a CCI, even if no action is taken.

9

Effectiveness Conclusion Changes made in 2015 resulted in a program that, while still providing some encouragement to monitor cross-cutting aspects, may be less responsive to cross-cutting behavior indicators, may provide only limited communication about cross-cutting themes, and introduced concerns about whether the program would proactively identify cross-cutting concerns consistent with the program objective.

Based on the information reviewed, the team determined that the CCI program continues to have value by providing a focus on patterns of safety culture behaviors, but program changes made in 2015 resulted in a program that appears to be less responsive to cross-cutting behavior indicators and introduced concerns about whether the program would proactively identify cross-cutting concerns consistent with the program objective. The team assessed seven instances of definitive performance decline in which cross-cutting concerns were present or were revealed via follow-up inspection, both in the pre-2015 and 2015+ timeframes, and the 2015+ CCI program would not have opened a CCI in any of the instances. Therefore, the 2015+ program did not provide a formal or transparent mechanism in which the agency could inform the licensee of concerns in cross-cutting areas such that action could be encouraged before significant performance issues occurred. Additionally, implementation of the requirement that a theme be present for three consecutive assessment letters before a CCI may be opened was a significant factor in not meeting CCI criteria in the 2015+ examples.

The data reviewed revealed that there is a predictive relationship between increasing numbers of cross-cutting aspects and future movement in the Action Matrix or occurrence of precursor events. This validates the premise of the CCI program as a forward-looking component of a largely reactive oversight process, the importance of which is validated by reflecting on the Davis-Besse vessel head degradation event in 2002 and the corresponding updates to the CCI program. A well-structured and well-implemented CCI program can be expected to add value to the agencys process of assessing plant performance and determining appropriate regulatory activities.

The team determined that the program continues to have value by providing a focus on patterns of safety culture behaviors, but changes made in 2015 resulted in a program that appears to be less responsive to cross-cutting behavior indicators and thus introduced concerns about whether the program would proactively identify cross-cutting concerns consistent with the program objective. The team explored potential program improvements to address these effectiveness observations.

Program Recommendations The team reflected on the CCI program objective and the Principles of Good Regulation to identify attributes of a well-functioning CCI program.

• Straightforward monitoring of cross-cutting insights with a threshold that balances the need for proactive sensitivity with the desire to minimize false positives and ensure actual trends rather than data scatter are identified

• Clear and concise criteria for opening a CCI with minimal subjective consideration available for unique circumstances with the rationale for CCI decisions described in assessment letters in a complete and open manner 10

• Consistent follow-up of CCIs via a pre-established inspection procedure with pre-established objectives on a timeline commensurate with completion of licensee actions to address the issue The team was open to any and all ideas when considering possible revisions to the CCI program to improve the ability of the CCI program objective to be effectively carried out.

Options considered by the team ranged from maintaining the status quo, eliminating the CCI program altogether, and a variety of potential program changes. In evaluating possible improvements to the CCI program the team considered the information reviewed, conclusions reached, and the Principles of Good Regulation. More detail about the dispositioning of the all options considered is available in ADAMS as Accession No. ML20239A836. A simple list of all options considered is shown in Table 3.

Table 3: Options Considered by the Team for CCI Program Improvements 1 Overall Concept of the CCI Program

a. Completely eliminate the CCI program

b. Status quo - maintain the same program

c. Incorporate CCI insights into significance determination process (SDP) evaluations for affected plants

d. Consider additional sources of data beyond just more than minor NRC-identified or self-revealed inspection findings 2 Monitoring of Findings for Cross-Cutting Insights

a. Monitor findings using only the three cross-cutting areas

b. Monitor findings using the nine safety culture traits

c. Trend findings over a longer rolling period of time

d. Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached

e. Assess for CCIs quarterly rather than every 6 months

f. Eliminate chilling effect letter and discrimination enforcement criteria for opening a SCWE CCI; treat SCWE comparably to the HU and PI&R cross-cutting areas

g. Eliminate the aspect-specific threshold for opening a CCI; use just backstop values 3 Criteria and Threshold for Opening a Cross-Cutting Issue

a. Establish a variable threshold that would adjust based on changes in the overall number of findings being issued

b. Return to the thresholds established under the 2006 program (4 findings per aspect)

c. Use the 2014 methodology with current data to establish new thresholds

d. Provide more detailed discussion of CCI insights and decisions in assessment letters 4 CCI Follow-Up

a. Timing of follow-up actions should be based on licensee readiness

b. Establish standardized CCI follow-up inspection guidance

c. CCI follow-up could be scoped into an existing planned supplemental inspection

d. CCIs should not be able to be closed solely based on a reduction in inspection findings The team determined that some of the options considered likely would not enhance the program in a manner that improves implementation of the program objective whereas others likely would.

The level of effort and complexity involved with implementation of these options varies along with the level of impact each option on its own would have. Based on a consideration of how each option would improve implementation of the program objective and remain consistent with 11

the principles of good regulation, and a qualitative consideration of the resource cost to implement the change and resource savings once the change is in place, the team identified options that are being carried forward as recommendations. The recommendations are divided into two groups, and while the team is recommending implementation of all options, individual options could be selected for implementation. A more complete understanding of each recommended option is available in ADAMS as Accession No. ML20239A836.

Group 1 Recommendations Group 1 recommendations are those that could be implemented with a lower level of resources with a relatively short implementation time. These recommendations are intended to make some improvement in the implementation of the program objective. The impact of these changes would not be expected to be as significant as the impact of the Group 2 changes.

• 3.d: Provide more detailed discussion of CCI insights and decisions in assessment letters

• 4.a: Timing of follow-up actions should be based on licensee readiness

• 4.b: Establish standardized CCI follow-up inspection guidance with objectives

• 4.c: CCI follow-up could be scoped into an existing planned supplemental inspection

• 4.d: CCIs should not be able to be closed solely based on a reduction in inspection findings A review of the resource implications of these changes qualitatively considered both resource costs to implement and downstream resource cost benefits once implemented.

The agency resource cost to implement these changes would be minimal.

Recommendations 3.d, 4.a, 4.c, and 4.d would be accomplished through updates to existing guidance. Recommendation 4.b would require relocation of existing follow-up guidance from an IMC to an inspection procedure (IP), and even development of standardized follow-up objectives is in the teams view not a significant effort as IP95001 objectives would offer a sound starting point. Once implemented, Recommendation 3.d may result in a very small resource increase to more completely document cross-cutting insights in assessment letters.

The remaining recommendations would result in resource savings since follow-up objectives would no longer need to be determined on a case-by-case basis, follow-up activities would not automatically scale up on 6-month intervals but rather based on the actual completion of licensee assessments or actions, and CCI follow-up could be scoped into an already planned supplemental inspection.

These recommendations are expected to have minimal, if any, resource impact on industry.

Recommendations 3.d and 4.d should have no impact on industry resources.

Recommendation 4.c, when used, would result in resource savings to industry.

Recommendations 4.a and 4.b might necessitate licensee procedure updates but should ultimately result in reduced resources spent in a more targeted manner.

Group 2 Recommendations Group 2 recommendations are intended to significantly improve the CCI program and make implementation of the program objective more efficient in a manner beyond the improvements made by Group 1 recommendations. These recommendations would require a higher resource commitment and more associated staff effort and time as well as 12

stakeholder interaction to be implemented. These recommendations build on the Group 1 recommendations.

• 2.b: Monitor findings using the nine safety culture traits (2.a is also an acceptable though less preferable approach; 2.g and 3.c are inherently included in this recommendation)

• 2.d: Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached Recommendation 2.b (or 2.a if that is selected instead) could be implemented through updates to existing guidance that would not be all that significant. Recommendation 2.a could be implemented with minimal change to the existing program structure, and even though Recommendation 2.b is a larger change, the translation from the current 23 cross-cutting aspects to the safety culture traits is already documented in IMC 0310.

Recommendation 2.d could be implemented with minor changes to existing guidance. Once in place, these recommendations would result in larger resource savings than the Group 1 recommendations since identifying the applicable cross-cutting aspect/area/trait would be more straightforward. In the case of Recommendation 2.b, both agency and industry safety culture program products would be using the same language and categories, which would streamline reviews of licensee evaluations and actions.

The agency resource cost for implementation of these recommendations is more significant than Group 1 as they would likely result in the need to develop a Commission Paper.

Determination, with public participation, of proper cross-cutting data thresholds to support Recommendation 2.a/2.b may also involve more than minimal resources to implement.

The resource impact to industry of these recommendations would be more significant than the Group 1 recommendations. While recommendation 2.a would not likely have a significant resource impact, recommendation 2.b would necessitate licensee procedure changes and potentially some program modifications to account for different cross-cutting categorization. Recommendation 2.d might necessitate licensee procedure updates and, depending on the actions a licensees procedures prescribe for cross-cutting themes and CCIs, this recommendation might result in more actions being taken sooner by licensees.

Even if more robust actions would be taken sooner, this may not be a resource cost since it could result in effectively addressing concerns sooner, thus clearing cross-cutting concerns sooner.

The team is not recommending implementation of option 1.d (consider additional sources of data beyond just more than minor NRC-identified or self-revealed inspection findings) at this time. However, it may be prudent for the agency to begin consideration of possible additional sources of data for cross-cutting insights. The CCI program uses inspection findings to identify concerns in cross-cutting areas but the trend in inspection findings issued per year has been steadily decreasing. From 2006-2014 the agency issued roughly 13-14 findings per site per year. Starting in 2015 a steady decline began and by 2019 the agency issued roughly 6 findings per site per year. The decreasing trend in inspection findings was the subject of a recent staff study (ADAMS Accession No. ML19225D281) conducted in response to Commission Staff Requirements Memorandum SRM-M190620 (ADAMS Accession No. ML19178A030). Considering the current CCI program thresholds and the recent decreasing trend in findings, a situation in which there is not enough data available to provide a meaningful assessment under the current program structure may soon arrive, particularly if recommendations made in this report are not implemented.

13

The recommendations in Group 1 are viewed by the team as a starting point for program improvements since they require less effort and time to implement. The Group 1 recommendations could be implemented in parallel with the more significant Group 2 recommendations as one suite of improvements, in advance of the Group 2 recommendations as a two-step approach, or on their own with possible future consideration of Group 2 recommendations.

A separate effort to perform a comprehensive review of the PI&R Program was underway at the time of this CCI Program effectiveness review. While these were separate efforts, there was recognition all along that there are links between the CCI and PI&R Programs and they can positively or negatively impact each other. Similar to the CCI Program, numerous internal and public stakeholders hold the view that the PI&R program has not been as efficient and effective as it could be. The recommendations in this report would result in more reliable CCI Program conclusions that would serve as a more meaningful input to an improved PI&R assessment process that is being recommended by the PI&R team in their report (ADAMS Package Accession No. ML20247J590).

Figure 3: Recommendations for Improving Implementation of the CCI Program Objective Group 1 Group 2 Lower level of effort More significant level of effort Smaller changes intended to Larger changes intended to improve the CCI program but would significantly improve the CCI not address all issues identified program

• More complete

• Simplify the monitoring of documentation in cross-cutting insights of assessment letters inspection findings

• Pre-establish follow-up

• Remove the theme waiting guidance and objectives period and open a CCI once a threshold is

• Timing of follow-up actions reached; offer limited consistent with completion subjective consideration of licensee actions The team considered the Principles of Good Regulation as options were assessed and recommendations developed. The team believes the full suite of recommendations is consistent with all five Principles and would result in a CCI program that is implemented in a manner consistent with all five Principles. Implementation of just a subset of recommendations would not be contrary to any of the Principles.

Independence: The program would continue to use an independent source of data (inspection findings) and independently assess that data to arrive at a determination of the appropriate agency response. The agency response would independently confirm whether the cross-cutting concerns have been sufficiently addressed. The team believes the current program fully reflects this Principle and implementation of some or all recommendations would result in a program that continues to fully reflect this Principle.

14

Clarity: Clarity in the CCI program would be improved through simplification of the monitoring for cross-cutting insights, removal of the concept of a cross-cutting theme, and pre-establishment of standard criteria for opening and closing CCIs. The team believes the current program does not fully reflect this Principle, which would be addressed by recommendations 2.b (or 2.a if selected instead), 2.d, and 4.b.

Openness: The program would improve openness through more comprehensive description of the basis for opening, maintaining open, or closing CCIs in public assessment letters. The team believes the current program does not fully reflect this Principle, which would be addressed by recommendations 3.d and 4.b.

Reliability: Program reliability would be enhanced through simplification of the monitoring for cross-cutting insights in a manner that removes the overlap and subjectivity between the current 23 cross-cutting aspects. This combined with ensuring thresholds are appropriately set to balance the accumulation of sufficient data with the desired sensitivity to emerging trends would result in more reliable, credible program conclusions. The team believes the current program does not fully reflect this Principle, which would be addressed by recommendations 2.b (or 2.a if selected instead) and 4.b.

Efficiency: Efficiency in the program would be enhanced by the simplified approach to monitoring for cross-cutting insights, which would reduce the resources spent dealing with the subjectivity and overlap in the current 23 cross-cutting aspects. Establishing standardized criteria for the follow-up and closure of CCIs would also result in more efficient use of resources as appropriate criteria would not need to be developed for each individual CCI. The team believes the current program does not fully reflect this Principle, which would be addressed by recommendations 2.b (or 2.a if selected instead), 2.d, 4.a, 4.b, and 4.c.

15

APPENDIX A: Data Analysis The teams data review consisted of several major parts:

1. Review of program historical data and major program changes

2. Review of program assessment of plants that were performance outliers

3. Review of the relationship between cross-cutting insights and licensee performance This appendix discusses each of the three main topics in detail.

The teams review of historical program data focused on the Human Performance (HU) and Problem Identification and Resolution (PI&R) cross-cutting areas since the criteria for a Safety Conscious Work Environment (SCWE) cross-cutting issue (CCI) (a chilling effect letter, certain escalated enforcement actions, or a single finding with a SCWE cross-cutting aspect) are substantially different from the criteria for a HU or PI&R CCI and do not involve looking for trends in cross-cutting aspects. There is not a meaningful amount of data to review for the SCWE cross-cutting area.

In some cases the teams review of data used greater-than-Green assessment inputs, the majority of which have been White, as a measure of declining performance. This was not done to imply when the CCI program should intervene but was used since it represents a readily retrievable, sizable population of data that could be assessed. The CCI program objective is to identify cross-cutting concerns and encourage licensee actions before more significant performance issues, which is otherwise undefined, emerge. At least as far back as 2003, Inspection Manual Chapter (IMC) 0305 has included in the list of definitions safety significant finding/performance indicator, which is defined as [a]n inspection finding with a safety significance greater than green or a performance indicator that is greater than green. Further, since roughly 98 percent of findings and PIs have been Green, White findings or performance indicators (PIs) are more significant than the vast majority of issues. Conversely, Figure 1 of IMC 0305 does not use the term significant degradation in safety performance until Column 4 of the Action Matrix. Thus, more significant performance issues could potentially be viewed as referring to movement to Column 2, especially since a plant may now remain in Column 2 with multiple White inputs, or movement to Column 3 or higher in the Action Matrix.

1. Review of program historical data and major program changes The information presented in Figure 4 shows the number of plants that reached one or more cross-cutting theme data thresholds and the number of plants that had one or more CCIs opened on an annual basis from 2007 to 2019. The 2015 changes to the CCI program changed the threshold for determining whether a plant had a cross-cutting theme in the HU or PI&R cross-cutting area. The dark blue bars show when a pre-2015 theme threshold was or would have been exceeded and the light blue bars show when a 2015+ theme threshold was or would have been exceeded. The number of plants that had a new cross-cutting issue opened is shown in orange.

The 2015+ theme thresholds were reached less often than the pre-2015 theme thresholds, which was expected since the thresholds were elevated in 2015. Since the 2015 changes were implemented, there have been 11 instances where a plant had a cross-cutting theme and one instance where a CCI was opened. However, the opened CCI was in the SCWE cross-cutting area and based on a long-standing Chilling Effect Letter. Thus, since 2015 the cross-cutting Enclosure 2

aspect theme threshold has been reached ten times and no CCI has been opened as a result of cross-cutting aspects.

Figure 4: Number of Plants with HU or PI&R Cross-Cutting Themes and CCIs from 2007-2019 The information in Table 4 shows plants that reached cross-cutting theme thresholds in the 2014-2019 timeframe using the pre-2015 thresholds and the 2015+ thresholds. Each plants position in the ROP Action Matrix is shown in parentheses. Because the objective of the CCI program is to inform of concerns in cross-cutting areas before significant performance issues emerge, the team was most interested in plants that were in Column 1. Column 1 plants that moved to a higher column in the Reactor Oversight Process (ROP) Action Matrix within the following year after a theme was identified are noted with an asterisk and shown in bold.

The information in Table 4 is summarized in Figure 5. From 2014-2019 the pre-2015 theme thresholds would have been reached far more often than the 2015+ theme thresholds.

Focusing on Column 1 plants, the pre-2015 theme thresholds would have been reached 17 times and of those eight (47 percent) subsequently moved in the Action Matrix within the following year. The 2015+ theme thresholds would have been reached by three Column 1 plants and of those two plants (67 percent) subsequently moved in the Action Matrix (the third plant reached the theme thresholds twice just after exiting the IMC 0350 process, which is not unexpected due to the one year look-back at cross-cutting aspects).

This information shows that the 2015+ program identified less plants that did not demonstrate performance concerns (i.e. less false positives), however the 2015+ program did not identify eight Column 1 plants that did subsequently show some performance decline (i.e. possible false negative). Discussion with staff responsible for oversight and assessment of these eight plants found that the staff believed it would have been appropriate for the CCI program to have identified them. This raised a potential concern that the 2015 changes resulted in a less responsive program that may not proactively identify cross-cutting concerns prior to more significant performance issues. To explore this further, the team identified plants that specifically showed performance declines or outlier performance and assessed how the CCI program treated them.

2

Table 4: Plants that Met Cross-Cutting Theme Thresholds from 2014-2019 Pre-2015 Theme Threshold Met 2015+ Theme Threshold Met 4 in same aspect 6 in same aspect 20 in HU area 12 in PI&R area 2014 ANO (4) Fort Calhoun Fort Calhoun Fort Calhoun Duane Arnold (3) (0350) (0350) (0350)

Fort Calhoun (0350)

Palo Verde (1)

Salem (2) 2015 Fort Calhoun (1) Fort Calhoun (1) Pilgrim (4)

Grand Gulf (1)* River Bend (2)

Pilgrim (4)

River Bend (2)

Surry (1)

Susquehanna (2) 2016 Browns Ferry (1)* Fort Calhoun (1) Clinton (1)*

Fermi (1)* Watts Bar (1)*

Hope Creek (1)*

Perry (1)*

Summer (1)

Susquehanna (1)

Wolf Creek (1)*

Grand Gulf (1)

Sequoyah (2) 2017 Cooper (1) ANO (4) Clinton (2) Pilgrim (4)

Davis-Besse (1)* Pilgrim (4) Pilgrim (4)

Browns Ferry (2)

Clinton (2)

Columbia (2)

Salem (2)

Wolf Creek (2)

ANO (4)

Pilgrim (4) 2018 Grand Gulf (2) Grand Gulf (2) 2019 ANO (1) Watts Bar (2)

Browns Ferry (1)*

Catawba (1)

(#) indicates position in the ROP Action Matrix

• indicates plant moved to higher Action Matrix column within the following year 3

Figure 5: Number of Plants with Themes in Pre-2015 and 2015+ CCI Programs

2. Review of program assessment of plants that were performance outliers In Section 1 the team looked at overall CCI program data. In this section, the team identified performance outliers in the 2015+ time period and reviewed how the 2015+ CCI program assessed those plants. Performance outliers were determined first by looking for the plants with the most greater-than-Green (GTG) issues and the most time outside Column 1. No plant transitioned beyond Column 2 in the 2015-2019 timeframe but two plants were identified as performance outliers. Both plants, referred to as Plant A and Plant B, transitioned to Column 2 and remained there for an extended period of time based on multiple GTG assessment inputs.

Both plants were also sustained outliers in terms of the number of cross-cutting aspects received. The team also identified five plants, referred to as Plants C through G, that met criteria for Column 4 or beyond in the Action Matrix since implementation of the post-Davis-Besse CCI program in 2006 and reviewed how the 2015+ CCI program would have assessed those plants.

Figures 6 and 7 show that Plants A and B were sustained outliers in terms of the number of HU cross-cutting aspects and incurred numerous GTG assessment inputs. Plant A reached the HU theme threshold twice, and almost four consecutive times, but never enough to meet CCI criteria. Plant B reached the HU theme threshold once after numerous GTG assessment inputs.

A review of the performance issues at both sites along with discussion with NRC personnel involved with carrying out the oversight process at these two sites suggested that agency staff did have concerns with HU at the time at both sites.

4

Figure 6: Plant A: Timeline of Cross-Cutting Aspect Trends and Performance Issues Figure 7: Plant B: Timeline of Cross-Cutting Aspect Trends and Performance Issues The team also explored how the CCI program would have assessed plants that met criteria for Column 4 of the Action Matrix - an indisputable significant performance decline. Since no plants met this criteria in the 2015+ time period, the team looked back to implementation of the 2006 version of the CCI program and identified five plants that were initially in Column 1 then at some point met criteria for transitioning to Column 4 or beyond. These plants met Column 4 criteria less than ten years ago, which means the dataset is relatively recent. While there have 5

been ROP implementation changes in recent years, such as clarifications to more than minor screening criteria, changes to the definitions of Column 3 and 4, and SDP guidance changes, the net result, whether direct or indirect, has been a reduction in documented findings of all colors. Since more findings were being documented at the time these five plants reached Column 4 criteria it would have been easier to reach the 2015+ theme thresholds and CCI criteria at that time than in recent years. The team concluded that the assessment of these plants is a realistic, perhaps even optimistic, reflection of how the current CCI program would assess a plant in a similar situation today. The ROP implementation changes make the review of the 2015+ plants more relevant than it might appear on the surface since, even though they only reached Column 2, Plant A would have transitioned to Column 3 with the same set of circumstances in the pre-2015 timeframe.

Plants C through G (Figures 8 through 12) are the five plants that met Column 4 criteria. In most cases the plant was a sustained outlier in the number of HU (Plants C, D, and E) or PI&R (Plants D and F) cross-cutting aspects ahead of the performance decline. In some cases (Plants F and G), review of inspection reports and lessons learned documents revealed ROP implementation issues that unnecessarily limited the data available to the CCI program. In the case of Plant E, the number of bins and inability to use the Other cross-cutting aspects for baseline inspection results contributed to the failure of the 2015+ program to identify the plant before performance declined.

Figure 8: Plant C: Timeline of HU Cross-Cutting Aspects and Plant Performance Issues The 95003 inspection report for Plant C (Agencywide Documents Access and Management System (ADAMS) Accession No. ML16161B279) discussed that the licensee identified many areas where human performance did not meet industry standards. The licensees evaluation of one Yellow finding found that five of the seven safety culture weaknesses identified were in the HU cross-cutting area and the evaluation of the second Yellow finding found that four of the six safety culture weaknesses identified were in the HU cross-cutting area. Further, the licensee determined that the most significant causes for declining performance were ineffective change 6

management with respect to resource reductions, and leadership behaviors that were not commensurate with a strong safety culture, which falls into the HU cross-cutting area.

Thus, despite the plant being a sustained outlier in HU cross-cutting aspects and the 95003 inspection confirming the licensees determination that aspects of the HU cross-cutting area made up the bulk of the safety culture contributions to the performance decline, the 2015+ HU cross-cutting theme thresholds or CCI criteria would not have been met in the lead-up to, during, or after the plant was in Column 4.

Figure 9: Plant D: Timeline of Cross-Cutting Aspect Trends and Performance Issues In 2012 Plant D met the Column 4 criteria when a 95002 inspection determined the performance issues could not be closed, though an ROP deviation was issued to keep the plant in Column 3.

The initial 95002 inspection report (ADAMS Accession No. ML12363A137) noted in section 4OA4.02.05.b that the licensee determined that weaknesses in management oversight of supplemental workers, decision making, work practices, and the CAP [corrective action program] were the most prevalent safety culture attributes. Three of these four attributes fall under the HU cross-cutting area.

Thus, despite the plant being a sustained outlier in HU cross-cutting aspects and the 95002 inspection concluding that most of the safety culture contributors to the performance decline were in the HU cross-cutting area, the 2015+ HU CCI criteria were never met before, during, or after the performance decline. The 2015+ HU cross-cutting theme threshold would have been met once, roughly 1.5 years ahead of the performance decline and would have nearly been met a second consecutive time. This would have been a significant missed opportunity for the 2015+ CCI program to identify concerns and encourage actions far enough ahead of the performance decline to have potentially successfully intervened.

7

Figure 10: Plant E Timeline of Cross-Cutting Aspect Trends and Performance Issues The lessons learned report for the IMC 0350 process at Plant E (ADAMS Accession No. ML14128A376) noted in Lesson Learned No. 3 that there were 103 findings identified from 2007-2011 but no CCIs were identified as the findings were spread across several cross-cutting aspects. Thus, despite a very high number of inspection findings and a number of HU cross-cutting aspects that represented outlier performance, 2015+ cross-cutting theme or CCI criteria would not have been met until after the plant was already in the IMC 0350 process.

Figure 11: Plant F Timeline of Cross-Cutting Aspect Trends and Performance Issues The lessons learned report (ADAMS Accession No. ML13270A473) for Plant F noted that the two 95002 inspections conducted prior to the plant reaching Column 4 acknowledged the licensee root causes had significant ties to safety culture, specifically in the areas of decision making, resources, work practices, work control, human performance and the CAP. The 8

lessons learned report discussed other aspects of ROP implementation that may have reduced the amount of data available to the CCI program.

Thus, despite being a sustained outlier in the number of PI&R cross-cutting aspects, and the 95003 lessons learned report documenting licensee root causes in both HU and PI&R, the 2015+ program thresholds would not have been met before, during, or after the plant was in Column 4.

Figure 12: Plant G Timeline of Cross-Cutting Aspect Trends and Performance Issues The 95003 inspection report for Plant G noted that the NRC team concluded that [the utilitys]

identification of nuclear safety culture as a fundamental problem was appropriate. More specifically, the licensee identified that a significant contributor to declining performance at the station was the failure of leaders to consistently demonstrate a commitment to emphasize nuclear safety over competing goals. One reason why the number of cross-cutting aspects did not identify a concern in the leadup to performance decline may be that the safety culture issues identified as a significant contributor to declining performance were divided between aspects in the HU cross-cutting area and the other cross-cutting aspects. The other aspects, defined and discussed in IMC 0310, cannot be used in the baseline inspection program under current program guidance; they are reserved only for supplemental inspections.

The team could not definitively conclude how many other aspects may have been assigned to inspection findings had they been available for use and thus could not definitively conclude how the 2015+ CCI program would have assessed Plant G had the full suite of cross-cutting aspects been available for use.

In summary, the team performed detailed review of seven plants that were performance outliers.

Review of inspection reports or lessons learned documents showed that safety culture concerns were present in all five Column 4 criteria plants and discussion with staff indicated that safety culture concerns were present for the 2015+ outlier plants as well. The 2015+ CCI program 9

would not have opened any CCIs before the performance decline occurred and in only one case would a 2015+ theme threshold have been met before the performance decline.

3. Review of the relationship between cross-cutting insights and licensee performance The next question the team had was whether the CCI program adds value to the operating reactor assessment program or whether the premise of the CCI program as a forward-looking component of the ROP can be achieved. The team used the risk ratio statistic, which quantifies the strength of association between two events, to explore the association between cross-cutting themes and licensee performance and determine whether accumulating increasing numbers of cross-cutting aspects is statistically related to declines in performance.

Performance decline was determined based on Action Matrix movement and the occurrence of precursors as determined by the Accident Sequence Precursor (ASP) program. Risk ratios are often used in epidemiological studies and low-frequency accident research because they are more appropriate to measure relationships when using count data than other statistics such as the pearson correlation coefficient. These statistical tests were not performed in previous evaluations of the CCI program and therefore the predictive value of the CCI program was not established statistically.

The risk ratio represents the relative risk of moving in the Action Matrix in the following year when exceeding a certain number of cross-cutting aspects. The team examined plants in Column 1 of the Action Matrix from 2006 to 2018 and calculated the risk of that plant moving from Column 1 to a higher column in the Action Matrix within the following year, given that the plant exceeded a certain number of cross-cutting aspects. Various thresholds for HU and PI&R cross-cutting aspects were used to test the sensitivity of the results.

Plants that accumulated enough cross-cutting aspects to reach a cross-cutting theme threshold were at least twice as likely to move to the right in the Action Matrix within the following year compared to plants that did not reach the cross-cutting theme thresholds. This association existed for both the pre-2015 and 2015+ cross-cutting theme thresholds. The team performed a similar data review using ASP precursor events rather than Action Matrix movement. A very similar statistically significant relationship was found in which plants that reach cross-cutting theme thresholds were at least 70 percent more likely to experience a precursor event.

Figure 15 compares the Action Matrix movement risk ratio statistic for the pre-2015 and 2015+

CCI program iterations over two time periods, 2007-2013 and 2014-2018. In both time periods the 2015+ cross-cutting theme thresholds prove more predictive of Action Matrix movement from Column 1 than the pre-2015 program. However, in the 2014-2018 time period only three Column 1 plants have exceeded the cross-cutting theme threshold. This presented two concerns; first, the sample size was too small to draw a meaningful conclusion and second, whether the theme threshold was too high to capture some plants in which the agency should be formally concerned about performance in a cross-cutting area.

10

Figure 13: Action Matrix Movement for Plants With and Without Cross-Cutting Themes Figure 14: Precursor Events for Plants With and Without Cross-Cutting Themes Figure 15: Risk Ratio Statistic for Action Matrix Movement and Cross-Cutting Themes 11

To determine whether the stronger predictive relationship also resulted in cross-cutting concerns being missed, the team binned the instances in which a cross-cutting theme threshold was reached into categories:

True Positive: Plant reached cross-cutting theme threshold and did move in the Action Matrix within the following year False Positive: Plant reached cross-cutting theme threshold but did not end up moving in the Action Matrix within the following year Miss/False Negative: Action Matrix movement occurred but no cross-cutting theme had been reached prior Figure 16: Assessment of Cross-Cutting Theme Occurrences Figure 16 shows the results of this binning, which revealed a strong statistical association. The team recognized that not all greater than Green findings are preceded by concerns in a cross-cutting area and, likewise, not all cross-cutting area concerns necessarily result in a GTG finding or PI. Furthermore, not all ASP precursors are even associated with performance deficiencies. As discussed above though, all White inputs to the Action Matrix represent some deviation from typical performance, and all precursors represent events with a risk significance roughly equivalent to at least a White finding. The team explored events resulting in the most significant Action Matrix movement in more detail as discussed above. For this larger subset of events necessary to extract meaningful analysis from the data, the team does not believe the extensive level of effort that would be necessary to explore the details of each possible cross-cutting concern and GTG finding, PI, or precursor is worth the limited return. Those inputs that were not tied to CCIs are captured by the true negatives in the graphs above and accounted for in the statistical analysis. It is also worth noting that findings, PIs, and precursors that may initially not appear to be related to CCIs, may later be seen differently when more information becomes available. 1 It is also possible that instances considered misses or false negatives may actually have been a program success in which actions were in fact encouraged and taken as trends began appearing but before program thresholds were reached. It is difficult to extract 1

Licensee Event Report 05000293-2013-003, (Pilgrim), Loss of Off-Site Power Events Due to Winter Storm Nemo, April 8, 2013, ADAMS ML13114A293, was determined to be a precursor by the ASP program with a CCDP of 8E-5, but the inspection program did not identify any performance deficiencies related with the event at the time. Following two additional loss of offsite power events, two White PIs, and then another winter storm event with similar risk significance, it became evident that there were in fact underlying CCIs that had contributed to the event and its risk significance.

12

either of these instances from the dataset, but their off-setting contributions would indicate that taken together the overall impact on the statistical analysis is not significant.

A key concern raised by stakeholders and shared by staff was minimizing, to the extent practical, the occurrences of false positives while not undermining the program by increasing false negatives. This was also one of the goals of the CCI program changes implemented in 2015. The information shown in Figure 16 reveals that the changes implemented in 2015 did essentially eliminate false positives. The one false positive identified using 2015+ theme thresholds in the 2014-2018 timeframe was a plant that had just recently transitioned out of the IMC 0350 process that, while technically a false positive by the teams definitions, was likely primarily driven by the extensive inspection effort of the IMC 0350 process. Even when applying the 2015+ thresholds to pre-2014 data a clear drop in the number of false positives is seen. By that measure, the changes implemented in 2015 accomplished the objective of reducing false positives.

However, the team was concerned by the expansion in the number of potential misses (false negatives). The pre-2015 thresholds applied to data up through 2013 showed that 57 percent (66 out of 116) of the data represented a possible miss or false negative. The 2015+ thresholds applied to the same set of data increased the number of possible misses or false negatives to 80 percent. Further, applying the 2015+ thresholds to more recent data (2014-2018) resulted in 86 percent possible misses or false negatives. In other words, 86 percent of Action Matrix movement out of Column 1 was not preceded by a cross-cutting theme threshold being reached. Even though it is expected that some GTG assessment inputs would not involve cross-cutting concerns, the magnitude of this result, when combined with the detailed review of plants that exhibited performance declines showing that CCIs would not have been opened, validates concerns about the CCI program potentially failing to identify plants for which the agency should have concerns in cross-cutting areas.

4. Data review conclusions The team came to the following conclusions based on the data reviewed:

• There was a statistically significant predictive association between accumulating cross-cutting aspects and future performance decline, supporting the premise of the CCI program.

• Whether looking at the pre-2015 or 2015+ CCI programs, plants that reached cross-cutting aspect theme thresholds were significantly more likely than plants without themes to exhibit declining performance, as measured by Action Matrix movement out of Column 1 or occurrence of a precursor event within the next year.

• Based on historical data review, the 2015+ CCI program is less susceptible to false positives or data scatter than the pre-2015 program.

• A review of the five plants that met criteria for entering Column 4 since 2006 revealed that the 2015+ program would not have identified a single CCI at any of the plants until performance had already declined.

13

APPENDIX B: Stakeholder Views of the Cross-Cutting Issues Program As part of the effectiveness review, the team actively sought views on the cross-cutting issues (CCI) program from internal and external stakeholders. Opportunities for the public to share their views on the CCI program were provided at Reactor Oversight Process (ROP) public meetings on August 28 and September 25, 2019. A Federal Register Notice for public comment on the ROP Enhancement initiative also resulted in public comments associated with the CCI program. Industry comments were provided at the August 28 and September 25, 2019 ROP public meetings and are generally encompassed by a slide presentation discussed at the September 25 meeting (Agencywide Documents Access and Management System (ADAMS)

Accession No. ML19266A601). Internal views were gathered from several internal surveys and feedback activities conducted in 2018 and 2019, such as applicable views shared on risk informed decision-making and exchanging of views on the future of the agency and its activities.

A list of specific comments received from all sources is available in ADAMS at Accession No. ML20239A837. A high level summary of the comments received from various stakeholder groups is included below. The staff considered all comments in its review but did not specifically disposition or respond to each comment.

Public Views of the CCI Program: Most comments received from members of the public were critical of the current format of the CCI program. Program threshold were increased without an increase in consequence once thresholds are reached, thresholds are reached significantly less than previously, and there seems to be a reluctance to open CCIs. A couple comments were also received questioning the value in even having a CCI program.

Industry Views of the CCI Program: Industry noted that the elevation of threshold in 2015 have allowed for licensees to see and respond to developing trends before thresholds are reached, which along with improved industry performance is why thresholds have been reached less often. Industry also believes the three assessment period delay before a CCI is opened allows for time for corrective actions to be effective, which would obviate the need for a CCI to be opened. The program would benefit from pre-established CCI closure criteria. Despite these comments, however, the industry overall believes the CCI program does not add unique value to the agencys assessment program and could be eliminated completely.

Internal Agency Views of the CCI Program: Internal feedback offered a wide range of views on whether to improve the CCI program or eliminate the CCI program, but the common thread in the vast majority of internal feedback was dissatisfaction with the current CCI program.

Current thresholds are viewed as too high to add any value, the program failed to identify adverse trends in some cases, the three assessment period delay has resulted in a reactive rather than proactive program, and there are too many bins to place findings.

Enclosure 3