ML20239A836

CCI Program Options Considered

ML20239A836
Person / Time
Issue date:	09/21/2020
From:	Garmon A NRC/NRR/DRO/IRAB
To:
Alex Garmoe, NRR/DRO/IRAB, 415-3814
Shared Package
ML20239A806	List: ML20239A834 ML20239A835 ML20239A836 ML20239A837
References
Download: ML20239A836 (12)

Text

1 2019 Cross-Cutting Issues Program Effectiveness Review Options Considered This document forms a portion of the complete package of information surrounding the 2019 Cross-Cutting Issues (CCI) Program effectiveness review. The effectiveness review report if available in the Agencywide Documents Access and Management System (ADAMS) as Accession No. ML20239A835 and the package of all information in available in ADAMS as Accession No. ML20239A806.

On several occasions before, during, and toward the conclusion of the effectiveness review effort, the team solicited views on the CCI program and any suggested revisions from internal and external stakeholders. Stakeholder views are discussed in ML20239A837. Options considered by the team ranged from maintaining the status quo, eliminating the CCI program altogether, and a variety of potential program changes. In evaluating possible improvements to the CCI program the team considered the information reviewed, conclusions reached, and the Principles of Good Regulation.

This document includes the teams assessment of each option and identifies those that ultimately became recommendations.

1 Overall Concept of the CCI Program

a. Completely eliminate the CCI program

b. Status quo - maintain the same program

c. Incorporate CCI insights into significance determination process (SDP) evaluations for affected plants

d. Consider additional sources of data beyond just more than minor NRC-identified or self-revealed inspection findings

a. Completely eliminate the CCI program One of the most fundamental questions the team asked itself during the effectiveness review was whether the CCI program was needed at all. The team approached this question by considering the program objective and history, how it fits into the overall assessment program, and whether data is available to show if the program is adding value to the assessment program or could add value with adjustments. In a performance-based (i.e. retrospective) Reactor Oversight Process (ROP), having some forward-looking, anticipatory component of the process is a worthy concept validated by the circumstances surrounding the Davis-Besse vessel head degradation event, in which the agencys assessment process viewed plant performance through all Green assessment inputs, but missed underlying cultural concerns that ultimately drove a transition directly from Column 1 to the Inspection Manual Chapter (IMC) 0350 process. While the team found the current implementation of the CCI program to be less responsive in identifying cross-cutting concerns than prior program iterations, risk ratio data did show a statistically significant predictive association between increasing cross-cutting aspects and future declines in performance, which validates the premise of the CCI program.

Independence Efficiency Clarity Reliability Openness

2 The team does not recommend this option.

b. Status quo - maintain the same program The team outlined in this report various reasons why the current CCI program has become less responsive and may not be sufficiently encouraging action. The team does not believe that, absent at least some of the changes identified in this report, the CCI program would be any more effective at implementing the program objective.

The team does not recommend this option.

c. Incorporate CCI insights into SDP evaluations for affected plants The team was chartered to review the case file for Differing Professional Opinion (DPO)

DPO-2017-009 (Agencywide Documents Access and Management System (ADAMS)

Accession No. ML19043A950). The DPO concerns were associated with when and how to use qualitative considerations, including programmatic insights, in the significance evaluation of inspection findings. The CCI program is one means by which programmatic-type concerns may be revealed. The team concluded that this is not within the scope of its effectiveness review of the CCI program since any changes to address the concern would likely come via SDP changes rather than CCI program changes. The team will submit an ROP Feedback Form transmitting the issues raised in the DPO to the SDP program owners. This effectiveness review team will support the ROP feedback form review as needed.

The team is forwarding this option via the ROP feedback process since it is outside the effectiveness review scope.

d. Consider additional sources of data beyond just more than minor NRC-id or self-revealed inspection findings A data trending program, which the CCI program ultimately is, relies on the presence of sufficient data in order to provide effective trending. For the CCI program the data is inspection findings, either NRC identified or self-revealed. There has been a notable decline in inspection findings which, while it does not invalidate the concept of the CCI program, could present challenges in how the program accomplishes trending. From 2006-2014 the agency issued roughly 13-14 findings per site per year. Starting in 2015 a steady decline began and by 2019 the agency issued roughly 6 findings per site per year.

Considering the current CCI program thresholds and the recent decreasing trend in findings, a situation in which there is not enough data available to provide a meaningful assessment under the current program structure may soon arrive, particularly if recommendations made in this report are not implemented. Team recommendations in other areas could mitigate this concern if implemented. Otherwise, it would be prudent for the agency to begin consideration of other potential sources of data should it find at some point that it is not possible to effectively monitor cultural attributes given the reduced dataset of findings at some point.

The team does not recommend program changes based on this option at this time.

However, it would be prudent to begin consideration of possible additional sources of data for cross-cutting insights that could be implemented in the future, if determined to be necessary.

3 2 Monitoring of Findings for Cross-Cutting Insights

a. Monitor findings using only the three cross-cutting areas

b. Monitor findings using the nine safety culture traits

c. Trend findings over a longer rolling period of time

d. Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached

e. Assess for CCIs quarterly rather than every 6 months

f.

Eliminate chilling effect letter and discrimination enforcement criteria for opening a SCWE CCI; treat SCWE comparably to the HU and PI&R cross-cutting areas

g. Eliminate the aspect-specific threshold for opening a CCI; use just backstop values The team thoroughly considered options to more efficiently and effectively monitor and trend findings for cross-cutting insights. Some of the challenges with the current method of using three cross-cutting areas subdivided into 23 cross-cutting aspects, with additional aspects not included in the three cross-cutting areas, include:

There is overlap between many of the aspects and in many cases more than one aspect could reasonably apply to a finding In cases where multiple cross-cutting aspects apply, additional resources are often spent debating internally and with licensees on the most appropriate cross-cutting aspect to assign Implementation of the definition of a cross-cutting aspect as the proximate cause of a performance deficiency has at times steered inspectors away from the most applicable cross-cutting aspect since it may be viewed as too close to the performance deficiency The overlap between aspects, which often results in multiple potentially applicable aspects, and implementation of the definition of a cross-cutting aspect could lead to questions about the credibility or reliability of a trend based on these overlapping aspects Assigning aspects at such a detailed level before the licensee may have completed causal evaluations could result in a decision based on incomplete information The combination of these factors could result in a licensees causal evaluation on a cross-cutting theme finding different causal factors than what the NRC documented The use of 23 aspects means finding an aspect-specific threshold that is both sensitive enough but not vulnerable to false positives and data scatter does not appear to be reasonably achievable There is a population of aspects only available for use in supplemental inspections. This serves no useful purpose since these aspects would never accumulate with insights from other inspections to reveal a trend The team believes current trending for a CCI using 23 cross-cutting aspects is complex, inefficient, and the overlap in existing cross-cutting aspects and implementation of the cross-cutting aspect definition raises questions about the reliability of program conclusions and often results in the expenditure of additional resources when multiple cross-cutting aspects may apply to a finding. The team identified two practical solutions to address the many issues seen with using the existing cross-cutting aspects and program structure: (1) eliminate trending via cross-cutting aspects and use only the three cross-cutting areas, or (2) eliminate trending via cross-cutting aspects and use the nine safety-culture traits. The assignment of cross-cutting aspects to inspection findings could continue for information purposes even if CCI trending is done at the area or trait level.

4

a. Monitor findings using only the three cross-cutting areas The team first considered this option for simplification of the monitoring for cross-cutting insights as it is the simplest option to implement. This approach would eliminate the aspect-specific threshold and use only the backstop thresholds (option 2.g). Whether cross-cutting aspects or simply cross-cutting areas are assigned to inspection findings, the level of effort required to assign the proper characterization to a finding would be expected to be reduced. The overlap between the 23 cross-cutting aspects, if still used to provide granularity, would likely not be the subject of much debate when any human performance or PI&R aspect rolls up and is counted as part of the overall cross-cutting area total. This option also provides greater flexibility in ensuring CCI thresholds balance proper sensitivity with the accumulation of sufficient data to reveal a real trend free from data scatter or false positives. Finally, the program already includes at least some trending to the cross-cutting areas through use of the HU and PI&R backstop thresholds.

However, this option would not address the Other cross-cutting aspects that are not part of the three cross-cutting areas and per program guidance are only available to be used in supplemental inspections. This option would also result in cross-cutting trends that may consist of unrelated data points that may not point to a particular concern given the broad range of issues that each of only three areas would need to cover. This would raise questions about the validity or value of trends identified by the program.

The team does not recommend this option and instead recommends option 2.b.

b. Monitor findings using the nine safety culture traits The team also considered this option for simplification of the monitoring for cross-cutting insights. While this appears to be a larger departure from the existing programs use of the three cross-cutting areas with 23 aspects, the framework for transitioning to this approach is already in place. Exhibit 1 of IMC 0310 contains a cross-reference between the existing 23 cross-cutting aspects and the nine safety culture traits. The Other cross-cutting aspects are also mapped to the nine traits; meaning these would be available for everyday use, making them a relevant portion of the CCI program. This cross-reference would allow already assigned cross-cutting aspects to be easily mapped to the traits, which would allow for an immediate assessment of insights over time upon transition to the traits. Additionally, the cross-reference reveals that the nine safety culture traits cover any activity currently covered by the three cross-cutting areas, and even more.

Assignment of one of the nine traits to a finding rather than one of the 23 cross-cutting aspects would be more efficient given that there is much less potential overlap between the traits. Using the traits, similar to using the three cross-cutting areas, would allow for CCI thresholds that balance proper sensitivity with the accumulation of sufficient data to reveal a real trend free from data scatter or false positives. Finally, the safety culture traits come from the safety culture common language initiative and are also used by licensees in their safety culture monitoring programs. This means that both NRC and licensee programs would be using the same language born from the same initiative. The common language between licensee and NRC programs may also assist with option 1.d if the downstream need to consider possible additional sources of cross-cutting insights arrives.

Moving to the nine safety culture traits for monitoring cross-cutting insights better serves the CCI program objective than moving to the three cross-cutting areas. This approach offers most of the benefits of shifting monitoring to the three cross-cutting areas and also

5 addresses the limitations of using the three cross-cutting areas. Specifically, the other cross-cutting aspects, which would remain unavailable for baseline inspection use with the three areas approach since they do not neatly fit into the three areas, are already mapped into the nine safety culture traits and could be seamlessly brought fully into everyday use within the baseline and supplemental inspection programs under the nine traits approach.

Also, the nine safety culture traits are each more focused than the three cross-cutting areas, so trends identified in a cross-cutting trait would consist of more related data points than under the three areas approach.

The team viewed every option both optimistically and critically and often found pros and cons, but in this case the only downside the team identified is the more significant change to the program. The existing three cross-cutting areas would no longer exist in their current form, but they would be inherently carried forward within the nine safety culture traits. Regardless of the method used for trending cross-cutting insights (aspects, areas, or traits), program definitions should be reviewed to ensure the most applicable and appropriate aspect, area, or trait is assigned, even if it is close to the performance deficiency.

The team recommends this option.

c. Trend findings over a longer rolling period of time The team considered trending findings over a longer time period as a way to address several items. One of the concerns raised by stakeholders was the impact of multiple large inspections over a short period of time, which might result in a short-term burst of findings at one site that may be more spread out at other sites. Monitoring findings over two years, for example, rather than one year would mitigate the peaks and valleys caused by the scheduling of traveling baseline inspection effort. Additionally, this could be an approach, along with option 1.d, to ensure sufficient program data for meaningful trending.

However, the approach would necessitate a change to the CCI program thresholds in order to maintain the thresholds effectively the same. Keeping the thresholds effectively the same for double the monitoring time would also not be as simple as doubling the threshold since the team found that the number of findings at an average site over two years is not double, but rather increases by roughly 1.8 times. If thresholds would need to be re-considered the team believes the effort is better spent on options 2.a or 2.b, which would provide much more program benefit.

The team does not recommend this option.

d. Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached The 2015 changes introduced the requirement that a theme be present for three consecutive assessment periods before a CCI can be opened. This accompanied the removal of the subjective criteria for opening a CCI and was intended to provide the licensee with time to address the theme before the agency opens a CCI. While the team agrees that limiting subjectivity was a positive change, the three consecutive theme requirement has introduced confusion and complexity to the program. The three consecutive assessment period requirement is one of the primary reasons the program has failed to identify plants in a timely manner in which there were concerns in cross-

6 cutting areas in recent years. Further, the intent of the three assessment period delay is duplicative of the impact of elevated program thresholds, which stakeholders noted provide licensees with time to see a developing trend and take action before a theme threshold is reached. Licensees are informed of cross-cutting themes and IMC 0305 prescribes agency review of licensee actions in response to themes, yet since themes are purely data-based and are not opened or closed like CCIs, the agency cannot hold open a theme if the number of findings drops like it could with a CCI, even if there are concerns with licensee actions. Further, the CCI program objective states that CCIs are identified to inform licensees of concerns in cross-cutting areas. The implication under the current program is that the agency cannot formally express concern about a cross-cutting theme until it has been present for three consecutive assessment periods and becomes a CCI, even if staff does in fact have such concerns earlier based on initial follow-up actions to themes.

In addition to confusion and complexity, the data review revealed that the three consecutive theme requirement was a contributor to not identifying a CCI in cases reviewed where plants exhibited Column shifts or outlier performance. The combination of elevated thresholds and the three assessment period waiting time, both introduced in 2015, undermine the portion of the program objective to encourage licensee actions since data has shown that reaching current thresholds for three consecutive assessment periods is practically unachievable even for a significant outlier with problems in cross-cutting areas. Thus, the program has introduced a potential encouragement to wait out the inevitable drop in findings rather than take proactive action.

Data review showed that theme thresholds were seldom reached from 2015 onward, and staff found that in general the plants that did reach theme thresholds would have been appropriately flagged by the CCI program. Given thresholds that allow for reaction to a developing trend before a theme is reached and current program guidance that calls for some staff effort to review actions even when a theme is first identified, the program could be simplified with little impact to licensees and the objective better implemented by removing the three consecutive theme waiting period and instead opening a CCI once the threshold is exceeded, with the same set of graded follow-up actions currently prescribed.

Implementation of this option would also involve implementation of option 4.d. Along with opening a CCI once a threshold is exceeded, the team believes a very limited degree of subjectivity should be allowed for the rare and unique circumstance in which a threshold might be exceeded yet the agency does not believe it is representative of cross-cutting concerns. This would be expected to be a rare exception.

The team recommends this option.

e. Assess for CCIs quarterly rather than every 6 months Since the CCI program objective includes informing of concerns and encouraging licensee actions before significant performance issues emerge, the team considered whether reviewing for cross-cutting trends more frequently than the current six month frequency would improve implementation of the program objective. The most practical alternative would be quarterly since some form of assessment review is done when quarterly performance indicators are submitted and quarterly integrated inspection reports are completed. However, review of cross-cutting issues after the first and third quarters would require additional review and discussion when thresholds are met and additional assessment follow-up letters. The team also believed that the six month review frequency

7 in lieu of something more frequent was not adversely affecting implementation of the program objective.

The team does not recommend this option.

f.

Eliminate chilling effect letter and discrimination enforcement criteria for opening a SCWE CCI; treat SCWE comparably to the HU and PI&R cross-cutting areas As part of the teams assessment of the thresholds and criteria for cross-cutting themes, CCIs, and follow-up actions, the team considered whether the CCI program objective would be better implemented by treating the SCWE cross-cutting area more consistently with the HU and PI&R cross-cutting areas. Existing SCWE theme and CCI criteria are a chilling effect letter, escalated enforcement involving discrimination, or a single finding with a SCWE cross-cutting aspect that is non-isolated (meaning more than one individual is impacted, as discussed in IMC 0305 section 14.01). The follow-up of a chilling effect letter or escalated enforcement with discrimination would involve similar actions as those prescribed by the CCI program. The team also speculated, though could not verify, that the threshold of one SCWE cross-cutting aspect may impact the willingness of inspectors to assign that cross-cutting aspect or managers to approve it.

Nevertheless, the SCWE cross-cutting area appears to reasonably balance a lack of false positives with the ability to identify plants in which staff has SCWE concerns. Taken in isolation, the team does not believe the criteria for a SCWE CCI need to change.

However, option 2.d to eliminate the three assessment period wait before opening a CCI (which is actually a four assessment period wait for SCWE) could be implemented for the SCWE cross-cutting area as well without implementation of this option. Additionally, it would be prudent to review the SCWE criteria should option 2.b be implemented since SCWE is one of the nine safety culture traits.

The team does not recommend this option.

g. Eliminate the aspect-specific threshold for opening a CCI; use just backstop values The team initially reviewed this option on its own but determined that it is inherently incorporated into options 2.a and 2.b.

The team recommends this option as part of option 2.a or 2.b.

3 Criteria and Threshold for Opening a Cross-Cutting Issue

a. Establish a variable threshold that would adjust based on changes in the overall number of findings being issued

b. Return to the thresholds established under the 2006 program (4 findings per aspect)

c. Use the 2014 methodology with current data to establish new thresholds

d. Provide more detailed discussion of CCI insights and decisions in assessment letters

a. Establish a variable threshold that would adjust based on changes in the overall number of findings being issued The team received feedback that a variable threshold might be a reasonable option to consider given that the CCI program has gone through two substantial iterations (2006 and 2015) with thresholds that were initially too sensitive then overcorrected. From a

8 purely data-driven perspective, a variable threshold makes sense. It would automatically adjust to changes in the number of findings being issued across the board based on factors such as changes in ROP implementation, for example. Thus, program sensitivity would remain consistent and the need for occasional step changes in the program would be diminished.

However, a variable threshold could also be perceived as chasing performance. If the number of findings were to undergo a sustained reduction due to improved performance, for example, having a threshold that automatically adjusts downward could provide the appearance of punishing improved performance. Additionally, the ROP is a stable regulatory regime that remains largely consistent year to year; licensees know the thresholds and standards to which they are regulated, and at which increased oversight occurs. The team views this as a program strength. Implementing a variable CCI threshold would run counter to the philosophy of stable regulation and a licensee would not know CCI thresholds in advance, which would challenge their own monitoring of cross-cutting insights.

The team does not recommend this option.

b. Return to the thresholds established under the 2006 program (4 findings per aspect)

Stakeholder feedback provided in advance of the effectiveness review indicated that a strength of the current program is that thresholds are high enough to allow for licensees to see real trends developing in a cross-cutting aspect or cross-cutting area and evaluate and take action before thresholds are exceeded. This feedback also included comments that the prior program iteration, with a threshold of 4 findings per aspect, was vulnerable to false positives and did not allow for licensees to conduct a meaningful evaluation and take action before the threshold was met since action would need to be taken on only two or three data points. While the team sought but was not provided with, and did not identify, plant-specific or general industry evidence that this proactive behavior occurs in an effective manner, such behavior would be consistent with the CCI program objective. The team agrees that any CCI threshold should reveal a real developing trend before the threshold is met and not be vulnerable to data scatter or false positives. Going back to the prior program thresholds would not support what the team and stakeholders view as a reasonable threshold.

The team does not recommend this option.

c. Use the 2014 methodology with current data to establish new thresholds Given that the thresholds developed in 2015 were developed based on a methodology using data from 2009-2013, it seems reasonable from a purely data perspective to periodically revisit the thresholds using the same methodology with more current data.

Unlike the core damage frequency and large early release frequency threshold for quantitatively assessed inspection findings, there is no agency policy that states what the CCI threshold must be. As such, there is no reason to believe the thresholds set in 2015 are any better than the thresholds they replaced, and in fact the teams data review shows that neither are effective. The teams review of Column 4 plants, in particular, showed that current thresholds would not have identified any of the plants before performance significantly declined.

9 Should either option 2.a or 2.b be implemented, this option should be implemented as well. Implementation of option 2.b (traits) would necessitate the development of thresholds for the traits, but even implementation of option 2.a (three areas) should be accompanied by a review of the thresholds.

The team recommends this option in conjunction with option 2.a or 2.b.

d. Provide more detailed discussion of CCI insights and decisions in assessment letters The team was provided with feedback from an NRC senior manager involved in the agencys assessment process that while assessment letters technically include discussion of cross-cutting themes and issues, this discussion is at times limited and does not always reflect the agencys complete assessment of cross-cutting concerns. The teams own review of assessment letters revealed at times just a cursory mention of cross-cutting themes, as noted in Figure 2 of the CCI Effectiveness Review Report, for example. More detailed discussion in assessment letters of cross-cutting themes and issues, whether the theme or issue represents agency concern in a cross-cutting area, what the licensee has done to date to address the issue, and agency actions planned and taken would enhance openness with the public and help licensees take actions to address specific agency concerns.

The team recommends this option.

4 CCI Follow-Up

a. Timing of follow-up actions should be based on licensee readiness

b. Establish standardized CCI follow-up inspection guidance with objectives

c. CCI follow-up could be scoped into an existing planned supplemental inspection

d. CCIs should not be able to be closed solely based on a reduction in inspection findings

a. Timing of follow-up actions should be based on licensee readiness Current CCI program guidance directs graded agency actions in response to cross-cutting themes or issues that escalate based on the agencys assessment reviews every six months. This timeline may not correspond with the licensees evaluation and implementation schedule for corrective actions. The team believes the CCI program would better encourage licensees to develop and implement meaningful corrective actions if the agencys review of those actions was conducted based on licensee readiness rather than on automatic six-month intervals. A time backstop should be implemented to ensure each follow-up action is completed reasonably promptly even if the licensee has not indicated readiness. Escalation of agency follow-up actions would occur once review of licensee actions determined that concerns had not been effectively addressed. Under this approach agency review of cross-cutting concerns would escalate based on whether licensee actions were effective rather than based on the time an issue is open.

The team recommends this option.

b. Establish standardized CCI follow-up inspection guidance with objectives The team received stakeholder feedback recommending the establishment of standardized objectives for follow-up and closure of CCIs. Current program guidance directs the regional office to establish criteria for closing a CCI and clearly describe that

10 criteria in an assessment letter. This may lead to inconsistent criteria for the follow-up and closure of CCIs. Development of standard objectives for the follow-up and closure of CCIs would ensure consistent follow-up and closure of CCIs and enhance the efficiency of resources expended by both the agency and licensees. The standard follow-up objectives could be contained in a CCI follow-up inspection module within the baseline inspection program that also prescribes the graded follow-up actions to take in response to cross-cutting themes and CCIs.

The team recommends this option.

c. CCI follow-up could be scoped into an existing planned supplemental inspection When follow-up actions are planned in response to a CCI or cross-cutting theme and a supplemental inspection is already or will be scheduled at an appropriate timeframe to also review the CCI, the CCI follow-up should be scoped into the supplemental inspection with a corresponding resource adjustment to the supplemental inspection. The teams review of supplemental inspection objectives found that follow-up of a CCI is consistent with the objectives. A greater-than-Green input should not be closed until confirmation that causal factors, which could be related to an open CCI, have been addressed.

Scoping CCI follow-up into a supplemental inspection would ensure this potential overlap between a CCI and causal factors is considered. Thus, this is an opportunity to provide follow-up in a more efficient and effective manner.

The team recommends this option.

d. CCIs should not be able to be closed solely based on a reduction in inspection findings Closure of CCIs should only be based on the established follow-up objectives and should not be based solely on a reduction in inspection findings. Should option 2.d not be implemented, guidance should be updated to allow a theme to be held open such that a reduction in findings does not automatically close the theme if the agency believe there are underlying concerns with the theme that remain present.

The team recommends this option.

In summary, the following options are proposed by the team as recommendations:

2.b:

Monitor findings using the nine safety culture traits (with 2.a as a less preferable alternative) 2.d:

Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached 2.g:

Eliminate the aspect-specific threshold for opening a CCI; use just backstop values (as part of 2.a or 2.b) 3.c:

Use the 2014 methodology with current data to establish new thresholds (as part of 2.a or 2.b) 3.d:

Provide more detailed discussion of CCI insights and decisions in assessment letters 4.a:

Timing of follow-up actions should be based on licensee readiness 4.b:

Establish a CCI follow-up inspection procedure with follow-up objectives 4.c:

CCI follow-up could be scoped into an existing planned supplemental inspection 4.d:

CCIs should not be able to be closed solely based on a reduction in inspection findings

11 The team recognized that the recommendations would require varying amounts of time and resources to implement. Thus, the team divided the recommendations into two groups representing those that would require limited time and resources for implementation versus those that would require more substantial time and resources for implementation.

Group 1 recommendations are those that could be implemented with a lower level of resources with a fairly short implementation time. These recommendations would incrementally improve the CCI program, address at least some of the shortcomings of the current program, and make some improvement in the implementation of the program objective.

3.d: Provide more detailed discussion of CCI insights and decisions in assessment letters 4.a: Timing of follow-up actions should be based on licensee readiness 4.b: Establish a standard CCI follow-up inspection procedure with objectives 4.c: CCI follow-up could be scoped into an existing planned supplemental inspection 4.d: CCIs should not be able to be closed solely based on a reduction in inspection findings Group 2 recommendations are intended to significantly improve the CCI program and make implementation of the program objective more efficient in a manner beyond the improvements made by Group 1 recommendations. These recommendations would require a higher resource commitment and more associated staff effort and time as well as stakeholder interaction to be implemented. These recommendations build on the Group 1 recommendations.

2.b: Monitor findings using the nine safety culture traits (2.a is also an acceptable though less preferable approach; 2.g and 3.c are inherently included in this recommendation) 2.d: Remove the requirement that a theme be exceeded for three consecutive periods before a CCI is opened; open a CCI once a threshold is reached The recommendations in Group 1 are viewed by the team as a starting point for program improvements since they require less effort and time to implement. The Group 1 recommendations could be implemented in parallel with the more significant Group 2 recommendations as one suite of improvements, in advance of the Group 2 recommendations as a two-step approach, or on their own with possible future consideration of Group 2 recommendations.

12 Figure 1: Recommendations for Improving Implementation of the CCI Program Objective Group 1 Lower level of effort Smaller changes intended to improve the CCI program but would not address all issues identified More complete documentation in assessment letters Pre-establish follow-up procedure and objectives Timing of follow-up actions consistent with completion of licensee actions Group 2 More significant level of effort Larger changes intended to significantly improve the CCI program Simplify the monitoring of cross-cutting insights of inspection findings Remove the theme waiting period and open a CCI once a threshold is reached; offer limited subjective consideration