ML20239A837
| ML20239A837 | |
| Person / Time | |
|---|---|
| Issue date: | 09/21/2020 |
| From: | Garmon A NRC/NRR/DRO/IRAB |
| To: | |
| Alex Garmoe, NRR/DRO/IRAB, 415-3814 | |
| Shared Package | |
| ML20239A806 | List: |
| References | |
| Download: ML20239A837 (5) | |
Text
1 2019 Cross-Cutting Issues Program Effectiveness Review Stakeholder Views of CCI Program This document forms a portion of the complete package of information surrounding the 2019 Cross-Cutting Issues (CCI) Program effectiveness review. The effectiveness review report is available in the Agencywide Documents Access and Management System (ADAMS) as Accession No. ML20239A835 and the package of all information in available in ADAMS as Accession No. ML20239A806.
Public Comments Received on CCI Program Comment Source The Union of Concerned Scientists expressed interest in why there has been a significant decrease in cross-cutting theme thresholds reached, even when carrying forward the old lower thresholds.
Comments made at the August 28 ROP public meeting. The data referred to is in ADAMS as Accession No. ML19238A042.
Moving to a threshold of six findings per cross-cutting aspect is too high and stronger actions by the agency should be taken if these higher thresholds are met.
Comment made at August 28 ROP public meeting.
If the NRC had an effective and timely PI&R inspection procedure, it would obviate the need for the cross-cutting issues process. The cross-cutting issues process is essentially an operator work-around for valid regulatory insights into corrective action programs at plants.
Comment received in response to FRN NRC-2019-0155 INPO 12-012 and NUREG-2165 are not identical and therefore there is still not a common language.
The cross cutting issue process is arbitrary and should be eliminated. The NRC has no business in regulating human performance and questioning attitude.
Comment received in response to FRN NRC-2019-0155 The inspectors seem reluctant to issue substantive cross cutter. Limited research indicates that the last SCCI was issued in 2014.
Comment received in response to FRN NRC-2019-0155 Industry Comments Received on CCI Program Comment Source More consistent and clear follow-up and closure criteria for CCIs is desired.
Comment at September 25 ROP public meeting Threshold of six findings per aspect allows time for licensees to identify a trend and take action before the threshold is exceeded and is less vulnerable to being tripped by data scatter NEI presentation at September 25 ROP public meeting Three successive periods before a cross-cutting issue is opened allows time for corrective actions to take effect NEI presentation at September 25 ROP public meeting
2 Addition of the backstop allows broad or persistent trends to be identified NEI presentation at September 25 ROP public meeting Reduction in cross-cutting themes and issues since 2015 reflect improved industry performance NEI presentation at September 25 ROP public meeting In 2015, NEI recommended that the NRC eliminate the cross-cutting issues program, saying in part, We see no evidence that the SCCI [Substantive Cross-Cutting Issue]
process is adding unique insights about safety culture or providing unique value as a regulatory tool that justifies continuing it in its present form or in the revised form proposed by the staff. In our view, this remains true today.
We believe the CCI Effectiveness Evaluation team should challenge itself to determine the unique value CCIs contribute as a regulatory tool and to compare that value to the resources expended by NRC and the industry to carry out the CCI program.
NEI letter dated October 7, 2019 If the CCI Effectiveness Evaluation Team recommends lowering the present thresholds, we would be very concerned that this would lead to the same unnecessarily burdensome problems experienced before 2015, diverting licensees attention away from more safety significant matters.
10/719 letter For the current CCI Effectiveness Evaluation to produce meaningful results, the team must specify the purpose of CCIs that is to be the basis for measuring effectiveness.
10/7/19 letter
[W]e believe the ROP offers all the tools the NRC needs to address plants the NRC perceives to be outliers in Column 1, even without the CCI program. If the present CCI Effectiveness Evaluation concludes otherwise, the evaluation team should be expected to show there is a logical and statistically meaningful relationship between CCIs and objective measures of licensee performance.
Such a showing should be considered sine qua non for NRC to continue using CCIs as if they are a predictive indicator of licensee performance.
10/7/19 letter
3 Internal Feedback Received on CCI Program Comment Source 34% of respondents indicated cross-cutting aspects are applied very or somewhat inconsistently in their region.
Written comments associated with this question included:
I believe unintended messages may have been received about the value of documenting a green finding. These may result from changing the cross cutting aspect thresholds to the point where themes are rarely identified cross-cutting aspects are very subjective and/or arbitrary. I see no value in them and I recommend we do away with them all together.
CCAs are so numerous and broadly defined about 3-5 could be made to apply to any given finding.
The cross-cutting aspects have lots of overlap - i.e.
multiple aspects may apply to one performance deficiency.
Given the current thresholds, recommend removing the cross-cutting aspect process entirely or making it more meaningful.
Based on the changes made to how the NRC takes action on excessive numbers of Cross-Cutting Aspects, there seems to be little residual value in this determination.
It's very hard to even define "consistent" application for cross-cutting aspects. Just about every PD can pretty reasonably have multiple cross cutting aspects assigned to it.
The cross-cutting aspects have been rendered essentially worthless as an inspection tool due to the change in number needed to reach a threshold.
2018 Division of Inspection and Regional Support survey on inspection program Cross-cutting issues do not seem to carry the weight as they used to Regional risk-informed decision-making survey response I have not found that Cross Cutting Issue (CCI) program to impact licensee behavior in great measure due to two factors. First, the sample size of any trend is extremely low when using cross cutting issues. Second, the timeliness of any CCI is so far removed from the event that any learning opportunity has long since passed. The result appears to be an aggravating circumstance to the original violation with little impact to the overall licensee behavior. However, NUREG 2165, Safety Culture Common Language, provides better opportunities for the NRC staff to consider safety culture weaknesses and to encourage licensees to take appropriate actions before significant performance degradation occ urs. Recent ratios of minor violations to more than minor have run about 10 to 1 in my inspections.
So instead of 4 to 5 CCI behavioral indicators per year, 40 to 50 behavioral traits are provided. This is a much larger Feedback provided directly to the team
4 sample size and more valid for trending. In addition, the proximate trait can be assessed immediately in most cases.
For plant [x], the licensee never got to the point of a CCI or SCCI, but they came very close (1 away from a Resources SCCI). They were definitely tracking it and were very sensitive to it anytime we had a finding we were screening and presented to them. Its the Senior Residents opinion that the CCI program didnt cause them to take any additional action besides question us and try to propose what they thought the cross-cutting issue should be. They were already aware of their procedural compliance and procedural inadequacy issues and the SRI do not believe the CCI program drove them to take action.
Feedback provided directly to the team For plant [y]:
- 1. The CCI program failed in identifying adverse behavioral trends at [y] CCISCWE issue.
- 2. The [y] CCI SCWE was identified through the high number of allegations received and review of [y]
Transcript Reports.
- 3. The licensee failed to correct the [y] SCWE Theme in timely manner and NRC issued a CCI-SCWE at
[y]
- 4. NRC issued a CAL and Confirmatory Order Feedback provided directly to the team The cross cutting issues program was supposed to provide you those forward looking insights and allow the NRC to engage and shape licensee behavior before it starts to degrade. Identify trends and be proactive. Instead it takes multiple assessment periods and higher and higher bars (and finer and finer divisions) to assign a cross cutting issue and engage the licensees. That is a very reactive process. A risk informed approach would identify safety culture weaknesses and allow us to be proactive and shift inspection resources to those areas to nip problems in the bud.
Possibly our CCA assessment is weak and is not identifying problem plants as currently implemented? Having more CCA bins and higher thresholds CCIs (formally SCCIs) and for MTM is making this assessment ineffective.
Clearly the changes to the CCI program haven't helped us be more responsive to addressing leading indicators of safety culture issues In my particular case, the CCI program worked. I wasn't a big fan when the criteria were changed and never viewed them as a way to be more responsive. I believe it made us less responsive and removed a tool inspectors could use to focus management attention on a plant with declining Comments provided during NRC Futures Jam discussions
5 performance. In it's current state, the CCI program will not help us be more responsive to safety culture issues because the entry criteria are too high.
The program as it exists now has been completely gutted because they have so many bins, need so many more findings to reach a theme criteria, and the assessment periods are now longer so the bar is unreachable in most cases.
CCIs as they are currently implemented produce few results for the amount of time expended. I would recommend removing CCAs from violations and performing a targeted safety culture look using the CCAs on an as needed basis.
I agree the CCI program needs attention. I don't believe it's serving its intended purpose - but I believe that for reasons that are different than most others. I believe the bar is set too high. The backstops for human performance and problem ID, along with the requirement for three consecutive assessment periods, is a bar that would likely not be reached until a plant's performance degrades significantly. By that point, we will likely have already engaged in a higher level of oversight for the involved plant. I also believe that conversations around the CCI program need to remain apart from the discussions of risk, which are appropriately covered in other aspects (i.e., inspection sample selection, SDP, MD8.3, etc.).
Effort applying cross-cutting aspects to findings can be minimized by allowing all applicable aspects to be assigned, then using big data analysis to identify trends.
The number of samples the baseline inspection obtains represents an incredibly small fraction of actual issues at plants. Some fraction of the findings identified in the samples that we take will include a cross-cutting aspect. So, a fraction of a small number of issues (compared to total site issues) are now to be used to predict future licensee performance. We should analyze the sample numbers we draw to determine whether they are adequate to provide confidence in our assessments. If the desired level of confidence is not achieved, look for ways to increase the number of issues we consider.
Feedback provided directly to team