ML19322A014

CCI Effectiveness Review Presentation November 20, 2019 ROP Public Meeting

ML19322A014
Person / Time
Issue date:	11/18/2019
From:	Alex Garmoe NRC/NRR/DRO
To:
Garmoe A
References
Download: ML19322A014 (37)

Text

Cross-Cutting Issues Effectiveness Review Reactor Oversight Process Public Meeting November 20, 2019 Alex Garmoe

Discussion Outline

• Why an Effectiveness Review?

• History of CCI Program

• 2014 Effectiveness Review and Resulting Program Changes

• 2019 Effectiveness Review

• How to measure effectiveness

• The recent environment

• Stakeholder views

• Preliminary conclusions

• Potential recommendations

• Next Steps

Why an Effectiveness Review?

• Significant cross-cutting issues program changes made in 2015 have been given sufficient run-time

• ROP Assessment Process (IMC 0307) includes effectiveness review of program changes

• ROP enhancement included recommendations to change or eliminate the cross-cutting issues program

• An effectiveness review in late 2019 helps inform both efforts

History of CCI program

• Three cross-cutting areas were a part of the ROP since inception

• Human performance, PI&R, SCWE

• Limited guidance and little structure initially

• Davis-Besse head degradation event provided momentum to improve the treatment of safety culture and SCWE in the ROP

• Commission direction to improve cross-cutting issues program

• Result was rolled out in 2006

• Cross-cutting areas subdivided into 9 cross-cutting components and 27 cross-cutting aspects

• Definition, criteria, and follow-up actions established for SCCIs

History of CCI Program

• Starting in 2011, following the Commissions Safety Culture Policy Statement, NRC and industry developed shared set of terms to describe safety culture

• Ten traits of a healthy nuclear safety culture (each with attributes and examples) were developed and described in NUREG-2165 and INPO 12-12

• Common language was rolled into the SCCI program in 2013

• Cross-cutting areas were re-defined based on the common language traits

• Cross-cutting components eliminated, cross-cutting aspects reduced to 23

• IMC 0310 references aspects to the applicable common language traits

2014 Effectiveness Review

• Staff efforts to look at enhancing the ROP in 2013 and the February 2014 ROP Independent Assessment recommended a review of the existing SCCI program

• Effectiveness review completed in April 2014 and concluded:

• Identification of SCCIs effective in communicating concerns to licensee

• Difficult to prove that licensee corrective actions prevented more significant performance issues. Thus, cannot say definitively whether SCCI program is either effective or ineffective.

• SCCIs are not a leading indicator for declining licensee performance

• Resource cost is significant without apparent commensurate increase in safety benefit

2015 SCCI Program Changes

• No changes to existing cross-cutting areas or aspects

• Cross-cutting theme re-defined as 6 findings with a common aspect

• Cross-cutting area backstops introduced and supported by data analysis (97th percentile)

• 20 total findings in human performance area

• 12 total findings in PI&R area

• Substantive label deemed negative and removed

• Subjective criteria for opening SCCI were removed

• CCI no longer opened right away - theme must be present for three assessment periods (four for SCWE)

2019 Effectiveness Review

• Multi-disciplinary team assigned from headquarters and multiple regions

• Alex Garmoe, NRR/DRO (team lead)

• William Schaup, Region III Senior Resident Inspector

• Harry Freeman, Region IV Senior Reactor Inspector

• Stephanie Morrow, RES Human Factors Engineer

• Rebecca Sigmon, NRR/DRO/IOEB

• Nick Hobbs, Region II Resident Inspector

• SES Sponsor: Mark Franke, Region II DRS Director

Measures of Effectiveness The NRC identifies a cross-cutting issue (CCI) to inform the licensee that the NRC has a concern with the licensees performance in the cross-cutting area and to encourage the licensee to take appropriate actions before more significant performance issues emerge

1. Cross-cutting aspects assigned in reliable and efficient manner

2. Cross-cutting themes are monitored at an appropriate frequency

3. CCIs identified to inform licensees of NRC concern

4. CCIs encourage licensee to take appropriate actions ahead of more significant performance issues

• Team also assessed effectiveness of 2015 changes given the bases for the changes and the impact of the changes on the program objective

The Environment Since 2015

• Cross-cutting themes and issues have been declining

The Environment Since 2015

• Average number of findings per site declined as thresholds raised

The Environment Since 2015

• Higher % of findings required per aspect/area to reach thresholds Aspect theme % plotted on left axis, backstop % plotted on right axis

What is Driving the Decline?

• August 15, 2019, Memorandum from S. Walker to H. Nieh, Analysis of Inspection Findings Trend at Nuclear Power Reactors Between 2015 and 2018 (ML19225D281)

• The data analyzed by the staff indicates that the decline in inspection findings can most likely be attributed primarily to implementation of the ROP

• Overall, NRR concluded that there has been an improvement in plant safety over the period (2000-2018) for several areas considered. However, the staff did not identify an appreciable improvement in performance in the period from 2015 to 2018 that could conclusively account for the decrease in inspection findings.

Staff analysis shows Decline in inspection Decline in CCIs not Consider whether CCI decline in findings not findings has simply due to program adjustments simply due to influenced decline in improved industry needed to meet improved industry CCIs performance objectives performance

Summary of Stakeholder Views

• Internal: elevated program thresholds and number of cross-cutting aspects has resulted in a program with limited to no value as currently implemented

• Industry: program is effective as currently implemented since it balances the need to accumulate sufficient data with the need to intervene in time to arrest a developing trend. Industry also expressed a desire for standard closure criteria.

• Public: program changes have reduced the agencys regulatory attention and impact on cross-cutting issues. For example, program changes have made it more difficult to reach cross-cutting thresholds yet the consequences when those thresholds are reached have not increased.

Insights From Spain

• Spain adopted the NRCs SCCI process but maintained monitoring at the component level

• 13 components are used - they include the 4 other components

• Roughly 80% of findings are assigned a cross-cutting component

• SCCI threshold set at 8 findings per component based on analysis of number of findings across the industry

• Identify declining plants but not too many (avoid too many false positives)

• Findings are analyzed for CCIs quarterly using a rolling 1-year timeframe

• CSN found licensees were monitoring to their own lower thresholds (i.e.

program was encouraging licensee action)

• If SCCI open for >1 year it is considered repetitive

• Independent safety culture assessment may be requested

Effectiveness Review Conclusions

• Cross-cutting aspects being assigned at a consistent rate

• Regions are assessing for cross-cutting themes

• IMC 0305 uses assessment cycle when it should use assessment period

• Licensees could be more meaningfully informed of NRC concerns

• There is evidence that licensees are encouraged to take actions in response to approaching or exceeding theme thresholds

Effectiveness Review Conclusions

• The 2015 changes improved the program in several ways

• Reduced the false positives that burdened the old SCCI program

• Reduced resources expended by ensuring focus was only placed on true performance outliers

• Reduced the subjectivity of CCI determinations (mixed resource impact)

• Introduced the backstops

• However, the 2015 changes implemented elevated program thresholds and delayed follow-up action to cross-cutting themes, which when combined with the complexity of the program and the decrease in inspection findings over the past several years has challenged the ability of the program to meet its objective.

Even if Its Effective, Is It Useful?

• Meaningful insights can be gained from CCI program as noted in 95003 lessons learned reports

• Unique value provided by CCI program based on review of Grand Gulf DPO

• Sites with more CCAs more likely to move in Action Matrix Remain in Column 1 of Change to Higher Column in Action Matrix Action Matrix Less than 8 Human 198 46 19%

Performance CCAs 8 or more Human 71 39 35%

Performance CCAs Remain in Column 1 of Change to Higher Column in Action Matrix Action Matrix Less than 6 PI&R 249 72 22%

CCAs 6 or more PI&R 20 13 39%

CCAs

So Where Do We Go With The Program?

• The team is evaluating possible recommendations to improve the program

• Underlying framework remains sound

• Assigning of cross-cutting insights to inspection findings

• Identifying trends, ideally ahead of significant performance issues

• Graded approach to follow-up and closure

• The team sees opportunities for improvement to ensure the program

• Meets the IMC 0305 stated objective

• Is an indicator ahead of significant performance issues

• Is simple and consistent

• Uses resources efficiently

Potential Enhancements to the CCI Program

• Simplify the program and improve efficiency

• Improve openness through more complete documentation of concerns

• Standardize and pre-establish closure criteria

• Review program thresholds to ensure program is responsive when appropriate, is sensitive to false positives, and allows sufficient data to accumulate to identify a meaningful trend

• Treat SCWE similar to other areas or traits

Simplify the Program

• Stakeholder feedback (internal and external) noted that current complexity results in subjectivity

• Overlap between current 23 aspects adversely impacts reliability and credibility of CCI program outcomes

• Negative impact on resources spent debating and selecting the appropriate cross-cutting aspect when multiple could be justified

Simplify the Program

• Option 1: Monitor based on the three cross-cutting areas

• Eliminate the 23 aspects and assign findings as simply HU, PI&R, or SCWE

• The CCI threshold would be based on total number of HU, PI&R, or SCWE findings

• Backstop thresholds should be reviewed for possible adjustment

• Licensees would assess for more detailed insights and causal factors Problem Identification &

Human Performance SCWE Resolution

Simplify the Program

• Option 1: Monitor based on the three cross-cutting areas

• Benefits:

• Within the same framework of the existing program

• Less resource burden assigning cross-cutting insights to findings

• Simple change to implement as program structure remains largely the same

• Challenges:

• Does not reveal a specific trend, and there may not even be a trend

• What to do with the other area?

Simplify the Program

• Option 2: Leverage the Common Language Initiative

• Monitor for cross-cutting insights using the traits of a healthy nuclear safety culture

• No areas or aspects, just the nine applicable traits

• Single threshold for each trait, no backstops Leadership Personal Continuous Effective Safety Questioning Accountability Accountability Learning Comm. Attitude Environment Respectful Work Decision PI&R for Raising Work Processes Making Concerns Environment

Simplify the Program

• Option 2: Leverage the Common Language Initiative

• Benefits:

• Current aspects fully covered by the traits

• Common language between agency and licensees should improve communications about CCIs

• Less overlap means less resources spent assigning traits to findings

• Meaningful trends would be revealed for licensee review and action

• Challenges:

• Larger change to the existing program

Improve Openness and Standardize

• Improve the discussion of CCIs in assessment letters

• Basis for opening a CCI and what the NRC concern is

• Basis for keeping a CCI open

• Basis for closure of a CCI

• Standardize and pre-establish closure criteria

• Response to stakeholder feedback

• Public and licensees would know the criteria for CCI closure ahead of time

• Consistent agency decision-making while assessing CCIs

• Consider follow-up actions based on licensee readiness rather than automatically every 6 months

Review Program Thresholds

• Program thresholds must be appropriate to ensure program can meet its objectives

• Too low: vulnerable to false positives and data-scatter

• Too high: does not identify trends or concerns ahead of performance declines

• Option 1 to monitor via three cross-cutting areas: are current backstops set appropriately?

• Option 2 to monitor via common language traits: what should the trait thresholds be?

Review Program Thresholds: Backstops

• Initial backstops applied looking at 97th percentile based on 2009-2013 data (Gold): 20 for Human Perf. and 12 for PI&R

• Team used the same methodology and looked at three time periods spanning 2015-2018 (Green, Peach, and Purple)

• Team would likely recommend 15 for Human Perf. and 6 for PI&R 2009-2013 (original 2015-2018 2017-2018 2018 baseline)

Human Perf PI&R Human Perf PI&R Human Perf PI&R Human Perf PI&R Average 7.9 3.6 6.1 2.0 5.2 1.7 5.1 1.4

+2 std dev 14.2 5.6 12.9 5.2 12.4 4.5 from mean Median 7 3 5 2 4.5 1 4 1 95% 18 9 15 6 12.4 5 12.2 5 97% 19.5 11 16 6.1 15 6 15 5.5

Review Program Thresholds: Backstops Black - Plants reaching backstops at 2015-2018 97th percentile (thresholds: HU 16, PIR 6)

Blue - Additional plants reaching backstops at 2018 97th percentile (thresholds: HU 15, PIR 5)

• Table of plants 2015 MOC Waterford - HU, PIR 2015 EOC Waterford - PIR 2016 MOC 2016 EOC 2017 MOC 2017 EOC 2018 MOC 2018 EOC River Bend - PIR River Bend - HU River Bend - PIR Diablo Canyon - HU Duane Arnold - PIR Salem - PIR Salem - PIR Grand Gulf - HU Grand Gulf - HU Grand Gulf - PIR Grand Gulf - HU, PIR Grand Gulf - HU, PIR Monticello - PIR Indian Point - PIR Susquehanna - HU Susquehanna - PIR Comanche Peak - PIR Comanche Peak - PIR Clinton - HU Clinton - HU Clinton - HU Watts Bar - HU Perry - HU Callaway - HU Fermi - PIR Wolf Creek - PIR Columbia - HU Columbia - HU Cooper - PIR Cooper - HU, PIR Three Mile Island - PIR Three Mile Island - PIR Millstone - PIR

Review Program Thresholds: Traits

• Team used IMC 0310 cross-references to re-assign all aspects issued since 2015 into the common language traits

• Reviewed 95th and 97th percentiles with and without Column 4 plants and multiple time periods since 2015

• Team initially envisioned maintaining a threshold of 6, though data supports a threshold of 5 CL CO DM LA PA PI QA RC WP w/Col 4 2015-2018 95% 2 0 3 3.95 1 5 4 0 6 97% 3 1 4 4 2 5 4 0 6.77 2017-2018 95% 2 0 3 3 1 4 4 0 5.95 97% 2 1 3.74 4 1.37 5 5 0 6.37 w/out Col 4 2015-2018 95% 2 0 3 4 1.35 4 4 0 6 97% 3 1 4 4 2 5 4 0 6 2017-2018 95% 2 0 3 3.15 1 4 4 0 5 97% 2 1 4 4 1.5 5 5 0 6

Review Program Thresholds: Traits Review Program Thresholds: Traits 2015 MOC 2015 EOC 2016 MOC 2016 EOC 2017 MOC 2017 EOC 2018 MOC 2018 EOC Diablo Canyon - WP Diablo Canyon - WP Comanche Peak - PI Indian Point -WP River Bend - PI River Bend - LA River Bend - PI Salem - PI Salem - WP Turkey Point - LA Waterford - WP Harris - WP Surry - WP Susquehanna - QA Susquehanna - WP Susquehanna - PI Wolf Creek - CL Clinton - LA Clinton - WP Watts Bar - WP, RC Watts Bar - WP Grand Gulf- WP, DM, Grand Gulf - WP Grand Gulf- WP PI Browns Ferry - WP Millstone - WP Perry - WP Wolf Creek - CL Wolf Creek - RC Wolf Creek - LA, RC Cooper - PI Cooper - WP, PI Three Mile Island - PI

Treat SCWE Consistent With Other Traits

• Since 2006 SCWE has been treated differently

• Threshold for an SCCI (old) or theme (current):

• One or more findings with SCWE aspect and impact is not isolated

• Chilling effect letter

• Escalated enforcement with discrimination

• Threshold for SCWE-related traits based on number of findings per trait - same concept and threshold as other traits

• Chilling effect letter and discrimination enforcement have their own process that is generally redundant to the CCI process

• Proposed graded follow-up approach for human performance and PI&R traits would also work for SCWE traits

Potential Enhancements to the CCI Program

• Simplify the program and improve efficiency

• Improve openness through more complete documentation of concerns

• Standardize and pre-establish closure criteria

• Review program thresholds to ensure program is responsive when appropriate, is sensitive to false positives, and allows sufficient data to accumulate to identify a meaningful trend

• Treat SCWE similar to other areas or traits

Potential Enhancements to the CCI Program

• The team believes the potential enhancements under consideration would result in a program that is better able to meet its objectives and meets all Principles of Good Regulation

• Remains INDEPENDENT

• Use independent data sources (findings) and arrive at independent conclusions

• More CLARITY

• Objective criteria for opening CCI and standardized criteria for closing a CCI

• More OPEN

• Publicly available standardized follow-up and closure criteria for CCIs, enhanced description of basis for CCI descriptions in assessment letters

• More RELIABLE

• Less overlap between areas or traits means more reliable binning and resulting trends

• More EFFICIENT

• Less overlap between areas or traits means less resources spent binning

• Standardized follow-up and closure: resources not needed for case-by-case treatment

Next Steps

• Continue consideration of program options

• Team will draft a report documenting its conclusions and recommendations

• Input into 2019 ROP assessment process

• Input into ROP enhancement phase 2

• Follow-up separate effort will further review the recommendations and consider the implementation strategy

• Will involve additional stakeholder outreach

• Review for appropriate level of Commission engagement

Questions