ML20064G915

Enclosures 1 and 2 - Cy 2019 ROP Metric Report Dashboard and Analysis

ML20064G915
Person / Time
Issue date:	03/11/2020
From:	Philip Mckenna NRC/NRR/DRO/IRSB
To:	Chris Miller NRC/NRR/DRO
Mayer A, NRR/DRO, 415-1081
Shared Package
ML20064G913	List: ML20064G914 ML20064G915
References
Download: ML20064G915 (16)

Text

Calendar Year 2019 Reactor Oversight Process Self-Assessment Metric Report (Enclosures 1 and 2)

Self-Assessment Metrics Overview/Dashboard Independence Metrics I-1 Completion of the I-2 Resident Inspector I-3 Resident Inspector I-5 Fully Qualified I-4 Inspector Objectivity I-6 Analysis of Metric Baseline Inspection Objectivity Through Objectivity Through Inspectors and Operator Reviews Permanent Site Staffing Program Rotation Policy Diverse Experience Licensing Examiners Result Green Green Green Green Green Green Openness Metrics O-3 Conduct of Annual O-5 Issuance of ROP O-1 Issuance of O-2 Issuance of Assessment Meetings or O-4 Reporting and O-6 Responsiveness to Metric Public Meeting Notices Inspection Reports Assessment Letters Other Engagement Dissemination of PI Data ROP Contact Us Forms and Summaries Activities Result Green Green Green Green Yellow (CY18: Green) Green Efficiency Metrics E-4 Completion of E-1 Completion of E-5 Completion of Final E-2 Initiation of Reactive E-3 Completion of Performance Deficiency E-6 Responsiveness to Metric Supplemental Significance Inspections Temporary Instructions Determinations ROP Feedback Forms Inspections Determinations Result Green Green Green Green Red (CY18: Green) Red (CY18: Red)

Clarity Metrics C-3 Traceability of C-1 Maintenance of ROP C-2 Corrections to ROP C-4 Maintenance of ROP Metric Greater-than-Green Web Pages Web Pages Governance Documents Inspection Findings Result Green Green Green Not Evaluated Reliability Metrics R-2 Predictability and R-1 Performance of R-3 Predictability of R-4 Consideration of Repeatability of Metric Lessons Learned Agency Actions and Operating Experience Significance Evaluations Response Insights Determination Results Result Green Green Green Green Enclosure 1

CALENDAR YEAR 2019 ROP SELF-ASSESSMENT METRICS REPORT REFERENCED TO INSPECTION MANUAL CHAPTER 0307, APPENDIX A 0307A-01 INDEPENDENCE PERFORMANCE METRICS (I)

I-1 Completion of Baseline Inspection Program Metric Criterion Met: Green Definition: The baseline inspection program is completed annually in accordance with program requirements.

Criteria: Green Yellow Red All regions and the Office 1 or more regions or of Nuclear Security and NSIR did not meet Incident Response N/A completion (NSIR) met completion requirements requirements

• Note: No less than 100% compliance for any region or NSIR.

Baseline inspection program completion is defined in Section 04.07 of Inspection Manual Chapter (IMC) 2515, Light-Water Reactor Inspection Program - Operations Phase (ADAMS Accession No. ML17079A202). Any region or office that does not complete the baseline inspection program per IMC 2515 is individually evaluated.

Analysis: For CY 2019, all regions and NSIR completed 100 percent of their baseline inspections as defined by IMC 2515 within the allocated resources. Each region and NSIR documented completion of the baseline inspection program via memorandum (Agencywide Documents Access and Management System (ADAMS) Accession Nos. ML20042E405 for Region I, ML20063H438 for Region II, ML20049H329 for Region III, ML20059N508 for Region IV, and ML20021A236 for NSIR (non-public)).

I-2 Resident Inspector Objectivity Through Rotation Policy Metric Criterion Met: Green Definition: Senior resident inspectors (SRIs) and resident inspectors (RIs) are stationed for a 7-year maximum tour length, unless specifically granted an extension per IMC 2515.

Criteria: Green Yellow Red 0 inspectors 1 or more inspectors exceeded 7 years N/A exceeded 7 years without an extension without an extension Enclosure 2

• Note: No less than 100% compliance for any region. Any region that has 1 or more inspectors exceed 7 years at a site without an extension is individually evaluated.

Analysis: For calendar year (CY) 2019, there was 100% compliance in all regions with no SRIs or RIs stationed at a plant beyond the maximum tour length without an extension. One region reported a minor clerical error with one extension memo, however the error was immediately corrected and there were no concerns regarding the RIs objectivity. As a result, the program office did not count this minor administrative error, as it does not indicate a loss of objectivity or a programmatic weakness in terms of RI tour lengths.

I-3 Resident Inspector Objectivity Through Diverse Experience Metric Criterion Met: Green Definition: Permanently-staffed SRIs and RIs spend a minimum of one week each year inspecting at another site.

Criteria: Green Yellow Red 3 noncompliant 4-5 noncompliant 6 noncompliant

• Note: No more than two non-compliances in any region. Any region that has more than two non-compliances is individually evaluated.

Analysis: For CY 2019, all but three permanently-staffed RIs and SRIs completed their objectivity visits by spending a minimum of one week inspecting at another plant. No region had more than two non-compliances.

I-4 Inspector Objectivity Reviews Metric Criterion Met: Green Definition: Line managers perform annual on-site employee performance and objectivity reviews of fully qualified inspectors assigned to an inspection branch.

Criteria: Green Yellow Red 3 noncompliant 4-5 noncompliant 6 noncompliant

• Note: No more than two non-compliances in any region or NSIR. Any region or office that has more than two non-compliances is individually evaluated.

Analysis: For CY 2019, all but three fully qualified inspectors received annual onsite performance and objectivity reviews in the regions and in NSIR.

No region or office had more than two non-compliances.

I-5 Fully Qualified Inspectors and Operator Licensing Examiners Metric Criterion Met: Green Definition: Inspectors and operator licensing examiners remain fully qualified in accordance with qualification requirements.

Criteria: Green Yellow Red 5 noncompliant 6-9 noncompliant 10 noncompliant

• Note: No more than three non-compliances in any region or NSIR.

Any region or office that has more than three non-compliances is individually evaluated.

Analysis: For CY 2019, all but four inspectors and/or operator licensing examiners remained fully qualified in accordance with IMC 1245 and its appendices. No region or office had more than three non-compliances.

I-6 Analysis of Permanent Site Staffing Metric Criterion Met: Green Definition: Permanent inspector staffing levels at each of the reactor sites for both SRIs and RIs are maintained to provide continuity of regulatory oversight.

Criteria: Green Yellow Red 95% < 95% AND 90% < 90%

• Note: No less than 90% compliance for any region. Any single site that falls below 90% is individually evaluated.

Inspectors assigned to the site permanently or through a rotation with a minimum duration of 6 weeks shall be counted. Inspectors on 6 week or longer rotational assignments will be identified as such. Inspectors assigned to the site for less than six weeks will not be counted, but should be indicated as such. Additionally, the regions shall indicate sites where permanently assigned resident or senior resident inspectors are away from the site for greater than 6 continuous weeks. Only inspectors who have attained at least a basic inspector certification status, as defined by Appendix A to Inspection Manual Chapter 1245, shall be counted.

Data will indicate number of days a qualified resident and senior resident inspector are permanently assigned to the site during the year divided by the number of days in the year. Number of days spent on training; meetings away from the site; participation in team inspections; leave; or other temporary duties (e.g. acting for branch chiefs in his/her absence) will not be counted against the metric unless the absence exceed 6 continuous weeks.

Analysis: For CY 2019, the average permanent inspector staffing was 97.1% for all regions with no single region falling below 97%. Overall, there were four sites that fell below the 90% threshold. A Region I site backfilled a vacant RI position with the only volunteer to respond to the rotation announcement, however, the volunteer did not complete his/her IMC 1245 Appendix A qualification prior to reporting to the site (achieved qualification at the end of the rotation), and so the time did not count for this metric. Two Region II sites fell below the 90% threshold due to extended rotations for one site and extended leave for a RI at another site. A Region III site had a RI go on a rotational assignment which was extended, and then upon return the RI went on unexpected sick leave.

Inspection support was provided, but not in the six-week increments that would count for this metric. In all cases, adequate site coverage was maintained. No further evaluation or action is required for these site since the events were not reasonably within the regions ability to foresee and prevent.

0307A-02 OPENNESS PERFORMANCE METRICS (O)

O-1 Issuance of Inspection Reports Metric Criterion Met: Green Definition: Reactor Oversight Process (ROP) inspection reports are issued within applicable timeliness goals.

Criteria: Green Yellow Red 25 late > 50 late

> 25 AND 50 late

(~ 95% timely) (~ 90% timely)

• Note: No more than 15 late inspection reports in any region or 5 late inspection reports in NSIR. Any region that has more than 15 late inspection reports or office that has more than 5 is individually evaluated.

Analysis: For CY 2019, there were seven inspection reports not issued on time.

No single region or NSIR had more than fifteen late inspection reports.

O-2 Issuance of Assessment Letters Metric Criterion Met: Green Definition: Annual and follow-up assessment letters are issued within the applicable timeliness goals.

Criteria: Green Yellow Red 2 late 3 late 4 late

• Note: No more than one late letter in any region. Any region that has more than one late letter is individually evaluated.

Analysis: For CY 2019, 100% of annual and follow-up assessment letters were issued on time.

O-3 Conduct of Annual Assessment Meetings or Other Engagement Activities Metric Criterion Met: Green Definition: Public assessment meetings or other engagement activities that discuss the results of the NRCs annual assessment of the licensees performance, are conducted annually for all sites within the applicable timeliness goals.

Criteria: Green Yellow Red 2 late 3 late 4 late

• Note: The level of public engagement and timeliness goals are determined by plant performance, as described in IMC 0305. No more than one late meeting/activity in any region. Any region that has more than one late meeting/activity is individually evaluated.

Analysis: For CY 2019, 100% of the public assessment meetings and other engagement activities met applicable timeliness goals.

O-4 Reporting and Dissemination of Performance Indicator (PI) Data Metric Criterion Met: Green Definition: Performance indicator (PI) data submittals by the licensees are posted to the NRCs external web site within the applicable timeliness requirements.

Criteria: Green Yellow Red 0 late web posting 1- 3 late web postings > 3 late web postings

• Note: Any licensee submittals that did not meet the timely reporting requirements will also be evaluated by NRC staff and discussed with industry to address corrective actions to prevent recurrence.

Analysis: For CY 2019, 100% of the licensee PI data submitted to the NRC was posted to the external ROP website on time. No licensee submitted data late to the NRC.

O-5 Issuance of ROP Public Meeting Notices and Summaries Metric Criterion Met: Yellow Definition: ROP-related public meetings are noticed prior to the meeting and meeting summaries are posted after the meeting within the applicable timeliness requirements.

Criteria: Green Yellow Red 95% < 95% AND 90% < 90%

• Note: ROP-related public meetings include ROP working group meetings, annual assessment meetings, and other ROP-related meetings conducted by NRC staff. No more than two late notices or summaries for any region or office. Any region or office that has more than two late notices or summaries is individually evaluated.

Analysis: For CY 2019, NRR, NSIR, and the regions noticed and summarized public meetings within established timeliness goals 91 percent of the time (the Green criterion is 95 percent). The large proportion of missed timeliness goals occurred because of inconsistencies between regions for how less formal public engagement activities (e.g., poster sessions, open houses) are handled in terms of meeting notices and summaries.

The program office staff has received clarification from the Office of Public Affairs and provided guidance to the regions on proper public meeting noticing and administration procedures for less formal public engagement activities that count as annual assessment meetings. The staff plans to revise IMC 0305 in CY 2020 to clarify the requirements for noticing less formal regional public engagement activities.

O-6 Responsiveness to ROP Contact Us Forms Metric Criterion Met: Green Definition: ROP Contact Us forms received through the public or internal website regarding the ROP are responded to within 45 days upon receipt.

Criteria: Green Yellow Red 95% < 95% AND 90% < 90%

Analysis: For CY 2019, 0 feedback forms were received from the NRCs external ROP Contact Us button. In CY 2019, 1 of 1 (100%) of all the completed NRCs internal ROP Contact Us feedback forms were responded to within the timeliness goals.

0307A-03 EFFICIENCY PERFORMANCE METRICS (E)

E-1 Completion of Supplemental Inspections Metric Criterion Met: Green Definition: Exit meetings for supplemental inspections are completed within 180 days from licensee notification of readiness.

Criteria: Green Yellow Red 1 late 2 late 3 late

• Note: No more than one late exit meeting in any region. Any region that has more than one late exit meeting is individually evaluated.

Analysis: For CY 2019, all exit meetings for supplemental inspections were completed within the timeliness goal.

E-2 Initiation of Reactive Inspections Metric Criterion Met: Green Definition: Entrance meetings for reactive inspections are conducted within 30 days of a determination that an event or specific circumstances require a reactive inspection.

Criteria: Green Yellow Red 2 late 3 late 4 late

• Note: No more than one late entrance meeting in any region. Any region that has more than one entrance meeting is individually evaluated.

Analysis: For CY 2019, 100% of entrance meetings for reactive inspections were completed within the timeliness goal. There was one reactive special inspection completed during CY 2019 at Fermi Power Plant, Unit 2, to evaluate degraded torus coatings and the potential for torus suction strainer blockage. The special inspection report is available at ADAMS Accession No. ML20031D253.

E-3 Completion of Temporary Instructions Metric Criterion Met: Green Definition: Temporary Instruction (TI) inspections associated with IMC 2201, Security Inspection Program for Operating Commercial Nuclear Power Reactors (ADAMS Accession No. ML18031B047), and IMC 2515 are completed within the required TI completion time.

Criteria: Green Yellow Red 97.5% of documented < 95% of documented

< 97.5% AND 95%

completions are timely completions are timely

• Note: No less than 95% completion for any region. Any region that falls below 95% is individually evaluated. The term documented completions refers to cases where TI results are documented in an inspection report.

Analysis: In CY 2019, 100% of TIs were documented as complete within the timeliness goals.

E-4 Completion of Performance Deficiency Determinations Metric Criterion Met: Green Definition: The time from the identification date (i.e., the date the issue of concern was brought to the licensees attention by the NRC, the date the performance deficiency was self-revealed, or the date the licensee documented the condition resulting from the performance deficiency in the corrective action program) to the start date used for consideration of inspection findings in the assessment process (as defined by IMC 0305) is within 120 days.

Criteria: Green Yellow Red 90% timely < 90% AND 75% < 75% timely

• Note: No more than two untimely occurrences for any region or NSIR.

Any region or office that has more than two untimely occurrences is individually evaluated.

Analysis: For CY 2019, one performance deficiency determination was completed in CY 2019 that was later finalized as a White finding (Watts Bar).

However, this finding was processed through a deterministic appendix of IMC 0609, Significance Determination Process (emergency preparedness), and these findings have not historically been counted for this metric. The effectiveness review of the Inspection Finding Resolution Management (IFRM) process pilot effort was completed in CY 2018 (ADAMS Accession No. ML18123A319), and the review team recommended that ROP metrics E-4 and E-5 be combined into a single 255-day SDP metric. This single 255-day SDP timeliness metric will be implemented for CY 2020 and provides updated guidance that all potentially Greater-than-Green findings, regardless of which IMC 0609 Appendix they are screened through, will count for this metric.

E-5 Completion of Final Significance Determinations Metric Criterion Met: Red Definition: Inspection items are finalized as greater-than-Green within 90 days since: (1) the date of initial licensee notification of the preliminary significance in an inspection report, or (2) the date the item was otherwise documented in an inspection report as an apparent violation or finding pending completion of a significance determination.

Criteria: Green Yellow Red 95% timely < 95% AND 90% < 90% timely

• Note: No more than one late finalized significance determination for any region or NSIR. Any region or office that has more than one late finalized significance determination is individually evaluated. NSIR will also meet the 90% timeliness expectation for security-related findings.

Analysis: For CY 2019, 0 of 1 (0 percent) of GTG findings were finalized within the timeliness goals. One White finding at Clinton Power Station (ADAMS Accession No. ML19092A212) was finalized in greater than 90 days. This extended timeline was due, in large part, to the time required for staff review of over 2,000 pages of supplemental information provided by the licensee after the Regulatory Conference.

Accordingly, the staff determined that this Red metric did not represent a programmatic weakness in the significance determination process program. As described in the analysis for metric E-4, one Watts Bar finding that was finalized as Greater-than-Green in CY 2019 was processed through a deterministic appendix of IMC 0609, Significance Determination Process (emergency preparedness), and these types of findings have not historically been counted for this metric.

Because of the low number of Greater-than-Green findings for CY 2019, any one finding that exceeded the E-5 timeliness goal would turn the metric to Red, since the Green threshold is 95 percent. In CY 2020, ROP metrics E-4 and E-5 will be combined into a single 255-day SDP metric. The Red/Yellow/Green criteria for the single 255-day SDP timeliness metric will be counted on a per-finding basis, instead of on a percentage basis, due to the decreasing number of Greater-than-Green findings that have been finalized each year.

E-6 Responsiveness to ROP Feedback Forms Metric Criterion Met: Red Definition: ROP feedback forms are completed within applicable timeliness goals.

Criteria: Green Yellow Red 90% timely < 90% AND 80% < 80% timely Analysis: For CY 2019, the staff did not meet the timeliness goals delineated in IMC 0801. However, the staff made significant improvements from the previous 2 years in this area. The staff dispositioned 62 percent of the feedback forms opened in CY 2019 (almost three times the CY 2018 closure rate), compared to the Green criterion of 90 percent. As of February 1, 2020, 114 feedback forms remain in the backlog, about a 50 percent decrease from CY 2018. Because the E-6 metric is not an effective measure of feedback form disposition timeliness, starting in CY 2020, the staff will measure its efficiency in dispositioning feedback forms via the new ROP data trending program. Metric E-6 will be removed from the formal set of ROP metrics in IMC 0307, Appendix A.

The staffs action to address this Red metric included issuance of revisions to IMCs and Inspection Procedures that closed several feedback forms. The staff also piloted an improved feedback form process that implements a semiannual regional screening and prioritization approach for all feedback forms.

0307A-04 CLARITY PERFORMANCE METRICS (C)

C-1 Maintenance of ROP Web Pages Metric Criterion Met: Green Definition: ROP pages on the public website are reviewed at least quarterly to ensure that the content on the page is up-to-date with accurate information.

Criteria: Green Yellow Red 90% Web pages < 80% Web pages

< 90% AND 80%

reviewed reviewed

• Note: All ROP-related Web pages will be reviewed for general content on a quarterly basis. This review will also include a sampling of hyperlinks for accuracy.

Analysis: For CY 2019, 100% of ROP Web pages on the public website were reviewed within timeliness goals.

C-2 Corrections to ROP Web Pages Metric Criterion Met: Green Definition: Broken hyperlinks or out-of-date content on the ROP internal or external Website are corrected within 30 days upon discovery.

Criteria: Green Yellow Red 95% corrected within < 90% corrected

< 95% AND 90%

30 days within 30 days Analysis: For CY 2019, 100% of the reported broken hyperlinks were corrected within the timeless goals. An average of three to five broken hyperlinks per quarter were identified and corrected.

C-3 Traceability of Greater-than-Green Inspection Findings Metric Criterion Met: Green Definition: Inspection findings are updated in the Reactor Program System (RPS) and posted to the ROP web page to ensure traceability of a greater-than-Green inspection finding from discovery to final resolution. When a report or letter follows up on an existing item (i.e., final significance determination letters and supplemental inspection reports), the RPS entry is updated to reflect the new information.

Criteria: Green Yellow Red

< 90% GTG 95% GTG findings

< 95% AND 90% findings traceable traceable

• Note: No more than one greater-than-Green inspection finding found to be untraceable for any region or NSIR. Any region or office that has more than one untraceable issue is individually evaluated.

Analysis: 2 of 2 (100 percent) Greater-than-Green inspection findings issued in CY 2019 were found to be traceable from discovery to final resolution on both the internal and external websites. Staff reviewed the documentation for each of the GTG findings finalized in CY 2019, focusing on ensuring inspection reports associated with the Greater-than-Green findings were accurately entered into RPS. Further, this review traced these findings on the ROP webpage, specifically the Plant Information Summary webpage, ensuring that external stakeholders have access to all of the publicly-available information pertaining to the findings.

C-4 Maintenance of ROP Governance Documents Metric Criterion Met: Not Evaluated Definition: Baseline inspection procedures and other ROP-related inspection procedures and manual chapters are reviewed at least once every 4 years.

Criteria: Green Yellow Red 95% reviewed within < 90% reviewed within past 4 years < 95% AND 90% past 4 years Analysis: This metric was not evaluated for CY 2019 due to inconsistences in how the metric has been counted and due to the fact that the metric definition is not in accordance with the NRR Office Instruction OVRST-102, NRR Procedures for Processing Inspection Manual Documents, which specifies that ROP governance documents should be reviewed and reissued every 5 years, not every 4 as metric C-4 measures. The staff has revised this metric, effective CY 2020, to conform to OVRST-102 with a 5-year IMC and IP periodic review requirement. Additionally, to ensure staff awareness and action in accordance with OVRST-102 and the revised ROP metric, the NRR Inspection Manual Coordinator will identify and track all IMCs and IPs that require periodic reviews in CY 2020 to ensure that they are completed in a five-year interval or as otherwise required.

0307A-05 RELIABILITY PERFORMANCE METRICS (R)

R-1 Performance of Lessons Learned Evaluations Metric Criterion Met: Green Definition: Lessons learned evaluations are performed, reports are issued, and recommendations are considered and entered into the tracking system for significant NRC activities to ensure their completion in accordance with program expectations.

Criteria: Green Yellow Red All required evaluations 1 or more evaluations completed and N/A not completed documented

• Note: All supplemental inspections conducted in accordance with inspection procedure (IP) 95003, Supplemental Inspection For Repetitive Degraded Cornerstones, Multiple Degraded Cornerstones, Multiple Yellow Inputs Or One Red Input, implementations of IMC 0350, Oversight Of Reactor Facilities In A Shutdown Condition Due To Significant Performance and/or Operational Concerns, Incident Investigation Team (IIT) responses, and Augmented Inspection Team (AIT) responses are individually evaluated for potential program improvements.

Timeliness expectations will be determined by senior management on a case-by-case basis.

Analysis: For CY 2019, there were no IP 95003, IMC 0350, AIT, or IIT inspections completed, and so there were no recommendations from lessons learned evaluations to enter into the ROP Lessons Learned Tracker.

The staff provided semiannual updates to DIRS management on the status of the ROP lessons learned, including the status of new lessons learned and timeliness of disposition for all lessons learned.

Accordingly, this metric is Green for CY 2019.

R-2 Predictability and Repeatability of Significance Determination Results Metric Criterion Met: Green Definition: Greater-than-Green inspection findings and the associated degraded conditions contain adequate detail to enable an independent auditor to trace through the available documentation and conclude that the significance characterization is reasonably justifiable from both programmatic and technical positions. This audit should be documented in a memo that is internally available to the NRC and referenced in the annual metric report.

Criteria: Green Yellow Red 0 deemed 1 deemed 2 deemed unpredictable unpredictable unpredictable

• Note: Any significance determination documentation determined by the auditor to be inadequate will be evaluated and appropriate programmatic changes will be considered.

Analysis: For CY 2019, the staff determined that 2 of 2 (100 percent) of the Greater-than-Green findings issued in the CY contained adequate detail to enable an independent auditor to trace through the available documentation and conclude that the significance characterization was reasonably justifiable from both programmatic and technical positions.

This audit is documented in a memorandum dated January 22, 2020 (ADAMS Accession No. ML20016A290, non-publicly available).

R-3 Predictability of Agency Actions and Response Metric Criterion Met: Green Definition: Deviations from the Action Matrix are expected to be infrequent to ensure reliable and predictable programmatic and technical positions.

Criteria: Green Yellow Red 1 deviations 2 - 3 deviations > 3 deviations

• Note: All deviations are individually evaluated for potential program improvements Analysis: For CY 2019, there were no ROP Action Matrix deviations.

R-4 Consideration of Operating Experience Insights Metric Criterion Met: Green Definition: A summary of recent operating experience insights is provided and discussed for the mid-cycle and end-of-cycle assessments for each region to inform inspection planning.

Criteria: Green Yellow Red Operating experience Operating experience not discussed during all N/A discussed during 1 or regional assessment more regional meetings assessment meetings Analysis: For CY 2019, 100% of all end-of-cycle assessment meetings included discussions of operating experience insights to inform inspection planning.