ML043350214

Attachment 1, Reactor Oversight Process Performance Metrics

ML043350214
Person / Time
Site:	Calvert Cliffs
Issue date:	12/03/2004
From:	Beverly Smith NRC/NRR/DIPM/IIPB
To:	Grant G, Gwynn T, Plisco L, Wiggins J Region 1 Administrator, Region 2 Administrator, Region 3 Administrator, Region 4 Administrator
Brain P. Smith
Shared Package
ML043350128	List: ML043350118 ML043350214
References
Download: ML043350214 (51)

Text

Reactor Oversight Process Performance Metrics PI-1 Consistent Results Given Same Guidance Definition: Independently verify PIs using Inspection Procedure (IP) 71151, PI Verification.

Count all PIs that cross a threshold because of discrepancies as noted in the resultant inspection report. Licensees are requested per Nuclear Energy Institute (NEI) 99-02 to report changes to PI colors as soon as practical upon discovery via a mid-quarter report and to annotate in the comments field an explanation for the change.

Criteria: Use the first year of data as a benchmark for future comparison and to establish acceptable range of variability.

Lead: Regions, IIPB 5

4 Number of Occurrences 3

2 1

0 1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter

1. Significant Discrepancies Analysis: The graph represents the number of significant deficiencies reported for each quarter. Significant discrepancies are issues identified by the NRC during a PI verification inspection that caused the PI to cross a threshold.

Metric Criterion Met: Yes. No significant deficiencies were reported during this assessment period.

Page 1

Reactor Oversight Process Performance Metrics PI-2 Questions Regarding Interpretation of PI Guidance Definition: Quarterly, count the number of frequently asked questions (FAQs).

Criteria: Expect low numbers (but not as low as metric PI-1), with a stable or decreasing trend.

Lead: IIPB 60 40 Number of FAQs 20 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter New FAQs Approved FAQs Open/Active FAQs Analysis: Each quarter represents the total number of new FAQs introduced and approved during the ROP NRC/Industry Working Group meetings held during the respective quarter. This metric was revamped after insights were gained from the first year of full implementation of the ROP. The improvements made to this metric provide for a more timely and accurate account of FAQs. Since this metric was reconstructed from historical data, the second and third quarter of 2000 contain estimates (some FAQ logs were unavailable).

Interpretation questions regarding the PI guidance in NEI 99-02 took an upward trend during the initial stages of the ROP. This upward trend was anticipated; however, as NRC inspectors and licensees became more familiar with the guidance, and as additional guidance was provided to clarify NEI 99-02, a lower and generally stable number of questions require evaluation. For this assessment period, the number of unresolved interpretation questions has increased slightly. However, preliminary results for 3Q/2004 indicate the number of open questions is dropping to the levels seen prior to this assessment period.

However, several of the FAQs continue to have remained open for a significant amount of time. As reported in the previous ROP self-assessment SECY paper, these FAQs are related to the Scrams with Loss of Normal Heat Removal PI.

NRC continues to work with stakeholders to resolve the open issues.

Page 2

Reactor Oversight Process Performance Metrics Specifically, an ROP Working Group Task Force has been formed to address the issues with the Scrams with Loss of Normal Heat Removal PI.

Metric Criterion Met: Yes. There are relatively low numbers of FAQs with a stable or declining trend Page 3

Reactor Oversight Process Performance Metrics PI-3 Timely Indication of Declining Safety Performance Definition: Quarterly, track PIs that cross multiple thresholds (e.g., green to yellow or red).

Evaluate and characterize these results to allow timely indication of declining performance.

Criteria: Expect low numbers (near zero).

Lead: IIPB 5

4 Number of Occurrences 3

2 1

0 1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter No.of PI Crossing Multiple Thresholds Analysis: For the given parameters that are monitored by the PIs, the PIs appear to provide timely indication of declining performance.

Metric Criterion Met: Yes. There were no occurrences of PIs crossing multiple thresholds during this assessment period.

Page 4

Reactor Oversight Process Performance Metrics PI-4 Minimize Potential for Licensee Actions Taken in Response to the Performance Indicator Program That Adversely Impact Plant Safety Definition: Survey stakeholders regarding PIs driving undesirable decisions. This question will be included in the overall Federal Register notice.

Criteria: Expect low numbers of unintended consequences reported, with a stable or decreasing trend.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 5

Reactor Oversight Process Performance Metrics PI-5 Timely PI Data Reporting Definition: Within 5 weeks of the end of each calendar quarter, track (count) late PI postings on the NRCs external Web site. Also note the number of late submittals from licensees that did not meet the 21-day timeliness goal.

Criteria: Expect a low number (near zero) of late PI submittals and postings on the NRCs external Web site.

Lead: IIPB 5

4 Number of Untimely Actions 3

2 1

0 1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter

1. of Late Postings Analysis: There have been no late PI data postings on the NRC's external web site since the inception of the ROP. However, during the 3rd quarter of 2003 one site submitted PI quarterly data past the expected due date set forth in NEI 99-02, "Regulatory Assessment Performance Indicator Guideline," Revision 2. The late submittal required additional NRC follow up but did not delay the web postings past the required 5 weeks from the end of the quarter.

Metric Criterion Met: Yes. There were no late PI submittals and postings on the NRCs external Web site.

Page 6

Reactor Oversight Process Performance Metrics PI-6 Stakeholders Perceive Appropriate Overlap of Inspection Program and PIs Definition: Survey stakeholders perceptions of overlap between PIs and the Inspection Program. This question will be included in the survey for internal stakeholders and the Federal Register notice for external stakeholders.

Criteria: Expect a low number of negative comments, with a stable or declining trend in the number of negative comments received.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 7

Reactor Oversight Process Performance Metrics PI-7 Reporting Conflict Reduction Definition: Survey licensees and other external stakeholders regarding the perceived overlap between reporting requirements, such as those promulgated by Institute of Nuclear Power Operations (INPO), the World Association of Nuclear Operators (WANO), and the Maintenance Rule. This question will be included in the Federal Register notice.

Criteria: Expect a low number of negative comments, with a stable or declining trend in the number of negative comments received.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 8

Reactor Oversight Process Performance Metrics PI-8 Clarity of PI Guidance - NEI-99-02 Definition: Survey external stakeholders perceptions regarding the clarity of the guidance contained in NEI 99-02. This question will be included in the Federal Register notice.

Criteria: Expect a low number of negative comments or examples of interpretation issues, with a stable or declining trend in the number of negative comments received.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 9

Reactor Oversight Process Performance Metrics IP-1 Percentage of Inspection Findings Documented in Accordance With Requirements Definition: Audit inspection reports in relation to program requirements (IMC 0612, Power Reactor Inspection Reports) for documenting green findings, greater-than-green findings, and violations. Report the percentage of findings that meet the program requirements. Each year, audit one resident/integrated report from each plant, 25 percent of all other baseline reports, and all reports resulting from inspections beyond the baseline program.

Criteria: Expect an improving trend in the percentage of findings documented in accordance with program requirements.

Lead: IIPB 100%

% Findings that Meet All Requirements 80%

60%

40%

20%

0%

1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 3Q/04 4Q/04 Calendar Quarter Moving avg for all regions Comments: The graph represents the cumulative average for all inspection reports reviewed by the IIPB staff during 2003. The average reported for any given quarter is the integrated average for the past 4 quarters.

Analysis: The review of the data for the integrated average for all inspection reports is ongoing for the calendar year 2004 since the staff is currently evaluating this data. The final results will be included in the CY 2004 annual report.

Metric Criterion Met: N/A Page 10

Reactor Oversight Process Performance Metrics IP-2 Number of Baseline Inspection Procedures Significantly Changed Definition: Review all issued changes to baseline inspection procedures and count those procedures whose scope or frequency of inspection changed, and count new inspectable areas that relate to risk-informing the inspection.

Criteria: Expect relatively few significant changes, with a stable or declining trend.

Lead: IIPB 10 Number of Procedures Changed 8

6 4

2 0

1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter Program Documents Analysis: There were three minor changes in the second quarter CY 2004 that affected the scope of the baseline inspection procedures. IP 71111.01 Adverse Weather Protection, has been revised to clarify sample sizes, minimum samples for completion and improve guidance provided in the inspection requirements. IP 71111.21 Safety System Design and Performance Capability, has been revised to provide information on using probabilistic analyses to select risk-significant systems and components. IP 71111.20 Refueling and Other Outage Activities, has been revised to add guidance for containment inspections, when possible, and to add a completion status section. Although the staff made these minor changes to the baseline inspection procedures, there were no significant changes to the scope or frequency made during the first two quarters of CY 2004.

Metric Criterion Met: Yes. This metric was met based on the relatively stable trend.

Page 11

Reactor Oversight Process Performance Metrics IP-3 Number of Feedback Forms per Document Definition: Count the number of feedback forms received for each program document each quarter. Use a histogram to chart the number of documents for which feedback forms were received. Highlight those documents against which the most forms are written.

Criteria: Expect a decreasing trend in the number of feedback forms received for program documents.

Lead: IIPB 30 120 Number of Program Documents 25 100 Number of Feedback Forms 20 80 15 60 10 40 5 20 0 0 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter 1-3 4-6 7-9 10-12 13-15 >15 Forms Received Analysis: The staff received 70 feedback forms from January 1, 2004 through June 31, 2004. Approximately 67% of all feedback forms received during this assessment period were related to issues in the following areas: (1) Significant Determination Process (IMC 0609); (2) Power Reactor Inspection Reports (IMC 0612); (3) Inspector Qualification Program for the Office of Nuclear Reactor Regulation Inspection Program (IMC 1245); and (4) Baseline Inspection Procedure: Maintenance Effectiveness (IP 71111.12). Each of these four documents received relatively an equal amount of forms. The rest of the documents each received about 33% of all feedback forms.

Metric Criterion Met: No. This metric data indicated that the feedback forms received during the first quarter through the second quarter CY 2004 increased slightly. The concentration of Page 12

Reactor Oversight Process Performance Metrics feedback forms in selected program areas indicated that there needs to be further improvement in these areas for CY 2004.

Page 13

Reactor Oversight Process Performance Metrics IP-4 Completion of Baseline Inspection Program Definition: Annual completion of baseline inspection program.

Criteria: Defined as per IMC 2515, Light-Water Reactor Inspection Program - Operations Phase.

Lead: IIPB, Regions Comment: The baseline inspection program completion data will be included in the CY 2004 annual report since data is collected and can only be analyzed on a yearly basis.

Analysis: None Metric Criterion Met: N/A Page 14

Reactor Oversight Process Performance Metrics IP-5 Inspection Reports are Timely Definition: Obtain RPS data on the total number of reports issued and the number issued within timeliness goals (Timeliness goals are defined in IMC 612, Power Reactor Inspection Reports).

Criteria: Expect 90 percent of inspection reports to be issued within program's timeliness goals.

Lead: IIPB, Regions 100%

Percentage of Timely Reports 80%

60%

40%

20%

0%

1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter R1 R2 R3 R4 Nat'l Total 90% Criteria Analysis: For inspections not conducted by a resident inspector, inspection completion is normally defined as the day of the exit meeting. For resident inspector and integrated inspection reports, inspection completion is normally defined as the last day covered by the inspection report.

A total of 274 inspection reports were issued between the first and second quarter in CY 2004. During the first quarter regions 1 and 2 did not meet the timeliness goals due to the moratorium on Safeguards reports. However in the second quarter all four regions met the timeliness goal. Overall as a program, 95.2 percent of all issued inspection reports were timely.

Metric Criterion Met: Yes. This metric was met based on more than 90 percent of inspection reports issued within program's timeliness goals.

Page 15

Reactor Oversight Process Performance Metrics IP-6 Temporary Instructions (TIs) are Completed Timely Definition: Audit the time to complete TIs by region. Compare the completion status in RPS to TI requirements. Report by region the number of TIs closed within goals.

Criteria: Expect all TIs to be completed within TI requirements.

Lead: IIPB 10 Number of TIs Closed Within Goals 8

6 4

2 0

1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter R1 R2 R3 R4 # to be closed Analysis: TI 2515/151 (Expanded Pilot Force-On-Force Exercise Evaluation) was completed during the first quarter of 2004 within the timeliness goals by all four regions. No other TIs are required to be completed during the first two quarters of this calender year. However, several TIs are due to be completed in the second half of 2004.

Metric Criterion Met: Yes. The metric met expectations because all TIs were completed within their goals.

Page 16

Reactor Oversight Process Performance Metrics IP-7 Public Communication Is Timely Definition: IIPB posts inspection reports to the NRC's external (public) Web site within ROP timeliness goals using electronic version of inspection reports entered into the Agency Document Access and Management System (ADAMS) by the regions.

IIPB also posts entries from the Plant Issues Matrix (PIM) to the NRC's public Web site using data entered into RPS by the regions. In addition, IIPB records the number of inspection reports not available in ADAMS and the number of PIM entries not updated in RPS, as well as the number of inspection reports and PIMs that are not posted to the NRC's public Web site within goals.

Within five weeks of the end of each quarter, IIPB posts issued inspection reports from the previous quarter, using the electronic version in ADAMS, and the associated PIM entries from RPS to the NRC's public Web site. Within nine weeks of the end of each quarter, IIPB posts additional inspection reports and PIM entries for those not yet issued by the 5-week posting to include all findings from the previous quarter.

Criteria: Expect few untimely postings of PIMs or inspection reports, with a stable or declining trend.

Lead: IIPB 100%

Percentage of Timely Postings 75%

50%

25%

0%

1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter R1 R2 R3 R4 Analysis: There continue to be very few untimely postings of PIMs and/or inspection reports to the ROP web page. The few exceptions appear to be isolated and not indicative of a systematic problem. However, the percentage of timely postings has consistently been at or very near100% for each quarter, with a stable trend in untimely postings.

Metric Criterion Met: Yes. The metric met expectations.

Page 17

Reactor Oversight Process Performance Metrics IP-8 Public Communication Is Accurate Definition: Each calendar quarter, sample information on the NRC's external (public) Web site and count the number of times and reasons for regions changing PIMs or inspection reports (i.e., inaccuracy, new information).

Criteria: Expect few inaccuracies, with a stable or declining trend.

Lead: IIPB, Regions 14 12 Number of Inaccuracies 10 8

6 4

2 0

1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National Total Analysis: The large number of inaccurate postings of Plant Issues Matrix (PIM) entries or inspection reports on the web during 1st quarter 2003 was attributed to errors by one region. This region has taken actions to reduce the number of inaccuracies as evidenced by very few inaccuracies in the subsequent quarters.

Metric Criterion Met: Yes. The metric was met because of the recently few inaccuracies in the first two quarters of CY 2004.

Page 18

Reactor Oversight Process Performance Metrics IP-9 Analysis of Inspection Hours Definition: Collect and analyze resource data (e.g.; direct inspection effort, preparation/documentation, plant status hours) for Baseline, Supplemental/Plant Specific, and Safety Issues Inspections, and other ROP activities.

Criteria: (1) Significant deviations are not expected on an annual basis. Explore reasons for any deviations that may be evident.

(2) Track and trend resource usage for the baseline inspection program and supplemental/plant-specific inspections. Analyze causes for any significant departure from established trend.

(3) Track and trend resource usage for preparation, documentation, and other ROP activities, and assess the effects on budgeted resources.

Lead: IIPB Analysis: This metric is intended primarily for tracking and trending resource usage for the ROP. The results are used to improve the efficiency and effectiveness of the ROP and to make management and budget decisions. The tabulated data for this metric will be included in the CY 2004 annual report since the data is ongoing and is currently being collected and reviewed.

Metric Criterion Met: N/A Page 19

Reactor Oversight Process Performance Metrics IP-10 Survey of ROP Users Definition: Survey inspectors and other NRC personnel implementing the ROP, asking whether the inspection program covers areas that are important to safety.

Criteria: Trend average level of agreement.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 20

Reactor Oversight Process Performance Metrics IP-11 Survey of Inspection Report Usefulness Definition: Survey external stakeholders, asking about the usefulness of inspection reports.

This question will be included in the Federal Register notice.

Criteria: Trend average level of agreement.

Lead: IIPB Comment: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 21

Reactor Oversight Process Performance Metrics SDP-1 The SDP Results Are Predictable and Repeatable and Focus Stakeholder Attention on Significant Safety Issues Definition: Quarterly audit of a representative sample of reported inspection findings against the standard criteria set forth in IMC609, Significance Determination Process.

Findings should contain adequate detail to enable an independent auditor to trace through the available documentation and reach the same significance color characterization.

Criteria: The target goal is at least 90% are determined to be predictable and repeatable.

Any SDP outcomes determined to be non-conservative will be evaluated and appropriate programmatic changes will be implemented.

Lead: RES for greater than green; DIPM/IIPB (reactor); DIPM/IOLB (non-reactor) 100%

Percentage of Comforming Findings 80%

60%

40%

20%

0%

1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National Total Analysis: This metric was not captured for the first two quarters of CY 2004. The data is currently under review. A complete analysis will be included in the CY2004 annual report when sufficient data has been collected and analyzed.

Metric Criterion Met: N/A Page 22

Reactor Oversight Process Performance Metrics SDP-2 SDP Outcome Is Risk-Informed and Accepted by Stakeholders Definition: Track the total number of appeals of final SDP results reported quarterly by the regions.

Criteria: Expect zero appeals of SDP significance that result in a final determination being overturned across all regions.

Lead: Regions, IIPB 3

Number of Appeals 2

1 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter SDP Appeals Successful Appeals Analysis: There was one appeal of final SDP decision during the first two quarters of CY 2004. A Green design control finding was successfully appealed at Calvert Cliffs, the licensee provides documentation in which demonstrated that the design had been accepted by NRR.

Metric Criterion Met: No. Performance during this assessment period did not meet program expectations because of this successful appeal.

Page 23

Reactor Oversight Process Performance Metrics SDP-3 Inspection Staff Is Proficient and Find Value in Using the SDP Definition: Survey internal stakeholders using specific quantitative survey questions that focus on training, effectiveness, and efficiency.

Criteria: Expect either a stable or an increasingly positive perception of the SDP process over time.

Lead: IIPB Comments: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 24

Reactor Oversight Process Performance Metrics SDP-4 SDP Tools for Evaluating Inspection Findings Reflect Current Plant Design and Licensee Operating Practices.

Definition: Monitor substantive revisions made to the risk-informed inspection notebooks due to non-conservative technical flaws by tracking the number of phase 2 inspection notebooks that are issued for use and subsequently withdrawn following onsite benchmarking activities.

Criteria: The target goal is zero notebook retractions due to non-conservative technical flaws.

Lead: IIPB 3

Number of Revisions 2

1 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter Worksheet Revisions Analysis: The staff completed benchmarking the risk-informed inspection notebooks for all sites, which included comparing the notebooks against licensee-developed risk models using similar assumptions. No (revision 1) notebooks have been retracted or returned to Brookhaven National Laboratories for immediate revision to limit potentially non-conservative outcomes during the assessment period.

Metric Criterion Met: Yes. Performance during this assessment period met program expectations.

Page 25

Reactor Oversight Process Performance Metrics SDP-5 Results of the Same Color are Perceived by the Public to Translate to the Same Level of Significance for All Cornerstones.

Definition: Publish a Federal Register notice to survey external stakeholders using specific questions asking for examples of where the SDP-determined significance of findings does not appear to be consistent across ROP cornerstones.

Criteria: Expect stable or increasingly positive perception of the SDP over time.

Lead: IIPB Comments: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 26

Reactor Oversight Process Performance Metrics SDP-6 The Resources (Direct Charges and Support Activities) Expended Are Appropriate Definition: Track the percentage of total inspection resource expenditures attributed to SDP activities. Calculate the effort expended by the regions in completing SDP evaluations as a percentage of the total regional direct inspection effort. Use RPS codes for SDP processing activities.

Criteria: Total SDP expenditures should not exceed 10 percent of the total regional direct inspection effort (DIE) with a stable or decreasing trend.

Lead: IIPB 12 Percentage of DIE for SDP Efforts 10 8

6 4

2 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National R1 R2 R3 R4 Analysis: Regional expenditures associated with SDP evaluations remain stable and below the target goal. There was a slight increase in the average due to significant resource expenditures on Hope Creek, Seabrook, Wolf Creek, Cooper and ANO fire protection issues. These issues contributed significantly to the large increase in expenditures in Regions I and IV. Region IIs high expenditures were also a result of fire protection issues, requiring the use of the new SDP which was issued in May 2004.

Metric Criterion Met: Yes. Performance during this assessment period met program expectations.

Page 27

Reactor Oversight Process Performance Metrics SDP-7 Appropriateness of Regulatory Impact from the SDP Definition: Monitor the trend of regulatory impact forms that are critical of the SDP and assessment processes.

Criteria: Expect a stable or decreasing trend.

Lead: IIPB Analysis: Regulatory impact forms analysis will be included in the CY 2004 annual report since the data is pending and the analysis is performed on an annual basis.

Metric Criterion Met: N/A Page 28

Reactor Oversight Process Performance Metrics SDP-8 Final Significant Determinations Are Timely Definition: Conduct a quarterly audit of RPS data to identify the total number of inspection items finalized as greater than green that were under review for more than 90 days since:

(1) the date of initial licensee notification of the preliminary significance in an inspection report, or (2) the date the item was formally transmitted to an NRR technical branch for SDP assistance, or (3) the item was otherwise documented in an inspection report as an unresolved item pending completion of a significance determination and not counted in either of the above categories.

Criteria: In FY 2003, at least 75% of all SDP results that are counted per the criteria above should be finalized within 90 days, increasing 5% per year to 90% in FY 2006. All issues greater than 90 days will be assessed to determine causal factors and to recommend process improvements.

Lead: IIPB 110%

100%

Actions Completed within 90 days 90%

80%

70%

60%

50%

40%

30%

20%

10%

0%

3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter Total R1 R2 R3 R4 Analysis: Timeliness of final significance determinations decreased from 73% in FY 2003 to 58% for the first two quarters of CY 2004. There was a total of 14 issues closed during this assessment period, and 8 of 14 issues were finalized within the 90 days. The issues that were late include Davis Besse, Comanche Peak, Point Beach and Oconee corrective actions related to pressurizer heaters. In accordance with IMC 0307, issues not meeting the timeliness metric are assessed to determine causal factors and to recommend process improvements.

SDP timeliness remains a challenge and continues to be addressed by the SDP Improvement Task Action Plan.

Page 29

Reactor Oversight Process Performance Metrics Metric Criterion Met: No. Performance during this assessment period did not met program expectations. About 6 of the 14 untimely items in the first two quarters of FY 2004 were greater than 200 days old. The average age of the untimely closed items was 247 days. The increased closure rate is a result of increased management attention.

Page 30

Reactor Oversight Process Performance Metrics SDP-9 SDP Results Are Communicated Accurately to the Public Definition: Each calendar quarter, track the number of inspection findings that are inaccurately communicated to the public (color of findings is inaccurately reported), by auditing the inspection findings summary information available on the NRC web. The detailed review will include item type, significance characterization, enforcement action status, and text descriptions of greater-than-green inspection findings prior to release to external stakeholders.

Criteria: The target goal is zero inaccuracies. All inaccuracies must be addressed.

Lead: IIPB 10 8

Number of Inaccuracies 6

4 2

0 2000 2001 2002 2003 2004 Calendar Quarter Inaccurately Reported Findings Analysis: During the current assessment cycle two instances were identified in which the findings had the wrong date entered. As a result of the wrong date the findings were not posted in a timely manner on the NRC's external web site when looking at Assessment Matrix information developed from the reported Plant Issues Matrix (PIM) data. The errors were corrected. Although not considered an inaccuracy per this metric, the IIPB remains concerned with the PIM entries on the web page being incomplete. Based on recommendations for the OIG and SDP task group in CY 2003, we revised IMC 0306 to require that new inspection report numbers be developed and the PIM entries revised each time the status changed. For example, for a greater-than-green finding there should be at least two reports discussed in and linked from the PIM entry: the original report that identified the issue and the final SDP letter that noted its color. The supplemental inspection report should be included as well, when available. Of the 21 greater-than-green findings open at the end of 2Q2004, only 2 of them followed the aforementioned guidance. Many of these remain incomplete. This change was made so that the PIM entry would provide traceability of an issue Page 31

Reactor Oversight Process Performance Metrics from discovery to final resolution. IMC 0306 is currently being revised to clarify these expectations and to provide examples of complete PIM entries.

Metric Criterion Met: Yes. Performance in this area met program expectations.

Page 32

Reactor Oversight Process Performance Metrics AS-1 Subjective Judgment Is Minimized and Is Not a Central Feature of the Process. Actions Are Determined by Quantifiable Assessment Inputs (Examine PIs and SDP Results)

Definition: Audit all assessment-related letters and count the number of deviations from the Action Matrix.

Criteria: Expect few deviations, with a stable or declining trend.

Lead: IIPB Analysis: There have been a total of five (5) deviations from the Action Matrix since the beginning of the Reactor Oversight Program in Calender Year 2000.

• Salem/Hope Creek deviation approved August 20, 2004, to provide heightened NRC oversight to closely monitor the licensees actions to address significant issues associated with safety conscious work environment (SCWE).

• Cooper deviation approved April 12, 2004, to provide heightened NRC oversight to monitor the actions confirmed by the Confirmatory Action Letter (CAL), dated January 30, 2003.

• Indian Point 2 deviation approved April 2, 2004, to closely monitor the utilitys performance following the stations recovery from longstanding problems.

• Indian Point 2 deviation approved March 18, 2003 to provide for heightened oversight of the facility.

• Oconee 1 deviation approved on August 23, 2002, to permit for agency actions consistent with the degraded cornerstone column, including the performance of an IP 95002 vice IP 95003 supplemental inspection.

2Q02 3Q02 4Q02 1Q03 2Q03 3Q03 4Q03 1Q04 2Q04

1. of ROP 0 1 0 1 0 0 0 0 2 Deviations Metric Criterion Met: Yes. The number of deviations is very low with a stable trend. The deviations have been reviewed for potentially ROP generic implications. Based on this review, no program recommendations have been made.

Page 33

Reactor Oversight Process Performance Metrics AS-2 The Program Is Well-defined Enough to Be Consistently Implemented Definition: Audit all assessment letters and count the number of significant departures from requirements in IMCs 0305, Operating Reactor Assessment Program, and 0350, Oversight of Operating Reactor Facilities in an Extended Shutdown as a Result of Significant Performance Problems. Timeliness goals are counted in metric AS-5.

Criteria: Expect few departures, with a stable or declining trend.

5 Number of Inappropriate Actions 4

3 2

1 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National Lead: IIPB Analysis: There were no significant departures from the requirements of IMC 0305 or 0350 as a result of an audit of assessment letters during the period between January and June 2004.

Metric Criterion Met: Yes. There were no departures from the requirement of IMC 0305 or 0350 and the trend is considered stable.

Page 34

Reactor Oversight Process Performance Metrics AS-3 Actions Taken Are Commensurate with the Risk of the Issue and Overall Plant Risk Definition: Review actions taken for greater-than-green inspection findings and PIs. Track the number of actions (or lack of actions) taken by the regions that are not appropriate for the significance of the issues and are non consistent with the Action Matrix.

Criteria: Expect few departures, with a stable or declining trend.

Lead: IIPB 5

Number of Inappropriate Actions 4

3 2

1 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National Analysis: All actions taken by the regional offices were consistent with the Action Matrix during the period between January 2004 and June 2004 based on the fact that all appropriate inspections were consistent with the action matrix.

Metric Criterion Met: Yes. No departures from the ROP regarding actions taken in response to greater-than-green findings or PIs were identified.

Page 35

Reactor Oversight Process Performance Metrics AS-4 The Number And Scope of Additional Actions Recommended as a Result of the Agency Action Review Meeting (AARM) Beyond Those Actions Already Taken Are Limited Definition: Review the results of the Agency Action Review Meeting (AARM).

Criteria: Few additional actions, with a steady or declining trend.

Lead: IIPB Analysis: The AARM was held on April 14, 2003, in Lansdowne, Virginia. The participants confirmed the appropriateness of agency actions for Point Beach 1 and 2 and Cooper. The participants recommended one additional action, beyond those already taken or planned, for Cooper. The recommendation was to consider updating the CAL for Cooper to incorporate actions associated with findings subsequent to the CAL issuance. The next Agency Action Review Meeting is scheduled for April 2005.

Metric Criterion Met: Yes. The AARM participants made only one recommendation for consideration.

Page 36

Reactor Oversight Process Performance Metrics AS-5 Assessment Program Results (Assessment Reviews, Assessment Letters and Public Meetings) Are Completed in a Timely Manner Definition: Track the number of instances in which timeliness goals established in IMC 0305 were not met. The regions will collect timeliness data for the conduct of quarterly reviews (within 5 weeks of the end of quarter); mid-cycle, and end-of-cycle reviews (within 6 weeks of the end of quarter); issuance of assessment letters (within 2 weeks of the quarterly review and 3 weeks of the mid-cycle and end-of-cycle reviews); assessment follow-up letters (on or before the next quarterly review); and public meetings (within 16 weeks of the end of the assessment period).

Criteria: Expect few instances in which timeliness goals were not met, with a stable or declining trend.

Lead: Regions, IIPB 5

4 Number of Untimely Actions 3

2 1

0 1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter National Analysis:

2Q/2004: All quarterly assessment reviews and all five assessment follow-up letters were completed within timeliness goals.

Page 37

Reactor Oversight Process Performance Metrics 1Q/2004: All sixty four1 end-of-cycle meetings, all sixty four1 annual assessment letters, and the two(2) assessment follow-up letter were completed within timeliness goals.

4Q/2003: All sixty six quarterly assessment reviews and three out of four assessment follow-up letters were completed within timeliness goals.

3Q/2003: All sixty six mid-cycle review meetings were conducted within timeliness goals.

Additionally, all sixty six mid-cycle letters and one assessment follow-up letter were completed within timeliness goals.

Metric Criterion Met: Yes. The timeliness goals were met and have a stable trend.

1 Previously, it was reported that sixty six (66) assessment meeting and letters were completed.

Prior to the 2003 Mid-Cycle Assessment, Indian Point was treated as two different sites with two assessment letters. For the 2003 Mid-Cycle Assessment and subsequent assessments, one assessment letter was issued for both Indian Point units. In addition, Davis-Besse is under the IMC 0350 process and is therefore does not receive an ROP Assessment letter.

Page 38

Reactor Oversight Process Performance Metrics AS-6 The Web Posting and Availability via ADAMS of Assessment Letters Is Timely Definition: Review the posting of letters to the NRC's external Web site and availability in ADAMS and compare to the timeliness goals. Record the number of letters not available in ADAMS and number of letters not posted to the Web site within goals.

Criteria: IIPB posts assessment letters to the NRC's external Web site using the electronic version in ADAMS within 10 weeks after the end of mid-cycle and end-of-cycle assessment periods and within 8 weeks of the end of intervening quarters.

Lead: IIPB 3

Number of Untimely Actions 2

1 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter R1 R2 R3 R4 Nat'l Avg.

Analysis: 2Q/2004: Four (4) assessment follow-up letters were not posted to the web within timeliness goals.

1Q/2004: Two (2) assessment follow-up letters were not posted to the web within timeliness goals.

4Q/2003: Three assessment follow-up letters were not posted to the web within timeliness goals.

3Q/2003: All of the sixty six mid-cycle letters were posted to the web within timeliness goals. One assessment follow-up letter was not posted to the web within timeliness goals.

Page 39

Reactor Oversight Process Performance Metrics Between January and June 2004, 92% of the assessment letters (end-of-cycle and follow-up) were posted within timeliness goals. This web posting timeliness self assessment metrics shows a slight decline from 94% during the previous review period (January - December 2003). IIPB will be taking additional steps to ensure that assessment follow-up letters are posted to the web in a timely manner. IIPB has recently began requesting that the regional offices provide their input to the action matrix summary web page prior to posting this information publically.

The breakdown of assessment letters posted within timeliness goals is as follows: 64 of 64 of end-of-cycle letters and 1 of 7 follow-up assessment letters.

There are two reasons the follow-up assessment letters are not being posted in a timely manner: (1) regions are not consistently sending the follow-up assessments to the IIPB email address (RidsNrrDipmlipb@nrc.gov) as specified in IMC 0306; and (2) IIPB is not consistently checking the inbox in the IIPB email address (RidsNrrDipmlipb@nrc.gov). The staff will monitor this metric closely for the second half of 2004.

Metric Criterion Met: Yes. The overall web posting timeliness goal was met. However, the data shows that the assessment follow-up letters, as a whole, are not being posted on the web in a timely manner.

Page 40

Reactor Oversight Process Performance Metrics AS-7 Assessment Program Procedures Are Stable Enough to Be Perceived as Predictable Definition: Count the number of revisions to IMCs 0305 and 0350.

Criteria: Expect few revisions, with a stable or declining trend.

Lead: IIPB 5

4 Number of Program Revisions 3

2 1

0 2000 2001 2002 2003 2004 Calendar Year

1. of IMC revisions Analysis: During calender year 2002, there was one revision to IMC 0305, Operating Reactor Assessment Program which was issued on February 19, 2003 and IMC 0350 Oversight of Operating Reactor Facilities in a Shutdown Condition Performance Problems which was issued on December 31, 2003. A revision to IMC 0305 is planned for late calendar year 2004.

Metric Criterion Met: Yes. Multiple revisions of the overall IMC 0305 program have not been required.

Page 41

Reactor Oversight Process Performance Metrics AS-8 The NRC's Response to Performance Issues Is Timely Definition: Count the number of days between issuance of an assessment letter discussing an issue of more than very low safety significance and completion of the supplemental inspection (by exit meeting date, not issuance of the inspection report).

Criteria: Expect a stable or declining trend.

Lead: Regions, IIPB 200 175 150 Number of Elapsed Days 125 100 75 50 25 0

1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter R1 R2 R3 R4 Nat'l Avg.

Analysis: Baseline data for this metric are still being collected. However, data collected to date indicates a relatively stable long term trend regarding the elapsed time between the issuance of an assessment letter and the completion of the corresponding supplemental inspection. The data represents an average timeliness for the supplemental inspections completed in each region in any given quarter. IIPB will continue to monitor this data set to determine if an adverse trend exists, although the first half of 2004 appears to show a stable or declining trend.

Metric Criterion Met: N/A Page 42

Reactor Oversight Process Performance Metrics AS-9 The Agency Takes Appropriate Actions to Address Performance Issues for Licensees Outside of the Licensee Response Column of the Action Matrix Definition: Solicit feedback on the appropriateness of regulatory attention given to licensees with performance problems via a survey question to both internal and external stakeholders.

Criteria: Expect stable or improved perception.

Lead: IIPB Comments: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 43

Reactor Oversight Process Performance Metrics AS-10 Information Contained in Assessment Reports Is Relevant, Useful, and Written in Plain Language Definition: Perform surveys to determine internal and external stakeholder views on assessment reports.

Criteria: Expect stable or improved perception of the relevance, usefulness, and understandability of assessment reports.

Lead: IIPB Comments: This metric was not captured for the first two quarters of CY 2004. The survey is conducted in a biennial basis according by IMC 0307, is ongoing, and will be included in the CY 2004 annual report.

Analysis: None Metric Criterion Met: N/A Page 44

Reactor Oversight Process Performance Metrics AS-11 Degradations in Plant Performance, as Measured in the Action Matrix, are Gradual and Allow Adequate Agency Engagement of the Licensees Definition: Track the number of instances each quarter in which plants move more than one column to the right in the Action Matrix (as indicated on the Action Matrix Summary).

Criteria: Expect few instances in which plant performance causes a plant to move more than one column to the right in the Action Matrix. Provide a qualitative explanation of each instance in which this occurs. Expect a stable or declining trend from the first-year benchmark.

Lead: IIPB 5

4 Number of Occurrences 3

2 1

0 1Q/01 2Q/01 3Q/01 4Q/01 1Q/02 2Q/02 3Q/02 4Q/02 1Q/03 2Q/03 3Q/03 4Q/03 1Q/04 2Q/04 Calendar Quarter

1. of Occurences Analysis: During the period of January through June 2004, there were no reactor plants that moved more than one column to the right in the Action Matrix.

Since the beginning of the ROP, a total of 18 reactor units have been in the degraded cornerstone or multiple/repetitive degraded cornerstone column of the Action Matrix. Eight of these units moved more than one column to the right in the Action Matrix in one quarter. This includes Indian Point 2 which began the ROP in the multiple/repetitive degraded cornerstone column and four reactor units that moved to these columns due to yellow or red inspection findings. The number of plants moving two or more columns to the right in the Action Matrix have been few and within the expected frequency.

Metric Criterion Met: Yes. No plants have moved two or more columns to the right during the period of January - June 2004.

Page 45