ML102500676
| ML102500676 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 03/23/2009 |
| From: | Division of Inspection and Regional Support |
| To: | Atomic Safety and Licensing Board Panel |
| SECY RAS | |
| Shared Package | |
| ML102500654 | List:
|
| References | |
| 50-282-LR, 50-306-LR, ASLBP 08-871-01-LR-BD01, RAS 18574 | |
| Download: ML102500676 (12) | |
Text
NRC000048 NRC INSPECTION MANUAL DIRS MANUAL CHAPTER 0307 REACTOR OVERSIGHT PROCESS SELF-ASSESSMENT PROGRAM 0307-01 PURPOSE The Reactor Oversight Process (ROP) is a regulatory framework that includes licensee performance indicator (PI) data, NRC inspection activity and determination of inspection finding significance, and licensee performance assessment. The ROP self-assessment program evaluates the overall effectiveness of the ROP through its success in meeting its preestablished goals and intended outcomes.
0307-02 OBJECTIVES 02.01 To establish the processes for collecting information and data to support the ROP self-assessment program.
02.02 To establish a process for objectively evaluating the effectiveness of the ROP in achieving the goals of being objective, risk-informed, understandable, and predictable as well as the applicable agency performance goals listed in the NRC=s Strategic Plan.
02.03 To provide timely, objective information to inform program planning and to develop recommended improvements to the ROP.
02.04 To inform the Commission, NRC senior management, and the public of the results of the ROP self-assessment program, including any conclusions and resultant improvement actions.
0307-03 DEFINITIONS 03.01 Audit. A periodic examination and checking of selected records or activities to verify their correctness or compliance with predetermined standards.
03.02 Effectiveness. The ability to achieve the desired outcomes of an activity, program, or process. A program is considered effective if it is meeting its goals and achieving the intended outcomes. Ensuring that NRC actions are effective (e.g., high quality, efficient, timely, and realistic) is one of the organizational excellence objectives in the NRCs Strategic Plan and is used to evaluate the overall success of the ROP. Overall ROP effectiveness is measured each year through its self-assessment program using predefined Issue Date: 03/23/09 1 0307
NRC000048 metrics and performance evaluations of key program areas in accordance with this manual chapter.
03.03 Objective. A desired program attribute in which decisions are based on factual information and uninfluenced by emotion, surmise, or personal prejudice. Being objective is one of the four program goals of the ROP used to evaluate its overall success and effectiveness.
03.04 Openness. The ability to perform in a manner that is accessible and unobstructed in order to maintain public confidence. The goal to ensure openness explicitly recognizes that the public must be informed about, and have a reasonable opportunity to participate meaningfully in, the NRC=s regulatory processes. Ensuring openness in our regulatory process is one of the organizational excellence objectives in the NRCs Strategic Plan and is used to evaluate the overall success and effectiveness of the ROP.
03.05 Operating Plan. A management tool used to ensure that planning, budgeting, and performance management are performed in an integrated and balanced manner. It includes a summary of NRR programs, projects, activities, and other items to be measured throughout the year and the metrics which will be used to monitor them. Programs and activities are aligned under the five agency Strategic Goals.
03.06 Predictable. A desired program attribute in which more than one individual can follow the same defined process and arrive at the same conclusion in a consistent manner (i.e., repeatable). Being predictable is one of the four program goals of the ROP used to evaluate its overall success and effectiveness.
03.07 ROP Inspectable Area Lead. Person assigned responsibility to oversee and manage the use of individual baseline inspection procedures or attachments to those procedures.
03.08 ROP Program Area Lead. Person assigned responsibility to oversee and manage the associated programs for the major elements of the Reactor Oversight Process. The ROP elements are the performance indicator (PI) program, the inspection program, the significance determination process (SDP), and the assessment process.
03.09 Risk-Informed. An approach to decision-making in which risk insights are considered along with other factors (such as engineering judgment, safety limits, redundancy, and diversity) to better focus licensee and regulatory attention on issues commensurate with their importance to health and safety. Being risk-informed is one of the four program goals of the ROP used to evaluate its overall success and effectiveness.
03.10 Survey. A sampling or partial collection of facts, figures, or opinions taken and used to approximate or indicate what a complete collection and analysis might reveal. The NRC utilizes internal and external surveys to gather stakeholder feedback and gauge stakeholder satisfaction with the effectiveness of the ROP.
03.11 Understandable. A desired program attribute in which the process and its results are clear and written in plain English. Being understandable is one of the four program goals of the ROP used to evaluate its overall success and effectiveness.
Issue Date: 03/23/09 2 0307
NRC000048 0307-04 RESPONSIBILITIES AND AUTHORITIES 04.01 Director, Office of Nuclear Reactor Regulation (NRR). Oversees and manages the Reactor Oversight Process.
04.02 Director, Office of Research (RES). Provides support and data as requested by the Director, NRR.
04.03 Director, Office of Nuclear Security and Incident Response (NSIR). Provides support and data as requested by the Director, NRR, for the Security and Emergency Preparedness cornerstones.
04.04 Regional Administrators. Provide data to support the ROP self-assessment program as requested by the Director, NRR.
04.05 Director, Division of Risk Assessment (DRA). Provides data to support the ROP self-assessment program as directed by the Director, NRR.
04.06 Director, Division of Inspection and Regional Support (DIRS).
- a. Oversees the implementation of the ROP self-assessment program.
- b. Develops policies for the ROP self-assessment program.
- c. Issues the annual ROP self-assessment report.
- d. Issues status reports to the Deputy Regional Administrators.
04.07 Chief, Performance Assessment Branch.
- a. Develops program guidance and procedures for the ROP self-assessment program.
- b. Ensures data from all sources are collected and consolidated to facilitate analysis.
- c. Recommends and implements improvements to the ROP self-assessment program.
- d. Monitors the effectiveness of corrective actions and improvements to the ROP that are developed in response to self-assessment findings.
- e. Develops the annual ROP self-assessment report.
- a. Collect self-assessment data each calendar quarter for assigned program area (e.g., PI, inspection, SDP, and assessment).
Issue Date: 03/23/09 3 0307
NRC000048
- b. Collect and analyze self-assessment data for the previous year, and develop the annual program evaluation for assigned program area.
04.09 ROP Inspectable Area Leads.
- a. Collect data and user experience for assigned inspectable areas and summarize the information for the annual self-assessment report.
- b. Annually review and evaluate the implementation of assigned inspection procedures.
NOTE: A complete listing of current ROP Program Area Leads and Inspectable Area Leads is available through the Apoints of contact@ link from the ROP Digital City Web page.
0307-05 DISCUSSION The ROP is the NRC's primary means of assuring that commercial nuclear power plants are operated safely, securely, and in accordance with applicable regulations. The ROP consists of inspections, performance indicators, significance determination processes, assessment, and enforcement. It is important that the ROP be periodically evaluated and improved when necessary to ensure continued achievement of its specified goals and intended outcomes.
As noted in IMC 0308, Reactor Oversight Process Basis Document, the objectives of the staff in developing the various components of the ROP were to provide tools for inspecting and assessing licensee performance in a manner that was more risk-informed, objective, predictable, and understandable than the previous oversight processes. Accordingly, the goals of the ROP include the four specific program goals of being objective, risk-informed, understandable, and predictable, as well as the applicable organizational excellence objectives (e.g., openness and effectiveness) from the NRC=s Strategic Plan for Fiscal Years 2008-2013. Each of these ROP goals support the NRCs mission and characterize the manner in which the agency intends to achieve its strategic goals of safety and security:
to ensure adequate protection of public health and safety and the environment, and to ensure adequate protection in the secure use and management of radioactive materials.
The six ROP goals are summarized below:
Objective - decisions are based on factual information and uninfluenced by emotion, surmise, or personal prejudice.
Risk-informed - risk insights are considered along with other factors (such as engineering judgment, safety limits, redundancy, and diversity) to better focus licensee and regulatory attention on issues commensurate with their importance to health and safety.
Understandable - the process and its results are clear and written in plain English.
Predictable - more than one individual can follow the same defined process and arrive at the same conclusion in a consistent manner (i.e., repeatable).
Issue Date: 03/23/09 4 0307
NRC000048 Open - the NRC appropriately informs and involves stakeholders in the regulatory process.
Effective - NRC actions are high quality, efficient, timely, and realistic, to enable the safe and beneficial use of radioactive materials.
The NRC utilizes a planning, budgeting, and performance management (PBPM) process and program-level operating plans, which include performance measures and targets, to directly ensure that the performance goals of the Strategic Plan are properly assessed and that key program outputs and outcomes are met. The PBPM process and associated operating plans are the primary means of determining whether the strategic performance goals are being met. The ROP self-assessment program is not meant to replicate or replace this activity; however, many of the ROP self-assessment program metrics are the same as or similar to measures and criteria of the PBPM.
The ROP self-assessment process utilizes program evaluations and performance metrics to determine its success in meeting the goals and intended outcomes of the ROP. The level of effectiveness of the ROP is determined by considering whether the program goals are met and the intended outcomes are achieved. The intended outcomes of the ROP, which help form its basis and are incorporated into the various ROP processes, include to successfully:
$ Monitor and assess licensee performance
$ Identify performance issues through NRC inspection and licensee PIs
$ Determine the significance of identified performance issues
$ Adjust resources to focus on significant performance issues
$ Evaluate the adequacy of corrective actions for performance issues
$ Take necessary regulatory actions for significant performance issues
$ Communicate inspection and assessment results to stakeholders
$ Make program improvements based on stakeholder feedback and lessons learned Periodically, the ROP self-assessment program collects information from various sources, including the Reactor Program System (RPS), the inspection program, the PI program, other industry-level indicators, periodic independent audits, stakeholder surveys, public comments, and other stakeholder interactions. Based on this information, the success of the ROP's major program areas (PIs, inspection program, significance determination process, and assessment) is assessed. These program area evaluations align directly with, and fulfill the intent of, the planned program evaluations for the ROP as stipulated in Appendix B to NRCs Strategic Plan for Fiscal Years 2008-2013. In addition, the ROP's overall effectiveness is assessed and recommendations for improvement are made.
05.01 Performance Metrics. A set of performance metrics associated with each of the program areas of the ROP was developed to assess performance with respect to the seven goals of the ROP mentioned above. A detailed description of these performance metrics is contained in Appendix A. The four primary ROP program areas include the PI program, the inspection program, the significance determination process, and the assessment program.
These program specific metrics are designated as the PI, IP, SDP, and AS metrics respectively.
Issue Date: 03/23/09 5 0307
NRC000048 Metrics of a more general nature are also monitored and analyzed, primarily using stakeholder feedback, to gauge overall performance of the ROP. In addition, general metrics were developed to provide the basis for the annual evaluations of the ROP resource expenditures, resident inspector demographics and staffing, the training and qualifications program, and the regulatory impact analysis. These more general metrics are designated as the O (overall) metrics.
Each metric in Appendix A includes its definition, the criteria to determine whether it is met, the organization responsible for gathering the data, and a cross-reference to those goals each metric is intended to support. Each lead organization outside of NRR/DIRS has been provided with instructions summarizing the data elements and metrics they are requested to support, the periodicity the data is needed, and any specific instructions necessary to clarify the scope of the data. For example, the regional counterparts have been provided with a detailed data collection and submittal form detailing the specific data elements needed to support each performance metric for which the regions have lead responsibility for data collection. The performance metrics will be reviewed as part of the annual ROP self-assessment process to evaluate their efficiency and effectiveness in providing a useful assessment of the ROP. Metrics may be added, deleted, or modified as necessary to provide a meaningful management tool.
05.02 Data Collection. NRR/DIRS has the overall responsibility for data collection. A variety of methods are used to collect data regarding the performance of the ROP. These methods include data from the RPS, internal and external stakeholder surveys, independent audits, responses to Federal Register notices, and information collected via program document reviews. In addition, RES, NSIR, the regional offices, and other NRR branches are tasked to provide data. To the extent possible, data collection is from agency databases and the need for ad hoc, manually developed data is minimized. Since the self-assessment program is relying heavily on the quality of the data contained in the RPS database, it is imperative that the regions ensure the accuracy and timeliness of the RPS data. As part of the annual metric review, NRR/DIRS will evaluate the need to modify or add permanent automated systems to obtain needed metric information to minimize the burden on the staff.
With the exception of stakeholder surveys and responses to Federal Register notices, data is collected quarterly. Data reporting is completed within 45 calendar days of the end of the quarter under review. Internal and external stakeholder surveys or Federal Register notices to collect stakeholder feedback are issued at least biennially. Also, periodic equipment trending reports issued by RES are reviewed to identify additional insights into ROP performance.
- a. ROP inspectable area leads remain cognizant of the implementation of their assigned procedures. Throughout the year, they collect feedback forms written against their assigned areas, they participate in industry meetings to gain insights into the industry's perceptions of their areas, and they visit regions and sites to perform or observe their inspections in the field and to discuss their areas with the inspectors and regional managers. The purpose for the ROP inspectable area leads to perform or observe inspections is (1) to assess the adequacy of the inspection procedure for possible improvements to its scope, focus, and guidance, (2) to assess the adequacy of the ROP program guidance, and (3) to collect Issue Date: 03/23/09 6 0307
NRC000048 comments on the ROP from inspectors and licensees. This programmatic self-assessment is not intended to audit the performance of the regions in implementing the ROP.
Each year, the inspectable area leads summarize the insights gained, significant issues with, and major changes to their assigned areas. The summary is given to the ROP program area lead responsible for the inspection program in time to support the inspection program evaluation for the annual self-assessment report.
In addition, a more detailed review and realignment of inspection resources will be performed at least biennially in accordance with Appendix B to this Chapter. The focus of this effort is to adjust existing inspection resources to improve the effectiveness of the inspection program in identifying significant licensee performance deficiencies.
- b. ROP program area leads remain cognizant of the implementation of their assigned programs. Throughout the year, they collect feedback forms written against their assigned areas, they visit regions and sites to discuss their areas with the inspectors and regional managers, and they participate in industry meetings to gain insights into the industry's perceptions of their areas. The program area leads collect self-assessment metric data for their areas each calendar quarter.
At the end of each year, the ROP program area leads collect metric data and other insights into their areas and analyze the data for the previous year. The analyses form the basis for the program evaluations and are included in the annual self-assessment report on the ROP.
05.03 Data Analysis and Recommendation Development. NRR/DIRS has the overall responsibility for analyzing program data and developing recommended improvements to the ROP. Data analysis consists of comparing performance metric data with preestablished criteria and writing a determination of its meaning or programmatic impact.
For example, criteria for acceptable ROP performance have been identified for each performance metric as detailed in Appendix A. Thus a favorable comparison of data to criteria would indicate the ROP met the process goals and objectives, and likely, no programmatic changes would be recommended. However, for an unfavorable comparison more analysis is required to determine causal factors and develop recommended process improvements.
The analysis of data also includes evaluating the feedback forms, the results of audits conducted on various aspects of the ROP, comments collected from internal and external stakeholders, and any other insights gained by ROP inspectable area leads and ROP program area leads.
Due to their direct experience with the inspection and oversight programs gained through their implementation of the procedures, the regions should be consulted during the data analysis and recommendation development process to ensure the regional insights are incorporated into the change process.
05.04 ROP Self-Assessment Reports. There are several types of periodic ROP self-assessment reports that serve different purposes as described below:
Issue Date: 03/23/09 7 0307
NRC000048
- a. Periodic Reports. As noted above, metric data are collected and analyzed on a quarterly basis, as applicable. Periodic reports may be issued as deemed necessary to address particular issues of concern resulting from the quarterly data and analysis.
- b. Annual Performance Metric Report. NRR/DIRS develops an annual ROP performance metric report after the conclusion of each calendar year. The overall summary report must discuss any metrics that did not meet their preestablished criteria, the staff=s analysis of the reasons for not meeting the criteria, and any actions taken or planned to change the program or improve its implementation.
The report may briefly discuss any other significant lessons from the analyses of the metrics, even if the lesson is related to a metric that did meet its criteria. The report will also identify any metrics not counted during the previous year and the reasons for that. The analysis may be included in a separate report or may be incorporated into the annual Commission paper discussed in 05.04.c.
- c. Annual Commission Paper. As directed by the Commission, the staff presents the results of its annual self-assessment of the ROP in a Commission paper. The Commission paper is written to support the Agency Action Review Meeting (AARM) and the Commission briefing on AARM results that follows the review meeting. This paper typically addresses any lessons learned from the previous year, effectiveness evaluations of any major changes made to the ROP, the status of issues discussed in the Commission paper from the previous year, and any other significant issues affecting the ROP.
The annual Commission paper includes evaluations of the four key program areas of the ROP: the PI program, the inspection program, the SDP, and the assessment program. In addition, the staff typically includes discussions and assessments of ROP communication and inspector training activities, ROP self-assessment and independent evaluations (including regulatory impact), ROP resources, and other topic areas, as warranted. The self-assessment focuses on the effectiveness of recent significant program changes, the strengths and weaknesses of the program, and additional planned actions to improve program effectiveness. The paper also presents the staff=s overall conclusions as to whether the ROP has been successful in meeting the goals and intended outcomes of the ROP. The staff also highlights those areas of the ROP that warrant focus in the upcoming year based on the self-assessment results and lessons learned. The paper typically includes several attachments with additional detail to support the staff=s assessment and conclusions.
- d. Consolidated Response to External Survey. To address a prior concern that the staff had been unresponsive to the external stakeholder survey comments, NRR/DIRS prepares a consolidated response to the external survey. This response consolidates all comments received in response to the survey by question and provides a comprehensive response to each question. The annual ROP self-assessment Commission paper, the annual ROP performance metric report, and the consolidated response to the external survey are posted to the ROP Web page and sent along with an acknowledgment letter to each survey respondent.
05.05 Customized Audits of the ROP. After each annual ROP cycle, NRR/DIRS may use Issue Date: 03/23/09 8 0307
NRC000048 the insights gained from the self-assessment to develop topics for audits that delve more deeply into those aspects of the ROP that show indications of weaknesses or areas for future development. The topics may be suggested by an analysis of the metrics, an analysis of the feedback forms, audits of inspection reports, survey responses, or information gathered at counterpart meetings. NRR/DIRS develops an audit plan that tailors the audit to each region and identifies the attributes to be verified and associated standards. The audits can verify consistency of program implementation among the regions, verify an aspect of the program over all four regions, or focus on specific areas within one or two regions. The audits are generally conducted by NRR/DIRS staff, who may ask for assistance from other branches or the regions if a particular expertise is needed.
END Appendix A, Reactor Oversight Process Self-Assessment Metrics Appendix B, ROP Realignment Process , Revision History for IMC 0307 Issue Date: 03/23/09 9 0307
NRC000048 NRC000048 ATTACHMENT 1 Revision History For IMC 0307 Commitment Issue Date Description of Change Training Training Comment Tracking Needed Completion Resolution Number Date Accession Number N/A 12/12/02 Revised significantly to include a more None N/A N/A detailed discussion of the role of inspectable and program area leads, the annual review of the baseline inspection program, and other aspects of the self-assessment program. The specific metrics for these roles were added to Appendix A.
N/A 12/12/03 Revised to provide greater detail for None N/A N/A documenting the results of the annual inspection procedures reviews, and some metrics in Appendix A were modified to better align with the operating plan metrics and other program commitments.
N/A 01/14/04 Based on a decision at the DRP/DRS None N/A N/A counterpart meeting held on December 17-18, 2003, metric IP-5 was revised to change the inspection report timeliness to 45 calendar days for all inspection reports, with exception of reactive inspection reports, which will stay at 30 days.
Issue Date: 03/23/09 Att1-1 0307
NRC000048 N/A 02/20/06 Revised to support the new safety None N/A ML060110235 performance measures of the NRC=s Strategic Plan, to better define the ROP goals and intended outcomes, and to consolidate and clarify several of the performance metrics.
Completed 4 year historical CN search.
N/A 11/28/06 Revised to measure the effectiveness of the None N/A safety culture enhancements to the ROP, to clarify expectations regarding the resident demographics and staffing metrics, and to include a discussion of the consolidated response to external survey questions.
N/A 01/10/08 Revised to eliminate and consolidate several None N/A ML073510410 CN 08-002 metrics, to separate Appendix A from the base IMC to serve as a stand-alone document, and to summarize and link to Appendix B on the ROP realignment process.
Revised to address the Commission SRM W200800299 03/23/09 dated June 30, 2008, to reflect the recently None N/A ML090300620 CN 09-010 issued Strategic Plan for FY 2008 - 2013, and to reincorporate the security cornerstone in the ROP self-assessment process, and some metrics were revised for clarification purposes while others were removed to eliminate redundancy or unnecessary burden.
Issue Date: 03/23/09 Att1-2 0307