ML20262G982

Assessment Option 1 Proposed Methodology

ML20262G982
Person / Time
Issue date:	11/12/2020
From:	Russell G NRC/NRR/DRO/IRSB
To:
Russell Gibbs, NRR/DRO/IRSB, 415-8578
Shared Package
ML20247J590	List: ML20247J599 ML20247J601 ML20247J602 ML20262G982 ML20274A133
References
Download: ML20262G982 (12)

Text

Proposed Method for Problem Identification and Resolution Assessment Using Evaluation Criteria and Graded Response Approach to Support Assessment Option 1 As discussed in Assessment Option 1 in Appendix B to the Comprehensive Review of the Reactor Oversight Process Problem Identification and Resolution (PI&R) Inspection Program report, the effectiveness of a PI&R program could be assessed through the evaluation of criteria for each of the PI&R assessment areas. These criteria, listed below, are not intended to be exhaustive but rather provide a starting point for evaluating each area during inspection activities and the licensees annual assessment meeting. The goal is an improved assessment process with a framework that uses 4 key assessment areas and specific criteria to determine in an objective manner whether a licensees PI&R performance is either (1) satisfactory, (2) satisfactory but needs improvement or (3) unsatisfactory.

Although Assessment Option 1 is not the teams recommended option, the team developed this strawman approach to provide additional considerations for the concept which is discussed at a high-level in Appendix B to the report. If Option 1 was selected as the path forward to improve assessment, a substantial amount of staff and industry effort would be needed to align on the assessment areas and specific performance criteria proposed in this document. This should be taken into account in reviewing the acceptability of the material presented.

1. Evaluation Criteria This standard considers a less than satisfactory finding when the identified performance issue(s) is/are programmatic in nature, and not isolated to one aspect, case, individual, etc., as applicable.

A. General/Definitions

1. If the review team identifies performance issues/deficiencies that lead to programmatic weaknesses, the team should ensure the licensee determines the cause(s) of the issues, which can be used as the basis for developing recommendations for corrective actions.

2. For the Assessment Areas that contain sub-elements, a single finding with respect to the overall performance relative to the area would be made by the review team.

3. An adverse trend for assessment and documentation purposes is defined as 4-6 or more related observations or findings (including documented minor violations) over a 12-month period which illustrate a gap in a sub element of the licensees PI&R program.

The inspectors should be able to reasonably conclude the issues share a common element such as training, procedures, knowledge, and including Human Performance cross cutting factors defined in IMC 0310. Adverse trends must be documented in a semiannual trend sample in a resident quarterly report or team inspection report.

4. A repetitive adverse trend for assessment and documentation purposes is defined as 6 or more additional related observations or findings that represent licensee performance after identification of the original adverse trend and after sufficient time whereby corrective actions could have been implemented. At least a Level 1 follow-up inspection 1

activity should have been conducted and assessment documented in an inspection report before assigning a repetitive adverse trend.

5. A significant programmatic weakness for assessment and documentation purposes is defined as an identified and documented weakness that has resulted in multiple adverse trends covering several sub elements within an area or covers multiple PI&R assessment areas. This observation should be supported by a large number of findings (10 or more) which share a common element such as training, procedures, knowledge, and including Human Performance cross-cutting factors defined in IMC 0310, and should not be localized to a single licensee organization. A documented safety conscious work environment (SCWE) cross-cutting issue (CCI) would be an example of a significant programmatic weakness because it has an adverse effect on multiple areas of the PI&R program. At least a PI&R annual sample should have been conducted and assessment documented in an inspection report before assigning a significant programmatic weakness.

6. Significant Condition Adverse to Quality (SCAQ). A significant condition adverse to quality is one which, if uncorrected, could have a serious effect on safety or operability.

(from NQA-1, ASME Nuclear Quality Assurrance-1).

7. Condition Adverse to Quality (CAQ). A failure, malfunction, deficiency, deviation, defect, or nonconformance associated with the performance of an activity affecting the safety-related function of a structure, system or component. (from 10 CFR 50 Appendix B Criterion XVI).

8. Condition Adverse to Regulatory Compliance (CARQ). A condition where the licensee is not in conformance with NRC regulations, inspection or enforcement processes (such as the Reactor Oversight Process), a failure to comply with a docketed commitment made to the NRC, a non-compliance with the licensee Quality Assurance program that does not consequently affect nuclear safety. Conditions Adverse to Regulatory Compliance are addressed with licensee corrective action programs. (from NEI 16-07, Improving the Effectiveness of Issue Resolution to Enhance Safety and Efficiency).

9. Corrective Actions to Preclude Repetition (CAPR): Corrective Actions developed to correct the root cause of an SCAQ and prevent repetition or minimize consequences if the cause cannot be determined. (from 10 CFR 50 Appendix B Criterion XVI)

10. These evaluation criteria shall be reviewed during quarterly and annual plant assessment review (IMC 0305) and during the biennial PI&R team inspection (IP71152B). This allows for continuous assessment of the PI&R program and incorporates all available data rather than solely relying upon sample selection during the Biennial PI&R team inspection to assess performance.

2

B. Assessment Area 1 - Identification of Problems (P1, P4, P5, and P6)

The organization identifies an issue using one of serval robust programs and enters these issues into the PI&R process in a timely manner.

Sub Elements:

a. CAP Identification (P1): The organization implements a corrective action program with a low threshold for identifying issues. Individuals identify issues completely, accurately, and in a timely manner in accordance with the program

b. Trending Program (P4): The organization periodically analyzes information from the corrective action program and other assessments in the aggregate to identify programmatic and common cause issues.

c. Use of Operating Experience (P5): The organization systematically and effectively collects, evaluates, and implements relevant internal and external operating experience in a timely manner.

d. Self-Assessments and Audits (P6): The organization routinely conducts self-critical and objective assessments of its programs and practices.

1. Satisfactory A finding of satisfactory is appropriate when a review demonstrates the presence of the following conditions:

(a) Licensee staff enters conditions into the licensees PI&R program at a low threshold.

Deficient conditions associated with safety system performance or regulatory compliance are promptly brought to the attention of main control room operators so operability and functionality can be assessed. When appropriate, Technical Specifications action statements or compensatory measures are initiated.

(b) Licensee has an effective trending program which uses PI&R program data and other applicable insights (e.g., Maintenance Rule Program, System Health Reports, etc.) to identify low level trends with equipment and human performance. The licensee addresses identified issues prior to them becoming more significant problems (c) Licensee has an effective Quality Assurance Audit and self-assessment program which identifies weaknesses and places those deficiencies and observations into the PI&R Program for resolution.

(d) License has an effective process for gathering, evaluating, and entering issues identified at other facilities into their PI&R program when appropriate. This includes NRC Information Notices, Part 21 reports, industry wide communications, and fleet and owners group reports and recommendations.

(e) Occasional findings and performance deficiencies, represented by 3 or fewer findings individually or 6 or fewer findings combined with a cross-cutting aspect of either P1, P4, P5, or P6 in the previous 12-month period, which are not risk/security significant and do not represent and adverse trend.

3

2. Satisfactory But Needs Improvement Consideration should be given to a finding of Satisfactory But Needs Improvement when a review demonstrates the presence of one or more of the following conditions:

(a) An example of a failure to identify or to enter a known SCAQ into the corrective action program, not resulting in a safety-significant or security-significant finding. This criterion is intended to follow up on a significant programmatic breakdown where the licensees program guidance was clearly met and not followed/implemented. A SCAQ misclassified as a CAQ would not meet this criterion.

(b) An adverse trend is identified in one sub-element. An adverse trend should be supported by several examples, represent a performance or programmatic weakness in that sub element alone, and be documented in an integrated Inspection report as a semiannual trend or in a stand-alone PI&R inspection report. A trend in this area would include:

i. Repetitive examples of licensee staff being aware of Conditions Adverse to Quality and failing to document them in the Corrective Action Process or other parallel PI&R process (IE Simulator Deficiency Logs) or CAQs affecting operability are not promptly brought to the MCR operators so operability can be determined and appropriate technical specifications entered when appropriate.

ii. Repetitive examples of issues not being documented with enough relevant detail, such that operators make the non-conservative operability decisions for identified CAQs or CARCs.

(c) A safety or security significant finding with a cross cutting aspect of P1, P4, P5, or P6.

(d) Greater than 3 findings with a cross cutting aspect of P1, P4, P5, or P6 in any 12-month period. (3 in same individual aspect)

(e) Greater than 6 cumulative findings cumulative findings with cross-cutting aspects of P1, P4, P5, or P6 in any 12-month period.

(f) Greater than 7 findings with a cross cutting aspect of P1, P4, P5, or P6 in any 36-month period. (7 in same individual aspect). If a 24-month inspection cycle is retained this criteria would change to 5 in a 24-month period.

3. Unsatisfactory Consideration should be given to a finding of Unsatisfactory when a review demonstrates the presence of significant performance issues with respect to the other indicators that are determined to be related to one or more of the following conditions:

(a) A safety- or security-significant finding with a documented performance deficiency of failing to identify a significant adverse condition.

(b) A significant programmatic weakness exists which results in widespread failure to enter SCAQ, CAQ, CARC into the corrective action program.

4

(c) A repetitive adverse trend in P1, P4, P5, or P6 documented in at least 3 consecutive semiannual trend reviews or consecutive PI&R area inspections and NRC management determines licensee corrective actions have been ineffective based upon follow-up inspection.

(d) Greater than 1 safety- or security-significant finding with a cross-cutting aspect of P1, P4, P5, or P6 in any 12-month period.

(e) Greater than 10 findings with a cross cutting aspect of P1, P4, P5, or P6 in any 12-month period. (10 in same individual aspect)

(f) Greater than 24 cumulative findings cumulative findings with cross-cutting aspects of P1, P4, P5, or P6 in any 12-month period.

(g) Area documented as ineffective during a Supplemental Inspection or IMC 0350 inspection.

C. Assessment Area 2 - Evaluation of Problems (P2):

The organization thoroughly evaluates issues to ensure that resolutions address causes and extent of conditions commensurate with their safety significance.

Sub Elements:

a. Prioritization of Problems (Assigning proper level of review)

b. Evaluation of Problem (Evaluation follows and meets CAP Guidance)

1. Satisfactory A finding of Satisfactory is appropriate when a review demonstrates the presence of the following conditions:

(a) The licensee appropriately prioritizes issues in accordance with the safety, security, and/or radiological significance so that licensee resources and oversight are assigned commensurate with the actual or potential consequences of the issue in accordance with their PI&R program.

(b) The licensee thoroughly evaluates issues to ensure that resolutions address causes and extent of conditions commensurate with their safety, security, and radiological significance issue in accordance with their Corrective Action Program Procedure and Quality Assurance Plan.

(c) Occasional findings and performance deficiencies, represented by 3 or fewer findings with a cross-cutting aspect of P2 in the previous 12-month period, in this area which are not risk/security significant and do not represent an adverse trend.

5

2. Satisfactory But Needs Improvement Consideration should be given to a finding of Satisfactory But Needs Improvement when a review demonstrates the presence of one or more of the following conditions:

(a) An example of a failure to evaluate an identified SCAQ and develop corrective actions to preclude repetition (CAPRs) resulting in recurrence of the SCAQ or equipment failures but not resulting in a safety significant or security significant finding.

(b) An adverse trend is identified in one sub element. An adverse trend should be supported by several examples and represent a performance or programmatic weakness in that sub element alone and be documented in an integrated Inspection report as a semiannual trend or in a stand-alone PI&R inspection report. A trend in this area would include:

i. Repetitive example of licensee staff failing to accurately prioritize issue in accordance with their safety or security significance and as a result CAQs and CARC are not corrected commensurate with their safety significance.

ii. Repetitive examples of licensee staff failing to perform an adequate evaluation as assigned in their corrective action program which either: (1) does not identify the correct cause, (2) does not conduct an extent of cause or extent of condition issues (when required) due to lack of rigor. As a result, corrective actions developed do not correct the identified CAQ commensurate with its safety/security significance.

iii. Repetitive examples of the licensee failing to follow the corrective action program requirements for when an evaluation is unable to determine a cause for a more significant event.

(c) A safety or security significant finding with a cross-cutting aspect of P2.

(d) Greater than 3 findings with a cross-cutting aspect of P2 (e) Greater than 7 findings with a cross-cutting aspect of P2 in any 36-month period (See area 1 comment).

3. Unsatisfactory Consideration should be given to a finding of Unsatisfactory when a review demonstrates the presence of significant performance issues with respect to the other indicators that are determined to be related to one or more of the following conditions:

(a) A safety or security finding of Yellow or Red significance with a documented performance deficiency of failing to evaluate a significant adverse condition or develop CAPRs.

(b) A significant programmatic weakness exists which results in widespread failure to adequate evaluate SCAQ, CAQ, and CARC and develop corrective actions to correct the conditions.

6

(c) A repetitive adverse trend in P2 documented in at least 3 consecutive semiannual trend reviews or consecutive PI&R area inspections and NRC management determines licensee corrective actions have been ineffective based upon follow up inspection.

(d) Greater than 1 safety or security significant finding with a cross cutting aspect of P2 in any 12-month period.

(e) Greater than 10 findings with a cross cutting aspect of P2 in any 12-month period.

(f) Area documented as ineffective during an IP 95002 or IP 95003 supplemental inspection or IMC 0350 inspection.

D. Assessment Area 3 - Timely and Effective Corrective Actions (P3):

The organization takes effective corrective actions to address issues in a timely manner commensurate with their safety significance Sub Elements:

a. Corrective actions scheduled commensurate with Safety Significance

b. Deferral of Corrective Actions evaluated and Justified.

c. Corrective actions address original performance deficiency/issue and restore compliance.

d. CAPRs for SCAQs are effective.

1. Satisfactory A finding of Satisfactory is appropriate when a review demonstrates the presence of the following conditions:

(a) The licensee effectively schedules and completes corrective actions development commensurate with their safety significance using the work control process.

(b) The licensee appropriately performs an evaluation in the event that corrective actions placed in the work control process are deferred or cancelled. This evaluation would include compensatory actions, bridging strategies, or alternative corrective actions to ensure the CAQ or CARC is corrected commensurate with its safety or security significance.

(c) The licensee ensures that the final corrective actions completed adequately address the original CAQ or CARC observed.

(d) The licensee tracks CAPRs for SCAQ to completion in the corrective action program.

(e) The licensee conducts effectiveness reviews for CAPRs for SCAQ and develops new corrective actions when appropriate.

(f) Occasional findings and performance deficiencies in this area, represented by 3 or fewer cumulative findings with cross cutting aspect of P3 in any 12-month period, which are not risk/security significant and do not represent an adverse trend.

7

2. Satisfactory But Needs Improvement Consideration should be given to a finding of Satisfactory But Needs Improvement when a review demonstrates the presence of one or more of the following conditions:

(a) An example of a failure to implement CAPRs in a timely manner and prevent repetition of an identified SCAQ, but not resulting in a safety or security significant finding.

(b) Examples where CAPRs for SCAQ are not being tracked or completed under the CAP.

(c) Examples of effectiveness reviews for CAPRs for SCAQ not being completed or actions taken if the review identified an issue.

(d) An adverse trend is identified in one sub-element. An adverse trend should be supported by several examples and represent a performance or programmatic weakness in that sub element alone and be documented in an integrated Inspection report as a semiannual trend or in a stand-alone PI&R inspection report.

i. Repetitive example of licensee staff failing to schedule corrective action assignments in a timely manner commensurate with the CAQs or CARCs safety- or security-significance as evidenced by repetitive failures of equipment or corrective actions not accurately being completed due to errors in the work management process.

ii. Repetitive examples of licensee staff failing to complete corrective actions assigned due to work orders or engineering change packages being deferred and the deferrals not being evaluated and results in unnecessary or uncompensated safety or security risk until the underlying CAQ or CARC is corrected.

iii. Repetitive examples of the licensee failing to identify that the actions completed did not actually correct the CAQ or CARC or restore compliance commensurate with its safety or security significance.

(e) A safety- or security-significant finding with a cross-cutting aspect of P3.

(f) Greater than 3 findings with a cross-cutting aspect of P3 (g) Greater than 7 findings with a cross-cutting aspect of P3 in any 36-month period.

(See area 1 comment)

3. Unsatisfactory Consideration should be given to a finding of Unsatisfactory when a review demonstrates the presence of significant performance issues with respect to the other indicators that are determined to be related to one or more of the following conditions:

(a) A safety- or security-significant finding with a documented performance deficiency of failing to correct a significant adverse condition.

8

(b) A significant programmatic weakness exists which results in widespread failure to correct SCAQ, CAQ, and CARC commensurate with their safety or security significance.

(c) A repetitive adverse trend in P3 documented in at least 3 consecutive semi-annual trend reviews or consecutive PI&R area inspections and NRC management determines licensee corrective actions have been ineffective based upon follow up inspection.

(d) Greater than 1 safety- or security-significant finding with a cross-cutting aspect of P3 in any 12-month period.

(e) Greater than 10 findings with a cross cutting aspect of P3 in any 12-month period.

(f) Area documented as ineffective during a Supplemental Inspection or IMC 0350 inspection.

E. Assessment Area 4 - Safety Conscious Work Environment (SCWE):

The organization ensures a work environment exists where employees feel free to raise safety concerns and where concerns are promptly reviewed, given the proper priority based on their potential safety significance, and appropriately resolved with timely feedback to the originator of the concerns and to other employees.

Sub Areas:

a) SCWE Policy: The organization effectively implements a policy that supports individuals rights and responsibilities to raise safety concerns, and does not tolerate harassment, intimidation, retaliation, or discrimination for doing so.

b) Alternate Process for Raising Concerns: The organization effectively implements a process for raising and resolving concerns that is independent of line management influence. Safety issues may be raised in confidence and are resolved in a timely and effective manner.

c) Free Flow of Information: Individuals communicate openly and candidly, both up, down, and across the organization and with oversight, audit, and regulatory organizations.

1. Satisfactory A finding of Satisfactory is appropriate when a review demonstrates the presence of the following conditions.

(a) The licensee has an established SCWE program verified by a review of programmatic documents.

(b) The licensee monitors for a SCWE through self-assessment using their proceduralized Nuclear Safety Culture Monitoring Programs, employee concerns program (ECP), and site-specific review boards to screen disciplinary actions for potential chilling implications.

9

(c) The licensee maintains a safety-conscious work environment as evident through discussions and interviews with licensee staff. Site employees appear willing to raise nuclear safety concerns through at least one of the several means available.

2. Satisfactory But Needs Improvement Consideration should be given to a finding of Satisfactory But Needs Improvement when a review demonstrates the presence of one or more of the following conditions which are potential indications of a SCWE concern and further monitoring is warranted.

(a) Allegations of Harassment, intimidation, retaliation, or discrimination (HIRD) received involving a senior licensee manager. Allegation being reviewed by OI or mediated through Alternative Dispute Resolution. (Potential for employees to be chilled by observation of this behavior)

(b) Allegations received alleging a chilled work environment.

(c) Interview results indicate that multiple employees in a work group are hesitant or unwilling to raise concerns to certain managers or at all.

(d) Licensee or third-party safety culture review/assessment identifies work groups of concern with respect to SCWE.

(e) An unsatisfactory in PI&R Performance areas 1, 2, or 3.

(f) A step-increase in the number of allegations received compared to the previous 3 years.

The step change should result in the total number of allegations being at least double the industry average for that year.

(g) High-volume of allegations where total allegations are greater than 3 times the industry average for at least 2 consecutive years.

3. Unsatisfactory Consideration should be given to a finding of Unsatisfactory when a review demonstrates the presence of significant performance issues with respect to the other indicators that are determined to be related to one or more of the following conditions:

(a) In any 18-month period there is a documented ROP finding with a cross-cutting aspect of SCWE, and the impact on SCWE was not isolated. For the purpose of this criteria not isolated means more than one individual is impacted (e.g., multiple individuals, functional groups, shift crews, or levels within the organization are affected).

(b) The licensee has received a chilling effect letter during the assessment period, or one remains open.

10

(c) The licensee has received correspondence from the NRC that transmitted (1) a SL I, II, or III enforcement action that involved discrimination or (2) a confirmatory order that involved discrimination. The theme applies only to the sites(s) where the discrimination occurred.

(d) Results of a supplemental inspection of IMC 0350 inspection document a SCWE concern.

(e) Results of SCWE focus group (PI&R samples) documented in 3 consecutive report show no improvement, declining SCWE, or indications that the concern is impacting multiple site organizations and NRC management has concerns about the effectiveness of their corrective actions taken to date and this concern is documented in an Annual Assessment letter.

2. Actions Taken When PI&R Area(s) Weaknesses Are Identified Satisfactory Licensees with satisfactory performance across all areas would continue to receive inspection as dictated by the Action Matrix.

Satisfactory But Needs Improvement The Regional staff shall conduct an annual PI&R sample to evaluate the licensees corrective actions to address the identified weakness(es) for specific areas rated as Satisfactory But Needs Improvement. It is expected that the licensee would place a documented programmatic weakness in their PI&R program and conduct an evaluation. This sample should be scheduled and coordinated with the licensee to allow adequate time for the licensee to takes actions to address the weakness in accordance with their PI&R program. PI&R sample selection should be discussed at the End of Cycle meetings but not included in the Annual Assessment Letter or RPS Report 22 (i.e., Inspection Activity Planned Report). This sample would be within the baseline inspection limits of the procedure. For a SCWE assessment area Satisfactory But Needs Improvement assessment, focus groups should be considered - additional annual sample guidance would be required for this option. Information provided by the licensee such as Allegation Process Requests for Information may be used in lieu of a site visit on a case-by-case basis.

Unsatisfactory For a licensee with 1-2 unsatisfactory areas a supplemental inspection may also be required by the Action Matrix. That inspection should be informed to include the evaluation of the unsatisfactory areas. If no supplemental procedures are occurring, an additional PI&R sample above nominal baseline levels if necessary or a standalone IP 93100, Safety Conscious Work Environment Issue of Concern Followup inspection utilizing qualified Safety Culture Assessors shall be completed (Assessment area 4 only). These inspection activities would focus on the licensees corrective actions for the identified unsatisfactory assessment area. Note: IP 93100 is an IMC 2515 Appendix C inspection which would require regional administrator approval and 11

be discussed in the Annual Assessment Letter and Report 22. Feeder to quarterly resident inspection report. The estimated level of effort for this inspection is 14-50 hours. Note:

supplemental inspections IP 95001 and IP 95002 would need to be updated to ensure the degraded PI&R programmatic element is evaluated and corrective actions followed up on. This is already an implied element of the procedure and would not change the level of effort for IP 95001 or IP 95002 in most cases. This is currently a direct element of IP 95003.

For a licensee with greater than 2 Unsatisfactory areas and in at least Column 2 of the Action Matrix or 1 unsatisfactory area and in Columns 3 or 4 of the NRC Action Matrix: Schedule and conduct an additional PI&R team inspection at the midpoint of the inspection cycle as allowed by IMC 0305. This would result in a standalone inspection report, discussion at the End of Cycle Meeting, and Annual Assessment Letter/Report 22. The estimated level of effort is 212-280 direct inspection hours assuming current resource expenditures.

For a plant with all 4 performance areas evaluated as Unsatisfactory and having completed all of the above follow-up inspections with documented unsatisfactory assessments and NRC management has documented ongoing concerns about the effectiveness of corrective actions for consecutive assessment periods, NRC management shall consider moving the licensee to Column 5 of the Action Matrix as discussed in IMC 0305 and its IMC 0308.

3. Basis for Suggested Graded Approach This evaluation approach is modeled after the format used in NRC Management Directive 5.6, Integrated Materials Performance Evaluation Program (IMPEP), dated July 24, 2019. This approach supports the philosophies discussed in IMC 0305 and IMC 0308 and the Commission direction specified in SRM-SECY-2000-0049. NRC response is performance-based and an appropriate level of management review is built into each level to ensure addition inspections are adequately considered. The lowest level response is with current baseline inspection guidelines. The next level is similar in scope to traditional enforcement follow-up discussed in Section 11 of IMC 0305 (IP 92722 and 92723). The assignment of an additional biennial inspection is currently discussed in Section 10 of IMC 0305 and supports the Commission position on the importance of licensee PI&R to the ROP and overall licensee performance. A review of plants that have been in Columns 3, or 4 or placed in the IMC 0350 process, shows that the majority of these plants had a significant weakness in their PI&R programs which led to this decline in performance. The assessment evaluation presented in this attachment is more prescriptive than the current assessment process but remains consistent with existing the principles of the ROP. Refer to the history and basis document for additional information.

12