ML102120044
| ML102120044 | |
| Person / Time | |
|---|---|
| Site: | Prairie Island  |
| Issue date: | 04/13/2010 |
| From: | Travieso-Diaz M Northern States Power Co, Pillsbury, Winthrop, Shaw, Pittman, LLP, Xcel Energy |
| To: | Atomic Safety and Licensing Board Panel |
| SECY RAS | |
| Shared Package | |
| ML102120039 | List: |
| References | |
| 50-282-LR, 50-306-LR, ASLBP 08-871-01-LR-BD01, RAS 18336, RCE01166830-01 QF-0433, Rev 2 | |
| Download: ML102120044 (62) | |
Text
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 1 of 31 RCE REPORT SCAQ - Inadequate CAP Resolution of Significant Issues Event Date: 1/26/2009 RCE01166830-01 CAP AR 01166830 RCE Team Members:
Management Sponsor: Tim Allen Business Support Manager Team Leader: Rick Myers Planning Manager Team Member: Fred Ericson Security Team Member: Roberta MilesBoysen Configuration Management Team Member: Gene Woodhouse Performance Assessment Team Member/Consultant: Lloyd Calvery WD Associates RCE Mentor: Gene Woodhouse Performance Assessment Approvals:
Richard Myers, Planning Manager RCE Team Leader Date Tim Allen, Business Support Manager Management Sponsor Date Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 2 of 31 Table of Contents Page #
I. Executive Summary.................................................................................................... 3 II. Event Narrative ........................................................................................................... 5 III. Extent of Condition Assessment................................................................................. 6 IV. Previous Similar Events:............................................................................................. 8 V. Operating Experience: ................................................................................................ 9 VI. Nuclear Safety Significance...................................................................................... 11 VII. Reports to External Agencies & the NSPM Nuclear Sites ....................................... 14 VIII.Data Analysis............................................................................................................ 15 A. Information & Fact Sources.................................................................................. 15 B. Evaluation Methodology & Analysis Techniques ................................................ 15 C. Data Analysis Summary ....................................................................................... 16 D. Failure Mode Summary ........................................................................................ 16 IX. Root Cause and Contributing Causes ....................................................................... 22 X. Corrective Actions .................................................................................................... 24 XI. References................................................................................................................. 29 XII. Attachments .............................................................................................................. 31 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 3 of 31 I. Executive Summary Problem: The Corrective Action Program at Prairie Island has not consistently resolved significant issues resulting in decreased equipment reliability, operation transients and repeat non-compliance with station procedures and regulatory requirements.
Event Synopsis:
A Focused Self Assessment, documented in AR 1165841, was performed on the effectiveness of the Corrective Action Process. The Focused Self Assessment process discovered numerous examples of Prairie Island failing to consistently resolve significant issues. The Focused Self Assessment was used to develop the problem statement for this Root Cause Evaluation.
Conclusions:
Using root cause evaluation methods, including evaluation of the Corrective Action Process, the team concluded that the Corrective Action Process is not managed effectively. The team identified four significant contributing factors which are driving ineffective resolution of significant issues:
- Work load is greater than available resources.
- Prioritization of individual work load is less than adequate (due date driven). This included the fact that there are no common site priorities to help manage work load.
- Failure to follow procedure (FP-PA-ARP-01)
- Due date is what is measured and monitored, not quality Nuclear Safety Significance:
Cross cutting investigations, self assessments and previous root causes have identified issues; however, the timeliness to resolve these issues has been less than adequate. Additionally, with over 1800 open action items there is a possibility that one or more of those open action items may pose a nuclear, radiological or personnel safety concern.
Root Cause:
- Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
- Contributing Cause #1 - An integrated site priority matrix that interfaces with other high resource plant processes and programs does not exist.
- Contributing Cause #2 - There are no highly visible CAP process measures for quality.
- Contributing Cause #3 - Failure to perform all requirements within the procedure.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 4 of 31 Corrective Action Synopsis:
- Develop and implement a CAP priority matrix similar or identical to the work order priority matrix.
- Develop and implement a Department CAP Health Indicator.
- Develop and implement a Site CAP resolution quality and timeliness Key Performance Indicator.
- Establish management expectations and accountability for CAP process implementation and timeliness of resolution.
- Revise FP-PA-ARP-01 to address identified issues and enhancements.
Reports to External Agencies:
No reports will be made to external agencies. The CAP identifying this issue was posted on the February 17, 2009 NSP Nuclear Department Internal OE Report.
When the RCE is complete and approved by PARB a follow-up posting will be made.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 5 of 31 II. Event Narrative A Focused Self Assessment (FSA), documented in AR 1165841 (D38), was performed on the effectiveness of the Corrective Action Process (CAP). The FSA checklist, QF-0402, provides numerous examples in support of the problem statement for this Root Cause Evaluation (RCE): The Corrective Action Program at Prairie Island has not consistently resolved significant issues resulting in decreased equipment reliability, operation transients and repeat non-compliance with station procedures and regulatory requirements.
What was observed during the evaluation and interview process was a cyclic problem within the identification and resolution process:
Start: Ineffective Problem Description Problem Reoccurs Ineffective Screening Ineffective Review Ineffective Resource Utilization Ineffective Analysis Inappropriate or Ineffective Corrective Actions Other observations indicated that the site has a mindset that if we meet due date and KPIs we are doing well (D27). Other issues identified are procedure-driven. For example, managers and supervisors default to a due date of 30 days/120 days because a longer time requires prioritization, consideration of time required for quality resolution, and documentation of the justification. A successful process makes doing the right thing the path of least resistance.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 6 of 31 The interview and evaluation process revealed that actions to address the CAP process shortfalls alone will not resolve the issues. A successful CAP process needs support and interaction with other site programs, integrated site priorities and management presence reinforcing standards and expectations.
III. Extent of Condition Assessment The extent of condition is defined as the extent to which an identified problem has impacted other plant processes, equipment or human performance. The following areas were considered for extent of condition.
Environment - environment has been determined not to be a factor for this extent of condition.
Equipment - equipment has been determined not to be a factor for this extent of condition.
People - this condition applies to all site personnel, including contractors, who perform any aspect of the corrective action process.
Organization - this condition is not limited to a single department or organization. It applies to all Prairie Island departments.
Process - similar processes include work management, and engineering work management.
A search of the CAP database was performed to determine if past problems have been identified that are associated with the RCE problem statement. The search results were reviewed looking for ARs related to CAP process effectiveness and for safety culture as it applies to the corrective action process.
The following applicable CAPs were found:
- RCE 1117841, WANO AFI OR.2-2 (D9), Managers frequently do not hold personnel accountable for using established processes to achieve high levels of performance. Ineffective or noncompliant use of processes has contributed to insufficient resolution of equipment problems and to delays in addressing emergent issues. Insufficient leadership focus on developing trust and confidence in the value of using established processes contributes to these shortfalls.
- RCE 1141755 (D7), The NRC identified three issues which have been assigned a crosscut to the Evaluation/Extent aspect of the Corrective Action Program component: These three crosscutting issues are as follows:
o TSC Ventilation configuration and functionality o SI-9-5 check valve testing deficiencies o 11 TD AFWP high bearing temperatures
- RCE 1165133 (D15), at the 4th Quarter NRC Exit, Prairie Island received notification of several additional actual and potential NRC violations, each Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 7 of 31 with cross-cutting aspects. Added to the existing NRC violations over the last 4 quarters, there are 3 separate cross-cutting aspects in the NRC Matrix with 3 violations in each aspect. They are in Work Practices -
Oversight, Decision-making - Systematic Processes and Decision-making
- Conservative Assumptions. The purpose of this root cause is to determine the cause of the Human Performance related NRC violations to occur in this quantity, creating the potential for a Substantive Cross-Cutting Issue finding by the NRC.
- Existing actions to address similar issues are contained in the site excellence plan within the CAP Gap and work management action plans that are intended to target and improve human performance with regards to meeting standards and expectations (i.e., understanding and following processes).
Results of Extent of Condition Assessment:
The evaluation of extent of condition has determined that similar issues have been identified and actions are in place to resolve each item. No additional actions are recommended to address other site processes.
Extent of Cause Assessment The extent of cause analysis determines if the root causes of this problem have impacted other plant processes, equipment or human performance. Five distinct areas are considered for extent of cause:
- Environment - environment has been determined not to be a factor for this extent of condition.
- Equipment - equipment has been determined not to be a factor for this extent of condition.
- People - Do the causes impact other personnel (other than those involved) or other human performance issues?
- Organization - Do the causes impact other crews, departments or organizations?
- Process - Are there identical or similar processes or procedures that may be impacted by the causes?
The methods used are to compare in the following manner:
- Identical to Identical (same - same);
- Identical to Similar (same - similar);
Results of Extent of Cause Assessment:
The goal of the extent of cause assessment for the root cause was to determine if the program procedure does not describe the methods required to successfully perform the various aspects of the CAP process. The environment and equipment were determined to not be significant issues for this cause. The evaluation focused on people and processes.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 8 of 31 It was determined that the causes identified are not limited to the CAP process. The discussion of Previous Similar Events illustrates how the existence of these and similar causes have led to significant problems at Prairie Island. It was determined that changes to the CAP process alone would not resolve this type of issues. Effective interface with other site processes and programs is required.
Overall, there were no significant conditions adverse to quality identified for the extent of this root cause.
IV. Previous Similar Events:
RCE01013473, D6 High Crankcase Pressure Resulting in Unit 2 Shutdown (D5) identified three Organizational Root Causes, in brief; inappropriate execution of the CAP program, current set of performance measures and reinforcement tools places a high importance on getting a task completed on time and less importance on the quality of the product, and no robust trending process. Review of the CAPRs and EFRs created reveal multiple failures to perform the actions, EFRs deemed as unsatisfactory in completion notes in Passport, but effective in EFR report [Attachment 10].
RCE 01085806, BKR 16-7, 12 SI Pump Breaker Inoperable (D6) identified a human performance failure in an error-prone process that was a longstanding work practice. Review of the CAPRs shows no corrective action developed to specifically address the organizational failures that allow this, and perhaps other, longstanding work practices.
RCE01099775, High Radiation Area, Locked High Radiation Area, and Very High Radiation Area Controls (D7) identified a root cause of weak enforcement and oversight of Radiation Protection rules and expectations have created a work environment that is more susceptible to radiological events. Review of CAPRs and EFR show failure to complete actions and questionable effectiveness [Attachment 10].
RCE01100615, CAPRs Closure Conflicts with Procedural Requirements (D8) identified previous attempts to resolve the issue have not been fully effective, with human performance failure modes including wrong assumptions, inadequate verifications, inadequate tracking and time and schedule pressure. From the report: Corrective actions to directly address the HU failures were considered, but not implemented. Previous efforts to address the HU aspects were not successful. The root cause cited wrong assumptions, inadequate verification and inadequate task management. The CAPR action cited to result in assurance that a plan exists to facilitate CAPR closure (a pre-planning meeting with requirement that owed to report the results) does not appear to have been proceduralized [Attachment 10].
RCE 01117841, WANO Peer Review AFI OR.2 (D9) states managers frequently do not hold personnel accountable for using established processes to achieve high levels of performance. The root cause identified was disciplined adherence to processes is not a site priority, with three contributing factors; managers have not effectively engaged the workforce to accomplish Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 9 of 31 site goals, priorities, or improvement initiatives, work groups do not clearly understand their roles and responsibilities and how to effectively work together to resolve issues, and managers have not set clear direction and priorities that are understood by the workforce, and have not aligned the organization to achieve common goals. This effort is ongoing, due April 2010.
RCE01141755-01, Identified NRC Crosscutting Issues (D7) concludes that the operational philosophy currently in place relies on skill sets and knowledge that no longer exist within the station and the organization is not placing appropriate focus on plant issues, strategies, and/or the appropriate priority when they are identified. The root cause identified is the roles and responsibilities previously held by Engineering to address plant issues have not been effectively transferred to Operations to promote a strong operational focus with contributing causes cited; high workload without proper prioritization and lack of critical skills throughout the organization. The created CAPRs, still in process, are to revise the ODMI procedure and train Operations and Engineering. CAs are in process to address the contributing causes. This effort is ongoing, due June 2010.
Results of Previous Similar Events Assessment:
The results of the previous similar events review and assessment validate the problem statement associated with this RCE: the corrective action program at Prairie Island has not consistently resolved significant issues.
V. Operating Experience:
Internal OE Assessment A review of the July-December 2008 Site Drum Report (D20) for missed opportunities and Internal OE provided insight that some issues are recognized in the Site and Departmental DRUM reports.
The Site DRUM Report concluded the station has communicated the vision for improved performance, but this has not been translated to improved individual contributor level performance. The execution of this vision may be attenuated by conflicting directions i.e. difficulty in prioritizing work activities.
The Issues that were identified in the DRUM Report are:
- The work load combined with limited qualified resource availability fosters due date driven, shallow evaluation/work completion which results in rework. RCE interviews confirmed this (D27).
- The KPI for event clock reset rate does not align with input from others such as the NRC. The KPI describes station performance as exceeding expectations.
- The corrective action program overall health is adequate with significant opportunities for improvement, including management of CAP inventory (particularly by the Engineering department) and Root/Apparent Cause evaluation grades.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 10 of 31
- Program challenges include addressing increasing CAP and CAP action inventory and addressing/evaluating issues with sufficient detail to eliminate ineffective and untimely corrective actions.
- Actions for previously identified trends are not completed. A FSA (D38) indicated 15 of the 20 AFIs were deemed to be off track.
- There were no new trends identified in this report; however there were three (3) outstanding trends from the previous report.
The Site Drum Report identified Actions to improve Human Performance.
These actions include:
- Establish and promulgate management expectations for supervisory oversight of pre-job briefings.
- The 2009 Site HU Improvement Plan was developed during the fourth quarter. Individual Departments are designed to support the site plan.
Results of Internal OE Assessment The RCE team came to the conclusion that individual departments recognize their issues and document them in the DRUM Reports, however, the effectiveness of resolution is not fully understood nor corrective actions completed (D20). Four (4) of the six (6) issues identified in the report align with and support the conclusion that the failure mode F6 (Inadequate Program Management) is a primary driver. This is consistent with the evaluations performed as a function of this RCE.
External OE Assessment Based on the RCE Charter problem statement a search was made of the Institute of Nuclear Power Operations (INPO) OE Database using the following keywords: safety culture and problem reporting and Event Descriptor: corrective action (PI). Results of the queries were reviewed for applicability to this RCE. The following summary from Harris was found to have a problem statement and causes closely related to this RCE.
OE27290 Harris identified that some important or longstanding problems related to organizational deficiencies are not driven to resolution through CAP implementation (D40).
ABSTRACT: In some cases, organizational aspects of problems are not adequately resolved through the corrective action program. Harris has failed to recognize opportunities for organizational learning. Evidence of missed learning opportunities exists in our performance indicator and system health reviews, our procedural guidance for investigation depth, and our oversight of self-evaluation.
CAUSES: The selected cause is that even though HNP has a bias for action, we are missing opportunities to learn. We need to identify typical situations that are indicative of a learning opportunity, to ask "how did we allow this to happen". Evidence of missed learning opportunities exists in our performance Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 11 of 31 indicator and system health reviews, our procedural guidance for investigation depth, and our oversight of self-evaluation.
Results of External OE Assessment Review of the above OE by the team determined that additional information from Harris would be helpful in determining actions to address the Prairie Island issue. Harris was contacted and a copy of their root cause evaluation (D45) was obtained. The RCE teams review of their identified causes and corrective actions was used as input in formulating actions to address the issue at Prairie Island.
VI. Nuclear Safety Significance For the purpose of keeping this summary to a desirable length, instances and examples of attribute deficiencies may be found in Focus Self Assessment Report, PI-FSA-09-01, CAP AR# 01166830 and the charter statement of this root cause evaluation report (D2, D4)
There is a potential for nuclear and radiological safety challenges due to the ineffective and untimely corrective actions. Some examples are:
- The reliability of D5/D6 Emergency Diesel Generators that supply power to safety related equipment upon a loss of offsite power. Although complete now, the resolution of this issue was not timely.
- TSC habitability in the event of an emergency at the station due to ventilation issues. Although complete now, the resolution of this issue was not timely.
- Radiation monitoring repetitive failures could affect EAL classification and subsequent actions to protect the health and safety of plant personnel and the public.
Cross-cutting investigations, self assessments and previous root causes have identified issues; however, the timeliness to resolve these issues has been less than adequate. Additionally, with over 1800 open action items there is a possibility that one or more of those open action items may pose a nuclear, radiological or personnel safety concern.
- SCWE (D21, D29, D30, D32)
Safety Conscious Work Environment is defined as an environment in which employees feel free to raise safety concerns, both to their management and to the NRC, without fear of retaliation and where such concerns are promptly reviewed, given the proper priority based on their potential safety significance, and appropriately resolved with timely feedback to employees.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 12 of 31 With the above-mentioned potential safety significant items, the larger scope of the Principles for a Strong Nuclear Safety Culture needs to be discussed further. A Safety Culture is characterized by an organizations values and behaviorsmodeled by its leaders and internalized by its membersthat serve to make nuclear safety the overriding priority.
As a result of the interviews performed the team concluded that the individual contributors felt comfortable bringing up concerns with their supervisors and management. This meets the first part of the definition of SCWE.
Some of the attributes pertinent to this Root Cause Evaluation are:
- Managers and supervisors practice visible leadership in the field by placing eyes on the problem, coaching, mentoring, and reinforcing standards. Deviations from station expectations are corrected promptly.
Interviews with individual contributors revealed a perception that managers and supervisors are not available to address questions, standards and priorities. With approximately 700 hours0.0081 days <br />0.194 hours <br />0.00116 weeks <br />2.6635e-4 months <br /> of management observations per month the validity of this perception comes into question. Reinforcing the perception that supervisors are not available is a cross-cutting issue with Human Performance inadequacies. The team determined that part of the issue is the definition of field observation.
Observing work in the field is usually considered watching an operator perform a surveillance test or a maintenance technician repair a pump.
Engineering work is usually done in an office environment but consists of work behaviors. As such the site management team needs to realize that observations in the field may constitute observing their direct reports in their cube or in the library. The result of manager/supervisor unavailability is that completion of A and B level corrective actions is hurried because of due date induced time pressure. (D27).
- The organization maintains a knowledgeable workforce to support a broad spectrum of operational and technical decisions. Outside expertise is employed when necessary.
There is evidence that the work force has changed and a loss of knowledge and skills has resulted. This is evident in the performance of apparent cause and root cause evaluations. A lack of knowledge of the problem sometimes causes the evaluation to be incomplete or incorrect.
A lack of knowledge of evaluation techniques has allowed less than adequate root cause evaluations resulting in as many as three different teams being chartered to address the same problem (D44 RCE01157726
- Rad Shipment).
- Leaders recognize that production goals, if not properly communicated, can send mixed signals on the importance of nuclear safety. They are sensitive to detect and avoid these misunderstandings.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 13 of 31 There was a consistent message from interviews that while quality was a concern of the individual in all their work activities, meeting the due date of a CAP corrective action became priority. The evaluation would be performed in a format to meet grading requirements; however, the result is a lack of in-depth evaluation of A and B level corrective actions. This is symptomatic of manager/supervisor unavailability to ensure that questions are answered and standards and priorities are understood and maintained. (D27)
- The bases, expected outcomes, potential problems, planned contingencies, and abort criteria for important operational decisions are communicated promptly to workers.
This attribute was not evident in interviews (D27) or document reviews (refer to reference list.)
- Senior management incentive programs reflect a bias toward long-term plant performance and safety.
On the contrary the incentives were based on work completion within the CAP by the due date. At times this prejudiced the completeness of problem evaluation and adequate corrective action development (D27).
- Decision-making practices reflect the ability to distinguish between allowable choices and prudent choices.
There is indication that conservative decision making for equipment issues allow addressing the symptoms such that they go away for the time although the cause is not addressed by corrective actions that restore the system to its original design or restore the margin of safety (refer to reference list).
- Plant personnel apply a rigorous approach to problem-solving.
Conservative actions are taken when understanding is incomplete.
There is strong supporting evidence that after the designation of a severity level and appropriate evaluation assignment, resources and management sponsors are not rigorously involved and approved procedures and processes not adhered to (D27).
- Single-point accountability is maintained for important safety decisions, allowing for ongoing assessment and feedback as circumstances unfold.
There is sufficient evidence that this attribute is not occurring. The failure to implement timely corrective actions or to address top ten items supports this (D27).
- Equipment is meticulously maintained well within design requirements.
In accordance with the charter problem statement this is not evident.
- Anomalies are recognized, thoroughly investigated, promptly mitigated, and periodically analyzed in the aggregate.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 14 of 31 Anomalies are recognized and documented in the CAP process but the actions are not promptly taken to mitigate the problem. There has been no discussion on the aggregate impact of longstanding degraded equipment or processes that are not effective.
- Workers identify conditions or behaviors that have the potential to degrade operating or design margins. Such circumstances are promptly identified and resolved.
Symptoms are addressed and corrected without restoring the design margin of safety. The concept of reduced margin of safety did not frequently come up in interviews (D27). Promptness is not evident as supported by the problem statement.
- Individuals are well informed of the underlying lessons learned from significant industry and station events, and they are committed to not repeating these mistakes.
Industry Operating Experience is not used proactively. Interviews determined that adequate incorporation of distributed OE was lacking due to work load in other areas (D27). Expectations for using OE are not being reinforced by first-line supervisors IAW FP-PA-OE-01, Attachment 7, Expectations for Using OE (D47).
- Expertise in root cause analysis is applied effectively to identify and correct the fundamental causes of events.
The training and qualification for root cause evaluators has not been an ongoing process. Some of the qualified personnel have not participated in Root Cause teams in several years and have had no refresher training.
Lack of highly qualified RCE personnel has resulted in PARB frequently requesting changes to drafted RCEs.
- Employees have confidence that issues with nuclear safety implications are prioritized, tracked, and resolved in a timely manner.
All personnel interviewed agreed with the problem statement of this RCE. Most suggested solutions that pointed to a lack of faith in the organization to provide the resources and tools to correct the problem.
This perception was determined to be based on high work load, conflicting priorities and the lack of timely resolution of issues. (D27).
VII. Reports to External Agencies & the NSPM Sites No reports will be made to external agencies. The CAP identifying this issue was posted on the February 17, 2009 NSP Nuclear Department Internal OE Report. When the RCE is complete and approved by PARB a follow-up posting will be made.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 15 of 31 VIII. Data Analysis The Root Cause Evaluation team was chartered and began work on March 19, 2009 (D4).
The multidisciplinary 6 person effort (including management sponsor, and an outside technical expert) comprised more than 167 years of work experience.
The team spent more than 770 person hours: 1) reviewing documents (CAP FSA, CAPs, procedures, selective previous RCEs and ACEs); 2) making observations of AR Screening Team Meeting and AR pre-screening; and 3) conducting interviews.
The team employed failure modes analysis methods to evaluate human performance, process, and organizational issues. The sites corrective action trend code manual, containing tables for human performance, process, and organizational and management failure modes, was used to bin the interview and evaluation results. The tables were also used to clarify and focus information from the various interviews and evaluations. Additionally, the why staircase was applied to the inappropriate actions identified in the interviews, observations and failure modes analysis. The results were then evaluated based on the AR process flow diagram to determine the areas where inappropriate actions and evaluation results indicated problems exist. The AR process review led to a barrier analysis based upon the procedural instructions.
Review, compilation of results, and evaluation provided insight to determine the root cause and significant contributing causes.
A. Information & Fact Sources Twenty-nine employees were interviewed ranging from managers to administrative assistants [ref. Attachment 2]. A standard set of questions was used to gain an understanding of motivators and drivers associated with the corrective action process and job task performance [ref.
Attachment 3]. Additionally AR 01165841, Focused Self Assessment (FSA), Determine the stations ability to effectively implement the corrective action program/process, RCE 01117841-01, WANO Peer Review AFI OR.2-2, RCE01141755-01, Identified NRC Crosscutting Issues, RCE01013473, RCE 01085806, RCE01099775, and RCE01100615, were reviewed and analyzed (D27 and D28).
B. Evaluation Methodology & Analysis Techniques Because of the nature of this RCE a typical Event and Causal Factor chart could not be used. Instead an AR Process Flow Analysis was used (Attachment 9). The team started with a review of the AR Process FSA (D38), previous RCEs and other documents identified above. The team developed a set of interview questions in an effort to determine motivators and drivers that would indicate where process shortcomings existed. The team compiled the information from the various sources, populated the information into Failure Modes Analysis work sheets, and developed a set of inappropriate actions for analysis using the WHY Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 16 of 31 staircase. Corroborative conclusions by both methods of analysis would be indicative of the root cause. Final analysis was to pinpoint the problems and conclusions on an AR process flow analysis (Attachment
- 9) to determine focal points for barrier analysis. The AR process review led to a barrier analysis based upon the procedural instructions. Review, compilation of results and evaluation provided insight to determine the root cause and significant contributing causes.
C. WHY Staircase Summary (Attachment 7)
After review of AR 1165841, Focused Self Assessment on CAP program, and completion of the interview process, the team identified several inappropriate actions. The inappropriate actions identified were:
- IA-1, CAP AR initiators are not providing sufficient detail in the action request as required by FP-PA-ARP-01, Section 5.2.
- IA-2, Screening Team does not effectively /consistently apply the attributes of a strong nuclear safety culture.
- IA-3, Plant personnel are driven by due date without proper focus on problem solving and resolution.
- IA-4, Effective corrective actions are not consistently identified, or if identified, are not consistently completed.
- IA-5, Completed A Level actions and B level CAPs are not consistently reviewed and are not reviewed in a timely manner.
Following the why logic the team determined that the underlying causes are as follows:
- 1. The Plant focus on raw data and KPIs overrides quality of corrective actions. KPIs are a known measurable item and the plant staff strives to meet expectations. This raises the question, Are we measuring the right stuff?
- 2. Inadequate program management and inadequate interface with other processes drive competing priorities and allow plant staff to focus on due date vs. working what potentially is the right stuff.
Both underlying causes can have the unintentional effect of:
- A and B level CAPs action evaluation may be compromised due to the volume of work and due date on lower value items.
- Completed A and B level CAPs actions are not consistently reviewed and are not reviewed in a timely manner.
This is symptomatic of manager/supervisor unavailability to ensure that questions are answered and standards and priorities are understood and maintained.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 17 of 31 D. Failure Mode Summary The Failure Mode analysis examined three areas: Human Performance, Organizational and Management, and Process. The failure modes are from the Fleet CAP Trend Code Manual, FG-PA-CTC-01, associated with the CAP Program.
Note: Only the more significant applicable failure modes are discussed in this section. The attached tables provide information on other failure modes determined to be applicable, yet have fewer or less significant supporting facts.
Process Failure Mode (Attachment 4)
Process failure modes were evaluated based upon AR Process FSA (D-38), previous RCEs and personnel interviews. Team input from the various data sources identified four (4) key process failures:
- No process monitors [AR3]. Through interviews the team discovered that effective resolution of issues the first time was foremost on the mind of every interviewee. However, all interviewees indicated that the primary focus was on meeting the due date without consideration to priority. It was reported and verified through review of the procedure and Key Performance Indicators (KPIs) that due date is what is monitored and due date is where the workers focus. Line personnel reported the desire to go the extra distance to ensure effective resolution but high work load and competing plant priorities kept the focus on solving the symptoms and meeting due dates. Process monitors for quality do not exist or are not present in the minds of the PI staff. Resolution of A and B level ARs is addressed through grading. Grading for A and B evaluations is at the front end of the process. Grading at the front end does not adequately measure if the evaluation and resolution effectively resolved the issue. The FSA for the CA process also concluded that grading is not being effective at addressing quality resolution of long term problems.
- Only monitoring problems (late due dates) [AR4]. During the interview process the key indicator referenced by interviewees was late due dates. It was noted by several interviewees that the only item which generated a personal interview with the site VP is being late on an AR action. When asked about A and B level AR grading, interviewees indicated that they were managing their work to the degree of what does it takes to get a passing grade? The focus is on completing action items, not quality.
- Actions not tied to another process when necessary [RR5].
Discussion with interviewees indicated that competing site priorities factored heavily in what work they were doing. Emergent plant issues draw resources from across the organization regardless of the departments primary roles and responsibilities. The organization Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 18 of 31 pulls in the expert to address issues. This cultural tendency to use the go to person significantly impacts the level of effort used to address AR evaluations and corrective actions.
- Actions conflict with another process [RR4]. Discussion with interviewees included individual work load and work priorities.
Many examples were discussed in which work management priorities did not align with the AR process priorities. Work management priorities have been improved to address the various aspects that impact equipment reliability and availability including:
o Work Priority 1 through 5 based on FP-WM-WOI-01, Work Identification, Screening, Validation and Cancellation, work significance.
o System Health Code 1 through 4 based on FG-E-02, Equipment Reliability Performance Indicators.
o PM Critical Indicator based upon FP-E-SE-02, Component Classification.
However the work management priority system does not interface with the CAP system. This leaves a void in every departments ability to effectively prioritize individual work tasks. Management defaults to crisis management, and workers default to due date.
Interviewees report that what gets worked depends on who is asking and the perceived urgency. This results in a highly motivated owner using personal networking to get some work done prior to potentially more important work.
Organizational and Management Failure Modes (Attachment 5)
Organizational and Management Failure Modes were evaluated based upon AR Process FSA (D-38), previous RCEs and personnel interviews.
Team input from the various data sources identified the following key organizational and management failures:
- F-1, Inadequate Communication within Organization Poor problem statements written by the initiator and an inadequate review by the supervisor are the first steps in the process breakdown.
Without manager/supervisor visibility with their eye on the problem within the department, production priorities and quality fall to the performer. The result is due date driven, with a target of meeting the minimum requirements.
- F-2, Inadequate Communication among Organizations Inadequate Communication among Organizations results in competing priorities, confusion, delays and rework. In any organization that has their resources stretched to the limits, confusion and rework is unacceptable.
- F-3, Inadequate Prioritization Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 19 of 31 Many examples were discussed in which work management priorities did not align with the AR process priorities. Work management priorities have been improved to address the various aspects that impact equipment reliability and availability including, Work Priority 1 through 5, System health Code 1 through 4, and PM Critical indicator. However the work management priority system does NOT interface with the CAP system. This leaves a void in every departments ability to effectively prioritize individual work tasks and results in competing priorities between departments.
- S-3, Insufficient Staffing.
Every interviewee from manager to front line employee reported that the work load is greater than available resources. High work load combined with a less than adequate priority matrices results in high frustration and creates the potential for ineffective and untimely resolution of issues.
- F-6 Inadequate Program Management.
Analysis indicates that F1, F2, F3 and S3 are all indicative of F6, Inadequate Program Management. Each of these failure modes is driving the other in an ever increasing competition for resources and resolution.
Human Performance Failure Mode (Attachment 6)
Interview data was used to populate the Human Performance Failure Mode table. However, upon evaluation of the data in conjunction with the RCE charter it was determined that all the applicable HU failure modes supported the Organizational and Management and Process Failure Mode analysis. No further analysis was performed.
Failure Modes Evaluation Conclusion (Attachment 8)
The organizational and management failure modes and the process failure modes were charted to evaluate drivers and impact. The chart shows that the underlying cause is inadequate program management which is impacted by five (5) key drivers (see Attachment 8):
- 1. F-2, Inadequate Communication among Organizations
- 2. F-3, Inadequate Prioritization
- 3. S-3, Inadequate Staffing
- 4. RR-4, Actions Conflict with Another Process (Priorities)
- 5. RR-5, Actions not Tied to Another Process when Necessary (Priorities)
The net result is that the lack of CAP process interface with other plant processes which are all in competition for limited resources results in inadequate prioritization of individual work load and ineffective process management.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 20 of 31 E. AR Process Flow Analysis (Attachment 9)
The data and conclusions from the WHY and the Failure Modes Analysis were evaluated and applied to the AR Process Flow Chart (see Attachment 9). This analysis identified three key failure points contributing to inadequate program management: initial problem identification, screening, and completion reviews. The evaluation then applied the elements of a strong safety culture in an effort to determine where the site is falling short of the mark. Results are as follows:
- Initial problem identification -
Attachment 13 of FP-PA-ARP-01 states that the problem statement in a CAP should be a clear statement of facts that includes the standard of performance, the observed or discovered performance, and how the performance deviates from the standard. Personal opinions not supported by fact should be avoided. Interviews indicated that this is almost never what the screening team sees. The Team concluded that inadequate problem statements and supporting information starts the CAP process off poorly because evaluations are focused on the problem statements and inadequate problem statements result in symptom solving, not problem solving. Poor problem statements written by the initiator and reviewed by the supervisor are the first step in the process breakdown. If the initial CAP write-up is less than adequate, the supervisor should return it to the initiator for clarification.
- Screening -
Poor problem statements and inadequate research or misunderstanding the problem results in screening results that are less than effective. Failure to understand the problem has resulted in wrong actions assigned, inadequate evaluations and repeat issues.
Although the number of holdovers may be increasing, the screening team seems reluctant to send things back for clarification. Instead of taking a best guess and compromising quality a poor problem statement should be returned to the supervisor. The number of holdovers and returns should be factored into CAP quality.
- Completion Reviews and Grading -
The focus during closure review and grading of A and B level CAPs does not appear to ask: Is the problem fully resolved?
Problems statements that are not always fully understood and organizational focus on meeting due dates result in problems that are not fully resolved. Completion reviews do not tie everything together and look beyond symptom solution. Effectiveness reviews can be removed from the problem by months or even years, only to find at the closeout review that we missed the boat completely. Failure to understand the problem results in inadequate evaluations, ineffective corrective actions and repeat issues. Completion reviews that Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 21 of 31 validate the symptom are resolved keep us in a circular pattern that results in repeat issues.
By shifting our focus to the attributes of a strong safety culture as they apply to job task execution during AR process implementation the plant will see an improvement in problem resolution and equipment availability and reliability.
F. CAP Process Procedure Barrier Analysis (Attachment 11)
NOS identified an adverse trend in CAP resolution of significant issues with examples in the four key CAP process elements, Issue Identification, Issue Screening, Issue Evaluation, and Issue Resolution/Timeliness of Resolution. The CAP procedure, FP-PA-ARP-01, and failure to adhere to the procedure, could be contributing to these significant issues. Evaluation of the procedure and areas for improvement are itemized in Attachment 11.
- Issue Identification -
Expectations for the level of detail to be included in the problem statement are clearly stated. The procedure defines five separate functions responsible for insuring the problem statement meets the stated expectations. Failure to follow the procedure in this area is causing inadequate problem statements.
- Issue Screening -
Several steps in the procedure provide direction for the screening team to reduce the number of actionable CAPs and focus site efforts on the most significant issues. The screening process is not screening out CAPs that do not need to be written, CAPs with similar issues where one can be closed and cross-referenced to the second, and CAPs that can be changed to a non-CAP. Failure of the screening process to require the Initiator and Initiators Supervisor/Manager to provide a high-quality problem statement inhibits recognition of the significance of problems and level of effort required. Time requirements are emphasized. Multi-disciplinary teams should, but dont appear to be, considered for evaluations other than RCEs.
Notes are not consistently being added addressing SCAQ, CAPR, and EFR requirements for all A CAPs. The composition of the screening team is not concretely defined to ensure the necessary knowledge and experience is available at screening team meetings.
- Issue Evaluation and Issue Resolution/Timeliness of Resolution -
The procedure focuses on due date over quality. The only KPI referenced is for timely resolution of conditions adverse to quality, found twice in the procedure. FP-PA-ARP-01, Revision 21, Attachment 13, page 80 of 85, is the first and only mention of the existence of standards of quality required to be met by evaluations and corrective actions in the CAP procedure. There is no reference Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 22 of 31 or definition of those established standards. The Technical Review Panel results are not being used as a quality of CAP action KPI.
The default due date is prescribed by the procedure. Consideration of the time required to accomplish an effective evaluation and corrective actions is not emphasized, measured, nor reinforced.
IX. Root Cause and Contributing Causes On the surface the problem statement would indicate that the cause of this condition is simply weak or inadequate evaluation of identified problems.
However, during the interview process all of the individual contributors interviewed clearly understood analysis, extent of cause and extent of condition. All the personnel interviewed indicated a desire to drive problems to completion. With this realization the Team needed to determine what the drivers are that prevent the site from consistently resolving significant issues.
The RCE problem statement would also indicate that the CAP process is broken and all we need to do is fix the procedure. The evaluation determined there are instances of less than adequate procedure use and adherence; therefore a determination could not be made that the process is inadequate.
The Procedure Review Barrier Analysis identifies areas for improvement within the procedure.
Working through the identified RCE evaluation methods including evaluating the CAP process the team concluded that the CAP process is not effectively managed. The team looked for the drivers that would be indicative of the root cause, and concluded there are four significant contributing factors driving ineffective resolution of significant issues:
- Work load is greater than available resources.
- Prioritization of individual work load is less than adequate (due date driven). This included the fact that there are no common site priorities to help manage work load.
- Failure to follow procedure (FP-PA-ARP-01).
- Due date is what is visibly measured and monitored, not quality.
The interesting thing about these contributors is that they drive each other.
Work load is high due to numerous functional areas in each department, all demanding resource time. With no effective way to integrate priorities between the work management system, engineering work management system and the CAP process, due date becomes the controlling factor. Failure of supervision to effectively prioritize individual work load results in each individual controlling their work based upon due date. This prioritization is also seen in not removing items from the work load which provide limited value to the site. All work essentially becomes equal with due dates being the only differentiator. Coupled with this, the most visible indicator of CAP progress is the due date. Although some quality indicators are in place, they have limited visibility to the process implementers. For more significant items on which work is started late, challenges to quality have occurred when Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 23 of 31 there is a last minute rush to complete the work. This creates a potential to miss procedural closure requirements. The four contributors have a spiraling effect that can be seen in backlogs increasing and repeat issues in the CAP process.
Root Cause:
The team concluded that failure to achieve effective and timely problem resolution is due to inadequate program management and a weak nuclear safety culture. This is primarily the result of:
- Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
- Contributing Cause #1 - An integrated site priority matrix that interfaces with all plant process and programs does not exist.
- Contributing Cause #2 - There are no highly visible CAP process measures for quality.
- Contributing Cause #3 - Failure to perform all requirements within the procedure.
The team concluded that the lack of integrated priorities results in C level items done at the expense of A and B level items which is indicative of a weak safety culture. The team determined that the principals are understood, however they are not demonstrated. The principles for a Strong Nuclear Safety Culture would indicate that an organizations values and behaviors, modeled by its leaders and internalized by its members, serve to make nuclear safety the overriding priority. The purpose of the corrective action process is to ensure that conditions adverse to quality are identified and effectively resolved in a timely manner.
Based on the evaluation of the safety culture attributes as discussed in Section VI, and the results of the various evaluations performed by the RCE, the team concluded that there is a breakdown in three of the attributes:
- 1) (attribute 1) Everyone is personally responsible for nuclear safety.
- 2) (attribute 2) Leaders demonstrate commitment to safety.
- Managers and supervisors practice visible leadership in the field by placing eyes on the problem, coaching, mentoring, and reinforcing standards. Deviations from station expectations are corrected promptly and,
- Leaders recognize that production goals, if not properly communicated, can send mixed signals on the importance of nuclear safety. They are sensitive to detect and avoid these misunderstandings.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 24 of 31
- 3) (attribute 4) Decision-making reflects safety first.
- Leaders recognize that production goals, if not properly communicated, can send mixed signals on the importance of nuclear safety. They are sensitive to detect and avoid these misunderstandings.
- Decision-making practices reflect the ability to distinguish between allowable choices and prudent choices.
Examples:
- Manager presence in the associated work area is less than needed resulting in employees setting their own priorities.
- CAP and work management priorities compete for limited resources.
- The engineering work management system does not interface with the plant work management system for establishing priorities and controlling work load.
- RCE grading routinely fails to meet quorum requirements.
- RCE team member assignments are not respected.
These attributes are reflected in an organization that is focused and has a set of clear standards and expectations, where management presence and interface with their staff is routine and ongoing, and success is measured by effective results and resolution of actions. As discussed above and identified in the evaluations and interviews, management availability is limited, priorities are unclear, changing and competing, work is due date driven and focused on the KPIs that measure due date not quality. Addressing the root and contributing causes will strengthen our safety culture in this area.
X. Corrective Actions Corrective Actions to Restore (broke-fix)
- No corrective actions required. This was not an equipment issue.
Interim Corrective Actions (mitigation)
- CA #1 (01166830-04) - Develop and implement a Site CAP resolution quality and timeliness Key Performance Indicator. The indicator should be a weighted formula based upon quality, due date and how long a CAP has been open. TRP results would be a key input to this indicator.
Responsible Group: Fleet CAP Manager, Completion Due Date: August 21, 2009 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 25 of 31
- CA #2 (01166830-05) - Establish management expectations and accountability for the following:
- PARB to define required departments for each root cause grading to ensure effective cross-organizational review.
- Change TRP review of completed actions to include.
o Review of Level A actions, not just the final CAP o A random sample of completed B evaluations/actions, using total population of completed B actions, Sample Size Determination Table (FP-G-CD-01 Attachment 1) and random number generator.
o Identify quality resolution / recognition
- The A and B Owed To is accountable for quality and timeliness.
- Institute a 14 day owed to review of all level A actions as they are closed
- Communicate all RCE issues and results to the site.
This is an ongoing activity that should be monitored through the paired observation program.
Responsible Group: Business Support Manager, Completion Due Date: June 15, 2009
- CA #3 (01166830-06) - Revise FP-PA-ARP-01 to address the following issues:
- Clearly state what function is responsible for writing the evaluation assignment statement - Section 5.5, 5.6, 5.7.
- Designate between SCAQs, CAQs and non-CAQs as part of the screening process and documented priority.
- Attachment 9 is only referenced in step Section 5.10, but includes information relevant to multiple other steps. Provide reference to Attachment 9 in all applicable sections.
- Identify Responsible Individual(s) in left column - Section 5.10.
- Screening charter states determine the level of effort. Clarify statement to state meaning of that phrase - determine if RCE, ACE, CE, or CA is needed.
- Composition of screening team allows minimum quorum (shall) that may not include representatives from appropriate departments (should) - Attachment 8. Re-evaluate quorum requirements such that all appropriate departments are represented.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 26 of 31
- Change Attachment 10, TRP Charter to require review of all closed A actions (not waiting for completion of CAP), and a random sample of completed B evaluations/actions (not waiting for completion of CAP), using total population of completed B actions, Sample Size Determination Table (FP-G-CD-01 Attachment 1), and random number generator. Add random number generator, Sample Size Determination Table to procedure.
- Change Attachment 10, TRP Charter to proceduralize chart, currently viewed only by TRP team, to be presented formally to PARB. Revise chart to use actions as units on y-axis.
Responsible Group: Fleet CAP Manager, Completion Due Date: June 30, 2009
- CA #4: (01166830-07) - Complete a site rollout of the Principles for a Strong Nuclear Safety Culture. This should include continuing exposure at Management meetings, Leadership Forums, D15 Team Notes write-ups, and observations, mentoring and coaching at all levels of management and supervision. (Closeout will be based on initial rollout. Use FSA to validate.)
Responsible Group: Pride Initiative Manager Completion Due Date: July 30, 2009
- CA #5 (01166830-08) - Once CAP priority matrix has been created, complete a department led review of current AR backlog to apply the new standards, and prioritizing / managing employee work load.
Responsible Group: Business Support Manager, Completion Due Date: June 20, 2009 Corrective Actions to Prevent Recurrence (CAPRs)
- CAPR #1 (01166830-09) - Develop and implement a CAP priority matrix designed to interface with the work management process and the engineering work management system.
Responsible Group: Business Support Manager, Completion Due Date: May 20, 2009
- CAPR #2 (01166830-10) - Develop and implement a Department CAP Health Indicator. The indicator should be a weighted formula based upon quality, due date and how long a CAP has been open.
Responsible Group: Fleet CAP Manager, Completion Due Date: June 15, 2009
- CAPR #3 (01166830-15) - Provide Root Cause Evaluation refresher training to all qualified RCE personnel.
Responsible Group: Business Support Manager, Completion Due Date: June 15, 2009 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 27 of 31 Other Corrective Actions
- CA #6 (01166830-11) - Change default sort settings for AT-0085 Supervisors Report to severity level, then due date. Initiate an ITAR and track to closure with this CA.
Responsible Group: Performance Assessment Supervisor, Completion Due Date: June 15, 2009
- CA #7 (01166830-12) - Change the Corrective Action Effectiveness Indicator (MRM) meets standard to 100%
Responsible Group: Fleet KPI Manager, Completion Due Date: June 15, 2009 Effectiveness Reviews:
- EFR #1 (01166830-13) - Complete a FSA on CAP process effectiveness based on EFR results of severity level A and B CAPs closed in the last half of 2009, newly developed site and department quality indicators, and NOS audit reports. Success will be indicated by: all CAPs are prioritized with the new codes, the new quality indicators are being utilized by departments to drive performance (suggest interviews of employees), and the new indicators are blue or green trending to blue for 1st quarter of 2010.
Responsible Group: Business Support Manager, Completion Due Date: March 31, 2010
- EFR #2 (01166830-14) - Complete a FSA on CAP process effectiveness by evaluating EFR #1 results, site and department quality indicators, and NOS audit reports. Success is effective resolution of significant issues as indicated by: increased trend in equipment reliability, reduced operation transients and no repeat non-compliance with regulatory requirements.
Measurement will be reflected in the MRM KPIs indicating Blue for the identified areas. Due date 4th quarter of 2010.
Responsible Group: Business Support Manager, Completion Due Date: December 31, 2010 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 28 of 31 Corrective Action Matrix Root Cause / CAPR / IA / FM Contributing Cause EFR / CA RC CAPR #1 IA-2, IA-3, IA-4, & IA-5 CAPR #2 F-2, F-3, F-6, RR-4, RR5, AR-3, CAPR #3 AR-4, & S-3 CA #5 EFR #1 EFR #2 CC #1 CAPR #1 IA-3 & IA-4 CA #5 F-2, F-3, F-6, RR-4, RR-5, & S-3 CC #2 CAPR #2 IA-3, IA-4, & IA-5 CA #1 F-2, F-3, F-6, AR-3, AR-4, RR-4, CA #2 RR-5, & S-3 CA #4 CC #3 CA #3 IA-1 & IA-5 F-2, F-3, F-6, AR-3, AR-4, & S-3 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 29 of 31 XI. References Doc # List Date/Rev 01/30/2009 Rev 1 FP-PA-ARP-01. CAP Action Request Process 21 CAP01166830, SCAQ -Inadequate CAP Resolution of Significant 2 01/26/09 Issues 3 FP-PA-RCE-01, ROOT CAUSE EVALUATION MANUAL 7/30/2008 Rev 14 4 RCE01166830 Charter RCE01013473, D6 High Crankcase Pressure Resulting in Unit 2 5 02/03/2006 Shutdown 6 RCE01085806, BKR 16-7, 12 SI PUMP Breaker Inoperable 04/03/2007 7 RCE01141755-01, Identified NRC Crosscutting Issues 07/02/2008 RCE01100615, CAPRs Closure Conflicts with Procedural 8 06/15/2007 Requirements 9 RCE01117841, WANO Peer Review AFI OR.2-2 01/02/2008 10 FP-PA-ARP-01. CAP Action Request Process Rev 19 11 FP-PA-ARP-01. CAP Action Request Process Rev 17 12 FP-PA-ARP-01. CAP Action Request Process Rev 15 13 AR01139790, PU.02.E.2.a, Work Initiation and Screening 06/03/2008 RCE01099775, High Radiation Area, Locked High Radiation Area, and 14 09/07/2007 Very High Radiation Area Controls 15 RCE01165133, Human Performance & Cross-Cutting NRC Violations 03/25/2009 OE28449 - Procedure Technical Content Issues (Palo Verde Nuclear 16 05/11/2007 Generating Station)
OE28283 - Corrective Action to Prevent Recurrence (CAPR) from a 17 01/06/2009 Root Cause was incorrectly documented as being complete (Pilgrim)
OE22639 - NRC Identifies Crosscutting Issue In The Area Of Problem 18 03/02/2006 Identification And Resolution - Columbia OE16574 - Lack of Completeness of Information Provided in Problem 19 06/02/2003 Identification and Resolution Documents at Davis Besse 20 Site Drum Report July -December 2008 02/22/2009 CERTREC - NRC Reactor Oversight Process - Safety Culture 21 2009 Vulnerabilities 22 AT-0075 AR Screening Report - 3/30/2009 03/30/09 23 AT-0075 AR Screening Results - 3/30/2009 03/30/2009 ACE01163206, A CAP action due date extension was appropriately 24 requested but not granted NRC Reactor Oversight Program Substantive Cross-Cutting Issues 25 03/04/2009 2008 Annual Assessment Letters 26 PINGP Screen Team Standard Agenda 03/30/2009 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 30 of 31 Doc # List Date/Rev 27 Interview Notes (all) 28 Interview Questions 29 Sorting Out Safety Culture (The Nuclear Professional) 30 The Culture Question (The Nuclear Professional) 31 INPO 05-003, Performance Objectives and Criteria May 2005 32 Principles for a Strong Nuclear Safety Culture - INPO 11/2004 33 WANO 2007 AFI PD Strength AR Matrix 01/01/2008 to 34 PARB Agenda Reviews 3/24/2009 35 PARB review of completed RCEs 2008 36 Department Clock Reset CAP01170951 4/3/2009 37 FP-E-ECR-01, Engineering Change Request 2-4-09 Rev 3 Focus Self Assessment Number: PI-FSA-09-01 AR Number:
38 Jan. 12-16, 2009 01165841 39 Sep08 - Feb09 TRP Summary Chart OE27290 - Harris identified that some important or longstanding 40 problems related to organizational deficiencies are not driven to 5/19/2008 resolution through CAP implementation 41 QF-0429, Rev 4, NSPM Screen Team Standard Agenda Rev 4 PII Commonly Seen Cause & Effect Relationships for Use During 42 Stream Analysis and Failure Mode Charts FG-PA-PAR-01, Rev. 9, Performance Assessment Review Board 2/20/2009 43 Guideline Rev 9 44 RCE01157726, Radioactive Material Shipment Exceeded DOT Limits 10/30/2008 45 CERTEC Common Cause Evaluation Report 2/17/2009 46 Harris RCE Report - Significant Adverse Condition Investigation 5/19/2008 Report 47 FP-PA-OE-01, Operating Experience Program 10/31/2008 Rev 10 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
QF-0433, Rev 2, (FG-PA-RCE-01) RCE Report Template Page 31 of 31 XII. Attachments Attachment 1 - RCE Charter Attachment 2 - Interview List Attachment 3 - Interview Questions Attachment 4 - Process Failure Modes Attachment 5 - Organizational and Management Failure Modes Attachment 6 - Human Performance Failure Modes Attachment 7 - Why Staircases Attachment 8 - Failure Modes Analysis Diagram Attachment 9 - AR Process Flow Analysis Attachment 10 - RCE Action Review Summary Attachment 11 - Procedure Review Barrier Analysis Form retained in accordance with record retention schedule identified in FP-G-RM-01.
4/13/2010 Last saved by n154161
Attachment 1 ROOT CAUSE EVALUATION MANUAL FG-PA-RCE-01 Root Cause Evaluation Charter CAP AR # 01166830 Manager Sponsor: Tim Allen Problem Statement:
The Corrective Action Program at Prairie Island has not consistently resolved significant issues resulting in decreased equipment reliabity, operational transients and repeat non-compliance with station procedures and regulatory requirements Investigation Scope:
The team will review data contained within the corrective action program and determine what aspect or aspects of the corrective action process are not functioning sufficiently to prevent rework and repeat conditions.
This investigation will include A and B severity level CAPs and CAP actions starting on January 1, 2008.
Data prior to that period will not reflect the effects of substantive changes in process Investigation Methodology:
Methodologies will include: event and causal factor charting, interviews, barrier analysis, change analysis, why staircase. The team will use the above methodologies to ensure the criteria identified n NRC IP95001 are satisfied.
Team Members:
Team Leader: Rick Myers Team Member: WD associates <Lloyd Calvery>
Team Member: Fred Ericson Team Member: Gene Woodhouse, Performance Assessment Team Member: Robbie MilesBoysen Consultant for CAPRs: Al Smith / Amanda Bierbauer Milestones:
Date Assigned: 1/27/09 Start: - 3/19/09 (Kickoff Meeting)
Status Update: PARB 4/8/09 Draft Report: 4/10/09 Final Report: 4/20/09 Communications Plan:
The results of the RCE determination will be communicated to the site via a D-15 communication.
Additional communiqué will be provided as the analysis uncovers opportunities.
Approved: ___Tim Allen Date: ___2/27/09 Management Sponsor Approved by: Screen Team / PARB on __________________
Date Page 1 of 1
Attachment 2 Interview List 1 Engineering Design Manager 1 2 Nuclear Plant Attendant 1 3 Inspection and Materials Supervisor 1 4 Engineer 11 5 Performance Assessment Supervisor 1 6 Plant Engineering Supervisor 1 7 Steam Generator Project Supervisor 1 8 Planning Supervisor 1 9 PI Security Analyst 1 10 Performance Assessment Coordinator 1 11 Elec/I&C Design Supervisor 1 12 Elec/I&C Supervisor 1 13 Work Control Center Manager 1 14 Work Week Manager 1 15 Plant Manager 1 16 Electrical Maintenance Technical Advisor 1 17 Ops Support Manager 1 18 CAP Coordinator 1 19 Operations Manager 1 Page 1 of 1
Attachment 3 Interview Questions Charter: The corrective action program at Prairie island has not consistently resolved significant issues resulting in decreased equipment reliability, operational transients and repeat non-compliance with station procedures and regulatory requirements.
- 1. What are your thoughts about the charter?
- 2. Do you have any examples for equipment reliability, operational transients and repeat non-compliance with station procedures and regulatory requirements?
- 3. Who owns the CAP process?
- 4. What is the key driver when performing a RCE/ACE or CE?
- 5. What factors are at play when you perform an ACE/CE/CA or other CAP action assignment? What controls the time and quality of the product? Do you feel like that time pressure impacts product?
- 6. When you write a CAP and corrective actions are assigned, does the performer of the CA discuss the issue with you?
- 7. In your own words what is extent of Cause?
- 8. In your own words what is extent of condition?
Page 1 of 2
Attachment 3
- 9. How much and what type of training have you had for performing RCE/ACE?
- 10. How is OE integrated into plant activities before a problems occur?
- 11. If a CAP / AR is ranked a D or closed to trend, is there communication with initiators as to why?
- 12. Last Question - One thing you would do to fix this problem?
Page 2 of 2
Attachment 4 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Failure Mode Definition Applicability Supporting / Refuting Evidence Actions Not Specified The action(s) that an Applicable S- CAP liaison duties are not proceduralized, A JFG is required (RR1) individual or group must S - QF 0430 Ace grading sheet perform to accomplish a task are not contained in Cause and effect: F6 Inadequate Program Management the document or instruction.
Actions Not Clear The action(s) that an Applicable S- Instructions or requested action on the AR are not clear - is not always (RR2) individual or group must resolved by the assignee. The assignee tries to determine what issue to perform to accomplish a resolve. (D-27) task are not clearly S- The AR screening team make-up and quorum are not specific and described in the constantly changing. . (D-27) document or instruction. S- Department clock reset AR # 01170951- explaining proper CAPR closeouts (D-36)
S- Inconsistent understanding of the CAP process (D-27)
Cause and effect: F6 Inadequate Program Management Actions not within The action(s) that an Applicable S- Actions are not able to be completed due to previous perquisite actions are Control of the individual or group must not completed . (D-27)
Individual (RR3) perform to accomplish a S- Interviewee stated do not always have time to properly analyze the problem task cannot be performed (D-27) as specified (physical S- CAP extensions are difficult to obtained (D-27) constraints, do not have authority to dictate results, etc.).
Actions Conflict with The action(s) that an Applicable S- ECR process has it own grading system for prioritization Another Process (RR4) individual or group must System health, work order priority (D-37,38) perform to accomplish a task conflict or contradict the actions specified by Cause and effect: F6 Inadequate Program Management another document or instruction.
Actions Not Tied to The action(s) contained Applicable S- CAP action priorities not tied into the work management matrix priorities Another Process When within one document or Necessary (RR5) instruction does not Cause and effect: F6 Inadequate Program Management reference supporting documents or instructions when necessary.
Page 1 of 4
Attachment 4 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Failure Mode Definition Applicability Supporting / Refuting Evidence Methods Not Clearly Action(s) are required by Applicable S- Interviewee stated the instruction manual for performing the HUIE, ACE Defined (RR6) the document or did not match-up with the associated forms, since then the instructional instruction, but the manuals have been revised to better match the associated forms. manual method to accomplish the instructions do not match-up with associated forms. (D-27) actions is not clearly S- Procedures do not reference use of tri folds for PASSPORT (D-27) specified by the document or instruction.
Unnecessary Actions The document or Not Applicable No evidence Required (RR7) instruction require the performance of certain actions that is not really necessary to successfully perform the action.
Wrong Information The information provided Applicable S- Trend codes in CAP process are inconsistently entered. (D-27)
(RR8) in the document or instruction is incorrect.
Critical Actions Not Critical actions required Applicable R- Purchase 100 rubber stamps to stamp critical steps in work procedures.
Verified (AR1) to successfully perform a Note: for improvement, forward to the procedure group the critical steps that task are not verified are stamped on a procedure for inclusion in next procedure revision. (D-27) within the process.
Page 2 of 4
Attachment 4 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Failure Mode Definition Applicability Supporting / Refuting Evidence Excessive Verifications The document or Not Applicable No evidence (AR2) instruction requires excessive verification of completed steps or tasks.
Actions are verified, regardless of criticality to the task or the task has multiple reviews and verifications instead of a single, specific review.
No Process Monitoring There is no established Applicable S- difficult to find/use a meaningful monitoring system for trending (D-27)
(AR3) means of monitoring the S- CAP actions not reviewed until CAP all actions are completed - may be success or failure of the picked up by TRP - no established policy (D-27) process. S- TRP conducts a sampling of A level actions and B and C level caps. This does not follow the guidelines in procedure FP-PA-ARP-01 Attachment 10 (D-27)
S- DRUM processes have not evolved to a workable tool that provides its intended purposes. (D-27)
R - grading system for RCR and ACE (D-27)
Cause and effect: F6 Inadequate Program Management Only Monitoring The only method of Applicable S- Monitoring by due date quality is lowered (D-27)
Problems (AR4) monitoring process S- Monitor by KPI (D-27) performance is to observe S- KPI are lagging indicators (D-27) problems when they occur. Cause and effect: F6 Inadequate Program Management No Acceptance Criteria No acceptable Applicable or Not R- grading process for RCE and ACE (D-27)
(AR5) performance parameters Applicable S- due dates used to monitor for KPI (D-27) have been established for S- acceptance criteria by TRP (D-27) the process, procedure or task.
Page 3 of 4
Attachment 4 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Failure Mode Definition Applicability Supporting / Refuting Evidence No One Specified to No one is specified Applicable or Not No evidence Perform Task (I1) (either by title, group, or Applicable other means) as responsible for completion of the actions required by a document or instruction.
More Than One Person More than one person or Not Applicable No evidence Specified to Perform group is specified (either Task (I2) by title, group, or other means) as responsible for completion of the actions required by a document or instruction.
Person Specified Not The person or group Applicable S - ACE assigned to unqualified person Able to Perform Task specified (either by title, S- Interviewees stated everyone owns the CAP process therefore no one owns (I3) group, or other means) as it.
responsible for the completion of the required actions in a document or instruction is unable to perform the action. Typically because they do not have the skill or knowledge.
Page 4 of 4
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Span of Horizontal organizational Not Applicable No evidence Control (S1) design - the number of personnel which a supervisor is responsible for is too large or too few for the groups oversight & responsibilities.
This often creates problems with task assignment and accountability.
Inadequate Levels in Vertical organizational design Applicable S - makeup of screening team not most knowledgeable, experienced, or the Organization (S2) - the number of levels or decision-makers (D27) layers, from senior manager to employee is too many or too Cause and effect few for the given activity.
- C3 INADEQUATE KNOWLEDGE Creates problems with communication of expectations.
Insufficient Staffing Comprehensive organizational Applicable S - cant fill open slots (D27)
(S3) design - the total number of Are we allocating resources appropriately? (D27) employees for which the Spots filled, but not enough for workload (D27) company or group is designed are not filled. Often causes Cause and effect staff work overload and poor
- C3 INADEQUATE KNOWLEDGE accountability.
- F3 INADEQUATE PRIORITIZATION
- F5 INADEQUATE EMERGING ISSUES MANAGEMENT Inadequate A breakdown in Applicable S - RCE team members (and site) dont hear outcome (D27)
Communication within communication (written or TRP results not communicated to site (D27) an Organization (F1) verbal) within one organization Site not aware when RCEs going on - the start (D27) or work group. Often leads to Work group - managers/supervisors not available (D27) important issues not being addressed and critical process Cause and effect breakdown.
- C1 INADEQUATE TRUST
- C2 INADEQUATE TEAMWORK
- F4 INADEQUATE PLANNING
- F6 INADEQUATE PROGRAM MANAGEMENT Page 1 of 6
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate A breakdown in Applicable S - actions not clear in CAP (D27)
Communication among communication (written or Closed to trend - no communication (D27)
Organizations (F2) verbal) among two or more organizations or work groups. Cause and effect Often leads to a breakdown in
- C1 INADEQUATE TRUST processes that require several
- C2 INADEQUATE TEAMWORK groups to participate.
- F4 INADEQUATE PLANNING
- F6 INADEQUATE PROGRAM MANAGEMENT Inadequate Deficiencies in determining Applicable S - Top 10 issues (D27)
Prioritization (F3) which work takes precedence 3 different RCE teams for rad shipment - NRC attention raised significance over other work. Often leads (D27) to unexpected equipment R - PR 1 set up to maintain and improve (D27) failures or failure to meet regulatory requirements. Cause and effect
- C1 INADEQUATE TRUST
- C2 INADEQUATE TEAMWORK
- C3 INADEQUATE KNOWLEDGE
- F4 INADEQUATE PLANNING
- F5 INADEQUATE EMERGING ISSUES MANAGEMENT
- F6 INADEQUATE PROGRAM MANAGEMENT
- S2 INADEQUATE LEVELS IN THE ORGANIZATION
- RR4 ACTIONS CONFLICT WITH ANOTHER PROCESS
- RR5 ACTIONS NOT TIED TO ANOTHER PROCESS WHEN NECESSARY Page 2 of 6
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Planning Deficiencies in determining Applicable S - due date driven (D27)
(F4) what work must be done, by Inadequate consideration of length of time needed (D27) whom, when, and how long it Shortage of qualified staff (reviews) (D27) will take. Often leads to staff work overload, budget over- Cause and effect runs and low morale.
- F1 INADEQUATE COMMUNICATION WITHIN AN ORGANIZATION
- F2 INADEQUATE COMMUNICATION AMONG ORGANIZATIONS
- F3 INADEQUATE PRIORITIZATION
- RR4 ACTIONS CONFLICT WITH ANOTHER PROCESS Inadequate Emerging Deficiencies in determining Applicable S - fighting fires; reactive, not proactive (D27)
Issues Management how to deal effectively with Focus on symptoms instead of root cause, crisis reoccurs (D27)
(F5) unexpected issues. Often leads Shift from engineering to operations focus (D27) to continual crisis management and low morale. Cause and effect
- F3 INADEQUATE PRIORITIZATION
- S2 INADEQUATE LEVELS IN THE ORGANIZATION
- RR4 ACTIONS CONFLICT WITH ANOTHER PROCESS
- RR5 ACTIONS NOT TIED TO ANOTHER PROCESS WHEN NECESSARY Page 3 of 6
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Program Inadequate oversight of critical Applicable S - Management sponsors not consistently engaged with RCE team (D27)
Management (F6) work processes to ensure they Owed to not effectively managing CAPRs from RCEs (D27) function smoothly and KPIs not measuring wrong thing - not measuring quality of actions (D27) effectively. Often results in PARB not consistently performing timely reviews of EFRs (D34) program degradation over time Difficulty in due date extensions result in quality degradation (D27) or increased problems within Ineffective alignment among CAP, Work Management, Site priorities (D?)
those processes. Passport rollout not managed effectively so lost internal OE (D27)
DRUM report doesnt provide meaningful information (D27)
CAPR due dates from different RCEs overlapping (D?)
New RCEs launched before EFRs complete on CAPRs for same issues (D?)
Multiple groups identifying same issues, no consolidation, multiple RCEs from multiple directions addressing the same issues (D?)
Shift from engineering to operations focus (D27)
Inattention to detail on the part of many management team members (RCE approval and CE performance) (D36)
PARB members dont all consistently review and comment on RCE (D35, D27)
Individual contributor challenged manager regarding procedure use and adherence and was advised to proceed in a manner not adherent to procedure (D27)
Cause and effect
- C1 INADEQUATE TRUST
- F1 INADEQUATE COMMUNICATION WITHIN AN ORGANIZATION
- F2 INADEQUATE COMMUNICATION AMONG ORGANIZATIONS
- F3 INADEQUATE PRIORITIZATION
- AR1 CRITICAL ACTIONS NOT VERIFIABLE
- AR3 NO PROCESS MONITORING
- AR4 ONLY MONITORING PROBLEMS
- RR1 ACTIONS NOT SPECIFIED
- RR2 ACTIONS NOT CLEAR
- RR4 ACTIONS CONFLICT WITH ANOTHER PROCESS
- RR5 ACTIONS NOT TIED TO ANOTHER PROCESS WHEN NECESSARY Page 4 of 6
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Trust (C1) A lack of confidence in the Applicable S - Top 10 Equipment List, important or not? (D27) workgroup or members of the Minor Mods not initiated due to belief Design Engineering wont get to it workgroup, or a disbelief in (D27) information shared. Often RCE - lack of management support (D27) results in fractured work RCE - belief management wont accept results of RCE efforts (D27) completion and stress levels. PARB - change root cause, contributing cause, corrective actions of RCE team shows lack of trust in root cause team results R - PARB member does trust the root cause process and outcome (D27)
Cause and effect
- F1 INADEQUATE COMMUNICATION WITHIN AN ORGANIZATION
- F2 INADEQUATE COMMUNICATION AMONG ORGANIZATIONS
- F3 INADEQUATE PRIORITIZATION
- F6 INADEQUATE PROGRAM MANAGEMENT Inadequate Teamwork Constant friction among the Applicable S - competing priorities (D27)
(C2) workforce, or an unwillingness Unclear roles and responsibilities (D27) to work with one another. This problem could exist within Cause and effect organizations or between
- C3 INADEQUATE KNOWLEDGE organizations. Results in
- F1 INADEQUATE COMMUNICATION WITHIN AN confusion within the ranks and ORGANIZATION a lack of information flow
- F2 INADEQUATE COMMUNICATION AMONG among the groups. ORGANIZATIONS
- F3 INADEQUATE PRIORITIZATION Page 5 of 6
Attachment 5 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Knowledge An inadequate Applicable S - Poor action descriptions (D27)
(C3) understanding of the Lack of initiator/owed to discussion with assignee (D27) work to be performed and CAP process doesnt tie into site priorities (D?)
how the work ties into the ACE - not given problem statement (D27) overall goals. Often Shift from engineering to Ops-led (D27) causes individual errors New people not familiar with process/priorities to occur. AR Screening team not necessarily the most knowledgeable, experienced, decision-maker SI 9-5 mechanical agitation, D5/D6, R11/R12, EH mod, Instrument Air (D27)
Cause and effect
- C2 INADEQUATE TEAMWORK
- F3 INADEQUATE PRIORITIZATION
- S2 INADEQUATE LEVELS IN THE ORGANIZATION
- S3 INSUFFICIENT STAFFING Lack of Commitment A lack of dedication to Applicable S - one example of individual knowingly closing incomplete action, (C4) the work. Often results challenged, admitted, and not corrected by individual (D27) in inconsistent or R - Evidence that with the exception noted above, all wish to do quality, unreliable performance timely, meaningful work (D27) by an individual or group.
Inadequate Self A failure to continually Applicable S - inconsistent follow-through to resolution (D27)
Assessment (C5) encourage feedback, R - ample evidence of self-assessment (D38) listen to customer input, or look at better ways to perform. Often creates a false sense of security and leads to complacency.
Page 6 of 6
Attachment 6 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inattention (A1) Not paying attention to Not Applicable No evidence Type - SB the task requirements.
Not paying attention to information in the immediate environment.
Bored (A2) Inadequate level of Not Applicable No evidence Type - SB mental activity due to performance of repetitive actions or lack of activity.
Habit / Reflex (A3) Ingrained or automated Not Applicable No evidence Type - SB pattern of actions attributed to the repetitive nature of a well-practiced task or a natural response.
Tired & Fatigued (A4) Degradation of physical Not Applicable No evidence Type - SB/RB/KB or mental abilities due to illness, a lack of rest, or influences associated with body rhythms.
Distracted & Conditions of task or the Applicable S - Emergent issues require work on CAP actions to be stopped and restarted Interrupted (A5) work environment and result in repeated extension request.
Type - SB require the individual to S - RCE team make-up (PI Rad Shipment RCE required 3 different teams to stop and restart a task, complete and RCE on D5 Tagging Issues team did not stay together until diverting the individuals report was complete.
attention from the task at hand.
Multi Tasking (A6) Performing two or more Not Applicable No evidence Type - SB tasks simultaneously and neglecting to perform a required element of one or more of the tasks.
Lapse of Memory (A7) Momentary loss of Not Applicable No evidence Type - SB memory regarding information previously learned and known.
Page 1 of 5
Attachment 6 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Tracking Method used to maintain Not Applicable No evidence (Place Keeping) (A8) control of information, Type - SB/RB necessary requirements, or status was not properly used.
Time & Schedule Urgency or excessive Applicable S - Due date driven for assignments (D27)
Pressure (A9) pace required to perform S - Due date extensions difficult to get (D27)
Type - SB/RB/KB the task. No spare time allotted or perception by the individual that a tight schedule exists.
Fear of Failure (A10) Apprehension regarding Applicable S - expectation is to meet due date and impression we are driven to meet KPIs Type - SB/RB/KB potential adverse consequences if the individual fails to perform at a high level, resulting in undesirable behaviors.
Imprecise Miscommunication Applicable S - lack of clear problem statement on ACE CAP Communication (A11) resulting from error of S - feedback to CAP initiator on CAPs closed to CTT, CTAT is LTA Type - SB/RB omission or commission R - feed back on RCE and ACE grading by the sender or receiver. S - feedback to site on significant issue (new RCE) and results of RCE This includes breakdowns of the three-part communication process.
Cognitive Overload Mental demands on the Not Applicable No evidence (J1) individual to maintain a Type - RB/SB high level of concentration while requiring recall of excessive amounts of information.
Spatial Disorientation Loss or misjudgment of Not Applicable No evidence (J2) place or time; wrong Type - SB/RB component, wrong train and wrong unit errors due to similarities in the environment.
Page 2 of 5
Attachment 6 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Mindset / Preconceived The tendency of an Applicable S - ACE performer did not like way CAP was written (1 example from Idea (J3) individual to make a interviews)
Type - RB judgment based upon a S - mechanical agitation of SI-9 was decided to perform in the field (RCE preconceived mental review) model or preconditioned R - AR screening meeting - ARs held over that members were not sure of or bias that is not based needed more information. Went back to group to get more info.
upon the current information, conditions or indications.
Wrong Assumptions Judgments are made Not Applicable No evidence (J4) without verification of Type - RB the facts and are usually based upon the individuals perception of recent experiences or events.
Inadequate Verification Insufficient verification Not Applicable No evidence (J5) of the facts, and is Type - RB usually based upon inaccurate information or the lack of information.
Inadequate Motivation Low morale or low Not Applicable R - all interviews indicate that problem solvers are motivated (J6) interest in performing Type - SB/RB/KB well.
Shortcuts Taken (J7) Actions to allow the job Applicable S - solved immediate issue & didnt go into extent of cause and condition to Type - RB to go easier or faster, the depth they should have based on due date. Due date driven drives them to contrary to prescribed cut it short (D27).
requirements. S - extent of cause and condition evaluated based on severity level (D27)
Work Around (J8) Compensatory or non- Not Applicable S - RCE manual and template disconnected Type - RB standard actions to meet a R - RCE manual & template have been revised so they align requirement are taken by the worker due to uncorrected material condition, programmatic deficiencies, or long-standing problems.
Page 3 of 5
Attachment 6 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Over Confident (K1) Underestimating the Not Applicable No evidence Type - KB/RB/SB difficulty or complexity of the task. Self-satisfaction or confidence with a situation in which actual hazards or dangers exist, but the worker is not aware of them.
Unfamiliar or Tasks that have not been Applicable S - members of RCE team are not always qualified or proficient to perform Infrequent Task (K2) performed before or are RCE. Dont have the required skill set (D27)
Type - KB performed infrequently. S - RCE task exceeds the skill set needed to perform the RCE by many of the members assigned to the team (D27).
Misdiagnosis (K3) Decisions made with Not Applicable No evidence Type - KB accurate information that is used or interpreted incorrectly when reaching a decision.
Tunnel Vision (K4) Decisions are made Not Applicable No evidence Type - KB without considering all the available options or information needed to adequately assess the situation.
Inadequate Knowledge Insufficient knowledge of Applicable S - RCE task exceeds the skill set needed to perform the RCE by many of the of Fundamentals (K5) fundamentals needed for members assigned to the team.
Type - KB task, such as heat transfer, fluid flow, structural analysis, etc.
Inadequate Knowledge Insufficient knowledge of Not Applicable No evidence of Standards (K6) codes, standards, design Type - KB basis, licensing basis, regulations, etc. needed to perform the task.
Page 4 of 5
Attachment 6 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Flawed Analytical Decisions based on a Not Applicable No evidence Process or Model (K7) flawed analysis, such as Type - KB/RB using qualitative versus quantitative data, insufficient determination of problem/solution scope, improper computer modeling, or inadequate sample scope.
Page 5 of 5
Attachment 7 WHY STAIRCASE Effect/Symptom IA-1 CAP AR initiators are not providing sufficient detail in the action request as required by Section 5.2. (Attachment 13 provides guidance on sufficient detail)
[Failure Modes F-2, F-3, F-6, S-3]
1.WHY Initiator is not held accountable to adhere to standard in Attachment 13 by Manager/Supervisor WHY Managers/Supervisors are not consistently following procedure - Section 5.3 WHY Do not apply procedure requirement WHY Not held accountable by screening process - Section 5.5 WHY Time pressure WHY Should complete the review in 1 day and shall not exceed three working days - Section 5.3 Step 3 WHY Supervisor/manager does not recognize risks and/or consequences
- 2. WHY Screening Team is not holding initiator/supervisor accountable to adhere to standard in Attachment 13 by Owed To, Screen Team Chair, or Plant Manager.
WHY Screening Team does not consistently apply procedure requirements -
Section 5.3 WHY Not held accountable to adhere to standard in Attachment 13 by Owed To WHY Time pressure WHY One day for lead to present to team for review - Section 5.5 Step 3 WHY Screening Team does not recognize risks and/or consequences
- 3. WHY Initiator, Supervisor, Screening Team not held accountable to adhere to standard in Attachment 13 by Owed To WHY Owed to doesnt consistently apply procedure requirement WHY Not held accountable to adhere to standard in Attachment 13 by Assigned To (pushback)
WHY Doesnt apply procedure requirement WHY Time pressure WHY Focusing on next due date WHY Time Pressure WHY Timely completion emphasized by mention of KPI -
Page 1 of 5
Attachment 7 Section 5.7 Step 4 WHY Establishing due date outside of default requires more time - analysis of level of effort, priority, justification -
Section 5.7 Step 4 WHY Workload WHY Screening process does not consistently screen out CAPs that do not need to be written - Section 5.2 Step 5 WHY Screening process does not consistently screen out CAPs with similar issues where one can be closed and cross-referenced to the second -Section 5.5 Step 6 WHY Screening process does not consistently screen out CAPs that can be changed to a non-CAP AR Section 5.5 Step 7 IA-1 Contributing Cause - Failure to perform all requirements within the procedure.
Effect/Symptom IA-2 Screening Team does not effectively/consistently apply the attributes of a strong nuclear safety culture.
[Failure Modes F-2, F-3, F-6, S-3]
- 1. WHY Problems are not fully understood at front end; possible or unknown reduced safety margins not considered, risk and consequences not considered WHY Problem statements are incomplete, non-specific or cluttered with non-specific data (refer to IA-1)
WHY Screening Team knowledge or diversity LTA WHY The screening team makeup is not specified in the procedure and is changed at management direction only WHY CAP Procedure Attachment 8 weak, too many shoulds WHY Human Behaviors: Time pressure creates brevity
- 2. WHY Entire Screening Team does not review all information provided with the problem statement WHY Work Load and Time pressure WHY Relevance to understanding the problem not recognized WHY Potential impact on attributes of a strong nuclear safety culture not recognized IA-2 Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
Page 2 of 5
Attachment 7 Effect/Symptom IA-3 Plant personnel are driven by due date without proper focus on problem solving and resolution.
[Failure Modes F-3, F-6, RR-4, RR-5, S-3]
WHY Individuals manage work load to prevent late actions WHY CAs due dates are monitored and therefore prioritized above quality WHY Easily measurable and self-revealing while ineffective cause and corrective actions are not immediately revealed WHY Late CAs are easily measured while ineffective solutions and correction are not WHY Plant focus on raw data and KPIs overrides quality of evaluations and corrective actions WHY Reduced Safety/Operating margins are not considered WHY Safety culture model is not clearly pictured or understood at PI WHY Managers and supervisors are not present in front of their direct reports (eye on the problem)
WHY Integrated site priority matrix nonexistent IA-3 Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
IA-3 Contributing Cause - An integrated site priority matrix that interfaces with other high resource plant processes and programs does not exist.
IA-3 Contributing Cause - There are no highly visible CAP process measures for quality.
Page 3 of 5
Attachment 7 Effect/Symptom IA-4 Effective corrective actions are not consistently identified, or if identified, are not consistently completed.
[Failure Modes F-2, F-3, F-6, RR-5]
- 1. WHY Corrective actions are designed to address symptoms, not cause WHY Cross-functional problems are not clearly understood by site management WHY Safety culture model is not clearly pictured or understood at PI WHY Clear expectations to address multi-functional issues are not well defined and difficult to implement WHY Safety culture model is not clearly pictured or understood at PI WHY Managers and supervisors are not present in front of their direct reports (eye on the problem)
- 2. WHY Individuals are driven by due date without proper focus on problem-solving and resolution [IA-3]
IA-4 Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
IA-4 Contributing Cause - An integrated site priority matrix that interfaces with other high resource plant processes and programs does not exist.
IA-4 Contributing Cause - There are no highly visible CAP process measures for quality.
Effect/Symptom IA-5, Completed A Level actions and B level CAPs are not consistently reviewed and are not reviewed in a timely manner.
[Failure Modes AR-3, AR-4, F-3, F-6, S-3]
- 1. WHY Passport tracking and cross-referencing poses problems (closeout is a daisy chain)
WHY Focus shifted to due dates from effectiveness WHY Work load/time management WHY Safety culture model is not clearly pictured or understood at PI Page 4 of 5
Attachment 7 WHY No integrated Site Priority system that would focus resource time on priorities WHY No quality measures/KPI
- 2. WHY Owed to CAP review does not occur until all corrective actions are complete WHY Procedure-driven WHY Safety culture model is not clearly pictured or understood at PI WHY No integrated Site Priority system that would focus resource time on priorities.
WHY No quality measures/KPI
- 3. WHY TRP sample size is not adjusted based on failure rate WHY Management direction in contradiction to procedure instructions WHY Work load / time management WHY Safety culture model is not clearly pictured or understood at PI WHY No integrated Site Priority system that would focus resource time on priorities WHY No quality measures/KPI IA-5 Root Cause - Management has failed to consistently enforce quality standards and set work priorities based upon procedural requirements and risk / benefit to the plant.
IA-5 Contributing Cause - There are no highly visible CAP process measures for quality.
IA-5 Contributing Cause - Failure to perform all requirements within the procedure.
Page 5 of 5
Attachment 8 Failure Modes Analysis Diagram F-6 Inadequate Program management Organizational And Management Failure Modes Process Failure Modes AR-3 F-1 No Process Monitors Inadequate Communication within Organization AR-4 F-2 Only Monitoring Problems Inadequate Communication Among Organizations RR-4 F-3 Actions Conflict with Inadequate Prioritization Another Process RR-5 S-3 Actions Not Tied To Inadequate Staffing Another Process When (resource usage) Necessary F-5 Inadequate Emergent Issue Management S-2 Inadequate levels in the Organization C-1 Inadequate Trust C-2 Inadequate Teamwork C-3 Inadequate Knowledge Page 1 of 1
Attachment 9 AR Process Flow Analysis KEY:
Process flow chart is provided in FP-PA-ARP-01, CAP Action Request Process, Section 5.0, Requirements.
This analysis takes the conclusions from the failure modes analysis, the inappropriate actions from the WHY staircase analysis, safety culture attributes and pictorially represents the process impact point. This evaluation pictorially illustrates the various analysis techniques and identifies the key points of leverage to put PI and PI culture back on track.
Organizational and Management Failures: INPO Safety Culture Attributes:
F-6, Inadequate Program Management is the primary cause 1. Everyone is personally responsible for nuclear safety.
which is induced by three key drivers: 2. Leaders demonstrate commitment to safety.
- 1. F-2, Inadequate Communication among organizations 3. Trust permeates the organization.
- 2. F-3, Inadequate Prioritization 4. Decision-making reflects safety first.
- 3. S-3, Inadequate Staffing 5. Nuclear technology is recognized as special and unique.
- 6. A questioning attitude is cultivated.
Additional Failure Modes:
- 7. Organizational learning is embraced.
- 4. F-1, Inadequate communication within organization 8. Nuclear safety undergoes constant examination.
- 5. F-4, inadequate planning
- 6. F-5, Inadequate emergent issue management Impact on process flow is indicated by BLUE circle and
- 7. C-3, Inadequate knowledge number.
Process Failures:
F-6, Inadequate Program Management is also driven by Inappropriate Actions:
process failure as follows: IA-1, CAP AR initiators are not providing sufficient detail in the
- 1. AR 3, No process monitors action request as required by Section 5.2.
- 2. AR-4 Only monitoring problems IA-2, Screening Team does not effectively/consistently apply the
- 3. RR-4, Actins conflict with another process attributes of a strong nuclear safety culture.
- 4. RR-5, Actions not tied to another process when IA-3, Plant personnel are driven by due date without proper necessary focus on problem solving and resolution.
RR-4 and RR-5 can be characterized as weakness in the IA-4, Effective corrective actions are not consistently identified, or if identified, are not consistently completed.
program instruction IA-5, Completed A Level actions and B level CAPs are not Impact on process flow is indicated by RED circles and consistently reviewed and are not reviewed in a timely boxes. manner.
Impact on process flow is indicated by Green circle and IA Number.
Page 1 of 3
Attachment 9 AR Process Flow Analysis 1 2 4 1 2 4 5.3 - AR Approval 5.6 - Initiation Non-SRO Review Evaluation A Assignment 5.1 - Identify Problem, 5.2 - Submit Action 5.5 - AR Screening Improvement or Request and Processing Request No 5.3 - AR Approval OPR SRO Review Required? 5.12 - Initiate B Work Assignment Yes, for INFO only IA-1 IA-2 Yes Perform 5.4 - Assign C Operability Operability A Make Operability Recommendation Problem statements Recommendation Determination per are not complete, to Responsible (FP -OP-OL-01) Group specific or cluttered with non-specific data. Problems are not fully Priorities and F-1, F-2 Emergent issues are not assessed for understood on the front end. impact on other site organizational impact. S-3, F-3 F-2, F-3, RR-4, RR-5 process not considered.
F-2, F-3, S-3, RR-4, RR-5 Page 2 of 3
Attachment 9 AR Process Flow Analysis A 1 IA-3 IA-5 1 2 4 1
IA-3 IA-4 IA-3 IA-4 IA-5 Plant personnel are due date driven No quality monitors, Effectiveness reviews are Necessary corrective without proper focus on problem only monitoring missing or untimely. actions are not being taken.
solving and resolution problems F-3, F-5, S-3, C-3, AR-3, F-3, F-5, S-3, C-3, AR-3, F-2, F-3, F-5, S-3, C-3, AR-3, AR-4, AR-3, AR-4 AR-4, RR-4, RR-5 AR-4, RR-4, RR-5 RR-4, RR-5 Priorities and impact on other site process not considered.
F-2, F-3, S-3, RR- Page 3 of 3 4, RR-5
Attachment 10 RCE Action Review Summary RCE CAPR 01013473-37 closed to CA 01059041-21 then closed to CAPR 01013473-37 with no action taken.
RCE CAPR 01013473-09 initial training and periodic refresher training shows no evidence of periodic refresher training being implemented RCE CAPR 01013473-10 develop and present a case study completed, no study actually done as evidenced by questions presented but not answered, and no further action taken.
RCE CAPR 01013473-36 no evidence in report benchmarked with industry standards RCE CAPR 01013473-38 notes say EFR called ineffective, but Sharepoint report doesnt identify by EFR or CAPR number additional actions and effective reviews. No way to easily determine if actions and effectiveness reviews were completed.
RCE CAPR 01013473-42 notes say EFR called ineffective, but EFR report in Sharepoint says effective.
RCE CAPR 01013473-11 changes indicators to balance quality and timeliness so that completion of CAPRs not due-date, but quality driven. EFR for this issue states The conclusion of this effectiveness review is that the actions completed were effective in addressing the problem described above, but there is still additional work needed by Management to actively discuss quality of CAP action completion over due dates. A copy of this effectiveness review will be provided to the Director of Engineering for use in discussing CAP action quality and due dates in Engineering Supervisor and Engineer All-Hands meetings. A follow up EFR appears to have been warranted, but not issued.
RCE CAPR 01013473-12 changes CAP extension requirements. EFR for this issue states The conclusion of this effectiveness review is that the actions completed were effective in addressing the problem described above, but there is still additional work needed by Management to actively discuss quality of CAP action completion over due dates. A copy of this effectiveness review will be provided to the Director of Engineering for use in discussing CAP action quality and due dates in Engineering Supervisor and Engineer All-Hands meetings. A follow up EFR appears to have been warranted, but not issued.
RCE CAPR 01013473-05 develop and implement an Equipment Trending Program status complete, with notes that this trending program exists. Either RCE evaluation missed this or assignment did not clearly specify a desired improvement in trending program, or some other disconnect.
RCE CAPR 01013473-08 creates TRP, was effective on A actions, directed later to focus more on C CAPs, no procedural direction to refocus on A actions should they degrade.
RCE CAPR 01099775-03 action is to define clear expectations and consequences for HRA, LHRA, and VHRA violations. Procedure changed to define clear consequences, but no indication that any action was taken to define clear expectations.
RCE CAPR 01099775-04 to increase observations was implemented by procedure change, completed November 2007. RCE CAPR 01099775-10 proposed to and agreed to by PARB to delete CAPR 01099775-04 October 2008.
RCE CAPR 01099775-09 EFR deemed effective despite evidence in the report refuting this.
Page 1 of 1
Attachment 11 Procedure Review Barrier Analysis Barrier - Procedure Use and Adherence FP-PA-ARP-01, Revision 21
- Screening process does not screen out CAPs that do not need to be written -
Section 5.2 Step 5
- Procedure states should complete the review in 1 day and shall not exceed three working days - Section 5.3 Step 3a, drives inappropriate results.
Section 5.3 Step 3b and Attachment 13
- One day for CAP lead to present team for review - Section 5.5 Step 3, drives inappropriate results.
- Screening process does not screen out CAPs with similar issues where one can be closed and cross-referenced to the second -Section 5.5 Step 6
- Owed To doesnt insure sufficient detail in CAP AR - Attachment 13 and Section 5.6 (Level A CAP ARs); Section 5.7 Step 1 doesnt make this responsibility clear
- Procedure does not clearly state what function is responsible for writing the evaluation assignment statement - Section 5.5, 5.6, 5.7
- Establishing initial due date outside of default requires more time - analysis of level of effort, priority, justification - Section 5.7 Step 4, drives inappropriate results.
- Assigned To doesnt determine immediately whether the assignment can be completed by the due date Section 5.8 Step 1
- Attachment 9 is only referenced in step Section 5.10, but includes information relevant to multiple other steps
- Left column doesnt identify Responsible Individual(s) - Section 5.10 Page 1 of 2
Attachment 11
- Establishing initial due date outside of default requires more time - analysis of level of effort, priority, justification - Section 5.13 Step 3, drives inappropriate results.
- Assigned To doesnt determine immediately whether the assignment can be completed by the due date Section 5.14 Step 1
- Owed to does not consistently insure issue has been properly resolved - Section 5.20 Step 1 and Attachment 9
- Screening charter states determine the level of effort. Could be clarified to state meaning of that phrase - determine if RCE, ACE, CE, or CA is needed.
- Screening charter states consideration should be given to using multi-discipline teams in performing evaluations of issues affecting multiple organizations. This is not limited to RCEs, and there is no evidence multi-disciplinary teams are considered except for RCEs - Attachment 8
- Screening team doesnt consistently add note for all A CAPs identifying SCAQ or not, CAPRs, EFRs required - Attachment 8
- Screening team does not consistently identify if feedback to Originator is needed
- Attachment 8
- Composition of screening team allows minimum quorum (shall) that may not include representatives from appropriate departments (should) - Attachment 8
Attachment 9
- Technical Review Panel is not performing of reviews of scope of CAPs defined in charter - Attachment 10
- Results of Technical Review Panel reviews are a measure of quality of CAP actions, but these results are not a KPI or published to the site - Attachment 10
- Established standards of quality for review and approval of evaluations and corrective actions are not established, known, enforced - Attachment 13
- Attachment 14 allows for CAP assignments to be excluded from CAP Performance Indicators, removing visibility to site Result:
- Inadequate problem statements
- Time pressure
- Inadequate evaluations and actions
- Workload increase
- Focusing on next due date Page 2 of 2