ML102120054

Applicant-Northard Exhibit 35-FG-PA-RCE-01, Revision 17, Root Cause Evaluation Manual.

ML102120054
Person / Time
Site:	Prairie Island
Issue date:	02/12/2010
From:	Butterworth H, Travieso-Diaz M Northern States Power Co, Pillsbury, Winthrop, Shaw, Pittman, LLP, Xcel Energy
To:	Atomic Safety and Licensing Board Panel
SECY RAS
Shared Package
ML102120039	List: ML102120040 ML102120041 ML102120042 ML102120043 ML102120044 ML102120045 ML102120046 ML102120047 ML102120048 ML102120049 ML102120050 ML102120051 ML102120052 ML102120053 ML102120054 ML102120055 ML102120056 ML102120057 ML102120058 ML102120059 ML102120060 ML102120061 ML102120063
References
50-282-LR, 50-306-LR, ASLBP 08-871-01-LR-BD01, RAS 18336 FG-PA-RCE-01, Rev 17
Download: ML102120054 (53)

Text

FG-PA-RCE-01 Revision: 17 Issue Date: 2/12/2010 Nuclear Fleet Guidance Document Page 1 of 53

Title:

ROOT CAUSE EVALUATION MANUAL Approval:

Henry H. Butterworth Director, Fleet Operations Standards

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 2 of 53 Table of Contents Page 1.0 PURPOSE......................................................................................................................... 3 2.0 APPLICABILITY ............................................................................................................... 3 3.0 RESPONSIBILITIES ......................................................................................................... 3 4.0 DEFINITIONS ................................................................................................................... 3 5.0 INSTRUCTIONS ............................................................................................................... 6 5.1 GENERAL INSTRUCTIONS ................................................................................... 6 5.2 RCE PREPARATION .............................................................................................. 7 5.3 RCE INFORMATION GATHERING ........................................................................ 7 5.4 ANALYZING INFORMATION ............................................................................... 10 5.5 EXTENT OF CONDITION AND CAUSE ............................................................... 11 5.6 OPERATING EXPERIENCE ................................................................................. 12 5.7 ROOT CAUSE DETERMINATION AND VALIDATION ........................................ 13 5.8 CORRECTIVE ACTION DEVELOPMENT ............................................................ 14 5.9 EFFECTIVENESS REVIEWS................................................................................ 15 5.10 RCE GRADING, PARB REVIEW AND FINAL PROCESSING............................. 16 6.0 RECORDS ...................................................................................................................... 17

7.0 REFERENCES

................................................................................................................ 17 7.1 SOURCE DOCUMENTS ....................................................................................... 17 7.2 REFERENCE DOCUMENTS ................................................................................ 17 7.3 COMMITMENTS.................................................................................................... 18 8.0 REVISION

SUMMARY

................................................................................................... 18 9.0 ATTACHMENTS ............................................................................................................. 18

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 3 of 53 1.0 PURPOSE 1.1 The purpose of this document is to provide guidance for personnel to effectively identify the root cause(s) of problems in order to ensure that proper corrective actions to prevent recurrence are implemented.

2.0 APPLICABILITY 2.1 It is the responsibility of personnel conducting a Root Cause Evaluation (RCE) to ensure that the investigation is performed in compliance with applicable station procedures or controls. This guideline establishes the framework for standards and expectations regarding RCE performance to ensure consistency, thoroughness and quality.

3.0 RESPONSIBILITIES None 4.0 DEFINITIONS 4.1 Causal Factor: An action, condition or circumstance that directly or in combination with another causal factor shaped the outcome of the situation and led to an inappropriate action.

4.2 Common Cause Analysis: An assessment method used to identify underlying causes for a number of similar events. Usually initiated based on a declining or adverse trend, the analysis generally uses a variety of statistical analyses, interviews, and surveys to help determine the underlying causes of the adverse trend. See Attachment 16.

4.3 Complete Picture (of the event) - an end-state of an investigation, when no action or condition exists (including conflicting information) that is not explained by facts.

4.4 Contributing Cause: A causal factor that may have increased the significance or likelihood of the event, complicated the response, or increased the discovery time, but taken alone was not sufficient to cause the event (i.e., if the contributing cause was not present, the event would likely still have occurred). Correction of Contributing Causes can improve the quality of the process or product, but they are not required to be corrected.

4.5 Corrective Action to Prevent Recurrence (CAPR): See FP-PA-ARP-01

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 4 of 53 4.6 Direct Cause: The causal factor that directly led to the inappropriate action. See list of potential direct causes in Attachment 17.

4.7 Effectiveness Review (EFR) - See FG-PA-CAE-01 4.8 Extent of Cause: The extent to which the root cause(s) of an identified problem have impacted or have a significant potential to impact other plant processes, equipment or human performance. These can be seen as causes and their corresponding effects.

4.9 Extent of Condition (EOC): See FP-PA-ARP-01 4.10 Failure Mode: An event causal factor that when identified will help identify the Root Cause(s) and Contributing Cause(s) for an event.

4.11 Inappropriate Action: Personnel action that is inappropriate for the situation and directly transforms the situation into an event. The behavior can be an action taken or failure to take an action. The term inappropriate is not intended to imply personnel fault.

4.12 Independent Reviewer - Qualified Root Cause Evaluator who independently reviews the Root Cause Evaluation. This individual is not the Root Cause Evaluator or the Sponsor.

4.13 Possible Cause - A postulated cause which may have been present during the event, but for which there is insufficient evidence to further confirm (or to eliminate) its presence. In the case of multiple possible causes, there is insufficient evidence to determine that one possible cause is significantly more likely to have been present than the others. Possible causes typically involve one or more assumptions about event conditions, for which there is little or no supporting or refuting objective evidence readily available.

4.14 Possible Failure Mode - A postulated failure mode which may have been present during the event, but for which there is insufficient evidence to further confirm (or to eliminate) its presence. In the case of multiple possible failure modes, there is insufficient evidence to determine that one was significantly more likely to have been present than the others.

4.15 Problem Statement: An objective description of the specific problem that will be investigated, including who was involved; what occurred; what standard, requirement or regulation was not met; when the event occurred; where it occurred; and what the actual and potential consequences are. The problem statement does not include information on why the event occurred.

4.16 RCE Charter: The document that defines the Problem, scope methodology and team members for the RCE.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 5 of 53 4.17 RCE Management Sponsor: The Management Sponsor is the designated owner of the problem and is responsible for the following:

1. Provides regular briefs to the Management team.

2. Provides briefs to outside agencies as necessary.

3. Obtaining PARB or CAP Screening Team approval of the charter.

4. Assembles RCE team with appropriate level of skills, knowledge and qualifications to evaluate incident.

5. Routinely interacts with team, attends team briefings and provides feedback on quality and progress.

6. Approves and is responsible for the quality of the completed RCE Report.

7. Ensures the Root Cause team is provided with the needed support and resources.

8. Provides feedback to the RCE Team regarding their performance.

4.18 RCE Team: A team comprised of an RCE Team Lead, Root Cause Investigator (RCI) and other members as appropriate to perform a Root Cause evaluation. The RCE Team Lead and the RCI may be the same individual.

4.19 RCE Team Lead: An individual designated to lead the RCE Team. (Reference Attachment 13). If the Team Lead is not RCI qualified, the RCI has authority for the cause analysis techniques used and quality and depth of the root cause and corrective action determination. The Team Lead has responsibility and authority to produce an acceptable overall product.

4.20 Root Cause: The most basic (underlying) cause(s) that management has control to fix and, if corrected, will prevent recurrence or significantly mitigate the consequences of the same or similar events. The level of determining when a root cause has been identified may be stopped when the proposed corrective actions would not change.

4.21 Root Cause Evaluation (RCE): An analysis technique that identifies the cause(s) of a problem or condition, and the actions needed to prevent recurrence.

4.22 Root Cause Investigator (RCI): An individual qualified in methods and techniques used to perform root cause investigation and analysis. A qualified investigator will have completed the Root Cause Evaluator Job Familiarization Guide FL-CAP-RCE-001G.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 6 of 53 4.23 Underlying Cause: Organizational or programmatic causal factors that failed to detect or correct a direct cause, or permitted a direct cause to exist, including apparent and root causes. See Attachments 2 (step 6) and 15 for a list of potential underlying causes (also known as process and organizational failure modes).

5.0 INSTRUCTIONS 5.1 GENERAL INSTRUCTIONS 5.1.1 The RCE report must have stand-alone quality by presenting facts and other data to clearly support the causes determined and providing corrective actions that will address the causes. Clear logic ties must exist from root cause through all causal factors to the inappropriate action and to the event. Each significant block or step in the causal analysis chart (e.g., Event and Causal Factor Chart or Why Staircase) must have document references that prove the block/steps validity.

5.1.2 All RCEs should have at least one Root Cause Investigator (RCI) assigned to the RCE Team.

5.1.3 Site management or CAP Screen Team should appoint a Management Sponsor who will provide oversight to the RCE, and an RCE Lead who will conduct the RCE.

5.1.4 A charter should be established, containing the elements identified in Attachment 11, RCE Charter Template.

5.1.5 The CAP Screen Team and/or the PARB should review and approve the RCE charter.

5.1.6 To be a qualified Root Cause Investigator (RCI), an individual must initially become qualified by completing the Root Cause Evaluator Job Familiarization Guide FL-CAP-RCE-001G. To keep this qualification, the RCI must participate as a full-time team member on a Root Cause Evaluation within the previous two (2) calendar years. This is tracked by completing an Attendance Sheet and the Team Lead of the RCE turning it in to LMS/Training.

5.1.7 Periodically during the RCE development, update appropriate PARB members on RCE conclusions and actions. This is recommended so that significant misalignment between the RCE Team and PARB does not occur. Rule of thumb: an RCE is half the report itself and half the communication of the report.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 7 of 53 5.2 RCE PREPARATION 5.2.1 Initiate the preparation process as soon as practicable after the evaluation is assigned.

The RCE Team Lead should complete the following:

1. Quarantine or sequester physical evidence for the event if appropriate using FP-PA-ARP-02, Attachment 1 for guidance.

2. With the Management Sponsor, determine the scope of the evaluation (through development of the RCE Charter).

3. Consider who should be interviewed and any schedule constraints that may impact the interviews (e.g., shift workers).

4. Draft team member assignments.

5. Conduct the RCE pre-job brief. The PJB will include: presenting the charter; working hours; collateral duties; timeline for analysis and report generation; protection of physical evidence; control of regular job duties/distractions; and any procedural requirements including but not limited to due dates of interim reporting requirements to outside agencies

6. If support from another department is involved, provide early notification.

7. Give early consideration to the need to correspond with outside organizations such as vendors, EPRI, other utilities, etc. Information requests/inquiries can take several days or weeks. Nuclear Network is one of the industry information exchange media for requesting information from other utilities that have experienced similar events.

8. Obtain documents of formal notifications to external agencies (NRC, DNR, EPA, Insurer, INPO, etc.).

9. Identify and review all station procedures and policies that govern the activities associated with the event.

10. IF performing an RCE on an incident that involves chemicals or chemical processes, THEN contact Industrial Health and Safety to ensure compliance with OSHA 1910.

5.3 RCE INFORMATION GATHERING 5.3.1 The RCE Team should gather information and data relating to the event/problem. This includes physical evidence, interviews, records, and documents needed to support the root cause.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 8 of 53 The team should keep track of the sequence of all events, the source of all facts used as evidence, basis for all assumptions and the sources (documents) used. Source documents should be indexed.

Some typical sources of information which may be of assistance include the following:

1. Interviews

2. Written statements

3. Removed equipment and/or replaced parts

4. Operating logs

5. Maintenance records

6. Inspection reports

7. Procedures and Instructions

8. Vendor Manuals

9. Drawings and Specifications

10. Equipment History Records

11. Trend Chart/Strip Chart Recordings

12. Sequence of Event Recorders

13. Radiological Surveys

14. Plant Parameter Readings

15. Sample Analysis and Results

16. Correspondence

17. Design Basis Information

18. Photographs/Sketches of Failure Site

19. Industry Bulletins

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 9 of 53

20. Previous internal and external operating experience or events

21. Turnover logs for affected groups (e.g., RP, Maintenance)

22. Task analysis and Lesson plans

23. Perform a physical walkdown, if warranted.

Written statements should be obtained prior to any critique which could alter the NOTE: perceptions of those personnel involved with the event whenever possible.

5.3.2 Use Attachment 1, "Personnel Statement," or a similar form to obtain written statements from personnel involved as soon as practical (preferably prior to leaving the site) following the event. Personnel statements are normally written separately by each individual rather than as a collaborative summary of the event.

Have each individual include name, date, position, department, and in their own words, describe their knowledge of the event facts, and their involvement in the event before, during, and after the final outcome. They should include any pertinent verbal communications and specify who they spoke with (by name and/or position) as well as other communications (pre-job brief, direct assignment, inter-department interfaces, etc.)

and who they spoke with. They should also list any pertinent procedural or equipment conditions relating to the event.

5.3.3 Conduct personnel interviews with involved parties as soon as practical following the event. Do not conduct interviews until sufficient information has been analyzed. See Attachment 4, "Interviewing Techniques."

5.3.4 IF it is suspected that the cause of the event may have been an intentional attempt to disrupt normal plant operation (e.g., tampering), THEN notify station management and Security in accordance with applicable station procedures.

5.3.5 The RCE Team should plan activities so that physical evidence and other important information is not altered, destroyed, or lost. Preservation of evidence must not interfere with or delay placing the plant systems in a safe condition.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 10 of 53 5.4 ANALYZING INFORMATION These are not the only methods available, but represent proven NOTE: techniques for evaluating various types of problems.

5.4.1 Using the facts identified by the evaluation, and reviewing the event as a whole, decide which of the facts or groups of facts are pertinent. Event and causal factor charting and why staircase, along with either barrier analysis or change analysis should be used. The analysis techniques should be used to establish tightly linked, evidence-based chains of cause and effect.

Analytical techniques that are available include:

1. Event and Causal Factor Charting (Attachment 2)

2. Task Analysis (Attachment 3)

3. Change Analysis (Attachment 5)

4. Barrier Analysis (Attachment 6)

5. Failure Modes and Effects Analysis (Attachment 7)

6. Why Staircase (Attachment 8)

7. Pareto Analysis (Attachment 9)

8. Common Cause Analysis (Attachment 16)

9. PIIs Equipment Root Cause of Failure Analysis 5.4.2 Compare the facts to an "acceptable standard" and determine if an unacceptable condition exists. Identify each inappropriate action and equipment failure.

5.4.3 It may be helpful to perform Streaming Analysis for a group of problems (e.g., INPO AFIs) as a prelude to RCE. See FG-BS-STR-01 5.4.4 Evaluate potential detrimental effects on associated plant equipment.

5.4.5 Organize the information into an overall description of the problem.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 11 of 53 5.4.6 Establish a start time and a finite end time to bound the event and prepare the team to begin developing the event and causal factor chart.

5.4.7 Determine the actual and potential safety significance of the event (nuclear, radiological, industrial, environmental, and regulatory). This may require formal analysis of the event by the Probabilistic Risk Assessment (PRA) group and they should be contacted early in the investigation.

5.4.8 For self-revealing or externally identified issues, state why the in-place normal performance monitoring process did not identify the issue before it became self-revealing and develop remedial actions accordingly.

5.4.9 Maintain objectivity to prevent jumping to conclusions. Do not assume that a technician used a procedure or that the procedure would work as written or that an alarm printed on an alarm recorder was heard by an operator. Likewise, you can't assume that instructions given over the phone were heard and understood by the receiver. Keep in mind that all human performance events are caused by personnel error. Avoid labeling a root cause as personnel error if at all possible; seek the underlying cause of the error instead.

5.4.10 IF new problems OR conditions adverse to quality that are not directly part of the causal chain for the RCE are found, THEN initiate a CAP i.a.w. FP-PA-ARP-01.

5.5 EXTENT OF CONDITION AND CAUSE 5.5.1 Extent of Condition Determine where the same or similar condition/problem exists within other plant processes, equipment or organizations and may be unknown, setting up the potential for latent errors to cause another event. Example: problem is a mechanical failure of a Safety Injection pump bearing caused by a manufacturing flaw. Extent of Condition, in simple terms, would consider:

Same-Same: Same type of bearing used in opposite train Safety Injection pump.

Same-Similar: Same type of bearing used in another safety related pump The more safety significant the potential consequence is, the further one should go with the extent of condition assessment.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 12 of 53 The scope of extent of condition is bounded when the cause of the problem is understood (i.e., do not check all pump bearings, but only those of the specific manufacturer; do not check all bearings from the manufacturer, but only those subject to the specific manufacturing flaw).

5.5.2 Extent of Cause Determine where the root cause could have had an impact within other plant processes, equipment or organizations that is yet unknown. Continuing the pump bearing example, it would be necessary to determine whether the manufacturer provided equipment other than the same pump bearing (e.g., other pump bearings or equipment), that could have a manufacturing flaw.

5.5.3 Document the scope of the extent of condition/cause, the basis for the scope selected, and the conclusions regarding actions needed as a result of the EOC analysis.

5.5.4 Example Problem - A nuclear facility found Control Room operator A to be color blind.

Extent of Condition - Examine all other operators for color blindness.

Extent of Cause - Direct cause was no annual test requirement in the procedure. Root cause was medical staff high turnover rate, resulting in an inadequate procedure.

x Check the procedure for other missing requirements.

x Check other critical activities performed by the department.

x Check other departments that have high turnover rates.

5.6 OPERATING EXPERIENCE The OE analysis has two objectives:

1. Determine if previous OE exists that should have prevented the event.

2. Determine if the corrective actions being developed for the RCE have appropriate depth and breadth. In other words, have other stations had success with specific corrective actions that are not included in the RCE?

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 13 of 53 If no applicable OE is found, include the OE search criteria in the final report (this can be included in an attachment and summarized in the report).

a. What databases were searched?

b. What keywords or combinations were used?

c. What was the number of hits and the number of relevant similar items?

5.6.1 Internal Operating Experience Search the CAP data base for key words and similar events.

If related events are identified, determine what the causes and corrective actions taken were, and why the corrective actions were ineffective. Use this knowledge to help develop new effective corrective actions.

5.6.2 External Operating Experience Search the INPO data base for key words and similar events.

Attach keyword Repeat to the parent CAP if the event should have been NOTE: prevented by previous OE.

If related events are identified, determine why the OE program did not prevent the event by checking how the event was processed by the OE program. Also determine what corrective actions were taken by the station(s) that submitted the event(s).

5.6.3 Conclusions Document conclusions on whether a problem exists with the OE program, and whether the proposed corrective actions for the RCE appropriately align with successful industry experience.

5.7 ROOT CAUSE DETERMINATION AND VALIDATION 5.7.1 Review the causal factors identified in the analytical techniques used. The root cause or causes are the most fundamental, underlying causes identified. When different causal chains (i.e., drill downs from different inappropriate actions) identify the same underlying cause, this provides more evidence of a root cause.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 14 of 53 5.7.2 The root cause must be concrete and specific such that someone not involved with the RCE can readily understand what the cause is, and would be able to develop appropriate corrective actions. Nebulous, generic or global cause statements will not allow SMART corrective actions.

5.7.3 Provide a narrative description of the primary causal chains and logic ties.

5.7.4 Validate the root causes using the following guidance:

Elimination of the root cause would have prevented the associated causal factors and inappropriate actions, and prevented the same or similar events. Accomplish the validation by working your way from the root cause up through the causal chain, back to the event. Verify logic ties for each step.

5.7.5 Determine if any of the safety culture components contributed to the deficiency. Evaluate against the safety culture components for each of the cross-cutting areas within NRC Inspection Manual, Chapter 0305 Operating Reactor Assessment Program, using form QF-0436 and Attachment 14. Attach the form in Sharepoint and provide a summary of the conclusions in the RCE report.

5.7.6 If possible causes are used, clearly identify them. Possible causes may be needed if it is not possible to independently determine their complete validity (i.e., their existence is inferred because they provide the most probable linkage between causal factors).

5.8 CORRECTIVE ACTION DEVELOPMENT The use of corrective actions that call for "assessment," "evaluation,"

NOTE: "consideration," "review," etc. should be avoided, unless the amount of analysis work is clearly outside the scope of effort of the RCE Team.

5.8.1 Corrective actions should meet SMARTS criteria from FP-PA-ARP-01.

5.8.2 Solutions must be identified and implemented that will correct the identified root cause(s).

5.8.3 Brainstorming and interviewing are good sources of CAPRs and involve appropriate group(s) to establish ownership as early as possible. See Attachment 10, "Development of Corrective Actions."

5.8.4 Obtain "buy-in" from the Manager of the group that will be responsible for performing the corrective action. If the action affects a Fleet process, obtain Peer Group concurrence.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 15 of 53 5.8.5 IF the RCE Team, Management Sponsor, or group responsible for implementing corrective actions is unable to reach agreement, THEN Site Management or Peer Group should facilitate a resolution.

5.8.6 Identify appropriate Interim or Compensatory Actions, if they are needed, and if they were implemented prior to the evaluation. Interim corrective actions should be specified if there is a high probability of recurrence in the time period between the initial problem and when the final corrective actions are implemented.

Entry of corrective actions into the Action Request database is independent of NOTE: PARB review.

5.8.7 Enter the corrective action into the Action Request system verbatim from the report and document the action number in the RCE report. Include the corrective action owners and due dates in the RCE report. You may keep the corrective action in INPROG status in Passport (remains editable) until the report is approved by PARB. Ensure mitigation related immediate actions are processed in a timely manner.

5.8.8 OE Coordinators will initiate an OEER for the other site within the fleet to review the root cause once it has been approved by PARB/CAP screening.

5.8.9 If corrective actions for contributing causes are not developed, provide an explanation why.

5.8.10 If an LER is associated with the event, verify the RCE conclusions and actions are consistent with LER report. If they are not, notify Regulatory Affairs.

5.9 EFFECTIVENESS REVIEWS If the event was caused by seasonal and/or environmental conditions (e.g. a summer algae bloom in the river), it might be prudent to perform an interim EFR NOTE: before the conditions repeat and then do a final EFR after the conditions have passed.

5.9.1 Effectiveness Reviews are required for CAPRs and are performed after corrective actions have been implemented to ensure the RCE identified and corrected the root cause(s).

This is a proactive assessment of the corrective actions versus waiting for an event challenge to determine effectiveness. The depth and duration of an effectiveness review should be commensurate with the significance and complexity of the problem. General guidance and examples of when an Effectiveness Review might be appropriate can be found in FG-PA-CAE-01, Corrective Action Effectiveness Review Manual.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 16 of 53 5.10 RCE GRADING, PARB REVIEW AND FINAL PROCESSING 5.10.1 Ensure a Complete Picture of the Event exists and draft the report using QF-0433.

1.Section VII.A of QF-0433, Information and Facts Sources, is intended to allow a narrative description of those items that may involve specialized knowledge, such as department procedures or technical concepts - this section provides appropriate background information that may not be commonly known by the audience.

2.Section XI, References, is intended to provide a list of all relevant reference documents associated with the RCE, such as procedures and policies.

3.Section XII, Attachments (list of supporting evidence documents) is a list of all documents that support and are referenced in individual items in the event and causal factor chart or why staircase (e.g., radiation survey record of RHR pump, or interview with Shift Manager).

5.10.2 When the draft report is complete, obtain an independent review (RCI qualified) and incorporate comments as necessary. The intent of this review is to determine whether the RCE is on the overall right track, and meets the basic principles and standards for a root cause evaluation.

5.10.3 Obtain approval of the Management Sponsor. When the Management Sponsor has approved, the RCE may be signed of as complete in Passport.

5.10.4 RCE Grading and final grade is determined by PARB. The final grade assigned to an RCE is documented in the RCE grading assignment by Performance Assessment. Use QF-0432, RCE Report Evaluation to evaluate the quality of the RCE.

5.10.5 The PARB review should be led by the Team Sponsor, Team Lead, or the RCI, and should focus on the timeline, the evidence that supports the causes (e.g., the event and causal factor chart), and the root cause/CAPR/EFR quality and linkage.

5.10.6 IF the RCE grade is 2 or below, THEN the RCE should be re-opened to address the key comments, AND a separate CAP AR should be written for the low-scoring RCE.

5.10.7 Grading should be made available to the evaluator, the RCE Team and the Management Sponsor after scoring.

5.10.8 When RCE has been approved and required PARB comments are incorporated, verify the due dates and wording of the corrective actions (Passport must match the report), and move them out of INPROG.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 17 of 53 6.0 RECORDS 6.1 The original hard copy of the completed RCE report may be retained for reference and informational purposes.

6.2 The electronic copy of completed RCE reports that are initiated by Action Reports are required to be retained as part of the applicable Action Reports for record purposes, as required by FP-PA-ARP-01. Closure of the RCE assignment/evaluation by the management sponsor is sufficient for the approval signature on the RCE report.

7.0 REFERENCES

7.1 SOURCE DOCUMENTS 7.1.1 FP-PA-ARP-01, CAP Action Request Process 7.1.2 QF-0432, RCE Report Evaluation 7.1.3 INPO 90-004, "Good Practice OE-207, Root Cause Analysis" 7.2 REFERENCE DOCUMENTS 7.2.1 FP-PA-ARP-01, CAP Action Request Process 7.2.2 FG-PA-CAE-01, Corrective Action Effectiveness Review Manual 7.2.3 QF-0432, RCE Report Evaluation 7.2.4 FP-PA-HU-01, Human Performance Program 7.2.5 FL-CAP-RCE-001G, Root Cause Evaluator Job Familiarization Guide 7.2.6 QF-0414, Human Performance Department and Site Clock Reset Notification -

(RED/YELLOW) Sheet 7.2.7 QF-0433, RCE Report Template 7.2.8 FP-PA-ARP-02, Augmented Incident Evaluation

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 18 of 53 7.2.9 QF-0436, Evaluation of Safety Culture Impacts 7.3 COMMITMENTS None 8.0 REVISION

SUMMARY

8.1.1 Section 4 - added definitions of direct cause and underlying cause to support common cause analysis tool 8.1.2 Section 5.10 - clarified the intent of the RCE independent review step 8.1.3 Moved list of potential Direct Causes from within Attachment 4 to its own new Attachment 17.

8.1.4 Added Attachment 16, Common Cause Analysis, with deletion of the CCE Manual.

9.0 ATTACHMENTS 9.1.1 Attachment 1 - Personal Statement 9.1.2 Attachment 2 - Event & Causal Factor Charts 9.1.3 Attachment 3 - Task Analysis 9.1.4 Attachment 4 - Interviewing Techniques 9.1.5 Attachment 5 - Change Analysis 9.1.6 Attachment 6 - Barrier Analysis 9.1.7 Attachment 7 - Failure Modes and Effects Analysis 9.1.8 Attachment 8 - Why Staircase 9.1.9 Attachment 9 - Pareto Analysis

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 19 of 53 9.1.10 Attachment 10 - Development of Corrective Actions 9.1.11 Attachment 11 - RCE Charter Template 9.1.12 Attachment 12 - Equipment Failure RCE 9.1.13 Attachment 13 - RCE Team Lead Checklist 9.1.14 Attachment 14 - Safety Culture Analysis 9.1.15 Attachment 15 - Organizational and Process Failure Modes 9.1.16 Attachment 16 - Common Cause Analysis 9.1.17 Attachment 17 - Direct Causes

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 20 of 53 ATTACHMENT 1 PERSONAL STATEMENT

/ /

Name (Print) Date Position Department General Instructions:

In your own words, describe your knowledge of the event facts, and your involvement in the event before, during, and after the final outcome. Include any pertinent verbal communications and specify who you spoke with (by name and/or position). Indicate the format of the communications (pre-job brief, direct assignment, inter-department interfaces, etc.), and who you spoke with. List any pertinent procedural or equipment conditions relating to the event. Use additional sheets as necessary.

Signature _________________________________

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 21 of 53 ATTACHMENT 2 EVENT & CAUSAL FACTOR CHARTS An event and causal factor chart (E&CF) is a graphic display of an event. The heart of the E&CF chart is the sequence of events plotted on a time line. Beginning and ending points are selected to capture all essential information pertinent to the situation.

The key to the E&CF chart is to identify the correct inappropriate actions (IAs) and then drill down through the causal factors to the root cause(s) (the why staircase is most commonly used as the drill down tool).

E&CF charts are particularly useful for complex and complicated situations, and can be more useful than long narrative descriptions. They allow you to separate the many causal factors associated with complex events.

The E&CF chart graphically displays the relationship between the sequence of events, inappropriate actions, barriers, changes, causes, and effects.

FORMATTING THE E&CF CHART:

x All events (actions or happenings) that occurred during some activity - rectangles x Causal factors that influenced the course of events - ovals x All events and conditions that are assumed or have not been confirmed - dotted line rectangles and ovals x Inappropriate actions - diamonds x Contributing Cause - oval with shaded grey x Root Cause - ovals with highlighted or bubbled area at one end (dark) x Terminal event (end point of the evaluation, typically this will be the consequence of the event) - circle x Other symbols may be used, as desired, to indicate barriers, broken barriers, process changes, or other items that contribute to the clarity. Provide an identification key for these symbols if used.

CRITERIA FOR EVENT DESCRIPTION Events should precisely describe a SINGLE action or happening (quantified) and be based on VALID information (facts). Use a short sentence, usually with just one subject noun and one action verb. A date (and time if pertinent) should be included.

Chart scope should range from beginning to end of the situation sequence:

x Each event should be derived directly from the event and conditions preceding it. When this is not the case, it is an indication that one or more events or conditions are missing.

x Each event should be in the appropriate time relationship to the preceding and succeeding event.

Detail of the event sequence MUST be sufficient to ensure a complete picture.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 22 of 53 ATTACHMENT 2 (Continued)

EVENT & CAUSAL FACTOR CHARTS HOW TO DEVELOP AN E&CF CHART STEP 1: Evaluate initial information and documentation x What were inappropriate actions and/or equipment failures?

x When did they occur (during what task/evolution)?

x How did they occur?

x What were the consequences?

STEP 2: Begin constructing the preliminary primary event line.

x Start early - use currently known facts x Use yellow sticky notes. The events, factors, and conditions will probably need to be revised and rearranged.

STEP 3: Define scope of chart from initial information.

x Initiating event, i.e., beginning point x Terminal event, i.e., the reason for the investigation STEP 4: Add new information to preliminary chart:

x Events, Conditions and Inappropriate Actions in chronological sequence on the timeline x Events

- Primary - directly leads to or follows a primary effect or inappropriate action

- Secondary - impacts primary event, but is not necessarily directly involved in situation. Plotted on horizontal lines parallel to primary events line x Conditions

- Initial

- During course of inappropriate actions or equipment failures After inappropriate actions or equipment failures STEP 5: Identify failed barriers, changes, and causal factors.

x Start with Direct Causes, Attachment 17.

x Analysis (Task, Change, Barrier, Why Staircase, Interviewing)

STEP 6: Additional models that can be used to identify causal factors are shown below.

These models are simple, but can often help to identify causal factors or valuable lines of inquiry.

x Human Performance Model

1. Select qualified individual (trained, qualified, knowledgeable of management expectations, familiar with the plant)

2. Perform pre-job preparations (review procedure, get parts, walk down job site)

3. Conduct pre-job briefing (critical steps, OE, traps, tools, contingencies)

4. Conduct work (HU tools)

5. Supervisory oversight (frequency and quality)

6. Post job critique (problems captured in CAP)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 23 of 53 ATTACHMENT 2 (Continued)

EVENT & CAUSAL FACTOR CHARTS x Accountability Model

1. Develop clear written standards and expectations

2. Communicate the standards and expectations in a manner that assures individuals are knowledgeable of them

3. Monitor conformance to the standards and expectations

4. Coach when behaviors do not match standards

5. Implement consequences in a manner that achieves behavior change and maintains a strong safety conscious work environment x Performance Analysis QF-444 This worksheet can help one decide if training (i.e., lack of knowledge or skill), or other non training related causes are involved. See FP-PA-HU-02.

x Process and Organizational Failure Modes in Attachment 15 STEP 7: Reference document numbers that validate the block in the chart (e.g., logs, forms, interviews, procedures, reports, recorders, photos, surveys, lesson plans, correspondence, etc)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 24 of 53 ATTACHMENT 2 (Continued)

EVENT & CAUSAL FACTOR CHARTS Big Pump-Little Pump Example A day shift maintenance crew had been assigned to repair sump pump P-119A. Operations was to remove this pump from service and place a danger tag on the electrical supply breaker.

Operations opened and tagged what they thought to be the breaker for pump P-119A. Actually they opened and tagged the breaker for P-119B. While maintenance was working on pump P-119A, it automatically started. Fortunately no one was hurt. The following information was obtained during the event investigation:

1. Near the end of the midnight shift, operations was assigned to tag out the pump. The shift supervisor approved a correct equipment clearance order at 0700 and instructed two operators to remove the pump from service.

2. Two pumps, P-119A and P-119B, are installed next to each other in the same sump. The breakers for pumps P-119A and P-119B were not labeled. These breakers are adjacent to each other and are located in the vicinity of the pumps.

3. The shift change time is 0800

4. The operators saw that one breaker was larger than the other (see picture). They also noticed that pump P-119A was larger that P-119B. Electrical cabling between the breakers and pumps was observable for the entire length except for a short distance where both cables ran together beneath a metal cover. The operators looked under the cover from one end and thought they could see that the cables did not change their relative positions as they passed under the cover. They also pulled on one cable and sensed movement on the corresponding cable at the other end of the cover. In fact, the cables changed position beneath the cover. The first operator traced the cable from P-119A and, based on his belief that the cables did not cross under the cover, arrived at the first breaker for P-119B. This happened to be the large breaker, which reinforced his conclusion that he had identified the correct breaker. He opened the breaker and place the danger tag on the breaker handle.

5. The second operator assisted the first operator in tracing the cable and signed the tag verifying correct tag placement.

6. Many breakers for other similar equipment also are not labeled. Operators have become accustomed to this lack of labeling.

7. The operators assigned to the task placed and verified the tag at about 0740. They knew they had to complete this task before going home.

8. The maintenance crew started work on P-119A at 0900. At 1000, after the maintenance crew had uncoupled the pump and motor, the motor started in response to a high sump level.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 25 of 53 ATTACHMENT 2 (Continued)

EVENT & CAUSAL FACTOR CHARTS

9. Plant policies (procedures) do not specify how the person who places tags and the person who verifies proper tagging are to maintain independence. Usually these individuals do not work together. However, on this day they worked together because they did not have much time.

Picture

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 26 of 53 ATTACHMENT 2 (Continued)

EVENT & CAUSAL FACTOR CHARTS REMEMBER The intent of this process is to understand the sequence of events, determine the inappropriate actions and the relationships of the conditions and causal factors.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 27 of 53 ATTACHMENT 3 TASK ANALYSIS Task analysis is a tool that is used on evaluations where problems during performance of tasks contributed to the event. Performing a Task Analysis will provide the RCI with:

x A clear understanding of how the task is normally performed and should be performed.

x Questions arising out of the analysis to be answered during the course of the evaluation, usually through interviewing.

One of the first priorities when entering an evaluation is to understand as much as possible about the activity that was being performed. It may be necessary to obtain the required expertise on the team to be able to perform the task analysis.

The task analysis will require a review of procedures, work orders, logs, technical manuals, and other documents in an effort to determine what the task is about and how it was to be performed.

This process is called the Task Analysis method.

1. Paper and Pencil - the task is broken down on paper into subtasks identifying:

x Sequence of actions x Instructions x Conditions x Tools x Other materials associated with the performance of the task This type of analysis consists of a review of logs, work documents, technical manuals, etc.,

to determine what the task was about and how it was to be performed. The steps, questions and concerns should be displayed on the preliminary event and causal factor chart.

2. Walk-Through - A step-by-step enactment of the task for an observer without carrying out the actual function. The observer makes notes of any differences between the actual performance enactment and the procedure steps. Personnel performing the walk-through should be people who actually do the tasks, but not people who were directly involved in the event. The walk-through should identify:

x How the task is "really" performed x Problem areas such as:

- Discrepancies in procedure steps

- Human factors design in the man-machine interface

- Training, knowledge, or skill weaknesses

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 28 of 53 ATTACHMENT 3 (Continued)

TASK ANALYSIS Steps in Walk-Through Task Analysis:

1. Obtain preliminary information to understand what happened during the event.

2. Determine the scope of what is to be included in the walk-through.

3. Obtain necessary information:

x procedures, work package, etc.

x drawings x interviews

4. Develop a guide for the walk-through to outline how the analysis will be conducted:

x identify key activities to be performed and observed x identify activities to be recorded

5. Determine exactly what information is going to be recorded and how - one technique is to check off each step as it occurs. Discrepancies and problems may be noted in the margin or in comment space provided adjacent to the step.

6. Select personnel to perform the task who normally perform it. If a crew is involved, crew members should perform their normal role.

7. Perform the walk-through while observing and recording. Note any discrepancies or problems.

x Try to re-create the situation to obtain a sense of how the actual event occurred.

x The walk-through may be done in slow motion, stopping to address questions.

The personnel performing the task may describe the activities from their perspective as they perform.

x The walk-through may be performed in real time to identify time-related problems.

x An actual task in the plant may be observed, but preparation as described above is necessary.

x A simulator or mock-up may be used.

8. Summarize and consolidate problems noted and questions to be answered during interviews. Identify possible contributors or causal factors for the event or failure.

Example of a Task Analysis Worksheet Procedure Who Required Component Tools Conclusion Step Action or equipment

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 29 of 53 ATTACHMENT 4 INTERVIEWING TECHNIQUES Statements made by the interviewee should be validated by multiple sources NOTE:

wherever possible.

Interview Preparation All interviews require preparation, no matter how simple the problem seems. Interviewing is a "fact" finding skill rather than a "fault" finding session.

1. Develop a set of questions. The questions can be derived from the Events and Causal Factors Chart, Change Analysis, Barrier Analysis, or the Direct Cause list.

2. Review applicable procedures and documents gathered in section 5.3 to determine what additional information needs to be obtained via the interview.

3. Consider the preferred sequence of interviews.

4. Make appointments.

5. Select an appropriate location.

6. Allow time between interviews to reconstruct notes.

Introduction/Opening The purpose of the introduction is to orient the interviewee and put him/her at ease.

1. Explain the purpose of the evaluation and the interview (to identify what happened, how it happened, why it happened, and what can be done to prevent recurrence).

2. Provide the interviewee with an overview of the material to be covered.

3. Show interest and get the interviewee involved.

4. Anticipate and answer the interviewees questions:

x What will happen with information (it will be used to determine root causes).

x Will my name be used (the report may include a list of interviewees).

x Why do you want to talk to me (we believe that you can help explain what happened)

Question/Answer The purpose of the interview is to obtain the interviewees recollection and understanding of the event. The following are some of the features of a successful interview:

x Begin with open-ended questions to allow the interviewee to provide his/her perspective.

x Listen carefully while taking notes. Do not interrupt.

x Keep questions short and to the point.

x Do not ask leading questions.

x Use primary questions (from the prepared list) to introduce a topic and use secondary questions to clarify information.

The Closing The closing accomplishes more than just concluding the interview. It provides an opportunity to validate information and obtain additional information.

x Summarize the information that was recorded.

x Set up the potential for a follow-up interview.

Thank the interviewee for his/her help

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 30 of 53 ATTACHMENT 5 CHANGE ANALYSIS ALWAYS ASK:

What was different about this situation from all the other times the same task or activity was carried out without an inappropriate action or equipment failure?

STEPS IN CHANGE ANALYSIS:

1. Analyze the situation containing the inappropriate action or equipment failure.

2. Analyze a comparable situation that did not have an inappropriate action or equipment failure.

3. Compare the situation containing the inappropriate action or equipment failure with the reference situation.

4. Write down all known differences whether they appear relevant or not. As the evaluation progresses, be alert to other differences that were not apparent during the initial review and add them to the list.

5. Evaluate the differences for effect on producing the event. This must be done with careful attention to detail, e.g., a change in color or finish may change the heat transfer parameters and consequently affect system temperature.

6. Integrate information relevant to the causes of, and contributors to, the inappropriate action or equipment failure into the investigative process via the E&CF chart.

WHEN SHOULD CHANGE ANALYSIS BE USED?

x When causes of inappropriate action or equipment failure are obscure x When you don't know where to start the evaluation x You suspect a change may have contributed to the inappropriate action/equipment failure EXAMPLES OF CHANGES TO CONSIDER:

x What - operating parameters (i.e., changes in temperature, pressure, flow, cycle time, etc.)

x When - plant status, time of day, day of week, season of year, times when specific conditions exist (i.e., why does it work some times but not others?)

x Where - physical location (i.e., why does it work in one location but not another?)

x How - how equipment is supposed to work (i.e., why it works in one application but not another?)

x Who - personnel involved (i.e., is an individual/crew using a different method or technique?)

Example of a Change Analysis Worksheet Problem Statement: Define the problem as it relates to performing a change analysis. e.g.,

Following the rebuild of D4 during last outage, the D4 ran without problem for full operating cycle.

Following rebuild of D5 during this outage, the D5 was slow to start and developed high crank case pressure.

Previous Condition Current Condition Change / Difference Impact or Assessment or event or event (List all possible (List comparable (List all differences (What effect did the change contributors one at a contributors.) without evaluation, have on the situation? What time, need not be in value judgment or were the actual/potential sequential order.) significance, whether consequences? Why was relevant or not.) change allowed?)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 31 of 53 ATTACHMENT 6 BARRIER ANALYSIS DISCUSSION Barriers are devices employed to protect and enhance the safety and performance of the plant.

They can be physical or administrative in form. Barriers are erected to ensure consistent and desired performance of the plant. A single barrier is rarely relied upon. Generally, barriers are diverse and numerous - a defense-in-depth concept. Some examples of barriers commonly found in nuclear power plants highlight the importance of these devices as follows:

PHYSICAL BARRIERS x Engineered Safety Features x Safety and Relief Devices x Conservative Design Allowances x Redundant Equipment x Locked Doors and Valves x Ground Fault Protection Devices x Radiation Shielding x Alarms and Annunciators x Fire Barriers and Seals ADMINISTRATIVE BARRIERS x Plant Operating and Maintenance Procedures x Policies and Practices x Training and Education x Maintenance Work Orders x Radiation Work Permits x Licensing of Operators x Qualification of Welders x Methods of Communication x Certification of Health Physicists and Technicians x Certification of Engineers x Technical Specifications x Regulations x Supervisory Practices x Work History BARRIER TYPES x Barriers that promote (good design, labeling, work planning, procedures) x Barriers that prevent (interlocks, locked doors, physical segregation) x Barriers that discourage (caution signs, rope barriers, notes/cautions in procedures/briefings) x Barriers that detect (hold points, check off lists, operator rounds, pre-job procedure reviews) x Barriers that compensate (tests done at shutdown/low power, notifying control room prior to task)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 32 of 53 ATTACHMENT 6 (Continued)

BARRIER ANALYSIS BARRIER ANALYSIS METHODOLOGY

1. Identify target - Result of the event (e.g., Rx scram, ESF actuation, personnel injury, valve mispositioned, etc.). Target could also be desired result (e.g., successful completion of test).

2. Identify a single hazard to the target -Typically start with the symptoms or failure mode(s) at the time the event occurred. This could also be "energy" applied to the system (e.g., monthly pump test).

3. Identify all barriers.

4. Integrate this information into the preliminary E&CF.

5. Identify all apparent barriers that failed and allowed the event to progress.

6. Determine HOW the barrier failed, e.g., the relief valve barrier failed because although the valve was functional the set point had drifted high.

7. Determine WHY the barrier failed, e.g., the relief valve set point had not been checked since original installation because it is non-safety related.

8. Validate the results of the analysis with information learned.

While barrier analysis identifies missing or defective barriers, it has a weakness. If the investigator does not recognize ALL failed barriers, the evaluation may be incomplete. Because using barrier analysis alone is very time consuming it is recommended that barrier analysis be used in conjunction with other techniques.

When an RCE evaluation is initiated, you must think in terms of barriers. Naturally, the barriers established in plants differ widely and evaluation of them is dependent upon your knowledge.

Regardless of variations in barriers at plants, RCE provides the framework for barrier assessment because it focuses on precise barrier categories that have proven to be critical in identifying equipment failures. Corrective actions from RCE evaluations usually include modification of existing barriers, but caution should be taken before considering additional barriers so that additional failure modes are not introduced.

Example of an Energy (Hazard)/Barrier/Target Analysis Worksheet Energy/Hazard Barrier Assessment Target (List one at a time, (Identify all applicable (Identify if barrier was (Identify all applicable need not be in physical and missing, weak, or targets such as individual sequential order.) administrative barriers ineffective and why.) organizations, equipment, for each consequence.) facilities, and processes.)

Monthly pump test Procedure No step to open Successful completion of discharge valve. test Operator New Operator. Did not QV&V or STAR Supervisor No oversight of first time evolution.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 33 of 53 ATTACHMENT 7 FAILURE MODES AND EFFECTS ANALYSIS THE FM & E ANALYSIS PROCESS

1. Develop a list of possible failure modes. Possible sources or references to develop the list might include the following:

x Previous failures from equipment history data bases x Known failures from industry user groups x Known failures from original equipment manufacturer x Previous failures from other stations x Failure diagnostic programs, guides, and tools (EPRI ERCAWS, computer aids, consultants, etc.)

Possible failure modes can be documented on a Fishbone Diagram where each failure mode is a major rib of the fish. As an alternative, the major ribs can be general categories such as Human Performance, Procedures, Equipment, and Facilities.

2. Collect physical evidence NOTE: This can be performed concurrently with Step 1.

x Physical evidence should be gathered to completely understand the WHAT and HOW of the failure.

Care must be taken while gathering evidence not to accidentally destroy other evidence. For example, if a component must be disassembled, care NOTE: must be taken to capture all "as found" conditions. Do not clean or contaminate fracture surfaces. Measurements, photographs, video tape, or other methods should be considered to preserve evidence x Evidence should be gathered to validate or refute the postulated failure modes. For example, IF one of the postulated failure modes is WATER HAMMER, THEN conduct a system walk down to look for evidence of water hammer, such as damage to small pipe or instrument connections, etc.

3. Evaluate each possible failure mode against the physical evidence to validate or refute it. In other words, determine whether the failure mode would have produced the physical evidence that exists.

4. Continue Steps 1 through 3, and through a process of elimination reduce the list to the single failure mode or the most probable failure mode(s).

5. Evaluate the single or most probable failure modes using the "Cause and Effect" process to determine the root cause(s).

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 34 of 53 ATTACHMENT 7 (Continued)

FAILURE MODES AND EFFECTS ANALYSIS HINTS ON USING THE FM & EA PROCESS A thorough understanding of the failed equipment is necessary in order to conduct FM & EA. A highly knowledgeable subject matter expert is needed. If the evaluation team does not possess a high level of knowledge, an expert needs to be recruited from elsewhere inside or outside of the organization.

x Possible failure modes should not be ruled out until physical evidence validates that it should be eliminated. The evaluation may need to look for a lack of evidence to eliminate a particular failure mode.

x The process may need to be repeated to identify intermediate failure modes until the primary failure mode is determined.

x Examination of physical evidence may need to be performed under laboratory conditions. If that is the case, it is important to get laboratory personnel involved as early as possible. It is highly recommended that laboratory personnel visit the location of the failure to understand layout, environmental conditions, history, etc., that may have contributed to the failure.

x If the component failure was catastrophic, physical evidence may have been lost or destroyed in the failure (for example, electrical insulation is destroyed by fire). If that is the case, other similar components can be examined. Also, possible corrective actions to consider are methods to capture and preserve physical evidence in future failures.

Failure Modes & Effects Analysis Example Environment Natural Maintenance Bird Feces Contamination Salt Spray Artificial Washing Line Sugar Cane Burning Smog Washing After Outages CB-EG Transmission Line Insulation Failure LineSectionalizing Failures Degradation Contamination Design Operation

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 35 of 53 ATTACHMENT 8 WHY STAIRCASE WHY STAIRCASE PRINCIPLES x All events have a cause. These events are the result of plant conditions, plant design, human performance, etc.

x A bond/relationship exists between cause and effect. The analysis usually starts at an undesired effect and goes back to the root cause. The key is to recognize that for each effect there is at least one condition and one action but typically there are more.

Fire Oxygen Fuel Ignition source For example - In the above diagram, IF oxygen and fuel are present as conditions, THEN the action of introducing the ignition source establishes the minimum necessary requirements for a fire.

x Root cause(s) can be found by examining the relationships. Ask "why?" usually five to seven times to determine "root" cause.

HINTS ON USING CAUSE AND EFFECT ANALYSIS x Often causes and effect analysis will lead to management-controlled root causes (also called Organizational and Programmatic causes).

x When more than one cause is responsible for an effect, each cause must be evaluated.

x Why staircase is most effective when used within the framework of the E&CF chart. It is not a stand alone method because the situation must first be unraveled to the point where ALL failure modes are identified. This is particularly true in situations involving multiple failures.

x This process of why staircase provides a logical, structured guide to maintaining the evaluation on track, but will require good judgment and experience to be effective.

REPEAT WHY STAIRCASE ANALYSIS UNTIL:

x The cause is outside of the control of the plant staff x The cause is determined to be cost prohibitive x The equipment failure is fully explained x There are no other causes that can be found that explain the effect being evaluated x Further cause and effect analysis will not provide additional benefit in correcting initial problem

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 36 of 53 ATTACHMENT 8 (continued)

WHY STAIRCASE Example Why Staircase Legend: D stands for document, RC stands for root cause, WS stands for why staircase, CC stands for contributing cause, IA stands for inappropriate action.

Why Staircase IA1 Particle was not detected during job coverage survey for removing and wrapping the fuel sipper from the spent fuel pool.

Why 1 Particle was difficult to detect with job coverage survey (D4, D5, D17, diagram)

Why 1.1 Dose rates measured with RO-2 at 30 cm were 20-30 mR/hr (i.e. not significant to general area dose rates) (D14, diagram)

Why 1.2 LAS of equipment and cables detected no particles (D14, D17)

Why 1.3 All discrete particles found in area were of low dose concern (D17, D23)

Why 2 Survey conducted for job area coverage per RPIP 1135 versus an equipment survey for rad shipment evaluation IAW D11.7 (D11, D25)

Why 2.1 Person doing job coverage survey was not required to be familiar with D11.7 and D11.7 was not referenced in work documentation (see WS4)

Why A Person was following RPIP 1122 - Hot Particle Program (D29)

Why a. RPIP 1122 is not prescriptive with respect to rad shipping concerns (RC1)

Why B Rad shipper brief was only about decontamination target levels, not discrete particles (D14, D23)

Why C RMSP is outside work management program Why a. See WS4 Why D Personnel completing surveys and packaging equipment are not qualified RWSC Why a. No training program exists to specifically describe shipping tasks x The training and qualification for rad shippers and RP personnel who perform shipping related activities does not meet industry standards (CC2, RC2)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 37 of 53 ATTACHMENT 9 PARETO ANALYSIS PURPOSE & DESCRIPTION Pareto Analysis is the process of determining the "Vital Few" factors responsible for a particular problem. A Pareto Chart is a bar chart of failures ordered by frequency of failure, cost of failure or contribution to system unavailability.

E D G " A " H o u rs U n a v a ila b le E D G " B " H o u rs U n a v a ila b le 500 100% 500 100%

9 8%

9 5% 94%

450 89% 90% 450 90%

8 2%

400 80% 400 81% 80%

74%

350 70% 350 70%

Hours Unavailable 66 %

Hours Unavailable 63 %

300 60% 300 60%

Percent Percent 24 6 250 50% 250 5 0% 50%

48%

204 200 40% 200 40%

139 30% 134 150 127 30% 150 30% 30%

107 102 89 89 100 20% 100 20%

64 53 48 44 50 10% 50 26 10%

16 0 0% 0 0%

1. Determine the Effect or Problem. What is the problem to be addressed? For example, is system reliability or availability of concern (or both)?

2. Decide how the Effect should be measured. Determine how to measure the problem. For example, frequency of failure is used to measure reliability; duration of failure is used to measure availability.

3. Decide how the Effect can be stratified. Failures are typically stratified by system equipment or component, although other strata may be used.

4. Interpret the Results. What does the Data Reveal? Which failures or causes are the "Vital Few?"

PARETO CHART CONSTRUCTION

1. Collect the data and group the events by category or strata.

2. Order the categories from highest to lowest (frequency, unavailability, etc.).

3. Draw bars for each category; the bar heights equal the categorys frequency/duration.

Develop the cumulative line, adding the impact of each category from left to right.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 38 of 53 ATTACHMENT 10 DEVELOPMENT OF CORRECTIVE ACTIONS IMPORTANT! Avoid the shotgun approach. Excessive and unnecessary corrective actions not only add burden to staff, but introduce the possibility for new failure modes. For proposed corrective actions, ask which root cause they will address. If they do not address a root cause, are they needed?

1. Corrective actions should be developed to address the following:

x root causes (prevent recurrence) x failure modes (repair what is broken) x symptoms (detect future degradation before failure) x common mode failures (other components, train, systems, unit, sites, departments, programs, etc.)

x effectiveness follow up (are actions effective)

2. Recommended corrective actions need to:

x address issues x be cost effective x be within control of site x meet or exceed industry standards x be created using the SMARTS criteria

3. Factors to consider:

x cost x risks and/or consequences of actions or inaction (new failure modes) x mitigation or prevention if addressing root cause is cost prohibitive

4. Constraints to consider:

x time (short-term vs. long-term, temporary fix vs. action to prevent recurrence) x resources x political realities

5. Reviewed by the individuals that will be implementing the corrective actions so that they can provide input and understand the reason and scope for the corrective actions.

6. Understand the following:

x requirements (CFR, codes, FSAR, etc.)

x commitments x goals and objectives x previous or similar situations x vested interests

7. Corrective Action Effectiveness from most effective at preventing recurrence to least effective x Design (plant and/or process/procedures) to prevent the cause x Remove the human element from being able to initiate the cause x Add/improve barriers x Warn the individuals before they can initiate the cause x Train the individual (adequate training is essential, but do not use training to compensate for an error-likely situation)

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 39 of 53 ATTACHMENT 11 RCE CHARTER TEMPLATE Root Cause Evaluation Charter CAP AR #________

RCE#________

Manager Sponsor: [Name of responsible manager]

Problem Statement: [Insert a statement of the problem that is to be investigated. Include bulleted areas where the problem(s) is/are noted. The problem should cover who was involved, what occurred, what was expected to occur, when the event occurred, where the event occurred. Consequences also should be described and bounded. The RCE will describe why the event occurred.]

Investigation Scope: [Develop a statement bounding the scope of the analysis. Include key lines of inquiry from PARB/CAP screening and any known areas for EO condition evaluation.]

Investigation Methodology:

[Insert the methodologies that the team expects to use to perform the investigation such as event and causal factor charting, interviews, barrier analysis, change analysis, why staircase, etc.]

[E&CF chart and why staircase, along with barrier or change analysis, should be used.]

Team Members:

[Consideration should be given as to whether or not the team or individual performing the evaluation requires a mentor.] [If conducting a team root cause, the team composition should be cross disciplinary.]

[The sponsor needs to obtain buy-in from the team members manager/group head that the team member will work on the evaluation full time, and other work is reassigned or deferred.]

Team Leader ________, Dept./Group Team Member ________, Dept./Group Team Member ________, Dept./Group Team Mentor ________, Dept./Group (if needed)

Milestones:

Date Assigned Date Status Update Date Draft Report Date Final Report Date Communications Plan: [If determined to be needed]

Initial communication to the station, communication to INPO, NRC, etc., follow up to the station, etc.

Approved:___________________________ Date:_____________________

Management Sponsor Approved by: Screen Team / PARB on ______________________

(circle one) Date

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 40 of 53 ATTACHMENT 12 EQUIPMENT FAILURE RCE Equipment Failure RCE The PII Equipment Root Cause Failure Modes Analysis methodology should be used for equipment RCEs. The manual is located on the Fleet PA web page.

1. Descriptions of other investigative methods, report content, etc.

2. The depth to which the equipment failure RCE is taken is based on the safety and economic significance of the failure.

3. Quarantine or preserve the failed equipment so that evidence is not destroyed or disturbed using FP-PA-ARP-02, Attachment 1 for guidance.

4. Determine potential failure modes.

5. Where appropriate develop a testing plan that utilizes the failure modes chart to prove or refute all the possible causes. The testing plan should prevent destruction of evidence as much as possible for future testing and should detail the expected resulting possibilities.

6. Through the testing sequence, different failure modes should be eliminated. The goal is to eliminate all but one failure mode. The failure mode is used to determine the root cause.

7. If testing shows that multiple event failure modes have taken place in the same event, then each must be considered for root cause and corrective actions should be applied to each unique root cause.

8. In these evaluations, the following additional items should be considered during the investigation. Findings in each of the items below should be provided in the final report:

x Current Operability Determination or evaluation.

x Reportability evaluation.

x Industry Operating Experience (OE) review.

x Internal Operating Experience (OE) review.

x Vendor experience/input.

x Organizational & Programmatic Deficiencies/Human Error contribution.

9. Investigation of the failure mode may require laboratory analysis. Many of these test results must be compared to the original design specifications to determine if the critical characteristics of the failed item meet design requirements. Tolerances should be included as this will often identify a mis-manufactured issue.

10. Successful equipment failure RCE is heavily dependent on a thorough and systematic evaluation of technical data. After collecting the data, perform simple analyses to eliminate possible scenarios. Watch for human error or programmatic problems.

Consult experts as required.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 41 of 53 ATTACHMENT 13 RCE TEAM LEAD CHECKLIST This list has been devised to assist team leads in leading a root cause team. Please use this as a guide. Additionally, please review and have working knowledge of the procedures and guidelines.

Thank you for your efforts and contributions to the Root Cause Evaluation process.

Initial assignment

1. Team lead assigned by Screen Team

2. Quarantines or sequesters physical evidence for the event if appropriate using FP-PA-ARP-02, Attachment 1 for guidance. (May have been completed during augmented event investigation process.)

3. Obtain written statements as soon as possible after the event. (May have been completed during augmented event investigation process.)

4. With the Management Sponsor, determine the scope of the evaluation (through development of the RCE Charter

5. With the Management Sponsor, assemble team members needed to perform evaluation.

Assure team makeup has members with necessary training or qualifications to participate on the team.

6. Attends PARB or Screening with Management Sponsor for approval of RCE charter.

Coordinate team logistics

7. Obtain and coordinate room space, equipment and communication lines for team members.

8. Obtain and coordinate supplies needed (notebooks, org charts, yellow stickies, highlighters)

9. Establish routine meeting times, places and agendas for team. Serves as meeting chair.

10. Create electronic files on common drive for team access

11. Coordinate meetings with the Management Sponsor (and PARB if necessary) to discuss impediments, needed resources or issues/delays in completing the evaluation.

Team leadership and coaching

12. Conduct RCE pre-job brief with team members.

13. Facilitate problem solving and collaboration. Keep the team focused and on track.

14. Consider who should be interviewed and any schedule constraints that may impact the interview (e.g., shift workers).

15. Draft team member assignments.

16. Ensure discussions and decisions lead toward closure. Capture those items that may need to be revisited later.

17. Ensure alignment (causes, actions, effectiveness reviews) with team.

18. Ensure the team addresses all relevant issues within the scope of the evaluation and match the completed sections of the evaluation with the grading criteria.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 42 of 53 ATTACHMENT 13 (Continued)

RCE TEAM LEAD CHECKLIST Communication

19. Provide early notification if support from other departments is required.

20. Give early consideration to the need to correspond with outside organizations such as vendors, EPRI, other utilities, etc. Information requests/inquiries can take several days or weeks. Nuclear Network is one of the industry information exchange media for requesting information from other utilities that have experienced similar events.

21. Identify or define the station acceptable performance criterion that meets applicable Industry Standards and Regulations.

22. If performing an RCE on an incident that involves chemicals or chemical processes, contact Industrial Health and Safety to ensure compliance with OSHA 1910.

23. Ensure alignment (causes, actions, effectiveness reviews) with Management Sponsor.

24. Attend PARB with Management Sponsor to support presentation of the Root Cause Evaluation.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 43 of 53 ATTACHMENT 14 SAFETY CULTURE ANALYSIS The purpose of this analysis is to determine if the root cause of the RCE may be related to a safety culture aspect. It is important to understand this so trends in safety culture can be identified and corrected, and so that the overall regulatory status of the station can be taken into account. The safety culture system is described in the NRC Inspection Manual Chapter 0305, and is basically the NRCs view of a management system.

This activity should not be performed until the root causes have been validated.

1. Review the safety culture aspects on form QF-0436. Determine if any of the root causes align with a safety culture aspect.

2. Document the analysis on form QF-0436

3. Consult with Regulatory Affairs to determine if additional actions are needed.

Actions could be needed depending on whether the RCE root causes align with (or dont align with) the NRCs safety culture aspect determination, or if new trends are identified.

4. Document the basis for the conclusions reached in the Safety Culture section of the RCE report.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 44 of 53 ATTACHMENT 15 PROCESS AND ORGANIZATIONAL FAILURE MODES (Also known as potential underlying causes)

Roles & Responsibilities Related Actions Not Specified The action(s) that an individual or group must perform to accomplish a task are not contained in the document or instruction.

x Many errors involving missed expectations x Administrative procedures not including applicability or responsibilities x Process missing necessary steps Actions Not Clear The action(s) that an individual or group must perform to accomplish a task are not clearly described in the document or instruction.

x Many errors involving execution of the process x Human error traps within the process x Technical procedure steps (what to do) are vague or complicated (multiple actions) x High procedure revision rate Actions Not W/in Control of the Individual The action(s) that an individual or group must perform to accomplish a task cannot be performed as specified (physical constraints, do not have authority to dictate results, etc.).

Actions Conflict W/Another Process The action(s) that an individual or group must perform to accomplish a task conflict or contradict the actions specified by another document or instruction.

x Conflicting requirements or directions between processes or procedures x Ineffective technical review or verification Actions Not Tied to Another Process When Necessary The action(s) contained within one document or instruction does not reference supporting documents or instructions when necessary.

x Many errors occur at interface points between processes or programs x Requests or feedback between processes or program are informal, not tracked Methods Not Clearly Described Action(s) are required by the document or instruction, but the method to accomplish the actions is not clearly specified by the document or instruction.

x Directions on how to accomplish the task are not clear.

Unnecessary Actions Required The document or instruction require the performance of certain actions that is not really necessary to successfully perform the action.

x Excessive number of controls to perform activity x Less safety significant work is required to be done with the same level of controls as safety significant work

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 45 of 53 ATTACHMENT 15 (continued)

PROCESS AND ORGANIZATIONAL FAILURE MODES x Controls or checks have little added value or quality Wrong Information x The information provided in the document or instruction is incorrect.

x Technical errors or sequencing errors Accountability Related Critical Actions Not Verified x Reviews and verification did not identify a problem and were not bypassed x Inadequate validation of the quality, completion, correctness and documentation of an activity x Verification should be required (forced by the process) at critical points to prevent a single error from causing a failure of the overall process Excessive Verifications x The document or instruction requires excessive verification of completed steps or tasks.

x Actions are verified, regardless of criticality to the task or the task has multiple reviews and verifications instead of a single, specific review.

No Process Monitoring x There is no established means of monitoring the success or failure of the process.

x Lack of program/process monitoring, evaluation & improvement x Extended period of lowering or poor performance Only Monitoring Problems x The only method of monitoring process performance is to observe problems when they occur.

No Acceptance Criteria x No acceptable performance parameters have been established for the process, procedure or task.

x No guidance specified for what constitutes acceptable or unacceptable performance x No guidance for when the task should be stopped Individual Related No One Specified to Perform Task x No one is specified (either by title, group, or other means) as responsible for completion of the actions required by a document or instruction.

x Boundaries of responsibility not properly defined - a gap exists

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 46 of 53 ATTACHMENT 15 (continued)

PROCESS AND ORGANIZATIONAL FAILURE MODES More Than One Person Specified to Perform Task x More than one person or group is specified (either by title, group, or other means) as responsible for completion of the actions required by a document or instruction.

x Boundaries of responsibility not properly defined - too much overlap exists Person Specified Not Able to Perform Task.

x The person or group specified (either by title, group, or other means) as responsible for completion of the required actions in a document or instruction is unable to perform the action, typically because they do not have the skill or knowledge (not trained or qualified).

x Personnel affected by the process are unaware of requirements, responsibilities or expectations x Fragmented responsibility for program/process Functional Issues Inadequate Communication within an Organization x A breakdown in communication (written or verbal) within one organization or work group.

Often leads to important issues not being addressed and critical process breakdown.

x Many committed actions not carried out in timely manner x Supervisors not having direct communication channels to senior management x No routine meetings to communicate standards & expectations x Problems not solved because they are not communicated or the right personnel are not involved Inadequate Communication among Organizations x A breakdown in communication (written or verbal) among two or more organizations or work groups. Often leads to a breakdown in processes that require several groups to participate.

x Lack of defined interface requirements, expectations or responsibilities x Lack of teamwork or trust amongst organizations or work groups x Problems that transcend multiple organizations or groups are not solved Inadequate Prioritization x Events recur due to slow implementation of corrective actions x High backlog of work x In conflict with station mission & goals x Work activity missing due dates or priority Inadequate Planning x Not clearly determining what work must be done, by whom, when, and how long it will take.

Often leads to staff work overload, budget over-runs and low morale.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 47 of 53 ATTACHMENT 15 (continued)

PROCESS AND ORGANIZATIONAL FAILURE MODES Inadequate Emerging Issues Management x Deficiencies in determining how to deal effectively with unexpected issues. Often leads to continual crisis management and low morale.

x Lack of a self-assessment process x Poor performance monitoring & trending x Lack of attention to emerging issues, significance not recognized Inadequate Program Management x Inadequate oversight of critical work processes to ensure they function smoothly and effectively. Often results in program degradation over time or increased problems within those processes.

x Insufficient support of program or process x Extended period of lowering or poor performance x Long term issues not adequately addressed or resolved x Line management unfamiliar with process Structural Issues Inadequate Span of Control x The number of personnel which a supervisor is responsible for is too large or too few for the groups oversight and responsibilities. This often creates problems with task assignment and accountability.

x Inadequate organizational structure and planning x Mixing short term & long term mission groups under a single manager Inadequate Levels in Organization x Vertical organizational design - the number of levels or layers, from senior manager to employee is too many or too few for the given activity. Creates problems with communication of expectations.

x Excessive layers of management x Inadequate organizational structure and planning Insufficient Staffing x The total number of employees for which the company or group is designed are not filled.

Often causes staff work overload and poor accountability.

x High time pressure x Work overload in an organization Cultural Issues Inadequate Trust x A lack of confidence in the workgroup or members of the workgroup, or a disbelief in information shared. Often results in fractured work completion and high stress levels.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 48 of 53 ATTACHMENT 15 (continued)

PROCESS AND ORGANIZATIONAL FAILURE MODES Inadequate Teamwork x Constant friction among the workforce, or an unwillingness to work with one another. This problem could exist within organizations or between organizations. Results in confusion within the ranks and a lack of information flow among the groups.

x Management infighting or friction between organizations x Requests for feedback, interaction or information are informal (no process), not x tracked or are lost x Organizational boundaries of responsibility not properly defined (either a gap exists or too much overlap exists)

Inadequate Knowledge x An inadequate understanding of the work to be performed and how the work ties into the overall goals. Often causes individual errors to occur.

x Not providing the workforce with the necessary skills & knowledge to do their job Lack of Commitment x A lack of dedication to the work. Often results in inconsistent or unreliable performance by an individual or group.

x Inadequate resources assigned to program or process x Excessive amount of time to implement/develop a program x Missing program or process elements (Process owner, sufficient staff, procedure, process requirements known by personnel, process performance monitoring) x Inadequate management support of program or process Inadequate Self Assessment x A failure to continually encourage feedback, listen to customer input, or look at better ways to perform. Often creates a false sense of security and leads to complacency.

x Ineffective process monitoring (monitoring areas include: backlog status, failure rate, resources available to support the process, effectiveness of process) x Long term issues not being resolved or addressed x Extended period of lowering or poor performance

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 49 of 53 ATTACHMENT 16 COMMON CAUSE ANALYSIS PURPOSE Common cause analysis (CCA) provides a systematic method to determine the causes of adverse trends, typically within ACEs or RCEs. Once the common causes are identified, other traditional cause analysis techniques (e.g., why staircase) can be used to drill down to an acceptable level of underlying cause.

Common cause analysis can also be performed for potential trends identified under C significance level CAPs.

The Performance Analysis worksheet (QF-0444) is an additional option for analyzing trends that collectively constitute a department or site wide performance deficiency.

METHOD Step 1 For potential trends, determine if an adverse trend exists by comparing actual performance to baseline or historical performance. Do conditions such as greater work hours during an outage explain the apparent change in performance?

Document the basis for your conclusion in the evaluation report. If an adverse trend is identified, verify the parent CAP is considered for B level by CAP screening team, and continue with this procedure.

Step 2 List the individual issues, events or deficiencies that make up the trend in a table with the following headers:

x CAP # & Event Date x Event Date x Event Description x Direct Cause x Additional columns as appropriate for specific parameters (e.g., parameters that may constitute trends or commonalities, such as involved department, level of position in organization, experience level of involved individual, etc.)

Step 3 Evaluate each event to determine what the direct cause was (see Attachment 17 for potential direct causes), using information from the CAP. If the CAP does not have sufficient information, contact the involved individuals.

Step 4 Review the direct causes identified and specific parameters chosen, and determine groupings or direct causes that are common to most individual events. Compare results with respect to consequence as well as frequency of occurrence.

Step 5 Include the table in the Event Description and Timeline section of the evaluation report. Provide a clear basis in the table for the direct causes selected for each event.

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 50 of 53 ATTACHMENT 16 (continued)

COMMON CAUSE ANALYSIS Step 6 Select one or more of the common causes/specific parameters for additional analysis, and provide a clear basis for what was selected. Use conventional causal analysis techniques such as why staircase or event and causal factor charting, and determine an underlying cause (or causes) for the common causes/specific parameters chosen.

Step 7 Identify any failed corrective actions from the individual events that may shed light on what future corrective actions will be effective. Craft the corrective actions with this insight in mind.

Example Tables CAP # Event Direct Basis for Direct Additional Columns for Specific and Description Cause Cause selected Parameters as needed Event Date Direct # of Occurrences Comments Cause/Specific Parameter

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 51 of 53 ATTACHMENT 17 DIRECT CAUSES This attachment lists direct causes that can potentially lead directly to an inappropriate action.

The list is intended to be used as a guide when developing event and causal factor charts to determine what lies on the causal chain between the inappropriate action and the underlying causes of an event. The direct causes of an event can also tie to other direct causes, before linking to more underlying organizational or programmatic issues.

1. Verbal Communications - examples includes pre job briefings, plant or equipment status communications, work direction, turnovers, and meetings.

x Instructions not complete or clear x Unaware of changes in job x Supervisor or new shift unaware of problem x Plant communication systems inadequate

2. Written Document - examples include procedures, work orders, vendor manuals, drawings, logs, night orders, RWPs, labels, tags.

x Documents technically inaccurate x Document incomplete x Document not current revision x Document did not match the job x Document not legible x Document not available x Document contained ambiguous language x Document missing precautions or prerequisites

3. Human Factors - examples include interfaces with equipment, instruments, computers, and recorders.

x Manipulation of incorrect equipment x Equipment not accessible x Instrument accuracy not adequate x Instrument precision not adequate x Instrument range of display not adequate x Incorrect instrument used to obtain data x Equipment layout is confusing x Inconsistent equipment layout between trains x Operating panel layout is confusing

4. Physical Environment - anything in the work location that hampers successful job completion.

x Lighting inadequate

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 52 of 53 ATTACHMENT 17 (continued)

DIRECT CAUSES x Housekeeping problems (water, oil, debris, etc.)

x Excessive noise or heat x Confined space x Too many workers for work location x Energized equipment accessible x Protective equipment impacted work

5. Work schedule and plan - examples include an individuals schedule, 13 week schedule, job schedule within or among shifts, schedule or job scope coordination, work package readiness, pre job preparations.

x Worked excessive hours x Fatigue rule or process non compliance x Work in one area impacted another job x Scope of work changed x Insufficient time to perform job x Insufficient resources scheduled x Inappropriate work schedule x Two jobs scheduled concurrently x No or inadequate pre job briefing x Job plan changed during pre job briefing x Unclear responsibility assignments or priorities x Lack of contingency planning x Parts not ready or available x Work package or plan not sufficient or appropriate for job

6. Work Practices - anything that the worker did not perform iaw accepted standards, including failure to correctly use a human performance tool to mitigate an error likely situation (remember the worker is the last line of defense).

x HU Tool not used or not used properly x Failure to use procedure x Failure to follow procedure x Inattention to detail x Failure to follow skill of the craft norms x Failure to record information correctly x Short cuts used

7. Training and Information Sharing - anything the worker does not know how to do related to formal training, informal training, and supervisory mentoring and day to day communications.

x Worker did not have required knowledge x Worker did not have required skill x Worker was not provided training x Continuing training not sufficient

ROOT CAUSE EVALUATION MANUAL Revision: 17 FG-PA-RCE-01 Page 53 of 53 ATTACHMENT 17 (continued)

DIRECT CAUSES x Time since job last performed too long x Training received too long ago x Training not realistic - didnt mimic real job properly x Supervisory information not communicated (e.g., D-15) x Procedure change not communicated x Worker unaware of new requirement x Standard not sufficiently or routinely reinforced x Routine job feedback not provided

8. Change Implementation - anything associated with properly managing a change.

x Change not recognized x Change Management Plan not prepared when required x Change Management Plan inadequate 3/4 Training not provided when needed 3/4 Procedure not revised when needed 3/4 Communication plan lacking x Change Management plan not implemented correctly

9. Risk Management - nuclear, radiological, industrial, environmental or regulatory risk not managed.

x Risks not identified x Risks not effectively assessed x Risk mitigation plan not sufficient x Risk mitigation plan not followed x Individual work practices not consistent with level of risk x Workers not aware of risk

10. Equipment problem - any problem associated with a piece of equipment not operating as required.

x Misalignment x Corrosion or pitting x Out of tolerance or specification x Seized or frozen x Overheating x Age related degradation x Faulty design x Deformation or collapse x Excessive wear x Manufacturing defect