ML102460630
Text
NSP000033 QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 1 of 49 RCE REPORT Prairie Island Nuclear Generating Plant 11 Turbine-Driven Auxiliary Feedwater Pump Discharge Pressure Switch Manifold Isolation Mispositioning Event Date: July 31, 2008 RCE 01146005 CAP AR 01146005 RCE Team Members:
Root Cause Investigator: Kelly Vincent Team Member: Ryan Baker Team Member: Andy Notbohm Team Mentor: Gene Woodhouse Team Leader: Jeff LeClair Team Sponsor: Len Clewett Approvals:
RCE Team Leader Date Management Sponsor Date Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 2 of 49 Table of Contents Page #
I. Executive Summary.................................................................................................... 3 II. Event Narrative ........................................................................................................... 7 III. Extent of Condition Assessment................................................................................. 9 IV. Previous Similar Events:........................................................................................... 10 V. Operating Experience: .............................................................................................. 12 VI. Nuclear Safety Significance...................................................................................... 14 VII. Reports to External Agencies & the NMC Sites....................................................... 14 VIII.Data Analysis............................................................................................................ 15 A. Information & Fact Sources.................................................................................. 15 B. Evaluation Methodology & Analysis Techniques ................................................ 15 C. Data Analysis Summary ....................................................................................... 16 D. Failure Mode Summary ........................................................................................ 16 IX. Root Cause and Contributing Causes ....................................................................... 17 X. Corrective Actions .................................................................................................... 18 XI. References................................................................................................................. 21 XII. Attachments .............................................................................................................. 22 Attachment 1 (Event and Causal Factor Chart) ........................................................ 23 Attachment 2 (Interview List)................................................................................... 24 Attachment 3 (Configuration Control Evaluation Matrix) ....................................... 25 Attachment 4 (Why Staircase) .................................................................................. 27 Attachment 5 (Barrier Analysis)............................................................................... 28 Attachment 6 (Configuration Control Barrier Analysis) .......................................... 30 Attachment 7 (Process Failure Modes Analysis)...................................................... 32 Attachment 8 (Organizational and Management Failure Modes)............................. 36 Attachment 9 (Human Performance Failure Modes)................................................ 41 Attachment 10 (DRUM Report Analysis) ................................................................ 48 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 3 of 49 I. Executive Summary Problem:
On 7/31/08, both Unit 1 auxiliary feedwater pumps (AFW Pumps) auto started following the Unit 1 reactor trip. The 11 turbine driven auxiliary feedwater pump (11 TDAFW Pump) tripped 42 seconds later. Subsequent investigation found the instrument manifold isolation valve for the discharge pressure switch to be out of position (closed vice open).
Isolation of the switch caused the pump to trip on a low discharge pressure.
Event Synopsis:
Between March 11, 2008 and July 31, 2008, an I&C technician or plant operator inadvertently operated the manifold block isolation valve for Pressure Switch PS-17700 (11 TDAFWP Lo Discharge Pressure Trip Pressure Switch). During that time period, there were seven surveillance procedures (SPs) completed (SP 1301, SP 1376, SP 1103, and 4 occurrences of SP 1102) that operated valves in the vicinity of the PS-17700 manifold isolation valve. These valves are identical in design to the PS-17700 manifold valve and in close proximity to the valve. There are no steps in the procedures to check the position of PS-17700 manifold isolation valve because none of the procedures operate this valve. None of the personnel interviewed remember any issues with the operation of the manifold valves during their associated surveillance procedures.
Figure 1: PS-17700 Valve Arrangement PS-17700 manifold isolation PS-17700 AF-195-1 AF-15-9 11 TDAFWP discharge pipe The Technical Specification required frequency to check the functionality of PS-17700 is every refueling outage, and was completed via SP 1301 during 1R25. The calibration of PS-17700 is completed every refueling outage using SP 1234A (completed on 23 February 2008). SP 1234A operated the PS-17700 manifold valve and required independent verification of the valve position. Additionally, C1.6A1-1 (Integrated Operations Checklist Prior to Heatup First Floor Turbine Building) verified the position of the PS-17700 manifold valve during the outage.
==
Conclusions:==
The function of this valve is to isolate PS-17700 (see Figure 1). This is the same function as AF-195-1, which is the root isolation valve for PS-17700.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 4 of 49 AF-195-1 has the following configuration controls (attachment 6) associated with it:
- It is identified in a drawing
- Exists in the equipment database
- It is labeled
- It has a safeguards hold card
- It utilizes Blocks Plus
- It has a locking device The PS-17700 manifold isolation valve has none of these configuration controls, even though it has the same functionality as AF-195-1.
There are many tools available that maintain configuration control of the plant. These tools include, but are not limited to:
- a. Drawings
- b. Procedures
- c. Valve Lineups
- d. Labeling
- e. Safeguards Hold Cards
- f. Maintenance Work Instructions and Clearance Order Instructions
- g. Equipment Database Entry
- h. Locking, Blocking, and/or Lock Wire The root cause evaluation determined that, since no procedures conducted after C1.6A.1-1 was last performed physically operate the manifold isolation valve for PS-17700, the most likely cause of the mispositioning was inadvertent operation. The most effective tool at avoiding inadvertent mispositioning would be locking the valve. Other tools would be effective at raising awareness of and controlling instrument isolation valves.
Examples of these tools are:
- Including the valve in the equipment database
- Complete valve lineups
- Inclusion on drawings
- Identification with Labels Nuclear Safety Significance:
Probabilistic Risk Assessment (PRA) Input The Auxiliary Feedwater (AFW) System is an important system in the PRA model. The modeled function of the AFW System is to supply makeup water to the steam generators following a reactor trip in which Main Feedwater is unavailable.
Following the trip of 11 TDAFW Pump on July 31st, 2008, the pump was declared unavailable and the risk impact was managed according to plant procedure H24.1. The Core Damage Frequency (CDF) and Large Early Release Frequency (LERF) risk levels were assessed as yellow and green, respectively.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 5 of 49 The risk impact of the 11 TDAFW Pump unavailability is currently being evaluated under the NRC Significance Determination Process (SDP). The significance of the issue is being divided into three parts: Internal events PRA, Fire PRA and Seismic PRA. All three areas are in the process of being evaluated and the overall risk significance of 11 TDAFW Pump unavailability will not be determined until completion of the SDP evaluations. The expected completion date is no later than the end of 2008.
Safety Conscious Work Environment (SCWE)
The investigation revealed no concerns with respect to the safety culture at Prairie Island.
There is no evidence that an environment exists in which employees would not feel free to raise concerns to their management and/or the Nuclear Regulatory Commission (NRC) without fear of retaliation. Employees are encouraged to raise concerns, and have many avenues available to allow them to do so. Prairie Islands policy for prohibiting harassment or retaliation for raising nuclear safety concerns is strictly enforced.
Root Cause:
Between March 11, 2008, and July 31, 2008, an I&C technician or plant operator inadvertently operated the manifold block isolation valve for PS-17700 while performing scheduled plant maintenance.
The root cause is inadequate configuration controls for components that have the potential to adversely impact the design function of safety related Structures, Systems and Components (SSCs).
Causal Factors:
o The mispositioned valve was not locked in the required position, making mispositioning more likely. SWI O-3 does not define which valves shall have locks, blocks or lock wires installed.
o SWI O-3 (Safeguards Hold Cards and Component Blocking or Locking) contains a definition of what components should not be controlled under SWI O-3, but does not contain a definition of which components should be controlled per the safeguards hold card program.
o The mispositioned valve was not labeled, bypassing barriers to make identification more likely. Site procedure 5AWI 3.10.5 is not aligned with INPO document 88-009 (Sections 6.3.1.1 and 6.3.1.3).
Evaluation of past recurring issues involving components that affect the function of safety related systems have not implemented corrective actions of sufficient scope to remedy problems in other safety related systems.
This event occurred partially as a result of a human performance issue. While analysis of the actual event could not be performed because the root cause team could not determine the actual time and date the discharge pressure switch was isolated, the following tools are in place to minimize the occurrence and severity of human performance errors:
- STAR / Self-Checking
- Peer Checking Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 6 of 49
- Procedural Adherence
- Place Keeping
- Verbal Communications
- Co-worker Coaching
- Are You Ready Checklist During interviews, significant time was spent discussing human performance tools and issues with personnel across different plant departments. All personnel indicated a strong familiarity with and use of human performance tools. Interviews indicated that the loud environment in the Auxiliary Feedwater Pump room makes verbal communication difficult and in one instance a junior operator did require and receive effective coaching.
Neither of these issues contributed to the occurrence of this event.
Contributing Cause:
An unrecognized operator burden exists, creating an error likely situation that has been present for several years. To correct a condition during limited plant conditions that over-ranges the turbine-driven pump suction gages, several procedures were changed that kept the gage isolated, except when needed for operation. This procedural change resulted in additional valve manipulations in the vicinity of PS-17700 manifold isolation valve.
o Based on interviews, at the time this issue was discovered the engineering change process did not have adequate means of prioritizing engineering change requests to ensure operator burdens had an adequate priority to be promptly resolved. Subsequent revisions to the EC procedure have corrected related deficiencies.
o The gage lines between the manifold isolation valves for the suction pressure gage and discharge pressure switch are crossed for 11 TDAFW Pump, but not for the other pumps. As a result, the increased frequency of valve operation combined with the crossover (error trap) increased the likelihood of this mispositioning event.
Corrective Action Synopsis:
The review of previous similar events, extent of condition and extent of cause assessments indicates that mispositioning of components are common and difficult to prevent throughout the industry.
Corrective actions from past evaluations have focused on human performance. These corrective actions have been effective at reducing valve misposition events in the short term, but have not been effective at long-term resolution of underlying issues.
Administrative controls have been instituted to eliminate valve mispositioning events, including the formation of valve misposition teams. These teams had the effect of raising focus on the issue, and were reactive in nature. They did little to install barriers to prevent component mispositioning and have not proven effective at minimizing the frequency or severity of mispositioning events.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 7 of 49 Evaluations have been completed that identified issues with components that affect system operation, but have been limited in scope to correct issues in individual systems.
This root cause is a concern in all plant safety-related systems and must be corrected accordingly. This is the primary focus of this evaluations corrective actions. Attachment 3 details the approach to grouping the effects of component mispositions. The evaluation demonstrates that there are currently adequate configuration control tools in place to control components in the plant, but that these controls have not been effectively implemented on all significant components in all systems. There are adequate human performance tools in place to minimize the number of valve mispositioning events when combined with effective controls.
The corrective actions to prevent recurrence (CAPRs) include revision of site configuration control procedures, specifically SWI O-3. These revisions are necessary to put in place the correct configuration control methodologies to ensure this issue is resolved.
CAPRs include formation of a project team to evaluate all safety related systems to determine if there are other components that if mispositioned, might prevent a safety related system from performing its design function. The team will also determine adequate procedural guidance for these components. To limit the scope and maximize effectiveness of the project team, this evaluation should specifically address the level B components as determined in the root cause evaluation matrix (Attachment 3). It is recommended that these level B components be included in the equipment database and drawings, locking devices be installed and that level B components be labeled in the field.
Interim action to mitigate the configuration control issue is to maintain the locking devices on the Auxiliary Feed Water system discharge and suction pressure switch manifold isolation valves until the project team formalizes the controls required for level B components.
Reports to External Agencies:
Inoperability of 11 TDAFW Pump is reportable per 10CFR50.73(a)(2)(i)(B) as a condition prohibited by Technical Specifications. It will likely also be reportable as a condition that could have prevented fulfillment of a safety function per 10CFR50.73(a)(2)(v). An evaluation is being completed under CAP 01146005-09, and is assigned to licensing.
An Operating Experience report will be submitted to INPO per corrective action assignment 01146597-03 once the root cause is completed and approved.
An internal Operating Experience report was submitted on August 5, 2008. [Provide information on any reports to external agencies]
II. Event Narrative Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 8 of 49 At the end of the Unit 1 outage (1R25) in February 2008, SP 1234A was performed on 11 TDAFW Pump to calibrate the discharge pressure switch on the pump. The final position of the isolation valve to the switch was open and independently verified and a valve lineup per C1.6A.1-1 verified the position on 11 March 2008.
SP 1301 was performed to test all trip and safety related functions of 11 TDAFW Pump and SP 1376 was performed to verify operation of system valves. SP 1103 was performed to verify post-outage AFWP flow. During each of the four subsequent months, SP 1102 was also performed to test manual pump operation. None of these procedures manipulated the PS-17700 isolation valve, but all required the operation of isolation valves in close proximity and of the same type as the isolation valve to the discharge pressure switch (see photograph below).
On July 31, 2008 unit 1 experienced a reactor trip due to a Foxboro unit failure during testing. Subsequent to the trip, 11 TDAFW Pump auto started as designed and tripped 42 seconds later on low discharge pressure. Local and remote indications did not indicate any abnormal operating conditions. A prompt investigation into 11 TDAFW Pump trip revealed the discharge pressure switch was isolated. A time delay in the pump protective circuitry is designed to trip the pump after 35 seconds of a low discharge pressure condition. Monthly SPs (SP 1102) to test the operability of the TDAFW Pump do not test the low discharge pressure trip function of the pump as the selector switch for 11 TDAFW Pump is in manual, bypassing the discharge low pressure trip.
An investigation into the cause of the isolation of 11 TDAFW Pump discharge pressure switch revealed the root cause was a failure of the site to adequately control components that affect safety related equipment. Although there were no repetitive or corrective maintenance activities that operated this valve since the outage, the monthly SP 1102 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 9 of 49 does operate isolation valves similar in location and identical in valve type. The operation of this valve in proximity to the valve in question for the SP was an unidentified operator work around. As part of the initial investigation into this event, the site determined that tampering was not evident with this incident.
II. Extent of Condition Assessment
- Extent of Condition -
Mispositioned minor valves occur frequently in nuclear power. The amount of configuration controls placed on minor valves is dependent on the consequences of the valve being out of position, as well as the frequency at which a valve is operated. Minor valves, such as instrument manifolds, may exist in a large number of systems throughout the plant; however their presence in situations that could affect the function of safety related systems (see Attachment 3) is much more limited. Mispositioned valves will continue to remain unidentified until they cause an adverse affect in their associated system since the majority of these valves on site are not controlled using physical barriers.
A review of Operations DRUM reports from the first quarter of 2007 to present show a substantial fraction of component mispositioning to be non-safety related [See 0].
- Extent of Cause -
The configuration control failures, as noted in this root cause evaluation (Attachment 3),
carry ramifications for all systems containing safety-related equipment. A review of procedures pertaining to proper control of safety-related equipment exposed a weakness in defining which components require control as well as the manner in which controls should be implemented. As a result, many minor, safety related components are not adequately controlled. Minor safety related components, such as manifold valves, are not currently included in equipment databases and are not shown on plant drawings containing safeguards equipment. Thus, this condition extends through all of the following:
- All systems containing safety-related equipment
- Procedures pertaining to proper control of safeguards equipment, to include:
-SWI O-3
-5AWI 3.10.5
- Site drawings containing safety-related systems
- Equipment databases While the extent of this condition is broad, the scope of its focus is limited to mechanical isolation components. Sufficient controls for minor electrical and electronic safety related components already exist in the form of:
- Physical barriers [locked cabinets, terminal box covers, etc]
- Clear and consistent labeling
- Inclusion on site drawings Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 10 of 49 III. Previous Similar Events:
- ACE00264715 (8/2002) o An Apparent Cause Evaluation (ACE) was performed to determine the cause for the increasing trend in component mispositionings. The apparent cause was determined to be a lack of consistent use of human performance tools, and that the use of human performance tools is not ingrained or accepted by all plant staff. Specifically, a questioning attitude and attention to detail were lacking in all of the events. The corrective actions resulting from this evaluation were to increase the involvement of supervisors in the field observing and setting standards; and implementation of new pre-job briefing requirements and expectations.
- ACE00400382 (1/2003) o An ACE was completed to determine the cause for an adverse trend in component mispositionings. During the month of January 2003, there were five mispositionings that occurred within a four week period. The evaluation determined the causes to be that error likely situations were created, and that human performance tools could have prevented these issues from happening. The corrective actions included a re-emphasis of expectations with respect to self-checking, and an increase in the number of field observations with an emphasis on maintaining configuration control.
- ACE00444932 (5/2003) o Between April 1, 2003 and May 22, 2003 there were nine component mispositionings at Prairie Island. An ACE was completed to determine the causes. The trends/causes were determined to be: Lack of a questioning attitude; poor use of STAR; components not identified on prints; no tracking method for component manipulations; and lack of operator knowledge. Corrective actions included: Providing a team notes article to reinforce the benefits of a questioning attitude; reinforce that ANY component manipulation is to be tracked; reinforce managements expectations of using STAR; and develop a method to track components manipulated during a shift. This last corrective action created the configuration control card.
- ACE00818979 (3/2005) o In February/March of 2005, the site had seven component mispositionings occur within a seven week timeframe. After examining the different situations the apparent cause was determined to be: Inconsistent use of human performance tools and techniques to prevent errors from occurring.
Corrective actions: For every job that is to be performed, there will be a pre-job brief; and an operations department human performance improvement plan was created. Also, a corrective action requiring operations personnel to read the pre-job brief AWI, and to reinforce the importance of pre-job briefs and the Stop When Unsure human performance tool.
- AR01000100 (10/2/2005)
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 11 of 49 o Root isolation valve for PI-11363 (21 CS pump suction pressure gage) was found closed during performance of an SP. The apparent cause was a poorly written procedure that didnt address opening and closing the valve as needed. The corrective action from this event was to correct the procedures being used.
- AR01006092 (12/1/2005) o AFW pump suction gage block test valves were inadvertently shut. The cause was determined to be unfamiliarity with the type of valve being operated. The corrective action taken was to train the department involved, and to replace the suction pressure gages with improved gages so that operation of the isolation valves (the valves that should have been closed instead of the block test valves) wouldnt be required as frequently. (This last action was not completed).
- AR01006490 (12/5/2005) o During TP 1533 (Fuel Oil system underground leak test), valves required to be opened were found already in the open position; instrument root valve (required to be open) was found closed. Causes determined to be operation of unlabeled components, lack of knowledge of requirements for operation of instrument block valves and a poorly written procedure. The corrective actions installed labels on the associated valves, changed the procedure, and issued an operating instruction to inform operators of proper block valve operating instructions.
- AR01089219 (4/25/2007) o The manifold isolation valves for 12 and 21 motor-driven AFW pumps were found closed. Interviews led to the conclusion that the valves were improperly closed during turbine building data collection. The operator involved was remediated, and the data collection form was revised to eliminate the requirement to check motor-driven and turbine-driven AFW pump suction pressure. The write-up for this event highlights the need for rigor in configuration control any time a component is manipulated.
- AR01036956 (06/23/2006) o CL-113-2 (Cooling Water to 12 AFWP) was found out of position during the application of the locking device. This was a category A component.
The site clock was reset and an ACE was performed to understand how this occurred, but there was no cause identified for the mispositioned valve. There was additional guidance written into procedure to ensure hold cards are attached to hand wheels versus operating chains.
Common causes present throughout these examples are: Poor utilization of human performance tools; poorly written procedures; and poor component tracking/configuration control.
In the majority of these cases, the corrective actions involved training and raising the issue to the departments involved; revising the associated procedure; and in some cases installing labels on components. While these corrective actions were moderately effective Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 12 of 49 in the particular system/situation to which they were applied, they were noticeably ineffective in preventing similar occurrences at the site.
The corrective actions from these previous events were ineffective because they corrected the apparent issues that lead to the mispositionings, but failed to eliminate the underlying causes.
IV. Operating Experience:
Internal OE A search was conducted on the CAP database at Prairie Island for recent events related to valve mispositioning.
- CAP 01000100 (02 October 2005):
o Containment Spray Suction root isolation inadvertently isolated. The corrective action was to add configuration controls for the isolation valves.
- CAP 01006092 (01 December 2005):
o 12/21 AFWP pressure gages incorrectly believed to have been mispositioned. 11/22 AFWP gages are normally isolated and 12/21 AFWP gages are normally valved in. The difference in valve positions between the pumps contributed to the issue. Corrective actions included resolving pressure over-ranging issues, improve training and correct labeling deficiencies.
- CAP 01006490 (05 December 2005):
o Instrument block valve (121 Diesel Generator Fuel Oil Storage Tank Pump discharge pressure instrument) mispositioned. Corrective actions were to label the valves.
- CAP 01089219 (24 April 2007):
o 12/22 MDAFWP Instrument Block Valves mispositioned. Corrective actions were to implement configuration controls on the valves.
o Adverse trend in component mispositionings (May 2003). Performed apparent cause on nine events in a two month period. Corrective actions included training on site standards pertaining to configuration control and development of a method to track components not formally tracked otherwise.
o Adverse trend in component mispositionings (March 2005). Performed apparent cause on seven events in a six week period. Corrective actions included site-wide training on site standards with respect to configuration control and to take measures to improve pre-job brief quality.
o Adverse trend in component mispositionings (January 2003). Performed apparent cause on five events in a one month period. Corrective actions included training on site standards with respect to configuration control and human performance improvement tools.
- RCE 000200:
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 13 of 49 o CARDOX unavailability due to valve mispositioning. Root causes included a lack of training/familiarity, inadequate labeling, and inconsistent expectations for valve position verification (no physical position verifications were made). Corrective actions included changing the method of applying locks, institution of blocks plus, and administrative controls to require Independent Verification (IV) for isolations and restorations.
External OE A search was performed on the INPO database for industry events related to valve mispositionings. The following is a summary of related industry events:
- SER 56-84:
o Three cases of valves and controls being inadvertently mispositioned, resulting in safety-related systems failing to meet operability requirements. Contributing causes included improper configuration controls on components and the failure of tag clearances and valve lineups to identify and correct mispositioned valves.
- SOER 85-2:
o Four cases studied by INPO with respect to Human Performance Error induced valve mispositioning. In one case and in the summary, a contributing cause was noted to be a lack of identification labels on valves. The summary also emphasizes the importance of tracking valves that affect the operation of systems required for safe operation.
- SOER 87-3:
o Five cases of reactor trips resulting from improper positioning of Instrumentation and Controls valves and switches. Statistically, 50% of mispositioning reviewed occurred during maintenance and testing.
- SER 29-88:
o Four cases of Inadvertent Trips and loss of safety system functions due to mispositioned instrument valves. One case involved the block valve for the Auxiliary Feedwater Pump discharge pressure switch being inadvertently isolated. The summary notes that the corrective actions implemented by many stations as a result of SOER 85-2 were ineffective at addressing the root causes.
- SER 6-92:
o Seven cases of valve mispositioning resulting in a loss of safety system function. Five of seven cases involved improper lineups after system maintenance. The report stresses the importance of post-maintenance testing in the identification of mispositioned valves.
Reviewing industry and site OE shows that valve mispositioning is common and affects nearly every system in the plant. Previous corrective actions have been limited in scope and have usually concentrated on the affected system(s) as opposed to processes. This root cause has determined that valve mispositioning is a symptom and the processes are the root cause.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 14 of 49 V. Nuclear Safety Significance Probabilistic Risk Assessment (PRA) Input The Auxiliary Feedwater (AFW) System is an important system in the PRA model. The modeled function of the AFW System is to supply makeup to the steam generators following a reactor trip in which Main Feedwater is unavailable.
Following the trip of 11 TDAFW Pump on July 31st, 2008, the pump was declared unavailable and the risk impact was managed according to plant procedure H24.1. The Core Damage Frequency (CDF) and Large Early Release Frequency (LERF) risk levels were calculated by Equipment Out of Service (EOOS) and recorded on the Phase 1 Risk Assessment Worksheet.
Not Including Severe Weather
Unit 1: CDF = 5.72E-05/yr (YELLOW), LERF = 4.35E-08/yr (GREEN) with a risk informed Allowed Outage Time (AOT) of 7.4 Days.
Including Severe Weather
Unit 1: CDF = 9.57E-05/yr (YELLOW), LERF = 4.35E-08/yr (GREEN) with a risk informed Allowed Outage Time (AOT) of 4.2 Days.
Work Week Managers and Operations continued to re-evaluate the CDF and LERF values for various configurations encountered during the unavailability for 11 TDAFW Pump.
The risk impact of the 11 TDAFW Pump unavailability is currently being evaluated under the NRC Significance Determination Process (SDP). The significance of the issue is being divided into three parts: Internal events PRA, Fire PRA and Seismic PRA. All three areas are in the process of being evaluated and overall risk significance of 11 TDAFW Pump unavailability will not be determined until completion of the SDP evaluations. The expected completion date is no later than the end of 2008.
Safety Conscious Work Environment (SCWE)
The investigation revealed no concerns with respect to the safety culture at Prairie Island.
There is no evidence that an environment exists in which employees would not feel free to raise concerns to their management and/or the Nuclear Regulatory Commission (NRC) without fear of retaliation. Employees are encouraged to raise concerns, and have many avenues available to allow them to do so. Prairie Islands policy for prohibiting harassment or retaliation for raising nuclear safety concerns is strictly enforced.
VI. Reports to External Agencies & the NMC Sites Inoperability of the 11 TDAFW Pump will be reportable per 10CFR50.73(a)(2)(i)(B) as a condition prohibited by Technical Specifications. It will likely also be reportable as a Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 15 of 49 condition that could have prevented fulfillment of a safety function per 10CFR50.73(a)(2)(v).
An Operating Experience (OE) report will be submitted to INPO per corrective action assignment 01146597-03.
An internal Operating Experience report has been filed (8/5/2008) as part of the plant trip and an additional OE report will be submitted pending the results of the root cause.
VII. Data Analysis A. Information & Fact Sources
- Interviews were conducted with the following groups:
o Operations o Instrumentation & Control o Engineering o Scheduling
- The following data sources were used to obtain information in support of this root cause evaluation:
o Site surveillance procedures o Site administrative work instructions (AWI) o Fleet procedures o Site operations procedures o Vendor technical manuals o Site drawings o Industry operating experience o INPO good practice guides o Photographs o Action Requests B. Evaluation Methodology & Analysis Techniques Data for this evaluation was collected primarily by interviews, procedures, maintenance records, and logic diagrams. Photographs of the associated 11 TDAFW Pump instruments were also provided to the team, showing their isolation valves and tubing configuration.
Analysis of the data was performed using the following methods:
- Barrier Analysis (Attachment 5)
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 16 of 49
- Failure Mode Analysis (Attachments 7, 8, and 9)
- Why Staircase Analysis (Attachment 4)
- Event and Causal Factors Analysis (Attachment 1)
C. Data Analysis Summary The root cause and contributing causes were determined using interviews (Attachment 2),
why staircase analysis, barrier failure analysis, failure mode analysis and event and causal factor charting. Why staircase analysis determined that the root cause is inadequate configuration controls for components that adversely impact the design function of safety related SSCs. Failure mode analysis and barrier analysis indicated that a contributing cause was an unresolved operator burden which created an error-likely situation that existed for several years.
D. Failure Mode Summary Human Performance Failure Modes (Attachment 9)
Inattention (A1) - The worker performing the SP failed to pay adequate attention to the valve manipulations they were performing.
Spatial Disorientation (J2) - Multiple, unlabeled block valves in close proximity could have led the operator to manipulate the wrong valve.
Mindset/Preconceived Idea (J3) - A preconceived mindset on the layout of the system pressure gage/pressure switch and associated isolation valves may have played a part in causing this event. The layout of the discharge pressure switch and the suction pressure gage is potentially confusing. The manifolds for these components are not located below the associated piece of equipment, as it typically is with other components of this type.
Wrong Assumptions (J4) - Personnel making the wrong assumption about the isolation valve associated with the suction pressure gage contributed to causing this event.
Inadequate Verification (J5) - The worker performing the procedure failed to properly verify the valve they were operating.
Work Around (J8) - The requirement to unisolate the suction pressure gage to take a reading, then re-isolate the gage, is an operator work around. The need for this step is to prevent over-ranging the gage. If gages with the proper range and accuracy were installed, this step would be unnecessary. The need for this step creates an error-likely situation.
Organizational and Management Failure Modes (Attachment 8)
Inadequate Prioritization(F3) - The Engineering Change (EC) process failed to identify and prioritize an operator burden in the operation of the PI-11054 gage isolation valve. This failed to eliminate a potential error-likely situation or the operator burden.
Inadequate Planning (F4) - The work to replace the suction pressure gages was not adequately planned or executed.
Process Failure Modes (Attachment 7)
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 17 of 49 Critical Actions not Verified (AR1) - The person performing this task failed to verify the valve they were manipulating.
No Process Monitoring (AR3) - SP 1102 does not test the functionality of the discharge pressure switch when in manual control. Running the pump in auto would have resulted in the discovery of the mispositioned valve before the pump was required for operation.
Only Monitoring Problems (AR4) - The site was not aware of the mispositioned valve until it caused a problem. Checks should have been in place to identify that this valve was out of position before it caused the pump to trip, instead of waiting for the trip to identify the valve being out of position.
VIII. Root Cause and Contributing Causes Root Cause:
Between March 11, 2008, and July 31, 2008, an I&C technician or plant operator inadvertently operated the manifold block isolation valve for PS-17700 (discharge pressure switch for 11 turbine driven auxiliary feed pump), while performing scheduled plant maintenance. The root cause is inadequate configuration controls for components that have the potential to adversely impact the design function of safety related SSCs.
Causal Factors:
o The mispositioned valve was not labeled, bypassing barriers to make identification more likely. 5AWI 3.10.5 is not aligned with INPO document 88-009 (Sections 6.3.1.1 and 6.3.1.3).
o SWI O-3 (Safeguards Hold Cards and Component Blocking or Locking) contains a definition of what components should not be controlled under SWI O-3, but does not contain a definition of which components should be controlled per the safeguards hold card program.
o The mispositioned valve was not locked in the required position, making mispositioning more likely. SWI O-3 does not define which valves shall have locks, blocks or lock wires installed.
Evaluation of past recurring issues involving components that affect the function of safety related systems have not implemented corrective actions of sufficient scope to remedy problems in other safety related systems.
This event occurred as a result of a human performance issue. While analysis of the actual event could not be performed because the root cause team could not determine the actual time and date the discharge pressure switch was isolated, the following tools are in place to minimize the occurrence and severity of human performance errors:
- STAR / Self-Checking
- Peer Checking
- Procedural Adherence
- Place Keeping
- Verbal Communications Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 18 of 49
- Co-worker Coaching
- Are You Ready Checklist During interviews (Attachment 2), significant time was spent discussing human performance tools and issues with personnel across different plant departments. All personnel indicated a strong familiarity with and use of human performance tools.
Interviews did indicate that the noisy environment in the Auxiliary Feedwater Pump room does make verbal communication difficult and in one instance a junior operator did require and receive effective coaching regarding communication in a noisy area while performing an evolution.
Contributing Cause:
An unrecognized operator burden exists, creating an error likely situation that has existed for several years. To correct a condition during specific conditions that over-ranges the turbine driven pump suction pressure gage, several procedures were changed that kept the gage isolated, except when needed. This action required additional valve manipulations in the vicinity of the PS-17700 manifold isolation valve.
o Based on interviews, at the time this issue was discovered the engineering change process did not have an adequate means of prioritizing engineering change requests to ensure operator burdens had an adequate priority to be promptly resolved. Subsequent revisions to the EC procedure have corrected related deficiencies.
o The gage lines between the manifold isolation valves for the suction pressure gage and discharge pressure switch are crossed for 11 TDAFW Pump, but not for the other pumps. As a result, the increased frequency of valve operation combined with the crossover (error trap) increased the likelihood of a mispositioning event.
IX. Corrective Actions Corrective Actions to Restore (Broke-Fix)
The following corrective actions have been implemented to correct the condition for 11 TDAFW Pump and other components subject to the same issue, per CAP 01146005-11 and -12:
- 2) Conducted SPs to verify system operability a) Operations completed SP 1102 with a temporary change request (TCR) to run the pump with the Control Switch in Auto.
b) I&C completed SP 1234A (Pressure Switch Calibration) on 11 TDAFW Pump to verify switch functionality per CAP 01146005-11 Interim Corrective Actions (Mitigation)
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 19 of 49
- 1) The positions of other valves with similar functions in AFW System have been checked. The suction and discharge pressure switch manifold isolation valves for all four Auxiliary Feed Water Pumps were lock-wired in the open position per CAP 01146005-12.
- 2) A sampling of valve positions was completed to verify the positions of valves with similar functions. Checklists C1.6A.3-1, C1.6A.1-2, and C1.6A.1-1 were completed with no discrepancies noted (CAP 01146005-12).
- 3) For outage 2R25, the checklists required in C1.6A-2 will be completed with both initial and independent verification for all components. (CAP 01146005-27)
Corrective Actions to Prevent Recurrence (CAPRs)
- 1. (AR 01146005-17) Utilize the five-phase process (per FG-E-MOD-02, rev 4 and http://pinet/businessplanning/PRG.htm), to conduct a comprehensive review of site configuration control standards. The RCE team has determined this activity to be greater than level of effort for the site due to the lack of representative drawings and database information.
a) Develop a process to review safety related systems to determine if there are any small components that may adversely affect the function of the safety related SSCs. Trial the methodology on a significant safety related system (recommend AFW) A proposed methodology is:
i) Determine safety related functions for each system ii) Determine inputs to ensure safety related functions via logic diagrams iii) Determine location of input devices to trips on logic diagrams.
iv) Determine various means of affecting input devices via system walkdowns v) Initiate Engineering Changes to input components that affect safety related components into the equipment database and make drawing changes. All components affecting the function of safety related SSCs shall be maintained in the equipment database and shall be included in the appropriate system P&ID drawing.
vi) Determine the appropriate checklist the valves should be on (typically SWI O-
- 3) and add the component into the checklist using the PCR process.
vii) Determine an appropriate means to lock the component in the correct position and install locking device on component viii) Label the device with unique identifier obtained from EC above.
Owner: Tom Verbout Due Date: 03/15/2009 b) For each safety related system, complete this process to systematically identify all components that may adversely affect safety related SSCs, and implement changes per the process developed in A) above.
Owner: Tom Verbout Due Date: 06/01/2010
- 2) (AR 01146005-18) Revise SWI O-3 to incorporate the following changes:
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 20 of 49 a) Rewrite the definition of components requiring control to be inclusive vice exclusive (current wording is of the procedure is that The use of Safeguards Hold Cards, with Locking or Blocking devices SHALL be limited to those components that could pose a threat to the safe operation of the reactor if inadvertently mispositioned during normal operation.)
b) Separate locking/blocking requirements from Safeguards Hold Card Requirements.
Implementation of these changes, including evaluation of training requirements, and additional process and/or procedure changes, will be included as part of the PCR process.
Owner: Terry Bacon Due Date: 03/15/2009 Other Corrective Actions
- 1) (AR 01146005-19) Initiate work request and complete work order to reroute piping between PI-11054 (Suction Pressure Gage) and its isolation manifold and between PS-17700 (Discharge Pressure Switch) to remove the human performance error trap.
Owner: Gary Wheelock Due Date: 03/15/2009
- 2) (AR 01146005-20) Complete EC 7454 to replace PI-11054 (Suction Pressure Gage) with a gage suitable for all expected operating conditions to remove risk of over-ranging.
- a. (AR 01146005-21) Revise all applicable procedures to maintain PI-11054 (Suction Pressure Gage) unisolated except when specifically required to be isolated for maintenance or testing Owner: Gary Wheelock Due Date: 06/15/2009
- 3) Revise 5AWI 3.10.5 to:
- a. (AR 01146005-22) Benchmark industry standards for labeling of small components that could affect the performance of safety-related systems (referred to as Class B components in attachment (3), evaluation matrix).
- b. (AR 01146005-23) Align with the requirements of INPO 88-009 Sections 6.3.1.1 and 6.3.1.3 (Uniquely identifying all instrument block valves) as pertaining to components that could adversely affect safety-related systems.
Owner: Terry Bacon Due Date: 03/15/2009
- 4) (AR 01146005-24) Revise SWI O-3 to enact the following changes:
- a. Benchmark industry standards with respect to control of components that affect safety-related systems and determine if proposed changes to SWI O-3 are aligned with industry best practices.
Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 21 of 49
- b. Benchmark use of Blocks Plus against the industry to determine if benefits outweigh potential error traps from allowing verification by use of colored blocks as opposed to physical verification.
- c. Recommended controls include entry into the equipment database, inclusion on drawings, and the use of labels and locking devices.
Owner: Steve Seilhymer Due Date: 06/15/2009 Effectiveness Reviews
- 1) Conduct trend review of Operations DRUM Reports for the next four quarters after the project is completed to identify trends in mispositioning of small components related to the operation of safety-related systems. Compare to trend of mispositioned valves before the corrective actions to prevent recurrence have been implemented to gain an understanding of the effectiveness of these corrective actions. An effective measure of no significant (group A or B components, as defined in this root cause) within the four quarters of reviewed drum reports, is acceptable.
Root Cause/Contributing Cause CAPR/CA EFR RC1 CAPR #1, CAPR #2, CA #4 EFR #1 CC1 CA #1, CA #2, CA #3 N/A Owner: Len Clewett Due Date: 06/15/2010 X. References
- FG-PA-RCE-01, Root Cause Evaluation Manual, Rev. 14
- FP-OP-COO-01, Conduct of Operations, Rev. 4
- FP-OP-TAG-01, Fleet Tagging, Rev. 5
- 5AWI 3.10.1, Methods of Performing Verification, Rev. 14
- 5AWI 3.10.5, Plant Equipment Labeling, Rev. 13
- 5AWI 3.10.8, Equipment Problem Resolution Process, Rev. 12
- 5AWI 15.5.1, Plant Equipment Control Process, Rev. 25
- C1.6A.1-1, Unit 1 - Integrated Operations Checklist Prior to Heatup First Floor Turbine Building, Rev. 10
- Crew Shift Schedule, 2008
- H3.1, Outplant Labeling Standards, Rev. 9
- H24.1, Assessment and Management of Risk Associated with Maintenance Activities, Rev. 13
- ICM-01.01, Instrument Control Manual, Rev. 14 (Monticello)
- INPO 88-009, System and Component Labeling, June 1991 Form retained in accordance with record retention schedule identified in FP-G-RM-01.
QF-0433, Rev 1, (FG-PA-RCE-01) RCE Report Template Page 22 of 49
- Nuclear Oversight 2nd Quarter of 2008 Assessment Report for Prairie Island, dated August 13th, 2008
- NRC Generic Letter No. 96-01, Testing of Safety-Related Logic Circuits
- Procedure 2161, Plant Prestart Checklist Process Instrumentation, Rev. 28 (Monticello)
- SP 1101, 12 Motor-Driven Auxiliary Feedwater Pump Quarterly Flow and Valve Test, Rev. 48
- SP 1193, Cycling AFWP and CLG Water MVs, Rev. 33
- SP 1301, 11 Turbine Driven Auxiliary Feedwater Pump Auto Start and Functional Refueling Outage Test, Rev. 22
- SP 1355A, Train A AFW Check Valve Testing, Rev. 11
- SWI O-3, Safeguards Hold Cards & Component Blocking or Locking, Rev.
77
- 11 TDAFW Pump Trip HU Review, AR01146005
- Technical Manual NX-48238-1, Hand valves and Manifold Instrumentation, Rev. 1
- Unit 1 Trip Sequence of Events (SOE)
- Work Order Package for WO00153953-01
- Work Order Package for WO00316496-01
- XH-106-229 Anderson, Greenwood, and Co., Manifold Valve XI. Attachments
- Attachment 1: Event and causal factor chart
- Attachment 2: Interview List
- Attachment 3: Evaluation Matrix
- Attachment 4: Why Staircase
- Attachment 5: Barrier Analysis
- Attachment 6: Configuration Control Barrier Analysis
- Attachment 7: Process Failure Modes Analysis
- Attachment 8: Organizational and Management Failure Modes
- Attachment 9: Human Performance Failure Modes
- Attachment 10: DRUM Report Analysis Form retained in accordance with record retention schedule identified in FP-G-RM-01.
RCE 01146005 Event and Causal Factor Chart Attachment 1 (1833) 23 09 March 2008 11 March 2008 12 March 16 March 24 March 26 March 2008 01 May 2008 29 May 26 June 0817:40 1323 / 31 0619 / 01 February SP 1193 C1.6A.1-1 2008 2008 2008 SP 1102 SP 1102 2008 2008 31 July Jul 2008 August 2008 Complete Complete. SP 1301 SP 1376 SP 1103 AFWP AFWP SP 1102 SP 1102 2008 C28-2 2008 SP 1234A Cycle AFWP Verify Block Complete. Complete. Complete. Monthly. Monthly. AFWP AFWP Unit 1 Complete. SP 1234A Complete. MOV. Valve Open. Suction/ Cycle PI- Cycle PI- Cycle PI- Cycle PI- Monthly. Monthly. Trip / 11 Found Complete 11 TDAFWP PI-11054 Discharge 11054 11054 11054 Isolation 11054 Isolation Cycle PI- Cycle PI- TDAFWP 17700 11 Suction/ Isolation Trip Test. Isolation Isolation Open then Open then 11054 11054 Start. Block TDAFWP Discharge Cycled Open 17704 Open then Open then shut. shut. Isolation Isolation Valve Suction/
Press Switch then Shut. Isolated and Shut. Shut. Open then Open then Shut. Discharge Calibration. un-isolated. shut. shut. 0818:23 Valve Pressure 31 July Opened. Switch 2008 Cal.
1 1 11 1 1 1 TDAFWP 1
1 Trip.
Verification (Low and Validation: Suction/
17700 (11 TDAFWP Discharge Pressure Switch) Isolation Valve Inadvertently Closed) Discharge Verification Independent Pressure) and Validation:
Verification Independent Verification Discharge Pressure Switch Block 1500 / 01 August Valve Open Discharge 2008 SP 1102 Pressure AFWP Monthly).
Switch Block Cycle PI-11054 Valve Closed Isolation Open then shut.
3 1 1
2 1 Valve was not locked (made mispo Unrecognized operator burden Mispositioned valve did not have SWI O-3 defines what cannot be more likely). SWI O-3 does not created an error-likely a label. (5AWI 3.10.5 Not in line controlled , but does not identify provide for locking device without situation that persisted for with INPO 88-009) which components should be Safeguards Hold Tag.
several years controlled 4
Lines for PI-11054 (Suction Inadequate Configuration Controls for components Pressure Gauge Line) and PS-that have the potential to adversely impact the 17700 (Discharge pressure switch) design function of safety related SSCs are crossed (creating error-likely situation).
Page 23 of 49
RCE 01146005 Attachment 2 Interview List Position Reason Person Time/Date Scheduled I&C technician Last SP 1234A 2/23/08, Mike Chapeau 8/18/08 1000 SP 1301 3/12/08 I&C technician Last SP 1234A 2/23/08, David Machaj 8/13/08 1300 SP 1301 3/12/08 Engineering Configuration Control Chuck Rizzo 8/13/08 0930 Engineering Aux Feed Pump engineer Gary Wheelock 8/14/08 0930 Engineering IST engineer Doug Lalone 8/15/08 1530 Operations Work Tagouts, IV methods, Pre- Jeff Baartman 8/13/08 0930 Control job Briefs, valve Supervisor manipulations, valve configuration and control I&C Supervisor Pre-job Briefs, valve Jason Tribe 8/13/08 1330 manipulations, valve configuration and control Engineering Configuration control, ECR Nate Bibus 8/14/08 1115 supervisor process, change process Outplant Operator SP 1376 3/16/08 Mike Baartman 8/15/08 0730 SP 1102 5/29/08 Outplant Operator SP 1102 6/26/08 Matt Lawrence 8/18/08 0900 Outplant Operator SP 1102 6/26/08, and Troy Halvorson 8/18/08 0800 3/26/08 Outplant Operator SP 1102 3/26/08 Scott 8/12/08 1300 Christianson Outplant Operator SP 1102 5/1/08 Dan Robinson 8/15/08 0930 Outplant Operator SP 1102 5/1/08 John Alpers 8/15/08 0930 Outplant Operator SP 1102 5/1/08 Mike 8/12/08 1345 Pauzauskie Outplant Operator SP 1102 5/29/08 Scott Jablonski 8/15/08 0730 Outplant Operator SP 1102 5/29/08 Ricky Kuhn 8/15/08 0730 Outage Scheduler Outage scheduling Ed Heineman 8/18/08 1030 Page 24 of 49
RCE 01146005 Attachment 3 Diagram A: Configuration Control Evaluation Matrix used to determine methodologies and extents for configuration control Diagram A: Target Component Configuration Control Major Minor Evaluation Matrix Component Component Safety Related A B Affected SSCs Non - Safety Related C D Configuration Control Methodology:
In this matrix, target components are the components that are being manipulated.
Affected components are the equipment or systems affected by the operation of the component.
Major components are components currently labeled and part of the database. Minor components are not typically manipulated when operating the plant and have varying degrees of configuration controls implemented. The disparity in configuration control of minor components is typically the result of implementing a corrective action. Many manifold valves are lock-wired.
There are several methods for maintaining configuration control:
- a. Drawings
- b. Procedures
- c. Valve lineups
- d. Labeling
- e. Safeguards hold tags
- f. Maintenance Work Instructions and Clearance Order Instructions
- g. Equipment Database Entry
- h. Locking, Blocking, and/or Lock Wire Which of the methods used for configuration control for any given component depends on the item that is controlled, and the effects of not controlling the item (i.e., a misposition event).
Page 25 of 49
RCE 01146005 Attachment 3 Area A: This area represents larger components that affect the operation of safety related systems. This area has the most rigorous application of controls. There is one instance of a misposition occurring in this area (ACE 01036956) in the last several years. The issue was corrected during the installation of the safeguards hold cards, therefore the installed configuration control methods were effective barriers at preventing mispositions of this type.
Area B: This area represents the focus for the root cause evaluation. Components in this category include (but are not limited to) air header and instrument manifold isolation valves. CAPs 01146005, 01006092, 01089219, show a relatively high frequency of occurrence. The root cause investigation indicates this is the most problematic area because the likelihood of occurrence is high and the consequences may be severe.
Area C: Components in this area include main feed pumps, main steam cutout valves, and motor control centers without safeguards influence. While not controlling these components might cause operational hardship and difficulty maintaining the plant, a lack of control of these components should not affect safeguards functions. Most of these components do have adequate configuration controls.
Area D: Components in this area are primarily installed to support maintenance. Control of these components using many of the methods of configuration control would be impossible to administer and prohibitively expensive. CAPs 00861714, 01006490 are indicative of these types of issues.
Page 26 of 49
RCE 01146005 Attachment 4 WHY STAIRCASE Effect/Symptom Cause/Reason 11 TDAFW Pump tripped The discharge pressure switch (PS-17700) was isolated at the gage isolation.
Why was the discharge An operator or I&C technician pressure switch isolated? inadvertently closed the isolation valve.
Why did an operator or I&C Crossed lines between technician inadvertently close TDAFWP suction gage and the isolation valve? discharge pressure switch created an error-likely situation.
Requirement to unisolate pump suction gage and re-isolate during plant evolutions created an unidentified operator burden and increased the likelihood of a mispositioning event Why were these issues able to There were inadequate contribute to the inadvertent configuration controls in place operation of the valve? for small components that affect the operation of safety related systems.
Page 27 of 49
RCE 01146005 Attachment 5 11 TDAFW PUMP DISCHARGE PRESS SWITCH BLOCK VALVE BARRIER ANALYSIS Hazard Barrier Assessment Target SP operating Valves Procedure No personnel Valve in correct in Vicinity interviewed position expressed concerns with involved procedures Qualified Workers All personnel were appropriately qualified Job Planning No indication of ineffective Pre-Job Briefs Verification and IV in SP1234A after Validation manipulation.
1C.6A.1-1 verify lineup prior to startup.
Would not prevent inadvertent operation.
Supervisory Effective oversight Oversight of operations work.
Worker Practices All personnel interviewed displayed understanding of site standards for procedure use and adherence and IV as well as required actions for finding a valve out of position Block/Lock Not used and could have provided backup to operators
/ technicians to ensure correct valve was operated Labels Not used and could have aided in prevention of inadvertent operation by allowing operator to verify operation against procedure.
Page 28 of 49
RCE 01146005 Attachment 5 Hazard Barrier Assessment Target Other Inadvertent Physical barrier to Minimize personnel in Operation area of TDAFW vicinity and control Pump access to the valve Block/Lock Not used and could have prevented inadvertent operation Labels Not used but would not have prevented inadvertent operation without a procedural control.
Verification and Inclusion on periodic Validation safeguards hold card verification could have identified mispositioned valve.
Checklist would not prevent inadvertent operation.
Page 29 of 49
RCE 01146005 Attachment 6 CONFIGURATION CONTROL BARRIER ANALYSIS WORKSHEET Undesirable Existing Failed? How Barrier Failed (in this case) Why Barrier Failed (in this Missing Situation Barrier (Yes/No) case) Barriers?
Lack of Valve No Did not present an adequate shield Maintenance performed on No configuration Lineups against valve misposition. This system after valve lineup.
control for barrier was not applicable to prevent Lineups only required after major components this undesirable situation maintenance.
Drawings Yes Not used as barrier Valve was not present on Yes drawings Equipment Yes Information on valve was not Valve was not present in database Yes Database maintained and therefore, Entry significance was not understood Procedures No Procedures for maintenance were Did not fail. All examples of SPs No considered robust, and maintained maintained configuration control.
configuration control. Configuration Administrative procedures control procedures are not adequate. regarding configuration control need revision Work Yes Inadequate control of components Components are not maintained No Instructions operated in the vicinity of the target in the database and are therefore and Clearance component. not available to planners for Order inclusion in WO and C/O.
Instructions Failure of this barrier is due to failure of other barriers.
Locking, Yes Components that may adversely Component lock wiring was not Yes Blocking, affect safety related systems are not used and lock physically controlled to prevent wiring inadvertent operation Page 30 of 49
RCE 01146005 Attachment 6 Undesirable Existing Failed? How Barrier Failed (in this case) Why Barrier Failed (in this Missing Situation Barrier (Yes/No) case) Barriers?
Safeguards Yes Component not properly identified or Component was not identified as No Hold Tags tagged one requiring a safeguards hold tag; requirements are not specific as to which components need safeguards hold cards Labels No Component was not labeled (ad hoc Not used, but operator knowledge Yes labels were used on manifold valves requires identity of component indicating need for labels) and trace line back to isolation.
Page 31 of 49
RCE 01146005 Attachment 7 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Definition Applicability Supporting / Refuting Evidence Failure Mode Actions Not Specified The action(s) that an Not Applicable There is no evidence that not specifying actions contributed to the occurrence (RR1) individual or group must of this event.
perform to accomplish a task are not contained in the document or instruction.
Actions Not Clear The action(s) that an Not Applicable Actions not being clear did not play a part in causing this event.
(RR2) individual or group must perform to accomplish a task are not clearly described in the document or instruction.
Actions not within The action(s) that an Not Applicable Actions not being within the control of the individual did not contribute to the Control of the individual or group must occurrence of this event.
Individual (RR3) perform to accomplish a task cannot be performed as specified (physical constraints, do not have authority to dictate results, etc.).
Actions Conflict with The action(s) that an Not Applicable There is not evidence that actions conflicted with another process.
Another Process (RR4) individual or group must perform to accomplish a task conflict or contradict the actions specified by another document or instruction.
Actions Not Tied to The action(s) contained Not Applicable There is no evidence that actions were not tied to another process when Another Process When within one document or necessary.
Necessary (RR5) instruction does not reference supporting documents or instructions when necessary.
Page 32 of 49
RCE 01146005 Attachment 7 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Definition Applicability Supporting / Refuting Evidence Failure Mode Methods Not Clearly Action(s) are required by Not Applicable There is no evidence to suggest that methods not being clearly defined Defined (RR6) the document or contributed to the occurrence of this event.
instruction, but the method to accomplish the actions is not clearly specified by the document or instruction.
Unnecessary Actions The document or Not Applicable There is no evidence to suggest that unnecessary actions required contributed Required (RR7) instruction require the to the occurrence of this event.
performance of certain actions that is not really necessary to successfully perform the action.
Wrong Information The information provided Not Applicable There is no evidence that wrong information played a role in causing this (RR8) in the document or event.
instruction is incorrect.
Critical Actions Not Critical actions required Applicable The person performing this task failed to verify the action they took (i.e., the Verified (AR1) to successfully perform a valve they operated).
task are not verified within the process.
Page 33 of 49
RCE 01146005 Attachment 7 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Definition Applicability Supporting / Refuting Evidence Failure Mode Excessive Verifications The document or Not Applicable There is no evidence that excessive verifications played a part in causing this (AR2) instruction requires event.
excessive verification of completed steps or tasks.
Actions are verified, regardless of criticality to the task or the task has multiple reviews and verifications instead of a single, specific review.
No Process Monitoring There is no established Applicable SP 1102 does not test the functionality of the discharge pressure switch when (AR3) means of monitoring the in manual control. Running the pump in auto would have resulted in the success or failure of the discovery of the mispositioned valve before the pump was required for process. operation.
Only Monitoring The only method of Applicable See No Process Monitoring Above Problems (AR4) monitoring process performance is to observe problems when they occur.
No Acceptance Criteria No acceptable Not Applicable There is no evidence that a lack of acceptance criteria played a role in causing (AR5) performance parameters this event.
have been established for the process, procedure or task.
No One Specified to No one is specified Not Applicable There is no evidence that suggests that no on was specified to perform the task, Perform Task (I1) (either by title, group, or therefore this is not a contributing cause.
other means) as responsible for completion of the actions required by a document or instruction.
Page 34 of 49
RCE 01146005 Attachment 7 Process Failure Modes I = Individual Related RR = Roles & Responsibilities Related AR = Accountability Related Definition Applicability Supporting / Refuting Evidence Failure Mode More Than One Person More than one person or Not Applicable Having more than one person specified to perform the task is not a Specified to Perform group is specified (either contributing factor to this event.
Task (I2) by title, group, or other means) as responsible for completion of the actions required by a document or instruction.
Person Specified Not The person or group Not Applicable The person(s) specified to perform this task were able to perform this task; Able to Perform Task specified (either by title, therefore this is not a contributing cause.
(I3) group, or other means) as responsible for the completion of the required actions in a document or instruction is unable to perform the action. Typically because they do not have the skill or knowledge.
Page 35 of 49
RCE 01146005 Attachment 8 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Definition Applicability Supporting / Refuting Evidence Failure Mode Inadequate Span of Horizontal organizational Not Applicable There is no evidence to suggest that an inadequate span of control Control (S1) design - the number of contributed to the occurrence of this event.
personnel which a supervisor is responsible for is too large or too few for the groups oversight & responsibilities.
This often creates problems with task assignment and accountability.
Inadequate Levels in Vertical organizational design Not Applicable There is no evidence to suggest that inadequate levels in the organization the Organization (S2) - the number of levels or played a role in causing this event.
layers, from senior manager to employee is too many or too few for the given activity.
Creates problems with communication of expectations.
Insufficient Staffing Comprehensive organizational Not Applicable There is no evidence to suggest that insufficient staffing contributed to this (S3) design - the total number of event.
employees for which the company or group is designed are not filled. Often causes staff work overload and poor accountability.
Inadequate A breakdown in Not Applicable There is no evidence to suggest that inadequate communication within the Communication within communication (written or organizations played a role in causing this event.
an Organization (F1) verbal) within one organization or work group. Often leads to important issues not being addressed and critical process breakdown.
Page 36 of 49
RCE 01146005 Attachment 8 Definition Applicability Supporting / Refuting Evidence Failure Mode Inadequate A breakdown in Not Applicable There is no evidence to suggest that inadequate communication among Communication among communication (written or organizations played a role in causing this event.
Organizations (F2) verbal) among two or more organizations or work groups.
Often leads to a breakdown in processes that require several groups to participate.
Page 37 of 49
RCE 01146005 Attachment 8 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Deficiencies in determining Applicable The EC process failed to identify and prioritize an operator burden in the Prioritization (F3) which work takes precedence operation of the PI-11054 gage isolation valve. This failed to eliminate a over other work. Often leads potential error likely situation or the operator burden.
to unexpected equipment failures or failure to meet regulatory requirements.
Inadequate Planning Deficiencies in determining Applicable The work to replace the suction pressure gages was not adequately planned (F4) what work must be done, by or executed.
whom, when, and how long it will take. Often leads to staff work overload, budget over-runs and low morale.
Inadequate Emerging Deficiencies in determining Not Applicable There is no evidence that inadequate management of emerging issues Issues Management how to deal effectively with contributed to the occurrence of this event.
(F5) unexpected issues. Often leads to continual crisis management and low morale.
Inadequate Program Inadequate oversight of critical Not Applicable There is no evidence that inadequate program management played a role in Management (F6) work processes to ensure they causing this event.
function smoothly and effectively. Often results in program degradation over time or increased problems within those processes.
Inadequate Trust (C1) A lack of confidence in the Not Applicable There is no evidence that inadequate trust is a contributing cause to this workgroup or members of the event.
workgroup, or a disbelief in information shared. Often results in fractured work completion and stress levels.
Page 38 of 49
RCE 01146005 Attachment 8 Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Teamwork Constant friction among the Not Applicable There is no evidence that inadequate teamwork contributed to the (C2) workforce, or an unwillingness occurrence of this event.
to work with one another. This problem could exist within organizations or between organizations. Results in confusion within the ranks and a lack of information flow among the groups.
Page 39 of 49
RCE 01146005 Attachment 8 Organizational and Management Failure Modes S = Structural Issues F = Functional Issues C = Cultural Issues Definition Applicability Supporting / Refuting Evidence Failure Mode Inadequate Knowledge An inadequate Not Applicable There is no evidence that inadequate knowledge contributed to the occurrence (C3) understanding of the of this event.
work to be performed and how the work ties into the overall goals. Often causes individual errors to occur.
Lack of Commitment A lack of dedication to Not Applicable There is no evidence that a lack of commitment contributed to the occurrence (C4) the work. Often results of this event.
in inconsistent or unreliable performance by an individual or group.
Inadequate Self A failure to continually Not Applicable There is no evidence that inadequate self assessment contributed to the Assessment (C5) encourage feedback, occurrence of this event.
listens to customer input, or look at better ways to perform. Often creates a false sense of security and leads to complacency.
Page 40 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inattention (A1) Not paying attention to Applicable The worker performing the SP failed to pay adequate attention to the valve Type - SB the task requirements. manipulations they were performing.
Not paying attention to information in the immediate environment.
Bored (A2) Inadequate level of Not Applicable There is no evidence to suggest that personnel performing the work were Type - SB mental activity due to bored.
performance of repetitive actions or lack of activity.
Habit / Reflex (A3) Ingrained or automated Not Applicable There is no evidence to suggest that this event was caused by habits or Type - SB pattern of actions reflexes.
attributed to the repetitive nature of a well-practiced task or a natural response.
Tired & Fatigued (A4) Degradation of physical Not Applicable There is no evidence to suggest that the personnel involved were tired or Type - SB/RB/KB or mental abilities due to fatigued.
illness, a lack of rest, or influences associated with body rhythms.
Distracted & Conditions of task or the Not Applicable There is no evidence to suggest that the personnel involved were distracted or Interrupted (A5) work environment interrupted.
Type - SB require the individual to stop and restart a task, diverting the individuals attention from the task at hand.
Page 41 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Multi Tasking (A6) Performing two or more Not Applicable There is no evidence to suggest that multi tasking played a role in causing this Type - SB tasks simultaneously and event.
neglecting to perform a required element of one or more of the tasks.
Lapse of Memory (A7) Momentary loss of Not Applicable There is no evidence to suggest that a lapse in memory contributed to the Type - SB memory regarding occurrence of this event.
information previously learned and known.
Inadequate Tracking Method used to maintain Not Applicable There is no evidence to suggest that inadequate tracking played a part in (Place Keeping) (A8) control of information, causing this event.
Type - SB/RB necessary requirements, or status was not properly used.
Time & Schedule Urgency or excessive Not Applicable There is no evidence to suggest that time/schedule pressure had any effect on Pressure (A9) pace required to perform causing this event.
Type - SB/RB/KB the task. No spare time allotted or perception by the individual that a tight schedule exists.
Fear of Failure (A10) Apprehension regarding Not Applicable There is no evidence to suggest that a fear of failure caused this event.
Type - SB/RB/KB potential adverse consequences if the individual fails to perform at a high level, resulting in undesirable behaviors.
Page 42 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Imprecise Miscommunication Not Applicable There is no evidence to suggest that imprecise communications contributed to Communication (A11) resulting from error of the occurrence of this event.
Type - SB/RB omission or commission by the sender or receiver.
This includes breakdowns of the three-part communication process.
Cognitive Overload Mental demands on the Not Applicable There is no evidence to suggest that cognitive overload played a role in (J1) individual to maintain a causing this event.
Type - RB/SB high level of concentration while requiring recall of excessive amounts of information.
Spatial Disorientation Loss or misjudgment of Applicable Multiple, unlabeled block valves in close proximity could have contributed to (J2) place or time; wrong this event.
Type - SB/RB component, wrong train and wrong unit errors due to similarities in the environment.
Page 43 of 49
RCE 01146005 Attachment 9 Mindset / Preconceived The tendency of an Applicable A preconceived mindset on the layout of the system pressure gage/pressure Idea (J3) individual to make a switch and associated isolation valves played a part in causing this event. The Type - RB judgment based upon a layout of the discharge pressure switch and the suction pressure gage is preconceived mental potentially confusing. The manifold for these components is not located below model or preconditioned the associated piece of equipment, as it typically is with other components of bias that is not based this type.
upon the current information, conditions or indications.
Page 44 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Wrong Assumptions Judgments are made Applicable Personnel making the wrong assumption about the isolation valve associated (J4) without verification of with the suction pressure gage contributed to causing this event.
Type - RB the facts and are usually based upon the individuals perception of recent experiences or events.
Inadequate Verification Insufficient verification Applicable The worker performing the procedure failed to properly verify the valve they (J5) of the facts, and is were operating.
Type - RB usually based upon inaccurate information or the lack of information.
Inadequate Motivation Low morale or low Not Applicable There is no evidence to suggest that inadequate motivation contributed to the (J6) interest in performing occurrence of this event.
Type - SB/RB/KB well.
Shortcuts Taken (J7) Actions to allow the job Not Applicable There is no evidence to suggest that shortcuts were taken that led to the Type - RB to go easier or faster, occurrence of this event.
contrary to prescribed requirements.
Work Around (J8) Compensatory or non- Applicable The requirement to unisolate the suction pressure gage to take a reading, then Type - RB standard actions to meet a re-isolate the gage, is an operator work around. The need for this step is to requirement are taken by prevent over ranging the gage. If gages with the proper range and accuracy the worker due to were installed, this step would be unnecessary. The need for this step creates uncorrected material an error-likely situation.
condition, programmatic deficiencies, or long-standing problems.
Page 45 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Over Confident (K1) Underestimating the Not Applicable There is no evidence to suggest that the personnel performing this work were Type - KB/RB/SB difficulty or complexity over-confident.
of the task. Self-satisfaction or confidence with a situation in which actual hazards or dangers exist, but the worker is not aware of them.
Unfamiliar or Tasks that have not been Not Applicable There is no evidence to suggest that unfamiliar or infrequently performed tasks Infrequent Task (K2) performed before or are played a role in causing this event.
Type - KB performed infrequently.
Misdiagnosis (K3) Decisions made with Not Applicable There is no evidence that misdiagnosis contributed to the occurrence of this Type - KB accurate information that event.
is used or interpreted incorrectly when reaching a decision.
Tunnel Vision (K4) Decisions are made Not Applicable There is no evidence to suggest that tunnel vision played a role in causing this Type - KB without considering all event.
the available options or information needed to adequately assess the situation.
Inadequate Knowledge Insufficient knowledge of Not Applicable There is no evidence to suggest that the personnel performing this work had an of Fundamentals (K5) fundamentals needed for inadequate knowledge of fundamentals.
Type - KB task, such as heat transfer, fluid flow, structural analysis, etc.
Page 46 of 49
RCE 01146005 Attachment 9 Human Performance Failure Modes A = Attentional Issues J = Judgment Issues K = Knowledge Issues Failure Mode Definition Applicability Supporting / Refuting Evidence Inadequate Knowledge Insufficient knowledge of Not Applicable There is no evidence to suggest that an inadequate knowledge of standards of Standards (K6) codes, standards, design contributed to the occurrence of this event.
Type - KB basis, licensing basis, regulations, etc. needed to perform the task.
Flawed Analytical Decisions based on a Not Applicable There is no evidence to suggest that a flawed analytical model played a role in Process or Model (K7) flawed analysis, such as causing this event.
Type - KB/RB using qualitative versus quantitative data, insufficient determination of problem/solution scope, improper computer modeling, or inadequate sample scope.
Page 47 of 49
RCE 01146005 Attachment 10 Drum Report Analysis Drum reports for the last six quarters were reviewed for information pertaining to this root cause. The following information was deemed relevant:
o Components out of position - an indicator of processes that control configuration management. A large number indicates trouble maintaining the configuration of the plant.
o Operator burdens - consists of operator work-arounds, operations concerns, control room deficiencies, temporary modifications, and, for the last two quarters, long term clearances. This is an indication or the rate at which issues which prevent normal operation of the plant are identified compared to the rate at which these issues are permanently resolved. The use of these processes indicates the short term corrective nature of the site. A low value would indicate that the site resolves issues quickly or that they fail to identify issues. A high value indicates a high level of issues is being identified due to material condition of the plant and/or, the long term corrections are not resolving the issue long term.
o Unplanned LCOs - this indicator is representative of equipment issues or the ability to operate the plant as planned.
o Clock Resets - This indicator demonstrates the relative success of the human performance program. Actions from clock resets are typically short term and are not required to address actions to prevent recurrence.
Drum # Components Out Operator Burdens / 10 Unplanned Clock of Position LCOs Resets 01-2007 6 3.1 5 5 02-2007 6 8.6 7 5 03-2007 1 9.8 1 1 04-2007 1 10.1 9 6 01-2008 6 9 10 17 02-2008 2 8.3 6 3 Page 48 of 49
RCE 01146005 Attachment 10 Operations Department KPI Analysis 18 Components out of position 16 Unplanned LCOs Clock Resets 14 Operator Burdens / 10
- of Occurrences 12 10 8
6 4
2 0
01-2007 02-2007 03-2007 04-2007 01-2008 02-2008 Drum Quarter Results of Analysis:
From the graph of the data, three of the KPIs are leading indicators (components out of position, unplanned LCOs, clock resets). The Operator burdens are a cumulative indicator. There was a dramatic rise in operator burdens, but this was an effort to lower the threshold for defining operator burdens after receiving feedback from the INPO assessment in 2007. This action brought the site in line with the rest of the industry for sensitivity to operator burdens.
A continued high level of operator burdens is indicative issues are not being resolved with adequate priority. In this issue, SP 1102 requires the un-isolation and isolation of the suction pressure gage. This is not how the plant was designed, and requires additional operator action to operate the plant. Per 5AWI 3.10.8, section 4.9, this is an unrecognized operator burden. The operation of a valve in the vicinity of other, identical valves with safety related significance has an unintended consequence of developing an error likely situation. This was not evaluated in any documentation regarding the changes requiring the gage to be isolated.
Page 49 of 49