ML20117K951
| ML20117K951 | |
| Person / Time | |
|---|---|
| Site: | Sequoyah  |
| Issue date: | 02/29/1996 |
| From: | TENNESSEE VALLEY AUTHORITY |
| To: | |
| Shared Package | |
| ML20117K930 | List: |
| References | |
| IRA-SQN-96-003, IRA-SQN-96-3, NUDOCS 9609120260 | |
| Download: ML20117K951 (12) | |
Text
._; _ _ _ _ . _.
..e
. /
ASSESSMENT IRA-SQN-96-003 SEQUOYAH RELIABILITY COMMON CAUSE ASSESSMENT February 1996 Performed by:
Sequoyah Independent Review and AnalysisDepartment 9
0 0
1 9609120260 960905 PDR ADOCK 05000327 P PDR n
q e' . *
/
l A. Maintenance and Modifications-I
- Supervisory methods do not provide necessary coaching or accountability.
j Craft knowledge and skill levels are not suf5cient for some tasks.
The craftsmen do not fully understand or apply self-checking and questioning attitude techniques (STAR, QV&V).
B. Operations ;
i The knowledge and expedence of on-shift operations crews relative to normal plant operation has declined in recent years.
The assessment also identified a relatively high level of knowledge based errors. As shown in figure 4, the SQN knowledge based error rate is approximately twice that of a typical good performing plant. Research by FPIIntemational has shown that worker knowledge level is a leading indicator of performance. Thus, the high level of knowledge based errors found in this ' assessment is particularly significant in that it is predicting future performance problems.
The detailed analysis and basis for the conclusions can be found in Section IV of this report.
III. Recommended Corrective Actions Recommended corrective actions are discussed in the Reliability Study Report.
IV. Investigation Details /Information Each of the PERs was analyzed to enable each inappropriate event to be categorized into a causing organization, an Organizational and Programmatic (O&P) failure mode, a Human
' Error /Inappropdate Actions (HEIA) failure mode, a work process, a key activity within the work process and human error type. Section A documents the results of the data analysis for the plant as a whole. A multidimensional analysis was then performed on the data. Sections B and C provide the result of the multidimensional analysis for the Maintenance and Operations departments. The analysis of the data was used as the basis for the interview questions. Interviews were used to validate and qualify the data before developing conclusions and making specific root cause determinations.
A. Sequoyah Site Common Cause Results .
Figures 1 through 6 show the distribution of the data for the entire site. The significant causing organizations were Maintenance and Modifications with 43 j i
Ii ercent of the PERs and Operations with 26 percent. Figure I shows the distribution Page 2
. '. o-l l
by causing organization.
The O&P failure modes for the site are shown graphically in figure 2. Forty-six percent of the PERs were related t'o failures that occurred within a single l organization. DeSciencies associated with a single program contributed to 32 percent l of the PERs. Thirteen percent of the PERs were associated with an organization's i interface with a single program.
HEIA failure modes are shown in Sgure 3. The most common HEIA failure mode was Misjudgment with 43 percent of the occurrences. Inadequate Skills or Knowledge followed with 27 percent and Inattention to Detail with 15 percent of the occurrences.
The distribution of human error type (skill, rule, or knowledge based) determined in this assessment is shown in figure 4, along with typical values for mature, good performing I
plants. The percentage of knowledge based errors is more than twice the typical values, and skill based errors are approximately half the typical value. Knowledge based errors are typically associated with inadequate training, not knowing supervisor's expectations, Hrst-time or infrequent task pedormance, or tasks that require memorizing or comprehension of complexing situations.
Research by FPIInternational has shown that worker knowledge level is a leading indicator ofperformance. Thus, the high level of knowledge based errors found in this assessment is particularly significant in that it is predicting future performance problems.
Conduct ofMaintenance was the most predominant work process at 31 percent followed by Modi 5 cations with 17 percent. Plant Operations and Clearances each accounted for nine percent of the work processes. The work process failures are shown graphically in ngure 5. Corrective Maintenance was the most predominant key activity at 11 percent
- followed by Conduct of Testing at 7 percent and Plant Monitoring at 6 percent. The distribution of key activities is shown in Figure 6.
Based on the above data the team concluded that the Maintenance and Operations departments were incurring signi5 cant human error (misjudgments) as a result of' program desciencies and inadequate communications within the departments. The team decided to concentrate on the Maintenance and Operations departments for further analysis. Section B below provides the details of the analysis of the Maintenance department and section C provides the details of the analysis of the Operations department.
B. Maintenance and Modifications.
The PERs caused by Maintenance and Modifications were analyzed to determine .,
common elements within the Maintenance Department. These PERs consisted of 39 PERs issued in 1995 and 5 PERs issued in 1994. To put this number in perspective, 11,501 WRs were closed during the calendar year 1995. This amounts to a rate of Page 3
1
,e..
I one significant PER per 295 WRs closed or 0.34 percent. Instmmentation Maintenance (MIG) was the causing organization for 31 percent of these PERs followed by 22 percent for Mechanical Maintenance (MMG). Twenty-two percent also were assigned to the Maintenance Manager (MAS) because the speci5c responsible sub-organization was unknown. Modifications accounted for 12 percent and 8 percent for Electrical Maintenance (MEG). Figure 8 shows the causing organizations in Maintenance and Modifications.
The Maintenance O&P failure mode results are shown on figure 9. The most predominant failure mode was Inadequate Job Skills, Work Practice, or Decision Making which accounted for 27 percent of the maintenance events. This was' '
followed by Inadequate Scope (16 percent) and Insufficient Detail (11 percent).
Inadequate Job Skills, Work Practice, or Decision Making is a breakdown within the Maintenance depanment and indicates excessive human errors. Typical root causes for this type of failure include inadequate supervision, inadequate training or staff qualification, inadequate vertical communication, and conflicting or unreasonable organizational goals.
~
Inadequate Scope is a programmatic failure that results in the omission of necessary functions in procedures. The typical root cause for this failure mode is inadequate program design or inadequate feedback from the field work force.
Insufficient Detail is also a programmatic failure that results in vague procedures. The typical root cause for this failure mode also is inadequate program design or inadequate feedback from the 6 eld work force.
The HEIA failure modes were analyzed and are shown on figure 10. The most predominant HEIA failure incurred was Misjudgment at 50 percent. That was followed by Inadequate Skills or Knowledge at 22 percent and Inattention to Detail at 13 percent. Rule based errors accounted for 70 percent of the inappropriate actions while. knowledge based and skill based errors each accounted for 15 percent. Figure 11 shows the distribution of the human error types.
Misjudgment is generally a rule based error. The predominant failures were wrong ,
assumption, mindset, and misinterpretation ofinformation. Typical root causes are inadequate training or inadequate man-machine interface. Mindset is decision making without seeking the facts and evidence objectively. Inadequate training. supervisory methods, or work practice are the typical root causes. Misinterpretation ofinformation results from information that is not used correctly in the decision-making process. Typical root causes are inadequate verbal or written communication or inadequate man-machine interface. .
I Inadequate Skills or Knowledge is typically a knowledge based error. Inadequate training Page 4
. 7
. . . 1
\
.. ..' l l i l
l l
and tunnel vision were the predominant failures. Inadequate Training results from a lack l of training for the needed skills or training that is not complete or detailed enough. The typical root cause for this failure is inadequate managerial methods. Tunnel Vision results from actions taken or decisions made without assessing the entire situation. The typical root cause for this failure is inadequate work practice (i.e., failure to question information, assumptions, etc.).
The most predominant work process was conduct of maintenance with 66 percent of the PERs, followed by modifications, surveillance testing, and planning & work control at six percent each. The most predominant key activity'was corrective maintenanc.e which accounted for 26 percent of the key activities. Figure 12 shows the work processes and Figure 13 shows the key activity distibution.
This indicates that Maintenance has the most problems with corrective maintenance. It l
also suggests that the scope and amount of detail found in the site procedures does not i match the skill and knowledge level of the Maintenance personnel which is resulting in human (misjudgment) errors. In reviewing the PERs it was noted that 42 percent of the corrective maintenance PERs occurred when equipment was being reassembled. Fifteen l
Maintenance and Modifications craftsmen and foremen were interviewed to confirm these
~ .
j .
conclusions.
l During the maintenance interviews, MMG and MIG personnel stated that they have ,
problems with the quality of work order instructions and some mis and M/AIs. They stated that 20 to 30 percent of the work orders had to.be revised. However, MPAC l data show that 3.2 percent of the WOs were retumed to Planning during the second half of1995 due to planning deficiencies. MEG and modifications personnel also stated they had problems with the mis and M/AIs. These instructions were said to be l too complicated and complex. M/AI-7.1, Cable Terminations and Repairing l Damaged Cables and M/AI-7.2, Iristallation ofHeat Sh:inkable Tubing were each cited as being too complex with large, complicated data sheets. Generic mis, such as the generic MI on relief valves, were cited as not having enough detail for working on .
I some of the less common reliefvalves.
It should be noted.that a common cause analysis of the planning department was conducted in November,1995. The result of that assessment was that the content of the work instructions did not match the skill and knowledge of the craftsman. Most of the maintenance personnelinterviewed for the present assessment stated that some j instructions did not meet their needs.
l Most craftsman and foreman did not know what QV&V is and/or could not explain the difference between QV&V and STAR. Many craftsman had a weak understanding of' STAR although all said that they use STAR all the time. MIG personnel were an exception with a good understanding of STAR and QV&V.
Page 5
X When given examples ofPER conditions attributable to maintenance most craftsman were familiar with the incidents and stated that the reason the incidents occurred was inexperience orimfhmiliarity. .
Most craftsmen and foremen stated that the first line supervisors are in the field observing thejobs for at least an hour.each day. While the amount of time supervisors spend in the field observing work is good, research data show that the effectiveness can be improved greatly with a minimal improvement in Held supervision time and quality. Figure 14 shows that errors can be reduced by a factor of 4 when the quality of supervision is improved and Figure 15 shows that errors in Maintenance can be reduced by a factor of 4 to 6 if 40 percent of the work is surveyed with the use of a checklist. Figures 16 and 17 show the attributes of good supervision and an example of a checklist for use by the supervisors in the Held.
The results of the above interviews supported and corroborated the results of the analysis of the PER data. Stream analysis was performed to detennine the root cause. It showed that human (misjudgment) errors are occurring in corrective maintenance activities (activities that are unique). The root causes are:
,- Root Causes
- 1. The craftsmen do not fully understand or apply self-checking and questioning attitude techniques (STAR, QV&V). .
- 2. Supervisory methods do not provide necessary coaching or accountability.
Contributine Cause A contributing cause was that the work orders, mis, M/AIs do not match the skill or expectation of the craft. Corrective action for this is being taken in response to the planning common cause assessment, as discussed above.
C. Operations The O&P common cause analysis results described below are shown graphically in Figure
- 18. The most predominant O&P failure mode was Inadequate Job Skill, Work Practice, or Decision Making. This failure mode accounted for 46 percent of the events for Operations. This failure mode was followed by Inadequate Communications Within the Organization (21 percent) and Insuf5cient Detail (11 percent). This distribution of O&P failure modes is indicative of problems with the worker knowledge and skill, with the ability to communicate within the organization, and with the procedures.
Inadequate Job Skill, Work Practice, or Decision Making is an organizational breakdown that occurs within a single organization. Typical root causes for this failure mode include inadequate supervision, inadequate training or staff qualification, inadequate vertical Page 6
~
o f
communication, and conflicting or unreasonable organization goals.
Inadequate Communications Within the Organization is also an organizational breakdown within a single organization. Typical root causes for this failure are inadequate information path (star-type communication pattem), lack of a teamwork culture, and inadequate physical setting.
Insufficient Detail is a programmatic desciency that results from vague procedures. The typical root cause for this failure mode is inadequate program design or inadequate feedback from the field workforce.
The O&P failures which are attributable to humalip'erformance were analyzed and it was determined that a majority of the Human Error / Inappropriate Actions (HE/IA) for Operations were in the areas ofInadequate Skills or Knowledge (40 percent),
Misjudgment (28 percent), Inattention to Detail (20 percent),and Committed Actions Not
- Canied Out (12 percent) (see Figure 19). It was also determined (Figure 20) that the majority of the Human perforInance problems were knowledge based (40 percent) and rule based (32 percent) errors. The types ofbreakdowns found in this area is indicative of a problem in worker, knowledge, training, or procedure clarity /speci5 city.
Inadequate Skills or Knowledge generally reflect knowledge based errors. The predominarit failure modes were Inadequate Training and Tunnel Vision. Inadequate Training results from a lack of training for the needed skills or training that is not complete or detailed enough. The typical root cause for this failure is inadequate managerial methods. Tunnel Vision results from actions taken or decisions made without assessing the entire situation. The typical root cause for this failure is inadequate work practice (i.e., failure to use QV&V).
Msjudgment and Committed Actions Not Carried Out normally are rule based errors.
The predominant failure mode under Msjudgment was misinterpretation ofinformation.
This indicated that information was not correctly used in the decision-making process.
The typical root causes are inadequate verbal or written communication or inadeqtiate man-machine interface. Committed Actions Not Carried Out include shortcuts evoked, task too complex, inappropriate order, on-the-job distractions, and inadequate tracking.
Typical root causes include inadequate work practices, inadequate work planning, inadequate supervisory methods, inadequate tracking program, and inadequate environmental conditions.
Inattention to Detail is typically a skill based error. The predominant failure mode was i f
unawareness. These errors result from not paying attention to alarms, signals, precautions, or information that are not contained in procedures or guidelines. Typical root causes for this failure mode are inadequate work schedule, inadequate work practice, 1 or inadequate communication.
Page 7 J
T.
l The Work Processes involved were also analyzed and it was found that conduct of l
operations (32 percent) and clearances (32 percent) were the most significant processes involved (see figure 21). The clearance process was not analyzed any further as it was the subject of a common cause assessment performed in November 1995. Conduct of ,
operations was analyzed further by reviewing the key activities for Operations. The j specific key activities identified include: Manipulation ofEquipment with 14 percent, and l Conduct of Tests, Procedure Preparauon and Revision, Plant Monitoring, and Clearance Preparation each with 11 percent (see figure 22). This data suggests a broad distribution of problems with Operations.
The inappropriate a,ctipns for the PERs categorized by the O&P Failure mode were i reviewed for the three top areas in further detail. Analysis of the data indicated possible problems in the area of training and procedures.
Inte views were conducted with an operations shift crew, plus three personnel from Operations Procedures Group, to validate the determination of possible problems in the areas of training and procedures. The principle focus of the Operations interviews was on procedures and training since these areas should indicate weaknesses related to inadequate skills or knowledge and misjudgments, which were the primary HEIA failure modes with 68% of the human errors. Several comments were made which support the failure modes which were observed:
Training
- " Don't spend enough time on system training."
- " Jeed more real OJT rather than pretty OJT."
- "l oo much naining on material that is of no benefit. Only train on objectives, other good material is missed."
Procedures
- "Some procedures are good some are not, validation (in plant)is not very good."
- " Procedures for complex evolutions are good, but problems exist in procedures for easier evolutions."
- " Operations Procedures group accessibility is not very good. You have to stay with it too make sure a change is made."
- " Intent changes take too long. Non-intent are done on-shift"
- "Should be allowed to use brain, should not have to sign off on every step."
- " Procedures requiring non-intent changes are not necessarily fixed prior to continuing with the evolution."
- "Need the ability to change procedures on-shift. The easier the process the more likely people are to use it."
The results of the interviews with the Operations crew were discussed with the Training department. The Training department concurred with the operators concerning the lack of systems training during the previous year and a half, and observed that operator training Page 8
has been largely reactive since 1993. Special training requirements (revised EOPs, Fairfield Glade, responses to plant and industry events) have reduced the amount of non-commitment training. A requalification training cycle was also suspended for one cycle in 1994 due to the need to support an outage. The Training department also recognized through operator feedback that the OJT training program needed more involvement and input from shift personnel and therefore implemented a revised methodology for conducting OJT. The revised method requires an SRO and RO to be with the AUO during the OJT evolution, which should improve the transfer of detailed operational knowledge and experience.
The interviews conducted with the operators also revealed that the Operations
~-
procedure change process does not promote ownership of the Operations procedures by I I
the operators on shift due to the perceived lack of timeliness of procedure revisions from submitted change requests. The operators expressed the need to either perform the ;
procedure changes themselves or to have ready accessibility to those who can make the changes.
The team also noted that has been a large influx of new on-shift supervisors (SOS's and ASOS's) that have never worked as AUO's or UO's at this station. Consequently, they are ;
being asked to operate systems and components that they have never had the opportunity to operate previously, or they have never had good OJT on these systems. Only 7 of19 )
new operators have completed the SQN training curriculum. The Sequoyah BOP is a J I
complex design and consequently requires an extensive skills, knowledge, and experience base to operate successfully.
1 Root Causes With respect to the operator knowledge level, the knowledge and experience of on-shift
- operations crews relative to normal plant operation has declined in recent years. This has been caused by two major factors
- An influx of new on-shift supervisors (SOS's and ASOS's) that have never worked as AUO's or UO's at this station.
- Operator training has been largely reactive since 1993. Introduction of special training requirements (revised EOPs, Fairfield Glades, responses to plant and industry events) and cancellation of training to support higher priority work (refueling outage) has reduced the amount of non-commitment training. As a result, the focus of OJT in operations has shifted from genuine transfer of knowledge to an exercisein documentation. The Auxiliary Building AUO routine qualification currently requires about 148 qual cards to complete. The Turbine Building routine requires about 140. The focus of requalification training has been regulatory compliance rather than good systems training. It should.be noted that Operations OJT has recently been changed to improve the knowledge and Page 9
experience with plant equipment by requiring a licensed RO and SRO to work jointly with each AUO during OJT.
The pdmary causes for the high levels of misjudgements and inattention to detail failures as determined in these and two other recent assessments (IRA-SQN-95-19, " Operations l Department Common Cause Assessment," and IRA-SQN-96-04, "Sequoyah Procedure Compliance Common Cause Assessment") are: l l
- Accountability for procedure compliance is insufficient.
- Supervisory weaknesses in the areas of field surveillance, providing needed information, and pre-job briefings.
- Human error reduction technique especially STAR and QV&V, have not been sufEciently reinforced by managers and supervisors.
With respect to procedural guidance, improvements in procedures that would be expected to occur over time have not been happening. This has resulted from:
- The overall perspective by operations personnel that intent changes to procedures take too long.
- The Operations Department procedure change process that does not allow intent procedure changes to be made on shift.
- A lack of feedback between the operations procedures group and on-shift operations personnel with respect to requested procedure changes.
Written A Date: A-/A 7/95 Reviewed: .
Date: 7b
\
t CC!nIDCD08t15c3SSCSSrnent Page 10
?
i . :
- ^
O e #
e ya 6 yg #
ATTACHMENT 3 s
(,- .
U ~
Production Schedule Performance -.
12 Week Schedule .:
Breakdown of WRsAVOs Removed
- During Schedule Development Process 50
.- 40 -
g 30 -
S
-s k
20 .
- ~
l tl 10 g -$ ,
' I I I I I I I I I I s M d O
W9 W10' W11 .W2 W3 W4 W5 W6 W7 WD W9 W10 W11 ' W12 W1 X-Axis W8 1G7 42 60 84 48 48 73 52 49 Total 9
~0 5- 11 0 0 3- 2 0 Engineerino 21 9 5 4 17 5 7 0 . 2 4 A Plannlop 15- 25 10 19 .13 18 15 1 10 i El Materials 45 2 0 12 3 3 0 1 O Manpower 1 10 21. -16 5 to 1 3 A Complete / Cancelled '19 '8' 0 -0 0 4 0 2 o 0 -1 DCN 0 8 0 0 2 0 0 1 0
-Funding
- 16 9- 0 4 3 1 0 8 1 Schedule Review ~
3 13 2 3 1 12 7 4 '
Outg/Dwnpwr Reg'd '6 8 -0 5 6 0 0 3 0 0 Minor Maintenanco 3 0 3- 0 0 0 4 0 0 Otheq Org Support Other' S 3 2 0 11 22 7 ~s
-l
, . .'o' ,
[Lastbat4 taken August 11,1995l
.:jf i
4
. _. a