ML20078C613
| ML20078C613 | |
| Person / Time | |
|---|---|
| Site: | Quad Cities  |
| Issue date: | 10/21/1994 |
| From: | Brunner J, Kroll T, Mccullough D COMMONWEALTH EDISON CO. |
| To: | |
| Shared Package | |
| ML20078C612 | List: |
| References | |
| PIF-94-1837, NUDOCS 9411010136 | |
| Download: ML20078C613 (36) | |
Text
.
Enclosure A: The Final Level 2 Investigation Report Addressing The Adequacy Of The Quad Cities FME Program.
FME Prograin Deficiencies Level 2 Investigation PIF 94-1837 ,
InveMigation Team Tom Kroll 'nvestigation Director Dale McCullough - Team leader Jack Brunner - Root Cause Specialist he Eenigenburg - Corporate Sponsor Dan Boehner - Quality Control Wayne Krause - HPES Coordinator Greg Donoho - Training LaVerne (Jim) Binzen - Mechanical Maintenance Jeff Odeen - Bechtel l l
Ken Parkinson - Mechanical Maintenance Scott Flaker - QV Overview Deb Ketcham - Clerical 9411010136 441021 PDR ADOCK 05000254 i p PDR l
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Executive Summary Problem Description On July 10, 1994, pieces of a plastic bag were found in Residual Heat Removal (RHR) valve 36A. Then on July 23,1994, a wire brush wheel and metal object were found in RHR Pump IC. The two events were discovered by routine surveillance testing. These two events are being investigated by separate investigations. After the occurrence of the second event, a Level 2 investigation of the Foreign Material Exclusion (FME) program was initiated.
An FME improvement program is being developed by the a multi-discipline team. The program will identify and correct FME weaknesses.
Two indications of Foreign Material introduction are fuel leaks and reactor water chemistry.
These parameters have improved over time. No fuel leaks have occurred in Unit I since cycle 8(1984) and Unit 2 has not had any fuel leaks since the cycle 4 (1978). For the first time Unit I reactor water chemistry averaged less than 0.1 micrombo for cycle 13 and Unit 2 is averaging less than 0.1 micromho for this cycle.
Evaluation / Concerns investigation results indicate that although the station responded to identified weaknesses in FME practices, corrective actions were not effectively implemented. The team has identified apparent weaknesses in defining, communicating, and re-enforcing expectations for FME practices, as well as monitoring the performance of work to determine if the expectations were understood.
These weaknesses lead to ineffective implementation of the FME program. ,
The inves'igation team drew the following conclusions:
Conclusion #1 The response to the SOER 90-02 " Nuclear Fuel Defects" was too narrow in scope, it did not address FME in other systems which discharge into the reactor.
Conclusion #2 The FME procedure, QCMP 307-04, Rev 0, was issued without being validated.
Conclusion #3 Unclear Expectation of what the FME program was to accomplish. FME barrier use was craftsman dependent and principally based on practices established prior to the issuance of QCGM 307-04.
Conclusion #4 Adequate FME material was not stocked in advance of FME program implementation.
[ i 1
l FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Conclusion #5 Station Line Management and SQV were inadequate in tracking corrective -
action effectiveness. Independent assessment did provide feedback on the program that could have prevented this but was not adequately addressed.
Conclusion #6 Training provided was inconsistently applied to the work force and did not match procedure owners expectations.
9.ecommendations The recommendations and conclusions identified in this investigation have been submitted to the FME Program Improvement Team.
The results of the investigation will be communicated to all Comed stations. The preliminary .
results of this investigation were presented to representatives for Corporate and each of the other stations in a meeting held at Quad Cities on 8/10/94.
I l
FAE PROGRAM DEFICIENCIES !
Level 2 Investigation j PIF 94-1837 1
I Introd u c t ion . . . . . .. . . . . . . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . , , , , , , 1 Problem Description ...... .........................,....... 1 )
l Narrative of FME Program History . . . . . . . . . . ..................... 2 Root Causes .......... ................. ................. 5 Evaluation of Previous Events .... .........
.................... .6 Evaluation of Issues Conclusions / Recommendations . . . . . . .............. 8 Issue #1 Response to SOER 90-02. . . . . . . . ................... ... 8 i Issue #2 ;
FME Procedure QCGM 307-04, Rev 0 Validation . . . ..... . . . 10 l i
Issue #3 i
Expectations .... .... . . ...... ....... ... . 13 i Issue h I Program Resources . . . . . . .... . . .... ...... ... . 15 )
Issue #5 Feedback / Monitoring . . .. . .. . . ... ... ....... . 18 I Issue #6 ' j Training .. .. .... ..... ............. ... . .. 19 1 1
Other issues .. . . ... . .... ....... ........ .......... 21 Attachment A . . . . . . . . ......... ......... . . ........ 22 ,
Charter Attachment B ........ .......... .......... .......... 24 Recommendations to " Foreign Material Exclusion", QCGM 307-G4, Rev 1, ;
Attachment "C" i Attachment C . ..... ......... ....... ...... . ...... ... 25 Investigation Team Members l l
Attachment D . . . . . . . . . . . . ..... ....... .... ..... ........ 26 ,
Human Error Inappropriate Actions i References ..... .....,......... ........................28
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Introduction The purpose of this investigation is to evaluate the Foreign Material Exclusion (FME) program to determine the root cause of the deficiencies in the FME program and its implementation.
Since this investigation is of a program versus an event the title of the report section headings will differ from those that appear in QCAP 2300-24 level 1,2,3 Root Cause Investigations.
Problem Description On July 10,1994, pieces of a plastic bag were found in RHR Valve 1-1001-36A (PIF 94-1745).
Then on July 23,1994, a wire brush wheel and metal object were found in RHR Pump 1-1002-C (PIF 94-1824). These two events are being investigated by separate level 3 investigations.
After the occurrence of the second event, station management determined that a level 2 investigation of the FME program was warranted. A Confirmatory Action I2tter was issued by the NRC indicating they have a significant concern due to these events which requires immediate attention.
The station has instituted a three phase immediate corrective action program, similar to that issued for rad worker practices. The purpose of the program is to determine the problems with FME practices through tight controls and observations, make correction to work practices and program controls, implement the new program and make observations to ensure that the new program is effective.
Evaluations are being performed to ensure Unit 1 safety systems and systems important to safety are free of foreign material prior to start up and that sirnilar Unit 2 systems are not degraded by foreign material.
In accordance with the investigation charter (Attachment A) the following areas will be evaluated in this report. l l
- 1. An evaluation of previous events (including the most recent occurrences) associated l with FME at Quad Cities.
- 2. Evaluate the Quad Cities policy (procedures and practices) for foreign material exclusion. I 1
- 3. Evaluate the training given to the work force regarding foreign material exclusion.
l
- 4. Provide recommendations to the FME program improvement team.
I 1
ADM\LEVELhrMEINV.WPF 1
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Narrative of FME Program History To evaluate the FME program a time line was constructed which includes important events related to development of the program. A time line and event and causal factor chart were developed to analyze program development. The chart is not part of the report but was part of the investigation of work materiais.
The following description provides the highlights of the time line regarding FME program development. Other significant events are included to give a sense of what was going on at the station during that time. Issues are not listed in order of significance but are numbered sequentially as they appear in the narrative.
In 1983, it was determined a procedure for cobalt reduction was needed based on EPRI Document NP-3220, " Cobalt Reduction Through Valve Maintenance" guidelines. After going through several iterations, the intent of the procedure changed to one which addressed cleanliness control for mechanical maintenance. In 1988, a procedure (QMMP 1500-34) was issued to establish cleanliness control for mechanical maintenance.
In 1990 Significant Operating Experience Report (SOER) 90-2 (Nuclear Fuel Defects) was issued. The SOER identified problems that had been encountered at many stations with fuel defects due to debris. The stations response to the SOER was to establish controls for foreign material exclusion on the refuel Door. This was done via procedure QCAP 270-1, " Conducting Activities in the Refuel Floor Foreign Materials Exclusion Area". ' Conclusion #1 The response to the SOER 90-02 " Nuclear Fuel Defects" was too narrow in scope, it did not address FME in other systems which discharge into the reactor).
In April 1993, a Corrective Action Record (CAR) 04-93-005, Severity Level II was written on the number of FME issues identified at the station. A response was made within a week including interim actions being taken and corrective actions to prevent recurrence. Interim actions were a memo to all maintenance personnel concerning the importance of FME (the memo was written but could not be located by the team) and a memo from the QC Supervisor to all QC personnel on April 30,1994, to note whether FME was satisfactory or unsatisfactory on all inspection reports. One of the long term actions was to implement an FME procedure.
On June 9,1993, the IIPCI Rupture Disk event occurred, focusing the stations efforts on addressing the issues associated with the event and presenting a significant managerial distraction.
Work on the FME procedure was started in June 1993. The procedure was being developed based on INPO and NOD guidelines as well as procedures from other stations.
July / August 1993, Bechtel replaced Stone & Webster as maintenance contractor.
In August 1993, an SQV follow-up on the FME CAR indicated that the training which was being developed for maintenance should be expanded to all site personnel.
ADM\LEVF1.2\r%IEINv.WPF 2
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 In August 1993, the Business Development Team (BDT), Diagnostic Evaluation Team (DET) and INPO were on-site requiring site resources to support these activities. Plant management changes were also started with a new Plant Manager being appointed. Other significant management changes occurred through the last quarter of 1993.
In September / October 1993, the FME procedure was distributed to the maintenance department heads for review. No feedback was received. During this time the site was responding to DET issues and preparing for the upcoming fall maintenance outages. The maintenance outages were being done to correct the stations' most significant materiel condition problems. (Conclusion
- 2 The FME procedure, QCMP 307-04, Rev 0, was issued without being validated).
In mid October, training briefs the maintenance shops that a new foreign material procedure is being issued.
On October 18,1993, the Foreign Material Exclusion Procedure QCGM 307-4 Rev 0 is issued.
(Conclusion #3 Unclear Expectation of what the FME program was to accomplish and Conclusion #4 Adequate FME material was not stocked in advance of FME program implementation).
In November 1993. a new Maintenance Superintendent is appointed.
In December 1993/ January 1994, extensive changes are made to the maintenance organization.
New organization was in effect February 4,1994.
In January,1994, the original sponsor of the FME procedure is reassigned from Maintenance Staff to the Valve Group. (Conclusion #5 Station Line Management and SQV were inadequate in tracking corrective action effectiveness, independent assessment did provide feedback on the program that could have prevented this but was not adequately addressed).
On January 12,1994, formal training begins on FME procedure. The training is given on the procedure and results in significant feedback from the work force. (Conclusion #6 Training provided was inconsistently applied to work force and did not match procedure owners expectations).
ADM\LEVE12JNEINv.WPF 3
9 FME PROGRAM DEFICIENCIES Level 2 Investigation FIF 94-1837 On January 27, 1994, Interim Procedure #466 on FME is issued. The interim procedure removed some excessive requirements (e.g. the procedure placed unrealistic requirements on the material for FME barriers without identifying anything that met them). Some requirements were relaxed, others strengthened.
On January 29, 1994, SQV meets with maintenance to discuss a concern that the FME CAR actions are ineffective based on SQV observations in the Geld. The following commitments were made by Maintenance in response to the CAR escalation:
e Ixtter dated March 3,1994, to department heads and training coordinators concerning importance of FME.
e Training of contractor pipefitters, Crane valve technicians and millwrights upon arrival at station.
e Mechanical and construction work planners instructed to add FME checklist to all work packages and add this requirement to work planner guide.
e Increased supervision by supervisor.
e FME barriers ordered or in stock, e FME device display in maintenance shop.
e Daily Link article to all people to increase FME expectations.
e Revise the FME procedure.
In March 1994, QlR13 starts.
In April 1994, the outage scope is reduced.
A Reactor Recirc pump ran up to full speed caused by a failure in the control loop circuitry resulting in a crud burst. This caused the in-plant dose rates to be much higher.
On April 28, 1994, Interim Procedure 665 replaces Interim Procedure 466. The procedure content was unchanged and it was issued because Interim Procedure 466 expired.
On June 15, 1994, Rad Worker Phase Program begins as a result of the high number of poor Rad Protection work practices.
On July 10, 1994, flow testing indicates low Dow through RIIR Valve 1-1001-36A. An l inspection reveals a plastic bag in the cage of the valve. ;
l i
On July 19, 1994, during surveillance test, low Dow noted in RHR Pump 1-1002-C.
ADMiLEVELT.FMEINV.WPF 4 l
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 ,
1 On July 23,1994, during a borescope inspection of RHR Pump 1-1002-C, found a flat piece of metal (approximately 4 x 1"). During a subseqcnt inspection, a wire wheel with its associated parts are found in RHR Pump 1-1002-C.
On July 25,1994, Maintenance Superintendent issued a memo to all maintenance and SEC work departments implementing immediate corrective actions to address FME problems subjecting all breaches greater than 3/4" to the most stringent requirements. The investigation of RHR Pump 1-1002-C is upgraded to a level 2 investigation.
On July 26, 1994, the decision is made to have 3 investigations; one each for RHR Pump 1-1002C, 36A Valve and one for the FME program. Confirmatory Action Letter is issued regarding FME to Quad Cities Station.
On July 27, 1994, the team for FME level 2 investigation is formed.
On July 28,1994, revision 1 to the FME procedure is issued.
On July 29,1994, Phase 1 requiring all boundary breaches to be subjected to the most stringent controls pending further guidance of the FME program was implemented. ,
Root Causes To determine the roo. cause of the problem the site is encountering with FME, the investigation team asked the following questions:
Were management expectations regarding FME clearly defined?
Were management expectations regarding FME clearly communicated?
Were worker practices monitored to determine if actions taken to improve FME practices were effective?
These questions were used because they are essential elements of effective program implementation. )
l The investigation concluded that although the station responded to identified weaknesses in the l FME practice, there was ineffective follow through on the corrective actions taken. The conclusion reached by the team identified weaknesses in all three areas which lead to ineffective implementation of the FME program.
ADMTLINEL2\FMEINv.WPF 5 :
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 To evaluate the conclusion for potential causal factors the investigation team using the root cause analysis checksheets determined the primary root cause to be Managerial Methods as described .
in QCAP 2300-24, Attachment C " Event and Causai Factors Causal Factor Worksheet". In I terms of causal factors as defined in QCAP 2300-24 these weaknesses are categorized as follows:
Managerial Methods Policy not adequately defined, communicated, monitored, or enforced.
- Contributing to these factors were:
Managerial Methods Known problems with FME were not adequately addressed, appropriate inputs were not included in the development of the program, the FME program was approved without adequate review, the FME program was not effectively monitored or assessed and interdepartmental communications were inadequate.
The inadequate program implementation resulted in the following:
FME training inadequacies.
FME procedure content inadequacies.
Not all necessary FME barrier materials readily available.
Evaluation of Previous Events The investigation team performed a review of level 2, 3 and 4 Problem Identification Forms (PIFs) initiated during the time period from January 1993, through July 1994, which involved ,
Foreign Material Exclusion. During this period, the search revealed 106 Foreign Material Exclusion causal factors related to the PIF's. A concentrated review from October,1993 ;
through July 1994, of PIF's totalled 66 of the 106. The 66 PIF's were divided into four categories which are: Human Inappropriate Actions (15), Materiel Condition (24), j Programmatic (Refuel Floor) (14), Housekeeping (13). It appears that the data base contains FME, housekeeping and materiel condition type items all under the heading of FME.
i Aouuvrunuwv.m 6
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 The 15 PIFs that are categorized as Inappropriate Actions are mainly debris or material created by the work activity. Ten cases of debris, slag, metal filings or sand found in pipes, valves and relays. Two were cases where something was dropped into equipment during the work activity.
The remaining three written for problems created because FME barrier material was left in the equipment.
The team's conclusion from this data is that FME is a problem that applies to all work groups on site. (Attachment D will list the 15 PIF's categorized as inappropriate actions). ,
it was also noted there was a high incidence of programmatic problems related to compliance with Refuel Floor FME requirements.
In August and September of 1993, the Integrated Reporting Program reporting process was changed to use the new Problem Identification Form (PIF). The station initiative was to improve reporting using one formal process. This resulted in a large number of PIF's being generated at inception of this program. Therefore, we can't determine if an adverse trend exists. The overall trending capability of the PIF process is still being developed.
ADWHVEU\FMEINV.WPF 7
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Evaluation of Issues Conclusions / Reconunendations Issue #.1 Response to SOER 90-02.
Evaluation SOER 90-02 " Nuclear Fuel Defects" had several issues requiring evaluation. One of which was
" Include guidance for maintaining cleanliness of those systems or areas that could introduce foreign material into the reactor core or fuel pool during maintenance or operation" This issue was evaluated by the system engineer responsible for the refuel Hoor. Consequently, the response was considered complete when a procedure was developed (QCAP 270-1) to control FME on the refuel Hoor. In late 1992, an effectiveness review of this SOER led to the consideration of FME for the remainder of the plant. The review believed that existing work practices were adequate.
It should be noted that no fuel leaks have occurred in Unit I since the eighth cycle and Unit 2 has not had any fuel leaks since the fourth cycle.
A review by this investigation team found that implementation of the existing program for FME on the refuel Door is inadequate based on the number of program compliance problems reported in the PIP data base.
Changes have been made in the Regulatory Assurance organization requiring that all affected organizations evaluate SOERs. Discussions with Regulatory Assurance personnel indicate that recent SOER response from the line organizations at the station have been adequate. Currently when SOERs are received, teams (Corporate and Station Personnel) are formed to provide a thorough evaluation. This should ensure that a broad evaluation of SOERs is performed.
Conclusion The response to the SOER 90-02 " Nuclear Fuel Defects" was too narrow in scope, it did not address FME in other systems which discharge Mto the reactor. Further follow-up effectiveness reviews inadequately concluded FME practices were adequate. i Potential Causal Factor (s)
Work Organization / Planning.
Inputs from all affected organizations was not included in the stations response to SOER 90-02.
l ADMM.EVFMVNFlNV.WPF 8
FME PROGRAM DEFICENCIES Level 2 Investigation PIF 94-1837 Recommendation 1A. Reopen the SOER to reflect and include the corrective actions of this investigation.
IB. Ensure that the Refuel Floor issues identified in the PIFs are considered on this evaluation.
4 k
)
b b
k ADM'11NE12EMEINv.WPF 9
4 FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Issue #2 FME Procedure QCGM 30744, Rev 0 Validation Evaluation The procedure was developed by using INPO and NOD guidelines as well as procedures from other plants. Several iterations of review occurred over a period of three months. The procedure was issued for comment to the maintenance department head but no comments were received. A contributing cause to the absence of comments was the activities at the station during the review period (DET response, preparation for maintenance outages).
The procedure owner did not consider it to be a "how to" procedure, therefore it did not go through the validation process. This resulted in the procedure being issued without input from the workers.
A review by the investigation team identified the following deficiencies with the procedure:
There is no QC cleanliness inspection hold point for non-safety systems that feed the reactor.
The FME review sheet (307-04, Attachment C) has the following inadequacies:
Procedure is mainly based on pipe, not components, gear boxes or pumps.
Electrical and instrumentation components are not addressed.
Meaning of signatures is unclear (what attribute are you attesting to when you are signing).
The worker would have to review a separate procedure, QCAP 307-10, Attachment E, to note changes to FME requirements. This form is typically utilized at the completion of a job to record root cause, as found and as left conditions, etc.
The system name is not spelled out (noun name more recognizable than MO-1-1001-36A).
Does not identify the sensitivity of the equipment or if the equipment is in a system that feeds the reactor. (what equipment is downstream?)
Does not have a place to log the installation and removal of FME barriers.
Does not provide guidance for establishing FME boundaries.
ADM1LEVE1.2VVE!NV.WPF 10
~
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Current checklist does not have provision to document closure of multiple boundaries. .
No provision in the procedure for cobalt reduction.
Cleanliness criteria was added based on construction flushing requirements. These criteria are not intended for systems that have been in service. Attachment C, which contains the cleanliness classes, has the following statement for class "B" which also applies to class "C"; for carbon steel material the following type of rust is not acceptable:
IIeavy rusting (continuous film of appreciable thickness which forms due to lengthy exposure to aerated water or condensed moisture).
No mention is made concerning methods of breaching systems. No guidance is provided to consider methods of cutting which minimize particulate introduction into systems.
The INPO/ NOD guidelines and FME procedure from other Comed stations do not contain these cleanliness classes.
Conclusions The FME procedure QCGM 307-04, Rev 0, was issued without being validated.
Potential Causal Factor (s)
Written Cont qunications t
Instructional / Informational Presentation Deficiencies, QCGM 307-04, Attachment C, (FME review sheet), contained information that was not applicable to an operating plant (cleanliness classes). The FME procedure was developed without input from the workforce, 8
f ADM;1EVEL2'JMEINV WPF 11
i 1
FME PROGRAM DEFICIENCIES Level 2 Investigation i PIF 94-1837 Recommendations 2A. Revise the FME review sheet (see attachment B).
2B. Delete the cleanliness classes from the report.
2C. Simplify the actual procedure guidance for the workers to what they need.
The other information on FME should be contained elsewhere.
2D. Ensure all new or significantly revised programs are validated.
2 E. Incorporate cobalt reduction on the checklist and in the procedure.
2F. Evaluate methods of breaching to determine the most appropriate methods / tool available. Incorporate these into the procedure.
2G. Expand the procedure applicability to all departments. In addition, we may have department specific procedures.
211. FME improvement team review inadequacies identified with QCGM 307-04 for procedure enhancement.
AISnum.u'DEINV.WPF 12
l FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Issue #3 Expectations Evaluation There was a mindset developed from QMMP 1500-34 in 1988, that bagging and taping breaches was the approved method. This has been the mindset until Phase I was implemented despite QMMP 1500-34 being replaced by QCGM 307-04 in October 1993.
The procedure owner intended for the procedure to be a guidelire of all FME options available not a descriptive "How to" For instance "IIere are some methods and here is how you do them". Also, it was not intended that the workers would have to determine the cleanliness classifications. They would be given the classification and an explanation. It was intended that the work analyst would do the classifying. The expectation was that the work analysts would check the classification and what barrier would be used. The worker would then have the information in the work package.
The work analyst is directed by the procedure that a temporary cover is required on all openings when work is not taking place at or in the opening. This was reflected with a check mark in the "Yes" box on Attachment C, " Temporary Cover Required" The procedure directs the work analyst to evaluate the remainder of the foreign material controls and consider their use. This was reflected with a check mark in the "No" box on Attachment C as indicated by a review of numerous work requests.
The work analysts felt the Supervisor could initiate tighter controls as the job evolved. QCGM 307-04, Rev 0 did not direct the Supervisor to do so.
The worker received the work package with the guidance from Attachment C to install a temporary cover on the component when it was left unattended. The workers interviewed felt that tape and plastic were a suitable temporary cover and had received no feedback that in some instances there are other preferred methods of establishing FME control.
The decision whether or not to install barriers during the course of the work as opposed to temporary covers when the job is unattended was left to Supervision / worker judgement.
Conclusion Unclear Expectation of what the FME program was to accomplish.
FME barrier use was craftsman dependent and principally based on practices established prior to the issuance of QCGM 30744.
ADMil.EvEt2FMEINv WPF 13
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Potential Causal Factors Managerial Methods Methods did not ensure inclusion of all appropriate inputs in the FME program.
Written Communications Procedure content had technical inaccuracies and some incorrect assumptions were made regarding how it would be used.
Recommendation 3A. Ensure that phased FME program currently in progress clearly communicates managements expectations regarding FME.
3B. Program owner clearly needs to define program expectations to training department, senior work planners, supervisors and workers to ensure its implementation.
3C. Senior management should communicate expectations to contractors during j in processing. Consider use of video recording to utilize for future use. !
l 3D. Add guidance for Cobalt reduction (work on stellite seats) and appropriate I system and component breach techniques to QCMP 307-04, Rev 1.
l I
i ADMEE"EL2\rMEINv.WPF 14
FME PROGRAM DEFICIENCIF,S Level 2 Investigation PIF 94-1837 Issue #4 Program Rescurces Evaluation Approximately six months ago, the Mechanical Maintenance Department fabricated a display board consisting of the various materials available for foreign material exclusion. The materials were mounted to the board with the uoun name and stores item number adjacent to each item on the board. The worker could select the appropriate material and order it from the store room.
The computer balance of the various material indicates stocked levels are too low to support the daily work activities. Most levels show zero balances in stock with minimum / maximum levels set at two (minimum) and twelve (maximum). Activity showed "No Activity."
Valve group frequently used inflatable air bladders and garlock gasket material that they specially ordered for this outage.
Tape and plastic, the items used most frequently by the station, are well stocked by the store room.
The vacuum cleaners are the best method for cleaning a component. Rad Protection has ownership of the vacuum cleaners. To obtain a vacuum cleaner, the worker must go to the Rad Protection office and fill out a request form. A Rad Technician accompanies the requester to ,
the fourth floor of the Reactor Building and issues the vacuum cleaner. This time consummg -
process has limited their use.
The team evaluated availability of tools necessary to perform enhanced FME inspections. We found that we have sufficient amount of equipment such as borescopes, fiber optic cameras, videos and monitors, j An evaluation of the tools used for cutting pipe at the station was done to determine if there are tools available to the workforce to support cutting of pipe in a manner that would minimize the amount of foreign material introduction. The standard practice is to use "Sawzalls" and i "Portabands" which results in metal debris going into piping systems.
1 Anuuvruuumv.wrr 15
i
\
FME PROGRAM DEFICIENCIES !
Level 2 Investigation PIF 94-1837 The maintenance shop has several clamshell type cutting tools which machine cut the pipe from the periphery without introducing small pieces of metal that occur when sawing. Due to the low number of clamshell tools available, their use is prioritized and can't support all jobs in which theit ase would be appropriate. An alternative is " blue lining" in which the outside diameter of the pipe is ground until the wall is very thin as indicated by a blue line, at this point the pipe can be severed by fatiguing.
Also, due to the absence of small tooling, many weld preps have to be done by grinding which can introduce small particulate into piping systems.
Discussions with mechanical maintenance indicate there are plans to have one central tool crib for contractors and maintenance personnel. This would provide controls for the types of tools which are used on specific jobs and would provide a means to ensure that all of the tools issued are returned.
Conclusion Adequate FME material was not stocked in advance of FME program implementation. Not all necessary FME materials were readily available.
There is an insufficient number of tools on site to cut pipe in a manner which would reduce the introduction of foreign material into piping systems.
Ilaving a central tool crib would provide better control of tool usage and inventory.
l'otential Causal Factor (s)
Change Management Adequate FME material was not stocked in advance of FME program implementation. .
i f
i 1
ADM'1 EVE 12ntEINV.WPF l6 '
i 1
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 ;
l Recommendation 4A. Evaluate current supplies of FME barriers considering demands of new FME requirements and order the appropriate amounts.
4B. Contact research industry informational sources for fire resistant FME barriers and decon and storage of barriers.
4C. Make vacuum cleaners more accessible for use.
4D. Consider making site visits by maintenance personnel to other plants that "
currently have high quality FME programs.
4 E. Procure additional clamshell cutting tools. Consider sharing tools that are expensive and aren't used frequently between Comed stations.
4F. Provide appropriate training to the workforce on the use of these tools.
4G. Establish a central tool crib for all site workers to provide better control and inventory of tool usage.
1 1
ADM\LEVE12FMEINV.WPF 17
1 FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Issue #5 FeedbacPJMonitoring Evaluation April 30,1993, QC Supervisor requested QC inspectors to formally document their evaluation of foreign material inspection via a comment on their inspection reports. A sample of 1000 out of approximately 7000 reports were reviewed. Of this 1000,130 had documented FME review, of which three had a rejectable observation. However, these reports document final inspections versus in process inspection.
The SQV organization continued to monitor the FME program and initiate FMR's to document their findings. Periodic follow-ups with Senior Maintenance Management formed the bulk of the feedback. Numerous actions were taken to address concerns identified. However, these actions collectively were ineffective as described in the event narrative.
PIF's have been written since early 1993, and have identified 106 FME occurrences. PIF trending data on subject of FME was not available. Br. sed on review by the investigation team, the PIF data base did not adequately identify FME issues.
No evidence could be identified that the individual / group that initiated the intended change in the FME program assessed in-field performance to ensure original expectations were satisfied.
Conclusion Station Line Management and SQV were inadequate in tracking corrective action effectiveness.
Independent assessment did provide feedback on the program tt ?t could have prevented this but it was not adequately addressed.
Causal Factors Management Methods inadequate self assessment culture.
Recommendation 5A. Assign clear responsibility and accountability for FME program implementation.
ADM\ LEVEL 2\l31EINv.WTF 18
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Issue #6 Training Evaluation Discussions with operating and operating training personnel indicate that no FME training was given other than the refuel floor procedure to the non-licensed operators. Interviews indicated that FME control was considered a maintenance activity.
FME Training is provided to contract personnel as part ofint xtrination training for new hires.
Additional training was given to the contractor personnel that would be performing the valve and motor operator work during the outage.
The training on " Foreign Material Exclusion", Procedure QCGM 307-04, Rev. O, was presented to the maintenance departments (EM, IM and MM) 'n the form of tailgate sessions in October 1993. This training was a short presentation givet by a member of the training department to introduce the new procedure (Training Request Number 93-1184). The training was a short presentation to the entire maintenance department and routed as a required reading package to maintenance shops.
At the request of the procedure owner, it was decided to include QCGM 307-04 as the procedure in the first week (Block 1) for 1994 Maintenance Continuing Training. No training request was generated for this as it was part of Block i training that is required by the Administration and Management Information (AMI), the controlling document for maintenance training. This training has been postponed many times and is in progress.
Topics selected for procedure training are trained utilizing a cover sheet that lists generic objectives and the training is conducted using the procedure. These training sessions are approximately 4 hours4.62963e-5 days <br />0.00111 hours <br />6.613757e-6 weeks <br />1.522e-6 months <br /> long and cover the procedure and include discussion with and by the trainees.
The procedure owner's expectation was for training to be conducted utilizing hands on, video and examples of different types of barriers / dams in the conduct of the training. The procedure owner thought that the work analysts would get the same training as maintenance. Ilowever, the work analysts did not receive the training.
Conclusion Training provided was inconsistently applied to the workforce and did not match procedure owners expectations.
Atmu EVFD\fMEINV.WPF 19
~
FME PROGRAM DEFICENCIES Level 2 Investigation PIF 94-1837 Potential Causal Factor (s)
Verbal / Written Communication ,
Expectations for FME training was not adequately communicated.
Recommendation 6A. Establish a requirement to have the program owner review and approve the proposed training prior to conduct of training and/or observe the first class for any procedure that establishes or significantly revises a program.
6 13 . Incorporate FME training in the generic and First Line Supervisor corporate and site specific programs.
6C. Provide hands-on training (mock-ups) for FME.
ADW.EVEU\FMEINv.WPF 20
t IME PROGRAM DEFICIENCIES IAvel 2 Investigation PIF 94-1837 Other Issues The probability of FME events would be reduced if problem reporting is further encouraged.
One method to increase self disclosure of problems is to create an environment which promotes sdf reporting. Results from level 2 investigations on Rad Worker Practices, Human Performance discussions held by team members and interviews indicate the perception exists that reporting infractions would result in severe discipline. We need clear communication that you can come forward and report what you lost without fear of retribution.
From interviews and discussions among team members, a perception exists that the site only focuses on one issue at a time. This issue will be addressed outside of the report.
l I
l ADM\LEVE12f%tEINV,WPF 21 I
1 l
l FME PROGRAM DEFICIENCIES Level 2 Investigation !
PIF 94-1837 l
Attachment A Charter i
TO: Guy Campbell, Station Manager July 26,1994 i
FROM: Tom Kroll, Investigation Director
SUBJECT:
Investigation Team Charter to evaluate the Foreign Material Exclusion (FME) program at Quad Cities. The investigation is being initiated in response to the following events:
As required by QCPP 2301, Integrated Reporting and Root Cause Analysis, the purpose of this memorandum is to define the charter for this Category 2 investigation. ,
The investigation into the specific causes of the subject events is being done under a separate investigation. The purpose of this investigation is to evaluate the FME program to detennine what barriers have failed previously causing events such as the subject events and develop corrective actions to prevent recurrence.
The investigation will be performed in accordance with Station procedures, to identify the root cause(s) and to provide recommendations for any Corrective Actions (C/A). The scope of the investigation will consist of the following, as a minimum:
- 1. An evaluation of previous events (including the most recent occurrences) associated with FME at Quad Cities.
- 2. Evaluate the Quad Cities policy (procedures and practices) for foreign material exclusion.
- 3. Evaluate the training given to the work force regarding foreign material excluson.
- 4. Corrective actions to prevent recurrence of FME events.
Provide a preliminary report with proximate causes by August 3,1994.
1 You will be advised of any additions to the original list of issues and concerns, as they arise.
ADEEvrmMWy.WPF 22
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Your authorization of reasonable ovenime for Investigation Team members is requested. Proper pre-authorization will be obtained if the guidelines of Generic Ixtter 82-12 need to be exceeded. In addition, you will be immediately notified if any other potential operability / safety concerns are identified.
Concurrence:
Tom Kroll Guy Campbell Maintenance Superintendent Station Manager Quad Cities Quad Cities ADM\LEVEINMEINV.WPF 23
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Attachment B Recommendations to " Foreign Material Exclusion", QCGM 307-04, Rev 1, Attachment "C" DELETE: System Cleanliness Class ADD: System Number and indicate if the following are applicable:
Reactor / Feeds / Flows to Reactor Y or N Sensitive Equipment Downstream Y or N Cleanliness Requirements (Work Analysts to specify)
Stellite Seats (Valve repair) - Cobalt control techniques mandatory.
(Work Analyst to specify)
ADD: Install FME barriers, if possible, with inlet and outlet protection and any enhancements which are deemed necessary.
ADD:
BREACIIES INSTALLATION REMOVAL Number of Barriers: Number of Barriers:
Init. & Date Init. & Date Type: Type:
Location: Location:
ADNBLEVE!JTMEINV.WPF 24 e' "
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 -
Attachment C Investigation Team Membess Tom Kroll - Investigation Director Dale McCullcugh - Team Leader Jack Brunner - Root Cause Specialist Joe Eenigenburg Corporate Sponsor Dan Boehner - Quality Control Wayne Krause -IIPEC Coordinator Greg Donoho - Training LaVerne (Jim) Binzen - Mechanical Maintenance Jeff Odeen - Valve Group Ken Parkinson - Mechanical Maintenance Scott Flaker - QV overview Deb Ketcham - Clerical <
l Aosruvrunmsv.wrr 25 <
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 Attachment D Human Error Inappropriate Actions 11/18/93 While lapping seat on 1-4901 valve, lapping disc came off and fell down pipe.
11/23/93 Dust and paint overspray on Unit 1 HPCI valves and motors affect operation of HPCI.
12/26/93 Found oil rag in 2D Cond Booster pump mote oil reservoir (twice) Rx feed pump (twice).
5/23/94 "B' SBGT found debris in relay flow switch - did not operate.
Various problems occurred during excavation of the 1/2B Fire diesel fuel l 6/10/94 return line - sand entering storm drain - no barrier.
l 6/16/94 Tape placed on holes left when turbine vibration detectors are removed - !
FME practice barrier. ,
I l
6/24/94 Cire Water piping - 3' piece of rebar and other debris found (probably i origmal construction). )
11/22/93 1/2 DG CWP Room cooler fan did not start - metal in contacts.
7/14/94 Shredded rag bag found in MOV 1-1001-36A.
In-progress 11/29/93 Gland seal condenser LCV does not work. Baffles in hotwell had been drilled and left metal and welding slag travelled to level control valve.
3/21/94 HPCI 1-2301-63B restricting orifice plugged (welding slag).
6/15/94 SBLC test tank (found metal filing in test tank).
ADhnLEVEL2wMEINv.WPF 26
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 1/15/94 Found 2B CRD suction filter reading 480 MREM at contact.
4/25/94 Wrench being used for turbine overhaul drcpped into the condenser.
6/23/94 2A Recirc pump motor upper bearing reservoir due to vent plug missing.
i ADM'1EVEL2WMEINV.WPF 27
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 References
~
In Process maintenance Cleanliness QMMP 1500-34, Rev 4 SOER 90-02 " Nuclear Fuel Defects" FME Procedures from all CECO Stations, Duane Arnold and Clinton Summary of FME Procedure history write-up by Maintenance 46 Field Monitoring Reports associated with FME written between 3/93 and 11/93 7000 QC Inspection Reports from 4/93 to present NWR's that were done on work under investigation Maintenance training done on FME in 1993 and 1994: N-MECT, 04MCPT Training request 93-1184 " Provide training in support of improvement of FME Practices and Implementation of QCGM 307-04 (FME)"
! April 3,1994, Tom IIall memo (TH-94-001) to Department Heads and Training Coordina' ors to discuss FME and the FME procedure July 20,1994, ConGrmatory Action Letter (CAL) Rill-94-006 for Quad Cities Unit 1 Quad Cities proposed strategy for answering CAL Quad Cities inspection Reports documenting Unsat FME 93-1955 (Unit Condenser Hatches), 93-1797 (Condenser llatches) and 93-3973 (North Torus Hatch)
Foreign Material Exclusion QCGM 307-G4, Rev 0, Rev 1 Foreign Material Exclusion Interim Procedures, IP 466 and IP 665 Zion Corrective Action Record 22-92-020 and associated NTS records (29510094002)
SQV 94-039 SQV witness points for torus coating preparation Draft reports of foreign material in RHR Pump 1-1002-C (25420094036) and valve 36A (25420094035) i ADM4.EVIUUNEINV.WPF 28
FME PROGRAM DEFICIENCIES Level 2 Investigation PIF 94-1837 PIF 94-1835 written on dropping pen while inspecting Drywell Torus Vacuum Breakers Chart of FME issues versus systems based on NTS data base as of 7/27/94 NOD-OP-22, FME Inpo Good Practice MA 315, FME 1994 Training of Contractor Personnel on FME package for various trrdes- >
Investigation Action Plan for RIIR Valve 36A SQV 94-039 dated 6/2/94, SQV to NWR Q07884 concerning SQV witness points for Torus Coating Apparatus Memo dated 4/30/93, from M. Neels (QC Inspector) to all QC Inspectors in response to CAR 04-93-005 Memo TK-7-25.wpf dated 7/25/94, from T. Kroll to all maintenance and SEC working departments regarding FME Immediate Actions Memo dated 2/16/93 to II. IIentschel from Mike Pacilio concerning covering 4KV breakers when not in use Memo dated 4/29/93, (JD-93-15) from J. Dierbeck to Jim Burkhead regarding maintenance and SEC corrective actions and actions to prevent recurrence for CAR 04-93-005 NRC IEN 92-85 NRC information notice: Potential failure of ECCS systems caused by FME Material 131ockage Summary of contractor valve group training (1994)
ADM'1DT12\FMENY.WPF 29
. l Enclosure B: Long Term Corrective Actions Associated With The Phased FME Program (including Implementation Schedule).
Quad Cities Station acknowledges that plant problems have occurred due to foreign material entering systems and equipment. The following actions are being implemented to minimize this risk.
- 1. A station policy has been developed to support Management expectations regarding FME and is to be in place in October 1994.
- 2. The new FME procedure has been written combining both the balance of plant and refueling floor FME procedures (QCGM 307-4 and QCAP 270-1 respectively), in addition to other applicable procedures utilizing the best FME practices throughout the industry. Validation will be accomplished by the FME performance improvement team -
during October 1994.
- 3. Training lesson plans have been developed based on task analysis utilizing the governing procedure and subject matter experts. This training is knowledge based and performance based with hands-on mock-ups. Methods of specific techniques and FME practices are being included in site training. This training will be validated, and all plant personnel and supervision directly involved in system breaches will be trained by the next unit refuel outage. Ul new employees and contractors directly involved in system breeches will receive this training during their initial "In Processing "
- 4. A material inventory of commonly used FME devices is being established and will be made easily accessible to all work groups from the tool crib. The use of these devices will be discussed during training and demonstrated to the maximum extent possible. The material inventory of FME devices will be established by the next unit refuel outage. -
- 5. A self-assessment of our effectiveness will be performed by Maintenance during the second month of the next unit refuel outage. Members of the Quad Cities FME team and FME representatives from other nuclear stations will assist Maintenance in this self-assessment.
- 6. The station has appointed an individual from the FME Performance Improvement team to act in the capacity of FME Program Manager. The FME Program Manager is tasked with coordinating and monitoring the actions of the respective disciplines, perform in-plant surveillances, maintain an adequate supply of FME materials and interface with training and management to ensure that expectations are being met.
- 7. The FME performance improvement team is continuing to present awareness of foreign material exclusion throughout the station. This is being accomplished through the use l of photographs, Daily Link articles, an FMEA hotline, and displays in the main hallway of the Service Building. ,
1 l
Enclosure C: A Description Of The Phase 2 And Phase 3 FME Program Controls FME Improvement Plan - Phase Two Control Criteria
- Phase Two FME Control Criteria Matrix establishes four Levels of Control based upon the.
System being worked on, and the potential for the activity to introduce foreign material which can not be immediately retrieved.
- FME Roles and Responsibilities for the First Line Supervisor (FLS), Monitor, QC Inspector, Overviewer, and Worker (s) must be fully understood.
- Operations will continue to perform work activities involving FME under the Operating Department Foreign Material Exclusions Expectations.
- Pre-Job Briefs conducted by the FLS, must be attended by those persons identified on the Phase Two FME Contro' Jteria Matrix. The brief shall:
Discuss and establish necesst ry FME requirements, techniques, and their relationship with work activity requirements (eg. Rad. Protection, Quality Control, Safety, etc.),
Describe the requirements oa QCGM 307-04, Rev.1, Attachment C in Special Instructions section.
- It may be proper to change the level of FME Controls during the course of work. -For example, a Ixvel I work activity could be reduced to Level 2, following installation of FME barriers that would prevent the entry of foreign material into the system or component during the work. The level shall be escalated back to Ixvel 1 prior to removal of the barriers.
- Overviewers for level 1 and 3 activities must be an individual that has been trained and qualified by the FME Performance Improvement Team. Completed QCPP 24f i Overview Sheets are to be submitted to the FME Performance Improvement Team. ,
- The worker shall perfonn as an Overviewer for their own work for 12 vel 4 activities.
- For work packages already issued to the working groups, the Work Analyst must determine the FME lxvel based on the matrix. The Matrix level must be identified on the QCGM 307-04, Rev.1, Attachment C in the special instructions section. If the Work Analyst is unable to detennine the appropriate Matrix Ixvel, The System Engineer, Department Head, or Superintendent must be contacted for resolution.
FME Imyrovement Plan - Phase Three Control Criteria The Phase Three includes control criteria from the Phase Two FME Improvement Plan plus the following.
- " FOCUS ON PREVENTION", ensure that foreign material does not find its way into areas where it could cause damage or problems.
- FME is a Work Practice. As jobs evolve, conditions and requirements change.
- Phase Three FME Control Criteria Matrix establishes four levels of Control based upon the Systems being worked on, and the potential for the activity to introduce foreign material which can not be immediately retrieved.
- Pre-Job Briefs conducted by the FLS, must be attended by those persons identified on the Phase Three FME Control Criteria Matrix.
- During work, each FME barrier used shall be listed in the Special Instructions section of QCGM 307-04, Rev. I, Attachment C. The following shall be recorded:
1 Barrier Type Location Installation Time and Date Removal Time and Date
- Phase Three FME Control criteria lifted the Phase Two requirement of having a QC Inspector present for all system breeches. QC Inspectors will be present if required by a hold point.
l l
i
.