ML20134C360
| ML20134C360 | |
| Person / Time | |
|---|---|
| Site: | Waterford  |
| Issue date: | 01/30/1997 |
| From: | Gaudet T ENTERGY OPERATIONS, INC. |
| To: | NRC OFFICE OF INFORMATION RESOURCES MANAGEMENT (IRM) |
| References | |
| W3F1-96-0198, W3F1-96-198, NUDOCS 9702030022 | |
| Download: ML20134C360 (23) | |
Text
_
- e Entargy gy,% %,%
--=-
Operations mu u 7 cme
- !504 739-6650 W3F1-96-0198 A4.05 PR January 30,1997 U.S. Nuclear Regulatory Commission ATTN
- Document Control Desk Washington, D.C. 20555
Subject:
Waterford 3 SES Docket No. 50-382 License No. NPF-38 Update of Actions to Address Fastener Discrepancies at Waterford 3 Gentlemen:
On October 3,1996, Waterford 3 submitted letter W3F1-96-0177 to the NRC providing the status of actions to address fastener discrepancies identified on motor operated valves (MOVs). The letter outlined the actions being taken and/or planned to address MOVs specifically as well as generic actions to bound the issue. Since then the Root Cause Analysis (RCA) Team has completed its investigation and has documented the results in a RCA report. The purpose of this letter is to provide an updated status of actions taken to address fastener discrepancies and to summarize the causes and corrective actions determined by the RCA Team.
As discussed in the October letter, a walkdown and external inspection of all safety-related and Category 1 (GL 89-10) non-safety-related MOVs identified a number of inconsistencies. This prompted the need to inspect other components for similar inconsistencies to bound the issue. The inspection scope was expanded to safety-related air operated valves (AOVs), Emergency Diesel Generators (EDGs), motor terminal boxes, electrical and I&C terminations, electrical and l&C mounting 9702030022 970130 PDR ADOCK 05000382 P
PDR l
)
Update of Actions to Address Fastener Discrepancies at Waterford 3 W3F1-96-0198 Page 2 January 30,1997 j
hardware, and mechanical supports and mounting hardware. The following is a brief summary of the scope and results of the inspections that have been completed thus far:
MOVs: This inspection looked at over 1200 fasteners on 78 t.iOVs with the following discrepancies / inconsistencies noted: 2 missing fasteners,53 non-carbon steel fasteners (18 in critical applications), and over 400 missing lock l
washers. Of the 18 non-carbon steel fasteners identified in critical applications, 12 were found to be originally supplied by Limitorque,4 were installed during construction /early startup, and only 2 were installed post-1985. This was evident j
by the paint on these fasteners. All identified non-carbon steel fasteners in critical applications have been replaced with the correct material and grade fasteners.
]
AOVs: The original AOV inspection consisted of a 10% sample of safety-related AOVs. A number of minor discrepancies was noted for this 10% sample, and the scope of AOV inspections was increased by 10%. The inspection of 20% of safety-related AOVs looked at approximately 2000 fasteners on about 48 AOVs with the following discrepancies / inconsistencies noted: 22 missing fasteners,37 1
material differences, and 32 missing lock washers. This number of discrepancies, although mostly minor, prompted an expansion of the inspection 3 ample to all safety-related AOVs (233 total).
. EDGs: This inspection looked at over 800 fasteners on both Emergency Diesel Generators with the following discrepancies / inconsistencies noted: 7 loose / stripped fasteners,4 material differences, and 35 missing /different lock washers.
I&C Components: Inspection of approximately 640 fasteners on 20 components e
has identified the following discrepancies / inconsistencies: 3 suspect fasteners,6 loose fasteners, and 25 missing washers.
Electrical Components: Inspection of approximately 2800 fasteners on 11 components has identified the following discrepancies / inconsistencies: 8 suspect fasteners,15 loose fasteners and 74 missing washers.
Mechanical Components: Inspection of approximately 2300 fasteners on 26 components identified the following discrepancies / inconsistencies: 26 suspect fasteners and 7 missing washers.
Update of Actions to Address Fastener Discrepancies at Waterford 3 W3F1-96-0198 Page 3 January 30,1997 in addition to the above field inspections, document searches were performed to evaluate the scope of fastener-related issues at Waterford 3. This included a review of Condition Reports (CRs) for the past two years, NRC Inspection Reports for the past several years, INPO field notes, a 1994 QA Audit on Fasteners, and numerous work packages including those associated with the identified fastener discrepancies on MOVs. The RCA Team evaluated the results of these reviews to determine if trends of fastener discrepancies had previously existed or been identified. The work package reviews were performed to determine if the MOV discrepancies were attributable to specific individuals, provide insight into the circumstances surrounding the discrepancies, and to determine if fasteners and other parts were requisitioned properly. Interviews of maintenance shop personnel were also conducted to ascertain knowledge levels regarding the use and replacement of fasteners.
The RCA Team evaluated the results of the field inspections and document reviews to identify the root and contributing causes. Attachment 1 consists of the Root Cause Determination, Generic Implications, and Recommended Corrective Action Plan sections from the CR-96-1528 RCA Report. In summary, the numerous fastener discrepancies / inconsistencies identified by the field inspections and document reviews suggest a generic deficiency in fastener-related work practices at Waterford 3. Comprehensive analysis of the results does not, however, suggest a universal disregard for fasteners but rather a lack of attention to detail for fasteners that are perceived to be non-critical. The issue can be appropriately characterized as a work practice problem with lock washers and auxiliary or non-critical fasteners.
As a result, there does not appear to be a need to perform an entire plant walkdown for fastener discrepancies at this time. However expectations have been set for Maintenance personnel to identify and correct fastener discrepancies as work is performed on components. It should also be noted that an independent review team was established to evaluate the issue of fastener controls at Waterford 3. The result of the team's assessment confirmed the conclusions drawn by the RCA Team, it is important to note that the fastener discrepancies identified have been determined to not affect component operability. While the identified discrepancies have been determined to be non-safety-significant, Waterford 3 has put forth an extensive effort to bound the issue and establish an aggressive and effective corrective action plan. Waterford 3 believes that this action plan will resolve the identified fastener discrepancies which currently exist in the plant and provide the necessary measures to ensure future discrepancies in this area are minimized.
1 Update of Actions to Address Fastener Discrepancies at Waterford 3 W3F1-96-0198 i
Page 4 January 30,1997 i
2 If you have any questions concerning the a' cove, please contact me at (504)739-6666.
Very truly yours, l
1
'+04k
)
l T.J. Gaudet Acting Director Nuclear Safety and Regulatory Affairs TJG/DFL/tjs Attachment cc:
E.W. Merschoff (NRC Region IV)
C.P. Patel (NRC-NRR)
R.B. McGehee N.S. Reynolds NRC Resident inspectors Office
l l
h L
i i
i 4
i CR-96-1528 Root Cause Determination Generic Implications Recommended Corrective Action Plan
Root Cause Determination Several analysis techniques were utilized in determining the root causes of the fastener discrepancies. These techniques included barrier analysis, Taproot @, and task analysis. To facilitate the analysis and to aid in accurately determining the root and contributing causes, the issues / discrepancies and associated evaluations were segregated into severallogical categories. These categories are as follows:
Missing Lock Washers Missing Fasteners (bolts, studs, nuts)
Fastener Material Differences (Stainless, Brass) 1 1
Additionally, the root cause analysis process did entail specific evaluation of the issues by component type (MOV, AOV, etc.). Although this further refinement in the evaluation process did not indicate any substantial differences in the root causes, the segregation was usefulin assessing the safety significance and in establishing the appropriate corrective actions.
4 Fastener paint was an additional consideration in determining the root causes. Those MOV fasteners with manufacturer paint underneath the white paint were supplied by limitorque and installed in the plant as part of the component assembly. Fasteners with white paint (but no manufacturer paint) were probab.y installed into the component during the construction or early startup era. (Plant components were painted white by 1985.)
Those fasteners with no paint were installed sometime after 1985. It should be noted that in no case was there sufficient information (work package reviews, interviews, etc.) to definitely identify when, how, or why the non-painted discrepant fasteners were installed.
MISSING LOCK WASHERS Identified Root Causes Training: Situation not Covered ( Corrective Actions #6, #7, #18 ) - Maintenance worker training for plant components typically does not include specific guidance on the use of lock washers. Neither the training lesson plans nor the skill demonstrations provide sufficient detail in the assembly and disassembly processes to specifically address lock washer use.
1 l
1 1
Root Cause Analysis Report Root Cause Determination
- l
~ ~
~ - ~ ~ -
~
~
Work Practices: Lack of Attention to Detail ( Corrective Actions #6, #7, #18).
Over the ten year period of operation since startup, there have been many periodic tasks (tests and PMs) performed on MOVs, AOVs, etc. Although procedures, training, and clear consistent management expectations are somewhat deficient in the area of lock washer use, there are tacit expectations that the " Tool Box Knowledge" of maintenance workers and the experience of supervisors and technical support personnel is sufficient to at least question why inconsistencies in the use or application of lock washers exists between identical /like components.
Contributing Causes Procedures: Level of Detail not Addressed (Corrective Actions #6, #7, #11, # 13,
=
- 14, #15, #18 ) - Procedures used in performing component maintenance work activities are inconsistent in specific requirements regarding installation of lock washers.
References to removal and replacement of washers is provided in some maintenance procedures and not in others. Additionally, diagrams in the vendor manuals and work instructions do not consistently depict the need or specify the requirement for lock
~
washers in the appropriate applications. Procedure ME-004-010 (Limitorque Motor Operator Maintenance for SMB-000), for example, contains several explicit illustrations and several steps indicating the requirement for lock washers. Other diagrams and steps within the procedure, however, make no reference to the need for lock washers.
Management Systems: Standards, Policies, and Administrative Controls were absent, confusing, or incomplete (Corrective Actions #6, #9, #13, #15 ) -
Although there may be tacit expectations that the " Tool Box Knowledge" of maintenance workers and the experience of supervisors would be sufficient to recognize the need for and to remedy deficient fastener conditions by installation of appropriate replacements, no clear explicit expectations regarding fasteners exist.
Management expectations, are not consistently clear and explicit on the installation and use of lock washers.
MOV MISClNG FASTENERS (BOLTS, STUDS, NUTS)
J Identified Root Causes Work Practices: Lack of Attention to Detail (Corrective Actions #6, #7, #12,
- 18). - During field inspections of MOVs, a missing top housing fastener on BAM-133 4
and a missing motor mount stud / nut combination on SI-125B were identified. Because of !!'e orientation of the fastener /MOV combinations, it is unlikely that these fasteners v.brued loose during operation.
i i
Root Cause Analysis Report Root Cause Determination e
Over the ten year period of operation rince startup, there have been many periodic tasks (tests and PMs) performed on these MOVs where replacement of fasteners removed during the maintenance activity could have been inadvertently omitted.
Neglecting to replace fasteners removed in the disassembly process indicates of a lack of attention to detail.
Failing to install new fasteneis when as-found conditions reflect a fastener deficiency also indicates a lack of attention to detail. There are tacit expectations that the ' Tool Box Knowledge" of maintenance workers and the experience of supervisors would be sufficient to recognize the need for and to remedy the deficient condition by installation of an appropriate replacement.
MOV FASTENER MATERIAL DIFFERENCES (stainless stell, Brass vs. carbon steel)
Identified Root Causes Documents: Procedures, Vendor Manuals, and Drawings either fail to or inconsistently reference fastener material requirements ( Corrective Actions #9,
- 11, #13 ) - Procedures, vendor manuals, and drawings used in performing MOV work activities are inconsistent or lacking in specific requirements regarding fastener material requirements. Niany of the MOV operator drawings used by planners, maintenance workers, and technical support personnel do not make specific reference to external fasteners or to the material requirements. (This type of information should at least be readily available in the vendor manuals or in drawings.)
Training: Situation not Covered ( Corrective Actions #6, #7, #18 ) - Worker training for work on MOVs does not include specific guidance on fastener material requirements. The trair 3 esson plans as well as the skill demonstrations do not l
provide sufficient detailin the assembly and disassembly processes to specifically delineate fastener material requirements.
Vendor Supplied Components: Deficiency in Original Supply ( Corrective Actions
- 3, #8 ) - Twelve (12) non-carbon steel fasteners in critical applications were identified as original equipment supplied with operator.
Construction Practices: Deficiency in Construction Parts Installation ( Corrective Actions #3, #8 ) - Four (4) non-c.arbon steel fasteners in critical applications and 35 non-carbon steel fasteners in non-critical applications were identified as having been installed during plant construction /early startup.
Roc:Cause Analysis Report Root Cause Determination
- Contributing Causes Management Systems: Standards, Policies, and Administrative Controls were absent, confusing, or incomplete ( Corrective Actions #6, #7, #13, #14, #15, #18).
- Although there may be tacit expectations that the " Tool Box Knowledge" of maintenance workers and the experience of supervisors would be sufficient to recognize the need for and to remedy deficient fastener conditions by insbilation of appropriate replacements, no clear explicit expectations Gregarding fasteners exist.
Additionally, neither the training lesson plans nor the training lectures and/or demonstrations make mention of fastener material requirements. (Thic contributing cause may have been a factor only in the installation of the two non-carbon steel fasteners that were installed post 1985.)
i MOV FASTENER MATERIAL DIFFERENCES (GRADE 2 VS. GRADE 5 or 8)
Identified Root Cause j
Equipment / Parts Defective: Vendor Supplied Equipment did not meet Vendor Specificatiorys ( Corrective Action #1, #3, #8, #10, #16 ) - The external fasteners supplied by Limitorque did not meet the Grade 5 requirements as specified by Limitorque.
Limitorque acknowledged that during the 70's and early to mid 80's they had limited control on their sub-suppliers and performed no testing on their sub-supplier's fasteners. The internal fasteners have also been found not to meet the Grade 5 or 8 requirements. The internal fasteners are not loaded very heavily. Engineering analysis has shown that fasteners meeting Grade 2 requirements have a high margin of safety. The intemal fasteners were only supplied by Limitorque.
AOV MISSING FASTENERS (BOLTS, STUDS, SCREWS, NUTS)
Field inspections of AOVs indicated that numerous non-critical fasteners were missing. No definitive evidence was available to determine the exact time or circumstances surrounding each AOV fastener discrepancy, however, several probable causes are likely. Since the AOV discrepancies are being treated collectively, there is reasonable assurance that the causes listed below are in fact root causes.
Root Cause Analysis Report Root Cause Determination
- Identified Root Causes Work Practices: Personal accountability for ass'igned responsibilities was less than adequate. (Corrective Actions #6, #7, #13, #18 ).
- During the ten year period of operation since startup, many periodic tasks (tests and PMs) have been performed on these AOVs where replacement of fasteners removed during the maintenance activity could have been inadvertently omitted. Neglecting to replace fasteners removed in the disassembly process and/or failure to install new fasteners when as-found conditions reflect a fastener deficiency indicates a lack of attention to detail or a lack of appreciation for the importance of fasteners in seemingly non-critical
]
applications.
Training: Situation not Covered ( Corrective Actions #6, #7, #13, #18 ) - Worker training for work on AOVs does not include specific guidance on fasteners and the need
)
for remedying deficient fastener conditions. There are implicit expectations that " Tool Box Knowledge" is sufficient in the area of fasteners, however, neither the training lesson plans nor the skill demonstrations provide sufficient fastener detail or emphasis in the assembly and disassembly processes.
AOV FASTENER MATERIAL DIFFERENCES (STAINLESS STEEL, BRASS) s identified Root Causes Documents: Procedures, Drawings, Vendor Manuals, and Work Instructions inconsistently or fall to reference fastener material requirements (Corrective Action #6, #7, #13, #18 ) - Procedures, drawings, vendor manuals, and work instructions used in performing AOV work activities are inconsistent or lacking in specific requirements regarding fastener material requirements. Many of the AOV operator drawings used by planners, maintenance workers, and technical support personnel do not make specific reference to external fasteners (or to the material requirements).
Training: Situation not Covered ( Corrective Action #6, #7, #18 ) - Worker training for work on AOVs does not include specific guidance on fastener material requirements. Neither the training lesson plans nor the skill demonstrations provide sufficient detail in the assembly and disassembly processes to specifically delineate fastener material requirements.
Root Cause Analysis Report Root Cause Determination
- Centributing Cause Management Systems: Standards, Policies, and Administrative Controls were absent, confusing, or incomplete ( Corrective Action #6, #7, #13, #14, #15, #18 ).
- Although there may be tacit expectations that the " Tool Box Knowledge" of maintenance workers and the experience of supervisors would be sufficient to recognize the need for and to remedy deficient fastener conditions by installation of appropriate replacements, no clear explicit expectations regarding fasteners (or the specific materials to be used for different fastener applications) exist.
GENERAL CONTRIBUTING CAUSES Procedures: Procedure UNT-005-002, Sections 3.14.3 & 5.1.2 provide the latitude for performing certain maintenance activities outside of the Cl process. (Corrective Action
- 14 ) - Procedure UNT-005-002, Sections 3.14.3 and 5.1.2 currently allow replacement of fasteners under the Minor Maintenance process. Excerpts from UNT-005-002 indicating circumstances where such latitude is allowed include
".. accessory parts that perform no operational function, such as nameplates, latols, l
cover plates, inspection plates.." (Section 5.1.2.10)
. if the component is safety related, the portion of a part worked does n_ot perform or affect a safety related function" (Section 3.14.3.1)
" Replacement parts on a safety related component do n_ot perform or affect a safety related function" (Section 3.14.3.3)
The latitude to perform such activities outside of the Cl process can be quite beneficial and should not necessarily be considered a deficiency. This latitude, in conjunction with the procedural and training deficiencies described earlier and the extended period of operation over which fastener discrepancies may have existed, may hcve conveyed something less than the tacit expectations that the proper material, combination, and arrangement of fasteners should be installed in all appropriate locations.
Equipment: Deficiencies in Vendor Supplied Components ( Corrective Actions #18 ) -
Limbrque operators do not always come with lock washers in applications where lock washers are appropriate. (Two out of six spare operators currently in the shop were received from Limitorque without limit switch cover lock washers. These inconsistencies in Original Equipment Manufacturer (OEM) supplied equipment are visually observable by the maintenance staff and may suggest that inconsistency in fasteners is acceptable.
Root Cause Analysis Report Root Cause Determination.
1 i
Documentation: Deficiencies in Vendor Supplied Documents ( Corrective Actions
- 13) - - Many diagrams, drawings, and sketches within the Limitorque manuals are not explicit with regards to lock washers or to material grades. 'Often, in cutaway or exploded views of the operators, no specific notation regarding the fastener or lock washer is made.
j This inconsistency, and perhaps deficiency, in Original Equipment Manufacturer (OEM) l supplied documents may contribute implicitly to a lower perceived expectation regarding l
fasteners.
Management Systems: Standards, Policies, and Administrative Controls were absent, j
confusing, or incomplete (Co: rective Action #6, #7, #13, #18 ). - Although there may be tacit expectations that the " Tool Box Knowledge" of maintenance workers and the experience of supervisors would be sufficient to recognize the need for and to remedy deficient fastener conditions by installation of appropriate replacements, no clear explicit expectations regarding l
fasteners exist.
Management expectations (e.g., " Zone Ownership"), are not consistently clear and explicit on the installation and use of lock washers. References to removal and replacement of washers is provided in some maintenance procedures and not in others. Additionally, diagrams in the vendor manuals and work instructions do not consistently depict the need or specify the requirement for lock washers in the appropriate applications.
Expectations regarding the proper material use for fasteners are not always explicitly or consistently communicated. For example, although Procedure ME-004-012 provides a note regarding material grade requirements for hex head cap screws and socket head cap screws, other procedures used in valve operator maintenance activities are noticeably deficient of such guidance. Additionally, neither the training syllabus nor the training lectures and/or demonstrations make mention of fastener material requirements.
OTHER Other potential root causes were evaluated during the investigation / analysis processes. No evidence was identified, however, to suggest other root or contributing causes to the fastener discrepancies. Several other areas that were specifically investigated with respect to the fastener discrepancies are as follows:
QC - The QC independent review process was not determined to be a factor primarily because QC does not typically cover activities on non-torque fasteners. The responsibility for quality workmanship in these cases, rests with maintenance staff personnel.
Root Cause Analysis Report Root Cause Determination
- Warehouse Controls - Document reviews, intentiews, and field inspections did not reveal sufficient evidence to suggest that the fastener discrepancies may have been caused by warehouse or material controls. Only 9 for the over 9800 fasteners inspected had discrepancies that were potentially related to material control.
Supervision - The lack of explicit expectations, the deficiencies in documentation, and the absence of fastener emphasis in skills training rendered maintenance supervision as deficient in fastener knowledge as craft personnel. It is unlikely that any substantial value (fewer discrepancies) might have been added by even the most conscientious supervision. Further, none of the evidence evaluated suggested a deficiency in supervision.
Self Checking - Lack of clear expectations and worker awareness regarding fasteners limit the value or benefit of even the most diligent practice of the STAR process.
Without the expectations or worker awareness, the self checking is inconsequential.
FUTURE INSPECTION RESULTS It is quite likely that the ongoing and planned inspections will identify additional fastener discrepancies. The root causes outlined above, however, willlikely be responsible for these additional discrepancies. In those cases where the above root causes do not apply, the station CR process will ensure proper identification of root causes and corrective actions, as appropriate.
Root Cause Analysis Report Root Cause Determination.
e e
Generic Implications The numerous fastener discrepancies identified by the inspections and investigations suggest a generic deficiency in fastener related work practices. As discrepancies were identif ed during the initial field inspections, the potential generic implications were recognized and Waterford 3 management progressively expanded the scope of the inspections. As inspection results were analyzed and root causes identified, an assessment as to the generic implications was made. This generic implications assessment was twofold. First, field inspection results and document review results were evaluated to bound or determine the actual scope of fastener related discrepancies. Second, the root causes or potential deficiencies contributing to the fastener discrepancies were also evaluated to determine the extent to which these deficiencies may have degraded the performance of plant equipment, processes, or personnel. The results of these assessments are summarized below.
FASTENER BEFICIENCY EVALUATION Comprehensive analysis of the MOV, AOV, and EDG inspection results does not suggest a universal disregard for fasteners but rather a lack of attention to detail for fasteners that are perceived to be non-critical. There is no appreciable evidence from either the results of the expanded scope inspections or the extensive documentation reviews to suggest that the less than adequate work practices have carried over into critical areas. Thorough evaluation of the investigation results suggests that this issue be characterized as a work practice problem with lock washers and auxiliary or non-critical fasteners. This position is further supported by the QA interviews with maintenance personnel which indicated a broad base of understanding for critical vs non-critical fasteners.
ASME Related Applications Pressure boundary connections are torqued connections and require second party verification.
Typically, no lock washers are required in torqued applications. Work affecting ASME related equipment requires more rigorous controls, have detailed documentation and specification and is performed by mechanical maintenance personnel with the high level of fastener awareness. These mitigating factors provide reasonable assurance that the fastener issue described herein is not applicable or generic to ASME applications or pressure boundary applications.
Root Cause Analysis Report Generic implications.
I Other Component Critical Fasteners The reviews of the documentation on AOVs, hydramotors,' diesel generators, and pumps i
showed that critical fastener information is easily obtainable. The component drawings usually show the critical fastener and give the specifications in the bill of material. If the drawing did not give the necessary information, it would be found in the vendor manual.
The availability of the necessary information and no critical fastener discrepancies identified in the field inspections, gave reasonable assurances that there was not a critical fastener problem in the plant. The critical fasteners on the MOVs was attributed to the lack of the necessary information on fastener specifications.
1 2
ROOT CAUSE / POTENTIAL DEFICIENCY EVALUATION Training 4
l The lack of adequate training in fasteners (a non-critical or seemingly non-significant aspect of the maintenance function) was cited as a root or contributing cause in the fastener issue.
i Investigation results provided no evidence that omissions in maintenance worker training j
carried over into significant or component functional aspects of maintenance. A more generic view of these omissions in the fastener area, however, might suggest the potential for similar i
omissions in other non-critical or seemingly non-significant aspects of the maintenance l
function. (e.g., personnel protective equipment, work area housekeeping, etc.) Maintenance Training has oeen advised of this an will evaluate other training areas.
l Materials identification (documentation deficiencies)
The lack of readily available, easy-to-use information on fasteners was cited as a root or i
contributing cause in the fastener issue. This difficulty in obtaining fastener related information j
might be viewed more generically to a lack of adequate parts information. Although no specific evidence was identified during these investigations to support this potential generic deficiency, it should be noted that efforts are underway to establish readily available lists of materials (Bills of Materials) for all components.
Root Cause Analysis Report Generic implications.
^-
Vendor Supplied Materials Fasteners deficiencies originating from the vendor (Limitorque) were identified during the investigations and were cited as a potential contributing cause in the fastener issue. One line of generic extrapolations might suggest that other plant components or parts may be deficient as a result of being supplied in a deficient state directly from the vendor. Rational arguments can be made, however, that plant equipment capability was verified during startup and is validated by successfulimplementation of the surveillance program. Another potential extrapolation of deficient vendor supply and the primary consideration in this deficiency being a potential contributor to the fastener discrepancies was the implication that a lower level of quality in non-critical applications is acceptable. Nothing in field inspections, document reviews, or personnel interviews, however, suggests that deficiencies in vendor supplied equipment was contributing to lower quality work in any particular area other than MOV non-l critical applications.
Management Expectations Deficiencies in management expectations relating to fasteners was cited as an underlying or contributing cause in the fastener issue. A more generic view of this deficiency suggests that 4
management expectations in other areas might also be deficient. Although this may be the case, none of the inspection results or other such evidence provided any basis for this extrapolation.
Material Control Fifty-three (53) fasteners made of materials other than those specified by design prompted the evaluation of material control as a potential generic issue. Several key observations made during the evaluation should be noted to put the issue into perspective. First, there was no objective evidence to suggest that non-carbon steel fasteners were issued for applications where carbon steel was sp'ecified. There were, however, several indications where fasteners were installed during WA controlled activities but no ROS was issued. Secondly, of the 53 wrong material fasteners identified during the MOV inspections, 35 were installed in non-criticallocations by either Limitorque or construction era personnel. Of the remaining eighteen (18) in critical applications, twelve (12) were part of the original supply from Limitorque and four (4) were installed by construction era personnel. Only two wrong material fasteners were potentially installed under the current material control processes and the current work control practices.
Root Cause Analysis Report Generic implications.
To characterize the material control aspect of this fastener issue, several investigations were performed and the results were evaluated. These investigation efforts are outlined as follows:
Extensive reviews of work authorizations / packages to determine if material control was responsible for the discrepant fasteners Field inspections to determine the extent of wrong material installations Research of stock codes in MMIS to determine if the wrong material may have been provided for issuance in the discrepant applications QA interviews of maintenance personnel QA audit of 150 safety related work authorizations to look for material issue problems.
Only 1 fastener problem was identified out of of 300 parts.
j The conclusions reached from the generic material control issue evaluation are as follows:
- 1. There is no evidence to suggest that the material issued from the warehouse was different from that requested on the ROS.
- 2. There are only two potential instances of wrong materials being installed on an MOV under current work practices. This oversight does not constitute a trend nor does it suggest a disregard for acquisition of fasteners of the proper material.
- 3. There is no evidence to suggest that wrong materials were either issued or installed in AOV applications. (Specifications from manufactures regarding AOV non-critical fasteners do not exist at the present time.)
- 4. Field inspections have been expanded beyond MOVs and AOVs to determine the full extent of fastener material reisted discrepancies in other component applications.
The number of discrepant fasteners that could be affected by poor material control weakness was very low. Of the over 9800 fasteners inspected, only 9 could be related to the material j
control process. The bulk of the discrepancies were missing lock washers. There were some suspect material fasteners installed in MOVs during construction. The specifications for these fasteners did not exist until 4 years after construction was completed. There were other I
suspect fasteners on AOVs. There is presently no specification requirements for those fasteners.
Through inspections and investigations the specific fastener material discrepancies have been bounded. Anslysis of the investigation results do not suggest a widespread disregard for materials when specific materials requirements exist.
Root Cause Analysis Report Generic implications e
i During the audit of the 150 work packages, nonfasteners related material discrepancies were identified. CR-96-1680 was generated to document that concern. It identified 9 parts where the end use evaluation could not be readily found. Further investigation reduced this to three items that were not acceptable.
The first item was copper fittings on the instrument air supply lines for BAM 141. A commercial grade evaluation should have been performed before installation. This issue was resolved by doing an after-the-fact evaluation.
The second item was a pneumatic booster relay installed in EFW 224B. There were two stock codes for this item. One had a revised commercial grade evaluation that required installation testing. The other stock code did not have the information advising the technician that testing was required. This issue was resolved by performing the required tests on the installed unit.
The third item was the fastener item. Quality Class 4 bolts were installed on the Containment Spray Pump B coupling guard. This issue was resolved by replacing the bolts with Q-1 bolts.
Three separate CR's were written for those items. This information plus an identified adverse trend in material control CR's, resulted in a root cause investigation team being set up just to address material controlissues.
I Root Cause Analysis Report Genericimplications e
s Recommended Corrective Action Plan As a result of the initial MOV field inspections several immediate corrective actions were initiated. These included remedial actions to correct critical fastener discrepancies and to expand the scope of the field inspections. During the subsequent inspections and as a result of the ongoing evaluations, additional corrective actions have been initiated. Some of these actions are complete. some are in progress and some are pending. The corrective actions listed below include those actions which were established as a direct result of the investigations and evaluations as well as those resulting from the root cause analysis herein.
1.
Parts Hold - A hold was placed on Limitorque parts in the warehouse. Acquisition of parts for Limitorque operators must receive an engineering evaluation until such time as a complete evaluation of Limitorque supplied inventory has been verified acceptable.
Resp. Group:
Materials Technical ECD:
Complete 2.
Briefing - Maintenance personnel were briefed on the fastener issue as an interim measure until such time as all appropriate corrective actions were established and implemented.
Resp. Group:
Maintenance ECD:
Complete 3.
Fastener Replacement - Non-carbon steel fasteners identified in critical applications on MOVs were replaced with Grade 5 carbon steel fasteners.
Resp. Group:
Maintenance ECD:
Complete i
4.
Expanded Field Inspections - Field inspections were expanded to encompass all safety related/GL 89-10 Program Category i MOVs, an inspection of 20% of the safety related AOVs and a general externalinspection of the site emergency diesel generator.
i Results from these inspections prompted further expansion of the field inspections to include spare MOVs in the warehouse, motor termination box fasteners, electrical terminations and mounting hardware, I&C instrument terminations and mounting hardware, and mechanical support and mounting hardware.
Resp. Group:
Maintenance ECD:
Complete Root Cause Analysis Report Recommended Corrective Action Plan e
~
,. i 5.
Cls initiated - Cls have been generated to remedy identified fasteners discrepancies.
Resp. Group:
Maintenance ECD:
Complete l
6.
Letter To Maintenance - A letter to maintenance department personnel will be issued J
providing explicit expectations on fasteners. It shall serve as an interim measure until such time as fastener training is provided. The letter will also give guidance on when to use corrective maintenance, when to write CRs an wh an to get Engineering assistance.
Resp. Group:
Maintenance ECD:
Complete 7.
Training - A special training module on fasteners will be developed and provided to maintenance personnel. This training will address:
policy on torqued vs non-torqued fasteners (use and application of lock washers) process for :;cquiring the proper replacement fasteners addressing as-found discrepant fastener conditions
, questioning inconsistencies in fastener applications in the field Resp. Group:
Training / Maintenance ECD:
12/31/96 8.
Fastener Replacement - All safety related and GL 89-10 Program Category i MOVs fasteners will be replaced with the specified fasteners (proper materials, combinations).
MOVs without high margins of safety will be competed by the end of RF8. Those MOVs with high margin of safety will be worked as they receive normal maintenance.
The internal fasteners will be replaced at the same time as the external fasteners.
Resp. Group:
Maintenance - Elect.
ECD:
6/1/97 -Identified MOVs 12/31/00 - Remaining MOVs 9.
"Like for Like" - The definition of "Like for Like" will be clarified to include material requirements in procedure UNT - 005 - 015, Work Authorization, Preparation, and implementation.
Resp. Group:
Maintenance ECD:
12/15/96 10.
Parts Verification - Remove limitorque parts from the warehouse that do not meet SAE Grade 5 or 8 specifications.
Resp. Group:
Maintenance Engineering ECD:
12/15/96 Root Cause Analysis Report Recommended Corrective Action Plan e
i 11.
VETIP Update - Revise the MOV Vendor Technical Manuals to provide a user friendly presentation of fastener requirements. VETIP Tracking # 96-369 Resp. Group:
Programs Engineering ECD:
In Progress (1/30/97) 12.
Worker Counseling - Because over a third of the electricians were involved with the most recent work performed on SI-332-B BAM 133 and SI 125B the entire shop will be counseled on the recognition and remediation of fastener discrepancies.
Resp. Group:
Maintenance-Electrical ECD:
12/1/96 13.
Fastener NWT - Establish a Natural Work Team (NWT) to formulate guidance on fasteners. This team should focus on those auxiliary or non-critical fasteners not typically addressed within vendor documentation. The recommendations resulting from this NWT will become corrective actions for CR 96-1528. This team will also determine the need for specific fastener guidance in procedures Resp. Group:
Maintenance ECD:
12/1/96 14.
Revise UNT-005-002 - UNT-005-002 will be revised to clarify the policy on minor maintenance on safety related component fasteners.
Resp. Group:
Maintenance ECD:
12/15/96 i
15.
Revise " Zone Ownership" Program - The Zone Ownership Program guidance will be revised to clarify that safety related component fastener replacements require proper documentation and material control.
Resp. Group:
Maintenance ECD:
11/30/96 16.
OE Notice - An operational experience notice will be issued to inform industry participants about the vendor related fastener material deficiencies.
Resp. Group:
Nuclear Safety ECD:
Complete 17.
Additional AOV inspection - The remaining AOV's will be inspected to current auxiliary fastener discrepancies.
Resp. Group:
Maintenance ECD:
6/1/97 Root Cause Analysis Report Recommended Corrective Action Plan e
1 18.
Training Long Term - The fastener training will be incorporated into the maintenance basic training modules. A training module for contractors will also be prepared.
j Resp. Group:
Training ECD:
4/1/97 19.
Additional MOV Internal Fastener inspections - Eight MOVs will receive an internal fastener inspections.
Resp. Group:
Maintenance ECD:
12/31/97 l
l NOTE:
)
A root / contributing cause of the identified discrepant fasteners was the lack of adequate j
guidance in procedures. In spite of this, no procedure revisions were recommended. The omission of any procedure revision recommendations was deliberate and was based on the l
following:
that level of detail is not warranted or even desired in procedures, and
" Tool Box Knowledge" should be sufficient or additional training should be provided to address fastener requirements.
4 1
Additionally, a more through evaluation of fasteners and associated strategies (by way of the NWT) should provide a better basis for procedural revisions, if any, or for the development of general / specific plant guidance on fastener use and requirements.
i i
4 e
a i
i 1
1 Root Cause Analysis Report Recommended Corrective Action Plan e
l t
Corrective Action Updates as of January 29,1997 i
7.
Training on fasteners - ECD has been extended to March 21,1997.
Lesson Plans are currently being developed.
9.
UNT-005-015 clarification - ECD has been extended to February 15, 1997. Procedure is in review / approval process.
10.
Parts verification - ECD has been extended to February 28,1997.
11.
VETIP update - This action has been completed.
12.
Worker counseling - This action has been completed.
13.
Fastener NWT - Team has been meeting, ECD on final results is February 28,1997.
14.
UNT-005-002 clarification - ECD has been extended to February 15, 1997. Procedure is in review / approval process.
15.
Zone Ownership Program - This action has been completed.
19.
MOV internal inspections -- This action has been completed. Identified discrepancies were documented on CR-96-1827.
i i
i