ML20237F239
| ML20237F239 | |
| Person / Time | |
|---|---|
| Site: | Rancho Seco |
| Issue date: | 12/17/1987 |
| From: | Coward G SACRAMENTO MUNICIPAL UTILITY DISTRICT |
| To: | Miraglia F Office of Nuclear Reactor Regulation |
| References | |
| AGM-TA-87-281, NUDOCS 8712290465 | |
| Download: ML20237F239 (143) | |
Text
-_
(VfSMun SACRAMENTO MUNICtPAL UTILITY DISTRICT O P. O. Box 15830, Sacramento CA 95852-1830,(916) 452-3211 AN ELECTRIC SYSTEM SERVING THE HEART OF CAllFORNIA DEC 17 : 1987 AGM/TA 87-281' L V. S. Nuclear Regulatory Commission Attn: Frank J.- Miraglia, Jr.
. Associate Director for Projects
~Philips Bldg.
7920 Norfolk Avenue
' Bethesda, MD. 20014 DOCKET.50-312 RANCHO SECO NUCLEAR GENERATING STATION LICENSE NO. DPR-54 CABLE DISCREPANCIES / ROOT CAUSE INVESTIGATIONS OF THE CABLE ISSUES
Reference:
SMUD Letter, Bob Croley to Frank J. Miraglia Jr., dated November 9, _1987
Dear Mr. Miraglia:
Please find attached the following investigations:-
" Appendix "R" Deficiencies"'
" Redundant Cabling In The Same Fire Area"
" Pulled Cables Stored In' Safety-Related Breaker Cubicles" !
" Cable Raceway Tracking System"
" Overweight Cable Trays"
" Unacceptable Intermixing Of Power / Control And Instrumentation Cables" These attached documents provide more detail, as requested by your Staff, than
'the summary of the investigations previously submitted with the referenced letter, j
8712290465 871217 ;
~ RANCHO SECO NUCLEAR GENERATING STATION O 14440 Twin Cities Road, Herald, CA 95638 9799;(209) 333-2935
)
l
- '=
., ya_-
EC 17 26/
AGM/TA'87-281 '
. Frank J. Miraglia, Jr. i.
o Please contact me if you have any questions. Members of your Staff with
'.2- 1 . questions requiririg additional information or clarification may contact
- John Atwell at (209)- 333-2935, extension 4917.
Sincerely, .
r .
~
orge A. Coward Assistant General Manager,.
Technical::and Administrative Services Attachment
.cc: ~ G. Kalman, NRC, Bethesda A. D' Angelo, NRC, Rancho Seco J. B. Martin, NRC, Walnut Creek l
I
. it. .
. .I '
INCIDENT ANALYSIS- 87-04 (ODR 87-12, ODR 87-61)
(LER 87-02, LER 87-10)
APPENDIX "R" DEFICIENCIES EXECUTIVE
SUMMARY
Evaluations performed by Nuclear Engineering Fire Protection resulted in the identification of two 10 CFR 50, Appendix ." R" deficiencies. One is the~ Diesel Generator (GEA) Breaker S4A08 control scheme.
~
The breaker's protective trip circuit could be actuated in a fire. This deficiency was reported as LER 87-02. The second deficiency was in the Reactor Coolant System High Point Vent Valves. A fire could cause spurious. operation of the valves and the loss of reactor coolant. This deficiency.was reported as LER-87-10.
Both of these deficiencies appear to have been caused by human error during the design and ' review - stage of past Engineering Change Notices. The fact that these problems were discovered during the Appendix R review of present-documentation indicates that the past Appendix R program deficiencies are being searched for, found,' and corrected.
i l
L L
L.
c - ,
. (.
l .
1
\,:
\
/
i INCIDENT ANALYSIS 87-04 (ODR 87-12, ODR 87-61)
(LER 87-02, LER 87-10)
APPENDIX "R" DEFICIENCIES DESCRIPTION OF EVENTS Two recent evaluations by the Fire Protection Group resulted in the discovery of existing electrical ,
circuits which do not meet the requirements of 10 CFR 50, ,
S Appendix R. These Appendix R deficiencies were reported in Licensee Event Reports (LERs) 87-02 and 87-10.
One is the Diesel Generator (GEA) Breaker S4A08 control scheme. The breaker's protective trip circuit could be actuated in a fire. This deficiency was reported as LER 87-02.
5
v .
.y .s n , C
,;l' INCIDENT ANALYSIS 87-04 pc '% j f '1; 9 h. ,
w c
) The second deficiency was in the Reactor Coolant System va \
x'High Point. Vent Valves. A fire could cause spurious '(
'operatiori of these valves and the loss of reactor coolant. This deficiency was reported as LER 87-10.
c A
Deficiency 1 '
1
.A v
)^
$ LER 87-02 describes a problem in the centrol scheme of 4 Diesel Generator (GEA) Breaker S4A08.
t.
Specifically, the .s
. /n icontrol sciheme. of S4A08 is susceptible to a " hot short"
)
which would actuate the breaker's protective ' trip
- <, I
(
.\;j circuitry and trip the .. breaker. A " hot short" is a' cond'icion .in which, 1) a fire burns off the insulation of a deenergized cable, 2) an energized cable falls across the deen rgized cable, and 3). the energized cable q induces voltagre;into the deenergized cable. Since a loss of offsite power is assumed to occur at the same time as a Control Room fire, the failure of breaker S4A08 would cause the loss of the only Appendix R allowed source of
'i' on-site power for safe shutdown. !
J-
.This deficiency was discovered by the Fire Protection Group during the Appendix R assessment of the final electrical distribution system.
/ 1 i
L J
)
f . y . ,
.l p.
I t
1 t 1 *
. TYCIDENT ANALYSIS l
Deficieng1_R LER ' 87-10 describes an' Append.ix R deficiency associated with the Reactor Conlent" System (RCS) High Point Vents.
i ,5 The conductors for both High Po$nt' Vent' Valves (of a set)
+
, ; 'T ar e routed in the same cablas. Additionally, the cables are routed in the same trays or vias. Because this commonality is downstream of the fuses for the High Point Vent Valves, the spurious operation of both High Point Vent Valves, due to a fire, is a credible accident. The spurious activation would be caused by a fire-induced
" hot short" from another energized cable.
In order to prevent accidental actuation of the High Point Vent Valve circuits the fuses are normally pulled.
This " hot short" deficiency was discovered by the Fire Protaction Group during an engineering /aluation of the electr. cal drawings for the RCS High Point Vents.
5
_ - _ _ - _ _ _ _- - - -___-_____ D
INCIDENT ANALYSIS CAUSES The direct cause of both of these deficiencies appears to be human error that occurred during the design and review completed for previous Engineering Change Notices (ECNs).
The fact that these problems were discovered during the Appendix R review of present documentation indicates that past Appendix R program deficiencies are being searched for, found, and corrected.
l
1 ..
' INCIDENT ANALYSIS INVESTIGATION.
.The investigation focused on two issues: ,
o Issue A: Were- the' Appendix R deficiencies described in LER 87-02 and LER 87-10' caused by a change to an acceptable design?
o -Issue B: :Is there a programmatic cause , of these conditions, or were they separate, isolated cases?
Issue A Were the' Appendix'R deficiencies described in LER 87-02 and LER 87-10 caused by a change to an acceptable design?
To determine whether- the Appendix R deficiencies described in LERs 87-02 and 87-10 were caused b'y a change to an acceptable design, the Incident Analysis Group (IAG) reviewed previous revisions to print E-204, sheet 1, " Elementary Diagram, Diesel Generator System." Based on that review the " hot short" deficiency has existed
.___ - ______ _ __-_ _ D
INCIDENT ANALYSIS since 1980. In 1980, Engineering Change Notice (ECN)
A2187 incorporated isolation switch (43) into the control scheme for Diesel Generator (GEA) breaker S4A08. ECN A2187 incorporated the required fire protection changes into the Diesel Generator (GEA) breaker and Auxiliary Feedwater Pumps.
As part of this investigation, the IAG reviewed the elementary diagrams of the Auxiliary Feedwater Pumps for deficiencies similar to those described in these LERs, no additional deficiencies were found.
The following prints were reviewed as part of the investigation of the high point vents deficiency:
- 1. E-203, sheet 105, " Steam Generator E-205B Vent Solenoid Valves HV-20534 and HV-20580"
- 2. E-203, sheet 106, " Pressurizer High Point Vent Solenoids HV-21528 and HV-21522"
- 3. E-203, sheet 107, " Steam Generator E-205A Vent Solenoids Valves HV-20533 and HV-20579" l
J
INCIDENT ANALYSIS The review of these prints indicates that the wiring commonality deficiency affects all three sets of the vent valves and has existed since 1983, when the High Point Vents were installed by ECN A-2934. Based on our review, the IAG concluded that the design deficiencies were inherent to the ECNs which effected the changes.
Issue B Is there a programmatic cause of these deficiencies, or were they separate, isolated cases?
As part of the determination as to whether there is a programmatic cause of these conditions or are these separate, isolated cases, the IAG reviewed ECN A2187
" Diesel Generator and Auxiliary Feedwater System Fire Protection Modification." ECN A2187 was reviewed in an attempt to determine if similar deficiencies existed.
Because the ECN review shows that the Auxiliary Feedwater Pump's circuitry had been properly designed for isolation from the Control Room, the IAG concludes that the Diesel Generator (GEA) breaker hot short deficiency is an isolated case.
INCIDENT ANALYSIS A review of ECN 7934 "High Point Vent Valves," revealed that all three sets of High Point Vent Valves exhibit the same deficiency and were not analyzed for an Appendix R event by the ECN. The Appendix R Evaluation for the High Point Vents, dated February 1985, states "RCS Vents are not considered in this report because their breakers are pulled during powered operation." The possibility of a hot short from an energized cable, during an Appendix R event, should have been, but was not considered.
Since both of these incidents have their basis in the failure to consider a hot short event, the IAG concluded that a random sample of Appendix R circuits should be reviewed for credible hot short events.
END l
I L-_____ _ )
4 ROOT CAUSE 86-10 REDUNDANT CABLING IN THE SAME FIRE AREA Based on LER 86-10 EXECUTIVE
SUMMARY
On May 22, 1986, two Engineering Department engineers discovered a cable routing error in Fire Area 17. The engineers found an empty conduit while looking for additional conduit space into the Control Room. The Cable Raceway Tracking System (CRTS) indicated that the conduit contained seven cables. A walkdown of the cable routing resulted in the discovery that the cables were in non-firewrapped conduit instead of firewrapped conduit.
The misrouted cables contained seven safe shutdown circuits that were routed from the Control Room to the Shutdown Panel through an isolation cabinet.
Construction Engineering had pulled the cables in accordance with the original routing; however, Nuclear Engineering revised the cable routing twice.
Construction Engineering did not pull back and repull the cables in agreement with the revised routings.
The direct cause of the failure to reroute the cables is personnel error. Although the Construction field engineer was aware of the routing revisions, he did not initiate the field installation of the "C" revision of the cable routing.
The underlying cause is that the Card Control Group (CCG) did not follow Engineering and Inspection Instruction (EII) EC-10 " Processing of Installation Cards."
A contributing cause of this event was Quality Control's failure to perform adequate inspections of cable installations.
The root cause is the failure of the Card Control Group cognizant engineer to implement adequate managerial !
controls. He did not require the use of adequate and formal procedures for the processing of installation cards by the CCG.
i l
DESCRIPTION On ' May 22, 1986, two Engineering Department engineers discovered a cable routing error in Fire Area 17. The engineers found an empty conduit while looking for additional conduit space into the Control Room. The Cable Raceway Tracking System-(CRTS) indicated that the conduit contained seven cables. A walkdown of the cable
. routing resulted in the discovery that the cables were in non-firewrapped conduit instead of firewrapped conduit.
The misrouted cables contained seven safe shutdown circuits that were routed from the Control Room to the Shutdown Panel through an isolation cabinet.
Construction Engineering had pulled the cables in accordance with the original routing; however, Nuclear Engineering revised the cable routing twice.
Construction Engineering did not pull back and repull the cables in agreement with the revised routings.
l CONCLUSION l
The~ direct cause of the failure to reroute the cables is
.i personnel error. Although the Construction field i l
i l
l ROOT CAUSE INVESTIGATION 86-10 l
engineer was' aware of the routing revisions, he did not initiate the field installation of the "C" revision of the cable-routing.
The underlying cause - is that the CCG did not follow Engineering and Inspection Instruction (EII) EC-10
" Processing of Installation Cards."
A contributing cause of this event was Quality Control's failure- to perform adequate inspections of cable installations.
The root cause is the failure of the Card Control Group cognizant engineer to implement adequate managerial controls. He did not require the use of adequate and
, formal procedures for the processing of installation cards by the CCG.
INVESTIGATION
'In its investigation of the cable routing error, the Incident Analysis Group (IAG) examined:
o The processing of installation cards
ROOT CAUSE 2: INVESTIGATION 86--10 o The Engineering Change Notice (ECN) for the seven circuits, ECN A-4942 o The cable routing revisions o The failure to reroute the cables o The submission of the revision "C" pull cards o The discovery of the Appendix R violations o Additional issues found in the course of the investigation.
o Why didn't the Construction Quality Control Group find the routing errors?
o Were signatures transcribed on the "C" Revision pull cards?
o Why are the CRTS generated installation cards and transmittal sheets for "B" &
"C" routing revisions to ECN A-4942 missing?
The processina of installation cards Nuclear Engineering uses installation cards to track cables, raceways, and conduits at Rancho Seco. There are three types of installation cards:
o Equipment / raceway
ROOT CAUSE INVESTIGATION 86-10 o Cable pull i o Cable termination (From and To).
The CRTS group generates all three types of cards and sends them to the Card Control Group (CCG) in the Modification Department.
l' The CCG controls the flow of the installation cards to and among the different construction groups involved in the installation and inspection of cables and raceways.
Procedure EII number EC-10 " Processing Of Installation Cards" was in effect from 1982 to 1986; however, the CCG did not use EC-10, or Figure 3 of EC-10 to process the cards. Instead of using EC-10 they processed installation cards in accordance with a flow chart j developed by the CCG engineering aide.
{
In 1981, the CCG engineering aide replaced the CCG ;
clerk. This changeout was performed with no turnover.
Due to the lack of a proper turnover, lack of training on the CCG process, and his perceived lack of procedures, he developed his own procedure for the processing of installation cards.
l I
4 ROOT CAUSE INVESTIGATION 86-10 The supervision provided the CCG engineering aide was of a general nature and did not include review of actual work . performance, nor did it include instruction on p'erformance of the CCG tasks. '
As previously stated, the CCG Engineering' .de did not '
know that procedure. EC-10 existed to control the
- processing of installation cards. Due to his lack of knowledge of ' the CCG process and not ' knowing of the existence of procedure EC-10, during the 1984/5 outage he- processed installation cards contrary to .the procedural requirements. Some of the deficiencies in processing of the cards were:
- 1. The CCG stored the pull cards until the Construction field engineers requested the CCG to send .them the cards. This practice was contrary to procedure EC-10 in that EC-10 required the CCG to inspect the cards and then send them to the Construction engineer for transmittal to Nuclear Power Services (NPS) c . .
ROOT CAUSE INVESTIGATION 86-10
- 2. After NPS ' pulled the cable, the Construction field engineers put the pull cards into a hold file instead of returning the pull cards to the CCG. This practice was contrary to procedure EC-10 which required' the Construction field engineer to return the pul?
cards to the CCG upon completion of the work.
In early 1986, Nuclear Engineering issued procedure NEP 6307 " Cable Installation." This procedure replaced
" Construction Methods and Procedures" and the EIIs related to cable installation and inspection. In December, 1986, the NEP-6300 series of - construction practices and procedures were converted to Modification Procedures / Inspection Standards (MP/IS). NEP 6307 became MP/IS 307. MP/IS 307 does not address QC cable
, pull inspections and approval in a manner such that it
.would assure that the inspections would be complete and timely. Additionally, the processing of installation cards is not addressed in MP/IS 307. These deficiencies have yet to be corrected.
ROOT CAUSE INVESTIGATION 86-10 The Encineerina Chance Notice for the seven circuits.
ECN A-4942 The 1984/1985 outage at Rancho Seco involved the installation of a significant number of cables and cable rerouting. During this period, Construction Engineering pulled roughly 292,000 feet of cable.
In order to assist in the processing of this large number of cables, and the reroutings, Nuclear Engineering issued Engineering Change Notice (ECN) A-4985 " Cable Pull." This ECN allowed all the cables to be pulled under one ECN. This practice has been (erroneously) referred to as " bulk" pulling of cable.
The cable terminations were accomplished under the ECNs associated with the individual circuits. It appears that if this process had been performed in conjunction with procedure EC-10 the misrouting would not have occurred.
In May 1984, Nuclear Engineering issued ECN A-4942 to isolate the Shutdown Panel (H2SD) from the Control Room.
This change was based on an analysis performed in i
l
E ROOT CAUSE INVESTIGATION 86-10 accordance with 10 CFR 50, Appendix R. Job number 4942 of ECN A-4895, routed seven circuits from the Shutdown Panel (H2SD) to isolation cabinet H4SIA, and from cabinet H4SIA to cabinet H4 CAL in the Control Room.
The cable routina revisions During the 1984/5 outage Nuclear Engineering revised the cable routing for these seven circuits three times. Two of the revisions occurred after Construction Engineering had already pulled and coiled the cables in agreement with revision "A" of Drawing Change Notices (DCNs) for ECN A-4942. Revision "A" involved only a change in the type of cable used, the cable code, and did not change the routing of the circuits.
In 1984/5 Nuclear Engineering reviewed all the Appendix R calculations. As a result of this review Nuclear Engineering issued revision "B" to the routing of the seven cables of interest in October 1984. The revision involved the partial pull back of the cables from both cabinet H4 CAL and the Shutdown Panel. After the pullback of the cables, one set was to be repulled through firewrapped conduit and the other cables were to be routed through a different fire area.
ROOT CAUSE INVESTIGATION 86-10 For Fire Area 17, the rerouting involved the removal of the cables from non-fire wrapped conduit and the pulling of the cables through fire-wrapped conduit X44154. The failure to perform this rerouting resulted in LER 86-10.
For Fire Area 36, the rerouting involved the removal of the cables from Fire Area 36 and the pulling of the cables through conduit X43318 in Fire Area 31. The failure to perform this rerouting resulted in LER 87-13.
In November 1984, Nuclear Engineering again changed the routing and issued routing revision "C." This revision changed two cable tray links in the cable routing from cabinet H4SIA to cabinet H4 CAL.
l The failure to reroute the cables Although the working documents (DCNs - pink 1010s) for revisions "B" and "C" had been issued, Construction l
Engineering did not reroute the cables.
It appears that the Construction field engineer did not request the revision "C" pull cards. His failure to j request the cards may have been caused by his belief ROOT CAUSE INVESTIGATION 86-10 that Nuclear Engineering intended to issue another revision. This supposition is _ supported by a_ note on the field engineer's pull tree about the issue of another revision.
On March 20, 1985, an Electrical Technician made the "from" termination for one of the cables. On March 21, 1985 the. Construction field engineer initialed the revision "A" pull card Supplemental Inspection Data Report (SIDRs). The field engineer's initials on the SIDR signified that he had verified that the pull card information was in agreement with the latest DCN.
l The submission of the revision ~"C" oull cards l
The sequence of events that resulted in the CCG submitting the revision "C" pull cards to the Construction Quality Control group cannot be determined; however, the following appears to have occurred:
\
l o On March 20, 1985, the CCG sent the Revision "A" pull cards to Construction Quality control 3 Group i
(
i
ROOT CAUSE INVESTIGATION 86-10 o A Construction Quality Control Inspector returned the revision "A" pull cards to the Construction field engineer after finding that the pull cards did not match the latest revision of the E-1010 o The Construction field engineer obtained hand typed revision "C" pull cards from the CCG o Someone in the CCG printed the name of the NPS foreman and the date of the completion of the revision "A" cable pull on the revision "C" pull cards o The construction field engineer initialed the revision "C" pull card SIDRs (the back of the pull cards) on March 21, 1985 o The Construction field engineer sent the revision "C" pull cards to the Construction Quality Control Group.
- - _ _ _ _ _ _ _ _ _ - _ - _ - _ - _ _ _ _ _ _ _ _ - _ _ __ _ A
1 ROOT CAUSE INVESTIGATION 86-10 I
The NPS foreman's name and the date. of the original cable pull on the revision "C" pull cards made it appear that the cables had been repulled in agreement with revision "C."
It could not be determined if the transfer of the printed name and completion date was a mistake resulting from the erroneous belief that NPS had repulled the cable or as .a deliberate falsification of records.
However, the transcribing of both information and names on installation cards was routinely done by NPS and Card Control Group personnel. Transcribing information from cards issued to the field to those to be retained by the CCG is addressed in a later section of this report.
On March 27, 1985, a Construction Quality Control Group Electrical Inspector inspected the cable installation for cabinet' H4SIA and cabinet H4 CAL and verified that the cable routing was in agreement with revision "C."
On April 1, 1985, another Electrical Inspector verified that the routing between the Shutdown panel and cabinet HASIA was in agreement with revision'"C."
1 i
7 ROOT CAUSE INVESTIGATION 86-10 During this period, the Construction Quality control Group . (CQCG) inspected cable pulls after completion of the termination of the cables. This occurred because they-were not aware of the work in progress. They did l
not receive information about the work until after the work had been completed. The practice of performing.QC inspections of the cables after completion of the termination violated a commitment made in the USAR to inspect the cable pulls before termination. ~ This commitment is in USAR section 8.2.2.11.H.14.c. A later section of ' this report discusses the failure of the Electrical inspectors in the Construction Quality Control Group to find the routing errors.
The discovery of the Aopendix R violations On May 22, 1986, two engineers discovered the routing error between isolation cabinet H4SIA and cabinet H4 CAL.
They found this error while looking for available space in Appendix R conduits into the Control Room. They found _ conduit X44154 empty and checked the CRTS data base. The CRTS data base indicated that conduit X44154 contained seven cables.
ROOT CAUSE INVESTIGATION 86-10 On February 4, 1987', the cable routing error between isolation cabinet H4SIA and the Shutdown Panel was discovered by contract personnel performing cable
.walkdowns. These walkdowns resulted from the original cabling error discovered May'22, 1986 in Fire Area 17.
SUMMARY
l^
Based on the-above, the IAG can state that:
o From 1982 to 1986,. the Construction Card Control Group (CCG) did not use procedure EII no. EC-10 " Processing of Installation Cards" l
to process installation cards o The CCG cognizant engineer did not provide adequate supervision to the CCG o The cognizant engineer did not assure that the CCG received. training on its work assignment
- j. o The cognizant engineer did not ensure that the CCG used approved procedures to process installation cards
ROOT CAUSE INVESTIGATION 86-10 o The cognizant engineer allowed the CCG to develop informal procedures for processing of installation cards. Additionally, these procedores were modified during the 1984/1985 outage j o Nuclear Engineering did not issue a procedure for the processing of installation cards when the NEPs replaced the Engineering and Inspection Instructions in 1986 o During the 1984/1985 outage, the Construction Quality Control inspections of cable pulls occurred after cable termination, but the USAR (section 8.2.2.11.H.14.c) required the QC verification be performed before the pulls i
were complete.
o Although the cognizant field engineer was aware of revision "C", he did not transmit the revision "C" pull cards to NPS
}
l
.i
l ROOT CAUSE INVESTIGATION 86-10 i
o The date of the original (revision "A") cable pull and the name of the NPS foreman who performed the work were transcribed onto the
-hand typed revision "C" pull cards. Because this information is recorded on the "C" pull cards it appeared that NPS had repulled the cables o Both the CCG and NPS personnel' routinely transcribed the NPS electrical foreman's printed name and the work completion dates on pull cards o The CRTS-generated pull and termination cards for the "B" and "C" routing revisions are missing. Additionally, there are no transmittal sheets for the transfer of those i cards between the CCG and the CRTS group, nor are there transmittal sheets for the transfer of the cards between the CCG and NPS.
ROOT CAUSE INVESTIGATION 86-10 Additional issues found in the course of the investigation o Why didn't the Construction Quality Control Group find the routing errors?
o Were signatures transcribed on the "C" Revision pull cards?
o Why are the CRTS generated installation cards and transmittal sheets for "B" & "C" routing revisions to ECN A-4942 missing?
Why didn't the Construction Ouality Control Group find the routina errors?
During the 1984/1985 outage, the Construction Quality Control Group inspectors did not always verify the complete cable routing to be in agreement with the routing drawings.
l' The Construction QC Coordinator was unaware that during the 1984/1985 outage:
i i
l i
L -_ _ _ - - _ _ _ _ _
4 -
ROOT CAUSE INVESTIGATION 86-10 o the Electrical QC Inspectors did not, in every case, verify the complete routing of cables o the CCG notified the QC inspectors to inspect pulled cables after cable terminations were made o .the Lead Electrical QC Inspector directed the Electrical -QC Inspectors to inspect cable routing "as far as possible from each end."
During the 1984/1985 outage, the CCG changed the processing of installation cards. The changes in the 1984/1985 outage resulted in the Construction Quality Control inspecting cable pulls after the cable had been pulled and terminated.
One of the changes in the 1984/1985 outage was that the field engineers stored the pull cards after pulling the i
cable. These cards were stored until a cable termination was made. After the cable termination was made the field engineer sent the pull cards to the CCG l
for transmittal to the Construction Quality Control l Group. This resequencing and storage of the pull cards
A ROOT CAUSE INVESTIGATION 36-10 I
resulted in the CCG notifying the Construction Quality {
Control Group to inspect pulled cable after the' pull was completed.
i The Electrical Construction Quality Control Inspectors were aware of the difficulty in verifying cable routing unless they were' present during the cable pull. The inspectors discussed this' problem with the Lead Electrical QC Inspector. He directed the inspectors to verify-routings "as far back as possible from each end."
It could not be determined when this direction was given to the inspectors.
1 Attachment 2 to procedure EII no. EC-11 " Cable During Installation Construction And Modification" required the inspection of cables as follows:
- 1. The QC inspector was to compare the pull cards !
against the latest DCNs and returned the cards to the field engineer if there was any .j discrepancy :
I j
i
. .. . _ _.____________._.____._________.______J
f t
)
ROOT CAUSE INVESTIGATION 86-10
- 2. The QC inspector was to verify:
" ... that cable size, number of conductors and routing are in accordance with drawings..." ,
and that
... the cable is present at correct "To and From" ends..."
Based on the above, many of the Electrical QC Inspectors interpreted that they were not required to witness a cable pull unless a tension measurina device was used.
I ECN A-4942 involved two cable pulls. One cable pull was from the Shutdown Panel to cabinet H4SIA, and the other cable pull was from cabinet H4SIA to cabinet H4 CAL in the control Room. In July 1984, NPS pulled the cables, and in March 1985, the CCG notified the Construction Quality Control Group to inspect the cables. The two 1 l cable pulls were inspected by different inspectors; l
l neither inspector found a routing error.
l l j
bW f t(
, ROOT CAUSE INVESTIGATION 86-10.
The Construction QC Coordinator stated that he was not aware that the Electrical QC Inspectors did not always verify the complete routing of cables. He stated that he believed the inspectors were verifying the complete cable routing by one of three methods:
V 4
% o witnessing the cable pull o the checking of the equipment used in the pull o walking down the cables after completion of the cable pull.
The Construction Quality Control Coordinator was not aware of the cable' inspection difficulties or that the inspectors were not always inspecting the entire cable !
routing. '
o-(y e Were sianatures transcribed on the "C" Revision Dull q
, cards?
O The printed name of the NPS foreman who supervised the l
-) revision "A" cable pulls and the date of the revision "A" cable pulls was transcribed to the revision "C" pull cards.
l 1
m .
ROOT CAUSE INVESTIGATION 86-10 Procedure EC-lO required the signature of the person completing the work. The NPS foreman printed his own name. Because the NPS foreman printed rather than signing the cards, the transcription by the NPS clerk and CCG personnel was not noticed.
The revision "C" pull cards had the name of the NPS electrical foreman in charge of the revision "A" cable installation of July 22, 1984 transcribed to them. The transcription of this information made it appear that the cables had been repulled in accordance with revision "C" when in fact they had not.
The Construction field engineer for the ECN A-4942 cables stated that he does not recall requesting the revision "C" pull cards and that he is not aware of how the NPS foreman's name and the revision "A" pull date were transferred to the revision "C" cards.
The routine transcribing of pull card information by the CCG created a situation where it was possible for the revision "A" pull card information to be mistakenly transcribed to the revision "C" pull cards.
O ROOT CAUSE INVESTIGATION 86-10 The direct cause of the transcribing of the revision "A" information to the ievision "C" pull cards cannot be determined at this point.
Why are the CRTS cenerated installation cards and transmittal sheets for "B" & "C" routino revisions to ECN A-4942 missina?
The IAG investigation was not able to determine the direct cause of the loss of the "B" and "C" cards.
However, the underlying cause of the loss of the cards is the failure to follow procedure EII no. EC-10
" Processing of Installation Cards." The CCG did not use transmittal sheets between the CCG and field installation personnel, and the CRTS group did not use transmittal sheets between the CRTS and the CCG.
Additionally, the CCG stored the installation cards instead of sending the cards to the cognizant field engineers as required by the procedure.
In the 1984/1985 outage, the lack of adequate l
installation card control resulted in " missing" CRTS generated pull and termination cards. It appears that 1
~
6 '
ROCT CAUSE INVESTIGATION 86-10 the revision "B" and"C" installation cards were lost after being sent to the CCG. The pull cards for these revisions were not transmitted to the field. The exact point of breakdown in the . pull card flow cannot be-determined. What is clear is that neither the CCG nor the CRTS group used transmittal documents between the CRTS and the CCG or between the CCG and the field.
Because the CCG was unaware of procedure EII No. EC-10
" Processing of Installation Cards," they established
' methods for the processing and control of installation cards.
In January 1986, NEPs replaced the EII Procedures, but an installation card processing procedure was not included in the replacement. As of June 1987, there is no procedure for the processing and control of installation cards. ,
I END I
ROOT CAUSE 87-02 PULLED CABLES STORED IN SAFETY-RELATED BREAKER CUBICLES Based on Licensee Event Report 87-16 EXECUTIVE
SUMMARY
On February 5, 1987, Electrical Maintenance personnel were working on Engineering Change Notice (ECN) R-0415B, " Installation Of Synch Check Relays On All Four (4) Diesel Generators And Four (4) Normal Supply Breakers." While performing the ECN, ,
Electrical Maintenance determined that the lock-out relays in two ;
safety-related cabinets could be inadvertently actuated if hit by cables coiled in the same cabinet.
On February 6, 1987, Nuclear Engineering determined that the presence of these coiled cables invalidated the seismic analysis for the cabinets. Nuclear Operations and Electrical Maintenance checked all the safety-related electrical cabinets and found cables coiled in four additional cabinets.
The direct cause of the coiled cables being left in all the cabinets, other than A405 and A409, is a planning error: the cable terminations should have been scheduled to occur immediately following the cable pull, but were not.
The direct cause of the coiled cables being left in cabinets 4A05 and 4A09 is a design error: the effects of the coiled spare cables on the seismic analysis were not considered.
The Root Cause of cables being left coiled and unrestrained in safety-related breaker cubicle cabinets is inadequate procedures for the installation of electrical cables.
ROOT CAUSE 87-02 DESCRIPTION
]
l l
On February 5, 1987, Electrical Maintenance personnel were ;
working on Engineering Change Notice (ECN) R-0415B, " Installation Of Synch Check Relays On All Four (4) Diesel Generators And Four (4) Normal Supply Breakers." While performing the ECN, Electrical Maintenance determined that the lock-out relays in two safety-related cabinets could be inadvertently actuated if struck by cables coiled in the same cabinet. This problem was reported via Occurrence Description Report (ODR)87-134.
On February 6, 1987, Nuclear Engineering determined that the presence of these coiled cables invalidated the seismic analysis for the cabinets. Nuclear Operations and Electrical Maintenance checked all the safety-related electrical cabinets and found cables coiled in four additional cabinets.
In accordance with 10 CFR 50.72 (2) (iii) (B), the NRC was notified immediately via the red phone and LER 87-16 was submitted.
i l
l t
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ - _ _ _ _ _ _J
1 L .
1 .
l ROOT CAUSE 87-02 CONCLUSIONS The direct cause of the coiled cables being left in all the cabinets,. other than A405 and A409, is a planning error: the cable terminations should have been scheduled to occur' immediately following the cable pull, but were not. .;
I.
The direct cause of the coiled cables being left in cabinets 4A05 and 4A09 is a design error: the April 22, 1983, Nuclear Engineering memo allowed . spared cables to end (coiled and unterminated) in equipment . without reviewing the effect of ~ the cables.on the. seismic analysis on the equipment.
The Root Cause of cables being left coiled and unrestrained in safety-related breaker cubicle cabinets is inadequate procedures for the installation of electrical cables. l e
l i
4 1
i i
e ROOT CAUSE 87-02 ,
l INVESTIGATION '
The Incident Analysis Group's investigation included:
o Consequences of the coiled cables o Cable installation practices o Plant procedures and the practice of storing coiled cables in equipment o Cables pulled and coiled in cabinets 4A08, 4A11, 4B01, and 4B11 o Cables declared as spares and coiled in cabi scs 4A05 and 4A09 o Commitments in Licensee Event Report 87-16.
k Consequences of the Colled Cables On February 5, 1987, Electrical Maintenance personnel were working on Engineering Change Notice (ECN) R-0415B, " Installation Of Synch Check Relays On All Four (4) Diesel Generators And Four (4) Normal Supply Breakers." While performing the ECN, Electrical Maintenance determined that the lock-out relays in two safety-related cabinets could be inadvertently actuated if hit by
l v
.i ROOT.CAUSE 87-02
, cables coiled in the same cabinet. This problem was reported via Occurrence Description Report (ODR)87-134.
On . February 6, 1987, Nuclear. Engineering determined that the presence of these coiled cables invalidated the seismic analysis for the cabinets. Nuclear Operations and Electrical Maintenance checked all the safety-related electrical cabinets and found cables coiled in four cabinets.
In - three of ' the affected cabinets - (4B01, 4B11, and 4A09) , . the breaker lock-out relays (the 486-2 relay) do not have a protective cover.
Attachment.1 is a copy of three photographs of the cables coiled in cabinet 4B01 and cabinet 4B11. In a seismic event, these coils of cables could have moved and struck the unprotected relays. The relays could have been inadvertently actuated if they had been struck by the cables.
Inadvertent actuation of the lock-out relays in cabinets 4B01 and 4B11 would have resulted in the loss of both normal and emergency power to the S4B bus. Actuation of the lock-out relay in cabinet A409 would have resulted in a loss of power to the S3A bus.
4 ROOT CAUSE 87-02 The cables found coiled in cabinets 4B01 and 4B11 were pulled
- into the cabinets on January 13, 1987. At the time these cables were discovered, they had been left coiled in the cabinets for twenty-four days.
The cable found coiled in cabinet 4A09 was pulled into the cabinet on December 10, 1982. At the time this cable was discovered it had been left coiled in the cabinet as a spare cable for four years.
In summary, the safety-related electrical cabinets that were affected by this event are:
o Cabinet 4B01: the Startup Transformer #2 connection to the S4B bus o Cabinet 4B11: the Diesel Generator GEB connection to the S4B bus o Cabinet 4A09: the S4B bus connection' to the :
Station Service transformer, X 43A (feed to the S3A bus) o Cabinet 4A05: the S4A bus connection to the DHS 1 Pump 261A o Cabinet 4A08: the Diesel Generator GEA connection to the S4A bus o Cabinet 4A11: electrical protection relays for the S4A bus.
1
_6_ 1 I
1
. - . - ________ )
4 ROOT CAUSE 87-02 Cable Installation Practices Typically, electrical cables are pulled and terminated by the combined efforts of the Construction and Electrical Maintenance.
Construction personnel, who work for the Modifications Department, pull, tie down, form and lug the cables. The Electrical Maintenance personnel terminate (land) the cables. In some cases, Electrical Maintenance personnel also tie down, form, and lug cables. This division of effort is common industry practice.
The coordination of tasks between the groups is not addressed in Plant procedures. Procedure AP.44, " Plant Modifications - ECN Implementation" should coordinate the efforts of the two groups, but does not. As an example, in January 1987, the lack of an adequate procedure resulted in safety-related cables not being scheduled for termination until several weeks after the cable pull.
Plant nrocedures and the Dractice of storina coiled cables in eauipment l
1 Procedure EM.187, " Control of Electrical Plant Modifications" allows the cables to be pulled into, coiled, and stored in plant I
e ROOT CAUSE 87-02 equipment. Procedure EM.187 also states that cables may not be terminated . . .unless the plant is in cold shutdown . . . .
" The phraseology of EM.187 encourages the pulling of cable into equipment with the termination to occur after Plant shutdown.
Procedures MP/IS 307 " Cable Installation, MP/IS 311 " Low Voltage Cable Terminations and Splices", and EM.163 " Installation of Permanent Plant Cables" are used to install and terminate cables.
Until recently, procedures MP/IS 307 and 311 referred to procedure EM.187.
None of the procedures for cable installation or termination address the condition that exists when cables are pulled into a cabinet and not immediately terminated. Additionally, there are no guidelines for what types of equipment may have cables stored in them until termination.
The construction procedure in effect from 1982 to 1986, Construction Methods & Practices, " Cable Installation" allowed the coiling of a spare cable in plant equipment. This practice is also addressed in an Engineering Department memo " Sparing Existing Plant Cables," dated April 22, 1983 (see attachment 2, Electrical Engineering memo of April 22, 1983). The memo also allows spared cables to end in equipment.
ROOT CAUSE 87-02 Neither the Construction Methods & Practices nor the April 22, 1983, Engineering Department memo address the class of equipment i
or seismic effect when cables are declared as spares. Because 1
the equipment class and the effect on seismic qualification were not proceduralized, cables were declared to be spares without regard to the class of the equipment or the effect of the cable on the seismic analysis of the equipment.
The Construction Methods & Practices procedures were replaced by Nuclear Engineering Procedures (NEPs) in 1986. These NEPs are now being replaced by Modification Procedures / Inspection Standards (MP/IS). However, because procedure MP/IS 307, " Cable I Installation" does not address the process by which cables are declared to be spares, the only guidance is in the April 22, 1983, memo.
Cables Pulled and Coiled in Cabinets 4A08. 4A11. 4B01 and 4B11 In January 1987, cables were pulled into and coiled in cabinets 4A08, 4A11, 4B01, and 4B11. These cables were part of ECNs R-1045 and R-415B. The Modifications Department pulled the cables for both ECNs into the equipment, and cable termination was scheduled to be performed by Electrical Maintenance during the i
ROOT CAUSE 87-02 Decay Heat. Outage in late March 1987. Leaving these cables in the cabinets was allowed by existing plant procedures.
The presence of- the coiled cables -invalidated the seismic analysis for the cabinets. In the case of ~ cabinets 4B01 and.
4B11, the-cables were in the cabinets for twenty-four days. The cables were in cabinets'4A08 for 30 days and in 4A11 for 24 days.
Cables Declared as Soares and Colled in Cabinets 4A05-and 4A09 The cables found in 4A05 and 4A09 had been declared spare cables and were left coiled in the equipment. The cable in 4A09 had been pulled Linto the cabinet in December 1982 and in'accordance
~
with existing plant procedures, was designated as a spare cable in July 1983. The Construction Methods and -Practices procedure <
" Cable ~ Installation" and the April 22, 1983,- memo, " Sparing
- Existing Plant - Cables" required the spare cable to be secured
- away from energized components; however, the memo also allowed the ending of the cable in equipment. When a cable was ended (coiled and not terminated) in active inservice equipment, the effect of the cable on the seismic analysis for the equipment was not addressed. )
I l.
q l
l l
..,l.
. .w.,........, . ~ . , , . . . . .. .. . . .. .. . . .
/
ROOT CAUSE 87-02 4
Commitments in LER 87-16 The District reported this event to the NRC via LER 87-16. LER 87-16 contained the following commitments:
- 1) The deficiency in the installation procedures for cables will be revised to directly address the intermediate configuration allowed while awaiting securing and termination as follows:
a) By March 23, 1987, MP/IS 307, " Cable Installation," will be revised to formalize the policy that cable pulls will be left secured outside of permanent plant equipment cubicles prior to final installation.
b) The following procedures will be revised to properly reference the new " Work Authorization &
Test" procedure (AP.90, to be issued by May 1, 1987):
i) MP/IS 311, " Low Voltage Cable Terminations and Splices," and; ii) EM.145, " Installation and Control of Temporary Electrical Cables," and; iii) EM.163, " Installation of Permanent Plant Cables," to give the Shift Supervisor a clear understanding of modification work being performed in permanent plant equipment.
l l
- 2. The distribution of responsibilities between the l construction forces and the electrical maintenance forces with respect to actually performing the desired terminations is acceptable. Beginning April 6, 1987, however, the planning group will ensure, via an explicit note in AP.44 (or successor procedures), that work requests for either cable:
a) securing, forming and lugging, or; b) termination i
are scheduled to follow immediately after work requests for the respective cable pull into the permanent plant ;
equipment cubicles.
l - __ _ _ _ _
4 l
ROOT CAUSE 87-02 Although the District's commitments prohibit the pulling of cable into equipment unless termination of the cable is to occur l immediately, these commitments conflict with procedure EM.187:
1 this procedure allows the practice of pulling cable without immediate termination.
ATTACHMENTS AND REFERENCES
- 1. Pictures of the cables found coiled in cabinets 4B01 and 4B12.
- 2. Electrical Engineering memo, " Sparing of Existing Plant Cables," dated April 22, 1983.
END
ROOT CAUSE 87-03 CABLE RACEWAY TRACKING SYSTEM EXECUTIVE
SUMMARY
As' a . result of Licensee Event Report 87-13 and several occurrence Description Reports related to the installation of cables at Rancho Seco, the Incident Analysis . ' Group initiated a review 'of the Rancho Seco Cable Raceway Tracking System (CRTS).. Although no specific cabling problems appear to have been directly-associated with or caused by the CRTS, there do appear to be significant indications of CRTS problems. These problems include:
o The NRC's questions about the quality of . the CRTS data base l
o Indications that the CRTS data base did not match the actual plant configuration o Allegations that the CRTS was not used correctly and that errors were being entered into the data base.
________________________j
(
i The Incident Analysis Group's investigation of the CRTS l confirms that significant problems did exist and that some problems continue to exist with the CRTS.
The root cause of the CRTS problems is that Nuclear Engineering management and the CRTS Supervisor were not adequately involved in the CRTS.
DESCRIPTION As a result of Licensee Event Report 87-13 and Occurrence Description Reports (ODR) related to the installation of cables at Rancho Seco, the Incident Analysis Group (IAG) initiated a review of the Rancho Seco Cable Raceway Tracking System (CRTS). Although no specific cabling problem appeared to have been directly associated with or caused by the CRTS, there did appear to be significant indications of CRTS problems. These problems include:
o The accuracy of the CRTS data base
)
I i
1 o Indications that the CRTS data base did not !
I match the actual plant configuration l l
o Allegations reporting that the CRTS was not used correctly and that errors were being entered into the data base.
)
1;.* ..\
ROOT CAUSE INVESTIGATION 87-03 CONCLUSION The root cause of the problems related-to the CRTS and its use is: That'neither Nuclear Engineering management or the CRTS Supervisor were - adequately involved in the
-CRTS.
INVESTIGATION The. investigation of the CRTS included examination of:
o -The development of the CRTS o Rancho Seco's cable raceway tracking computer software programs EE553 Transition from EE553 to CRTS
- 'The CRTS in 1980 through 1986 o The CRTS 1986 to 1987
- o. Procedures EE553 Procedures CRTS Procedures o CRTS supervision
ROOT CAUSE INVESTIGATION 87-03 The Development of the Cable Raceway Trackina System (CRTS)
As a part of the investigation the IAG reviewed the development of the CRTS that is summarized below.
Chronoloav 1977 The District decides to replace the cable tracking system (EE553) - used at Rancho Seco 1978 Control Data Corporation (CDC) receives contract to develop a cable tracking system 1978-1979 CRTS Project Engineer adds engineering design capabilities to the CDC cable tracking system 1980 CRTS replaces the Bechtel cable tracking system as the inventory of installed cables and raceways at Rancho Seco 1980-1981 The CDC Computer Analyst makes minor corrections to the CRTS software 1982 Nuclear Engineering begins to use the CRTS as an engineering tool l
)
1
- L .j s
4 i
l ROOT CAUSE INVESTIGATION 87-03 .
-1986 IBM PC/AT computers are interfaced with the CDC main frame computer 1987 The CRTS data base and software are transferred to an on-site computer
-The items listed in this chronology are described below.
During plant construction and until 1980, a Bechtel cable tracking system, EE553, was used to provide an inventory of installed cables, conduits and raceways at Rancho Seco.
In 1977, the District decided to replace the EE553 system. This decision was based on two reasons. First, the cost of EE553 reports was high: at that time,' the cost of updating the EE553 database and issuing a report exceeded $25,000. Second, EE553 was not used often: from 1975 to 1977 less than 50 cables had been installed at Rancho Seco.
In December 1977, the District issued a Request for Proposal (RFP) for the development of a cable tracking system. This RFP requested bidders to develop a cable tracking system that would:
_ _ _ _ _ _ _ _ _ _ _ _ - _ - - _ _ _ _ _ - a
ROOT CAUSE INVESTIGATION 87-03 o Provide a computer based inventory for the i installed cables and raceways at Rancho Seco o Include editing, updating, and reporting features o- Allow monthly updates of the data base files o Generate new cable and raceway reports at least every two months o Allow processing of data in a batch mode (batch processing).
In March 1978, the District accepted a proposal from Control Data Corporation (CDC) for the development of a cable tracking computer program. The cable tracking system was operated on a CDC CYBER computer which was used on a time share basis.
Development of the cable tracking software took almost two years, during which time the District Project Engineer expanded the scope of the cable tracking system:
working with the CDC System Analyst the Project Engineer added engineering design capabilities to CRTS. The l
t I
ROOT CAUSE INVESTIGATION 87-03 l intended purpose of the expanded scope was to ensure that L
the system could be used as an inventory system and as an engineering design tool.
Included among the engineering design capabilities that were added to the system were:
o An increased capability to check for overfilled raceway
- o. A check for Safeguard channel bridging o A check to verify that the cable raceways in a routing actually connected.
The RFP~ specifies that the CRTS was intended to maintain cable and raceway records at Rancho Seco. Documents related to using the CRTS as an engineering tool have not been found. Although there are no guidelines which establish the capabilities, functions, or design basis of the CRTS as a design tool, the CRTS has been and is being used as an engineering design tool. The CRTS is currently being used to check the design of Class 1E and non- Class 1E electrical cable routings.
9
~ ROOT CAUSE INVESTIGATION 87-03 In the- summer- of ~ 1980, CRTS replaced the EE553 as the- ,
- inventory of installed cables and raceways : at Rancho
. Seco.
Problems were identified in the CRTS data base and software in 1980 and 1981. The CRTS software problems were corrected by the CDC Computer Analyst. The CRTS data base problems were corrected by the District Project Engineer and the CDC Computer Analyst.
~
However, there' is little documentation of the changes which these individuals made to . the software and data base during this time frame.
In 1982, the method used to enter routing information into the' CRTS data base was changed. Before the change the CRTS staff entered routing information into the CRTS data base in the "as built" mode. After the change, information was entered into the data base in the
" proposed" mode.
Nuclear Engineering began to use the CRTS as an
- engineering tool for designing cable routings in 1982, at
. the time that it began entering routing information in l . the " proposed" mode rather than in the as-built mode.
l~
- l. lis. .. . r.. . .. .. . . . . . . . . . _ . . . . . _ . . _ . . . _ . . . . . . _ . _ _ _ . . . . _ . . . . . . . _ . . . . . . . . . . .. . . . . , , , . . . . , . . . . . .
,__,g .
l ROOT CAUSE INVESTIGATION 87-03 l In 1986, IBM PC/AT computers were interfaced with the J CYBER computer to provide increased flexibility in using the CRTS data base. 4 In 1987, the CRTS data base and software were transferred to an on-site supermini computer. Although the major reason for this change was to redace-operating costs, it also reduced the data processing time.
The CRTS system was designed, purchased, and has been operated outside of the Quality Assurance program.
Quality Assurance (QA) was not involved in the purchase or development of the CRTS. The lack of QA involvement with CRTS appears to be based on two points:
o The CRTS was designed to be an inventory of installed cables, raceways and equipment. As an inventory, the CRTS's use did not affect the quality of safety related systems o Although the District was purchasing a computer software program, the District's contract with CDC was a service contract.
Because the District-CDC contract was for
?: '.
L ROOT CAUSE INVESTIGATION 87-03 services rather than materials and the CRTS was non-Class 1, there was no requirement for a QA receipt inspection of purchased material.
Rancho Seco's Cable Raceway Trackina Computer Software Programs As a part of the investigation, the IAG reviewed the history of the computer programs used to track cables and raceways.
Chronoloav 1969-1974 The plant was under construction: Bechtel uses the EE553 for cable tracking and design checks 1974-1978 Early Plant operation: District uses the EE553 for cable tracking and design checks 1978-1980 CDC develops the CRTS: the EE553 data base is not updated.
1980-1982 CRTS is used for cable tracking: cable information is entered into the CRTS data base in the "as built" mode.
1982 1986 Cable . information is entered into CRTS
- _ -_______-_ ______ _ a
1 ROOT CAUSE INVESTIGATION 87-03 data' base in the " proposed" mode. CRTS is also used as a design tool.
l May/1986-1987 Cabling information is processed on personal computers and batch loaded to mainframe computer.
Bechtel's EE553 Procram During the construction of Rancho Seco from -1969 until
-1974, Bechtel controlled the installation and tracking of cables and raceways. Bechtel used the cable and raceway scheduling program EE553 to track both the cables and the raceways. Specifically, the EE553' program was designed to.
o Provide construction with reports that aided in the scheduling and installation of cable and raceway work o Provide a record of installed cables and raceways o Provide construction status reports.
' ROOT CAUSE INVESTIGATION 87-03 l Additionally, the supplemental (that is satellite) software program used the EE553 data base to provide reports and checks which included:
o Verification that circuits were not routed in l
more than one safeguards channel o Verification that there were no cable bridges between two or more safeguards channels.
o Checks on raceway fill After construction and until 1978, the Bechtel EE553 was used as the inventory of the installed cables and raceways at Rancho Seco. Bechtel retained control of the data base and the District provided the data changes.
I l
During the period 1974 to 1978, less than 50 cables were pulled.
From 1978 until the implementation of the CRTS in 1980, the District did not update the EE553 data base. This situation created a backlog of cable information that had to be entered into the database. This backlog was not resolved until late 1982.
l I
l
=
ROOT CAUSE INVESTIGATION 87-03 The Transition from EE553 to CRTS In 1980, the EE553 data base was transferred to the CRTS.
After the EE553/CRTS data base transfer, the CRTS was used to generate a 1,592 page error report. Most of these discrepancies resulted from a difference between how EE553 and CRTS recognized raceway sections. For example, the CRTS software required a point-to-point linkage of raceways, and the EE553 did not. In addition, many of the errors were carry-overs of errors that had existed in the EE553 data base since plant turnover.
Examples of these are:
o Cable trays filled more than 40%
o The intermixing of power and control cables with instrument cables.
Nuclear Engineering reviewed the data base error report in 1980. As a result of that review, the lack of manpower, and budget constraints, the Nuclear Engineering Supervising Electrical Engineer decided to postpone the resolution of these errors.
f ROOT CAUSE INVESTIGATION 87-03 When the District initiated a critical review of CRTS in 1986, the only available copy of-the error report was one the CDC Computer Analyst had kept for his files. In April 1987, the original error report was located in the Rancho Seco QA vault.
The CRTS 1980 - 1986 Information can be entered into the CRTS data base on either a " proposed" or an "as-built" mode. If information is entered into the CRTS data base on a
" proposed" basis, a CRTS-detected routing error will result in both an error message and the rejection of the input. However, if information is entered in the CRTS on an "as-built" basis, CRTS will generate an error message but will accept-the input into the data base.
From 1980 to early 1982, cable information was entered into the CRTS data base from the yellow " work complete" drawings. This information was entered in the "as built" mode. The two CRTS data input personnel from this period state that they entered data that resulted in error i flags. Because they did not have any procedural guidance on the error flags, they discussed the problem with their immediate supervisor. He instructed them to ignore the ROOT CAUSE INVESTIGATION 87-03 errors because the information was from the yellow I drawings which reflected the as-built condition of the plant. Therefore, between 1980 and early 1982, errors.
which were not corrected were entered into the CRTS data base.
Because the District had not updated the cabling data base since 1978, a backlog of about 2,000 cable entries existed in 1980. This backlog led to a- decision to
" force-feed" . the cable information into the CRTS, until _
the backlog was cleared.
By.early 1982, the CRTS staff completed the entry of the backlog of cable information and used the proposed mode of entry to begin entering cable information from pink
" working" documents. The use of the pink drawing for CRTS data entry proved unsatisfactory because the CRTS staff received the information about field-routing changes after the cable pull was complete. Because the cable pull was complete, the information was incorporated into the data base in the as-built mode. Use of this mode of entry hindered the use of the CRTS as an engineering tool. J l
1 i
_ _ _ - _ - _ . . _O
ROOT CAUSE INVESTIGATION 87-03 In late 1982, another change to the CRTS input j methodology was made. The CRTS staff began entering cable routing data from the white " proposed" documents, j Entering data from the proposed documents allowed CRTS to 1
check for cable tray overfills, cable intermixes, and l
routing errors before Construction Engineering pulled the cables.
When " proposed" cable and raceway information is entered into the CRTS data base, it is flagged as such in a status column. The " proposed" status flag is removed when the CRTS staff receives the completed cable and raceway installation card from Construction Engineering.
With the change in CRTS input methodology, the CRTS had the capability of checking the cable routing for errors before the cable was installed. CRTS " approval" became a j mandatory hold point for all cable routing DCNs. CRTS staff approval was required on all cable or raceway routings before Site Document Control (SDC) would issue a cable / raceway DCN. This requirement was implemented both by memo and word of mouth.
I ROOT CAUSE INVESTIGATION 87-03 The CRTS staff reviewed and approved all DCNs for:
o E-1008 (raceway routing) o E-1010 (cable routing)
- o. E-1026 (raceway codes) l l o E-1027 (cable codes) l o E-1028 (equipment installation drawings).
When problems were encountered, the CRTS staff dealt directly with the design engineers to resolve the problems.
This methodology remains in use today. However, until .i January, 1987, Field Engineers were able to bypass the CRTS system by making changes- to pull cards without i changing the CRTS data base.. Nuclear Engineering closed this loophole by . using Field Problem Reports and by requiring the use of comouter-orinted null cards to pull cable.
If the CRTS does not receive the completed cable and raceway installation cards from Construction, the
" proposed" status flag is not removed from the CRTS data base. Because of the missing cable and raceway installation cards, the " proposed" status field of the
______-_______A
ROOT CAUSE INVESTIGATION 87-03 l
l CRTS data base is not accurate. That is, there are closed ECNs with cables and raceway which have a proposed status in the CRTS data base.
The CRTS 1986 to 1987 In 1986, the District initiated a program to evaluate and correct the CRTS data base and to correct operating problems. The purpose of this program is to assure that the CRTS data base is accurate and that cable problems related to CRTS will not recur.
The key points of the program are the evaluation and resolution of CRTS data base and software problems.
o In April 1986, the District contracted with Impell Corporation (Task 241) to implement the program. In August 1986, Task 271 was added to the program. The changes made to date as a part of Task 241/271 allow more effective data manipulation and have enhanced the use of the CRTS as a design tool.
o The EE553/CRTS conversion error report was reviewed and the problems were documented. As
l l
ROOT CAUSE I!NESTIGATION 87-03 a result of this review 1,113 conversion problems (CONS) were identified.
o The CRTS data base was reviewed and the problems found were documented as CRTS Problem Reports (CPRs). As of May 1987, 932 CPRs have been identified. This review should have identified any errors resulting from entering data in the "as built" mode.
Steps to be completed will resolve the known Class I problems and non-Class I overfill and intermixing problems. The resolution of all the remaining known non-Class 1 problems has not been scheduled. The validation and verification of the CRTS software has been assigned as Action Item # 10 in the Rancho Seco Wire and Cable Program Report.
In 1986, the District contracted with CDC to purchase a l supermini computer and shift the operation of the CRTS to this on-site computer. In January 1987, the District i
transferred the CRTS database to the on-site computer and on February 6, 1987, the District began to use the new computer for the CRTS.
ROOT CAUSE INVESTIGATION 87-03 Procedures Procedure for use of the EE553 The 'EE553 is a proprietary Bechtel computer program.
Bechtel wrote and approved procedures for use of the EE553 system. Bechtel's procedures consist of:
o EE553 Users / Theoretical Manual o EE553 Programmers / Maintenance Manual o EE553 Validation Manual CRTS Procedures Until June 1987, procedures for the use of the CRTS did not exist. Nuclear Engineering Department's use of the CRTS was controlled by both memorandum and the CRTS User's Guide.
The original User's Guide was written by CDC in 1980. It did not contain sufficient information to provide an in-depth understanding of the CRTS. Further, it did not provide a clear explanation of the program's objectives or capabilities, and required a thorough understanding of l
ROOT CAUSE INVESTIGATION 87-03 the CDC CYBER operating system. Because the user's guide was inadequate, the CRTS users had to rely on verbal information from the CDC system analyst. A Rancho Seco CRTS User's Guide has never been formally approved.
Additionally, the original CRTS Use?'s Guide did not contain a detailed explanation of error messages. The lack of a detailed explanation of the error flags confused the CRTS personnel and resulted ic input errors during the first years of operation. In 1982, the CRTS input clerk requested and received an upgraded error message list from the CDC system analyst.
In 1984, the CDC-supplied CRTS User's Guide was revised by the CRTS data input clerk. This revision incorporated the upgraded error message list and information obtained from the CDC System Analyst. The revised User's Guide was not an approved procedure and received only limited distribution. One copy was used by the CRTS staff and one copy was given to Nuclear Engineering Department Electrical.
In January 1986, the CRTS data input clerk again revised the CRTS User's Guide. Although this second revision of l .
ROOT CAUSE INVESTIGATION 87-03 the User's Guide has not been approved, it is now used by the CRTS staff.
l As of May 1987, two new procedures have been written for 1 the CRTS. These are the " Engineers User's Guide" and Nuclear Engineering Administrative Procedure (NEAP) 4127
" Cable Raceway Tracking System." The Engineers User's ,
Guide was written as a desk top guide for the Nuclear Electrical Engineering staff as an overview of the operation of CRTS. Procedure NEAP-4127 addresses the document flow for the CRTS-related drawing changes and includes the mandatory CRTS review of Drawing Change Notices (DCNs) for E-1008, E-1010, E-1026, E-1027 and E-1028 drawings. Procedure NEAP-4127 was issued on June 15, 1987.
Although procedure NEAP-4127 includes the CRTS review of DCNs for certain electrical drawings, procedure NEP-4109
" Configuration Control" does not address this review. In l May 1985, an NED memorandum discussed this issue, but I corrective action was not taken.
l l
)
4
_ _ _ __.__________________________________j
e l 1
ROOT CAUSE INVESTIGATION 87-03 1
In May 1987, the Nuclear . Engineering Department l l
Electrical group held mandatory briefing sessions for all ~
Nuclear Engineering Department Electrical Engineers on:
o Procedure NEAP-4127 and the Engineers User's Guide o The interface between the Electrical Design Group and the CRTS staff.
l I
1 Attendees of the briefing were given copies of the Engineers User's Guide and a draft of procedure NEAP-4127.
CRTS Supervision During the period 1982 to 1986, there appear to be problems related to the supervision of the CRTS group.
These problems are demonstrated by the following:
o The CRTS engineering aide was the main point of contact with the CDC System Analyst o There was no procedure for the use of the CRTS l
4 ROOT CAUSE INVESTIGATION 87-03 o Known errors in the CRTS data base were not i corrected o Field engineers could bypass the CRTS system o The use of the CRTS as a DCN hold point was not included in the NEPs o The methodology for processing CRTS input was changed without documentation or the development of procedures o Adequate standards, policies and administrative controls were never implemented I
for the CRTS.
END 1
l l
l
_. _ __ _ _ _ _ ___ _ _ ____________ ______ _ _ u
!;'s /( ,
ROOT CAUSE 87-05 OVERWEIGHT CABLE TRAYS Based on LER 87-24 i i
EXECUTIVE
SUMMARY
On February 25, 1987 it was determined that up to seven cable trays may exceed the 50 pounds per linear foot limit of the USAR. This event was reported to the NRC .
via LER 87-24. None of these trays contained Class 1 l circuits.
Cable tray weights for all of the 3,459 cable trays at Rancho Seco were calculated. These preliminary calculations indicated no more than seven cable trays were potentially overweight.
Subsequent calculations determined that only one tray, 740BG1A, was overweight. In a final calculation, tray 740BG1A was calculated to weigh 50.35 lb/ft. This is overage is acceptable due to the conservatism of the calculation.
The direct cause of the overweight cable tray problem is inadequate procedural guidance.
The root cause of the overweight cable tray problem is the failure to ensure adequate procedural implementation of USAR requirements.
DESCRIPTION OF THE EVENT On February 25, 1987 it was determined that up to seven cable trays may exceed the 50 pounds per linear foot limit of the Updated Safety Analysis Report (USAR).
These potential overweight cable trays were reported to j the NRC via LER 87-24 Revision 0.
1 Although none of these trays contained Class 1 circuits, I
the supports for one of the trays also supported two )
l Class 1 trays. l Cable tray weight for all of the 3,459 cable trays at i
Rancho Seco were calculated. These preliminary I calculations indicated that seven cable trays were potentially overweight.
The District's commitment to limit the cable tray weight {
is found in USAR Section 5.1.2.1.8.C.1 " Structures and l
Containment System," which states: l I
l (a) Regardless of the cable tray or busduct function, all supports are designed to meet the requirements of seismic category I by dynamic analysis,...
i I
-ROOT CAUSE INVESTIGATION 87-05
~
(b) . Cable tray loading of 50 pounds per linear foot . is used throughout regardless of tray width or anticipated
. weight . of wire and cable. In no case does actual weight - of wire,4 cable and-tray exceed.this figure.
Subsequent calculations determined that only one ' tray, 740BG1A, was " overweight." The final' analysis of tray
.740BG1A indicates that it - may weigh as much as 50.35 lb/ft. Engineering. considers this value to be acceptable because the calculation is conservative.
CONCLUSIONS The exact limits for ' cable tray weight and fill are defined-in the Rancho Seco USAR. These limits are:
o 50 lb/ft for cable tray weight o 40% fill for cable trays associated with the Reactor Protection System and Safety Features Actuation System.
Electrical Engineering uses a fill-to-weight correlation to prevent exceeding these limits. Although the use of the fill-to-weight correlation appears acceptable; 19 ROOT CAUSE INVESTIGATION 87-05 Class 1E and 171 non-Class 1E cable trays were filled beyond the applicable fill limit without verification that the cable tray weight limits had not been exceeded.
The direct cause of the overweight-cable tray problem is inadequate procedural guidance'.
The root cause of the overweight cable tray problem is the failure to ensure adequate implementation of USAR
. requirements.
INVESTIGATION In this investigation the Independent Investigation /
Review Group (IIRG) examined:
o The USAR requirements o The fill-to-weight correlation o Ampacity concerns o Overfill concerns 1
o Applicable procedures j l
o cable and Raceway Tracking Systems. I j
ROOT CAUSE INVESTIGATION 87-05
-lhe USAR Requirements The USAR was examined to determine the design basis associated with cable tray fill' and weight. The USAR states that the maximum cable tray loading is 50 lb/ft and the maximum fill percentage is 40% for cable trays associated with the RPS - and SFAS. Specifically, USAR Chapter 5 " Structures and Containment System" section 5.1.2.1.8.C.1 states:
(a) Regardless of the cable tray or busduct function, all supports are
-designed to meet the requirements of seismic category I by dynamic analysis,...
(b) - Cable tray loading of 50 pounds per linear foot is used throughout regardless of _ tray width or anticipated weight of wire and cable. In no case.does actual weight of wire, cable and tray exceed this figure.
I Additionally USAR Chapter 8 " Electrical Systems" section 8.2.2.11, paragraph H, Item 9, states "the maximum percentage fill in redundant trays is 40%, and wherever possible, it is kept to a much lower value."
l _ _ _ _ _ _ __ _ ________________ _ __.____ _ i
F ',
ROOT CAUSE INVESTIGATION 87-05 The Fill-to-Weicht Correlation Based on calculations performed by Bechtel, a fill-to ,
weight correlation .was developed specific to Rancho Seco's cable population. This plant specific correlation was based on the assumption that as long as a cable tray is filled less than 40% the tray weight can not exceed 50 lb/ft. Engineering relied on this fill-to-weight correlation to ensure that the cable tray weight loading limit (USAR) of 50 lb/ft would not be exceeded.
Ampacity There-are two concerns associated with overfilled cable trays: ampacity and tray weight.
Ampacity affects the ability of a cable to carry the starting and full load current. As tray fill increases, the cables in that tray are able to carry less current.
Ampacity is of concern only for power and control trays.
Ampacity checks were performed on all power and control trays with fills greater than 40%, and no problems were found.
ROOT CAUSE INVESTIGATION 87-05 Overfill Concerns Engineering has controlled tray weight by a- fill-to-
' weight correlation. The basis of the correlation is that as long as power / control trays are not filled greater than 40% the tray weight will be less than 50 lb/ft.
Civil Engineering states that the 40% figure for 24 inch wide power / control trays appears in Bechtel TPO Design Guide C2.7 " Seismic. Category 'I Cable Tray and Conduit Raceway Support Systems." Additionally, Appendix D
" Methodologies For Determination of Cable Weights" of Design Guide C2.7 states that the maximum ultimate fill is 40% for control trays and 30% for power trays.
However, according to Appendix D,- neither of these i
conditions governs tray support design. ,
l I
The 40% fill limit for the instrumentation trays was i l
established to minimize mounding in the trays. Due to the light weight of the instrumentation cables, a fill- l l
to-weight correlation for instrumentation cable trays was not used.
During initial construction, Bechtel established a
____._--__-__a
5 4 ROOT CAUSE INVESTIGATION 87-05 procedural . limit of 40% for instrumentation and power / control cable trays. However, instrumentation cable tray fills of between 40% and 60% were allowed based on a generic Bechtel calculation. These cable tray fill limits were controlled administratively.
Bechtel stated that all pre-turnover cable trays filled in excess of the cable tray fill limits had been checked to verify that the tray loading was less than 50 lb/ft.
At turnover 12 Class 1E and 49 non-Class 1E power / control-cable trays exceeded the 40% fill limit._ Documentation _
cannot be found that shows that these trays were less than the 50 lb/ft limit when turned over.
From plant turnover to .1983 the District used a design fill limit of 40% for both control / power trays and instrumentation trays, i
In 1983 a memorandum was sent to all Electrical Engineers revising the cable tray fill limits. This memorandum raised the fill limits for instrumentation trays from 40%
to 50%. This 50% fill limit for instrumentation trays is not in agreement with the USAR statements of maximum fill for PS and SFAS trays.
ROOT CAUSE IlWESTIGATION 87-05 Currently the fill limit for power and control cable trays is 40% and 50% for instrumentation trays.
In the past Engineering has considered the cable tray fill limits to be guidelines which could be exceeded as necessary. The IIRG was unable to determine the method or procedure used by Engineering to authorize exceeding the fill limits. Based upon interviews, it appear that Engineers made informal checks to verify that the trays were not overweight and supervisory approval was neces-sary. However, neither the informal checks or the supervisory approval were proceduralized nor were they documented. 10 CFR 50.59 Safety Analysis Reviews to allow the 40% fill limit to be exceeded can not be found.
In July of 1987, 19 Class 1E cable trays and 171 non-Class 1 cable trays were listed as exceeding the 40% fill limit. Although 190 cable trays exceed the 40% fill limit only one was found to barely exceed the 50 lb/ft weight limit. The fact that only one of the overfilled cable trays, 740BG1A, appeared to exceed the weight limit gives credence that the fill-to-weight correlation is both satisfactory and conservative.
ROOT CAUSE INVESTIGATION 87-05 PROCEDURES Trackina and Reportina of Overfi11s/Overweichts
,)
During Rancho Seco's construction, the Bechtel TPO Design Guides, the Rancho Seco Design Guide and the E-700 series of drawings were used by Bechtel electrical engineering in the design and construction of the plant's electrical systems.
Bechtel TPO Design Guide C2.7 Appendix D gives the 1
maximum ultimate fill of control trays as 40% and power trays as 30%.
The Rancho Seco Design Guide discussed ampacity, but did not give specific guidance or limits on cable tray fill.
Note 14 of drawing E-701, Sheet 1 " General Construction Notes and Legends" states: " Tray fill shall be a maximum of 40%. This percentage fill may be exceeded at the discretion of the engineer."
- c. .
ROOT CAUSE INVESTIGATION 87-05 From 1975-until 1982, Rancho Seco design ' engineers used Bechtel' TPO design guides and the E-700 series of drawings for design guidance.
In 1982, Bechtel provide Rancho Seco a design l' guide / procedure on cable sizing and selection. In Section 7 " Selection of Cable Size" the following guidance is given: " Randomly filled trays shall have a maximum 40% fill."
In 1985 the Bechtel provided design guide was replaced by procedure NEP 5304.22 " Cable System Design, General."
Section 5.2 - of that procedure states: "The established limits-are conservative and a particular circumstance may merit the extra engineering' review necessary to justify design beyond these limits." Section 5.2.1.2 " Tray With Random Fill" states: " Randomly filled cable trays should l not exceed a maximum fill of 40%."
Since it was issued in 1985 procedure NEP 5204.22 has been revised only once. The revision to NEP 5204.22 allowed instrumentation trays to be filled to 50%. This fill limit is in conflict with the USAR limit of 40% for redundant trays.
- r. ; ..
1 ROOT CAUSE INVESTIGATION 87-05 Procedures Related to the Fill Limit The electrical design procedures do not provide guidance on the source of the fill limit, nor do they adequately explain the consequences of exceeding the limit.
Although the electrical design procedures state that the fill limits are conservative and may be exceeded with extra engineering review, the extra engineering review is not described or proceduralized.
The USAR fill limit for the Reactor Protection System (RPS) and Safety Features Actuation System (SFAS) cable trays is not presently contained in the electrical design procedures.
CABLE AND RACEWAY TRACKING SYSTEMS During the initial construction of Rancho Seco, Bechtel used the EE553, a cable and racevay tracking program, as a design and construction aid.
The EE553 Electrical Raceway report indicated the percent f fill of each cable tray, indicated those trays that were l
ROOT CAUSE INVESTIGATION 87-05 filled 30% or greater, and produced a special listing of the overfilled cable trays.
The District used Bechtel's EE553 program until 1978. In 1980 the EE553 program was replaced by CRTS.
One of the CRTS reports is " Tray with Included Cables."
This report lists each cable tray, each cable in the tray, the total cable area, and the percent fill. The report also indicates trays which exceed.40% fill, but CRTS does not calculate cable tray weight.
END 1
l
i 9 l
..- a ,
o ROOT CAUSE 87-09 UNACCEPTABLE INTERMIXING OF POWER / CONTROL AND INSTRUMENTATION CABLES )
J EXECUTIVE
SUMMARY
\ k l l I
J In March 1987, the District determined that 56 Class 1E instrument cables were intermixed with either power cables or control cables. The intermixing involved 20 ,
power cables and 14 control cables which were routed in '
I instrumentation cable trays, and 6 instrumentation cables routed in power / control cable trays. In addition, a total of 15 Reactor Protection System / Safety Features Actuation System (RPS/SFAS) instrumentation cables were found routed in instrument trays instead of conduit.
The majority of these problems were identified during a 1986/87 review (Impell Task 241/271) of the Cable and Raceway Tracking System (CRTS) data base. Eleven i
power / control cable intermixes were identified by l walkdowns performed as a part of the Cable and Wire Sampling Plan.
The intermixing of power / control cables with instrumentation cables and the routing of RPS/SFAS instrumentation cables in trays rather than conduitconstitute installations contrary to Section 8.2.2.11.H of the Updated Safety Analysis Report (USAR).
The intermixing of the power / control and instrumentation cables is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (ii) , (v) and (vii) . The routing of RPS/SFAS instrumentation cables in raceways rather than conduits is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (v) . These events were reported to the NRC via Licensee Event Report 87-26, " Unacceptable Intermixing of Power / Control and Instrumentation Cables."
The root cause of the cable intermixing problems is the failure to have and/or use procedures for cable design, installation, inspection, and repulling. Comprehensive design procedures did not exist until 1986. The present procedures do not address USAR requirements.
From 1981 to 1986, the Electrical QC Inspectors did not always inspect cable routing in accordance with procedure 1
e s,. .*,
d EC-11 " Cable Installation During Construction and Major Modifications."
The procedures for cable installation have not and do not specifically address the process used to pull back and repull cables in a revised routing.
DESCRIPTION In March 1987, the District determined that 56 Class 1E instrument cables were intermixed with either power cables or control' cables. The intermixing involved 20 power cables and 14 control cables which were routed in instrumentation cable trays, and 6 instrumentation cables routed in power / control cable trays. In addition, a total of 15. Reactor Protection System / Safety Features Actuation - System (RPS/SFAS) instrumentation cables were found routed in instrument trays instead of conduit.
The majority of these problems were identified during a 1986/87 review (Impell Task 241/271) of the Cable and Raceway Tracking System (CRTS) data base. Eleven power / control cable intermixes were identified by walkdowns performed as a part of the Cable and Wire Sampling Plan.
a,..
ROOT CAUSE INVESTIGATION 87-09 The intermixing of power / control cables with instrumentation cables and the routing of RPS/SFAS instrumentation cables in trays rather than conduit constitute installations contrary to Section 8.2.2.11.H l
l of the Updated Safety Analysis Report (USAR).
1 1
-The intermixing of the power / control and instrumentation cables is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (ii) , (v) and (vii). .The routing of RPS/SFAS instrumentation cables in raceways rather than conduits I is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (v) . These events were reported to the NRC via Licensee Event Report 87-26, " Unacceptable Intermixing of Power / Control and Instrumentation Cables."
CONCLUSIONS Direct Causes J l
1 P_qw_er/ Control Cables Associated with Cable Trays A28AA3 and A28AB3 I
The direct cause of the intermixing of cables in cable trays A28AA3 and A28AB3 is a construction error. The 1
j I
_ _ _ _ _ - _ - _ _ - _ - _ - _ _ - _ _ _ _ _ _ - _ 1
l i. 4, f
1 .
ROOT CAUSE INVESTIGATION 87-09 cables were repulled into their original routing instead of the revised routing.
The underlying cause is the failure to follow procedure l
EC-10 " Processing of Installation Cards." The pull cards were signed by the field engineer instead of the craft foreman who performed the work. Procedure EC-lO required the signature of the field installation personnel performing the work.
A contributing cause was the failure of the Electrical QC Group to identify the construction error. The error was not identified because of the failure of the electrical QC Inspectors to follow the requirements of procedure EC-11 " Cable Installation During Construction and Major Modifications. This was also identified in RC 86-10.
Power / Control Cables Associated with Cable Tray A28AN3 The direct cause of the intermixing of instrumentation cables with power / control cables in cable tray A28AN3 is a design error. The two power cables were not scheduled to be removed following the predesignation of the tray.
4, e, e
ROOT CAUSE INVESTIGATION 87-09 The contributing cause of the intermixing is a lack of a comprehensive service level designator in the cable tracking systems used. The tracking systems could not be used to check for intermixing conditions when the routing for the instrumentation cables was laid out; therefore, an intermixing condition would not be identified.
Power / Control Cables Associated with Cable Tray A28AA1 The direct cause of the intermixing in cable tray A28AA1 is either a design or construction error during construction of the Plant.
The contributing cause of the intermixing in cable tray A28AA1 is a lack of a comprehensive service level designator in the cable tracking systems.
Routina of 15 Instrument Cables in Trays Instead of Dedicated Conduit The direct cause of the RPS and SFAS instrumentation cables being routed in cable trays rathe'r than dedicated conduit is design error. This design error is present in
. h, a
ROOT CAUSE INVESTIGATION 87-09 1
two ECNs; however, the cable routings for both ECNs were performed by the same group.
The underlying cause of the cables being misrouted is the lack of training of NED personnel on USAR requirements.
Cables Associated with the Decay Heat System Cross Tie Flow Transmitters The direct cause of the intermixing of the 6 Decay Heat System cross tie flow instrumentation cables is design error. This design error occurred during the original design phase of Rancho Seco. The original design error was exacerbated when the routing was incorrectly redesigned in 1984.
The underlying causes are the use of only contract engineers as the Physical Layout Group and their lack of training on USAR requirements.
Cables Associated with the Reactor Coolant System Flow Transmitters The direct cause of the intermixing of the J4 Reactor
.y ROOT CAUSE INVESTIGATION 87-09 Coolant System flow transmitter cables is design error.
This design error occurred during the original design l
phase of' Rancho Seco. The original design error was l
exacerbated when . the routing was incorrectly redesigned in 1983.
l The underlying .cause was the use of only contract engineers for the Physical Layout Group and their lack of training on plant-specific USAR requirements.
Root Cause The root cause of the cable intermixing problems is the
- failure to have and/or use . procedures for cable design, I
l installation, inspection, and repulling.
Comprehensive design procedures did not exist until 1986.
The present procedure system does not address USAR requirements. ,
i, m,
3#'
In the 1983 outage, installation cards were not processed in accordance with procedure EC-10 " Processing of Installation Cards."
1
_. mm-.--_ - _ . . _ _ - ._-___._______.__m- _-_--__ _ . _ _ _ _ _ _ _ _ _
) .
ROOT CAUSE INVESTIGATION 87-09 l From 1981 to 1986, the Electrical QC Inspectors did not always inspect cable routing in accordance with procedure l
EC-11 " Cable Installation During Construction and Major 1
Modifications."
The procedures for cable installation have not and do not specifically address the process used to pull back and repull cables in a revised routing.
INVESTIGATION The Problem As of October 16, 1987, 56 Class 1E instrument cables have been found intermixed with either power cables or control cables. The intermixing involved 20 power cables and 14 control cables which were routed in instrumentation cable trays, and 6 instrumentation cables routed in power / control cable trays. In addition, a total of 15 RPS/SFAS instrumentation cables were identified as being routed in instrument trays instead of conduit.
_____________-_-______O
t >
e h ROOT CAUSE INVESTIGATION 87-09 l~
The majority of the cable problems were identified during a review (Impell Task 241/271) of the CRTS data base. An additional eleven power / control cable intermixes were
, identified by walkdowns performed as a part of the cable and Wire Sampling Plan. Two of these eleven intermixes were also identified by the review of the CRTS data base.
The intermixing of power / control cables with instrumentation cables and the routing of RPS/SFAS instrumentation cables in trays rather than in conduit constitute installations contrary to Section 8. 2. 2.11.H of the USAR, Amendment 4. The USAR states:
The separation of redundant cables of the reactor protection system and safety features actuation system circuits is accomplished by spatial separation in accordance with the following criteria:
- 1. . . .C1 ass I instrumentation circuits are routed in rigid metal conduits as explained in (2) below.
- 2. Reactor protection system and safety features actuation system instrumentation each have their channels routed in separate conduits are physically separated from each other throughout the plant.
n, .o, ROOT CAUSE INVESTIGATION 87-09
- 5. Power and control circuits are not mixed with instrumentation circuits in any raceway for any system.
There has been some discussion whether Section 8.2.2.11.H.5 of the USAR, Amendment 4 is applicable to all' plant systems or to just to RPS and SFAS. This i i
question was clarified by the issuance of amendment 5 to the USAR. However, the intermixing of power / control and instrumentation cables is not an acceptable Industry Practice.
The intermixing of the power / control cables and instrumentation cables is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (ii) , (v) and (vii).
The routing of RPS/SFAS instrumentation cables in raceways is reportable to the NRC in accordance with 10 CFR 50.73 (a) (2) (v) .
The cable intermixing problems can be separated into six groups:
- 1. The 15 power / control cables in instrumentation cable trays A28AA3 and A28AB3
ROOT CAUSE INVESTIGATION 87-09
- 2. The two power / control cables in instrumentation cable tray A28AN3
- 3. The two power / control cables in instrumentation cable tray A28AA1
- 4. The 15 RPS/SFAS instrumentation cables installed in cable trays instead of conduit.
- 5. The cable intermixing associated with the Decay Heat System cross tie flow transmitters
- 6. The cable intermixing associated with the Reactor Coolant System flow transmitters Since the causes of the intermixing for each group are different, they will be addressed separately.
The Scope of the Investigation The IIRG investigated the following:
c Procedures for the design, installation, and inspection of cables (Attachment 1) o Commitments for the inspection of cable routing (Attachment 1) o Changes in the Rancho Seco cable population from 1975 to 1986
=. * .
4 ROOT CAUSE INVESTIGATION 87-09 o Design, installation, and inspection processes from 1975 through 1986 o Tracking of cable routings o Modification 11 " Equipment Qualification" o The 15 power / control cables in instrumentation cable trays A28AA3 and A28AB3 o The two power / control cables in cable tray A28AN3 o The two power / control cables in cable tray A28AA1 o The 15 RPS/SFAS instrumentation cables l
l installed in trays instead of conduits.
o The six instrument cables associated with the Decay Heat System cross tie flow transmitters o The 14 power / control cables associated with the Reactor Coolant System flow transmitters The Chances in the Rancho Seco Cable Population As of October 1987, there are approximately 23,000 electrical cables installed at Rancho Seco. Nine thousand of these cables were installed since commercial operation.
L_________------
e ROOT CAUSE INVESTIGATION 87-09 The amount of cable installed each year since commercial operation has varied, but the maj ority of the installations occurred from 1983 to 1986. From 1975-to 1978, less than 45 Class 1E cables were installed at l
Rancho Seco.
The 1978 outage was the first outage with significant cable installation work. The majority of the cables l installed were security cables.
The ' 1981 outage involved the installation of cables for the short term NUREG-0737 (TMI) modifications.
i The 1983 outage began a three year period during which a l'
large number of cable installations were made. Five times more cable was installed in 1983 than 1981. The majority of this work was for the long term NUREG-0737 modifications.
The largest number of new cables was installed during the 1985 outage. The majority of the work again was for the long term NUREG-0737 modifications.
ROOT CAUSE INVESTIGATION 87-09 The Desian. Installation, and Inspection Processes from 1975 throuch 1986 NOTE: The procedures for the design, installation, and inspection are included in Attachment 1 to this report.
The desian orocess The Nuclear Engineering Department (NED) is responsible for establishing the design criteria, preparation of the design package, and supporting the installation of a modification. Since commercial operation, the processes (procedures and practices) used by NED to support these responsibilities have changed.
From 1975 to 1979 there were a limited number of modifications. As a result, the NED Electrical Design Group was small (three engineers). Additional engineering support was provided by contractor engineerc (usually from Bechtel).
ROOT CAUSE INVESTIGATION 87-09 Cable installation-The Modifications Department is responsible for the installation of modifications. Since commercial operation, contractor personnel have been used to install cables. Bechtel Corporation supplied most of the field engineers. Nuclear Power Services (NPS) supplied most of the craft labor.
Cable inspection Quality Assurance is responsible for the inspection of cable installations. Since commercial operation, the electrical inspection process has changed. Before 1979, cable inspection was supported by two District Quality Control (QC) inspectors.
If additional inspectors were needed, the District typically used inspectors supplied by Bechtel. During 1983, 1984, and 1985, the Electrical QC Group consisted entirely of vendor-supplied inspectors.
The District's cable inspection commitments:
e ROOT CAUSE INVESTIGATION 87-09 The District has committed to follow WASH 1284 " Guidance on Quality Assurance Requirements During the Operations Phase of Nuclear Power Plants" dated October 26, 1973.
WASH-1284 refers to ANSI N45.2.4-1972 " Installation, Inspection and Testing of Nuclear- Power Plant Instrumentation and Electrical Equipment During the Construction of Nuclear Power Generating Stations."
Section 5.11 of ANSI N45.2.4-1972 " Inspections to Verify Correctness of Installation" states:
Inspection shall- he made to verify that equipment is being located, installed, assembled, and/or connected to comply with the latest approved-for-construction drawings, manufactor's instructions, and installation specificaticas. Sugh inspections shall include, as - appropriate, vr.rification of:
...(3) Proper location and routing of cables and sensing lines.
Cable inspection procedures:
In 1981, procedure EC-11 " Cable Installation During Construction and Major Modifications" was written by the Lead Electrical QC Inspector. Section 7.5 of procedure EC-11 states:
1 I
L .
{
ROOT CAUSE INVESTIGATION 87-09 The QCE shall verify that cable size, number of conductors, cable code and routing are in accordance with drawings.
There have been questions about the interpretation of the phrase "... and routing are in accordance with drawings.
"These questions appear to be based on guidance for the inspection of cables that the Lead Electrical 'QC Inspector gave to electrical QC Inspectors. This procedure was written by a Bechtel engineer who was acting as the Construction QC Coordinator from 1979 to 1982 and as the Lead Electrical QC Inspector from 1982 to 1986. He stated that when he wrote procedure EC-11, he intended that the entire cable routing would be checked by QA. He also stated he was directed by SMUD to verify cable - routing only at the two ends. He recalled this direction being given to him sometime in 1979, but was unable to recall specifically who gave him this direction. He did provide a list of individuals who may have given this guidance to him.
The IIRG contacted all but one of the individuals. None recalled giving direction to check only the ends of the cable routing or were aware that this direction had been given.
i ROOT CAUSE INVESTIGATION 87-09 Based on the above, it appears District supervisors were not aware that from 1981 to 1986, QC Inspectors were not always verifying the complete cable routing. In interviews, the District Construction QC Coordinator (from 1982 to the present) stated that he understood the complete routing was being checked.
The Resident Engineer during the 1984/85 outage recalled discussions during the outage concerning whether an inspector needed to walk along during a cable pull or check the pulling equipment in order to verify the routing. He stated that it was his understanding that Class 1E cable routing had been checked 100%.
The Nuclear Engineering Lead Electrical Engineer from 1976 to 1986 stated that he believed the QC Inspectors were witnessing the cable pulls.
Electrical QC inspector training on District procedures:
From 1978 to 1982, a Bechtel Quality Engineer was the acting Construction QC Coordinator. In 1981, he wrote the Engineering Installation and Inspection (EII) procedures. In 1982, the Construction QC Coordinator was
l
. 1 l
ROOT CAUSE INVESTIGATION 87-09 replaced by a District employee, and he then became the i'
In late 1982, the District began using QC Inspectors from sources other than Bechtel. Several of these inspectors were assigned to the Electrical QC Group. The Lead Electrical QC Inspector provided training to Bechtal QC Inspectors on District procedures and policies, but not to the other inspectors. It cannot be determined what training the non-Bechtel electrical QC Inspectors received.
The Lead Electrical QC Inspector states that he did not provide training for electrical QC Inspectors who were not Bechtel employees.
The Construction QC Coordinator stated that he was not aware that the non-Bechtel electrical QC Inspectors did not receive training on District procedures. He believed that the Lead Electrical QC Inspector was providing training to all of the inspectors in the Electrical QC Group.
e t ROOT CAUSE INVESTIGATION 87-09 Inspection of cable routing from 1981 to 1986:
Root Cause 86-10 " Redundant Cabling in the Same Fire Area" reported that the Lead Electrical QC Inspector directed the electrical QC Inspectors during the 1984/85 outage to " check cable routings as far as possible from each end."
Based on additional interviews with several of these QC Inspectors, the practice of not always checking the complete cable routing apparently began sometime in 1981 l or 1982.
The QC Inspectors stated that sometime in 1981 or 1982, the Lead Electrical QC Inspector gave them the following guidance:
o Witness cable pulls if you are notified and have time o If you get to a point where you can't physically check the routing, stop o You can't afford to spend a lot of time inspecting cable routing, if a cable ROOT CAUSE INVESTIGATION 87-09 inspection gets to the coint where it takes a lot of time, stop.
The inspectors stated that they questioned this direction on several occasions, and they attempted to verify the complete cable routing, but were unable to do so in some cases.
Summary:
o Not all of the vendor supplied-electrical QC Inspectors received training on District procedures.
o From 1981 to 1986, the training for electrical QC Inspectors was inadequate in that the complete cable routing of cables being l installed were not always verified (inspected) as required by District procedures.
o Starting in 1981 or 1982, the electrical QC Inspectors were directed to "stop" checking the routing when the check became physically difficult or time consuming.
i 1
ROOT CAUSE INVESTIGATION 87-09 o Starting in 1981 or 1982, the electrical QC Inspectors were directed to check routings "as far back as possible" from each end.
l I
o From 1981 to 1986, the electrical QC Inspectors did not always inspect cable routing in accordance with the requirements of procedure EC-11 " Cable Installation During Construction and Major Modifications."
o From 1981 to 1986, the District did not always verify the routing of cables in accordance with commitments to follow WASH-1284.
i i
i As addressed in RC 86-10 " Redundant Cables in the Same Fire Area," the failure to follow the requirements of )
i procedure EC-11 " Cable Installation During Construction and Major Modifications" and the failure of the I
Construction QC Coordinator to provide adequate supervision to the Electrical QC Group were contributory causes to the failure to detect construction errors.
1 l
1
--___L _
e ROOT CAUSE INVESTIGATION 87-09 The Trackina of Cable Routinas Rancho Seco has used two systems to track cable routing.
During plant construction and until 1980, a Bechtel cable tracking system (EE553) was used to provide an inventory of installed cables, conduits and raceways. In the summer of 1980, CRTS replaced the EE553.
Neither tracking system is capable of determining the level of service (purpose of the cable, e.g. power). The alphanumeric cable designators do not definitively identify the use of cable. The service level is identified for some cables, but not all.
The lack of a definitive designator for the service level of a cable prevents the use of the CRTS to check for intermixing of power / control and instrumentation cables.
In the original design process, Bechtel used twisted shielded pair (TSP) or coax /triax cables for all circuits considered instrumentation. This practice allowed designers to identify instrumentation cables by examination of the type of cable used. This process i
1
l ROOT CAUSE INVESTIGATION 87-09 i
cannot be used by the CRTS because some low voltage power cables are TSP.
The CRTS generates installation cards. The system does l not generate a specific rework card for cable reroutings.
1 When a revised routing involves the pull back and repulling of the cable, the CRTS generated pull card shows only the new routing. Field engineers must refer to their drawings and make written comments on the cards for the repull and inspection of the cable.
There is.no procedural guidance for the marking of the l pull cards- for repulling and inspection of' repulled cables.
Modification 11 "Ecuipment Qualification" l
Modification 11 (Mod.011) " Equipment Qualification was issued in 1982 to upgrade existing hardware, conduit, trays, and cables to meet the environmental requirements of!the following.
o IE Bulletin 79-01B o Regulatory Guide 1.97 4 6 1
ROOT CAUSE INVESTIGATION 87-09 o NUREGs 0696 and 0737 I o 10 CFR 50.49. I To accomplish Mod.011, NED prepared the conduit rouH.ng l l
and tray change Drawing Change Notices (DCNs), electrical
]
schematic DCNs, and the equipment qualification documentation.
The ECNs associated with Mod.011 are:
o ECN A-3651 " Install Electrical Equipment to meet L.O.C.A. Conditions" o ECN A-3652 " Upgrade IE Equipment per 79.01B" o ECN A-4760 " Replacement of Transfer Switches" o ECN A-4786 " Replacement of Unqualified Transmitters" In order to support Mod.011, additional instrumentation cable trays were needed inside containment. Since space in some areas was limited, Nuclear Engineering searched for and located several power and control cable trays with a low percent of fill. These trays were to be converted to instrumentation trays.
i
1 ROOT CAUSE INVESTIGATION 87-09 ECN A-3065 " Bulk Raceway Installation" was issued for work on these trays. The existing cables in the power / control trays were to be pulled out and repulled in other power / control cable trays. The cable trays would then be redesignated as instrumentation cable trays, and instrumentation cables pulled into them. Three of these affected cable trays were L28AA3, L28AB4, and L28AN3.
The field engineers for Mod.011:
There were two field engineers assigned to Mod.011. Each field engineer had a NPS crew pulling cables. The NPS crews worked 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> on, 12 hours1.388889e-4 days <br />0.00333 hours <br />1.984127e-5 weeks <br />4.566e-6 months <br /> off. Cable pulls, started on one shift, were continued on the next shift.
The crew who completed a cable pull also completed the documentation for the cable pull.
One of the two construction field engineers for Mod.011 was a District engineer, the other was Bechtel engineer.
l For the District engineer, Mod.011 was the first assignment as a field engineer.- This engineer states that she:
{
I ROOT CAUSE INVESTIGATION 87-09 o received very little work direction from her immediate supervisor o depended on her colleagues for work direction and advice o received little training on her field engineer duties o learned the field engineer job through "on the job" training from her colleagues o frequently filled in for the Lead Electrical Engineer sr immediate supervisor) l o made decisions in the Lead Electrical Engineer's place o did not walk down the vias o checked the two end points, not the routing o depended on QC to catch problems and mistakes o marked the installation cards differently than the Bechtel engineer.
In addition, the Lead Electrical Engineer did not check her work, and she depended on the craft foreman to tell her when the work was done.
L_ ___ - -
1 ROOT CAUSE INVESTIGATION 87-09 Based on the above, it can be concluded that the training and supervision of the District's Mod.011 field engineer was less than adequate.
The Mod.011 installation cards:
The Construction Inspection Data Reports (CIDRs) for ECNs A-3651 and A-3652 were reviewed, and 610 installation cards (mainly pull cards) for these CIDRs were examined.
Based on these reviews, the following can be stated:
o one of the Mod.011 field engineers signed the installation cards instead of the field installation personnel as required by l procedure EC-10, " Processing of Installation
, Cards".
I o The field engineer's signature was found on 134 installation cards. The 134 cards included 70 deletict cards and 53 pull cards.
o One hundred and twenty five (125) of these signatures are dated in a four day period;
4 ROOT CAUSE INVESTIGATION 87-09 6/3/83 to 6/6/83. The QC cable inspection portion of the CIDRs was completed on 6/6/83.
o Procedure EC-10 required the QC Inspector to verify that the cards were properly filled out and signed. Twenty eight (28) of the cards were signed by a QC Inspector without a corresponding "wozk completed by" signature.
One inspector was responsible for 24 of these.
o There is significant variation in the marking of the cards for repull, and inspection of repulled cable.
i l
o One cable (1R1C240D) was found without an inspection signature. The pull card contains the statement: "did not inspect ???? pullback already repulled" )
l I
l o In several cases, the installation card files contained both a' completed hand typed card and ]
1 an uncompleted CRTS generated pull cards !
i n
l t
l
, ,. 2 9 -
- ,e ,
i
%E O
l 1
ROOT CAUSE INVESTIGATION 87-09 l
l o The two Mod.011 construction field engineers used different techniques to reflect cable I
repulls and were not consistent in their ;
, marking of the pull cards.
l l
[
Based on the above, it can be concluded that the processing of installation cards during Mod.011 was not always in accordance with the requirements of procedure EC-10 " Processing of Installation Cards."
Item 1 of 6: Power / Control Cables Associated with Cable Trays A28AA3 and A28AB3 Problem During the review (Impell Task 271) of the CRTS data base and the cable tracing walkdowns, three power cables and twelve control cables were found routed in instrumentation cable trays A28AA3 and A28AB3. These cable trays also contained 36 instrumentation cables.
The intermixing condition had existed since the instrumentation cables were installed in 1983 as a part of Mod.011. '"he intermixing of power / control cables with
_ - _ - - . ~ .
ROOT CAUSE INVESTIGATION 87-09 instrumentation cables is contrary to the requirements of USAR Section 8.2.2.11.H.5.
Investigation The IIRG reviewed the following areas:
o The Mod.011 work in cable trays A28AA3 and A28AB3 o The predesignation .of cable trays L28AA3 and L28AB3 o Pull cards /DCNs for the 15 power / control cables o The failure to repull the cables in the new routing o The signing'of the pull cards o The failure to find the cable routing error during the QC inspection The Mod.011 work in cable trays A28AA3 and A28AB3:
The installation of the 36 instrumentation cables and the rerouting of the 15 cables were a part of the work {
associated with Mod.011.
. ... l
ROOT CAUSE INVESTIGATION 87-09 The 15 power / control cables were to be pulled out of cable trays L28AA3 and L28AB3, and then repulled'through cable trays L28AA4 and L28AB4. After the trays were redesignated, the instrumentation cables were to be pulled into the redesignated cable trays.
The predesignation of cable trays L28AA3 and L28AB3:
In February 1983,.as a part of ECN A-3065, cab?.c trays L28AB3 and L28AA3 were redesignated as A28AB3 and A28AA3 (instrumentation cable trays) . The power / control cables
~in these trays were scheduled to be pulled out and repulled in power trays L28AA4 and L28AB4.
Pull cards /DCNs for the 15 power / control cables:
The IIRG was not able to locate pull cards or DCNs for five of the 15 power and control cables. The District identified the problem of missing installation cards in their review of the CRTS data base.
l The IIRG examined the pull cards and DCNs that were i l'
available. The DCNs have clear instructions marked on them for the pull back and repulling of these cables.
___ _ d
^l
-1 ROOT CAUSE INVESTIGATION 87-09 The cards do not have any CRTS generated repull markings or instructions. Some of the cards have written comments on them for the repulling of the cables. The marking of .
J' the cards for cable rerouting is inconsistent and would be hard for both field installation and QC personnel to 1
follow.
The pull cards for these power / control cards all have the signature of the Bechtel field engineer for Mod.011 in the " work completed by" block. Procedure EC-10
" Processing of Installation Cards" requires the field installation personnel to sign in this block.
The pull cards were signed by QC Inspectors indicating that the cables had been inspected and the work was acceptable. Of the 10 cards, seven were signed on the same day or next day, and two cards were signed three days before the field engineer's signature.
The QC Inspector who performed nine of the inspections stated he used the 1010 DCNs to verify routing and that upon the receipt of the cards, he checked his 1010 DCN, and signed off the cards. This action would explain the large number of inspector signatures on 6/5 and 6/6/83.
l ROOT CAUSE INVESTIGATION 87-09 The failure to repull the cables in the new routing:
I Based on the markings on several of the cards and on the recollections of one of the Electrical QC Inspectors, the power / control cables were apparently pulled out of cable trays A28AA3 and A28AB3. The inspector recalled a period during Mod.011 when the four cable trays were empty and the cables were pulled back and coiled.
l l
It would appear that the cables were mistakenly repulled into cable trays A28AA3 and A28AB3 instead of the revised l routing.
The signing of the pull cards:
Based on interviews, the end of the 1983 outage was a hectic time and there was a push to get the Mod.011 modification completed in time for testing. I During the outage, the majority of Mod.011 installation cards were signed by the field installation personnel who l l
performed the work. However ,during a four day period at {
the end of the outage, one of the Mod.011 field engineers signed 134 installation cards (including the cards for
)
l I
l 1
ROOT CAUSE INVESTIGATION 87-09 the cables in question) as the person completing the work. At the end of that four day period, the inspection portion for cable installations of the CIDR package was also completed. The' field engineer's signature was contrary to the requirements of procedure EC-10.
The failure to find the cable routing error during the QC inspection:
It cannot be determined if the QC inspector checked the entire cable routing. If the inspector. used 1010 drawings to check the cables and signed the pull card.on the day he received the card, then the card does not reflect the day the cable was inspected. Sixty six of the 134 cards signed by the field engineer during the four day period previously mentioned were signed the same day or the next day by QC inspectors.
Causes l
During the Mod.011 installation there was a lack of field engineer involvement in the installation of cables.
The field engineer who was interviewed stated:
t-
0 ROOT CAUSE INVESTIGATION 87-09 o She rarely went into the Reactor Building to check on work o Her craft foreman would tell her when a cable installation was completed o She depended on QC to catch problems.
The direct cause of the _ intermixing ' of cables in cable trays A28AA3 and A28AB3 is a construction error. The cables were pulled back into their ' original routing instead of the revised routing.
The underlying cause is the failure to follow procedure EC-10, " Processing of Installation Cards." The pull ,
cards were signed by the field engineer instead of the craft foreman who performed the work. Procedure EC-10 required the signature of the field installation personnel performing the work.
A contributing cause was the failure of the Electrical QC Group to identify the construction error. The error was not identified because of the failure of the electrical QC Inspectors to follow the requirements of procedure EC-11 " Cable Installation During Construction and Major
ROOT CAUSE INVESTIGATION 87-09 Modifications. This was also identified in RC 86-10
" Redundant Cabling in the Same Fire Area."
Item 2'of 6: Power / Control Cables Associated with Cable Trav A28AN3 l
-Problem l During the review' (Impell Task 271) of the CRTS data j
\ 1 base, one power cable and one control cable were found routed in instrumentation cable tray A28AA1. Cable tray A28AA1 also contained four instrumentation cables.
l l
The' intermixing condition in cable tray A28AA1 existed in l excess of 12 years. The intermixing of control / power 1
cables with instrumentation cables is contrary to the requirements of USAR Section 8.2.2.11.H.5.
Investigation l
The July 12, 1975 EE553 report indicated that cable tray L28AN3 contained four power cables. In late 1975, two of l the cables werte removed, leaving two power cables in the l
tray. >
1 I
L ROOT CAUSE INVESTIGATION 87-09 In February 1983,_ the cable tray was redesignated as a instrumentation cable tray as a part of ECN A-3065. The power / control cables were not scheduled to be pulled out j of the cable tray. The routing for the two cables was last revised in December 1975. No cables were scheduled to be pulled into this tray during the 1983 outage. The l
IIRG was unable to find any design ' documents showing a revision to the circuits following the 1975 revision.
In 1984/85, four instrument cables were added to the cable. tray as _ a part of ECN A-3652. The addition of these cables resulted in the mixing of power / control and instrumentation cables.
Causes The direct cause of the intermixing of in;trumentation cables with power / control cables in cable tray A28AN3 is a design error. The two power cables were not scheduled to be removed following the predesignation of the tray.
A contributing cause of the intermixing is a lack of comprehensive service level designator in the cable tracking systems used. The tracking systems could not be
V ,
ROOT.CAUSE INVESTIGATION 87-09 used to check for intermixing conditions when the routing for the instrumentation cables was; therefore, an intermixing condition would not be identified.
Item 3 of 6: Power / Control Cables Associated with Cable Trav A28AA1 Problem During the review (Impell Task 271) of the CRTS data base, one power cable and seven control-cables were found routed in instrumentation cable tray A28AA1. Cable tray A28AA1 also contained 11 instrumentation cables.
1 The intermixing of control / power cables with instrumentation cables is contrary to the requirements of USAR Section 8.2.2.11.H.5.
Investigation The EE553 report for July 12, 1975 lists cable tray A28AA1 as containing one power and seven control cables.
The eight cables were installed in cable tray A28AA1 during initial plant construction. The direct cause of
, '*[
ROOT CAUSE INVESTIGATION 87-09
.the routing of power / control cables in cable tray A28AA1 cannot be determined at the present time.
The routing of power / control cables in an instrumentation tray was not identified following plant turnover, and in 1985, five instrumentation cables were added to tray A28AA1 without the presence of the power / control cables in the tray being identified.
The failure to -; identify intermixing is related to the method used to identify cables.
Causes The direct cause of the intermixing in cable tray A28AA1' is either a design or construction error during construction of the Plant.
The underlying cause of the intermixing in cable tray A28AA1 is lack of a comprehensive service level designator in the cable tracking systems. !
ROOT CAUSE INVESTIGATION 87-09 Item 4 of 6: Routina of 15 Instrument Cables in Travs Instead of Dedicated Conduit Problem Fifteen RPS/SFAS instrumentation cables located in the Reactor Building were routed in cable trays instead of dedicated conduit as required by the USAR. Section 8.2.2.11.H.2 of the USAR, Amendment 4 states:
" Reactor protection system and safety features actuation system instrumentation each have their channels routed in separate conduits..."
I Contrary to this requirement, 15 RPS/SFAS instrumentation cables were routed into cable trays in 1983. This condition would not be an issue if a Safety Analysis Report (SAR) had been prepared and approved addressing the new cable routings.
1 1 _ . _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
ROOT CAUSE INVESTIGATION 87-09 Investigation The investigation involved the ECNs which accomplished the routing changes and the requirement to route the cable in dedicated conduit.
In 1983, the 15 cables in question were routed by Mod.011 ECNs A-2932 and .A-3652. ECN A-2932 installed the ,
necessary equipment to satisfy the NUREG-0737 requirement for instrumentation to detect inadequate core cooling.
ECN A-3652 ensured that electrical equipment in the Reactor Building met the post accident operability conditions required by NRC IE Bulletin 79-01B.
The IIRG reviewed the two ECNs to determine if the SAR or Design Basis Reports (DBR) had addressed the USAR cable routing instrumentation cable. The SARs and DBRs for the ECNs do not address the USAR cable routing requirements for instrumentation cable.
Procedure ECP-1 " Configuration Contiol", Revision 4 (dated July 1982) makes the Cognizant Engineer responsible for the preparation of the DBR based on details of the design. A Licensing Engineer then reviews
' ROOT CAUSE INVESTIGATION 87-09
-the DBR to determine if the USAR must be updated and prepares the SAR based on the DBR.
~
The SARs for'the two ECNs in question.did not adequately address USAR requirements in the review of the design.
l The cables were routed in cable trays instead of conduit, contrary to the USAR.
Specific procedures providing detailed design.
' requirements did not exist in 1982 and 1983. Procedure ECP only provides administrative information on preparing an ECN.
The FSAR was reviewed to determine the source - of the requirement to route the SFAS and RPS cables in dedicated conduit. Section 8.2.2.11.H.1 of the FSAR states:
Class I 600-volt power and control cables of -
one channel are run together in trays that belong to. the same channel. Class I 4 instrumentation circuits are routed in rigid )
metal conduits as explained in paragraph 2 below. These two different and . independent routing methods preclude the possibility of fire in a tray carrying power and control cables propagating to instrumentation circuits.
Based on the above, the routing the SFAS and RPS cables I
l L - :
ROOT CAUSE INVESTIGATION 87-09 1
was due to fire protection considerations. With the advent of Appendix "R" to 10 CFR 50, the routing of ,
instrumentation' cables in dedicated conduit was no longer required for fire protection considerations.
In 1982, the USAR replaced the FSAR. This change deleted the statement "These two different and independent routing methods preclude the possibility of fire . in a tray carrying power and control cables propagating to instrument circuits." The statement on routing the instrumentation cables in dedicated conduit was not deleted.
Causes The direct cause of the RPS and SFAS instrumentation cables being routed in cable tray rather than dedicated conduit is design error. This design error is present in two ECNs; however, the cable routings for both ECNs were performed by the same group.
A contributing cause of the cables being misrouted is the i lack of training of NED personnel on USAR requirements.
ROOT CAUSE INVESTIGATION 87-09
. Item 5 of 6: _C_gpb_les Associated with the Decay Heat System Cross Tie Flow Transmitters Problem Six Decay Heat System (DHS) instrumentation cables were found routed in power / control raceway by Impell's review of the CRTS data base. The intermixing of power / control cables and instrumentation cable is contrary to the requirements of USAR Section 8. 2. 2.11.H. 5 which states:
" Power and control circuits are not mixed with instrumentation circuits in any raceway for any system."
Investigation The investigation into this cable intermixing problem concentrated on establishing when this intermixing occurred and the design process used.
The present cables were installed by ECN A-4786
" Replacement of Unqualified Transmitters and Switches."
This ECN replaced the DHS cross tie flow transmitters and installed new cable. ECN A-4786 was written in 1984 and completed during the 1985 refueling and as part of the Mod.011 package.
ROOT CAUSE INVESTIGATION 87-09 ECN A-4786 was reviewed to determined if the intermixing was discussed in the SAR or DBR. Neither the SAR nor the DBR discuss cable intermixing. Section II.C of the DBR
" Design Crit'eria Used" states that " existing conduits, terminal boxes, cable trays, etc., shall be used where possible."
Based on interviews with the Design Engineer for ECN A-4786 the following can be concluded:
o There was no ' training given to NED personnel on the USAR requirements for their discipline.
o Block diagrams were supplied to the Physical Layout Group in NED. This group was to design the routings.
o The "where possible" use of the existing trays and conduit was not unique to ECN A-4786.
o The Physical Layout Group, which designed the cable routings, were all contractors.
Based on the design criteria in the DBR, the routing of the previously installed instrumentation cable was reviewed. The review showed that the previous instrumentation cable routing was also installed in
i ROOT CAUSE INVESTIGATION 87-09 i
power / control raceway. The new cable routing is not l
exactly the same as the previous routing, but many of.the '
t same vias are used.
i d
The FSAR was reviewed to determine when the intermixing j criteria was established. Section 8.2.2.11.H.5 was added to the FSAR by Amendment 5 in 1972; therefore, the original cable routings were contrary to the design requirements of the FSAR. How the original design error j occurred cannot be determined.
1 Causes I
The direct cause of the intermixing of the six DHS cross tie flow instrumentation cables is design error. This design error occurred during the original design phase of Rancho Seco. The original design error was exacerbated 1 when the routing was incorrectly redesigned in 1984. ;
l 1
Contributing causes are the use of only contract
)
engineers as the Physical Layout Group and their lack of j
)
training on USAR requirements. i
_ _ _ _ . _ _ _ _ _ _ - _ _ _ _ _ _ l
- 1. '
i i
ROOT CAUSE INVESTIGATION 87-09 Item 6 of 6: Cables Associated with the Reactor t Coolant System Flow Transmitters Problem Fourteen of the 16 power cables for the Reactor Coolant System flow transmitters are routed in instrumentation conduit in the Reactor Building. These conduits - also contained signal cables. This condition is contrary to the requirements of USAR Section 8.2.2.11.H.5.
Investigation i
The investigation into the intermixing of the 14 RCS flow transmitter cables concentrated on establishing when the i
intermixing occurred. l 1
The original design of the RCS flow transmitter cables routed the signal and power cables in the same conduit from the penetration to the transmitters. Bechtel design engineers state that in the past, it was considered I
acceptable practice to route the power and signal cables )
for an instrument in the same conduit. This practice was contrary to Section 8.2.2.11.H.5 of the FSAR. Section
]
l j
l l
____________________________a
i, ROOT CAUSE INVESTIGATION 87-09
- 8. 2. 2.11. H. 5 was added to the FSAR in'1972. Why this design error occurred cannot be determined.
.In 1983, the original installation was redesigned as part of Mod. 011. The new design routed the cables from the penetration to a junction box and from the junction box to the flow instruments. In two cases, the.Section of cable from the penetration was correctly designed. The 14 incorrectly designed routings indicate a lack of-attention to detail.
It appears that some of the cables were replaced when in l 1985 the transmitters were upgraded as part of ECN A-I 4786; however, the cable routings were not revised when I the cables were replaced. .
r )
l Causes
]
l 1
l The direct cause of the intermixing of the 14 RCS flow transmitter cables is design error. This design error ,
1 occurred during the original design phase of Rancho Seco. I I
The original design error was exacerbated when the '
routing was incorrectly redesigned in 1983. l 1
~ ROOT.CAUSE INVESTIGATION 87-09 A- contributing cause was. the use of only. contract engineers for the Physical Layout Group and their lack of
. training on plant-specific USAR requirements.
I l
i 1
l t
I I
_. ..m _ . _ . . _ __.____ _ __-.____.__
1 1 .
1 I
l ROOT CAUSE INVESTIGATION 87-09 1
l ATTACHMENT l
Procedures for the Desian, Installation. and Inspection of Cables I
1 Desian Drocedures from 1975 to 1986 l l
]
From 1975 to 1985, the design process for electrical equipment / cable used the following documents:
o Procedure ECP-1 " Configuration Control" o Electrical Drawing E-701" General Construction Notes and Legends" o Bechtel. Thermal Power Organization (TPO)
Standards and Guides
- TPO E2.6.1, " Cable Systems Design, General" o Rancho Seco Design Manual, Rev. 2, July 1970 o Industry Standards (IEEE) o The National Electrical Code.
From 1975 to 1985, procedure ECP-1 was used for the initiation and approval of plant modifications. In 1986, procedure ECP-1 was replaced by procedure NEP 4109,
" Configuration Control."
I L
ROOT CAUSE INVESTIGATION 87 !:
From 1985 to 1986, the design process for electrical equipment / cable used the following documents:
o NEP-4109 " Configuration Control" o NEPM-5204.22 " Cable Systems Design, General" o NEPM-5204.23 " Cable Pulling Calculations and (
i Raceway Design." ]
Until June 1987, CRTS Use procedures did not exist. Use of the CRTS was controlled by memorandum and the - CRTS User's Guide.
-1 Installation Drocedures from 1975 to 1986 From 1975 to 1982, the process for the installation of cable involved the following documents:
o ECP-1 " Configuration Control" o Electrical Drawing E-701 " General Construction Notes and Legends" o EM-145 " Installation and Control Of Temporary-Electrical Cables" o EM-163 " Installation of Permanent Plant Cables."
I
1 t ,i '
)
.- l ROOT CAUSE-INVESTIGATION'87-09 1
l From 1982 to - 19 8 5, the process for the installation of cables involved the following documents:
o ECP-1 " Configuration Control" o Electrical Drawing E-701 " General Construction Notes and Legends" o EM-145 " Installation and Control Of Temporary Electrical Cables" o EM-163 ~ " Installation of- Permanent Plant Cables" o Construction Methods and Procedures Sections:
- 6.0 " Cable Installation-Pulling"
- 7.0 " Cable Installation-Pulling Definitions and Examples"
- 8.0 " Cable Installation-Pull Calculations and Examples"
- 9.0 " Criteria for Overpulling of Cables in Conduits Already Occupied" o Engineering and Inspection Instructions (EIIs) i
- EC-10 " Processing of Installation Cards"
- EC-11 " Cable Installation During Construction and Major Modification"
- EC-15 " Cable Terminations and j Connections"
- - _ _ - _ _ _ _ _ . _ _ _ . _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __s
!, 1 3
.. . 1 1
ROOT CAUSE INVESTIGATION 87-09 In 1986, the Construction Methods and Procedures and the EIIs were replaced by the NEPs. The NEPs applicable to the Modifications' group were replaced in December 1986 by the Modification Procedures / Inspection Standards (MP/ISs). When_the EIIs were replaced, a procedure for the processing of installation cards was not issued. At the present time, there is no procedure for the ;
processing of installation cards. f In 1986, the process for the installation of cables involved the following documents:
o NEP-4109 " Configuration Control" o EM-187 " Control of Electrical Plant l
Modifications" o Electrical Drawing E-701 " General Construction i l
Notes and Legends" o EM-145 " Installation and Control Of Temporary !
l Electrical Cables" o EM-163 " Installation of Permanent Plant ,
Cables" o MP/IS-307 " Cable Installation" o MP/IS-311 " Low Voltage Cable Terminations &
Splices" l
O. ';
-ROOT CAUSE INVESTIGATION 87-09 o MP/IS-315 " Medium Voltage Cable Terminations &
Splices" o NEPM-5204.63 " Cable Installation Notes and Installation Details" o NEPM-5304.8C " Electrical Cable Installation and Termination."
)
The procedures for cable installation have not and do not specifically address the process used to pull back and repull cables in a revised routing.
i Inspection crocedures from 1975 to 1986 1
From 1975 to 1981, the process for the inspection of electrical cable used the following documents:
o QAP-29 " Construction Inspection" o QCI-107 " Installation Inspection of Electrical Cable, Wire, and Conduit" o Electrical Drawing E-701 " Construction Notes and Legends.
l l From 1981 to 1986, the process for the inspection of l
clectrical cable used the following documents:
l l
L
1 4 1 l o ROOT CAUSE INVESTIGATION 87-09 o QAP-29 " Construction Inspection" o QCI-107 " Installation Inspection of Electrical Cable, Wire, and Conduit" o QCP-330 " Conduit and Cable Installation Inspection" o QCP-340 " Cable Tray and Cable Inspection" o Electrical Drawing E-701 " Construction Notes and Legends."
In 1981, the EIIs were issued, and the applicable EIIs are: 's o EC-2 " Inspection of Raceway Systema" (
o G-4 "Procesc of Construction Reports" ,
o 'EC-10 "Prtcors,ing of 'Ir.stallation Cardb" o "r.C-11 "Cuale Installati% During Construilion
? ,
end Yajor Modifice. tion # i
,. ; 7, o EC-15 " Cable Teninatf ans < avj Conn Actions 8 .t f
/ ,
s O/ t ,\ ,
\
- i ,
s ,
I '
'. s I p, y Procedu. e QAF-6 " Inspection Planninc/ ' wit s issued in lp{si,. , f i )
/
, / /
In 1985, the EII.P were r placed , bi the EEF s , c e.ad in) ( , ,t
/ -
f /, I \ b' 1,
) , , ,.
December 1386, th'c' NEPs for cable inf.alh. tion were ,
r' i
- l s
-io - 7 3
x
)
i
< 1 / l
1 I
\ .
l .
T- hl I , ,
A. . p 8 g ;,. .),
s.
s .\ s( e s s* i ,
y
- k. {;
( t (i
' ' 'Q J}00T CAUSE INVESTICAT70N 87-09
, 3 ..
< ').
B ch #
I 6; (, s +-
~\ (
, c%.y replaced by the MP/ISs. The '!!P/IS procedures contain Inspection Reports (IRS). The applicable MP/ISs are: j
- s. .,
I. ' k ;. I 44 . o. MP/IS-307 " Cable Instbcllation" t , .-
- MP/IS3311 KLow Voltagn, CablesTerminations &
, lo
' - ~,
, 3 3 .
~
f Splices" q l s u .
o .MP/IS-315 " Medium Voitage Cable Terminations &
? Splices."
i s k The commitments fgr.the inspection of cable routina i
s V ,
a
,[
In a September 1976 Ietter to Mr. Redert W.
^
% i.
Reid, Chief f
Operating Reactors Brai.ch No. 4, regarding the~QA program (
f at Rancho Seco, the District committed toU include
! , (.
l < guidance on WASH -12M ,
" Guidance onc @tality Adsurance 1
{' A RequirementsDuringtheOperationsPhas[LofiNuclear 1
.t Power Plants,"tdaten October 26, 1973. '
; :, T ,
s N.
v -
z ,_
, , s
's WASI$I284 refers $othefollowintfdocuments:
.c s.
s
,s o Appendix S to 10 CFR Part 50 " Quality Assurance l Criteria for Nudear Po;mr Plants and Fuel Processing Plants" s t ...
- n. I, h, I l
l / # , S.
J M
- s 5
t.n k 1; $
c ROOT CAUSE INVESTIGATION 87-09 o Regulatory Guide 1.30 " Quality Assurance Requirements for the Installation, Inspection, and Testing of Instrumentation and Electrical Equipment"
\
o ANSI N45.2.4-1972 " Installation, Inspection and Testing of Nuclear Power Plant Instrumentation and ;
Electrical Equipment During the Construction of l Nuclear Power Generating Stations" (also designated j
,n :
ANSI N45.2.4-1972 states in Section 5.11. " Inspections to Verify Correctness of Installation": (
l Inspection shall be made to verify that equipment is being located, installed, assembled, and /or connected to comply with the latest approved-for-construction drawings, ]
manufactor's instructions, and installation i specifications. Such inspections shall .
I include, as appropriate, verification of*
...(3) Proper location and routing of cables and sensing lines." !
I END l
4 L