ML18100A540
| ML18100A540 | |
| Person / Time | |
|---|---|
| Site: | Salem  |
| Issue date: | 08/06/1993 |
| From: | Durr J, Ruland W NRC OFFICE OF INSPECTION & ENFORCEMENT (IE REGION I) |
| To: | |
| Shared Package | |
| ML18100A539 | List: |
| References | |
| 50-272-93-81, 50-311-93-81, NUDOCS 9308170041 | |
| Download: ML18100A540 (64) | |
See also: IR 05000272/1993081
Text
U.S. NUCLEAR REGULATORY COMMISSION
REGION I
REPORT/DOCKET NOS.
50-272/93-81
50-311193-81
LICENSE NOS.
LICENSEE:
FACILITY:
INSPECTION DATES:
Public Service Electric and Gas Company
P.O. Box 236
Hancocks Bridge, New Jersey 08038
Salem Nuclear Generating Station
June 6-28, 1993
INSPECTORS:
Stephen Barr, Resident Inspector, Salem, DRP (Asst. Team Leader)
M. Eugene Laz3.rowitz, Reactor Engineer, DRS
TEAM LEADER:
APPROVED BY:
Hukam Garg, Sr. Electrical Engineer, NRR/IDCB
Larry Scholl, Reactor Engineer, DRP
W. H. Ruland, Chief, Electrical Section,
Engineering Branch, DRS
Jacque P. Durr, Chief, Engineering
Branch, Division of Reactor Safety
9308170041 930811
ADOCK 05000272
G
PD.R
' DAte
I
Date
2
Areas Inspected: An Augmented Inspection Team (AIT), consisting of personnel from
Region I and NRR, inspected those areas necessary to ascertain the facts and determine
probable causes of numerous rod control system electronic component failures that occurred
in May and June 1993. The team assessed the safety significance of potential inadvertent
outward rod motion that occurred as the result of a single component failure. The adequacy
of the licensee's maintenance and troubleshooting practices relative to the rod control system
was reviewed. The decision process concerning attempted plant startups following the
failures and repairs was evaluated as well as the possibility for any potential generic
implications posed by the Salem rod control problems.
Results: See Executive Summary
TABLE OF CONTENTS
EXECUTIVE SUMMARY ...................................... 3
1.0
INTRODUCTION ....................................... 4
1.1
Rod Control System Failure Overview . . . . . . . . . . . . . . . . . . . . . . 4
1.2
Augmented Inspection Team (AIT) Formation ... ~ . . . . . . . . . . . . . . 4
1.3
Rod Control System Description . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.0
GENERAL SEQUENCE OF EVENTS . . . . . . . . . . . . . . . . . . . . . . . . . . 6
3.0
ROD CONTROL SYSTEM CORRECTIVE
MAINTENANCE/MODIFICATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.0
EVALUATION OF PREVENTIVE MAINTENANCE ACTIVITIES . . . . . . . .
8
4.1
Vendor Work Scope. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.2
Work Accomplished ............................ *. . . . . . 9
4. 3
PSE&G Oversight of Vendor Work . . . . . . . . . . . . . . . . . . . . . . .
10
4.4
Independent NRC Inspections ............... .' . . . . . . . . . . .
11
4.5
Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
12
5.0
EVALUATION OF TROUBLESHOOTING ACTIVITIES
. . . . . . . . . . . . .
12
5.1
Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . .
12
5.2
PSE&G Activities Prior to AIT Arrival . . . . . . . . . . . . . . . . . . . . .
13
5. 3
Actions After AIT Arrival . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
15
5.3.1 Component Failure Analysis . . . . . . . . . . . . . . . . . . . . . . . .
16
5.3.2 PSE&G Conclusions ........... ~ . . . . . . . . . . . . . . . . .
16
5. 3. 3 Corrective Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
16
5. 3 .4 AIT Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
17
5.3.5 General Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
5 .4
Summary of Component Failures . . . . . . . . . . . . . . . . . . . . . . . . .
18
6.0
RESTART DECISION PROCESSES . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
6.1
Description of PSE&G Actions . . . . . . . . . . . . . . . . . . . . . . . . . .
18
6.2
Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
6.3 * Conclusions ......... '. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
7.0
ROOT CAUSE POLICY AND PROCEDURES . . . . . . . . . . . . . . . . . . . .
21
8.0
COMPENSATORY MEASURES/JCO.......................... 22
i
9.0
SAFETY SIGNIFICANCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
10.0
CAUSES OF ROD CONTROL SYSTEM PROBLEMS . . . . . . . . . . . . . . .
25
10.1
HARDWARE PROBLEMS . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
10.1.1 Multiple Integrated Circuit and Output Transistor Failures .....
10 .1. 2 Regulation Circuit Board Short Circuits . . . . . . . . . . . . . . . .
10.1.3 I/O AC Amplifier, I/O Receiver and Slave Cycler Logic Card
25
25
Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
10.2
10.3
10.1.4 Q9 Transistor Failures ................... *. . . . . . . .
26
10.1.5 Failure Detector Card Resistor . . . . . . . . . . . . . . . . . . . . . .
26
ROD CONTROL SYSTEM DESIGN . . . . . . . . . . . . . . . . . . . . . .
27
10.2.1 Single Failure Vulnerability . . . . . . . . . . . . . . . . . . . . . . . .
27
10.2.2 Voltage Suppression Circuit . . . . . . . . . . . . . . . . . . . . . . .
27
10.2.3 Group Step Counter Compatibility
. . . . . . . . . . . . . . . . . . .
28
LICENSEE POLICY AND PERFORMANCE . . . . . . . . . . . . . . . . .
28
11.0
POTENTIAL GENERIC IMPLICATIONS . . . . . . . . . . . . . . . . . . . . . . .
28
12.0
OTHER FINDINGS/OBSERVATIONS . . . . . . . . . . . . . . . . . . . . . . . . .
29
13.0
EXIT MEETING . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
30
ATIACHMENT 1 - AIT Charter
ATTACHMENT 2 - Rod Control System Block Diagram
ATIACHMENT 3 - Sequence of Events
A TI ACHMENT 4 - Circuit Card Locations Experiencing Multiple Failures
ATIACHMENT 5 - List of Failures/Cause
A TI ACHMENT 6 - Exit Meeting Slides
ATIACHMENT 7 - Exit Meeting Attendees
ATIACHMENT 8 - Persons Contacted
ATIACHMENT 9 - Acronyms and Initialisms
ii
EXECUTIVESUMJ.\\fARY
NRC established an Augmented Inspection Team (AIT) on June 5, 1993, after Salem Unit 2
experienced numerous component failures of the rod control system (RCS). The team's
charter required detailed fact-finding, identification of root causes and potential generic
issues, and review of Public Service Electric and Gas (PSE&G) restart decisions.
The team found several causes of the RCS failures. The predominant cause of the failures
was that the solid state components were subjected to high voltage spikes produced by the
group step counters. During the outage, several counters were replaced during preventive
maintenance work. These counters had different electrical characteristics and resulted in the
generation of a higher reverse electromotive force (EMF) than the original counters.
Coincident with the higher reverse EMF, a surge suppression circuit was most likely lost and
permitted high voltage spikes to stress electrically and ultimately fail solid-state components
in the rod control system logic cabinet circuits.
The rod drop event, which occurred on May 28, 1993, was the result of short circuits on a
printed circuit card due to slivers of solder bridging adjacent circuit traces. Additional
component failures were the result of poor work practices during system troubleshooting and
testing that introduced momentary short circuits in the system. One failure was the result of
a spare circuit card that had an incorrect resistor installed.
PSE&G did not provide for the programmatic determination of a root cause for the RCS
problems. Had such a determination been performed, the repetitive startups and shutdowns
of Unit 2 might have been avoided. This weakness contributed to the troubleshooting effort
lacking clear leadership and delegation of responsibilities. The troubleshooting effort focused
too narrowly on identifying failed components and did not put adequate emphasis on finding
and correcting the root cause of the failures. For example, testing conducted by PSE&G,
after the AIT arrived, showed that. one of their original root cause.s was in error. However,
the team found that, for each of the five startups, the licensee satisfied all procedural and
technical specification (TS) requirements prior to initiating a Unit 2 startup, and that
operators conservatively and properly responded to the RCS problems. Licensee
management responded to, and devoted adequate resources to, each individual RCS event,
developed and resolved the apparent causes for each event, and required the testing of the
RCS before authorizing each startup.
The inspection also revealed that the rod control system is vulnerabl.e to a single component
failure that could result in inadvertent outward control rod motion when inward motion is
selected. Although the reactor protection system is independent of the rod control system
logic and, therefore, the scram function is not compromised,. there remains a concern that a
previously unanticipated single-failure mechanism may exist in the control system that can
initiate or aggravate reactivity excursions and result in fuel failure. On June 21, 1993, the
NRC issued Generic Letter 93-04, "Rod Control System Failure and Withdrawal of Rod
Control Cluster Assemblies," to notify all licensees of the problem and to request information
describing their plant specific findings and actions taken related to this issue.
DETAILS
1.0
INTRODUCTION
1.1
Rod Control System Failure Overview
Salem Unit 2 experienced multiple failures of the rod control system (RCS) during startup
following the cycle 7 refueling outage. Five plant startups were attempted with the last one
on June 3 being successful. However, on the first three attempts the RCS failed to move
correctly and maintain the control rods in the desired position. During the fourth startup, the
rod insertion limit computer did not respond as expected. The most serious failure occurred
during the second startup attempt on May 27, 1993, when during an attempt to withdraw
shutdown bank A, the operator observed that the analog rod position indication (ARPI) did
not indicate that the control rods were being withdrawn although the group step counter
showed demanded outward motion. The operator then attempted to insert shutdown bank A
to step 6; however, one control rod (1SA3) withdrew 8 steps while the group step counter
counted down from 20 steps to 6 steps. The operator then inserted the shutdown bank to
zero. At that time the ARPI indication for control rod 1SA3 indicated 15 steps. The plant
operators removed the power from the rod by pulling fuses and rod 1SA3 dropped to the
zero position by ARPI indication.
Preliminary NRC inspection revealed that the aberrant rod motion during the second startup
attempt may have been caused by a previously unrecognized single failure of the rod control
system. In response to NRC questions about the single failure issue and after consultations
with the RCS vendor (Westinghouse), PSE&G, on June 4, 1993, decided to bring Unit 2 to
Mode 3, hot standby. All the rods were inserted in the core and the reactor trip breakers
were opened to prevent any spurious rod withdrawal. Unit 1 continued to operate at 100%
power. The licensee submitted a justification for continued operation (JCO) for Unit 1 that
was accepted by NRC. Unit 1 tripped on June 8, 1993, due to an unrelated problem with
the circulating water system. Unit 1 started up on June 18, 1993, after NRC approval of the
PSE&G safety evaluation.
On June 22, 1993, PSE&G started RCS system retesting after extensive printed circuit card
replacements. During the test more failures occurred in the RCS. These failures were
unrelated to the previous failures and were most likely caused by a technician while he was
installing and removing jumpers (a device used to connect normally-isolated portions of a
circuit). On June 28, 1993, Salem Unit 2 commenced a reactor startup and the main
generator was connected to the electric grid on July 3, 1993.
1.2
Augmented Inspection Team (AIT) Formation
On June 5, 1993, senior NRC managers determined that an AIT was warranted to gather
information on the rod control system failures at Salem Unit 2. The AIT met with PSE&G
management and staff on June 6. PSE&G gave a presentation for the AIT on the facts
known at that time. A charter was formulated for the AIT and transmitted to the team on
June 7, 1993 (Attachment 1).
5
On June 8, 1993, NRC Region I issued a confirmatory action letter (CAL) that documented
the verbal commitments made by the licensee to the NRC regarding the rod control system
problems, the continued operation of Unit 1 and the subsequent restart of Unit 2.
The AIT completed initial inspection activities on June 17, 1993. PSE&G met with the team
and NRC management on June 18 to discuss their findings, corrective actions, and restart
plans. Additional on-site inspection was performed from June 23 through June 27 when
additional component failures occurred. This included the witnessing of final rod control
system testing and resolution of hardware problems. The actions agreed to in the CAL were
completed on June 27, 1993. The AIT charter was completed and the inspection terminated
on June 28, 1993.
1.3
Rod Control System Description
The Westinghouse rod control system (RCS) uses solid state logic and power electronics
packages to provide control and drive signals to the electro-mechanical ("Mag-Jack") control
rod drive mechanisms (CRDMs). (Refer to Attachment 2 for a system diagram). The
system commands are either a manual in or out signal generated by the reactor operator, or
an automatic signal generated by a difference in turbine power and its associated
temperature/pressure requirements, the actual reactor power, and the average temperature of
the reactor coolant (T.vJ*
The system develops a pulse train based on desired control rod direction and speed. The
system fixes the single speed for manual operation. If the signal comes from the automatic
part of the system, the difference between the required T.vg and the actual T.vg , generates the
speed signal. A higher difference in temperature* will cause a higher rod speed. This pulse
train ultimately generates the current pulses to the three coils in each CRDM. Because of
the core pattern, banks of rods consisting of one or two groups move in a predetermined
sequence. The logic cabinet provides the pulse train and the logic of which banks and
groups are to move. The logic cabinet also provides the memory to know where the rods are
in the sequence.
The logic cabinet also provides a signal to the control room group step counters to show
where the RCS has commanded the rods to be within the core. A separate analog or
individual rod position indication (ARPI) system that monitors actual position of each rod
(within a limited accuracy) shows the operator the actual rod location. There are limits on
the allowable mismatch between the RCS commanded position and the ARPI. These limits
require the operator to take corrective action to return the rods to the required position.
'-._
The power cabinets take the logic pulse train and convert it into a set of firing orders for
semiconductor power devices. These devices control the flow of power to the three CRDM
coils associated with each rod in a control rod group. When control rods are not being
moved, the cabinet powers the stationary holding coil to hold the rod at the desired position.
When control rod motion is required, the cabinets supply power in sequence to the movable,
6
lift and the stationary holding coils for each rod in the group. The "in" or "out" motion
command determines the sequence in which power is supplied to each coil, which then
controls which direction the rods move.
2.0
GENERAL SEQUENCE OF EVENTS
On March 16, 1993, Salem Unit 2 experienced an automatic reactor trip when steam
generator water levels reached their low level setpoint following the loss of one steam
generator feedwater pump. The trip occurred four days prior to the scheduled start date for
the Unit 2 seventh refueling outage. Plant management decided to enter the refueling outage
early and the unit was not restarted following the trip.
PSE&G had scheduled the implementation of a design change package (DCP) involving the
RCS power cabinet power supplies and a Westinghouse preventive maintenance service on
the RCS during the outage. The power supply DCP, which involved power supply
replacements and the rearrangement of the grounds on four power supplies within each of the
five power cabinets, was completed without incident. The Westinghouse preventive
maintenance service consisted of an inspection and repair, as necessary, of all of the circuit
cards in the RCS logic and power cabinets. During this work, four of the control rod group
counters in the main control room were replaced. One of the new counters came from
PSE&G's inventory and the other three were acquired from Florida Power and Light's
Turkey Point Station. During the retest of the new counters, several solid state components
were found to have failed on RCS system printed circuit boards. Repairs were made and the
system testing satisfactorily completed.
At the end of the outage, just prior to unit startup, the licensee conducted ARPI calibrations
on May 24 and 25. During these calibrations, additional problems with group counters for
control bank A, group 2; control bank C, group 1; and control bank B, groups 1 and 2 were
identified and repairs made by PSE&G.
On the evening of May 25, PSE&G initiated the startup of Unit 2 and began to withdraw the
control rods prior to diluting the primary coolant boron concentration to achieve criticality.
The licensee suspended the startup when the group counter for control rod control bank C,
group 1, stopped counting out while that group was being withdrawn. All control rods were
reinserted by the operators, and troubleshooting was initiated. The licensee identified and
repaired a number of problems, including shorted diodes on circuit cards in the RCS logic
cabinet. Subsequent to a RCS retest, PSE&G attempted a second startup on May 27.
During this startup, when operators attempted to withdraw shutdown bank A, no rod motion
was indicated on the ARPis and when the operator demanded a rod insertion for the bank,
rod 1SA3 actually stepped out 15 steps. The licensee removed power from the rod to return
it to the bottom of the core. Following this aborted startup, PSE&G identified and repaired
7
two additional failures on RCS circuit cards. After consultation with the RCS vendor,
Westinghouse, the licensee performed additional testing to demonstrate system operability
and, on May 28, undertook a third startup.
During the third startup, all control rods performed satisfactorily as they were withdrawn to
their critical position. As operators began the dilution to criticality, all four rods in control
bank C, group 1, fell completely into the core. Unit 2 operators entered the Abnormal
Procedure for a dropped rod event and subsequently manually tripped the plant, returning all
rods completely into the core. At this point, the Salem General Manager initiated a
Significant Event Response Team (SERT) to investigate the root cause of the RCS problems,
and a round-the-clock troubleshooting effort commenced to repair the RCS. The licensee
undertook an extensive investigation into the RCS and found faults on several more circuit
cards. These cards were repaired, and the licensee concluded that, because the faults were
similar to those that had been found and repaired previously, they had propagated a fault
through the RCS through their earlier troubleshooting efforts. Following this investigation,
PSE&G developed a root cause for the RCS failures and conducted additional testing to,
again, verify RCS operability. Satisfied that the system had been returned to normal,
PSE&G conducted a fourth startup on June 2, 1993.
The control rods behaved normally during the fourth startup; however, operators noted that
the pulse-to-analog converter and the rod insertion limit computer were not registering rod
motion while control banks Band D were being withdrawn. PSE&G management elected to
shut the unit down to investigate. The licensee found a defective component on a circuit
card in the RCS logic cabinet and attributed the problem to incomplete troubleshooting
following the third startup attempt. Once this additional card was repaired, operators started
up Unit 2 on June 3, 1993. This fifth startup took place without incident; however, PSE&G
decided to voluntarily shut down the unit on June 4 when it became evident that the problems
encountered on the second startup (when rod 1SA3 withdrew on an insertion signal) may
have been the result of a single failure, an event that is outside the plant's licensing basis.
Beginning on June 4, PSE&G, with assistance from Westinghouse, began an in-depth
investigation into all the problems that had been encountered with the Unit 2 RCS and an
analysis of the plant licensing basis. Additionally, the NRC dispatched an Augmented
Inspection Team (AIT) to Salem on June 5, 1993, to determine the cause of the RCS
problems that had occurred at Unit 2, to assess PSE&G's performance, and to develop any
generic applicability of those problems.
The AIT completed their initial on-site inspection effort on June 17, and PSE&G presented *
the results of their investigation to NRC Region I management on June 18, 1993. The root
.-
causes of the system failures had been identified, and the system repairs and modifications
were completed on June 21, 1993. System retesting commenced on June 22, 1993, during
which two additional transistor failures were identified. This resulted in additional circuit
cards being replaced. During the testing of the new circuit cards, one circuit board was
found to have an incorrect resistor installed and was then replaced with a spare card. System
8
testing was completed satisfactorily on June 27, 1993. The root cause for the transistor
failures was identified and Unit 2 commenced a reactor startup on June 28, 1993. See
Attachment 3 for a more comprehensive sequence of events.
3.0
ROD CONTROL SYSTEM CORRECTIVE MAINTENANCE/MODIFICATIONS
During the refueling outage, the only modification to the RCS was Design Change No. 2EC-
3136. PSE&G replaced obsolete power supply PS3 and PS4 (Lambda Model No. LM-261)
with new Lambda Model No. LCS-A-24-6795 for all five power supply cabinets. PSE&G
also replaced existing daisy chain neutral wiring between Lambda power supplies PS 1 to PS2
and PS3 to PS4 with an individual neutral wire for each power supply. This change was
done to minimize the risk of tripping the reactor during rework or replacement of any power
supply.
This change is similar to the change that was done at Salem Unit 1 under Design Control
Change No. lEC-3113. Unit 1 has been operating with this change without any problem
noted with the RCS. The team walked down the changes made to the power cabinets under
this design change and did not identify any differences in the installation between Unit 1 and
2. Based on the modification package review and the system walkdown, the team concluded
that the post-modification electrical configuration was equivalent to the original design and
that this design change did not cause the Unit 2 RCS failures.
In addition to the design change, the licensee had performed maintenance on the RCS. The
team reviewed all work orders (WOs) completed on the rod control system during the
outage. In'particular, WOs on an ARPI cabinet relay light (WO 930301114), insulation
resistance check on #2 CRDM (WO 931029004), and the repair/replacement of cracked outer
insulation of CRDM cables P-12 and P-8 for control rod 2SA3 and 1B4 (WO 930330102)
were reviewed. The team found no connections between these activities completed by
May 12, 1993, and the subsequent rod control system failures.
Work order 930421129 replaced four group step counters. The problems with the step
counters are discussed in Section 5.0.
4.0
EVALUATION OF PREVENTIVE MAINTENANCE ACTIVITIES
4.1
Vendor Work Scope
Approximately one month prior to the start of the Unit 2 refueling outage, PSE&G decided
to perform preventive* maintenance on the rod control system to ensure continued system
reliability. The work scope for the current outage was expanded beyond the routine
refueling outage preventive maintenance and testing to include a contract with Westinghouse
to perform a detailed cleaning, inspection, and testing of the rod control system. Minor
repairs and adjustments were to be performed by Westinghouse technicians. The
Westinghouse work scope also included the performance of static functional testing of the
..
9
installed rod control system printed circuit (PC) cards. Westinghouse had performed this
service at numerous reactor plants, including Salem Unit 1 in 1991.
4.2
Work Accomplished
The rod control system maintenance was performed in accordance with Westinghouse
procedure NSID-EIS-85-11, "Full Length Rod Control System Maintenance," which was
revised and approved for use at Salem Unit 2 as station procedure VSC.IC-PT.RCS-0003(Q),
"Full Length Rod Control System Maintenance." A summary of the work accomplished:
Pre-maintenance, as-found checks of the rod speed voltage, the oscillator-pulser
period, and resistance readings from the power supply neutral buses and the zero volts
bus to ground. The as-found readings were satisfactory.
A general precleaning inspection of the rod control cabinets and bus duct was
performed, the cabinets were cleaned, and a more detailed inspection was performed.
The inspection attributes included damaged or overheated components, wires touching
heat sinks, solder joint integrity, terminal tightness and correct lamp and fuse
installation. The identified deficiencies and the corrective actions taken were
documented in the procedure.
Following the cabinet cleaning, the power supply neutral bus and zero bus resistances
to ground were checked. Resistance read~gs were also taken on the current sensing
resistors. All readings were satisfactory.
Power supply checks were performed, and minor adjustments were made to ensure
the as-left voltages were within the procedure specifications. The overvoltage
protectors in the logic and power cabinets were also checked and minor adjustments
made to restore the trip points to the desired value.
The operation of the urgent and non-urgent alarms was checked, and the oscillator
and pulser periods were adjusted.
Various system logic tests were performed, and the operation of the control rod group
step counters was checked. Four group step counters failed the test acceptance
criteria and were replaced and satisfactorily retested; however, several circuit card
failures were also identified and corrected.
Current traces were taken using dummy coils in place of the normal lift, stationary,
and moving coils. All traces were as expected.
10
The Westinghouse work on the PC cards included the following:
Verification that each PC card is the correct configuration for the site. No problems
were identified with card configurations.
Visual inspection of the condition of the components on the PC cards. These
inspections identified cracked or pitted solder joints on 70 of the 184 (174 installed
cards and 10 spare) PC cards. The technicians repaired all deficiencies.
Performance of a static functional test of installed cards and spare cards using factory
test equipment to new card test acceptance criteria. Twenty-two cards failed the test
and were repaired and retested (this included 3 spare cards).
Performance of minor repairs on failed cards on-site using Westinghouse supplied
components and documentation of the failures. The cards that had problems identified
during the visual inspection and/ or functional tests were repaired by the Westinghouse
technicians.
PC cards requiring in-depth troubleshooting were to be returned to Pittsburgh for
repair or be replaced with a site spare. Three spare cards were utilized when
problems were encountered following the replacement of the group counters as
discussed above.
The Westinghouse technicians estimated that the number of repairs required to the circuit
boards was typical based on their experiences at other plants.
A more detailed review by
the SERT determined that the number of card repairs on Unit 2 were significantly higher
than the number of repairs required on Unit 1 .. Seventy Unit 2 cards required solder
connection repairs as compared to 28 on the Unit 1 cards. On both units the static card
testing failure rate was approximately 11 % .
4.3
PSE&G Oversight of Vendor Work
The maintenance procedure provided by Westinghouse was reviewed by PSE&G and revised
to ensure the appropriate station controls were incorporated and the procedure was approved
as a Salem station procedure. The PSE&G Instrumentation and Controls (I&C) department
supervisor ensured that the vendor technicians met the station requirements and were
qualified to work on the rod control system. The work was properly authorized by the shift
supervision, and the I&C supervisor coordinated the processing of the work order controlling
the maintenance activities. The rod control system work was performed primarily by
Westinghouse technicians who reported problems and status of the work to a PSE&G I&C
supervisor. The decision of what minor repairs were to be made was left to the discretion of
the Westinghouse technicians. The technicians documented the test data and deficiencies as
11
well as any repairs and adjustments that were made to resolve out-of-specification data and
hardware deficiencies. The completed work package was reviewed and approved by the I&C
supervisor.
PSE&G did not give Westinghouse specific direction as to the standards to which they
expected the work to be performed. For example,. the licensee did not specify criteria that
would be applied during the visual examination of the circuit cards. PSE&G expectations as
to how work was to be performed--such as soldering, installation of jumpers and working on
energized equipment--were not specified. PSE&G did not review the Westinghouse soldering
procedure to ensure that it met PSE&G standards. PSE&G personnel did not have a clear
understanding of the capabilities of the circuit card test consoles and were not aware that
some individual components on a card could be failed and not be identified during the bench
test.
4.4
Independent NRC Inspections
The NRC inspectors performed an independent inspection of several PC boards that were
installed in the logic cabinet to assess the effectiveness .of the preventive maintenance work
performed during the outage. The inspector also inspected four PC boards that were
removed from the power and logic cabinets and replaced with spare cards following the rod
drop event, which occurred on May 28, 1993. The cards were examined visually using a
JOx magnifying glass. The inspector identified solder connections that did not meet the
quality required by the PSE&G and Westinghouse soldering procedures; however, all of the
connections appeared to be adequate to ensure the circuit operability. On the stationary
gripper regulation circuit card, which was replaced as a result of the control rod drop
problem, the inspector identified degradation of the circuit board solder run. The circuit
traces had areas where the edge of the solder run had slivers of solder partially separated
from the trace. Most of the slivers were attached to the trace on one end and were parallel
to the solder run. However, several of the slivers had been displaced perpendicular to the
solder run to which they were attached and two appeared to bridge the gap between two
solder runs resulting in potential short circuits on the boards. PSE&G evaluated this
condition and determined that short circuits on these portions of the circuit board would have
resulted in the dropped control rods without a coincident failure alarm.
The inspectors reviewed the PSE&G technician training program for performing soldering of
circuit components. The course required the technicians to make high quality solder
connections of various configurations in order to become qualified. The inspectors reviewed
training records for several technicians involved in repair work performed on the rod control
system following the preventive maintenance effort and verified the technicians had
completed the training. An in-progress circuit repair was observed by the inspectors and
found to be properly performed resulting in a good quality repair.
12
4.5
Conclusions
The decision to perform the rod control system preventive maintenance effort was prudent.
Similar preventive maintenance previously performed on Salem Unit 1 had been performed
and subsequent system operation was reliable. Numerous deficiencies, which did not prevent
the system from operating in the past, were identified and corrected to avoid degradation to
an extent where future failures may occur. The technicians performing the work were well-
qualified, and the quality of their work was generally good. The identified deficiencies and
corrective actions were clearly documented in the work package.
The team concluded that PSE&G exerted minimal control over the vendor work effort. To a
large degree, the vendor technicians worked independently and applied their own standards
when evaluating system deficiencies. PSE&G did not specify inspection criteria to be used
during visual inspections of cards and components and for inspection of work performed such
as soldering.
The PSE&G personnel were not aware of the capabilities of the circuit card test consoles and
did not realize that the testers may not locate all failed .components on a circuit card. Cards
installed in the system after a successful bench test were assumed to be 100 % operable when
they could have failed components, such as shorted diodes.
The team concluded that preventive maintenance work was not the direct cause of the
numerous circuit card failures. However, the inspector concluded that a more detailed
circuit card inspection should have identified the degraded solder traces on the regulation
card and thereby avoided the rod drop event. Although the replacement of the group step
counters introduced a different model counter that contributed to the component failures, the
inspectors did not identify any inappropriate actions during the replacement activity.
5.0
EVALUATION OF TROUBLESHOOTING ACTIVITIES
5 .1
Background
As discussed in section 4.0, PSE&G decided to perform extensive preventive maintenance on
the rod control system during the refueling outage. During this work, on May 14, 1993,
Westinghouse identified that two step counters on the c.;ontrol room RCS panel failed an
operability test and two others marginally passed.
On May 16, 1993, a Westinghouse technician replaced the four counters with one unit from
the PSE&G stock and three obtained from the Turkey Point Nuclear Plant. During the
testing of the new counters, the first set of problems with the printed circuit cards occurred.
Westinghouse and PSE&G personnel performed the system troubleshooting, which involved
in-cabinet measurements and swapping of suspected bad printed circuit cards with good
circuit cards. The Westinghouse circuit card testers had been shipped to another site
following the completion _of that phase of the preventive maintenance work and, therefore,
13
were not available for troubleshooting. This effort found several failed components on the
supervisory data logging (SDL) and relay driver cards (RDCs).
During subsequent surveillance testing and \\plant startup attempts, additional failures were
encountered. The test consoles were returned to Salem on May 20, 1993, and PSE&G and
Westinghouse personnel continued to troubleshoot, repair, and replace printed circuit cards.
The problems included the failure of numerous integrated circuits, small signal diodes, and
power transistors. The integrated circuit compon~nts were from the Motorola High
Threshold Logic (HTL) family, a noise-resistant design. The failed logic circuits included
NAND, NOT and AND gates.
This effort culminated with PSE&G informing the NRC on June 2, 1993, that they had found
and repaired all of the failures, and that the reactor was ready for restart.
5.2
PSE&G Activities Prior to AIT Arrival
PSE&G experienced over thirty failures of solid state devices between May 16 and
June 5, 1993. When failures occurred in several areas in the logic cabinet, PSE&G decided
that the failures might be related and commenced a troubleshooting effort to identify the
relationship between the failures. Refer to Attachments 4 and 5 for a list of failures,
locations, and the final AIT determination of probable causes of the failures.
During the troubleshooting effort that began on May 16, 1993, the Westinghouse circuit *
cards testers were not on site, and the troubleshooting was performed by replacing suspected
failed cards with spare cards and the interchanging of similar cards within the logic cabinet.
The replacement of integrated circuit chips on failed cards was also performed. The
. capability to test functionally suspect cards was added to the troubleshooting methods on
May 20, 1993, when the card test units were returned to Salem. The troubleshooting was
performed on all shifts by PSE&G and Westinghouse persc;mnel. The Westinghouse role was
not specifically defined by procedure or contract and the technicians worked with the PSE&G
personnel as peers. The troubleshooting efforts continued through June 3, 1993, and resulted
in the identification and repair of numerous and often repeat component failures. The
troubleshooting for each rod control system malfunction ended with the identification and
repair of the specific component that caused the malfunction, even after repeat failures
continued to occur.
A PSE&G maintenance engineer initially concluded that the root cause of the failures was a
bad blocking diode on a relay driver circuit card. The cause of the initial failed diode was
attributed to random failure. PSE&G also concluded that, during the troubleshooting efforts
as the circuit card with the bad diode was moved from slot to slot, it caused failure of
several supervisory data logging circuit cards. The failed integrated circuit on the SDL
circuit card was then assumed to have caused additional diode failures. PSE&G finally
concluded that the interaction between failed diodes and integrated circuits was the cause of
all of the component failures except for a blown fuse in the 100 VDC power supply and a
14
failed auctioneering diode in the -15 VDC power supply. These failures were attributed to a
short circuit caused by technicians during the installation of test equipment.
Prior to the arrival of the AIT, PSE&G had not performed any safety evaluation to assess the
significance of the rod control system malfunctions and failures. Specifically, when control
rod 1SA3 withdrew during an insert signal, PSE&G did not investigate the potential that a
single component failure could have caused the event, thereby resulting in a condition that
was outside the design basis of the plant.
The plant management held a meeting to discuss the cause of the failures. During this
meeting, no one contradicted the root cause as the diodes, although subsequent interviews
revealed that several people had different theories. PSE&G informed the NRC of their root
cause determination during a June 2, 1993, phone call.
An AIT member went to the site on June 3, 1993, when PSE&G reported that another SDL
printed circuit card had experienced a failure during a plant startup. The inspector reviewed
the results of the troubleshooting and repair efforts and interviewed involved personnel to
determine whether the root causes identified by PSE&G were consistent with the available
information. The station troubleshooting procedure, SC.IC-GP.ZZ-0006(Q), "Controls
Equipment - Troubleshooting" was also reviewed.
The AIT member also reviewed the system design to determine whether a single component
failure could have resulted in the 1SA3 control rod moving out when rod in-motion was
demanded.
Conclusions
The team concluded that, prior to the AIT arrival, the troubleshooting efforts lacked clear
leadership and delegation of responsibilities. Personnel on each shift would isolate and
eliminate the cause of the most recent system malfunction; however, no one had the overall
responsibility to identify and correct the root cause, even after repeat failures continued to
occur. The troubleshooting procedure does not require the identification of the reason for a
component failure. When a root cause was presented, it was accepted by plant management
even though there was no analysis or testing performed to ensure it was a valid explanation
for the failures.
The team also concluded PSE&G had not evaluated the rod 1SA3 withdrawal event for a
single failure design vulnerability since it was assumed that the problem observed had been
the result of two component failures. The team found that the failure of one logic gate on a
slave cycle decoder circuit card would cause the same CRDM current signals as those
observed during the inadvertent outward rod motion on rod 1SA3.
15
5.3
Actions After AIT Arrival
After the arrival of the AIT, PSE&G formed a task force composed of elements of their
Engineering, Operations, Systems, Maintenance, and Training departments, as well as
Westinghouse Engineering and Licensing department personnel. PSE&G tasked this group
with finding the root cause of the failures and investigating the reason why one rod in a bank
of eight moved out and the other seven did not.
PSE&G, with Westinghouse assistance, used the utility's control rod system simulator in the
training center to investigate the failures. They introduced the component failures found in
the plant RCS and were able to reproduce the current traces to the CRDM coils that had
been taken in the plant after the single rod out event. Further, they demonstrated that a
single failure would cause outward rod motion with an inward demand. Again, the current
traces observed during this demonstration matched those taken in the plant as well as the
traces predicted by Westinghouse Engineering.
PSE&G also simulated the diode failure, assumed to be the cause of the component failures,
in the training center. They did not observe any integrated circuit failures similar to the
plant failures. This finding contradicted PSE&G's original root cause of the integrated
circuit failures and validated the AIT conclusion that the original root cause determination
was in error.
PSE&G put in place a troubleshooting team with a single leader and, utilizing PSE&G and
Westinghouse personnel and the Westinghouse card testing equipment, a bench test unit was
set up in an effort to validate suspected root causes.
During this testing, PSE&G learned that the new group step counters that were installed on
May 16, 1993, had different electrical characteristics than the previously installed units.
These different characteristics resulted in higher back electromotive forces (EMFs), when the
counter coils were pulsed, than those seen with the original counters. A surge suppression or
"wheeling" diode is installed to shunt this back EMF around the coil to protect the solid state
components that drives the coil. This diode is on the relay driver circuit (RDC) board and is
not related to the blocking diodes that failed. PSE&G learned that the old counter EMF was
such that, even without the wheeling diode in the circuit, the voltage spike produced
probably would not damage the connected transistor, nor would it propagate to any other
parts of the circuit. Without the wheeling diode, the new counter would exceed the transistor
rating and would sometimes cause the transistor to saturate or not tum off when the signal
was removed from the transistor. PSE&G noted that they had seen evidence of this
condition, in the form of condensation on the group counter face due to coil heatup, during
their troubleshooting during the time prior to the team's arrival. The coil heating resulted
when the transistor saturated causing continuous current flow through the coil instead of the
normal momentary pulses. Initially, PSE&G did not identify any relationship between the
condensation on the group counter face and the component failures. The connection was
made when a counter exhibited the same condensation condition during bench testing.
16
PSE&G also noted that the pulse without the wheeling diode was strong enough to propagate
to the integrated circuit on the SDL card output. They noted voltages of about 26 volts at
this point during the shop simulation. Since the integrated circuit is rated at a maximum of
20 volts for one second, the 26 volt or greater spikes could cause the integrated circuits to
degrade and fail.
5.3.1 Component Failure Analysis
PSE&G sent several failed integrated circuits to the manufacturer, Motorola, for a failure
analysis. Motorola's conclusion was that an over-voltage event caused the damage seen in
the integrated circuits. They found some integrated circuits failed, while others were
degraded and starting into failure. PSE&G also sent one integrated circuit to Westinghouse
and several diodes were sent to Phillips Electronics. The organizations, with one exception,
listed the cause of the failure as voltage overstress. In one case, electrostatic discharge
(ESD) was identified as the likely failure mechanism. Motorola informed PSE&G that,
while they used the term BSD, that did not necessarily mean the failure was caused by
handling, but by some voltage greater than 100 volts. These observations were consistent
with the events seen at the plant. Motorola also noted that voltage spikes can also degrade or
fail integrated circuits that are not in the immediate circuit subjected to the voltage spike.
5.3.2 PSE&G Conclusions
Based on the above, PSE&G concluded that the most likely root cause of the solid-state
component failures was the addition of the new counters, coupled with a bad printed circuit
card connection in the RCS logic cabinet that prevented the suppression diodes from
connecting to the circuit. The counters obtained from Turkey Point function properly in the
system as long as the surge suppression feature is present.
The potentially bad pin connection would have likely been in the connector in the RCS
cabinet. The pin connector on the printed circuit card was visually inspected during the
original Westinghouse maintenance work. PSE&G noted that they had straightened a few
pins during the maintenance and troubleshooting, but were not sure if they had checked or
straightened this particular pin. No pin problems were identified during inspections
performed subsequent to the identification of the voltage spike problem.
5.3.3 Corrective Actions
PSE&G replaced the 74 printed circuit cards in the logic cabinet to ensure that all damaged
or degraded components, as a result of the voltage spikes, were eliminated from the system.
Additional suppression diodes were also added directly at the counter coil terminals to
eliminate the possible loss of the suppression circuit should the pin connection degrade. The
system was then retested using the applicable portions of the test used during the
Westinghouse preventive maintenance work. During the testing, PSE&G noted that an
"Urgent Alarm" that should have annunciated at the shutdown bank C and D (SCD) power
17
cabinet did not illuminate. They tested several printed circuit cards and found that the signal
processing card and the alarm card in the SCD cabinet each had a failed transistor (Q9). The
transistors were replaced and system testing was satisfactorily completed.
All of the circuit cards had been bench tested satisfactorily prior to installation and had
performed normally during the initial system retest. The failed transistors were sent to a test
facility and were both found to have been damaged by an apparent overcurrent event that
resulted in the catastrophic failure of the transistor emitter. PSE&G then reviewed the
circuits involved as well as work that was in progress at the estimated time of failure. They
determined that the transistor failures occurred at about the time that two failure detector
cards were reinstalled. These cards had been removed earlier as directed by the test
procedure. During the time the cards are removed, a "card removed" alarm signal is
disabled by the installation of a jumper into the backplane of the power cabinet card cage.
At the time of the failure both Westinghouse and licensee personnel were involved in the test
activities. A licensee instrumentation and controls (I&C) technician was tasked with the
installation and removal of the two jumpers. The jumper installation and removal was
normally performed by Westinghouse technicians. The jumpers are installed in the card
connection adjacent to the signal processing and alarm cards. In one case, the access space
to the card cage connector is only about two inches wide making the exact placement of the
jumper difficult. The jumpers are small "h"-shaped assemblies that are designed to jumper*
across two card connector pins. The size and shape of the jumpers also contributed to the
difficulty of the jumpering task and increased the probability of installing the jumper across
the wrong set of pins. After reviewing the affected circuitry on the signal processing and
alarm cards, the transistor failure mode, the jumper installation and removal technique and
the proximity of the jumpering activity, PSE&G determined that the most likely cause of the
transistor failures was momentary short circuits caused by the jumpering activity. PSE&G
determined that, in one case, the I&C technician inadvertently shorted two pins in the back
plane of the card cage and, during another step, shorted two adjacent traces on the back of
the printed circuit card. PSE&G determined that these incidents caused a misbiasing of the
transistors and the resultant failures. Due to the low voltage and current levels involved, the
momentary short circuits caused by the jumpers would not have been readily discernible by
the technician.
As a result of the transistor failures, PSE&G replaced 13 circuit cards in the SCD power
cabinet. During the retest of these cards an incorrect resistor was identified on one of the
new cards. See section 10.1.5 for details. This card was again replaced and the system
retested satisfactorily.
5.3.4 AIT Assessment
The team reviewed the PSE&G troubleshooting results, component failure analyses, and
engineering evaluations associated with the RCS. The team concluded that unsuppressed
18
voltage spikes generated by the new group counters were the likely cause of the multiple
solid state component failures. The team also concluded that the corrective actions taken by
PSE&G were appropriate.
The team evaluated the PSE&G root cause evaluation for the Q9 transistor failures and
concluded that it was a reasonable explanation for the failed transistors. The team noted that
the poor jumpering techniques, which PSE&G initially failed to identify and correct, were
eliminated by the use of a modified circuit card in the place of the jumpers.
5.3.5 General Observations
The team occasionally observed the technicians handling the circuit cards by capacitors or
transistors mounted on the cards. This practice can mechanically stress solder connections
and may have caused some of the cracked solder joints observed by the team.
The team also noted that PSE&G technicians and the vendor technicians handled the printed
circuit card such that their hands contacted the traces and components on the board.
Although the component families selected for these boards are resistant in the area of
electrostatic discharge (BSD) damage, the handling could cause damage to the solid state
devices on the board.
PSE&G has since implemented proper handling techniques.
5.4
Summary of Component Failures
Attachment 4 summarizes the repeat failures of printed circuit cards in the logic cabinet slots
Al13, Al14, A713, and A714. Slots A113 and A114 are locations of the supervisory data
logging (SDL) cards, and slots A713 and A714 are locations of the slave cycler decoder
(SCD) cards. Attachment 5 is a list of failures in the RCS that occurred after the completion
of preventive maintenance activities. This attachment includes the corrective actions taken
and the most probable root cause of each failure. The root causes listed were determined by
PSE&G following the arrival of the AIT. The team independently reviewed the available
information and concluded that the root causes assigned by PSE&G were the most likely
causes of the failures. In one case, for the bank overlap card, PSE&G had not determined
the cause of the failure and the card was sent to Westinghouse for a failure analysis. The
team concluded that electrical overstress was the most likely cause of this failure.
6.0
RESTART DECISION PROCESSES
6.1
Description of PSE&G Actions
~.*~*
~-
Following the work performed on the RCS during the Unit 2 seventh refueling outage,
including the Westinghouse preventive maintenance on the circuit cards in the RCS logic and
power cabinets and the successful performance of ARPI calibrations, Salem management had
19
no apparent reason to suspect the operability of the RCS. The initial startup conducted on
May 25, 1993, was performed with I&C engineers present on 12-hour shifts, providing
normal startup coverage. It was these engineers who initiated the troubleshooting subsequent
to the shutdown performed when the control bank C, group 1, counter failed. PSE&G found
a number of component problems on the circuit cards in the RCS, including several shorted
blocking diodes. PSE&G determined the root cause of this first evidenced problem to be the
shorted diodes and that the movement of those failed diodes throughout the RCS during
troubleshooting had caused the large number of other problems. After repairs, PSE&G
verified the proper operation of the RCS and the group counters by stepping the counter for
each control rod group "OUT" for the full range of 228 steps and then cycling the counter
"IN" for 228 steps. Following the successful completion of that test, the Salem General
Manager (GM) authorized a unit restart.
The second Unit 2 startup was abandoned when shutdown bank A did not withdraw after
being commanded 20 "OUT" steps and then rod 1SA3 withdrew 15 steps when the bank of
rods had been given 20 "IN" commands. Additional licensee troubleshooting identified two
slave cycler decoder cards with bad ICs, the effect of which produced irregular current order )
signals to the control rod drive mechanisms (CRDMs) .. PSE&G repaired the defective cards
and consulted with Westinghouse to confirm that the action of shutdown bank A was
expected given the two identified failures. Westinghouse explained to PSE&G that the
current orders produced by the failed circuit cards would not have damaged the CRDMs.
Once the faulty cards were repaired, the licensee exercised the RCS and the CRDMs by
stepping every control rod group "OUT" 20 steps *and then "IN" 20 steps. When all group
counter and ARPI indications agreed and the rods behaved as expected, the Salem GM
authorized a unit startup. The GM later recounted to the inspector that he believed the
problems seen with the RCS were still due the propagation of faults through the movement of
the initial failed logic cards. The GM also related that PSE&G had not considered the 1SA3
rod withdrawal event to be contradictory to the single failure criteria for the RCS because
two failures had occurred.
After the four rods in control bank C, group 1, dropped into the core during the startup
attempt on May 28, and a manual reactor trip was performed, the Salem GM chartered a
Significant Event Response Team (SERT) to investigate the root cause of the RCS problems
and to assess the GM's decisions to restart the unit following each failure. Following the
May 28 event, the licensee also expanded their troubleshooting effort to a three shift round-
the-clock team approach with each shift consisting of a maintenance engineer, I&C
technicians and supervisors, system engineering personnel, PSE&G training center personnel,
and Westinghouse personnel. In accordance with Salem procedure AD-16, "Post-Reactor
Trip/Safety Injection Review Report," a root cause determination was required before restart
could be authorized by the GM following the plant trip. The licensee's troubleshooting effort
found circuit card component failures similar to those identified previously and separate
evidence of a faulty firing card in the RCS power cabinet. All logic cabinet cards were
repaired, and the licensee replaced the identified firing card and three other cards associated
with the group of dropped rods. The root cause determination for the RCS failures
20
submitted to and accepted .by the Salem GM cited the original shorted blocking diode and.
subsequent associated troubleshooting as one root cause, and a random and intermittent
failure of the firing card as another root cause. The GM accepted this root cause
determination and, after additional extensive functional testing of the RCS, authorized
another Unit 2 startup.
The failure encountered during the fourth startup of Unit 2 involved the pulse-to-analog
converter and the rod insertion limit computer input for control banks B and D. PSE&G
identified a failed chip in a location identical to a number of previously failed ICs. Station
personnel believed this failure to be a previously existing failure that was missed during the
extensive troubleshooting efforts of May 28-June 1. The card was repaired, and the rods
were again exercised to assure proper rod behavior and group counter and ARPI agreement.
With the testing satisfactorily completed, PSE&G performed the fifth startup of Unit 2. This
startup was completed successfully, and low power physics testing was underway when
PSE&G decided to voluntarily shut down the unit pending the full investigation into the
single failure and licensing basis issues.
Prior to each Unit 2 startup, the Salem Operations Department performed the requirements of
integrated operating procedure IOP-3, "Hot Standby To Minimum Load." This procedure
requires the senior nuclear shift supervisor to obtain all necessary Department sign-offs
certifying that the unit is prepared for startup and to obtain the Operations Manager's
authorization to change the unit's operating mode.
6.2
Assessment
A team member monitored the licensee's actions from May 25 to June 4, 1993. During that
period, the inspector maintained contact with Salem plant management and observed many of
the licensee's actions. Subsequent to June 4, the team reviewed operators' logs and
procedures used during the five startups and shutdowns, and interviewed the operators, shift
supervisors and plant management that had been involved with the events.
The team determined through direct observation and later review that Salem operators had
complied with all applicable procedures and Technical Specifications (TSs) during the five
startup and shutdown events. The operators displayed good awareness of plant conditions
and quickly identified and responded to the RCS abnormalities during the startups. Salem
management properly supported the shutdown of the unit once the abnormalities were
identified. The inspector verified that the licensee had satisfied all procedural and TS
requirements prior to the restart of the plant subsequent to each RCS event.
The team noted that PSE&G management recognized and responded to each RCS problem as
it occurred. PSE&G had prepared for potential problems by having I&C engineers on 12-
hour shifts for startup coverage and expanded the troubleshooting effort subsequent to each
startup to resolve the RCS problems. Salem management remained involved in those
troubleshooting efforts and challenged the technical staff to find and repair the cause of each
21
event. Licensee management also properly involved the system vendor to assist in problem
investigation and to verify system operability following the unexpected rod 1SA3 withdrawal
event. The inspector determined that Salem management, including the GM, devoted
adequate resources to the investigation of each RCS event and that management did not
initiate any unit restart before they thought that each individual problem had been identified
and resolved.
Despite the Salem management and technical stafrs attention to and resolution of each
individual RCS problem, the team identified a weakness in the licensee's pursuit and
consideration of a root cause for the sustained problems encountered with the RCS. See
section 7. 0 for further discussion.
As the inspection progressed, the team noted a significant improvement in the licensee's
control of troubleshooting and identification of root cause for component failures. This was
particularly evident during the resolution of the Q9 transistor failures of June 23 and the
unexpected urgent alarm that occurred on June 25, 1993. The systematic troubleshooting and
in-depth root cause analyses identified a plausible explanation for the transistor failures and
identified a specific incorrect circuit card component as .the cause for the unexpected alarm.
6.3
Conclusions
The team determined that PSE&G satisfied all procedural and TS requirements prior to
initiating a Unit 2 startup, that operators properly responded to the RCS problems, and
station management responded to and devoted adequate resources to each RCS event.
Proximate causes were developed for each event, and testing of the RCS was performed
before each unit startup. The team also concluded, however, that the PSE&G restart process
did not provide for the programmatic determination of a root cause for the RCS problems
and, had such a determination been performed, that the repetitive startups and shutdowns of
Unit 2 could have been avoided.
7.0
ROOT CAUSE POLICY AND PROCEDURES
The team reviewed the station policy and procedures relative to root cause determination
particularly as it applies to troubleshooting, maintenance and unit restart.
Station procedure SC.IC-GP.ZZ-0006(Q), "Controls Equipment - Troubleshooting," is used
to control troubleshooting of instrumentation and controls systems. A similar procedure
exists for mechanical sys~ems. Sectio~ 5.9 of the procedure require~ the user to "Determine ft.
the cause of the malfunction by recordmg as found data and companng to the expected
performance as derived from the procedures, ICD Cards, valve data cards or vendor
manuals." This is the extent of guidance provided in this procedure relative to determining
root cause of the equipment failures.
The work control system at Salem requires the completion of a cause section per procedure
NC.NA-AP.ZZ-:0009(Q), "Work C<;mtrol System." This procedure lists thirteen alternatives
22
for the cause of an electrical/electronic controls problem; however, no guidance is provided
for the user of the procedure. Absent guidance, the maintenance craft--with supervisory
help--would have to determine cause and classify that cause within a limited set of general
categories.
The AD-16, "Post-Reactor-Trip/Safety Injection Review," report (see section 6.1) specifies
that the root cause be determined for the component failure prior to startup. By this .
procedure, it is left to the General Manager to accept Engineering's determination of root
cause. For failures not resulting in a plant trip or safety injection, there is no guidance in
the station procedures on when a root cause determination should be required and whether a
determination should be completed before considering the system operational.
The team identified instances where the cause was simply listed as componetit failure or
random failure without any explanation for what may have caused the component failure.
When a shorted blocking diode was found during the initial RCS troubleshooting, the root
cause was attributed to random failure. As discussed in section 5.2, the PSE&G
management then attributed integrated circuit failures to the failed diodes. As discussed in
section 5. 3, subsequent testing in the shop and the training center showed this cause was
incorrect. The multiple card and diode failures, which were identified, were assumed to be
the result of moving circuit cards around to various locations during the troubleshooting and
thereby promulgating additional failures. The initial troubleshooting focused primarily on
finding and replacing the failed components with minimal focus on establishing why the
particular component failed. Postulated root causes were not tested to ensure they had a
sound engineering basis. PSE&G eventually concluded that the circuit card with the failed
diode would function normally even with the diode short circuited and that there was no
plausible explanation for why the shorted diode would result in other component failures.
Further, three members of the plant staff involved with the troubleshooting stated to the team
that they did not believe the root cause as described in the post-trip review document for the
May 28, 1993, manual reactor trip. They also reported that they did not air their concerns
to management at the time.
The team concluded that there was no clear station policy or expectations on how, when and
to what extent to perform root cause analysis for component failures. PSE&G management
expected the plant staff to use engineering judgement to determine how far to pursue the root
cause of component failures. There also is no clear policy on whether or not a system can
be considered operable without having determined the root cause of any failures.
8.0
COMPENSATORY l\\1EASURES/JCO
PSE&G decided to shut down Unit 2 on June 4, 1993, to assure the safety of the plant while
the root cause, potential safety-significant effects and licensing basis relevance of the RCS
problems were investigated. At the time, Unit 1 was operating at 100% power, and the
licensee had not experienced any problems with the Unit 1 RCS. On June 8, 1993, the NRC
Region I Office issued a Confirmatory Action Letter (CAL) to PSE&G, which affirmed the
l
23
licensee's commitment to maintain Unit 2 shut down until the above-mentioned investigation
was satisfactorily completed and to prepare a Justification for Continued Operation (JCO) to
assure the continued safe operation of Unit 1.
Between June 4 and June 8, PSE&G prepared the JCO for Unit 1, which was approved by
the Salem Station Operations Review Committee (SORC) on June 8, 1993. The JCO
considered such factors as the current plant status, RCS alarms and indications, operator
training, the unit licensing basis and accident analyses, and demonstrations of RCS
operability. The JCO also considered compensatory actions to be taken by unit operators,
including the placing of the RCS into the MANUAL mode in order to prevent automatic rod
motion without operator cognizance. The SORC approved the JCO for Unit 1 operation in
Mode 1 (Power Operation); however, within an hour and a half of the approval, Unit 1
tripped off-line due to the loss of the main condenser system when river debris clogged the
circulating water system travelling screens. With Unit 1 forced to enter Mode 3 (Hot
Standby), the JCO was no longer valid, because it did not consider the effects of restarting
the plant.
On June 15, 1993, SORC reconvened to consider a new JCO, one that did include an
analysis of unit restart. This second JCO considered all the factors of the first JCO and the
additional accident analyses related to startup events. The new analysis showed that, even
given the possibility of an inadvertent rod withdrawal accident, protection of the reactor core
and fuel would be assured in all cases except an asymmetric rod withdrawal from a
subcritical condition. The licensee committed in the JCO that any startup would be
performed by first withdrawing control rods and then achieving criticality by dilution of the
primary coolant boron concentration, thereby avoiding the possibility of an inadvertent rod
withdrawal from a subcritical condition.
With the SORC approval of the second JCO, and its subsequent approval by NRC Region I
and NRR, PSE&G initiated a Unit 1 startup on June 18, 1993. Prior to the startup, the
licensee implemented a design change to the RCS in which additional suppression diodes
were placed in each group counter circuit to assure the blocking of the EMF pulses back into
the RCS circuitry. This design change, additional RCS surveillance testing and the
compensatory measures of the JCO were completed, and Unit 1 achieved criticality on June
19 without event.
The team monitored licensee performance throughout the period in which the JCOs and
compensatory measures were developed and implemented. The team noted very"good
cooperation between the Salem Operations and Technical Departments and PSE&G
Engineering and Plant Betterment in the development of the additional accident analyses and
compensatory measures. A team member attended all SORC meetings associated with the
JCO preparation and was present in the control room for Unit 1 startup.
24
The team concluded that the licensee's approach to the continued operation of Unit 1 was
performed in a deliberate and conservative manner.
9.0
SAFETY SIGNIFICANCE
The identified failure in the Salem RCS could have potentially resulted in the operation of
either of the Salem units outside their design basis. Licensee evaluation of the failure, in
accordance with 10 CFR 50.59, concluded that the potential single failure was an
Unreviewed Safety Question (USQ). PSE&G subsequently performed an engineering
evaluation to ensure the safe restart and operation of both Units 1 and 2.
The applicable portion of the Salem licensing basis, as expressed in the Salem Updated Final
Safety Analysis Report (UFSAR) sections 4.3 and 15.3.5, stated that multiple failures in the
RCS would be required for a single rod withdrawal to occur. The single rod withdrawal
event was, therefore, allowed to be considered as an ANSI N18.2 Condition III event
(Infrequent Faults), for which the acceptance criteria allowed a small percentage ( <5%) of
fuel failure due to the low probability of event occurrence. With the determination that a
single failure could cause an inadvertent rod withdrawal accident, the event had to be
reevaluated against the criteria of a Condition II event (Events of a Moderate Frequency).
These criteria required that the Departure From Nucleate Boiling Ratio (DNBR) limit is not
exceeded, and, therefore, fuel design limits are maintained. The licensee was also required
to demonstrate that the plant still met the intent of the General Design Criteria (GDC) 25 of
10 CFR 50 Appendix A, which requires the protection system be designed to assure that fuel
design limits not be exceeded for any single malfunction of the reactivity control systems
such as accidental withdrawal of control rods.
The PSE&G safety evaluation demonstrated that no fuel design limits would be exceeded for
any of the affected operating transients other than the asymmetrical rod withdrawal from
subcriticality. PSE&G addressed the potential for exceeding fuel design limits as a result of
asymetrical rod withdrawal from subcriticality through administrative controls and, thereby,
resolved the USQ. PSE&G, therefore, concluded that both Salem units complied with the
requirements of Condition II events and GDC 25. PSE&G's evaluation was predicated on
several factors: the relevant failure does not affect the ability of the reactor protection
system to perform its intended safety function; the failure is detectable based on periodic
surveillance testing and operator verification of control rod position; although not taken
credit for in the evaluation, alarms, administrative controls, and compensatory measures
implemented specifically for the event provided further assurance that the discovered failure
would not adversely affect the safety of the plant; and the evaluation bounded all possible rod
motions considered in the evaluation. The licensee recognized the evaluation was an interim
document and could be affected by design changes to the plant (i.e., the installation of digital
electronic group counters) or further industry-wide revelations and developments.
The team reviewed the Salem licensing basis, observed portions of the licensee's deliberation
on the matter, and reviewed the final versions of the PSE&G 10 CFR 50.59 documentation
25
and their engineering safety evaluation. The team verified that this event did not provide a
challenge to the reactor coolant system or the containment boundary and further concluded
that, although the safety significance of the issue is not high, PSE&G acted properly and
prudently in their investigation into and resolution of all licensing basis concerns. The team
concluded that PSE&G's safety determination was accurate and conservative.
10.0
CAUSES OF ROD CONTROL SYSTEM PROBLEMS
The team reviewed available information and the sequence of events to arrive at the root
cause of the rod control system failures. Where warranted, the team's determination is
qualified based on the available information.
10.1
HARDWARE PROBLEMS
10.1.1 Multiple Integrated Circuit and Output Transistor Failures
As discussed in section 5.3, the IC failures began to occur when three of the group counters
were replaced with models that had different electrical characteristics and generated higher
reverse EMFs than the original counters. The higher voltage spikes coincident with the loss
of the surge suppression capability resulted in the circuit components being subjected to
voltages significantly in excess of their design ratings. The surge suppression circuitry was
suspected to have been lost as the result of an open circuit caused by spread pins on the card
cage connector. These voltage spikes also exceeded the voltage rating of the output
transistors on the relay driver cards. While a bad pin connection could not be definitely
identified, symptoms of the loss of the suppression circuit were identified during the
troubleshooting and operation. For example, on one occasion condensation was seen on a
group counter face in the main control room.
This same condition was observed during *
bench testing when the suppression circuit was disconnected and the circuit card output
transistor went into saturation resulting in continuous energization and subsequent heatup of
the counter coil.
The team concluded the unsuppressed voltage spikes were the most probable cause of the IC
and transistor failures. Attachment 5 contains a list of failed components including those
likely to have failed as a result of these voltage spikes (electrical overstress).
10.1.2 Regulation Circuit Board Short Circuits
As discussed in section 4.4, the stationary gripper regulation circuit card was found to have
short circuits on the back of the card that was caused by slivers of the solder trace crossing
between parallel solder runs. The slivers appeared to have been present as a result of
problems during the manufacturing process and may have been disturbed during card
cleaning as part of the preventive maintenance work. PSE&G evaluated potential effects of
the short circuits and determined that they would have resulted in reduced current to the
stationary gripper coils and at the same time would have prevented an urgent alarm. The
26
reduced stationary gripper coil current could result in dropped control rods. The slivers
causing the short circuits were very fine and, therefore, susceptible to movement due to
expansion 'effects of temperature and physical orientation of the card, thereby maldng the
possibility of an intermittent fault more likely.
The team concluded that the circuit card short circuits were the most probable cause of the
dropped rods on May 28, 1993.
10.1.3 1/0 AC Amplifier, 1/0 Receiver and Slave Cycler Logic Card Failures
May 29, 1993, troubleshooting was being performed in an effort to determine the cause of
the dropped control rods that occurred on May 28. During troubleshooting, the 100 VDC
power supply fuses were found to have blown and a -15 VDC power supply auctioneering
diode was found to have failed. It is suspected that technicians installing test equipment to
support this troubleshooting caused a short circuit when, at one point, they were routing
leads through the power cabinet while having one end already connected. The short may
have resulted in 100 volts being applied to the -15 volt bus and would explain the failure of
components on several I/O AC amplifier circuit boards, the I/O Receiver card, and the slave
cycler logic card.
Based on the proximity of these failed components, relative to the counter and their
associated voltage spikes, the team concluded that a short circuit of the power supply was the
most likely cause of these failures. However, the effects of the unsuppressed voltage spikes
could not be absolutely eliminated.
10.1.4 Q9 Transistor Failures
As discussed in section 5.3.3, during testing of the rod control system following the
replacement of the logic cabinet circuit cards, a failed transistor was identified on both the
signal processing and the alarm cards in the SCD power cabinet. The test procedure directs
the removal of two failure detector cards and the installation of jumpers in their associated
pin connectors. The location of these cards and the associated jumpering is immediately
adjacent to the cards that experienced the failures. The jumper installation and removal is
performed with the cabinet energized and the licensee has identified locations where
momentary mislocation of the jumper or contacting the circuit cards with the jumper would
result in high currents through the transistors resulting in their failure.
The team concluded that high transistor currents, as the result of jumpering activities, was
the likely cause of the Q9 transistor failures.
10.1.5 Failure Detector Card Resistor
Due to the failure of the Q9 transistors, (discussed in sections 5.3 and 10.1.4), PSE&G
replaced thirteen circuit cards in the SCD power cabinet, including the failure detector cards.
27
On July, 25, 1993, during the system retest, an urgent alarm was received when attempting
to obtain current traces with dummy coils installed in place of the control rod drive coils.
Troubleshooting efforts isolated the problem to the failure detector card, one of the
replacement cards, which was then removed for inspection and bench testing. The followup
investigation by PSE&G identified that the value of the Rl 8 resistor was 50 kn instead of the
intended design value of 511 kn. With the incorrect resistor installed, the failure detector
card permitted the urgent alarm to actuate prematurely. The card had been obtained from
the plant spare parts inventory and apparently had_ never been installed in the RCS since with
the wrong resistor installed it would have caused urgent alarms during normal system
operation.
The team concluded that the most probable cause of this failure was a manufacturing error,
although the possibility that PSE&G replaced the resistor at some previous date could not be
eliminated.
10.2
ROD CONTROL SYSTEM DESIGN
The team reviewed the design of the RCS and concluded that the following design features
were contributing factors to the 1SA3 rod withdrawal problem and the hardware failure root
causes discussed in section 10.1.
10.2.1 Single Failure Vulnerability
On May 27, 1993, one control rod withdrew while control rod insertion was being
attempted. This unexpected out motion was the result of a logic circuit component failure
that resulted in the movable gripper coil, the lift coil, and the stationary coil being energized
simultaneously contrary to what occurs during normal control rod motion. PSE&G
subsequently confirmed that the failure of a single gate on an integrated circuit could result in
the current signals experienced on May 27 and, therefore, could result in rod out motion
when in motion is desired. This postulated failure was installed in the training center rod
control system; and, during testing, the single failure was verified to cause outward rod
motion with insert demand. The potential for a single failure to challenge the fuel design
limits may be contrary to the requirements of General Design Criteria 25 of 10 CFR 50,
Appendix A. The safety significance of this vulnerability is discussed in section 9.0, and the
potential generic implications are discussed in section 11. 0 of this report.
10.2.2 Voltage Suppression Circuit
As discussed in section 5.3, the surge suppression diodes, which are used to protect the solid
state components from voltage spikes that are generated by the group step counters during
normal operation are mounted on the relay driver cards. Several diodes have a common
connection that utilizes pin 4 on the relay driver circuit card as the connection point to the
circuit. In the event a poor connection develops between the card pins and the pins in the
card cage connector, the suppression diode is prevented from performing its function and
28
high voltage spikes are seen by the solid state components. These voltage spikes will result
in the degradation and failure of the components. The effects of the loss of the pin 4
connection may not impact the system operation and, therefore, be undetectable until solid
state component failures occur and result in RCS malfunctions. The licensee has eliminated
this vulnerability to a poor pin connection by installing additional suppression diodes directly
on the group step counter terminals and is considering the installation of digital counters in
the future. The digital counters do not contain coils and, therefore, would remove the source
of the high voltage spikes and eliminate the need for a suppression circuit.
10.2.3 Group Step Counter Compatibility
As discussed in section 5.3, during rod control system preventive maintenance and testing
during the refueling outage, four group step counters were replaced. Three of the
replacement counters are a different model than the installed counters. PSE&G reviewed this
difference with the counter manufacturer and Westinghouse and concluded that they were
suitable replacements. The coils in the new counters have the same design operating voltage;
however, the coil resistance and inductance ratings of the coils are different. The difference
in the electrical characteristics resulted in higher reverse EMFs being generated than those
developed by the original counters. These higher reverse EMFs, coincident with the loss of
the surge suppression circuit, result in damage to the solid state components. While under
normal conditions the new model counter is compatible with the existing rod control system,
the new counters make the surge suppression circuitry essential.
10.3
LICENSEE POLICY AND PERFORMANCE
The Salem station does not have well-defined policies and procedures governing the
determination of root causes of equipment failures. The lack of such a policy and associated
procedures contributed to the faulty root cause determination and management decisions to
attempt startups without the problem being resolved. Refer to section 7 .0 of this report for
additional discussion.
11.0
POTENTIAL GENERIC IMPLICATIONS
The inadvertent withdrawal of a single rod (1SA3) during the May 27, 1993, startup of
Salem 2, could have been caused by a single failure. This is in conflict with the Salem
Updated Final Safety Analysis Report Section 15.3.5.1, which states that no single failure
could cause a single Rod Control Cluster Assembly (RCCA) withdrawal. Also 10 CFR 50,
Appendix A, "General Design Criterion for Nuclear Power Plants," Criterion 25 requires
that fuel design limits should not be exceeded as the result of a single malfunction of a
reactivity control system.
The rod control system installed at Salem 2 is used at all Westinghouse designed plants
except the Haddam Neck plant. Hence, the failure that occurred at Salem 2 has generic
implications. The NRC issued Information Notice 93-46 on June 10, 1993, to inform
29
licensees about the potential problem with Westinghouse designed rod control systems. The
NRC has also activated the Westinghouse Owners Group (WOG) Regulatory Response Group
(RRG) to address the concerns. On June 11, 1993, Westinghouse issued a Nuclear Safety
Advisory Letter (NSAL)93-007, to all domestic Westinghouse nuclear power plants. A
meeting was also held with the WOG at NRC headquarters to discuss their preliminary
findings.
According to the WOG, the failure of the RCS at Salem 2 is a Condition III event and not a
Condition II event, as defined by ASNI Standard Nl8.2, based on the fact that a small
fraction of failed fuel is an acceptable consequence for this event. Based on the analysis
done by Westinghouse, the WOG considers that the RCS meets the ANS condition III
acceptance criteria. The NRR staff has issued Generic Letter 93-04 on June 21, 1993, to all
Westinghouse plants for action and B&W and C-E plants for information. The staff is
requesting licensees who operate Westinghouse designed plants to ensure that the licensing
basis for their plant is satisfied and provide a supporting discussion for that assessment. If
the licensing basis is not satisfied, then the licensees are requested to provide the short-term
and long-term, corrective actions that will be taken. Thus, the generic aspects of this event
will be reviewed by NRR for all Westinghouse plants. NRR is also evaluating if this event
could also occur at B &Wand C-E plants.
12.0
OTHER FINDINGS/OBSERVATIONS
In addition to the findings discussed in previous sections, the AIT had the following
observations:
Work orders did not always contain the necessary engineering evaluations for closure.
Specifically, during work order 931029004, a question was raised about the insulation
resistance value between the cable shield and ground and the completed work package .
did not document the resolution of this question. Also, during work on CRDM cable
P-12 (WO 930330102), the cable was repaired using fiberglass tape, when the work
package specified Raychem heat shrink, and an engineering disposition for the
alternate repair method was not documented.
Printed circuit cards were handled in a manner that could result in stressing and
potentially damaging solder connections and/or components. (Section 5.3.5)
Precautions were not taken to avoid damage to circuit card components that could
result from electrostatic discharge. (Section 5.3.5)
The team had the following positive observations:
Plant housekeeping and the material condition of the plant was excellent, in particular,
the relay room and rod control cabinets.
30
The PSE&G technical training center was a valuable resource for the investigation
and confirmation of the rod control system problems. The center was well-equipped,
staffed and managed.
The control room operators performed well when responding to the numerous rod
control system (RCS) problems encountered during the attempted startups. The
operators displayed good plant awareness during the May 25 and May 27 startups by
promptly recognizing the abnormal RCS behavior and properly implementing the
abnormal and emergency procedures for the dropped control rod event of May 28.
13.0
EXIT MEETING
A formal, public exit meeting was held with PSE&G management on July 7, 1993, at the
Artificial Island Processing Center. The slides used during the meeting are included as
Attachment 6. Exit meeting attendees are included as Attachment 7. People contacted
during the inspection are listed in Attachment 8. The team submitted written questions to
PSE&G during the inspection. The.se questions will be placed in the Public Document Room
under a separate cover.
Attachments:
1. Memo dated June 7, 1993, from Thomas T. Martin to Wayne W. Hodges
2. Logic Cabinet Automatic and Manual
3. Sequence of Events
4. Circuit Card Locations Experiencing Multiple Failures
5. List of Failures/Cause
6. NRC Augmented Inspection Team Exit Meeting
7. Exit Meeting Attendees
8. Persons Contacted
9. Acronyms and Initialisms
Docket Nos. 50-272
50-311
ATTACHMENT 1
UNITED STATES
NUCLEAR REGULATORY COMMISSION
REGION I
475 ALLENDALE ROAD
KING OF PRUSSIA, PENNSYLVANIA 19406-1415
MEMORANDUM FOR:
Wayne M. Hodges, Director, Division of Reactor Safety
FROM:
Thomas T. Martin, Region Administrator
SUBJECT:
AUGMENTED INSPECITON TEAM CHARTER FOR REVIEW
OF ROD CONTROL SYSTEM FAILURES AT SALEM 2
Due to control rod system failures at Salem Unit 2, NRR and AEOD senior management and
I have determined that an Augmented Inspection Team (Afl) inspection should be conducted to
review the causes, safety implications, and associated licensee actions which led to or resulted
in the failure of the control rod system to function as expected to support the startup of Salem
Unit 2; and verify the circumstances and evaluate the significance of this event.
The Division of Reactor Safety (DRS) is assigned the responsibility for the overall conduct of
this Augmented Inspection. William Ruland, Chief, Electrical Section, DRS, is appointed as
Augmented Inspection Team Leader (Other AIT members are identified in Enclosure 2). The
Division of Reactor Projects (DRP) is assigned the responsibility for resident and clerical
support, as necessary; and the coordination with other NRC offices, as appropriate. Further,
the Division of Reactor Safety, in coordination with DRP, is responsible for the timely issuance
of the inspection report, the identif1cation and processing of potentially generic issues, and the
identification and completion of any enforcement action warranted as a result of the team's
review.
Enclosure 1 represents the charter for the Augmented Inspection Team and details the scope of
the inspection.
The inspection shall be conducted in accordance with NRC Management
Directive (MD) 8.3, NRC Inspection Manual 0325, Inspection Procedure 93800, Regional Office
Instruction 1010.1, and this memorandum.
This Augmented Inspection Team is being established and chartered in response to repeated
failures of the Rod Control System to function as expected and required during the startup of
Salem Unit 2 between May 25 and June 4, 1993.
Based on available information and
preliminary review by the licensee and their vendor, Westinghouse, it is not evident that the root
cause and circumstances leading to these repeated malfunctions have been sufficiently understood
or analyzed. Further, there is the potential that the licensee's own troubleshooting activities may
have caused or exacerbated the malfunctions. Additionally, there is also a possibility, as
demonstrated by one of the failures, that a single failure could result in the inadvertent and
unexpected withdrawal of a single control rod cluster assembly when an insert demand signal
is present. This conflicts with the design basis of the specific facility and may have safety
implications involving other Westinghouse designed facilities that use similar rod control
systems.
2
An AIT to review this matter is appropriate since the event involves possible adverse generic
implications; and is complicated, difficult to understand, and has an unknown probable cause.
Enclosures:
1.
Augmented Inspection Team Charter
2.
Team Membership
cc w/encls:
J. Taylor, EDO
J. Sniezek, OEDO
T. Murley, NRR
J. Partlow, NRR
W. Russell, NRR
J. Calvo, NRR
- c. Rossi, NRR
F. Miraglia, NRR
C. Berlinger, NRR
J. Wermeil, NRR
J. Richardson, NRR
A. Thadani, NRR
B. Grimes, NRR
B. Boger, NRR
E. Jordan, AEOD
D. Ross, AEOD
L. Wheeler, OEDO
J. Larkins, ACRS
J. Lieberman, OE
W. Kane, DRA, RI
C. Hehl, DRP, RI
J. Wiggins, DRP, RI
J. White, DRP, RI
. R. Cooper, DRSS, RI
S. Shankman, DRSS, RI
T. Johnson, SRI, Salem/Hope Creek
W. Ruland, DRS, RI
J. Durr, DRS, RI
W. Hodges, DRS, RI
L. Bettenhausen, DRS, RI
E. Wenzinger, DRP, RI
K. Abraham, PAO, RI
M. Miller, SW, RI
~~?
Thomas T. Martin
Regional Administrator
ENCWSUREl
Salem Generatin~ Station. Unit 2
Rcxl Control System Failures
Au~mented InS£>eci:ion Te.am <Am Charter
The general objectives of this AIT are to:
1.
Determine the specific circumstances and causes of e.ach individual rod control system
component failure.
2.
Based on the evaluations of the specific component failures, determine the root causes
of the multiple rod control system failures.
3.
Develop a detailed se.quence of events related to the failures.
4.
Determine and evaluate any changes made in the design, maintenance, testing, or
operation of the rod control system prior to the failures.
5.
Assess the safety significance of the rod control system failures, including any postulated
single failures that could result in a single control rod withdrawal when a control rod
insert signal is present.
6.
Determine the adequacy of Public Service Electric & Gas Company's (PSE&G)
maintenance and troubleshooting practices relative to the rod control system, including
vendor interface and control.
7.
Evaluate PSE&G's decision-making concerning attempted plant restarts after the failures
and repairs.
8.
Identify any potential generic implications posed by the rod control problems experienced
at Salem.
9.
Prepare a report documenting the results of this review for signature by the Regional
Administrator within 30 days of the completion of the inspection.
ENCWSURE2
Salem AIT Membership
William Ruland, AIT Leader, Section Chief, Division of Reactor Safety (DRS), Region I (RI)
M. Eugene Lu.arowitz, Reactor Engineer, DRS, RI
Steve Barr, Resident Inspector, Salem, DRP, RI
Hukam C. Garg, NRR
Qbserver
Dave Vann, New Jersey Bureau of Nuclear Engineering
Other NRC personnel, consultants, or contractors will be engaged in this AIT, as needed.
)
ATTACHMEN,J 2
)
'
LOGIC CABINET AUTOMATIC AND MANUAL
Rod Barile
S1l..ctor Swilch
rf
Min..
~.CSA
seo..
...cge
SBC~ Q ,.tee
sea*
- C8D
SBA*
' I
I
Ccwilro4 &
--
SDRCICIBril
~
-
Alarm -
Cktl
n
hn Urgtn*I
Flitufe
~
- ...J
I Alarm I
~ ' , ,
,,..,.,
~
sco
- r
Logic ~
I low~ I
!mi
Bank
Sel*ctor
-
SID Bank - Al + A2
Bl + B2
C + D
Control Bank - Al + A2
Bl + B2
Cl + C2
e91 + D2
'
In
--
Oul
~ .
In Out
.
, t
I *
Supemaor
Circuit
- ~ i*
Pulaer *
Slave
'
Pow*r
Cab
2SCD
SID C + D
Speed
Supervisor
I
Ci'cult In/Ou! I
-
Pulser - 48-432
1
' Pulstt
Slave
In/Out
I '
Select
.
MHltr
-
-
C~cJer
~
In/Ou'
I
coo
Oo
-
Counter
Mulliple1e
~c
--
Pulles
I
In/Out
'
, ' L. Slave
Stave
Cycler
Cyder
1BD
Cooent
Orders
1 *
1 Ir
Power t.J1ower
Cab
2.JAC
Cont Al
Cont Cl
SID Al
Cab
2.IBO
Cont Bl
Cont Dl
SID Bl
-
......
-
L. Sl~;n.
~ .
Stave
Cycler
Cycler
2AC
I t
Power
Cab
2.2AC
Cont A2
Cont C2
SID A2
2BD
Multipfeic
I t
-
~ I
Power
Cab
2280
Cont B2
Cont D2
SID B2
'
1 . .
Bank
Overtap
Unit
-
C31oup
Select
. ,
LOGIC
CABINET
'
t
POWER
CABINET
ATTACHMENT 3
Seguence of Events
Note: This sequence of events was developed from information collected during the course of
the inspection through interviews with PSE&G and contractor personnel, reviews of
documents and information provided by the SERT.
Mq,rch 1~. 1993
Unit 2 reactor trip occurred and the refueling outage commenced.
March 18, 1993
Westinghouse letter sent to PSE&G specifying terms of offer for rod control system
preventive maintenance work. PSE&G subsequently contracted Westinghouse to
perform work.
March 26, 1993
Westinghouse commenced circuit board inspection as part of the preventive
maintenance work on the rod control system.
April 6, 1993
Inspection, repair and bench test of logic cabinet and power cabinet circuit cards was
completed. System logic tested satisfactorily. Rod control system motor-generator
set (MG) preventive maintenance commenced.
April 10, 1993
Westinghouse technicians left Salem site when their work was delayed due to power
availability problems for the rod control systems. Printed circuit card testers were
removed from site for use at another facility.
May JO, 1993
Westinghouse _technicians returned to the Salem site t.o resume rod control system
functional testing.
May 14, 1993
Group step counters SDA-1 and CBA-1 failed their functional test. Counters CBB-1
and CBC-1 marginally passed the test.
May 16, 1993
Group step counters SDA-1, CBA-1, CBB-1 and CBC-1 were replaced with one
counter obtained from Salem spare parts inventory and three counters obtained from
the Turkey Point plant spares. During the retest of the counters several circuit card
components were found to be failed and were replaced.
May 20, 1993
Westinghouse circuit card tester returned to Salem site.
May 24, 1993
MG set testing was completed and individual rod position indicator (ARPI)
calibrations and rod drop testing commenced. CBA Group 1 did not move and an
Urgent Failure alarm was reeeived. CBC Group 1 counter responded to in rod
motion but not out. A short circuit at an indicating light was repaired and a circuit
card required replacement.
May 25, 1993
During ARPI calibrations, CBB Group 1 and 2 counters did not operate properly and
were replaced with two of the old counters which had been removed on
May 16, 1993. ARPI calibrations and rod drop testing was completed and a reactor
startup commenced.
May 26, 1993
Salem operators completed withdrawal _of all shutdown bank control rods to 225 steps
and commenced control bank rod withdrawal. CBC Group 1 step counter stopped
advancing at 31 steps. All control rods were inserted. Several failed integrated
circuits (IC) and diodes were identified. Numerous card repairs and replacements
were performed to correct problems.
May 27, 1993
Following repairs to circuit cards, the control rod group counters were tested by
performing a 228 step withdrawal and insertion without latching the control rods.
During a subsequent startup attempt, SBA was withdrawn to 20 steps as indicated on
the group counter with no corresponding response observed on the ARPI indicators.
During rod insertion, the ARPI indication for control rod 1SA3 indicated the rod had
withdrawn to 15 steps. The fuses for the l_SA3 control rod stationary gripper coil
were removed and the ARPI position indication went to zero.
May 28, 1993
Troubleshooting determined that failures on the circuit cards resulted in the movable
gripper coil and lift coils being energized simultaneously. Additional circuit board
problems were identified and repaired.
PSE&G personnel conferred with Westinghouse representatives via phone and were
advised that control rod 1SA3 *would not have been damaged during the improper
operation on May 27, 1993.
Rod control system testing was performed satisfactorily by performing a 20 step
withdrawal and insertion for each rod control bank, and a reactor startup commenced.
The control rods were withdrawn without incident.
While diluting the reactor coolant boron concentration to achieve criticality, all four
CBC Group 1 control rods dropped to the fully inserted position. The remaining
control rods were then manually tripped by the reactor operator as required by plant
procedures.
May 29-30, 1993
Troubleshooting was performed, but the cause of the dropped rods could not be
determined. Four circuit cards whose failure could have caused the dropped rods
were replaced.
During troubleshooting, the 100 VDC and -15 VDC power supplies were apparently
short circuited, resulting in blown fuses in the 100 VDC supply and a failed
auctioneering diode in the -15 VDC supply. The fuses and failed diode were
replaced. A transistor was found failed and replaced as well as failed components on
several other circuit cards.
May 31 and June 1, 1993
Additional group step counter problems continued to occur during troubleshooting and
testing. Numerous circuit card problems were identified and repaired.
June 2, 1993
Rod control system testing was performed, including counter operation, current order
and ARPI checks. Testing was completed satisfactorily.
Reactor startup commenced. Pulse-to-analog converter discovered to be failed when
Rod Insertion Limit annunciator failed to clear during CBD rod withdrawal.
June 3, 1993
The shift supervisor directed all rods to be inserted while troubleshooting was
performed. Faulty IC chips on data logging card were identified and replaced. The
system was retested satisfactorily and a reactor startup commenced. Criticality was
achieved at 3:05 p.m.
NRC Region I inspector arrives on site.
- June 4, 1993
Plant management decided to shut the plant down when it became evident that the
improper rod motion on rod 1SA3, encountered on May 27, 1993, may have been the
result of a single failure, resulting in operations outside the plant design basis.
June 5, 1993
NRC Augmented Inspection Team (AIT) arrived on site to investigate rod control
system problems.
June 8, 1993
The NRC issued a Confirmatory Action Letter (CAL) to PSE&G which affirmed the
licensee's commitment to 1) maintain Unit 2 shut down until the rod control system .
problems were investigated and satisfactorily resolved and 2) to prepare a Justification
for Continued Operation (JCO) for Unit 1 to documel)t their technical basis for
assuring that Unit 1 will continue to operate safely.
Unit 1 tripped due to loss of circ.ulating water and was maintained in hot standby
while the Unit 2 rod control system problems continued to be investigated.
June 17, 1993
AIT completed its initial on site inspection.
June 18, 1993
PSE&G met with NRC Region I management to present the results of their
investigation of the rod control system failures and planned corrective actions.
Unit 1 startup commenced following the addition of surge suppression diodes on the
group step counters.
June 19, 1993
Unit 1 achieved criticality. Rod control system operation normal.
June 20 and 21, 1993
PSE&G installed new circuit cards in logic cabinet, installed surge suppression diodes
on the group step counters, inspected and repaired all power cabinet circuit cards.
June 22, 1993
Rod control system testing commenced which required jumper installation and
removal in the power cabinets.
June 23, 1993
Transistor failures discovered on the signal processing and alarms cards in the SCD
power cabinet. Transistors replaced and system testing completed.
Region I NRC inspector arrived on site to review problem with failed transistors and
to observe current trace tests.
June 24, 1993
Current traces completed satisfactorily. Root cause evaluation for failed transistors
continued.
June 25, 1993
Thirteen circuit cards were replaced in the SCD power cabinet and system testing
commenced. An Urgent Failure alarm occurred while attempting to record current
traces with dummy coils installed. Testing stopped and troubleshooting deferred until
the following morning.
June 26, 1993
Troubleshooting was performed and the cause of the Urgent Alarm found to be an
incorrect resistor installed on one of the new sen cabinet circuit cards.
June 27, 1993
System testing and current traces were completed satisfactorily.
CAL actions complete.
June 28, 1993
AIT inspection completed.
Unit 2 reactor startup commenced, control rods functioned normally.
-.
May 16
May24
May 26
May 27
May 31
June 1
ATIACHMENT 4
CIRCUIT CARD LOCATIONS EXPERIENCING MQLTIPLE FAILURES
SUPERVISORY DATA LOGGING CARD CA113 Location)
Replaced card Westinghouse Serial Number (WSN) 1S3 with WSN 217
Repaired card WSN 217 by replacing integrated circuit chips ZS, Z9
and Z12
Replaced card WSN 217 with WSN 6014
Replaced card WSN 6014 with WSN 039
Reinstalled card WSN 6014 after replacing Z3 and Z6
Replaced Z2, ZS and ZS on card WSN 6014
Installed card WSN 039 after replacing Z2, Z3, and Z8
Repaired card. WSN 0039 -- replaced Z2
Card WSN 039 failed again -- replaced Z2 and Z3
Repaired card WSN 6014 (replaced Z2, Z5 and Z6) and then tested
WSN6014 and WSN 039 at the training center. Replaced one chip on
WSN6014
Installed card WSN 0039
Replaced the card with WSN 6014 (After replacing Z3 and Z6)
Repaired WSN ~14 -- replaced Z3 and Z5
Repaired WSN 6014 -- replaced Z3
Repaired WSN 6014 -- replaced Z3
Repaired WSN 6014 -- replaced Z3
SUPERVISORY DATA LOGGING CARD (Al 14 Location)
May 16
Repaired card WSN 216 -- replaced Z3
May 27
Replaced card WSN 216 with WSN 0039
Repaired and reinstalled card WSN 0039 -- replaced Z3
Replaced the card with WSN 216
May 31
Replaced the card with WSN 0183 (Z3 failed)
Repaired WSN 0183 -- replaced Z3
June 1
Repaired WSN 0183 -- replaced Z3
June 3
Repaired WSN 0183 -- replaced Z3
May 16
May26
May27
May 16
May26
May 27
- * *
IN/OUT RELAY DRIVER CARD (A 713 Location)
Replaced card WSN 132 with WSN 120
Swapped with card WSN 0133 from the A714 location
Replaced the card with WSN 701
Installed card WSN 0120
Installed card WSN 0345
Swapped with the card in A714 location
Cage pin 34 spread open -- closed pin
IN/OUT RELAY DRIVER CARD (A714 Location)
Replaced card WSN 139 with WSN 133
Swapped with card WSN 120 from A713 location
Replaced the card with WSN 695
Replaced the card with WSN 681
Replaced the card with WSN 695
Restored with card WSN 681
Installed card WSN 0342
Swapped with the card in A713 location
Reinstalled card WSN 0342
ATTACHMENT 5
LIST OF FAILURES/CAUSE
DATE/TIME/EVENT
FAILURE
CORRECTIVE ACTION
CAUSE
May 14
Group counters:
Replaced with one with
Normal wear
spare and three from
Preventive Maintenance
CBA-GRl
Turkey Point
CBB-GRl
CBC-GRl
SBA-GR2
May 16
Al 13 (Note l)
Replaced card
Electrical overstress (The
new counters and loss of
Counter Testing
A114
Replaced Z3 (Note 2)
suppression circuitry resulted
in solid state components
A113
Replaced Z8, Z9 & Zl2
being subjected to voltage
spikes in excess of their
A713
Replaced card
design rating)
A714
Replaced card
May24
Resistor in 22AC
Replaced resistor and
Short circuit caused by
Power cabinet
light socket
installation of the wrong type
Operations Surveillance
of light socket during
preventive maintenance.
A113
Replaced card
Electrical overstress
May 25
Group counters
Replaced with counters
Transistor saturation due to
CBB-GRl and
previously removed on
loss of suppression circuit
ARPI Calibration
CBB-GR2
May 16
May 26
A714
Replaced A714 card with
Electrical overstress
spare
2:00 a.m.
Plant Startup
DATE/TIME/EVENT
FAILURE
CORRECTIVE ACTION
CAUSE
May 26
Al13
Replaced card
Electrical overstress
8:00 a.m.
A713
Replaced the cards
A714
Troubleshooting
A113
Replaced the card
A113
Replaced Z2,Z5,Z8
A113
Replaced Z2
Al13
Replaced Z2,Z3
May 26
Al13
Replaced Z2,Z3
Electrical overstress
11:50 p.m.
Troubleshooting
May 27
A713
Swapped A713 and
Spread pin in A 713 card
A 714, corrected spread
cage
5:00 a.m.
pin
Troubleshooting
A113
Replaced with spare
Electrical overstress
May 27
Al14
Replaced with spare card
Electrical overstress
then replaced Z3 when
8:40 a.m.
problem occurred again
Troubleshooting
A113
Replaced Z3, Z6
May 27
A710
Replaced diode
Electrical overstress
1:05 p.m.
Troubleshooting
May 27
A114
Replaced with Spare
Electrical overstress
14:30
Troubleshooting
DATE/TIME/EVENT
FAILURE
CORRECTIVE ACTION
CAUSE
May 27
Rod 1SA3 moved
Replaced with new cards
Electrical overstress
15 steps up while
18:44
other stayed in
bottom when A511
Reactor Startup
and A501 failed
May 28
Control bank C -
Replaced firing, phase,
Short circuits on regulation
Group 1 dropped
regulation and 1/0 ac amp
card caused by slivers of
6:12 p.m.
cards associated with this
solder run
rod group.
Reactor Startup
May 30
Fuses FUll and
Replaced fuses
Technicians caused a short
FU6 blown on 100
circuit while installing test
3:00 a.m.
VDC power supply
equipment
Troubleshooting
-15 VDC
Replaced diode
Auctioneering diode
May 30
A803
Swapped card from A805
Technicians shorted 100
slot
VDC power to -15 VDC
. 4:30 p.m.
during test equipment
installation (Note 3)
Troubleshooting
May 30
Failed bench test:
Cards repaired
100 VDC to -15 VDC short
circuit (Note 3)
8:40 p.m.
A808
A812
Troubleshooting
A813
A814
May 31
Failed bench test:
Replaced transistor Q12
100 VDC to -15 VDC short
circuit (Note 3)
2:45 a.m.
A809
Troubleshooting
May 31
MXR2 Relay
Found and corrected
Spread pin in 22BD cabinet
spread pin
8:40 a.m.
Troubleshooting
DATE/TIME/EVENT
FAILURE
CORRECTIVE ACTION
CAUSE
May 31
Failed bench test:
10:45 a.m.
A114
Replaced card (Z3 failed
Electrical overstress
however card not
Troubleshooting
repaired due to lifted
trace)
Al13
Replaced Z3 and Z5
May 31
A113
Replaced Z3
Electrical overstress
1:00 p.m.
Al14
Replaced Z3
Troubleshooting
May 31
Al13
Replaced Z3
Electrical overstress
3:10 p.m.
Troubleshooting
May 31
A514
Replaced with spare
100 VDC to -15 VDC short
circuit (Note 3)
9:35 p.m.
Troubleshooting
June 1
A113
Replaced Z3
Electrical overstress
6:05 a.m.
Al14
Replaced Z3
Troubleshooting
June 1
A207
Replaced with spare
Electrical overstress
7:15 p.m.
Troubleshooting
June 3
A114
Replaced Z3
Electrical Overstress
Reactor Startup
DATE/TIME/EVENT
FAILURE
CORRECTIVE ACTION
CAUSE
June 23
SCD cabinet signal
Replaced Q9 Transistor
Technician caused short
processing and
on each card
circuit during installation of
12:30 a.m.
alarm cards
jumper
System Retest
June 25
Failure detector
Replaced card
Wrong value Rl8 resistor
card in power
installed at time of
10:40 p.m.
cabinet SCD
manufacture (50 kn versus
511 kn)
SCD Cabinet Retest
NOTE 1:
NOTE2:
NOTE 3:
Failures designated AXXX refer to the failure of a printed circuit card in associated with a
specific logic cabinet card location:
Al 13 - Supervisory Data Logging
Al14 - Supervisory Data Logging
A207 - Bank Overlap Logic
A501 - Slave Cycler Decoder
A511 - Slave Cycler Decoder
A514 - Slave Cycler Logic
A 710 - In/Out Relay Driver
A 713 - In/Out Relay Driver
A803 - In/Out AC Amplifier
A808 - In/Out AC Amplifier
A809 - In/Out Receiver
A812 - In/Out AC Amplifier
A813 - In/Out AC Amplifier
A814 - In/Out AC Amplifier
Replacement of components designated ZX refer to the replacement of a particular type of
integrated circuit on a printed circuit card.
While this is the most likely cause for these failures electrical overstress could not be
completely eliminated as a possible cause.
- --
ATTACHMENT 6
NRC AUGMENTED INSPECTION TEAM
EXIT MEETING
SALEM UNIT 2 ROD CONTROL SYSTEM PROBLEMS
ARTIFICIAL ISLAND PROCESSING CENTER
JULY 7, 1993, 10:00 A.M.
AGENDA
I.
INTRODUCTIONS
M. WAYNE HODGES
II.
PURPOSE OF INSPECTION
M. WAYNE HODGES
III. SEQUENCE OF EVENTS
WILLIAM H. RULAND
IV.
SCOPE OF REVIEW
V.
MOST PROBABLE CAUSES
VI.
PRELIMINARY FINDINGS
VII. SAFETY SIGNIFICANCE
VIII. CONCLUDING NRC REMARKS
M. WAYNE HODGES
IX.
PSE&G REMARKS
STEVEN MILTENBERGER
III. SEQUENCE OF EVENTS
UNIT 2 OUTAGE STARTED MARCH 16. WESTINGHOUSE
CONTRACTED FOR ROD CONTROL SYSTEM PREVENTIVE
MAINTENANCE.
PROBLEMS ENCOUNTERED WITH ROD GROUP POSITION
INDICATORS - REPAIRS. MADE.
IIL SEQUENCE OF EVENTS (CON'T)
MAY 25 - GROUP COUNTER STOPPED COUNTING DURING
ROD MOTION FOR START-UP. REPAIRS MADE.
MAY 27 - ONE ROD NOTED MOVING OUT WITH IN MOTION
REQUESTED. REPAIRS MADE.
MAY 28 - ALL FOUR BANK C, GROUP 1, RODS DROPPED
INTO THE CORE. PSE&G INVESTIGATION STARTED.
REPAIRS MADE, ROOT CAUSE(S) IDENTIFIED BY PSE&G,
TESTING COMPLETED.
JUNE 1 - ROD INSERTION LIMITS NOT RECORDING.
ATTRIBUTED TO INCOMPLETE
TESTING/TROUBLESHOOTING. REPAIRS MADE.
JUNE 3 - STARTUP STOPPED DUE TO SINGLE FAILURE
QUESTIONS.
III.
SEQUENCE OF EVENTS (CON'T)
AIT SENT ON JUNE 5 TO REVIEW EVENTS.
CONFIRMATORY ACTION LETTER ISSUED TO PSE&G ON
JUNE 8.
PSE&G/NRC MANAGEMENT MTG ON JUNE 18.
JUNE 23 A.M. - AFfER CARD REPLACEMENT, FOUND TWO
CIRCIDT CARDS WITH DAMAGED TRANSISTORS. REPAIRED
AND TESTED.
JUNE 25 - URGENT FAILURE ALARM. FOUND CARD WITH
INCORRECT COMPONENT (RESISTOR).
CLOSEOUT OF CONFIRMATORY ACTION LETTER ON JULY 1.
-
JULY 3 - GENERATOR BREAKER CLOSED, OUTAGF.; ENpS.
IV.
SCOPE OF REVIEW
TEAM COMPOSITION - s*MEMBERS - IN EXCESS OF 400
HOURS DIRECT INSPECTION EFFORT
INTERVIEWED MAINTENANCE PERSONNEL,
ENGINEERS, EQUIPMENT VENDOR, AND PSE&G
MANAGEMENT
OBSERVED TROUBLESHOOTING ACTIVITIES, ROD
CONTROL SYSTEM TESTING, AND IMPLEl\\IBNTATION
OF COMPENSATORY l\\IBASURES
OBSERVED OPERATION OF THE ROD CONTROL
SYSTEM SIMULATOR IN THE PSE&G TRAINING CENTER
REVIEWED CHANGES MADE TO THE ROD CONTROL
SYSTEM-
REVIEWED PSE&G RECORDS MADE DURING AND
AFTER THE EVENTS, INCLUDING DETAILED
DOCUMENTATION OF COMPONENT FAILURES
V.
MOST PROBABLE CAUSES
- voLTAGE SPIKES FROM REPLACEMENT COUNTERS WITH
AN OPEN CIRCUIT TO A SUPPRESSION DIODE REPEATEDLY
DAMAGED SEMICONDUCTOR DEVICES ON PRINTED
CIRCUIT (PC) CARDS.
DAMAGED OVER 30 COMPONENTS.
LED TO THE ABORTED STARTUPS ON MAY 25, MAY 27
ANDJUNE2.
LED TO THE DISCOVERY OF A POTENTIAL SINGLE
FAILURE VULNERABILITY AND POSSIBLE FAILURE TO
MEET GENERAL DESIGN CRITERION 25 (PROTECTION
SYSTEM REQUIREMENTS FOR REACTIVITY CONTROL
SYSTEMS).
- ~
-: .. ,
,,
V.
MOST PROBABLE CAUSES (CON'T)
CONDUCTIVE WIIlSKER SHORTED PC CARD TRACES
TOGETHER, LEADING TO DROPPED RODS*(MAY 28).
JUMPER REMOVAL SHORTED* CONNECTOR PINS AND PC
CARD TRACES TOGETHER, DAMAGING TRANSISTORS
(JUNE 23).
WRONG VALUE RESISTOR FOUND ON PC CARD (JUNE 25).
VI. PRELIMINARY FINDINGS
- *
DESIGN WEAKNESS - DIODE TO * -*
PROTECT CIRCUITS SUBJECT TO A
MECHANICAL FAILURE OF
CONNECTOR.
APPLICATION OF ROD CONTROL
SYSTEM DESIGN BASIS KNOWLEDGE
INADEQUATE.
CAUSE DETERMINATION POLICY AND
EXPECTATIONS NOT ESTABLISHED.
-. ~-
- .
NO CLEAR CHAIN OF COMMAND AND
RESPONSIBILITIES DURING
TROUBLESHOOTING ACTIVITIES.
VII.
SAFETY SIGNIFICANCE
SINGLE FAILURE - HAD NO ANALYSIS
TO SUPPORT COMPLIANCE WITH THE
GENERAL DESIGN CRITERIA FOR ROD
CONTROL SYSTEMS. ISSUE GENERIC
TO MOST WESTINGHOUSE NUCLEAR
PLANTS.
ROD CONTROL SYSTEM FAILURES
COULD INITIATE TRANSIENTS.
ACTUAL EVENT DAMAGED NON-
SAFETY EQUIPMENT ONLY. REACTOR
PROTECTION SYSTEM UNAFFECTED.
-- ~-
~
NRC ISSUED GENERIC LETTER 93-04 TO*
ALERT LICENSEES AND REQUIRE
REPORTS OF APPLICABILITY AND
ACTIONS PLANNED.
PSE&G
Alfieri, J.
Bachman, M.
Burricelli, R.
Burzstein, M.
Butz, M.
Camp, M.
Chranoski, R.
Cohen, S.
Fogelson, S.
Hagan, J.
LaBruna, S.
Lambert, C.
LaFevre, M.
Mannon, S.
McTigue, W.
Miller, L.
Miltenberger, S.
O'Donnell, P.
Rajkowski, L.
. Shedlock, M.
Thomson, F.
Vondra, C.
Wray, J.
ATTACHMENT 7
Exit Meeting Attendees
Barr, s.
Garg, H.
Hodges, W.
Johnson, T.
Lazarowitz, M.
Ruland, W.
Scholl, L.
Schoppy, J.
Stone, J.
White, J.
Zimmerman, J.
Baird, T.
Baker, T.'
Huckabee, J.
Others
Crowe, P. (Salem Today's Sumbeam)
Duca, P. (Delmarva Power)
Lang, M. (Salem Today's Sunbeam)
Milford, P. (Wilmington News Journal)
Oakes, R. (Atlantic Electric)
Robb, J. (Philadelphia Electric)
Ryan, G. (Radio Station WilC)
Vann, D. (State of New Jersey DEPE/BNE)
Yount, M. (Radio Station WILM)
r
ATTACHMENT 8
Persons Contacted
Public Service Electric and Gas Company
Amicucci, J. I&C Supervisor
Bailey, J., Nuclear Engineering Sciences Manager
Best, D., System Engineer
Budzik, D., Maintenance Engineer
Bursztein, M., Nuclear Electrical Engineering Manager
Carrier, T., Maintenance Engineer
Chranowsk:i, R., Technical Engineer
Cianfrani, W., Onsite Safety Review Engineer
Gallager, R., Operations Engineer
Hagan, J., Vice President - Nuclear Operations
Rambert, C., Manager - Nuclear Engineering Design
Heaton, R., I&C Supervisor
Karimian, S., Technical Consultant
Kent, R., Nuclear Fuels Engineer
LaBruna, S., Vice President - Nuclear Engineering
Lowry, W. D., System Engineer
Miller, L., SERT Manager
Miltenberger, S., Vice President and Chief Nuclear Officer
Panko, M., Maintenance Engineer
Pastva, J., LER Coordinator
Pike, K., Technical Manager (Acting)
Rajkowski, L., I&C Supervisor (Acting) E&PB
Robinson, E., Nuclear Training Department
Ronafalny, J., SERT Member
Shedlock, M., Maintenance Manager
Thomas, B., Licensing Engineer
Thompson, F., Manager - Licensing and Regulation
Villar, E., Licensing Engineer
Vondra, C., General Manager - Salem Operations
Baker, T., Nuclear Safety Licensing Engineer
King, T., Field Service Technician
Kisic, J., Field Service Technician
Huckabee, J., Salem Site Representative
Pysnick, J., Rod Control System Engineer
Katz, D., Design Engineer
GDC
GM-SO
IRPI
JCO
LER
OHA
P-250
PC
PSE&G
RCCA
RD
SCD
SDL
SERT
SNSS
SORC
TS
ATI'ACHMENT 9
Acronyms and Initialisms
. Augmented Inspection Team
Confirmatory Action Letter
Code of Federal Regulations
Control Rod Drive Mechanism
Design Change Package
Departure From Nucleate Boiling Ratio
Electromotive Force
General Design Criteria
General Manager - Salem Operator
Instrumentation and Control
Individual Rod Position Indication
Justification for Continued Operation
Licensee Event Report
Nuclear Shift Supervisor
Overhead Annunciator System
Plant Process Computer
Preventive Maintenance
Printed Circuit
Public Service Electric and Gas
Rod Control Cluster Assembly
Rod Control System
Relay Driver
Reactor Protection
Slave Cycle Decoder
Supervisory Data Logger
Safety Evaluation
Sequential Events Recorder
Significant Event Response Team
Senior Nuclear Shift Supel'Visor
Site Operating Review* Committee
Significant Operating Event Report
Technical Specification
Updated Final Safety Analysis Report
Unreviewed Safety Question
Work Order